* Kernel 4.19 network performance - forwarding/routing normal users traffic
@ 2018-10-31 21:57 Paweł Staszewski
2018-10-31 22:09 ` Eric Dumazet
` (2 more replies)
0 siblings, 3 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-10-31 21:57 UTC (permalink / raw)
To: netdev
Hi
So maybee someone will be interested how linux kernel handles normal
traffic (not pktgen :) )
Server HW configuration:
CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
Server software:
FRR - as routing daemon
enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa
node)
enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
Maximum traffic that server can handle:
Bandwidth
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
\ iface Rx Tx Total
==============================================================================
enp175s0f1: 28.51 Gb/s 37.24 Gb/s
65.74 Gb/s
enp175s0f0: 38.07 Gb/s 28.44 Gb/s
66.51 Gb/s
------------------------------------------------------------------------------
total: 66.58 Gb/s 65.67 Gb/s
132.25 Gb/s
Packets per second:
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
- iface Rx Tx Total
==============================================================================
enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
------------------------------------------------------------------------------
total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
After reaching that limits nics on the upstream side (more RX traffic)
start to drop packets
I just dont understand that server can't handle more bandwidth
(~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
side are increasing.
Was thinking that maybee reached some pcie x16 limit - but x16 8GT is
126Gbit - and also when testing with pktgen i can reach more bw and pps
(like 4x more comparing to normal internet traffic)
And wondering if there is something that can be improved here.
Some more informations / counters / stats and perf top below:
Perf top flame graph:
https://uploadfiles.io/7zo6u
System configuration(long):
cat /sys/devices/system/node/node1/cpulist
14-27,42-55
cat /sys/class/net/enp175s0f0/device/numa_node
1
cat /sys/class/net/enp175s0f1/device/numa_node
1
ip -s -d link ls dev enp175s0f0
6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
UP mode DEFAULT group default qlen 8192
link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity 0
addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
gso_max_segs 65535
RX: bytes packets errors dropped overrun mcast
184142375840858 141347715974 2 2806325 0 85050528
TX: bytes packets errors dropped carrier collsns
99270697277430 172227994003 0 0 0 0
ip -s -d link ls dev enp175s0f1
7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
UP mode DEFAULT group default qlen 8192
link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0
addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
gso_max_segs 65535
RX: bytes packets errors dropped overrun mcast
99686284170801 173507590134 61 669685 0 100304421
TX: bytes packets errors dropped carrier collsns
184435107970545 142383178304 0 0 0 0
./softnet.sh
cpu total dropped squeezed collision rps flow_limit
0 3961392822 0 1221478 0 0 0
1 3701952251 0 1258234 0 0 0
2 3879522030 0 1584282 0 0 0
3 3731349789 0 1529029 0 0 0
4 1323956701 0 2176371 0 0 0
5 420528963 0 1880146 0 0 0
6 348720322 0 1830142 0 0 0
7 372736328 0 1820891 0 0 0
8 567888751 0 1414763 0 0 0
9 476075775 0 1868150 0 0 0
10 468946725 0 1841428 0 0 0
11 676591958 0 1900160 0 0 0
12 346803472 0 1834600 0 0 0
13 457960872 0 1874529 0 0 0
14 1990279665 0 4699000 0 0 0
15 1211873601 0 4541281 0 0 0
16 1123871928 0 4544712 0 0 0
17 1014957263 0 4152355 0 0 0
18 2603779724 0 4593869 0 0 0
19 2181924054 0 4930618 0 0 0
20 2273502182 0 4894627 0 0 0
21 2232030947 0 4860048 0 0 0
22 2203555394 0 4603830 0 0 0
23 2194756800 0 4921294 0 0 0
24 2347158294 0 4818354 0 0 0
25 2291097883 0 4744469 0 0 0
26 2206945011 0 4836483 0 0 0
27 2318530217 0 4917617 0 0 0
28 512797543 0 1895200 0 0 0
29 597279474 0 1532134 0 0 0
30 475317503 0 1451523 0 0 0
31 499172796 0 1901207 0 0 0
32 493874745 0 1915382 0 0 0
33 296056288 0 1865535 0 0 0
34 3905097041 0 1580822 0 0 0
35 3905112345 0 1536105 0 0 0
36 3900358950 0 1166319 0 0 0
37 3940978093 0 1600219 0 0 0
38 3878632215 0 1180389 0 0 0
39 3814804736 0 1584925 0 0 0
40 4152934337 0 1663660 0 0 0
41 3855273904 0 1552219 0 0 0
42 2319538182 0 4884480 0 0 0
43 2448606991 0 4387456 0 0 0
44 1436136753 0 4485073 0 0 0
45 1200500141 0 4537284 0 0 0
46 1307799923 0 4534156 0 0 0
47 1586575293 0 4272997 0 0 0
48 3852574 0 4162653 0 0 0
49 391449390 0 3935202 0 0 0
50 791388200 0 4290738 0 0 0
51 127107573 0 3907750 0 0 0
52 115622148 0 4012843 0 0 0
53 71098871 0 4200625 0 0 0
54 305121466 0 4365614 0 0 0
55 10914257 0 4369426 0 0 0
PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
26.78% [kernel] [k] queued_spin_lock_slowpath
9.09% [kernel] [k] mlx5e_skb_from_cqe_linear
4.94% [kernel] [k] mlx5e_sq_xmit
3.63% [kernel] [k] memcpy_erms
3.30% [kernel] [k] fib_table_lookup
3.26% [kernel] [k] build_skb
2.41% [kernel] [k] mlx5e_poll_tx_cq
2.11% [kernel] [k] get_page_from_freelist
1.51% [kernel] [k] vlan_do_receive
1.51% [kernel] [k] _raw_spin_lock
1.43% [kernel] [k] __dev_queue_xmit
1.41% [kernel] [k] dev_gro_receive
1.34% [kernel] [k] mlx5e_poll_rx_cq
1.26% [kernel] [k] tcp_gro_receive
1.21% [kernel] [k] free_one_page
1.13% [kernel] [k] swiotlb_map_page
1.13% [kernel] [k] mlx5e_post_rx_wqes
1.05% [kernel] [k] pfifo_fast_dequeue
1.05% [kernel] [k] mlx5e_handle_rx_cqe
1.03% [kernel] [k] ip_finish_output2
1.02% [kernel] [k] ipt_do_table
0.96% [kernel] [k] inet_gro_receive
0.91% [kernel] [k] mlx5_eq_int
0.88% [kernel] [k] __slab_free.isra.79
0.86% [kernel] [k] __build_skb
0.84% [kernel] [k] page_frag_free
0.76% [kernel] [k] skb_release_data
0.75% [kernel] [k] __netif_receive_skb_core
0.75% [kernel] [k] irq_entries_start
0.71% [kernel] [k] ip_route_input_rcu
0.65% [kernel] [k] vlan_dev_hard_start_xmit
0.56% [kernel] [k] ip_forward
0.56% [kernel] [k] __memcpy
0.52% [kernel] [k] kmem_cache_alloc
0.52% [kernel] [k] kmem_cache_free_bulk
0.49% [kernel] [k] mlx5e_page_release
0.47% [kernel] [k] netif_skb_features
0.47% [kernel] [k] mlx5e_build_rx_skb
0.47% [kernel] [k] dev_hard_start_xmit
0.43% [kernel] [k] __page_pool_put_page
0.43% [kernel] [k] __netif_schedule
0.43% [kernel] [k] mlx5e_xmit
0.41% [kernel] [k] __qdisc_run
0.41% [kernel] [k] validate_xmit_skb.isra.142
0.41% [kernel] [k] swiotlb_unmap_page
0.40% [kernel] [k] inet_lookup_ifaddr_rcu
0.34% [kernel] [k] ip_rcv_core.isra.20.constprop.25
0.34% [kernel] [k] tcp4_gro_receive
0.29% [kernel] [k] _raw_spin_lock_irqsave
0.29% [kernel] [k] napi_consume_skb
0.29% [kernel] [k] skb_gro_receive
0.29% [kernel] [k] ___slab_alloc.isra.80
0.27% [kernel] [k] eth_type_trans
0.26% [kernel] [k] __free_pages_ok
0.26% [kernel] [k] __get_xps_queue_idx
0.24% [kernel] [k] _raw_spin_trylock
0.23% [kernel] [k] __local_bh_enable_ip
0.22% [kernel] [k] pfifo_fast_enqueue
0.21% [kernel] [k] tasklet_action_common.isra.21
0.21% [kernel] [k] sch_direct_xmit
0.21% [kernel] [k] skb_network_protocol
0.21% [kernel] [k] kmem_cache_free
0.20% [kernel] [k] netdev_pick_tx
0.18% [kernel] [k] napi_gro_complete
0.18% [kernel] [k] __sched_text_start
0.18% [kernel] [k] mlx5e_xdp_handle
0.17% [kernel] [k] ip_finish_output
0.16% [kernel] [k] napi_gro_flush
0.16% [kernel] [k] vlan_passthru_hard_header
0.16% [kernel] [k] skb_segment
0.15% [kernel] [k] __alloc_pages_nodemask
0.15% [kernel] [k] mlx5e_features_check
0.15% [kernel] [k] mlx5e_napi_poll
0.15% [kernel] [k] napi_gro_receive
0.14% [kernel] [k] fib_validate_source
0.14% [kernel] [k] _raw_spin_lock_irq
0.14% [kernel] [k] inet_gro_complete
0.14% [kernel] [k] get_partial_node.isra.78
0.13% [kernel] [k] napi_complete_done
0.13% [kernel] [k] ip_rcv_finish_core.isra.17
0.13% [kernel] [k] cmd_exec
ethtool -S enp175s0f1
NIC statistics:
rx_packets: 173730800927
rx_bytes: 99827422751332
tx_packets: 142532009512
tx_bytes: 184633045911222
tx_tso_packets: 25989113891
tx_tso_bytes: 132933363384458
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 74630239613
tx_nop: 2029817748
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 173730800927
rx_csum_unnecessary: 0
rx_csum_none: 434357
rx_csum_complete: 173730366570
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 38260960853
tx_csum_partial: 36369278774
tx_csum_partial_inner: 0
tx_queue_stopped: 1
tx_queue_dropped: 0
tx_xmit_more: 748638099
tx_recover: 0
tx_cqes: 73881645031
tx_queue_wake: 1
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 14441066823
rx_cache_full: 51126004413
rx_cache_empty: 21297344082
rx_cache_busy: 51127247487
rx_cache_waive: 21298322293
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 24603119858
ch_poll: 25180949074
ch_arm: 24480437587
ch_aff_change: 75
ch_eq_rearm: 0
rx_out_of_buffer: 669685
rx_if_down_packets: 61
rx_vport_unicast_packets: 173731641945
rx_vport_unicast_bytes: 100522745036693
tx_vport_unicast_packets: 142531901313
tx_vport_unicast_bytes: 185189071776429
rx_vport_multicast_packets: 100360886
rx_vport_multicast_bytes: 6639236688
tx_vport_multicast_packets: 32837
tx_vport_multicast_bytes: 2978810
rx_vport_broadcast_packets: 44854
rx_vport_broadcast_bytes: 6313510
tx_vport_broadcast_packets: 72258
tx_vport_broadcast_bytes: 4335480
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 142532004669
rx_packets_phy: 173980375752
rx_crc_errors_phy: 0
tx_bytes_phy: 185759204762903
rx_bytes_phy: 101326109361379
tx_multicast_phy: 32837
tx_broadcast_phy: 72258
rx_multicast_phy: 100360885
rx_broadcast_phy: 44854
rx_in_range_len_errors_phy: 2
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 59
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 148328738
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 36551843112
rx_65_to_127_bytes_phy: 65102131735
rx_128_to_255_bytes_phy: 5755731137
rx_256_to_511_bytes_phy: 2475619839
rx_512_to_1023_bytes_phy: 2826971156
rx_1024_to_1518_bytes_phy: 42474023107
rx_1519_to_2047_bytes_phy: 18794051270
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 48
rx_prio0_bytes: 101316322498995
rx_prio0_packets: 173711151686
tx_prio0_bytes: 185759176566814
tx_prio0_packets: 142531983704
rx_prio1_bytes: 47062768
rx_prio1_packets: 228932
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 12434759
rx_prio2_packets: 83773
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 288843134
rx_prio3_packets: 982102
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 699797236
rx_prio4_packets: 8109231
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 1385386738
rx_prio5_packets: 9661187
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 317092102
rx_prio6_packets: 1951538
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 7015734695
rx_prio7_packets: 99847456
tx_prio7_bytes: 0
tx_prio7_packets: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
ch0_events: 936264703
ch0_poll: 963766474
ch0_arm: 930246079
ch0_aff_change: 0
ch0_eq_rearm: 0
ch1_events: 869408429
ch1_poll: 896099392
ch1_arm: 864336861
ch1_aff_change: 0
ch1_eq_rearm: 0
ch2_events: 843345698
ch2_poll: 869749522
ch2_arm: 838186113
ch2_aff_change: 2
ch2_eq_rearm: 0
ch3_events: 850261340
ch3_poll: 876721111
ch3_arm: 845295235
ch3_aff_change: 3
ch3_eq_rearm: 0
ch4_events: 974985780
ch4_poll: 997781915
ch4_arm: 969618250
ch4_aff_change: 3
ch4_eq_rearm: 0
ch5_events: 888559089
ch5_poll: 912783615
ch5_arm: 883826078
ch5_aff_change: 2
ch5_eq_rearm: 0
ch6_events: 873730730
ch6_poll: 899635752
ch6_arm: 868677574
ch6_aff_change: 4
ch6_eq_rearm: 0
ch7_events: 873478411
ch7_poll: 899216716
ch7_arm: 868693645
ch7_aff_change: 3
ch7_eq_rearm: 0
ch8_events: 871900967
ch8_poll: 898575518
ch8_arm: 866763693
ch8_aff_change: 3
ch8_eq_rearm: 0
ch9_events: 880325565
ch9_poll: 904983269
ch9_arm: 875643922
ch9_aff_change: 2
ch9_eq_rearm: 0
ch10_events: 889919775
ch10_poll: 915335809
ch10_arm: 885110225
ch10_aff_change: 4
ch10_eq_rearm: 0
ch11_events: 962709175
ch11_poll: 983963451
ch11_arm: 958117526
ch11_aff_change: 2
ch11_eq_rearm: 0
ch12_events: 941333837
ch12_poll: 964625523
ch12_arm: 936409706
ch12_aff_change: 2
ch12_eq_rearm: 0
ch13_events: 914996974
ch13_poll: 937441049
ch13_arm: 910478393
ch13_aff_change: 4
ch13_eq_rearm: 0
ch14_events: 888050001
ch14_poll: 911818008
ch14_arm: 883465035
ch14_aff_change: 4
ch14_eq_rearm: 0
ch15_events: 947547704
ch15_poll: 969073194
ch15_arm: 942686515
ch15_aff_change: 4
ch15_eq_rearm: 0
ch16_events: 825804904
ch16_poll: 840630747
ch16_arm: 822227488
ch16_aff_change: 2
ch16_eq_rearm: 0
ch17_events: 861673823
ch17_poll: 874754041
ch17_arm: 858520448
ch17_aff_change: 2
ch17_eq_rearm: 0
ch18_events: 879413440
ch18_poll: 893962529
ch18_arm: 875983204
ch18_aff_change: 4
ch18_eq_rearm: 0
ch19_events: 896073709
ch19_poll: 909216857
ch19_arm: 893022121
ch19_aff_change: 4
ch19_eq_rearm: 0
ch20_events: 865188535
ch20_poll: 880692345
ch20_arm: 861440265
ch20_aff_change: 3
ch20_eq_rearm: 0
ch21_events: 862709303
ch21_poll: 878104242
ch21_arm: 859041767
ch21_aff_change: 2
ch21_eq_rearm: 0
ch22_events: 887720551
ch22_poll: 904122074
ch22_arm: 883983794
ch22_aff_change: 2
ch22_eq_rearm: 0
ch23_events: 813355027
ch23_poll: 828074467
ch23_arm: 809912398
ch23_aff_change: 4
ch23_eq_rearm: 0
ch24_events: 822366675
ch24_poll: 839917937
ch24_arm: 818422754
ch24_aff_change: 2
ch24_eq_rearm: 0
ch25_events: 826642292
ch25_poll: 842630121
ch25_arm: 822642618
ch25_aff_change: 2
ch25_eq_rearm: 0
ch26_events: 826392584
ch26_poll: 843406973
ch26_arm: 822455000
ch26_aff_change: 3
ch26_eq_rearm: 0
ch27_events: 828960899
ch27_poll: 843866518
ch27_arm: 825230937
ch27_aff_change: 3
ch27_eq_rearm: 0
ch28_events: 7
ch28_poll: 7
ch28_arm: 7
ch28_aff_change: 0
ch28_eq_rearm: 0
ch29_events: 4
ch29_poll: 4
ch29_arm: 4
ch29_aff_change: 0
ch29_eq_rearm: 0
ch30_events: 4
ch30_poll: 4
ch30_arm: 4
ch30_aff_change: 0
ch30_eq_rearm: 0
ch31_events: 4
ch31_poll: 4
ch31_arm: 4
ch31_aff_change: 0
ch31_eq_rearm: 0
ch32_events: 4
ch32_poll: 4
ch32_arm: 4
ch32_aff_change: 0
ch32_eq_rearm: 0
ch33_events: 4
ch33_poll: 4
ch33_arm: 4
ch33_aff_change: 0
ch33_eq_rearm: 0
ch34_events: 4
ch34_poll: 4
ch34_arm: 4
ch34_aff_change: 0
ch34_eq_rearm: 0
ch35_events: 4
ch35_poll: 4
ch35_arm: 4
ch35_aff_change: 0
ch35_eq_rearm: 0
ch36_events: 4
ch36_poll: 4
ch36_arm: 4
ch36_aff_change: 0
ch36_eq_rearm: 0
ch37_events: 4
ch37_poll: 4
ch37_arm: 4
ch37_aff_change: 0
ch37_eq_rearm: 0
ch38_events: 4
ch38_poll: 4
ch38_arm: 4
ch38_aff_change: 0
ch38_eq_rearm: 0
ch39_events: 4
ch39_poll: 4
ch39_arm: 4
ch39_aff_change: 0
ch39_eq_rearm: 0
ch40_events: 4
ch40_poll: 4
ch40_arm: 4
ch40_aff_change: 0
ch40_eq_rearm: 0
ch41_events: 4
ch41_poll: 4
ch41_arm: 4
ch41_aff_change: 0
ch41_eq_rearm: 0
ch42_events: 4
ch42_poll: 4
ch42_arm: 4
ch42_aff_change: 0
ch42_eq_rearm: 0
ch43_events: 4
ch43_poll: 4
ch43_arm: 4
ch43_aff_change: 0
ch43_eq_rearm: 0
ch44_events: 4
ch44_poll: 4
ch44_arm: 4
ch44_aff_change: 0
ch44_eq_rearm: 0
ch45_events: 4
ch45_poll: 4
ch45_arm: 4
ch45_aff_change: 0
ch45_eq_rearm: 0
ch46_events: 4
ch46_poll: 4
ch46_arm: 4
ch46_aff_change: 0
ch46_eq_rearm: 0
ch47_events: 4
ch47_poll: 4
ch47_arm: 4
ch47_aff_change: 0
ch47_eq_rearm: 0
ch48_events: 4
ch48_poll: 4
ch48_arm: 4
ch48_aff_change: 0
ch48_eq_rearm: 0
ch49_events: 4
ch49_poll: 4
ch49_arm: 4
ch49_aff_change: 0
ch49_eq_rearm: 0
ch50_events: 4
ch50_poll: 4
ch50_arm: 4
ch50_aff_change: 0
ch50_eq_rearm: 0
ch51_events: 4
ch51_poll: 4
ch51_arm: 4
ch51_aff_change: 0
ch51_eq_rearm: 0
ch52_events: 4
ch52_poll: 4
ch52_arm: 4
ch52_aff_change: 0
ch52_eq_rearm: 0
ch53_events: 4
ch53_poll: 4
ch53_arm: 4
ch53_aff_change: 0
ch53_eq_rearm: 0
ch54_events: 4
ch54_poll: 4
ch54_arm: 4
ch54_aff_change: 0
ch54_eq_rearm: 0
ch55_events: 4
ch55_poll: 4
ch55_arm: 4
ch55_aff_change: 0
ch55_eq_rearm: 0
rx0_packets: 7284057433
rx0_bytes: 4330611281319
rx0_csum_complete: 7283623076
rx0_csum_unnecessary: 0
rx0_csum_unnecessary_inner: 0
rx0_csum_none: 434357
rx0_xdp_drop: 0
rx0_xdp_redirect: 0
rx0_lro_packets: 0
rx0_lro_bytes: 0
rx0_ecn_mark: 0
rx0_removed_vlan_packets: 7284057433
rx0_wqe_err: 0
rx0_mpwqe_filler_cqes: 0
rx0_mpwqe_filler_strides: 0
rx0_buff_alloc_err: 0
rx0_cqe_compress_blks: 0
rx0_cqe_compress_pkts: 0
rx0_page_reuse: 0
rx0_cache_reuse: 1989731589
rx0_cache_full: 28213297
rx0_cache_empty: 1624089822
rx0_cache_busy: 28213961
rx0_cache_waive: 1624083610
rx0_congst_umr: 0
rx0_arfs_err: 0
rx0_xdp_tx_xmit: 0
rx0_xdp_tx_full: 0
rx0_xdp_tx_err: 0
rx0_xdp_tx_cqes: 0
rx1_packets: 6691319211
rx1_bytes: 3799580210608
rx1_csum_complete: 6691319211
rx1_csum_unnecessary: 0
rx1_csum_unnecessary_inner: 0
rx1_csum_none: 0
rx1_xdp_drop: 0
rx1_xdp_redirect: 0
rx1_lro_packets: 0
rx1_lro_bytes: 0
rx1_ecn_mark: 0
rx1_removed_vlan_packets: 6691319211
rx1_wqe_err: 0
rx1_mpwqe_filler_cqes: 0
rx1_mpwqe_filler_strides: 0
rx1_buff_alloc_err: 0
rx1_cqe_compress_blks: 0
rx1_cqe_compress_pkts: 0
rx1_page_reuse: 0
rx1_cache_reuse: 2270019
rx1_cache_full: 3343389331
rx1_cache_empty: 6656
rx1_cache_busy: 3343389585
rx1_cache_waive: 0
rx1_congst_umr: 0
rx1_arfs_err: 0
rx1_xdp_tx_xmit: 0
rx1_xdp_tx_full: 0
rx1_xdp_tx_err: 0
rx1_xdp_tx_cqes: 0
rx2_packets: 6618370416
rx2_bytes: 3762508364015
rx2_csum_complete: 6618370416
rx2_csum_unnecessary: 0
rx2_csum_unnecessary_inner: 0
rx2_csum_none: 0
rx2_xdp_drop: 0
rx2_xdp_redirect: 0
rx2_lro_packets: 0
rx2_lro_bytes: 0
rx2_ecn_mark: 0
rx2_removed_vlan_packets: 6618370416
rx2_wqe_err: 0
rx2_mpwqe_filler_cqes: 0
rx2_mpwqe_filler_strides: 0
rx2_buff_alloc_err: 0
rx2_cqe_compress_blks: 0
rx2_cqe_compress_pkts: 0
rx2_page_reuse: 0
rx2_cache_reuse: 111419328
rx2_cache_full: 1807563903
rx2_cache_empty: 1390208158
rx2_cache_busy: 1807564378
rx2_cache_waive: 1390201722
rx2_congst_umr: 0
rx2_arfs_err: 0
rx2_xdp_tx_xmit: 0
rx2_xdp_tx_full: 0
rx2_xdp_tx_err: 0
rx2_xdp_tx_cqes: 0
rx3_packets: 6665308976
rx3_bytes: 3828546206006
rx3_csum_complete: 6665308976
rx3_csum_unnecessary: 0
rx3_csum_unnecessary_inner: 0
rx3_csum_none: 0
rx3_xdp_drop: 0
rx3_xdp_redirect: 0
rx3_lro_packets: 0
rx3_lro_bytes: 0
rx3_ecn_mark: 0
rx3_removed_vlan_packets: 6665308976
rx3_wqe_err: 0
rx3_mpwqe_filler_cqes: 0
rx3_mpwqe_filler_strides: 0
rx3_buff_alloc_err: 0
rx3_cqe_compress_blks: 0
rx3_cqe_compress_pkts: 0
rx3_page_reuse: 0
rx3_cache_reuse: 215779091
rx3_cache_full: 1720040649
rx3_cache_empty: 1396840926
rx3_cache_busy: 1720041127
rx3_cache_waive: 1396834493
rx3_congst_umr: 0
rx3_arfs_err: 0
rx3_xdp_tx_xmit: 0
rx3_xdp_tx_full: 0
rx3_xdp_tx_err: 0
rx3_xdp_tx_cqes: 0
rx4_packets: 6764448165
rx4_bytes: 3883101339142
rx4_csum_complete: 6764448165
rx4_csum_unnecessary: 0
rx4_csum_unnecessary_inner: 0
rx4_csum_none: 0
rx4_xdp_drop: 0
rx4_xdp_redirect: 0
rx4_lro_packets: 0
rx4_lro_bytes: 0
rx4_ecn_mark: 0
rx4_removed_vlan_packets: 6764448165
rx4_wqe_err: 0
rx4_mpwqe_filler_cqes: 0
rx4_mpwqe_filler_strides: 0
rx4_buff_alloc_err: 0
rx4_cqe_compress_blks: 0
rx4_cqe_compress_pkts: 0
rx4_page_reuse: 0
rx4_cache_reuse: 1930710653
rx4_cache_full: 6490815
rx4_cache_empty: 1445028605
rx4_cache_busy: 6491478
rx4_cache_waive: 1445022392
rx4_congst_umr: 0
rx4_arfs_err: 0
rx4_xdp_tx_xmit: 0
rx4_xdp_tx_full: 0
rx4_xdp_tx_err: 0
rx4_xdp_tx_cqes: 0
rx5_packets: 6736853264
rx5_bytes: 3925186068552
rx5_csum_complete: 6736853264
rx5_csum_unnecessary: 0
rx5_csum_unnecessary_inner: 0
rx5_csum_none: 0
rx5_xdp_drop: 0
rx5_xdp_redirect: 0
rx5_lro_packets: 0
rx5_lro_bytes: 0
rx5_ecn_mark: 0
rx5_removed_vlan_packets: 6736853264
rx5_wqe_err: 0
rx5_mpwqe_filler_cqes: 0
rx5_mpwqe_filler_strides: 0
rx5_buff_alloc_err: 0
rx5_cqe_compress_blks: 0
rx5_cqe_compress_pkts: 0
rx5_page_reuse: 0
rx5_cache_reuse: 7283914
rx5_cache_full: 3361142463
rx5_cache_empty: 6656
rx5_cache_busy: 3361142718
rx5_cache_waive: 0
rx5_congst_umr: 0
rx5_arfs_err: 0
rx5_xdp_tx_xmit: 0
rx5_xdp_tx_full: 0
rx5_xdp_tx_err: 0
rx5_xdp_tx_cqes: 0
rx6_packets: 6751588828
rx6_bytes: 3860537598885
rx6_csum_complete: 6751588828
rx6_csum_unnecessary: 0
rx6_csum_unnecessary_inner: 0
rx6_csum_none: 0
rx6_xdp_drop: 0
rx6_xdp_redirect: 0
rx6_lro_packets: 0
rx6_lro_bytes: 0
rx6_ecn_mark: 0
rx6_removed_vlan_packets: 6751588828
rx6_wqe_err: 0
rx6_mpwqe_filler_cqes: 0
rx6_mpwqe_filler_strides: 0
rx6_buff_alloc_err: 0
rx6_cqe_compress_blks: 0
rx6_cqe_compress_pkts: 0
rx6_page_reuse: 0
rx6_cache_reuse: 96032126
rx6_cache_full: 1857890923
rx6_cache_empty: 1421877543
rx6_cache_busy: 1857891399
rx6_cache_waive: 1421871110
rx6_congst_umr: 0
rx6_arfs_err: 0
rx6_xdp_tx_xmit: 0
rx6_xdp_tx_full: 0
rx6_xdp_tx_err: 0
rx6_xdp_tx_cqes: 0
rx7_packets: 6935300074
rx7_bytes: 4004713524388
rx7_csum_complete: 6935300074
rx7_csum_unnecessary: 0
rx7_csum_unnecessary_inner: 0
rx7_csum_none: 0
rx7_xdp_drop: 0
rx7_xdp_redirect: 0
rx7_lro_packets: 0
rx7_lro_bytes: 0
rx7_ecn_mark: 0
rx7_removed_vlan_packets: 6935300074
rx7_wqe_err: 0
rx7_mpwqe_filler_cqes: 0
rx7_mpwqe_filler_strides: 0
rx7_buff_alloc_err: 0
rx7_cqe_compress_blks: 0
rx7_cqe_compress_pkts: 0
rx7_page_reuse: 0
rx7_cache_reuse: 17555187
rx7_cache_full: 3450094595
rx7_cache_empty: 6656
rx7_cache_busy: 3450094849
rx7_cache_waive: 0
rx7_congst_umr: 0
rx7_arfs_err: 0
rx7_xdp_tx_xmit: 0
rx7_xdp_tx_full: 0
rx7_xdp_tx_err: 0
rx7_xdp_tx_cqes: 0
rx8_packets: 6678640094
rx8_bytes: 3783722686028
rx8_csum_complete: 6678640094
rx8_csum_unnecessary: 0
rx8_csum_unnecessary_inner: 0
rx8_csum_none: 0
rx8_xdp_drop: 0
rx8_xdp_redirect: 0
rx8_lro_packets: 0
rx8_lro_bytes: 0
rx8_ecn_mark: 0
rx8_removed_vlan_packets: 6678640094
rx8_wqe_err: 0
rx8_mpwqe_filler_cqes: 0
rx8_mpwqe_filler_strides: 0
rx8_buff_alloc_err: 0
rx8_cqe_compress_blks: 0
rx8_cqe_compress_pkts: 0
rx8_page_reuse: 0
rx8_cache_reuse: 71006578
rx8_cache_full: 1879380649
rx8_cache_empty: 1388938999
rx8_cache_busy: 1879381123
rx8_cache_waive: 1388932565
rx8_congst_umr: 0
rx8_arfs_err: 0
rx8_xdp_tx_xmit: 0
rx8_xdp_tx_full: 0
rx8_xdp_tx_err: 0
rx8_xdp_tx_cqes: 0
rx9_packets: 6709855557
rx9_bytes: 3849522227880
rx9_csum_complete: 6709855557
rx9_csum_unnecessary: 0
rx9_csum_unnecessary_inner: 0
rx9_csum_none: 0
rx9_xdp_drop: 0
rx9_xdp_redirect: 0
rx9_lro_packets: 0
rx9_lro_bytes: 0
rx9_ecn_mark: 0
rx9_removed_vlan_packets: 6709855557
rx9_wqe_err: 0
rx9_mpwqe_filler_cqes: 0
rx9_mpwqe_filler_strides: 0
rx9_buff_alloc_err: 0
rx9_cqe_compress_blks: 0
rx9_cqe_compress_pkts: 0
rx9_page_reuse: 0
rx9_cache_reuse: 108980215
rx9_cache_full: 1822730121
rx9_cache_empty: 1423223623
rx9_cache_busy: 1822730594
rx9_cache_waive: 1423217187
rx9_congst_umr: 0
rx9_arfs_err: 0
rx9_xdp_tx_xmit: 0
rx9_xdp_tx_full: 0
rx9_xdp_tx_err: 0
rx9_xdp_tx_cqes: 0
rx10_packets: 6761861066
rx10_bytes: 3816266733385
rx10_csum_complete: 6761861066
rx10_csum_unnecessary: 0
rx10_csum_unnecessary_inner: 0
rx10_csum_none: 0
rx10_xdp_drop: 0
rx10_xdp_redirect: 0
rx10_lro_packets: 0
rx10_lro_bytes: 0
rx10_ecn_mark: 0
rx10_removed_vlan_packets: 6761861066
rx10_wqe_err: 0
rx10_mpwqe_filler_cqes: 0
rx10_mpwqe_filler_strides: 0
rx10_buff_alloc_err: 0
rx10_cqe_compress_blks: 0
rx10_cqe_compress_pkts: 0
rx10_page_reuse: 0
rx10_cache_reuse: 3489300
rx10_cache_full: 3377440977
rx10_cache_empty: 6656
rx10_cache_busy: 3377441216
rx10_cache_waive: 0
rx10_congst_umr: 0
rx10_arfs_err: 0
rx10_xdp_tx_xmit: 0
rx10_xdp_tx_full: 0
rx10_xdp_tx_err: 0
rx10_xdp_tx_cqes: 0
rx11_packets: 6868113938
rx11_bytes: 4048196300710
rx11_csum_complete: 6868113938
rx11_csum_unnecessary: 0
rx11_csum_unnecessary_inner: 0
rx11_csum_none: 0
rx11_xdp_drop: 0
rx11_xdp_redirect: 0
rx11_lro_packets: 0
rx11_lro_bytes: 0
rx11_ecn_mark: 0
rx11_removed_vlan_packets: 6868113938
rx11_wqe_err: 0
rx11_mpwqe_filler_cqes: 0
rx11_mpwqe_filler_strides: 0
rx11_buff_alloc_err: 0
rx11_cqe_compress_blks: 0
rx11_cqe_compress_pkts: 0
rx11_page_reuse: 0
rx11_cache_reuse: 1948516819
rx11_cache_full: 17132157
rx11_cache_empty: 1468413985
rx11_cache_busy: 17132820
rx11_cache_waive: 1468407772
rx11_congst_umr: 0
rx11_arfs_err: 0
rx11_xdp_tx_xmit: 0
rx11_xdp_tx_full: 0
rx11_xdp_tx_err: 0
rx11_xdp_tx_cqes: 0
rx12_packets: 6742955386
rx12_bytes: 3865747629271
rx12_csum_complete: 6742955386
rx12_csum_unnecessary: 0
rx12_csum_unnecessary_inner: 0
rx12_csum_none: 0
rx12_xdp_drop: 0
rx12_xdp_redirect: 0
rx12_lro_packets: 0
rx12_lro_bytes: 0
rx12_ecn_mark: 0
rx12_removed_vlan_packets: 6742955386
rx12_wqe_err: 0
rx12_mpwqe_filler_cqes: 0
rx12_mpwqe_filler_strides: 0
rx12_buff_alloc_err: 0
rx12_cqe_compress_blks: 0
rx12_cqe_compress_pkts: 0
rx12_page_reuse: 0
rx12_cache_reuse: 30809331
rx12_cache_full: 3340668106
rx12_cache_empty: 6656
rx12_cache_busy: 3340668333
rx12_cache_waive: 0
rx12_congst_umr: 0
rx12_arfs_err: 0
rx12_xdp_tx_xmit: 0
rx12_xdp_tx_full: 0
rx12_xdp_tx_err: 0
rx12_xdp_tx_cqes: 0
rx13_packets: 6707028036
rx13_bytes: 3813462190623
rx13_csum_complete: 6707028036
rx13_csum_unnecessary: 0
rx13_csum_unnecessary_inner: 0
rx13_csum_none: 0
rx13_xdp_drop: 0
rx13_xdp_redirect: 0
rx13_lro_packets: 0
rx13_lro_bytes: 0
rx13_ecn_mark: 0
rx13_removed_vlan_packets: 6707028036
rx13_wqe_err: 0
rx13_mpwqe_filler_cqes: 0
rx13_mpwqe_filler_strides: 0
rx13_buff_alloc_err: 0
rx13_cqe_compress_blks: 0
rx13_cqe_compress_pkts: 0
rx13_page_reuse: 0
rx13_cache_reuse: 14951053
rx13_cache_full: 3338562710
rx13_cache_empty: 6656
rx13_cache_busy: 3338562963
rx13_cache_waive: 0
rx13_congst_umr: 0
rx13_arfs_err: 0
rx13_xdp_tx_xmit: 0
rx13_xdp_tx_full: 0
rx13_xdp_tx_err: 0
rx13_xdp_tx_cqes: 0
rx14_packets: 6737074410
rx14_bytes: 3868905276119
rx14_csum_complete: 6737074410
rx14_csum_unnecessary: 0
rx14_csum_unnecessary_inner: 0
rx14_csum_none: 0
rx14_xdp_drop: 0
rx14_xdp_redirect: 0
rx14_lro_packets: 0
rx14_lro_bytes: 0
rx14_ecn_mark: 0
rx14_removed_vlan_packets: 6737074410
rx14_wqe_err: 0
rx14_mpwqe_filler_cqes: 0
rx14_mpwqe_filler_strides: 0
rx14_buff_alloc_err: 0
rx14_cqe_compress_blks: 0
rx14_cqe_compress_pkts: 0
rx14_page_reuse: 0
rx14_cache_reuse: 967799432
rx14_cache_full: 982704312
rx14_cache_empty: 1418039639
rx14_cache_busy: 982704789
rx14_cache_waive: 1418033206
rx14_congst_umr: 0
rx14_arfs_err: 0
rx14_xdp_tx_xmit: 0
rx14_xdp_tx_full: 0
rx14_xdp_tx_err: 0
rx14_xdp_tx_cqes: 0
rx15_packets: 6641887441
rx15_bytes: 3742874400402
rx15_csum_complete: 6641887441
rx15_csum_unnecessary: 0
rx15_csum_unnecessary_inner: 0
rx15_csum_none: 0
rx15_xdp_drop: 0
rx15_xdp_redirect: 0
rx15_lro_packets: 0
rx15_lro_bytes: 0
rx15_ecn_mark: 0
rx15_removed_vlan_packets: 6641887441
rx15_wqe_err: 0
rx15_mpwqe_filler_cqes: 0
rx15_mpwqe_filler_strides: 0
rx15_buff_alloc_err: 0
rx15_cqe_compress_blks: 0
rx15_cqe_compress_pkts: 0
rx15_page_reuse: 0
rx15_cache_reuse: 1920227538
rx15_cache_full: 19386129
rx15_cache_empty: 1381335137
rx15_cache_busy: 19387693
rx15_cache_waive: 1381329825
rx15_congst_umr: 0
rx15_arfs_err: 0
rx15_xdp_tx_xmit: 0
rx15_xdp_tx_full: 0
rx15_xdp_tx_err: 0
rx15_xdp_tx_cqes: 0
rx16_packets: 5420472874
rx16_bytes: 3079293332581
rx16_csum_complete: 5420472874
rx16_csum_unnecessary: 0
rx16_csum_unnecessary_inner: 0
rx16_csum_none: 0
rx16_xdp_drop: 0
rx16_xdp_redirect: 0
rx16_lro_packets: 0
rx16_lro_bytes: 0
rx16_ecn_mark: 0
rx16_removed_vlan_packets: 5420472874
rx16_wqe_err: 0
rx16_mpwqe_filler_cqes: 0
rx16_mpwqe_filler_strides: 0
rx16_buff_alloc_err: 0
rx16_cqe_compress_blks: 0
rx16_cqe_compress_pkts: 0
rx16_page_reuse: 0
rx16_cache_reuse: 2361079
rx16_cache_full: 2707875103
rx16_cache_empty: 6656
rx16_cache_busy: 2707875349
rx16_cache_waive: 0
rx16_congst_umr: 0
rx16_arfs_err: 0
rx16_xdp_tx_xmit: 0
rx16_xdp_tx_full: 0
rx16_xdp_tx_err: 0
rx16_xdp_tx_cqes: 0
rx17_packets: 5428380986
rx17_bytes: 3080981893118
rx17_csum_complete: 5428380986
rx17_csum_unnecessary: 0
rx17_csum_unnecessary_inner: 0
rx17_csum_none: 0
rx17_xdp_drop: 0
rx17_xdp_redirect: 0
rx17_lro_packets: 0
rx17_lro_bytes: 0
rx17_ecn_mark: 0
rx17_removed_vlan_packets: 5428380986
rx17_wqe_err: 0
rx17_mpwqe_filler_cqes: 0
rx17_mpwqe_filler_strides: 0
rx17_buff_alloc_err: 0
rx17_cqe_compress_blks: 0
rx17_cqe_compress_pkts: 0
rx17_page_reuse: 0
rx17_cache_reuse: 1552266402
rx17_cache_full: 5947505
rx17_cache_empty: 1155981856
rx17_cache_busy: 5948870
rx17_cache_waive: 1155976345
rx17_congst_umr: 0
rx17_arfs_err: 0
rx17_xdp_tx_xmit: 0
rx17_xdp_tx_full: 0
rx17_xdp_tx_err: 0
rx17_xdp_tx_cqes: 0
rx18_packets: 5529118410
rx18_bytes: 3254749573833
rx18_csum_complete: 5529118410
rx18_csum_unnecessary: 0
rx18_csum_unnecessary_inner: 0
rx18_csum_none: 0
rx18_xdp_drop: 0
rx18_xdp_redirect: 0
rx18_lro_packets: 0
rx18_lro_bytes: 0
rx18_ecn_mark: 0
rx18_removed_vlan_packets: 5529118410
rx18_wqe_err: 0
rx18_mpwqe_filler_cqes: 0
rx18_mpwqe_filler_strides: 0
rx18_buff_alloc_err: 0
rx18_cqe_compress_blks: 0
rx18_cqe_compress_pkts: 0
rx18_page_reuse: 0
rx18_cache_reuse: 67438840
rx18_cache_full: 1536718472
rx18_cache_empty: 1160408072
rx18_cache_busy: 1536718932
rx18_cache_waive: 1160401638
rx18_congst_umr: 0
rx18_arfs_err: 0
rx18_xdp_tx_xmit: 0
rx18_xdp_tx_full: 0
rx18_xdp_tx_err: 0
rx18_xdp_tx_cqes: 0
rx19_packets: 5449932653
rx19_bytes: 3148726579411
rx19_csum_complete: 5449932653
rx19_csum_unnecessary: 0
rx19_csum_unnecessary_inner: 0
rx19_csum_none: 0
rx19_xdp_drop: 0
rx19_xdp_redirect: 0
rx19_lro_packets: 0
rx19_lro_bytes: 0
rx19_ecn_mark: 0
rx19_removed_vlan_packets: 5449932653
rx19_wqe_err: 0
rx19_mpwqe_filler_cqes: 0
rx19_mpwqe_filler_strides: 0
rx19_buff_alloc_err: 0
rx19_cqe_compress_blks: 0
rx19_cqe_compress_pkts: 0
rx19_page_reuse: 0
rx19_cache_reuse: 1537841743
rx19_cache_full: 9920960
rx19_cache_empty: 1177208938
rx19_cache_busy: 9922299
rx19_cache_waive: 1177203401
rx19_congst_umr: 0
rx19_arfs_err: 0
rx19_xdp_tx_xmit: 0
rx19_xdp_tx_full: 0
rx19_xdp_tx_err: 0
rx19_xdp_tx_cqes: 0
rx20_packets: 5407910071
rx20_bytes: 3123560861922
rx20_csum_complete: 5407910071
rx20_csum_unnecessary: 0
rx20_csum_unnecessary_inner: 0
rx20_csum_none: 0
rx20_xdp_drop: 0
rx20_xdp_redirect: 0
rx20_lro_packets: 0
rx20_lro_bytes: 0
rx20_ecn_mark: 0
rx20_removed_vlan_packets: 5407910071
rx20_wqe_err: 0
rx20_mpwqe_filler_cqes: 0
rx20_mpwqe_filler_strides: 0
rx20_buff_alloc_err: 0
rx20_cqe_compress_blks: 0
rx20_cqe_compress_pkts: 0
rx20_page_reuse: 0
rx20_cache_reuse: 10255209
rx20_cache_full: 2693699571
rx20_cache_empty: 6656
rx20_cache_busy: 2693699823
rx20_cache_waive: 0
rx20_congst_umr: 0
rx20_arfs_err: 0
rx20_xdp_tx_xmit: 0
rx20_xdp_tx_full: 0
rx20_xdp_tx_err: 0
rx20_xdp_tx_cqes: 0
rx21_packets: 5417498508
rx21_bytes: 3131335892379
rx21_csum_complete: 5417498508
rx21_csum_unnecessary: 0
rx21_csum_unnecessary_inner: 0
rx21_csum_none: 0
rx21_xdp_drop: 0
rx21_xdp_redirect: 0
rx21_lro_packets: 0
rx21_lro_bytes: 0
rx21_ecn_mark: 0
rx21_removed_vlan_packets: 5417498508
rx21_wqe_err: 0
rx21_mpwqe_filler_cqes: 0
rx21_mpwqe_filler_strides: 0
rx21_buff_alloc_err: 0
rx21_cqe_compress_blks: 0
rx21_cqe_compress_pkts: 0
rx21_page_reuse: 0
rx21_cache_reuse: 192662917
rx21_cache_full: 1374120417
rx21_cache_empty: 1141972100
rx21_cache_busy: 1374120891
rx21_cache_waive: 1141965665
rx21_congst_umr: 0
rx21_arfs_err: 0
rx21_xdp_tx_xmit: 0
rx21_xdp_tx_full: 0
rx21_xdp_tx_err: 0
rx21_xdp_tx_cqes: 0
rx22_packets: 5613634706
rx22_bytes: 3240055099058
rx22_csum_complete: 5613634706
rx22_csum_unnecessary: 0
rx22_csum_unnecessary_inner: 0
rx22_csum_none: 0
rx22_xdp_drop: 0
rx22_xdp_redirect: 0
rx22_lro_packets: 0
rx22_lro_bytes: 0
rx22_ecn_mark: 0
rx22_removed_vlan_packets: 5613634706
rx22_wqe_err: 0
rx22_mpwqe_filler_cqes: 0
rx22_mpwqe_filler_strides: 0
rx22_buff_alloc_err: 0
rx22_cqe_compress_blks: 0
rx22_cqe_compress_pkts: 0
rx22_page_reuse: 0
rx22_cache_reuse: 12161531
rx22_cache_full: 2794655567
rx22_cache_empty: 6656
rx22_cache_busy: 2794655821
rx22_cache_waive: 0
rx22_congst_umr: 0
rx22_arfs_err: 0
rx22_xdp_tx_xmit: 0
rx22_xdp_tx_full: 0
rx22_xdp_tx_err: 0
rx22_xdp_tx_cqes: 0
rx23_packets: 5389977167
rx23_bytes: 3054270771559
rx23_csum_complete: 5389977167
rx23_csum_unnecessary: 0
rx23_csum_unnecessary_inner: 0
rx23_csum_none: 0
rx23_xdp_drop: 0
rx23_xdp_redirect: 0
rx23_lro_packets: 0
rx23_lro_bytes: 0
rx23_ecn_mark: 0
rx23_removed_vlan_packets: 5389977167
rx23_wqe_err: 0
rx23_mpwqe_filler_cqes: 0
rx23_mpwqe_filler_strides: 0
rx23_buff_alloc_err: 0
rx23_cqe_compress_blks: 0
rx23_cqe_compress_pkts: 0
rx23_page_reuse: 0
rx23_cache_reuse: 709328
rx23_cache_full: 2694279000
rx23_cache_empty: 6656
rx23_cache_busy: 2694279252
rx23_cache_waive: 0
rx23_congst_umr: 0
rx23_arfs_err: 0
rx23_xdp_tx_xmit: 0
rx23_xdp_tx_full: 0
rx23_xdp_tx_err: 0
rx23_xdp_tx_cqes: 0
rx24_packets: 5547561932
rx24_bytes: 3166602453443
rx24_csum_complete: 5547561932
rx24_csum_unnecessary: 0
rx24_csum_unnecessary_inner: 0
rx24_csum_none: 0
rx24_xdp_drop: 0
rx24_xdp_redirect: 0
rx24_lro_packets: 0
rx24_lro_bytes: 0
rx24_ecn_mark: 0
rx24_removed_vlan_packets: 5547561932
rx24_wqe_err: 0
rx24_mpwqe_filler_cqes: 0
rx24_mpwqe_filler_strides: 0
rx24_buff_alloc_err: 0
rx24_cqe_compress_blks: 0
rx24_cqe_compress_pkts: 0
rx24_page_reuse: 0
rx24_cache_reuse: 57885119
rx24_cache_full: 1529450077
rx24_cache_empty: 1186451948
rx24_cache_busy: 1529450553
rx24_cache_waive: 1186445515
rx24_congst_umr: 0
rx24_arfs_err: 0
rx24_xdp_tx_xmit: 0
rx24_xdp_tx_full: 0
rx24_xdp_tx_err: 0
rx24_xdp_tx_cqes: 0
rx25_packets: 5414569326
rx25_bytes: 3184757708091
rx25_csum_complete: 5414569326
rx25_csum_unnecessary: 0
rx25_csum_unnecessary_inner: 0
rx25_csum_none: 0
rx25_xdp_drop: 0
rx25_xdp_redirect: 0
rx25_lro_packets: 0
rx25_lro_bytes: 0
rx25_ecn_mark: 0
rx25_removed_vlan_packets: 5414569326
rx25_wqe_err: 0
rx25_mpwqe_filler_cqes: 0
rx25_mpwqe_filler_strides: 0
rx25_buff_alloc_err: 0
rx25_cqe_compress_blks: 0
rx25_cqe_compress_pkts: 0
rx25_page_reuse: 0
rx25_cache_reuse: 5080853
rx25_cache_full: 2702203555
rx25_cache_empty: 6656
rx25_cache_busy: 2702203807
rx25_cache_waive: 0
rx25_congst_umr: 0
rx25_arfs_err: 0
rx25_xdp_tx_xmit: 0
rx25_xdp_tx_full: 0
rx25_xdp_tx_err: 0
rx25_xdp_tx_cqes: 0
rx26_packets: 5479972151
rx26_bytes: 3110642276239
rx26_csum_complete: 5479972151
rx26_csum_unnecessary: 0
rx26_csum_unnecessary_inner: 0
rx26_csum_none: 0
rx26_xdp_drop: 0
rx26_xdp_redirect: 0
rx26_lro_packets: 0
rx26_lro_bytes: 0
rx26_ecn_mark: 0
rx26_removed_vlan_packets: 5479972151
rx26_wqe_err: 0
rx26_mpwqe_filler_cqes: 0
rx26_mpwqe_filler_strides: 0
rx26_buff_alloc_err: 0
rx26_cqe_compress_blks: 0
rx26_cqe_compress_pkts: 0
rx26_page_reuse: 0
rx26_cache_reuse: 26543335
rx26_cache_full: 2713442485
rx26_cache_empty: 6656
rx26_cache_busy: 2713442737
rx26_cache_waive: 0
rx26_congst_umr: 0
rx26_arfs_err: 0
rx26_xdp_tx_xmit: 0
rx26_xdp_tx_full: 0
rx26_xdp_tx_err: 0
rx26_xdp_tx_cqes: 0
rx27_packets: 5337113900
rx27_bytes: 3068966906075
rx27_csum_complete: 5337113900
rx27_csum_unnecessary: 0
rx27_csum_unnecessary_inner: 0
rx27_csum_none: 0
rx27_xdp_drop: 0
rx27_xdp_redirect: 0
rx27_lro_packets: 0
rx27_lro_bytes: 0
rx27_ecn_mark: 0
rx27_removed_vlan_packets: 5337113900
rx27_wqe_err: 0
rx27_mpwqe_filler_cqes: 0
rx27_mpwqe_filler_strides: 0
rx27_buff_alloc_err: 0
rx27_cqe_compress_blks: 0
rx27_cqe_compress_pkts: 0
rx27_page_reuse: 0
rx27_cache_reuse: 1539298962
rx27_cache_full: 10861919
rx27_cache_empty: 1117173179
rx27_cache_busy: 12091463
rx27_cache_waive: 1118395847
rx27_congst_umr: 0
rx27_arfs_err: 0
rx27_xdp_tx_xmit: 0
rx27_xdp_tx_full: 0
rx27_xdp_tx_err: 0
rx27_xdp_tx_cqes: 0
rx28_packets: 0
rx28_bytes: 0
rx28_csum_complete: 0
rx28_csum_unnecessary: 0
rx28_csum_unnecessary_inner: 0
rx28_csum_none: 0
rx28_xdp_drop: 0
rx28_xdp_redirect: 0
rx28_lro_packets: 0
rx28_lro_bytes: 0
rx28_ecn_mark: 0
rx28_removed_vlan_packets: 0
rx28_wqe_err: 0
rx28_mpwqe_filler_cqes: 0
rx28_mpwqe_filler_strides: 0
rx28_buff_alloc_err: 0
rx28_cqe_compress_blks: 0
rx28_cqe_compress_pkts: 0
rx28_page_reuse: 0
rx28_cache_reuse: 0
rx28_cache_full: 0
rx28_cache_empty: 2560
rx28_cache_busy: 0
rx28_cache_waive: 0
rx28_congst_umr: 0
rx28_arfs_err: 0
rx28_xdp_tx_xmit: 0
rx28_xdp_tx_full: 0
rx28_xdp_tx_err: 0
rx28_xdp_tx_cqes: 0
rx29_packets: 0
rx29_bytes: 0
rx29_csum_complete: 0
rx29_csum_unnecessary: 0
rx29_csum_unnecessary_inner: 0
rx29_csum_none: 0
rx29_xdp_drop: 0
rx29_xdp_redirect: 0
rx29_lro_packets: 0
rx29_lro_bytes: 0
rx29_ecn_mark: 0
rx29_removed_vlan_packets: 0
rx29_wqe_err: 0
rx29_mpwqe_filler_cqes: 0
rx29_mpwqe_filler_strides: 0
rx29_buff_alloc_err: 0
rx29_cqe_compress_blks: 0
rx29_cqe_compress_pkts: 0
rx29_page_reuse: 0
rx29_cache_reuse: 0
rx29_cache_full: 0
rx29_cache_empty: 2560
rx29_cache_busy: 0
rx29_cache_waive: 0
rx29_congst_umr: 0
rx29_arfs_err: 0
rx29_xdp_tx_xmit: 0
rx29_xdp_tx_full: 0
rx29_xdp_tx_err: 0
rx29_xdp_tx_cqes: 0
rx30_packets: 0
rx30_bytes: 0
rx30_csum_complete: 0
rx30_csum_unnecessary: 0
rx30_csum_unnecessary_inner: 0
rx30_csum_none: 0
rx30_xdp_drop: 0
rx30_xdp_redirect: 0
rx30_lro_packets: 0
rx30_lro_bytes: 0
rx30_ecn_mark: 0
rx30_removed_vlan_packets: 0
rx30_wqe_err: 0
rx30_mpwqe_filler_cqes: 0
rx30_mpwqe_filler_strides: 0
rx30_buff_alloc_err: 0
rx30_cqe_compress_blks: 0
rx30_cqe_compress_pkts: 0
rx30_page_reuse: 0
rx30_cache_reuse: 0
rx30_cache_full: 0
rx30_cache_empty: 2560
rx30_cache_busy: 0
rx30_cache_waive: 0
rx30_congst_umr: 0
rx30_arfs_err: 0
rx30_xdp_tx_xmit: 0
rx30_xdp_tx_full: 0
rx30_xdp_tx_err: 0
rx30_xdp_tx_cqes: 0
rx31_packets: 0
rx31_bytes: 0
rx31_csum_complete: 0
rx31_csum_unnecessary: 0
rx31_csum_unnecessary_inner: 0
rx31_csum_none: 0
rx31_xdp_drop: 0
rx31_xdp_redirect: 0
rx31_lro_packets: 0
rx31_lro_bytes: 0
rx31_ecn_mark: 0
rx31_removed_vlan_packets: 0
rx31_wqe_err: 0
rx31_mpwqe_filler_cqes: 0
rx31_mpwqe_filler_strides: 0
rx31_buff_alloc_err: 0
rx31_cqe_compress_blks: 0
rx31_cqe_compress_pkts: 0
rx31_page_reuse: 0
rx31_cache_reuse: 0
rx31_cache_full: 0
rx31_cache_empty: 2560
rx31_cache_busy: 0
rx31_cache_waive: 0
rx31_congst_umr: 0
rx31_arfs_err: 0
rx31_xdp_tx_xmit: 0
rx31_xdp_tx_full: 0
rx31_xdp_tx_err: 0
rx31_xdp_tx_cqes: 0
rx32_packets: 0
rx32_bytes: 0
rx32_csum_complete: 0
rx32_csum_unnecessary: 0
rx32_csum_unnecessary_inner: 0
rx32_csum_none: 0
rx32_xdp_drop: 0
rx32_xdp_redirect: 0
rx32_lro_packets: 0
rx32_lro_bytes: 0
rx32_ecn_mark: 0
rx32_removed_vlan_packets: 0
rx32_wqe_err: 0
rx32_mpwqe_filler_cqes: 0
rx32_mpwqe_filler_strides: 0
rx32_buff_alloc_err: 0
rx32_cqe_compress_blks: 0
rx32_cqe_compress_pkts: 0
rx32_page_reuse: 0
rx32_cache_reuse: 0
rx32_cache_full: 0
rx32_cache_empty: 2560
rx32_cache_busy: 0
rx32_cache_waive: 0
rx32_congst_umr: 0
rx32_arfs_err: 0
rx32_xdp_tx_xmit: 0
rx32_xdp_tx_full: 0
rx32_xdp_tx_err: 0
rx32_xdp_tx_cqes: 0
rx33_packets: 0
rx33_bytes: 0
rx33_csum_complete: 0
rx33_csum_unnecessary: 0
rx33_csum_unnecessary_inner: 0
rx33_csum_none: 0
rx33_xdp_drop: 0
rx33_xdp_redirect: 0
rx33_lro_packets: 0
rx33_lro_bytes: 0
rx33_ecn_mark: 0
rx33_removed_vlan_packets: 0
rx33_wqe_err: 0
rx33_mpwqe_filler_cqes: 0
rx33_mpwqe_filler_strides: 0
rx33_buff_alloc_err: 0
rx33_cqe_compress_blks: 0
rx33_cqe_compress_pkts: 0
rx33_page_reuse: 0
rx33_cache_reuse: 0
rx33_cache_full: 0
rx33_cache_empty: 2560
rx33_cache_busy: 0
rx33_cache_waive: 0
rx33_congst_umr: 0
rx33_arfs_err: 0
rx33_xdp_tx_xmit: 0
rx33_xdp_tx_full: 0
rx33_xdp_tx_err: 0
rx33_xdp_tx_cqes: 0
rx34_packets: 0
rx34_bytes: 0
rx34_csum_complete: 0
rx34_csum_unnecessary: 0
rx34_csum_unnecessary_inner: 0
rx34_csum_none: 0
rx34_xdp_drop: 0
rx34_xdp_redirect: 0
rx34_lro_packets: 0
rx34_lro_bytes: 0
rx34_ecn_mark: 0
rx34_removed_vlan_packets: 0
rx34_wqe_err: 0
rx34_mpwqe_filler_cqes: 0
rx34_mpwqe_filler_strides: 0
rx34_buff_alloc_err: 0
rx34_cqe_compress_blks: 0
rx34_cqe_compress_pkts: 0
rx34_page_reuse: 0
rx34_cache_reuse: 0
rx34_cache_full: 0
rx34_cache_empty: 2560
rx34_cache_busy: 0
rx34_cache_waive: 0
rx34_congst_umr: 0
rx34_arfs_err: 0
rx34_xdp_tx_xmit: 0
rx34_xdp_tx_full: 0
rx34_xdp_tx_err: 0
rx34_xdp_tx_cqes: 0
rx35_packets: 0
rx35_bytes: 0
rx35_csum_complete: 0
rx35_csum_unnecessary: 0
rx35_csum_unnecessary_inner: 0
rx35_csum_none: 0
rx35_xdp_drop: 0
rx35_xdp_redirect: 0
rx35_lro_packets: 0
rx35_lro_bytes: 0
rx35_ecn_mark: 0
rx35_removed_vlan_packets: 0
rx35_wqe_err: 0
rx35_mpwqe_filler_cqes: 0
rx35_mpwqe_filler_strides: 0
rx35_buff_alloc_err: 0
rx35_cqe_compress_blks: 0
rx35_cqe_compress_pkts: 0
rx35_page_reuse: 0
rx35_cache_reuse: 0
rx35_cache_full: 0
rx35_cache_empty: 2560
rx35_cache_busy: 0
rx35_cache_waive: 0
rx35_congst_umr: 0
rx35_arfs_err: 0
rx35_xdp_tx_xmit: 0
rx35_xdp_tx_full: 0
rx35_xdp_tx_err: 0
rx35_xdp_tx_cqes: 0
rx36_packets: 0
rx36_bytes: 0
rx36_csum_complete: 0
rx36_csum_unnecessary: 0
rx36_csum_unnecessary_inner: 0
rx36_csum_none: 0
rx36_xdp_drop: 0
rx36_xdp_redirect: 0
rx36_lro_packets: 0
rx36_lro_bytes: 0
rx36_ecn_mark: 0
rx36_removed_vlan_packets: 0
rx36_wqe_err: 0
rx36_mpwqe_filler_cqes: 0
rx36_mpwqe_filler_strides: 0
rx36_buff_alloc_err: 0
rx36_cqe_compress_blks: 0
rx36_cqe_compress_pkts: 0
rx36_page_reuse: 0
rx36_cache_reuse: 0
rx36_cache_full: 0
rx36_cache_empty: 2560
rx36_cache_busy: 0
rx36_cache_waive: 0
rx36_congst_umr: 0
rx36_arfs_err: 0
rx36_xdp_tx_xmit: 0
rx36_xdp_tx_full: 0
rx36_xdp_tx_err: 0
rx36_xdp_tx_cqes: 0
rx37_packets: 0
rx37_bytes: 0
rx37_csum_complete: 0
rx37_csum_unnecessary: 0
rx37_csum_unnecessary_inner: 0
rx37_csum_none: 0
rx37_xdp_drop: 0
rx37_xdp_redirect: 0
rx37_lro_packets: 0
rx37_lro_bytes: 0
rx37_ecn_mark: 0
rx37_removed_vlan_packets: 0
rx37_wqe_err: 0
rx37_mpwqe_filler_cqes: 0
rx37_mpwqe_filler_strides: 0
rx37_buff_alloc_err: 0
rx37_cqe_compress_blks: 0
rx37_cqe_compress_pkts: 0
rx37_page_reuse: 0
rx37_cache_reuse: 0
rx37_cache_full: 0
rx37_cache_empty: 2560
rx37_cache_busy: 0
rx37_cache_waive: 0
rx37_congst_umr: 0
rx37_arfs_err: 0
rx37_xdp_tx_xmit: 0
rx37_xdp_tx_full: 0
rx37_xdp_tx_err: 0
rx37_xdp_tx_cqes: 0
rx38_packets: 0
rx38_bytes: 0
rx38_csum_complete: 0
rx38_csum_unnecessary: 0
rx38_csum_unnecessary_inner: 0
rx38_csum_none: 0
rx38_xdp_drop: 0
rx38_xdp_redirect: 0
rx38_lro_packets: 0
rx38_lro_bytes: 0
rx38_ecn_mark: 0
rx38_removed_vlan_packets: 0
rx38_wqe_err: 0
rx38_mpwqe_filler_cqes: 0
rx38_mpwqe_filler_strides: 0
rx38_buff_alloc_err: 0
rx38_cqe_compress_blks: 0
rx38_cqe_compress_pkts: 0
rx38_page_reuse: 0
rx38_cache_reuse: 0
rx38_cache_full: 0
rx38_cache_empty: 2560
rx38_cache_busy: 0
rx38_cache_waive: 0
rx38_congst_umr: 0
rx38_arfs_err: 0
rx38_xdp_tx_xmit: 0
rx38_xdp_tx_full: 0
rx38_xdp_tx_err: 0
rx38_xdp_tx_cqes: 0
rx39_packets: 0
rx39_bytes: 0
rx39_csum_complete: 0
rx39_csum_unnecessary: 0
rx39_csum_unnecessary_inner: 0
rx39_csum_none: 0
rx39_xdp_drop: 0
rx39_xdp_redirect: 0
rx39_lro_packets: 0
rx39_lro_bytes: 0
rx39_ecn_mark: 0
rx39_removed_vlan_packets: 0
rx39_wqe_err: 0
rx39_mpwqe_filler_cqes: 0
rx39_mpwqe_filler_strides: 0
rx39_buff_alloc_err: 0
rx39_cqe_compress_blks: 0
rx39_cqe_compress_pkts: 0
rx39_page_reuse: 0
rx39_cache_reuse: 0
rx39_cache_full: 0
rx39_cache_empty: 2560
rx39_cache_busy: 0
rx39_cache_waive: 0
rx39_congst_umr: 0
rx39_arfs_err: 0
rx39_xdp_tx_xmit: 0
rx39_xdp_tx_full: 0
rx39_xdp_tx_err: 0
rx39_xdp_tx_cqes: 0
rx40_packets: 0
rx40_bytes: 0
rx40_csum_complete: 0
rx40_csum_unnecessary: 0
rx40_csum_unnecessary_inner: 0
rx40_csum_none: 0
rx40_xdp_drop: 0
rx40_xdp_redirect: 0
rx40_lro_packets: 0
rx40_lro_bytes: 0
rx40_ecn_mark: 0
rx40_removed_vlan_packets: 0
rx40_wqe_err: 0
rx40_mpwqe_filler_cqes: 0
rx40_mpwqe_filler_strides: 0
rx40_buff_alloc_err: 0
rx40_cqe_compress_blks: 0
rx40_cqe_compress_pkts: 0
rx40_page_reuse: 0
rx40_cache_reuse: 0
rx40_cache_full: 0
rx40_cache_empty: 2560
rx40_cache_busy: 0
rx40_cache_waive: 0
rx40_congst_umr: 0
rx40_arfs_err: 0
rx40_xdp_tx_xmit: 0
rx40_xdp_tx_full: 0
rx40_xdp_tx_err: 0
rx40_xdp_tx_cqes: 0
rx41_packets: 0
rx41_bytes: 0
rx41_csum_complete: 0
rx41_csum_unnecessary: 0
rx41_csum_unnecessary_inner: 0
rx41_csum_none: 0
rx41_xdp_drop: 0
rx41_xdp_redirect: 0
rx41_lro_packets: 0
rx41_lro_bytes: 0
rx41_ecn_mark: 0
rx41_removed_vlan_packets: 0
rx41_wqe_err: 0
rx41_mpwqe_filler_cqes: 0
rx41_mpwqe_filler_strides: 0
rx41_buff_alloc_err: 0
rx41_cqe_compress_blks: 0
rx41_cqe_compress_pkts: 0
rx41_page_reuse: 0
rx41_cache_reuse: 0
rx41_cache_full: 0
rx41_cache_empty: 2560
rx41_cache_busy: 0
rx41_cache_waive: 0
rx41_congst_umr: 0
rx41_arfs_err: 0
rx41_xdp_tx_xmit: 0
rx41_xdp_tx_full: 0
rx41_xdp_tx_err: 0
rx41_xdp_tx_cqes: 0
rx42_packets: 0
rx42_bytes: 0
rx42_csum_complete: 0
rx42_csum_unnecessary: 0
rx42_csum_unnecessary_inner: 0
rx42_csum_none: 0
rx42_xdp_drop: 0
rx42_xdp_redirect: 0
rx42_lro_packets: 0
rx42_lro_bytes: 0
rx42_ecn_mark: 0
rx42_removed_vlan_packets: 0
rx42_wqe_err: 0
rx42_mpwqe_filler_cqes: 0
rx42_mpwqe_filler_strides: 0
rx42_buff_alloc_err: 0
rx42_cqe_compress_blks: 0
rx42_cqe_compress_pkts: 0
rx42_page_reuse: 0
rx42_cache_reuse: 0
rx42_cache_full: 0
rx42_cache_empty: 2560
rx42_cache_busy: 0
rx42_cache_waive: 0
rx42_congst_umr: 0
rx42_arfs_err: 0
rx42_xdp_tx_xmit: 0
rx42_xdp_tx_full: 0
rx42_xdp_tx_err: 0
rx42_xdp_tx_cqes: 0
rx43_packets: 0
rx43_bytes: 0
rx43_csum_complete: 0
rx43_csum_unnecessary: 0
rx43_csum_unnecessary_inner: 0
rx43_csum_none: 0
rx43_xdp_drop: 0
rx43_xdp_redirect: 0
rx43_lro_packets: 0
rx43_lro_bytes: 0
rx43_ecn_mark: 0
rx43_removed_vlan_packets: 0
rx43_wqe_err: 0
rx43_mpwqe_filler_cqes: 0
rx43_mpwqe_filler_strides: 0
rx43_buff_alloc_err: 0
rx43_cqe_compress_blks: 0
rx43_cqe_compress_pkts: 0
rx43_page_reuse: 0
rx43_cache_reuse: 0
rx43_cache_full: 0
rx43_cache_empty: 2560
rx43_cache_busy: 0
rx43_cache_waive: 0
rx43_congst_umr: 0
rx43_arfs_err: 0
rx43_xdp_tx_xmit: 0
rx43_xdp_tx_full: 0
rx43_xdp_tx_err: 0
rx43_xdp_tx_cqes: 0
rx44_packets: 0
rx44_bytes: 0
rx44_csum_complete: 0
rx44_csum_unnecessary: 0
rx44_csum_unnecessary_inner: 0
rx44_csum_none: 0
rx44_xdp_drop: 0
rx44_xdp_redirect: 0
rx44_lro_packets: 0
rx44_lro_bytes: 0
rx44_ecn_mark: 0
rx44_removed_vlan_packets: 0
rx44_wqe_err: 0
rx44_mpwqe_filler_cqes: 0
rx44_mpwqe_filler_strides: 0
rx44_buff_alloc_err: 0
rx44_cqe_compress_blks: 0
rx44_cqe_compress_pkts: 0
rx44_page_reuse: 0
rx44_cache_reuse: 0
rx44_cache_full: 0
rx44_cache_empty: 2560
rx44_cache_busy: 0
rx44_cache_waive: 0
rx44_congst_umr: 0
rx44_arfs_err: 0
rx44_xdp_tx_xmit: 0
rx44_xdp_tx_full: 0
rx44_xdp_tx_err: 0
rx44_xdp_tx_cqes: 0
rx45_packets: 0
rx45_bytes: 0
rx45_csum_complete: 0
rx45_csum_unnecessary: 0
rx45_csum_unnecessary_inner: 0
rx45_csum_none: 0
rx45_xdp_drop: 0
rx45_xdp_redirect: 0
rx45_lro_packets: 0
rx45_lro_bytes: 0
rx45_ecn_mark: 0
rx45_removed_vlan_packets: 0
rx45_wqe_err: 0
rx45_mpwqe_filler_cqes: 0
rx45_mpwqe_filler_strides: 0
rx45_buff_alloc_err: 0
rx45_cqe_compress_blks: 0
rx45_cqe_compress_pkts: 0
rx45_page_reuse: 0
rx45_cache_reuse: 0
rx45_cache_full: 0
rx45_cache_empty: 2560
rx45_cache_busy: 0
rx45_cache_waive: 0
rx45_congst_umr: 0
rx45_arfs_err: 0
rx45_xdp_tx_xmit: 0
rx45_xdp_tx_full: 0
rx45_xdp_tx_err: 0
rx45_xdp_tx_cqes: 0
rx46_packets: 0
rx46_bytes: 0
rx46_csum_complete: 0
rx46_csum_unnecessary: 0
rx46_csum_unnecessary_inner: 0
rx46_csum_none: 0
rx46_xdp_drop: 0
rx46_xdp_redirect: 0
rx46_lro_packets: 0
rx46_lro_bytes: 0
rx46_ecn_mark: 0
rx46_removed_vlan_packets: 0
rx46_wqe_err: 0
rx46_mpwqe_filler_cqes: 0
rx46_mpwqe_filler_strides: 0
rx46_buff_alloc_err: 0
rx46_cqe_compress_blks: 0
rx46_cqe_compress_pkts: 0
rx46_page_reuse: 0
rx46_cache_reuse: 0
rx46_cache_full: 0
rx46_cache_empty: 2560
rx46_cache_busy: 0
rx46_cache_waive: 0
rx46_congst_umr: 0
rx46_arfs_err: 0
rx46_xdp_tx_xmit: 0
rx46_xdp_tx_full: 0
rx46_xdp_tx_err: 0
rx46_xdp_tx_cqes: 0
rx47_packets: 0
rx47_bytes: 0
rx47_csum_complete: 0
rx47_csum_unnecessary: 0
rx47_csum_unnecessary_inner: 0
rx47_csum_none: 0
rx47_xdp_drop: 0
rx47_xdp_redirect: 0
rx47_lro_packets: 0
rx47_lro_bytes: 0
rx47_ecn_mark: 0
rx47_removed_vlan_packets: 0
rx47_wqe_err: 0
rx47_mpwqe_filler_cqes: 0
rx47_mpwqe_filler_strides: 0
rx47_buff_alloc_err: 0
rx47_cqe_compress_blks: 0
rx47_cqe_compress_pkts: 0
rx47_page_reuse: 0
rx47_cache_reuse: 0
rx47_cache_full: 0
rx47_cache_empty: 2560
rx47_cache_busy: 0
rx47_cache_waive: 0
rx47_congst_umr: 0
rx47_arfs_err: 0
rx47_xdp_tx_xmit: 0
rx47_xdp_tx_full: 0
rx47_xdp_tx_err: 0
rx47_xdp_tx_cqes: 0
rx48_packets: 0
rx48_bytes: 0
rx48_csum_complete: 0
rx48_csum_unnecessary: 0
rx48_csum_unnecessary_inner: 0
rx48_csum_none: 0
rx48_xdp_drop: 0
rx48_xdp_redirect: 0
rx48_lro_packets: 0
rx48_lro_bytes: 0
rx48_ecn_mark: 0
rx48_removed_vlan_packets: 0
rx48_wqe_err: 0
rx48_mpwqe_filler_cqes: 0
rx48_mpwqe_filler_strides: 0
rx48_buff_alloc_err: 0
rx48_cqe_compress_blks: 0
rx48_cqe_compress_pkts: 0
rx48_page_reuse: 0
rx48_cache_reuse: 0
rx48_cache_full: 0
rx48_cache_empty: 2560
rx48_cache_busy: 0
rx48_cache_waive: 0
rx48_congst_umr: 0
rx48_arfs_err: 0
rx48_xdp_tx_xmit: 0
rx48_xdp_tx_full: 0
rx48_xdp_tx_err: 0
rx48_xdp_tx_cqes: 0
rx49_packets: 0
rx49_bytes: 0
rx49_csum_complete: 0
rx49_csum_unnecessary: 0
rx49_csum_unnecessary_inner: 0
rx49_csum_none: 0
rx49_xdp_drop: 0
rx49_xdp_redirect: 0
rx49_lro_packets: 0
rx49_lro_bytes: 0
rx49_ecn_mark: 0
rx49_removed_vlan_packets: 0
rx49_wqe_err: 0
rx49_mpwqe_filler_cqes: 0
rx49_mpwqe_filler_strides: 0
rx49_buff_alloc_err: 0
rx49_cqe_compress_blks: 0
rx49_cqe_compress_pkts: 0
rx49_page_reuse: 0
rx49_cache_reuse: 0
rx49_cache_full: 0
rx49_cache_empty: 2560
rx49_cache_busy: 0
rx49_cache_waive: 0
rx49_congst_umr: 0
rx49_arfs_err: 0
rx49_xdp_tx_xmit: 0
rx49_xdp_tx_full: 0
rx49_xdp_tx_err: 0
rx49_xdp_tx_cqes: 0
rx50_packets: 0
rx50_bytes: 0
rx50_csum_complete: 0
rx50_csum_unnecessary: 0
rx50_csum_unnecessary_inner: 0
rx50_csum_none: 0
rx50_xdp_drop: 0
rx50_xdp_redirect: 0
rx50_lro_packets: 0
rx50_lro_bytes: 0
rx50_ecn_mark: 0
rx50_removed_vlan_packets: 0
rx50_wqe_err: 0
rx50_mpwqe_filler_cqes: 0
rx50_mpwqe_filler_strides: 0
rx50_buff_alloc_err: 0
rx50_cqe_compress_blks: 0
rx50_cqe_compress_pkts: 0
rx50_page_reuse: 0
rx50_cache_reuse: 0
rx50_cache_full: 0
rx50_cache_empty: 2560
rx50_cache_busy: 0
rx50_cache_waive: 0
rx50_congst_umr: 0
rx50_arfs_err: 0
rx50_xdp_tx_xmit: 0
rx50_xdp_tx_full: 0
rx50_xdp_tx_err: 0
rx50_xdp_tx_cqes: 0
rx51_packets: 0
rx51_bytes: 0
rx51_csum_complete: 0
rx51_csum_unnecessary: 0
rx51_csum_unnecessary_inner: 0
rx51_csum_none: 0
rx51_xdp_drop: 0
rx51_xdp_redirect: 0
rx51_lro_packets: 0
rx51_lro_bytes: 0
rx51_ecn_mark: 0
rx51_removed_vlan_packets: 0
rx51_wqe_err: 0
rx51_mpwqe_filler_cqes: 0
rx51_mpwqe_filler_strides: 0
rx51_buff_alloc_err: 0
rx51_cqe_compress_blks: 0
rx51_cqe_compress_pkts: 0
rx51_page_reuse: 0
rx51_cache_reuse: 0
rx51_cache_full: 0
rx51_cache_empty: 2560
rx51_cache_busy: 0
rx51_cache_waive: 0
rx51_congst_umr: 0
rx51_arfs_err: 0
rx51_xdp_tx_xmit: 0
rx51_xdp_tx_full: 0
rx51_xdp_tx_err: 0
rx51_xdp_tx_cqes: 0
rx52_packets: 0
rx52_bytes: 0
rx52_csum_complete: 0
rx52_csum_unnecessary: 0
rx52_csum_unnecessary_inner: 0
rx52_csum_none: 0
rx52_xdp_drop: 0
rx52_xdp_redirect: 0
rx52_lro_packets: 0
rx52_lro_bytes: 0
rx52_ecn_mark: 0
rx52_removed_vlan_packets: 0
rx52_wqe_err: 0
rx52_mpwqe_filler_cqes: 0
rx52_mpwqe_filler_strides: 0
rx52_buff_alloc_err: 0
rx52_cqe_compress_blks: 0
rx52_cqe_compress_pkts: 0
rx52_page_reuse: 0
rx52_cache_reuse: 0
rx52_cache_full: 0
rx52_cache_empty: 2560
rx52_cache_busy: 0
rx52_cache_waive: 0
rx52_congst_umr: 0
rx52_arfs_err: 0
rx52_xdp_tx_xmit: 0
rx52_xdp_tx_full: 0
rx52_xdp_tx_err: 0
rx52_xdp_tx_cqes: 0
rx53_packets: 0
rx53_bytes: 0
rx53_csum_complete: 0
rx53_csum_unnecessary: 0
rx53_csum_unnecessary_inner: 0
rx53_csum_none: 0
rx53_xdp_drop: 0
rx53_xdp_redirect: 0
rx53_lro_packets: 0
rx53_lro_bytes: 0
rx53_ecn_mark: 0
rx53_removed_vlan_packets: 0
rx53_wqe_err: 0
rx53_mpwqe_filler_cqes: 0
rx53_mpwqe_filler_strides: 0
rx53_buff_alloc_err: 0
rx53_cqe_compress_blks: 0
rx53_cqe_compress_pkts: 0
rx53_page_reuse: 0
rx53_cache_reuse: 0
rx53_cache_full: 0
rx53_cache_empty: 2560
rx53_cache_busy: 0
rx53_cache_waive: 0
rx53_congst_umr: 0
rx53_arfs_err: 0
rx53_xdp_tx_xmit: 0
rx53_xdp_tx_full: 0
rx53_xdp_tx_err: 0
rx53_xdp_tx_cqes: 0
rx54_packets: 0
rx54_bytes: 0
rx54_csum_complete: 0
rx54_csum_unnecessary: 0
rx54_csum_unnecessary_inner: 0
rx54_csum_none: 0
rx54_xdp_drop: 0
rx54_xdp_redirect: 0
rx54_lro_packets: 0
rx54_lro_bytes: 0
rx54_ecn_mark: 0
rx54_removed_vlan_packets: 0
rx54_wqe_err: 0
rx54_mpwqe_filler_cqes: 0
rx54_mpwqe_filler_strides: 0
rx54_buff_alloc_err: 0
rx54_cqe_compress_blks: 0
rx54_cqe_compress_pkts: 0
rx54_page_reuse: 0
rx54_cache_reuse: 0
rx54_cache_full: 0
rx54_cache_empty: 2560
rx54_cache_busy: 0
rx54_cache_waive: 0
rx54_congst_umr: 0
rx54_arfs_err: 0
rx54_xdp_tx_xmit: 0
rx54_xdp_tx_full: 0
rx54_xdp_tx_err: 0
rx54_xdp_tx_cqes: 0
rx55_packets: 0
rx55_bytes: 0
rx55_csum_complete: 0
rx55_csum_unnecessary: 0
rx55_csum_unnecessary_inner: 0
rx55_csum_none: 0
rx55_xdp_drop: 0
rx55_xdp_redirect: 0
rx55_lro_packets: 0
rx55_lro_bytes: 0
rx55_ecn_mark: 0
rx55_removed_vlan_packets: 0
rx55_wqe_err: 0
rx55_mpwqe_filler_cqes: 0
rx55_mpwqe_filler_strides: 0
rx55_buff_alloc_err: 0
rx55_cqe_compress_blks: 0
rx55_cqe_compress_pkts: 0
rx55_page_reuse: 0
rx55_cache_reuse: 0
rx55_cache_full: 0
rx55_cache_empty: 2560
rx55_cache_busy: 0
rx55_cache_waive: 0
rx55_congst_umr: 0
rx55_arfs_err: 0
rx55_xdp_tx_xmit: 0
rx55_xdp_tx_full: 0
rx55_xdp_tx_err: 0
rx55_xdp_tx_cqes: 0
tx0_packets: 5868971166
tx0_bytes: 7384241881537
tx0_tso_packets: 1005089669
tx0_tso_bytes: 5138882499687
tx0_tso_inner_packets: 0
tx0_tso_inner_bytes: 0
tx0_csum_partial: 1405330470
tx0_csum_partial_inner: 0
tx0_added_vlan_packets: 3247061022
tx0_nop: 83925216
tx0_csum_none: 1841730552
tx0_stopped: 0
tx0_dropped: 0
tx0_xmit_more: 29664303
tx0_recover: 0
tx0_cqes: 3217398842
tx0_wake: 0
tx0_cqe_err: 0
tx1_packets: 5599378674
tx1_bytes: 7272236466962
tx1_tso_packets: 1024612268
tx1_tso_bytes: 5244192050917
tx1_tso_inner_packets: 0
tx1_tso_inner_bytes: 0
tx1_csum_partial: 1438007932
tx1_csum_partial_inner: 0
tx1_added_vlan_packets: 2919765857
tx1_nop: 79661231
tx1_csum_none: 1481757925
tx1_stopped: 0
tx1_dropped: 0
tx1_xmit_more: 29485355
tx1_recover: 0
tx1_cqes: 2890282176
tx1_wake: 0
tx1_cqe_err: 0
tx2_packets: 5413821094
tx2_bytes: 7033951631334
tx2_tso_packets: 1002868589
tx2_tso_bytes: 5089549008985
tx2_tso_inner_packets: 0
tx2_tso_inner_bytes: 0
tx2_csum_partial: 1404186175
tx2_csum_partial_inner: 0
tx2_added_vlan_packets: 2822670460
tx2_nop: 77115408
tx2_csum_none: 1418484285
tx2_stopped: 0
tx2_dropped: 0
tx2_xmit_more: 29321129
tx2_recover: 0
tx2_cqes: 2793351019
tx2_wake: 0
tx2_cqe_err: 0
tx3_packets: 5479609727
tx3_bytes: 7116904107659
tx3_tso_packets: 1002992639
tx3_tso_bytes: 5154225081979
tx3_tso_inner_packets: 0
tx3_tso_inner_bytes: 0
tx3_csum_partial: 1415739849
tx3_csum_partial_inner: 0
tx3_added_vlan_packets: 2842823811
tx3_nop: 78060813
tx3_csum_none: 1427083971
tx3_stopped: 0
tx3_dropped: 0
tx3_xmit_more: 28575040
tx3_recover: 0
tx3_cqes: 2814250785
tx3_wake: 0
tx3_cqe_err: 0
tx4_packets: 5508297397
tx4_bytes: 7127659369902
tx4_tso_packets: 1007356432
tx4_tso_bytes: 5145975736034
tx4_tso_inner_packets: 0
tx4_tso_inner_bytes: 0
tx4_csum_partial: 1411271000
tx4_csum_partial_inner: 0
tx4_added_vlan_packets: 2882086825
tx4_nop: 78433610
tx4_csum_none: 1470815825
tx4_stopped: 0
tx4_dropped: 0
tx4_xmit_more: 28632444
tx4_recover: 0
tx4_cqes: 2853456464
tx4_wake: 0
tx4_cqe_err: 0
tx5_packets: 5513864156
tx5_bytes: 7165864145517
tx5_tso_packets: 1014046485
tx5_tso_bytes: 5192635614477
tx5_tso_inner_packets: 0
tx5_tso_inner_bytes: 0
tx5_csum_partial: 1420810473
tx5_csum_partial_inner: 0
tx5_added_vlan_packets: 2861370556
tx5_nop: 78481355
tx5_csum_none: 1440560083
tx5_stopped: 0
tx5_dropped: 0
tx5_xmit_more: 28222467
tx5_recover: 0
tx5_cqes: 2833149758
tx5_wake: 0
tx5_cqe_err: 0
tx6_packets: 5560724761
tx6_bytes: 7210309972086
tx6_tso_packets: 994050514
tx6_tso_bytes: 5171393741595
tx6_tso_inner_packets: 0
tx6_tso_inner_bytes: 0
tx6_csum_partial: 1414303265
tx6_csum_partial_inner: 0
tx6_added_vlan_packets: 2905794177
tx6_nop: 79353318
tx6_csum_none: 1491490912
tx6_stopped: 0
tx6_dropped: 0
tx6_xmit_more: 31246664
tx6_recover: 0
tx6_cqes: 2874549217
tx6_wake: 0
tx6_cqe_err: 0
tx7_packets: 5557594170
tx7_bytes: 7223138778685
tx7_tso_packets: 1013475396
tx7_tso_bytes: 5241530065484
tx7_tso_inner_packets: 0
tx7_tso_inner_bytes: 0
tx7_csum_partial: 1438604314
tx7_csum_partial_inner: 0
tx7_added_vlan_packets: 2873917552
tx7_nop: 79057059
tx7_csum_none: 1435313239
tx7_stopped: 0
tx7_dropped: 0
tx7_xmit_more: 29258761
tx7_recover: 0
tx7_cqes: 2844660578
tx7_wake: 0
tx7_cqe_err: 0
tx8_packets: 5521254733
tx8_bytes: 7208043146297
tx8_tso_packets: 1014670801
tx8_tso_bytes: 5185842447246
tx8_tso_inner_packets: 0
tx8_tso_inner_bytes: 0
tx8_csum_partial: 1431631562
tx8_csum_partial_inner: 0
tx8_added_vlan_packets: 2872641129
tx8_nop: 78545776
tx8_csum_none: 1441009567
tx8_stopped: 0
tx8_dropped: 0
tx8_xmit_more: 29106291
tx8_recover: 0
tx8_cqes: 2843536748
tx8_wake: 0
tx8_cqe_err: 0
tx9_packets: 5528889957
tx9_bytes: 7191793816058
tx9_tso_packets: 1015955476
tx9_tso_bytes: 5207232047828
tx9_tso_inner_packets: 0
tx9_tso_inner_bytes: 0
tx9_csum_partial: 1421266796
tx9_csum_partial_inner: 0
tx9_added_vlan_packets: 2869523921
tx9_nop: 78586218
tx9_csum_none: 1448257125
tx9_stopped: 0
tx9_dropped: 0
tx9_xmit_more: 29483347
tx9_recover: 0
tx9_cqes: 2840042245
tx9_wake: 0
tx9_cqe_err: 0
tx10_packets: 5556351222
tx10_bytes: 7254798330757
tx10_tso_packets: 1028554460
tx10_tso_bytes: 5246179615774
tx10_tso_inner_packets: 0
tx10_tso_inner_bytes: 0
tx10_csum_partial: 1430459021
tx10_csum_partial_inner: 0
tx10_added_vlan_packets: 2881683382
tx10_nop: 79139584
tx10_csum_none: 1451224361
tx10_stopped: 0
tx10_dropped: 0
tx10_xmit_more: 29217190
tx10_recover: 0
tx10_cqes: 2852467898
tx10_wake: 0
tx10_cqe_err: 0
tx11_packets: 5455631854
tx11_bytes: 7061121713772
tx11_tso_packets: 992133383
tx11_tso_bytes: 5089419722682
tx11_tso_inner_packets: 0
tx11_tso_inner_bytes: 0
tx11_csum_partial: 1395542033
tx11_csum_partial_inner: 0
tx11_added_vlan_packets: 2852589093
tx11_nop: 77799857
tx11_csum_none: 1457047060
tx11_stopped: 0
tx11_dropped: 0
tx11_xmit_more: 29559927
tx11_recover: 0
tx11_cqes: 2823031110
tx11_wake: 0
tx11_cqe_err: 0
tx12_packets: 5488286808
tx12_bytes: 7137087569303
tx12_tso_packets: 1006435537
tx12_tso_bytes: 5163371416750
tx12_tso_inner_packets: 0
tx12_tso_inner_bytes: 0
tx12_csum_partial: 1414799411
tx12_csum_partial_inner: 0
tx12_added_vlan_packets: 2841679543
tx12_nop: 78387039
tx12_csum_none: 1426880132
tx12_stopped: 0
tx12_dropped: 0
tx12_xmit_more: 28607526
tx12_recover: 0
tx12_cqes: 2813073557
tx12_wake: 0
tx12_cqe_err: 0
tx13_packets: 5594132290
tx13_bytes: 7251106284829
tx13_tso_packets: 1035172061
tx13_tso_bytes: 5251200286298
tx13_tso_inner_packets: 0
tx13_tso_inner_bytes: 0
tx13_csum_partial: 1443665981
tx13_csum_partial_inner: 0
tx13_added_vlan_packets: 2916604799
tx13_nop: 79670465
tx13_csum_none: 1472938818
tx13_stopped: 0
tx13_dropped: 0
tx13_xmit_more: 27797067
tx13_recover: 0
tx13_cqes: 2888809352
tx13_wake: 0
tx13_cqe_err: 0
tx14_packets: 5548790952
tx14_bytes: 7194211868411
tx14_tso_packets: 1021015561
tx14_tso_bytes: 5231483708869
tx14_tso_inner_packets: 0
tx14_tso_inner_bytes: 0
tx14_csum_partial: 1427711576
tx14_csum_partial_inner: 0
tx14_added_vlan_packets: 2875288572
tx14_nop: 78900224
tx14_csum_none: 1447576996
tx14_stopped: 0
tx14_dropped: 0
tx14_xmit_more: 30003496
tx14_recover: 0
tx14_cqes: 2845286732
tx14_wake: 0
tx14_cqe_err: 0
tx15_packets: 5609310963
tx15_bytes: 7271380831798
tx15_tso_packets: 1027830118
tx15_tso_bytes: 5229697431506
tx15_tso_inner_packets: 0
tx15_tso_inner_bytes: 0
tx15_csum_partial: 1429209941
tx15_csum_partial_inner: 0
tx15_added_vlan_packets: 2940315402
tx15_nop: 79950883
tx15_csum_none: 1511105462
tx15_stopped: 0
tx15_dropped: 0
tx15_xmit_more: 28820740
tx15_recover: 0
tx15_cqes: 2911496633
tx15_wake: 0
tx15_cqe_err: 0
tx16_packets: 4465363036
tx16_bytes: 5769771803704
tx16_tso_packets: 817101913
tx16_tso_bytes: 4180172833814
tx16_tso_inner_packets: 0
tx16_tso_inner_bytes: 0
tx16_csum_partial: 1136731404
tx16_csum_partial_inner: 0
tx16_added_vlan_packets: 2332178232
tx16_nop: 63458573
tx16_csum_none: 1195446828
tx16_stopped: 0
tx16_dropped: 0
tx16_xmit_more: 23756254
tx16_recover: 0
tx16_cqes: 2308423025
tx16_wake: 0
tx16_cqe_err: 0
tx17_packets: 4380386348
tx17_bytes: 5708702994526
tx17_tso_packets: 813638023
tx17_tso_bytes: 4130806014947
tx17_tso_inner_packets: 0
tx17_tso_inner_bytes: 0
tx17_csum_partial: 1133007164
tx17_csum_partial_inner: 0
tx17_added_vlan_packets: 2277314787
tx17_nop: 62377372
tx17_csum_none: 1144307623
tx17_stopped: 0
tx17_dropped: 0
tx17_xmit_more: 23731361
tx17_recover: 0
tx17_cqes: 2253584638
tx17_wake: 0
tx17_cqe_err: 0
tx18_packets: 4450359743
tx18_bytes: 5758968674820
tx18_tso_packets: 815791601
tx18_tso_bytes: 4179942688909
tx18_tso_inner_packets: 0
tx18_tso_inner_bytes: 0
tx18_csum_partial: 1137649257
tx18_csum_partial_inner: 0
tx18_added_vlan_packets: 2314556550
tx18_nop: 63271085
tx18_csum_none: 1176907293
tx18_stopped: 0
tx18_dropped: 0
tx18_xmit_more: 23055770
tx18_recover: 0
tx18_cqes: 2291501928
tx18_wake: 0
tx18_cqe_err: 0
tx19_packets: 4596064378
tx19_bytes: 5916675706535
tx19_tso_packets: 825788649
tx19_tso_bytes: 4208046929921
tx19_tso_inner_packets: 0
tx19_tso_inner_bytes: 0
tx19_csum_partial: 1150666569
tx19_csum_partial_inner: 0
tx19_added_vlan_packets: 2450567026
tx19_nop: 65468504
tx19_csum_none: 1299900457
tx19_stopped: 0
tx19_dropped: 0
tx19_xmit_more: 23846250
tx19_recover: 0
tx19_cqes: 2426722127
tx19_wake: 0
tx19_cqe_err: 0
tx20_packets: 4424935388
tx20_bytes: 5757631205901
tx20_tso_packets: 804875006
tx20_tso_bytes: 4156262736109
tx20_tso_inner_packets: 0
tx20_tso_inner_bytes: 0
tx20_csum_partial: 1134144916
tx20_csum_partial_inner: 0
tx20_added_vlan_packets: 2294839665
tx20_nop: 63023986
tx20_csum_none: 1160694749
tx20_stopped: 0
tx20_dropped: 0
tx20_xmit_more: 23393201
tx20_recover: 0
tx20_cqes: 2271447623
tx20_wake: 0
tx20_cqe_err: 0
tx21_packets: 4595062285
tx21_bytes: 5958671993467
tx21_tso_packets: 821936215
tx21_tso_bytes: 4187977870684
tx21_tso_inner_packets: 0
tx21_tso_inner_bytes: 0
tx21_csum_partial: 1143339787
tx21_csum_partial_inner: 0
tx21_added_vlan_packets: 2457167412
tx21_nop: 65697763
tx21_csum_none: 1313827625
tx21_stopped: 0
tx21_dropped: 0
tx21_xmit_more: 23858345
tx21_recover: 0
tx21_cqes: 2433310348
tx21_wake: 0
tx21_cqe_err: 0
tx22_packets: 4664446513
tx22_bytes: 5931429292082
tx22_tso_packets: 814457881
tx22_tso_bytes: 4148607956533
tx22_tso_inner_packets: 0
tx22_tso_inner_bytes: 0
tx22_csum_partial: 1127284783
tx22_csum_partial_inner: 0
tx22_added_vlan_packets: 2548650146
tx22_nop: 66299909
tx22_csum_none: 1421365363
tx22_stopped: 0
tx22_dropped: 0
tx22_xmit_more: 23800911
tx22_recover: 0
tx22_cqes: 2524850415
tx22_wake: 0
tx22_cqe_err: 0
tx23_packets: 4416221747
tx23_bytes: 5721472587985
tx23_tso_packets: 823538520
tx23_tso_bytes: 4163520218617
tx23_tso_inner_packets: 0
tx23_tso_inner_bytes: 0
tx23_csum_partial: 1135996006
tx23_csum_partial_inner: 0
tx23_added_vlan_packets: 2292404120
tx23_nop: 62709432
tx23_csum_none: 1156408114
tx23_stopped: 0
tx23_dropped: 0
tx23_xmit_more: 22299889
tx23_recover: 0
tx23_cqes: 2270105487
tx23_wake: 0
tx23_cqe_err: 0
tx24_packets: 4420014824
tx24_bytes: 5740767318521
tx24_tso_packets: 820838072
tx24_tso_bytes: 4183722948422
tx24_tso_inner_packets: 0
tx24_tso_inner_bytes: 0
tx24_csum_partial: 1138070059
tx24_csum_partial_inner: 0
tx24_added_vlan_packets: 2289043946
tx24_nop: 62797341
tx24_csum_none: 1150973887
tx24_stopped: 0
tx24_dropped: 0
tx24_xmit_more: 22744690
tx24_recover: 0
tx24_cqes: 2266300568
tx24_wake: 0
tx24_cqe_err: 0
tx25_packets: 4413225545
tx25_bytes: 5716162617155
tx25_tso_packets: 808274341
tx25_tso_bytes: 4138408857714
tx25_tso_inner_packets: 0
tx25_tso_inner_bytes: 0
tx25_csum_partial: 1134587898
tx25_csum_partial_inner: 0
tx25_added_vlan_packets: 2297149310
tx25_nop: 62958238
tx25_csum_none: 1162561412
tx25_stopped: 0
tx25_dropped: 0
tx25_xmit_more: 24463552
tx25_recover: 0
tx25_cqes: 2272686971
tx25_wake: 0
tx25_cqe_err: 0
tx26_packets: 4524907591
tx26_bytes: 5865394280699
tx26_tso_packets: 807270022
tx26_tso_bytes: 4148754705317
tx26_tso_inner_packets: 0
tx26_tso_inner_bytes: 0
tx26_csum_partial: 1130306933
tx26_csum_partial_inner: 0
tx26_added_vlan_packets: 2402682460
tx26_nop: 64474322
tx26_csum_none: 1272375527
tx26_stopped: 1
tx26_dropped: 0
tx26_xmit_more: 23316186
tx26_recover: 0
tx26_cqes: 2379367502
tx26_wake: 1
tx26_cqe_err: 0
tx27_packets: 4376114969
tx27_bytes: 5683551238304
tx27_tso_packets: 809344829
tx27_tso_bytes: 4124331859270
tx27_tso_inner_packets: 0
tx27_tso_inner_bytes: 0
tx27_csum_partial: 1124954937
tx27_csum_partial_inner: 0
tx27_added_vlan_packets: 2267871300
tx27_nop: 62213214
tx27_csum_none: 1142916363
tx27_stopped: 0
tx27_dropped: 0
tx27_xmit_more: 23369974
tx27_recover: 0
tx27_cqes: 2244502686
tx27_wake: 0
tx27_cqe_err: 0
tx28_packets: 3
tx28_bytes: 266
tx28_tso_packets: 0
tx28_tso_bytes: 0
tx28_tso_inner_packets: 0
tx28_tso_inner_bytes: 0
tx28_csum_partial: 0
tx28_csum_partial_inner: 0
tx28_added_vlan_packets: 0
tx28_nop: 0
tx28_csum_none: 3
tx28_stopped: 0
tx28_dropped: 0
tx28_xmit_more: 0
tx28_recover: 0
tx28_cqes: 3
tx28_wake: 0
tx28_cqe_err: 0
tx29_packets: 0
tx29_bytes: 0
tx29_tso_packets: 0
tx29_tso_bytes: 0
tx29_tso_inner_packets: 0
tx29_tso_inner_bytes: 0
tx29_csum_partial: 0
tx29_csum_partial_inner: 0
tx29_added_vlan_packets: 0
tx29_nop: 0
tx29_csum_none: 0
tx29_stopped: 0
tx29_dropped: 0
tx29_xmit_more: 0
tx29_recover: 0
tx29_cqes: 0
tx29_wake: 0
tx29_cqe_err: 0
tx30_packets: 0
tx30_bytes: 0
tx30_tso_packets: 0
tx30_tso_bytes: 0
tx30_tso_inner_packets: 0
tx30_tso_inner_bytes: 0
tx30_csum_partial: 0
tx30_csum_partial_inner: 0
tx30_added_vlan_packets: 0
tx30_nop: 0
tx30_csum_none: 0
tx30_stopped: 0
tx30_dropped: 0
tx30_xmit_more: 0
tx30_recover: 0
tx30_cqes: 0
tx30_wake: 0
tx30_cqe_err: 0
tx31_packets: 0
tx31_bytes: 0
tx31_tso_packets: 0
tx31_tso_bytes: 0
tx31_tso_inner_packets: 0
tx31_tso_inner_bytes: 0
tx31_csum_partial: 0
tx31_csum_partial_inner: 0
tx31_added_vlan_packets: 0
tx31_nop: 0
tx31_csum_none: 0
tx31_stopped: 0
tx31_dropped: 0
tx31_xmit_more: 0
tx31_recover: 0
tx31_cqes: 0
tx31_wake: 0
tx31_cqe_err: 0
tx32_packets: 0
tx32_bytes: 0
tx32_tso_packets: 0
tx32_tso_bytes: 0
tx32_tso_inner_packets: 0
tx32_tso_inner_bytes: 0
tx32_csum_partial: 0
tx32_csum_partial_inner: 0
tx32_added_vlan_packets: 0
tx32_nop: 0
tx32_csum_none: 0
tx32_stopped: 0
tx32_dropped: 0
tx32_xmit_more: 0
tx32_recover: 0
tx32_cqes: 0
tx32_wake: 0
tx32_cqe_err: 0
tx33_packets: 0
tx33_bytes: 0
tx33_tso_packets: 0
tx33_tso_bytes: 0
tx33_tso_inner_packets: 0
tx33_tso_inner_bytes: 0
tx33_csum_partial: 0
tx33_csum_partial_inner: 0
tx33_added_vlan_packets: 0
tx33_nop: 0
tx33_csum_none: 0
tx33_stopped: 0
tx33_dropped: 0
tx33_xmit_more: 0
tx33_recover: 0
tx33_cqes: 0
tx33_wake: 0
tx33_cqe_err: 0
tx34_packets: 0
tx34_bytes: 0
tx34_tso_packets: 0
tx34_tso_bytes: 0
tx34_tso_inner_packets: 0
tx34_tso_inner_bytes: 0
tx34_csum_partial: 0
tx34_csum_partial_inner: 0
tx34_added_vlan_packets: 0
tx34_nop: 0
tx34_csum_none: 0
tx34_stopped: 0
tx34_dropped: 0
tx34_xmit_more: 0
tx34_recover: 0
tx34_cqes: 0
tx34_wake: 0
tx34_cqe_err: 0
tx35_packets: 0
tx35_bytes: 0
tx35_tso_packets: 0
tx35_tso_bytes: 0
tx35_tso_inner_packets: 0
tx35_tso_inner_bytes: 0
tx35_csum_partial: 0
tx35_csum_partial_inner: 0
tx35_added_vlan_packets: 0
tx35_nop: 0
tx35_csum_none: 0
tx35_stopped: 0
tx35_dropped: 0
tx35_xmit_more: 0
tx35_recover: 0
tx35_cqes: 0
tx35_wake: 0
tx35_cqe_err: 0
tx36_packets: 0
tx36_bytes: 0
tx36_tso_packets: 0
tx36_tso_bytes: 0
tx36_tso_inner_packets: 0
tx36_tso_inner_bytes: 0
tx36_csum_partial: 0
tx36_csum_partial_inner: 0
tx36_added_vlan_packets: 0
tx36_nop: 0
tx36_csum_none: 0
tx36_stopped: 0
tx36_dropped: 0
tx36_xmit_more: 0
tx36_recover: 0
tx36_cqes: 0
tx36_wake: 0
tx36_cqe_err: 0
tx37_packets: 0
tx37_bytes: 0
tx37_tso_packets: 0
tx37_tso_bytes: 0
tx37_tso_inner_packets: 0
tx37_tso_inner_bytes: 0
tx37_csum_partial: 0
tx37_csum_partial_inner: 0
tx37_added_vlan_packets: 0
tx37_nop: 0
tx37_csum_none: 0
tx37_stopped: 0
tx37_dropped: 0
tx37_xmit_more: 0
tx37_recover: 0
tx37_cqes: 0
tx37_wake: 0
tx37_cqe_err: 0
tx38_packets: 0
tx38_bytes: 0
tx38_tso_packets: 0
tx38_tso_bytes: 0
tx38_tso_inner_packets: 0
tx38_tso_inner_bytes: 0
tx38_csum_partial: 0
tx38_csum_partial_inner: 0
tx38_added_vlan_packets: 0
tx38_nop: 0
tx38_csum_none: 0
tx38_stopped: 0
tx38_dropped: 0
tx38_xmit_more: 0
tx38_recover: 0
tx38_cqes: 0
tx38_wake: 0
tx38_cqe_err: 0
tx39_packets: 0
tx39_bytes: 0
tx39_tso_packets: 0
tx39_tso_bytes: 0
tx39_tso_inner_packets: 0
tx39_tso_inner_bytes: 0
tx39_csum_partial: 0
tx39_csum_partial_inner: 0
tx39_added_vlan_packets: 0
tx39_nop: 0
tx39_csum_none: 0
tx39_stopped: 0
tx39_dropped: 0
tx39_xmit_more: 0
tx39_recover: 0
tx39_cqes: 0
tx39_wake: 0
tx39_cqe_err: 0
tx40_packets: 0
tx40_bytes: 0
tx40_tso_packets: 0
tx40_tso_bytes: 0
tx40_tso_inner_packets: 0
tx40_tso_inner_bytes: 0
tx40_csum_partial: 0
tx40_csum_partial_inner: 0
tx40_added_vlan_packets: 0
tx40_nop: 0
tx40_csum_none: 0
tx40_stopped: 0
tx40_dropped: 0
tx40_xmit_more: 0
tx40_recover: 0
tx40_cqes: 0
tx40_wake: 0
tx40_cqe_err: 0
tx41_packets: 0
tx41_bytes: 0
tx41_tso_packets: 0
tx41_tso_bytes: 0
tx41_tso_inner_packets: 0
tx41_tso_inner_bytes: 0
tx41_csum_partial: 0
tx41_csum_partial_inner: 0
tx41_added_vlan_packets: 0
tx41_nop: 0
tx41_csum_none: 0
tx41_stopped: 0
tx41_dropped: 0
tx41_xmit_more: 0
tx41_recover: 0
tx41_cqes: 0
tx41_wake: 0
tx41_cqe_err: 0
tx42_packets: 0
tx42_bytes: 0
tx42_tso_packets: 0
tx42_tso_bytes: 0
tx42_tso_inner_packets: 0
tx42_tso_inner_bytes: 0
tx42_csum_partial: 0
tx42_csum_partial_inner: 0
tx42_added_vlan_packets: 0
tx42_nop: 0
tx42_csum_none: 0
tx42_stopped: 0
tx42_dropped: 0
tx42_xmit_more: 0
tx42_recover: 0
tx42_cqes: 0
tx42_wake: 0
tx42_cqe_err: 0
tx43_packets: 0
tx43_bytes: 0
tx43_tso_packets: 0
tx43_tso_bytes: 0
tx43_tso_inner_packets: 0
tx43_tso_inner_bytes: 0
tx43_csum_partial: 0
tx43_csum_partial_inner: 0
tx43_added_vlan_packets: 0
tx43_nop: 0
tx43_csum_none: 0
tx43_stopped: 0
tx43_dropped: 0
tx43_xmit_more: 0
tx43_recover: 0
tx43_cqes: 0
tx43_wake: 0
tx43_cqe_err: 0
tx44_packets: 0
tx44_bytes: 0
tx44_tso_packets: 0
tx44_tso_bytes: 0
tx44_tso_inner_packets: 0
tx44_tso_inner_bytes: 0
tx44_csum_partial: 0
tx44_csum_partial_inner: 0
tx44_added_vlan_packets: 0
tx44_nop: 0
tx44_csum_none: 0
tx44_stopped: 0
tx44_dropped: 0
tx44_xmit_more: 0
tx44_recover: 0
tx44_cqes: 0
tx44_wake: 0
tx44_cqe_err: 0
tx45_packets: 0
tx45_bytes: 0
tx45_tso_packets: 0
tx45_tso_bytes: 0
tx45_tso_inner_packets: 0
tx45_tso_inner_bytes: 0
tx45_csum_partial: 0
tx45_csum_partial_inner: 0
tx45_added_vlan_packets: 0
tx45_nop: 0
tx45_csum_none: 0
tx45_stopped: 0
tx45_dropped: 0
tx45_xmit_more: 0
tx45_recover: 0
tx45_cqes: 0
tx45_wake: 0
tx45_cqe_err: 0
tx46_packets: 0
tx46_bytes: 0
tx46_tso_packets: 0
tx46_tso_bytes: 0
tx46_tso_inner_packets: 0
tx46_tso_inner_bytes: 0
tx46_csum_partial: 0
tx46_csum_partial_inner: 0
tx46_added_vlan_packets: 0
tx46_nop: 0
tx46_csum_none: 0
tx46_stopped: 0
tx46_dropped: 0
tx46_xmit_more: 0
tx46_recover: 0
tx46_cqes: 0
tx46_wake: 0
tx46_cqe_err: 0
tx47_packets: 0
tx47_bytes: 0
tx47_tso_packets: 0
tx47_tso_bytes: 0
tx47_tso_inner_packets: 0
tx47_tso_inner_bytes: 0
tx47_csum_partial: 0
tx47_csum_partial_inner: 0
tx47_added_vlan_packets: 0
tx47_nop: 0
tx47_csum_none: 0
tx47_stopped: 0
tx47_dropped: 0
tx47_xmit_more: 0
tx47_recover: 0
tx47_cqes: 0
tx47_wake: 0
tx47_cqe_err: 0
tx48_packets: 0
tx48_bytes: 0
tx48_tso_packets: 0
tx48_tso_bytes: 0
tx48_tso_inner_packets: 0
tx48_tso_inner_bytes: 0
tx48_csum_partial: 0
tx48_csum_partial_inner: 0
tx48_added_vlan_packets: 0
tx48_nop: 0
tx48_csum_none: 0
tx48_stopped: 0
tx48_dropped: 0
tx48_xmit_more: 0
tx48_recover: 0
tx48_cqes: 0
tx48_wake: 0
tx48_cqe_err: 0
tx49_packets: 0
tx49_bytes: 0
tx49_tso_packets: 0
tx49_tso_bytes: 0
tx49_tso_inner_packets: 0
tx49_tso_inner_bytes: 0
tx49_csum_partial: 0
tx49_csum_partial_inner: 0
tx49_added_vlan_packets: 0
tx49_nop: 0
tx49_csum_none: 0
tx49_stopped: 0
tx49_dropped: 0
tx49_xmit_more: 0
tx49_recover: 0
tx49_cqes: 0
tx49_wake: 0
tx49_cqe_err: 0
tx50_packets: 0
tx50_bytes: 0
tx50_tso_packets: 0
tx50_tso_bytes: 0
tx50_tso_inner_packets: 0
tx50_tso_inner_bytes: 0
tx50_csum_partial: 0
tx50_csum_partial_inner: 0
tx50_added_vlan_packets: 0
tx50_nop: 0
tx50_csum_none: 0
tx50_stopped: 0
tx50_dropped: 0
tx50_xmit_more: 0
tx50_recover: 0
tx50_cqes: 0
tx50_wake: 0
tx50_cqe_err: 0
tx51_packets: 0
tx51_bytes: 0
tx51_tso_packets: 0
tx51_tso_bytes: 0
tx51_tso_inner_packets: 0
tx51_tso_inner_bytes: 0
tx51_csum_partial: 0
tx51_csum_partial_inner: 0
tx51_added_vlan_packets: 0
tx51_nop: 0
tx51_csum_none: 0
tx51_stopped: 0
tx51_dropped: 0
tx51_xmit_more: 0
tx51_recover: 0
tx51_cqes: 0
tx51_wake: 0
tx51_cqe_err: 0
tx52_packets: 0
tx52_bytes: 0
tx52_tso_packets: 0
tx52_tso_bytes: 0
tx52_tso_inner_packets: 0
tx52_tso_inner_bytes: 0
tx52_csum_partial: 0
tx52_csum_partial_inner: 0
tx52_added_vlan_packets: 0
tx52_nop: 0
tx52_csum_none: 0
tx52_stopped: 0
tx52_dropped: 0
tx52_xmit_more: 0
tx52_recover: 0
tx52_cqes: 0
tx52_wake: 0
tx52_cqe_err: 0
tx53_packets: 0
tx53_bytes: 0
tx53_tso_packets: 0
tx53_tso_bytes: 0
tx53_tso_inner_packets: 0
tx53_tso_inner_bytes: 0
tx53_csum_partial: 0
tx53_csum_partial_inner: 0
tx53_added_vlan_packets: 0
tx53_nop: 0
tx53_csum_none: 0
tx53_stopped: 0
tx53_dropped: 0
tx53_xmit_more: 0
tx53_recover: 0
tx53_cqes: 0
tx53_wake: 0
tx53_cqe_err: 0
tx54_packets: 0
tx54_bytes: 0
tx54_tso_packets: 0
tx54_tso_bytes: 0
tx54_tso_inner_packets: 0
tx54_tso_inner_bytes: 0
tx54_csum_partial: 0
tx54_csum_partial_inner: 0
tx54_added_vlan_packets: 0
tx54_nop: 0
tx54_csum_none: 0
tx54_stopped: 0
tx54_dropped: 0
tx54_xmit_more: 0
tx54_recover: 0
tx54_cqes: 0
tx54_wake: 0
tx54_cqe_err: 0
tx55_packets: 0
tx55_bytes: 0
tx55_tso_packets: 0
tx55_tso_bytes: 0
tx55_tso_inner_packets: 0
tx55_tso_inner_bytes: 0
tx55_csum_partial: 0
tx55_csum_partial_inner: 0
tx55_added_vlan_packets: 0
tx55_nop: 0
tx55_csum_none: 0
tx55_stopped: 0
tx55_dropped: 0
tx55_xmit_more: 0
tx55_recover: 0
tx55_cqes: 0
tx55_wake: 0
tx55_cqe_err: 0
tx0_xdp_xmit: 0
tx0_xdp_full: 0
tx0_xdp_err: 0
tx0_xdp_cqes: 0
tx1_xdp_xmit: 0
tx1_xdp_full: 0
tx1_xdp_err: 0
tx1_xdp_cqes: 0
tx2_xdp_xmit: 0
tx2_xdp_full: 0
tx2_xdp_err: 0
tx2_xdp_cqes: 0
tx3_xdp_xmit: 0
tx3_xdp_full: 0
tx3_xdp_err: 0
tx3_xdp_cqes: 0
tx4_xdp_xmit: 0
tx4_xdp_full: 0
tx4_xdp_err: 0
tx4_xdp_cqes: 0
tx5_xdp_xmit: 0
tx5_xdp_full: 0
tx5_xdp_err: 0
tx5_xdp_cqes: 0
tx6_xdp_xmit: 0
tx6_xdp_full: 0
tx6_xdp_err: 0
tx6_xdp_cqes: 0
tx7_xdp_xmit: 0
tx7_xdp_full: 0
tx7_xdp_err: 0
tx7_xdp_cqes: 0
tx8_xdp_xmit: 0
tx8_xdp_full: 0
tx8_xdp_err: 0
tx8_xdp_cqes: 0
tx9_xdp_xmit: 0
tx9_xdp_full: 0
tx9_xdp_err: 0
tx9_xdp_cqes: 0
tx10_xdp_xmit: 0
tx10_xdp_full: 0
tx10_xdp_err: 0
tx10_xdp_cqes: 0
tx11_xdp_xmit: 0
tx11_xdp_full: 0
tx11_xdp_err: 0
tx11_xdp_cqes: 0
tx12_xdp_xmit: 0
tx12_xdp_full: 0
tx12_xdp_err: 0
tx12_xdp_cqes: 0
tx13_xdp_xmit: 0
tx13_xdp_full: 0
tx13_xdp_err: 0
tx13_xdp_cqes: 0
tx14_xdp_xmit: 0
tx14_xdp_full: 0
tx14_xdp_err: 0
tx14_xdp_cqes: 0
tx15_xdp_xmit: 0
tx15_xdp_full: 0
tx15_xdp_err: 0
tx15_xdp_cqes: 0
tx16_xdp_xmit: 0
tx16_xdp_full: 0
tx16_xdp_err: 0
tx16_xdp_cqes: 0
tx17_xdp_xmit: 0
tx17_xdp_full: 0
tx17_xdp_err: 0
tx17_xdp_cqes: 0
tx18_xdp_xmit: 0
tx18_xdp_full: 0
tx18_xdp_err: 0
tx18_xdp_cqes: 0
tx19_xdp_xmit: 0
tx19_xdp_full: 0
tx19_xdp_err: 0
tx19_xdp_cqes: 0
tx20_xdp_xmit: 0
tx20_xdp_full: 0
tx20_xdp_err: 0
tx20_xdp_cqes: 0
tx21_xdp_xmit: 0
tx21_xdp_full: 0
tx21_xdp_err: 0
tx21_xdp_cqes: 0
tx22_xdp_xmit: 0
tx22_xdp_full: 0
tx22_xdp_err: 0
tx22_xdp_cqes: 0
tx23_xdp_xmit: 0
tx23_xdp_full: 0
tx23_xdp_err: 0
tx23_xdp_cqes: 0
tx24_xdp_xmit: 0
tx24_xdp_full: 0
tx24_xdp_err: 0
tx24_xdp_cqes: 0
tx25_xdp_xmit: 0
tx25_xdp_full: 0
tx25_xdp_err: 0
tx25_xdp_cqes: 0
tx26_xdp_xmit: 0
tx26_xdp_full: 0
tx26_xdp_err: 0
tx26_xdp_cqes: 0
tx27_xdp_xmit: 0
tx27_xdp_full: 0
tx27_xdp_err: 0
tx27_xdp_cqes: 0
tx28_xdp_xmit: 0
tx28_xdp_full: 0
tx28_xdp_err: 0
tx28_xdp_cqes: 0
tx29_xdp_xmit: 0
tx29_xdp_full: 0
tx29_xdp_err: 0
tx29_xdp_cqes: 0
tx30_xdp_xmit: 0
tx30_xdp_full: 0
tx30_xdp_err: 0
tx30_xdp_cqes: 0
tx31_xdp_xmit: 0
tx31_xdp_full: 0
tx31_xdp_err: 0
tx31_xdp_cqes: 0
tx32_xdp_xmit: 0
tx32_xdp_full: 0
tx32_xdp_err: 0
tx32_xdp_cqes: 0
tx33_xdp_xmit: 0
tx33_xdp_full: 0
tx33_xdp_err: 0
tx33_xdp_cqes: 0
tx34_xdp_xmit: 0
tx34_xdp_full: 0
tx34_xdp_err: 0
tx34_xdp_cqes: 0
tx35_xdp_xmit: 0
tx35_xdp_full: 0
tx35_xdp_err: 0
tx35_xdp_cqes: 0
tx36_xdp_xmit: 0
tx36_xdp_full: 0
tx36_xdp_err: 0
tx36_xdp_cqes: 0
tx37_xdp_xmit: 0
tx37_xdp_full: 0
tx37_xdp_err: 0
tx37_xdp_cqes: 0
tx38_xdp_xmit: 0
tx38_xdp_full: 0
tx38_xdp_err: 0
tx38_xdp_cqes: 0
tx39_xdp_xmit: 0
tx39_xdp_full: 0
tx39_xdp_err: 0
tx39_xdp_cqes: 0
tx40_xdp_xmit: 0
tx40_xdp_full: 0
tx40_xdp_err: 0
tx40_xdp_cqes: 0
tx41_xdp_xmit: 0
tx41_xdp_full: 0
tx41_xdp_err: 0
tx41_xdp_cqes: 0
tx42_xdp_xmit: 0
tx42_xdp_full: 0
tx42_xdp_err: 0
tx42_xdp_cqes: 0
tx43_xdp_xmit: 0
tx43_xdp_full: 0
tx43_xdp_err: 0
tx43_xdp_cqes: 0
tx44_xdp_xmit: 0
tx44_xdp_full: 0
tx44_xdp_err: 0
tx44_xdp_cqes: 0
tx45_xdp_xmit: 0
tx45_xdp_full: 0
tx45_xdp_err: 0
tx45_xdp_cqes: 0
tx46_xdp_xmit: 0
tx46_xdp_full: 0
tx46_xdp_err: 0
tx46_xdp_cqes: 0
tx47_xdp_xmit: 0
tx47_xdp_full: 0
tx47_xdp_err: 0
tx47_xdp_cqes: 0
tx48_xdp_xmit: 0
tx48_xdp_full: 0
tx48_xdp_err: 0
tx48_xdp_cqes: 0
tx49_xdp_xmit: 0
tx49_xdp_full: 0
tx49_xdp_err: 0
tx49_xdp_cqes: 0
tx50_xdp_xmit: 0
tx50_xdp_full: 0
tx50_xdp_err: 0
tx50_xdp_cqes: 0
tx51_xdp_xmit: 0
tx51_xdp_full: 0
tx51_xdp_err: 0
tx51_xdp_cqes: 0
tx52_xdp_xmit: 0
tx52_xdp_full: 0
tx52_xdp_err: 0
tx52_xdp_cqes: 0
tx53_xdp_xmit: 0
tx53_xdp_full: 0
tx53_xdp_err: 0
tx53_xdp_cqes: 0
tx54_xdp_xmit: 0
tx54_xdp_full: 0
tx54_xdp_err: 0
tx54_xdp_cqes: 0
tx55_xdp_xmit: 0
tx55_xdp_full: 0
tx55_xdp_err: 0
tx55_xdp_cqes: 0
ethtool -S enp175s0f0
NIC statistics:
rx_packets: 141574897253
rx_bytes: 184445040406258
tx_packets: 172569543894
tx_bytes: 99486882076365
tx_tso_packets: 9367664195
tx_tso_bytes: 56435233992948
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 141297671626
tx_nop: 2102916272
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 141574897252
rx_csum_unnecessary: 0
rx_csum_none: 23135854
rx_csum_complete: 141551761398
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 127934791664
tx_csum_partial: 13362879974
tx_csum_partial_inner: 0
tx_queue_stopped: 232561
tx_queue_dropped: 0
tx_xmit_more: 1266021946
tx_recover: 0
tx_cqes: 140031716469
tx_queue_wake: 232561
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 16625975793
rx_cache_full: 54161465914
rx_cache_empty: 258048
rx_cache_busy: 54161472735
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 40572621887
ch_poll: 40885650979
ch_arm: 40429276692
ch_aff_change: 0
ch_eq_rearm: 0
rx_out_of_buffer: 2791690
rx_if_down_packets: 74
rx_vport_unicast_packets: 141843476308
rx_vport_unicast_bytes: 185421265403318
tx_vport_unicast_packets: 172569484005
tx_vport_unicast_bytes: 100019940094298
rx_vport_multicast_packets: 85122935
rx_vport_multicast_bytes: 5761316431
tx_vport_multicast_packets: 6452
tx_vport_multicast_bytes: 643540
rx_vport_broadcast_packets: 22423624
rx_vport_broadcast_bytes: 1390127090
tx_vport_broadcast_packets: 22024
tx_vport_broadcast_bytes: 1321440
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 172569501577
rx_packets_phy: 142871314588
rx_crc_errors_phy: 0
tx_bytes_phy: 100710212814151
rx_bytes_phy: 187209224289564
tx_multicast_phy: 6452
tx_broadcast_phy: 22024
rx_multicast_phy: 85122933
rx_broadcast_phy: 22423623
rx_in_range_len_errors_phy: 2
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 920161423
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 412006326
rx_65_to_127_bytes_phy: 11934371453
rx_128_to_255_bytes_phy: 3415281165
rx_256_to_511_bytes_phy: 2072955511
rx_512_to_1023_bytes_phy: 2415393005
rx_1024_to_1518_bytes_phy: 72182391608
rx_1519_to_2047_bytes_phy: 50438902587
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 48
rx_prio0_bytes: 186709842592642
rx_prio0_packets: 141481966007
tx_prio0_bytes: 100710171118138
tx_prio0_packets: 172569437949
rx_prio1_bytes: 492288152326
rx_prio1_packets: 385996045
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 22119952
rx_prio2_packets: 70788
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 546141102
rx_prio3_packets: 681608
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 14665067
rx_prio4_packets: 29486
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 158862504
rx_prio5_packets: 965307
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 669337783
rx_prio6_packets: 1475775
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 5623481349
rx_prio7_packets: 79926412
tx_prio7_bytes: 0
tx_prio7_packets: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
ch0_events: 1446162630
ch0_poll: 1463312972
ch0_arm: 1440728278
ch0_aff_change: 0
ch0_eq_rearm: 0
ch1_events: 1384301405
ch1_poll: 1399210915
ch1_arm: 1378636486
ch1_aff_change: 0
ch1_eq_rearm: 0
ch2_events: 1382788887
ch2_poll: 1397231470
ch2_arm: 1377058116
ch2_aff_change: 0
ch2_eq_rearm: 0
ch3_events: 1461956995
ch3_poll: 1475553146
ch3_arm: 1456571625
ch3_aff_change: 0
ch3_eq_rearm: 0
ch4_events: 1497359109
ch4_poll: 1511021037
ch4_arm: 1491733757
ch4_aff_change: 0
ch4_eq_rearm: 0
ch5_events: 1387736262
ch5_poll: 1400964615
ch5_arm: 1382382834
ch5_aff_change: 0
ch5_eq_rearm: 0
ch6_events: 1376772405
ch6_poll: 1390851449
ch6_arm: 1371551764
ch6_aff_change: 0
ch6_eq_rearm: 0
ch7_events: 1431271514
ch7_poll: 1445049729
ch7_arm: 1425753718
ch7_aff_change: 0
ch7_eq_rearm: 0
ch8_events: 1426976374
ch8_poll: 1439938692
ch8_arm: 1421392984
ch8_aff_change: 0
ch8_eq_rearm: 0
ch9_events: 1456160031
ch9_poll: 1468922870
ch9_arm: 1450930446
ch9_aff_change: 0
ch9_eq_rearm: 0
ch10_events: 1443640165
ch10_poll: 1456812203
ch10_arm: 1438425101
ch10_aff_change: 0
ch10_eq_rearm: 0
ch11_events: 1381104776
ch11_poll: 1393811057
ch11_arm: 1376059326
ch11_aff_change: 0
ch11_eq_rearm: 0
ch12_events: 1365223276
ch12_poll: 1378406059
ch12_arm: 1359950494
ch12_aff_change: 0
ch12_eq_rearm: 0
ch13_events: 1421622259
ch13_poll: 1434670996
ch13_arm: 1416241801
ch13_aff_change: 0
ch13_eq_rearm: 0
ch14_events: 1379084590
ch14_poll: 1392425015
ch14_arm: 1373675179
ch14_aff_change: 0
ch14_eq_rearm: 0
ch15_events: 1531217338
ch15_poll: 1543353833
ch15_arm: 1526350453
ch15_aff_change: 0
ch15_eq_rearm: 0
ch16_events: 1460469776
ch16_poll: 1467995928
ch16_arm: 1456010194
ch16_aff_change: 0
ch16_eq_rearm: 0
ch17_events: 1494067670
ch17_poll: 1500856680
ch17_arm: 1489232674
ch17_aff_change: 0
ch17_eq_rearm: 0
ch18_events: 1530126866
ch18_poll: 1537293620
ch18_arm: 1525476123
ch18_aff_change: 0
ch18_eq_rearm: 0
ch19_events: 1499526149
ch19_poll: 1506789309
ch19_arm: 1495161602
ch19_aff_change: 0
ch19_eq_rearm: 0
ch20_events: 1451479763
ch20_poll: 1459767921
ch20_arm: 1446360801
ch20_aff_change: 0
ch20_eq_rearm: 0
ch21_events: 1521413613
ch21_poll: 1529345146
ch21_arm: 1517229314
ch21_aff_change: 0
ch21_eq_rearm: 0
ch22_events: 1471950045
ch22_poll: 1479746764
ch22_arm: 1467681629
ch22_aff_change: 0
ch22_eq_rearm: 0
ch23_events: 1502968393
ch23_poll: 1510419909
ch23_arm: 1498168438
ch23_aff_change: 0
ch23_eq_rearm: 0
ch24_events: 1473451639
ch24_poll: 1482606899
ch24_arm: 1468212489
ch24_aff_change: 0
ch24_eq_rearm: 0
ch25_events: 1440399182
ch25_poll: 1448897475
ch25_arm: 1435044786
ch25_aff_change: 0
ch25_eq_rearm: 0
ch26_events: 1436831565
ch26_poll: 1445485731
ch26_arm: 1431827527
ch26_aff_change: 0
ch26_eq_rearm: 0
ch27_events: 1516560621
ch27_poll: 1524911010
ch27_arm: 1511430164
ch27_aff_change: 0
ch27_eq_rearm: 0
ch28_events: 4
ch28_poll: 4
ch28_arm: 4
ch28_aff_change: 0
ch28_eq_rearm: 0
ch29_events: 6
ch29_poll: 6
ch29_arm: 6
ch29_aff_change: 0
ch29_eq_rearm: 0
ch30_events: 4
ch30_poll: 4
ch30_arm: 4
ch30_aff_change: 0
ch30_eq_rearm: 0
ch31_events: 4
ch31_poll: 4
ch31_arm: 4
ch31_aff_change: 0
ch31_eq_rearm: 0
ch32_events: 4
ch32_poll: 4
ch32_arm: 4
ch32_aff_change: 0
ch32_eq_rearm: 0
ch33_events: 4
ch33_poll: 4
ch33_arm: 4
ch33_aff_change: 0
ch33_eq_rearm: 0
ch34_events: 4
ch34_poll: 4
ch34_arm: 4
ch34_aff_change: 0
ch34_eq_rearm: 0
ch35_events: 4
ch35_poll: 4
ch35_arm: 4
ch35_aff_change: 0
ch35_eq_rearm: 0
ch36_events: 4
ch36_poll: 4
ch36_arm: 4
ch36_aff_change: 0
ch36_eq_rearm: 0
ch37_events: 4
ch37_poll: 4
ch37_arm: 4
ch37_aff_change: 0
ch37_eq_rearm: 0
ch38_events: 4
ch38_poll: 4
ch38_arm: 4
ch38_aff_change: 0
ch38_eq_rearm: 0
ch39_events: 4
ch39_poll: 4
ch39_arm: 4
ch39_aff_change: 0
ch39_eq_rearm: 0
ch40_events: 4
ch40_poll: 4
ch40_arm: 4
ch40_aff_change: 0
ch40_eq_rearm: 0
ch41_events: 4
ch41_poll: 4
ch41_arm: 4
ch41_aff_change: 0
ch41_eq_rearm: 0
ch42_events: 4
ch42_poll: 4
ch42_arm: 4
ch42_aff_change: 0
ch42_eq_rearm: 0
ch43_events: 4
ch43_poll: 4
ch43_arm: 4
ch43_aff_change: 0
ch43_eq_rearm: 0
ch44_events: 4
ch44_poll: 4
ch44_arm: 4
ch44_aff_change: 0
ch44_eq_rearm: 0
ch45_events: 4
ch45_poll: 4
ch45_arm: 4
ch45_aff_change: 0
ch45_eq_rearm: 0
ch46_events: 4
ch46_poll: 4
ch46_arm: 4
ch46_aff_change: 0
ch46_eq_rearm: 0
ch47_events: 4
ch47_poll: 4
ch47_arm: 4
ch47_aff_change: 0
ch47_eq_rearm: 0
ch48_events: 4
ch48_poll: 4
ch48_arm: 4
ch48_aff_change: 0
ch48_eq_rearm: 0
ch49_events: 4
ch49_poll: 4
ch49_arm: 4
ch49_aff_change: 0
ch49_eq_rearm: 0
ch50_events: 4
ch50_poll: 4
ch50_arm: 4
ch50_aff_change: 0
ch50_eq_rearm: 0
ch51_events: 4
ch51_poll: 4
ch51_arm: 4
ch51_aff_change: 0
ch51_eq_rearm: 0
ch52_events: 4
ch52_poll: 4
ch52_arm: 4
ch52_aff_change: 0
ch52_eq_rearm: 0
ch53_events: 4
ch53_poll: 4
ch53_arm: 4
ch53_aff_change: 0
ch53_eq_rearm: 0
ch54_events: 4
ch54_poll: 4
ch54_arm: 4
ch54_aff_change: 0
ch54_eq_rearm: 0
ch55_events: 4
ch55_poll: 4
ch55_arm: 4
ch55_aff_change: 0
ch55_eq_rearm: 0
rx0_packets: 5861448653
rx0_bytes: 7389128595728
rx0_csum_complete: 5838312798
rx0_csum_unnecessary: 0
rx0_csum_unnecessary_inner: 0
rx0_csum_none: 23135855
rx0_xdp_drop: 0
rx0_xdp_redirect: 0
rx0_lro_packets: 0
rx0_lro_bytes: 0
rx0_ecn_mark: 0
rx0_removed_vlan_packets: 5861448653
rx0_wqe_err: 0
rx0_mpwqe_filler_cqes: 0
rx0_mpwqe_filler_strides: 0
rx0_buff_alloc_err: 0
rx0_cqe_compress_blks: 0
rx0_cqe_compress_pkts: 0
rx0_page_reuse: 0
rx0_cache_reuse: 2559
rx0_cache_full: 2930721512
rx0_cache_empty: 6656
rx0_cache_busy: 2930721765
rx0_cache_waive: 0
rx0_congst_umr: 0
rx0_arfs_err: 0
rx0_xdp_tx_xmit: 0
rx0_xdp_tx_full: 0
rx0_xdp_tx_err: 0
rx0_xdp_tx_cqes: 0
rx1_packets: 5550585106
rx1_bytes: 7255635262803
rx1_csum_complete: 5550585106
rx1_csum_unnecessary: 0
rx1_csum_unnecessary_inner: 0
rx1_csum_none: 0
rx1_xdp_drop: 0
rx1_xdp_redirect: 0
rx1_lro_packets: 0
rx1_lro_bytes: 0
rx1_ecn_mark: 0
rx1_removed_vlan_packets: 5550585106
rx1_wqe_err: 0
rx1_mpwqe_filler_cqes: 0
rx1_mpwqe_filler_strides: 0
rx1_buff_alloc_err: 0
rx1_cqe_compress_blks: 0
rx1_cqe_compress_pkts: 0
rx1_page_reuse: 0
rx1_cache_reuse: 2918845
rx1_cache_full: 2772373453
rx1_cache_empty: 6656
rx1_cache_busy: 2772373707
rx1_cache_waive: 0
rx1_congst_umr: 0
rx1_arfs_err: 0
rx1_xdp_tx_xmit: 0
rx1_xdp_tx_full: 0
rx1_xdp_tx_err: 0
rx1_xdp_tx_cqes: 0
rx2_packets: 5383874739
rx2_bytes: 7031545423967
rx2_csum_complete: 5383874739
rx2_csum_unnecessary: 0
rx2_csum_unnecessary_inner: 0
rx2_csum_none: 0
rx2_xdp_drop: 0
rx2_xdp_redirect: 0
rx2_lro_packets: 0
rx2_lro_bytes: 0
rx2_ecn_mark: 0
rx2_removed_vlan_packets: 5383874739
rx2_wqe_err: 0
rx2_mpwqe_filler_cqes: 0
rx2_mpwqe_filler_strides: 0
rx2_buff_alloc_err: 0
rx2_cqe_compress_blks: 0
rx2_cqe_compress_pkts: 0
rx2_page_reuse: 0
rx2_cache_reuse: 2173370
rx2_cache_full: 2689763744
rx2_cache_empty: 6656
rx2_cache_busy: 2689763998
rx2_cache_waive: 0
rx2_congst_umr: 0
rx2_arfs_err: 0
rx2_xdp_tx_xmit: 0
rx2_xdp_tx_full: 0
rx2_xdp_tx_err: 0
rx2_xdp_tx_cqes: 0
rx3_packets: 5456494012
rx3_bytes: 7120241119485
rx3_csum_complete: 5456494012
rx3_csum_unnecessary: 0
rx3_csum_unnecessary_inner: 0
rx3_csum_none: 0
rx3_xdp_drop: 0
rx3_xdp_redirect: 0
rx3_lro_packets: 0
rx3_lro_bytes: 0
rx3_ecn_mark: 0
rx3_removed_vlan_packets: 5456494012
rx3_wqe_err: 0
rx3_mpwqe_filler_cqes: 0
rx3_mpwqe_filler_strides: 0
rx3_buff_alloc_err: 0
rx3_cqe_compress_blks: 0
rx3_cqe_compress_pkts: 0
rx3_page_reuse: 0
rx3_cache_reuse: 2120123
rx3_cache_full: 2726126628
rx3_cache_empty: 6656
rx3_cache_busy: 2726126881
rx3_cache_waive: 0
rx3_congst_umr: 0
rx3_arfs_err: 0
rx3_xdp_tx_xmit: 0
rx3_xdp_tx_full: 0
rx3_xdp_tx_err: 0
rx3_xdp_tx_cqes: 0
rx4_packets: 5475216251
rx4_bytes: 7123129170196
rx4_csum_complete: 5475216251
rx4_csum_unnecessary: 0
rx4_csum_unnecessary_inner: 0
rx4_csum_none: 0
rx4_xdp_drop: 0
rx4_xdp_redirect: 0
rx4_lro_packets: 0
rx4_lro_bytes: 0
rx4_ecn_mark: 0
rx4_removed_vlan_packets: 5475216251
rx4_wqe_err: 0
rx4_mpwqe_filler_cqes: 0
rx4_mpwqe_filler_strides: 0
rx4_buff_alloc_err: 0
rx4_cqe_compress_blks: 0
rx4_cqe_compress_pkts: 0
rx4_page_reuse: 0
rx4_cache_reuse: 2668296355
rx4_cache_full: 69311549
rx4_cache_empty: 6656
rx4_cache_busy: 69311769
rx4_cache_waive: 0
rx4_congst_umr: 0
rx4_arfs_err: 0
rx4_xdp_tx_xmit: 0
rx4_xdp_tx_full: 0
rx4_xdp_tx_err: 0
rx4_xdp_tx_cqes: 0
rx5_packets: 5474372232
rx5_bytes: 7159146801926
rx5_csum_complete: 5474372232
rx5_csum_unnecessary: 0
rx5_csum_unnecessary_inner: 0
rx5_csum_none: 0
rx5_xdp_drop: 0
rx5_xdp_redirect: 0
rx5_lro_packets: 0
rx5_lro_bytes: 0
rx5_ecn_mark: 0
rx5_removed_vlan_packets: 5474372232
rx5_wqe_err: 0
rx5_mpwqe_filler_cqes: 0
rx5_mpwqe_filler_strides: 0
rx5_buff_alloc_err: 0
rx5_cqe_compress_blks: 0
rx5_cqe_compress_pkts: 0
rx5_page_reuse: 0
rx5_cache_reuse: 626187
rx5_cache_full: 2736559674
rx5_cache_empty: 6656
rx5_cache_busy: 2736559929
rx5_cache_waive: 0
rx5_congst_umr: 0
rx5_arfs_err: 0
rx5_xdp_tx_xmit: 0
rx5_xdp_tx_full: 0
rx5_xdp_tx_err: 0
rx5_xdp_tx_cqes: 0
rx6_packets: 5533622456
rx6_bytes: 7207308809081
rx6_csum_complete: 5533622456
rx6_csum_unnecessary: 0
rx6_csum_unnecessary_inner: 0
rx6_csum_none: 0
rx6_xdp_drop: 0
rx6_xdp_redirect: 0
rx6_lro_packets: 0
rx6_lro_bytes: 0
rx6_ecn_mark: 0
rx6_removed_vlan_packets: 5533622456
rx6_wqe_err: 0
rx6_mpwqe_filler_cqes: 0
rx6_mpwqe_filler_strides: 0
rx6_buff_alloc_err: 0
rx6_cqe_compress_blks: 0
rx6_cqe_compress_pkts: 0
rx6_page_reuse: 0
rx6_cache_reuse: 2325217
rx6_cache_full: 2764485756
rx6_cache_empty: 6656
rx6_cache_busy: 2764486011
rx6_cache_waive: 0
rx6_congst_umr: 0
rx6_arfs_err: 0
rx6_xdp_tx_xmit: 0
rx6_xdp_tx_full: 0
rx6_xdp_tx_err: 0
rx6_xdp_tx_cqes: 0
rx7_packets: 5533901822
rx7_bytes: 7227441240536
rx7_csum_complete: 5533901822
rx7_csum_unnecessary: 0
rx7_csum_unnecessary_inner: 0
rx7_csum_none: 0
rx7_xdp_drop: 0
rx7_xdp_redirect: 0
rx7_lro_packets: 0
rx7_lro_bytes: 0
rx7_ecn_mark: 0
rx7_removed_vlan_packets: 5533901822
rx7_wqe_err: 0
rx7_mpwqe_filler_cqes: 0
rx7_mpwqe_filler_strides: 0
rx7_buff_alloc_err: 0
rx7_cqe_compress_blks: 0
rx7_cqe_compress_pkts: 0
rx7_page_reuse: 0
rx7_cache_reuse: 2372505
rx7_cache_full: 2764578151
rx7_cache_empty: 6656
rx7_cache_busy: 2764578403
rx7_cache_waive: 0
rx7_congst_umr: 0
rx7_arfs_err: 0
rx7_xdp_tx_xmit: 0
rx7_xdp_tx_full: 0
rx7_xdp_tx_err: 0
rx7_xdp_tx_cqes: 0
rx8_packets: 5485670137
rx8_bytes: 7203339989013
rx8_csum_complete: 5485670137
rx8_csum_unnecessary: 0
rx8_csum_unnecessary_inner: 0
rx8_csum_none: 0
rx8_xdp_drop: 0
rx8_xdp_redirect: 0
rx8_lro_packets: 0
rx8_lro_bytes: 0
rx8_ecn_mark: 0
rx8_removed_vlan_packets: 5485670137
rx8_wqe_err: 0
rx8_mpwqe_filler_cqes: 0
rx8_mpwqe_filler_strides: 0
rx8_buff_alloc_err: 0
rx8_cqe_compress_blks: 0
rx8_cqe_compress_pkts: 0
rx8_page_reuse: 0
rx8_cache_reuse: 7522232
rx8_cache_full: 2735312581
rx8_cache_empty: 6656
rx8_cache_busy: 2735312836
rx8_cache_waive: 0
rx8_congst_umr: 0
rx8_arfs_err: 0
rx8_xdp_tx_xmit: 0
rx8_xdp_tx_full: 0
rx8_xdp_tx_err: 0
rx8_xdp_tx_cqes: 0
rx9_packets: 5482212354
rx9_bytes: 7169663341718
rx9_csum_complete: 5482212354
rx9_csum_unnecessary: 0
rx9_csum_unnecessary_inner: 0
rx9_csum_none: 0
rx9_xdp_drop: 0
rx9_xdp_redirect: 0
rx9_lro_packets: 0
rx9_lro_bytes: 0
rx9_ecn_mark: 0
rx9_removed_vlan_packets: 5482212354
rx9_wqe_err: 0
rx9_mpwqe_filler_cqes: 0
rx9_mpwqe_filler_strides: 0
rx9_buff_alloc_err: 0
rx9_cqe_compress_blks: 0
rx9_cqe_compress_pkts: 0
rx9_page_reuse: 0
rx9_cache_reuse: 37279961
rx9_cache_full: 2703825961
rx9_cache_empty: 6656
rx9_cache_busy: 2703826215
rx9_cache_waive: 0
rx9_congst_umr: 0
rx9_arfs_err: 0
rx9_xdp_tx_xmit: 0
rx9_xdp_tx_full: 0
rx9_xdp_tx_err: 0
rx9_xdp_tx_cqes: 0
rx10_packets: 5524679952
rx10_bytes: 7248301275181
rx10_csum_complete: 5524679952
rx10_csum_unnecessary: 0
rx10_csum_unnecessary_inner: 0
rx10_csum_none: 0
rx10_xdp_drop: 0
rx10_xdp_redirect: 0
rx10_lro_packets: 0
rx10_lro_bytes: 0
rx10_ecn_mark: 0
rx10_removed_vlan_packets: 5524679952
rx10_wqe_err: 0
rx10_mpwqe_filler_cqes: 0
rx10_mpwqe_filler_strides: 0
rx10_buff_alloc_err: 0
rx10_cqe_compress_blks: 0
rx10_cqe_compress_pkts: 0
rx10_page_reuse: 0
rx10_cache_reuse: 2049666
rx10_cache_full: 2760290055
rx10_cache_empty: 6656
rx10_cache_busy: 2760290310
rx10_cache_waive: 0
rx10_congst_umr: 0
rx10_arfs_err: 0
rx10_xdp_tx_xmit: 0
rx10_xdp_tx_full: 0
rx10_xdp_tx_err: 0
rx10_xdp_tx_cqes: 0
rx11_packets: 5394633545
rx11_bytes: 7033509636092
rx11_csum_complete: 5394633545
rx11_csum_unnecessary: 0
rx11_csum_unnecessary_inner: 0
rx11_csum_none: 0
rx11_xdp_drop: 0
rx11_xdp_redirect: 0
rx11_lro_packets: 0
rx11_lro_bytes: 0
rx11_ecn_mark: 0
rx11_removed_vlan_packets: 5394633545
rx11_wqe_err: 0
rx11_mpwqe_filler_cqes: 0
rx11_mpwqe_filler_strides: 0
rx11_buff_alloc_err: 0
rx11_cqe_compress_blks: 0
rx11_cqe_compress_pkts: 0
rx11_page_reuse: 0
rx11_cache_reuse: 2617466268
rx11_cache_full: 79850284
rx11_cache_empty: 6656
rx11_cache_busy: 79850504
rx11_cache_waive: 0
rx11_congst_umr: 0
rx11_arfs_err: 0
rx11_xdp_tx_xmit: 0
rx11_xdp_tx_full: 0
rx11_xdp_tx_err: 0
rx11_xdp_tx_cqes: 0
rx12_packets: 5458907385
rx12_bytes: 7134867867515
rx12_csum_complete: 5458907385
rx12_csum_unnecessary: 0
rx12_csum_unnecessary_inner: 0
rx12_csum_none: 0
rx12_xdp_drop: 0
rx12_xdp_redirect: 0
rx12_lro_packets: 0
rx12_lro_bytes: 0
rx12_ecn_mark: 0
rx12_removed_vlan_packets: 5458907385
rx12_wqe_err: 0
rx12_mpwqe_filler_cqes: 0
rx12_mpwqe_filler_strides: 0
rx12_buff_alloc_err: 0
rx12_cqe_compress_blks: 0
rx12_cqe_compress_pkts: 0
rx12_page_reuse: 0
rx12_cache_reuse: 2650214169
rx12_cache_full: 79239303
rx12_cache_empty: 6656
rx12_cache_busy: 79239523
rx12_cache_waive: 0
rx12_congst_umr: 0
rx12_arfs_err: 0
rx12_xdp_tx_xmit: 0
rx12_xdp_tx_full: 0
rx12_xdp_tx_err: 0
rx12_xdp_tx_cqes: 0
rx13_packets: 5549932912
rx13_bytes: 7232548705586
rx13_csum_complete: 5549932912
rx13_csum_unnecessary: 0
rx13_csum_unnecessary_inner: 0
rx13_csum_none: 0
rx13_xdp_drop: 0
rx13_xdp_redirect: 0
rx13_lro_packets: 0
rx13_lro_bytes: 0
rx13_ecn_mark: 0
rx13_removed_vlan_packets: 5549932912
rx13_wqe_err: 0
rx13_mpwqe_filler_cqes: 0
rx13_mpwqe_filler_strides: 0
rx13_buff_alloc_err: 0
rx13_cqe_compress_blks: 0
rx13_cqe_compress_pkts: 0
rx13_page_reuse: 0
rx13_cache_reuse: 2417696
rx13_cache_full: 2772548505
rx13_cache_empty: 6656
rx13_cache_busy: 2772548760
rx13_cache_waive: 0
rx13_congst_umr: 0
rx13_arfs_err: 0
rx13_xdp_tx_xmit: 0
rx13_xdp_tx_full: 0
rx13_xdp_tx_err: 0
rx13_xdp_tx_cqes: 0
rx14_packets: 5517712329
rx14_bytes: 7192111965227
rx14_csum_complete: 5517712329
rx14_csum_unnecessary: 0
rx14_csum_unnecessary_inner: 0
rx14_csum_none: 0
rx14_xdp_drop: 0
rx14_xdp_redirect: 0
rx14_lro_packets: 0
rx14_lro_bytes: 0
rx14_ecn_mark: 0
rx14_removed_vlan_packets: 5517712329
rx14_wqe_err: 0
rx14_mpwqe_filler_cqes: 0
rx14_mpwqe_filler_strides: 0
rx14_buff_alloc_err: 0
rx14_cqe_compress_blks: 0
rx14_cqe_compress_pkts: 0
rx14_page_reuse: 0
rx14_cache_reuse: 1830206
rx14_cache_full: 2757025703
rx14_cache_empty: 6656
rx14_cache_busy: 2757025958
rx14_cache_waive: 0
rx14_congst_umr: 0
rx14_arfs_err: 0
rx14_xdp_tx_xmit: 0
rx14_xdp_tx_full: 0
rx14_xdp_tx_err: 0
rx14_xdp_tx_cqes: 0
rx15_packets: 5578343373
rx15_bytes: 7268484501219
rx15_csum_complete: 5578343373
rx15_csum_unnecessary: 0
rx15_csum_unnecessary_inner: 0
rx15_csum_none: 0
rx15_xdp_drop: 0
rx15_xdp_redirect: 0
rx15_lro_packets: 0
rx15_lro_bytes: 0
rx15_ecn_mark: 0
rx15_removed_vlan_packets: 5578343373
rx15_wqe_err: 0
rx15_mpwqe_filler_cqes: 0
rx15_mpwqe_filler_strides: 0
rx15_buff_alloc_err: 0
rx15_cqe_compress_blks: 0
rx15_cqe_compress_pkts: 0
rx15_page_reuse: 0
rx15_cache_reuse: 2317165
rx15_cache_full: 2786854266
rx15_cache_empty: 6656
rx15_cache_busy: 2786854519
rx15_cache_waive: 0
rx15_congst_umr: 0
rx15_arfs_err: 0
rx15_xdp_tx_xmit: 0
rx15_xdp_tx_full: 0
rx15_xdp_tx_err: 0
rx15_xdp_tx_cqes: 0
rx16_packets: 4435773951
rx16_bytes: 5766665272007
rx16_csum_complete: 4435773951
rx16_csum_unnecessary: 0
rx16_csum_unnecessary_inner: 0
rx16_csum_none: 0
rx16_xdp_drop: 0
rx16_xdp_redirect: 0
rx16_lro_packets: 0
rx16_lro_bytes: 0
rx16_ecn_mark: 0
rx16_removed_vlan_packets: 4435773951
rx16_wqe_err: 0
rx16_mpwqe_filler_cqes: 0
rx16_mpwqe_filler_strides: 0
rx16_buff_alloc_err: 0
rx16_cqe_compress_blks: 0
rx16_cqe_compress_pkts: 0
rx16_page_reuse: 0
rx16_cache_reuse: 2033793
rx16_cache_full: 2215852927
rx16_cache_empty: 6656
rx16_cache_busy: 2215853179
rx16_cache_waive: 0
rx16_congst_umr: 0
rx16_arfs_err: 0
rx16_xdp_tx_xmit: 0
rx16_xdp_tx_full: 0
rx16_xdp_tx_err: 0
rx16_xdp_tx_cqes: 0
rx17_packets: 4344087587
rx17_bytes: 5695006496323
rx17_csum_complete: 4344087587
rx17_csum_unnecessary: 0
rx17_csum_unnecessary_inner: 0
rx17_csum_none: 0
rx17_xdp_drop: 0
rx17_xdp_redirect: 0
rx17_lro_packets: 0
rx17_lro_bytes: 0
rx17_ecn_mark: 0
rx17_removed_vlan_packets: 4344087587
rx17_wqe_err: 0
rx17_mpwqe_filler_cqes: 0
rx17_mpwqe_filler_strides: 0
rx17_buff_alloc_err: 0
rx17_cqe_compress_blks: 0
rx17_cqe_compress_pkts: 0
rx17_page_reuse: 0
rx17_cache_reuse: 2652127
rx17_cache_full: 2169391411
rx17_cache_empty: 6656
rx17_cache_busy: 2169391665
rx17_cache_waive: 0
rx17_congst_umr: 0
rx17_arfs_err: 0
rx17_xdp_tx_xmit: 0
rx17_xdp_tx_full: 0
rx17_xdp_tx_err: 0
rx17_xdp_tx_cqes: 0
rx18_packets: 4407422804
rx18_bytes: 5741134634177
rx18_csum_complete: 4407422804
rx18_csum_unnecessary: 0
rx18_csum_unnecessary_inner: 0
rx18_csum_none: 0
rx18_xdp_drop: 0
rx18_xdp_redirect: 0
rx18_lro_packets: 0
rx18_lro_bytes: 0
rx18_ecn_mark: 0
rx18_removed_vlan_packets: 4407422804
rx18_wqe_err: 0
rx18_mpwqe_filler_cqes: 0
rx18_mpwqe_filler_strides: 0
rx18_buff_alloc_err: 0
rx18_cqe_compress_blks: 0
rx18_cqe_compress_pkts: 0
rx18_page_reuse: 0
rx18_cache_reuse: 2156080239
rx18_cache_full: 47630941
rx18_cache_empty: 6656
rx18_cache_busy: 47631161
rx18_cache_waive: 0
rx18_congst_umr: 0
rx18_arfs_err: 0
rx18_xdp_tx_xmit: 0
rx18_xdp_tx_full: 0
rx18_xdp_tx_err: 0
rx18_xdp_tx_cqes: 0
rx19_packets: 4545554180
rx19_bytes: 5905277503466
rx19_csum_complete: 4545554180
rx19_csum_unnecessary: 0
rx19_csum_unnecessary_inner: 0
rx19_csum_none: 0
rx19_xdp_drop: 0
rx19_xdp_redirect: 0
rx19_lro_packets: 0
rx19_lro_bytes: 0
rx19_ecn_mark: 0
rx19_removed_vlan_packets: 4545554180
rx19_wqe_err: 0
rx19_mpwqe_filler_cqes: 0
rx19_mpwqe_filler_strides: 0
rx19_buff_alloc_err: 0
rx19_cqe_compress_blks: 0
rx19_cqe_compress_pkts: 0
rx19_page_reuse: 0
rx19_cache_reuse: 11112455
rx19_cache_full: 2261664379
rx19_cache_empty: 6656
rx19_cache_busy: 2261664601
rx19_cache_waive: 0
rx19_congst_umr: 0
rx19_arfs_err: 0
rx19_xdp_tx_xmit: 0
rx19_xdp_tx_full: 0
rx19_xdp_tx_err: 0
rx19_xdp_tx_cqes: 0
rx20_packets: 4397428553
rx20_bytes: 5757329184301
rx20_csum_complete: 4397428553
rx20_csum_unnecessary: 0
rx20_csum_unnecessary_inner: 0
rx20_csum_none: 0
rx20_xdp_drop: 0
rx20_xdp_redirect: 0
rx20_lro_packets: 0
rx20_lro_bytes: 0
rx20_ecn_mark: 0
rx20_removed_vlan_packets: 4397428553
rx20_wqe_err: 0
rx20_mpwqe_filler_cqes: 0
rx20_mpwqe_filler_strides: 0
rx20_buff_alloc_err: 0
rx20_cqe_compress_blks: 0
rx20_cqe_compress_pkts: 0
rx20_page_reuse: 0
rx20_cache_reuse: 2168116995
rx20_cache_full: 30597061
rx20_cache_empty: 6656
rx20_cache_busy: 30597281
rx20_cache_waive: 0
rx20_congst_umr: 0
rx20_arfs_err: 0
rx20_xdp_tx_xmit: 0
rx20_xdp_tx_full: 0
rx20_xdp_tx_err: 0
rx20_xdp_tx_cqes: 0
rx21_packets: 4552564821
rx21_bytes: 5944840329249
rx21_csum_complete: 4552564821
rx21_csum_unnecessary: 0
rx21_csum_unnecessary_inner: 0
rx21_csum_none: 0
rx21_xdp_drop: 0
rx21_xdp_redirect: 0
rx21_lro_packets: 0
rx21_lro_bytes: 0
rx21_ecn_mark: 0
rx21_removed_vlan_packets: 4552564821
rx21_wqe_err: 0
rx21_mpwqe_filler_cqes: 0
rx21_mpwqe_filler_strides: 0
rx21_buff_alloc_err: 0
rx21_cqe_compress_blks: 0
rx21_cqe_compress_pkts: 0
rx21_page_reuse: 0
rx21_cache_reuse: 2295681
rx21_cache_full: 2273986474
rx21_cache_empty: 6656
rx21_cache_busy: 2273986727
rx21_cache_waive: 0
rx21_congst_umr: 0
rx21_arfs_err: 0
rx21_xdp_tx_xmit: 0
rx21_xdp_tx_full: 0
rx21_xdp_tx_err: 0
rx21_xdp_tx_cqes: 0
rx22_packets: 4629499740
rx22_bytes: 5924206566499
rx22_csum_complete: 4629499740
rx22_csum_unnecessary: 0
rx22_csum_unnecessary_inner: 0
rx22_csum_none: 0
rx22_xdp_drop: 0
rx22_xdp_redirect: 0
rx22_lro_packets: 0
rx22_lro_bytes: 0
rx22_ecn_mark: 0
rx22_removed_vlan_packets: 4629499740
rx22_wqe_err: 0
rx22_mpwqe_filler_cqes: 0
rx22_mpwqe_filler_strides: 0
rx22_buff_alloc_err: 0
rx22_cqe_compress_blks: 0
rx22_cqe_compress_pkts: 0
rx22_page_reuse: 0
rx22_cache_reuse: 1407527
rx22_cache_full: 2313342088
rx22_cache_empty: 6656
rx22_cache_busy: 2313342341
rx22_cache_waive: 0
rx22_congst_umr: 0
rx22_arfs_err: 0
rx22_xdp_tx_xmit: 0
rx22_xdp_tx_full: 0
rx22_xdp_tx_err: 0
rx22_xdp_tx_cqes: 0
rx23_packets: 4387124505
rx23_bytes: 5718118678470
rx23_csum_complete: 4387124505
rx23_csum_unnecessary: 0
rx23_csum_unnecessary_inner: 0
rx23_csum_none: 0
rx23_xdp_drop: 0
rx23_xdp_redirect: 0
rx23_lro_packets: 0
rx23_lro_bytes: 0
rx23_ecn_mark: 0
rx23_removed_vlan_packets: 4387124505
rx23_wqe_err: 0
rx23_mpwqe_filler_cqes: 0
rx23_mpwqe_filler_strides: 0
rx23_buff_alloc_err: 0
rx23_cqe_compress_blks: 0
rx23_cqe_compress_pkts: 0
rx23_page_reuse: 0
rx23_cache_reuse: 2013280
rx23_cache_full: 2191548717
rx23_cache_empty: 6656
rx23_cache_busy: 2191548972
rx23_cache_waive: 0
rx23_congst_umr: 0
rx23_arfs_err: 0
rx23_xdp_tx_xmit: 0
rx23_xdp_tx_full: 0
rx23_xdp_tx_err: 0
rx23_xdp_tx_cqes: 0
rx24_packets: 4398791634
rx24_bytes: 5744875564632
rx24_csum_complete: 4398791634
rx24_csum_unnecessary: 0
rx24_csum_unnecessary_inner: 0
rx24_csum_none: 0
rx24_xdp_drop: 0
rx24_xdp_redirect: 0
rx24_lro_packets: 0
rx24_lro_bytes: 0
rx24_ecn_mark: 0
rx24_removed_vlan_packets: 4398791634
rx24_wqe_err: 0
rx24_mpwqe_filler_cqes: 0
rx24_mpwqe_filler_strides: 0
rx24_buff_alloc_err: 0
rx24_cqe_compress_blks: 0
rx24_cqe_compress_pkts: 0
rx24_page_reuse: 0
rx24_cache_reuse: 2143926100
rx24_cache_full: 55469496
rx24_cache_empty: 6656
rx24_cache_busy: 55469716
rx24_cache_waive: 0
rx24_congst_umr: 0
rx24_arfs_err: 0
rx24_xdp_tx_xmit: 0
rx24_xdp_tx_full: 0
rx24_xdp_tx_err: 0
rx24_xdp_tx_cqes: 0
rx25_packets: 4377204935
rx25_bytes: 5710369124105
rx25_csum_complete: 4377204935
rx25_csum_unnecessary: 0
rx25_csum_unnecessary_inner: 0
rx25_csum_none: 0
rx25_xdp_drop: 0
rx25_xdp_redirect: 0
rx25_lro_packets: 0
rx25_lro_bytes: 0
rx25_ecn_mark: 0
rx25_removed_vlan_packets: 4377204935
rx25_wqe_err: 0
rx25_mpwqe_filler_cqes: 0
rx25_mpwqe_filler_strides: 0
rx25_buff_alloc_err: 0
rx25_cqe_compress_blks: 0
rx25_cqe_compress_pkts: 0
rx25_page_reuse: 0
rx25_cache_reuse: 2132658660
rx25_cache_full: 55943584
rx25_cache_empty: 6656
rx25_cache_busy: 55943804
rx25_cache_waive: 0
rx25_congst_umr: 0
rx25_arfs_err: 0
rx25_xdp_tx_xmit: 0
rx25_xdp_tx_full: 0
rx25_xdp_tx_err: 0
rx25_xdp_tx_cqes: 0
rx26_packets: 4496003688
rx26_bytes: 5862180715503
rx26_csum_complete: 4496003688
rx26_csum_unnecessary: 0
rx26_csum_unnecessary_inner: 0
rx26_csum_none: 0
rx26_xdp_drop: 0
rx26_xdp_redirect: 0
rx26_lro_packets: 0
rx26_lro_bytes: 0
rx26_ecn_mark: 0
rx26_removed_vlan_packets: 4496003688
rx26_wqe_err: 0
rx26_mpwqe_filler_cqes: 0
rx26_mpwqe_filler_strides: 0
rx26_buff_alloc_err: 0
rx26_cqe_compress_blks: 0
rx26_cqe_compress_pkts: 0
rx26_page_reuse: 0
rx26_cache_reuse: 8
rx26_cache_full: 2248001581
rx26_cache_empty: 6656
rx26_cache_busy: 2248001836
rx26_cache_waive: 0
rx26_congst_umr: 0
rx26_arfs_err: 0
rx26_xdp_tx_xmit: 0
rx26_xdp_tx_full: 0
rx26_xdp_tx_err: 0
rx26_xdp_tx_cqes: 0
rx27_packets: 4341849333
rx27_bytes: 5678653545018
rx27_csum_complete: 4341849333
rx27_csum_unnecessary: 0
rx27_csum_unnecessary_inner: 0
rx27_csum_none: 0
rx27_xdp_drop: 0
rx27_xdp_redirect: 0
rx27_lro_packets: 0
rx27_lro_bytes: 0
rx27_ecn_mark: 0
rx27_removed_vlan_packets: 4341849333
rx27_wqe_err: 0
rx27_mpwqe_filler_cqes: 0
rx27_mpwqe_filler_strides: 0
rx27_buff_alloc_err: 0
rx27_cqe_compress_blks: 0
rx27_cqe_compress_pkts: 0
rx27_page_reuse: 0
rx27_cache_reuse: 1748188
rx27_cache_full: 2169176223
rx27_cache_empty: 6656
rx27_cache_busy: 2169176476
rx27_cache_waive: 0
rx27_congst_umr: 0
rx27_arfs_err: 0
rx27_xdp_tx_xmit: 0
rx27_xdp_tx_full: 0
rx27_xdp_tx_err: 0
rx27_xdp_tx_cqes: 0
rx28_packets: 0
rx28_bytes: 0
rx28_csum_complete: 0
rx28_csum_unnecessary: 0
rx28_csum_unnecessary_inner: 0
rx28_csum_none: 0
rx28_xdp_drop: 0
rx28_xdp_redirect: 0
rx28_lro_packets: 0
rx28_lro_bytes: 0
rx28_ecn_mark: 0
rx28_removed_vlan_packets: 0
rx28_wqe_err: 0
rx28_mpwqe_filler_cqes: 0
rx28_mpwqe_filler_strides: 0
rx28_buff_alloc_err: 0
rx28_cqe_compress_blks: 0
rx28_cqe_compress_pkts: 0
rx28_page_reuse: 0
rx28_cache_reuse: 0
rx28_cache_full: 0
rx28_cache_empty: 2560
rx28_cache_busy: 0
rx28_cache_waive: 0
rx28_congst_umr: 0
rx28_arfs_err: 0
rx28_xdp_tx_xmit: 0
rx28_xdp_tx_full: 0
rx28_xdp_tx_err: 0
rx28_xdp_tx_cqes: 0
rx29_packets: 0
rx29_bytes: 0
rx29_csum_complete: 0
rx29_csum_unnecessary: 0
rx29_csum_unnecessary_inner: 0
rx29_csum_none: 0
rx29_xdp_drop: 0
rx29_xdp_redirect: 0
rx29_lro_packets: 0
rx29_lro_bytes: 0
rx29_ecn_mark: 0
rx29_removed_vlan_packets: 0
rx29_wqe_err: 0
rx29_mpwqe_filler_cqes: 0
rx29_mpwqe_filler_strides: 0
rx29_buff_alloc_err: 0
rx29_cqe_compress_blks: 0
rx29_cqe_compress_pkts: 0
rx29_page_reuse: 0
rx29_cache_reuse: 0
rx29_cache_full: 0
rx29_cache_empty: 2560
rx29_cache_busy: 0
rx29_cache_waive: 0
rx29_congst_umr: 0
rx29_arfs_err: 0
rx29_xdp_tx_xmit: 0
rx29_xdp_tx_full: 0
rx29_xdp_tx_err: 0
rx29_xdp_tx_cqes: 0
rx30_packets: 0
rx30_bytes: 0
rx30_csum_complete: 0
rx30_csum_unnecessary: 0
rx30_csum_unnecessary_inner: 0
rx30_csum_none: 0
rx30_xdp_drop: 0
rx30_xdp_redirect: 0
rx30_lro_packets: 0
rx30_lro_bytes: 0
rx30_ecn_mark: 0
rx30_removed_vlan_packets: 0
rx30_wqe_err: 0
rx30_mpwqe_filler_cqes: 0
rx30_mpwqe_filler_strides: 0
rx30_buff_alloc_err: 0
rx30_cqe_compress_blks: 0
rx30_cqe_compress_pkts: 0
rx30_page_reuse: 0
rx30_cache_reuse: 0
rx30_cache_full: 0
rx30_cache_empty: 2560
rx30_cache_busy: 0
rx30_cache_waive: 0
rx30_congst_umr: 0
rx30_arfs_err: 0
rx30_xdp_tx_xmit: 0
rx30_xdp_tx_full: 0
rx30_xdp_tx_err: 0
rx30_xdp_tx_cqes: 0
rx31_packets: 0
rx31_bytes: 0
rx31_csum_complete: 0
rx31_csum_unnecessary: 0
rx31_csum_unnecessary_inner: 0
rx31_csum_none: 0
rx31_xdp_drop: 0
rx31_xdp_redirect: 0
rx31_lro_packets: 0
rx31_lro_bytes: 0
rx31_ecn_mark: 0
rx31_removed_vlan_packets: 0
rx31_wqe_err: 0
rx31_mpwqe_filler_cqes: 0
rx31_mpwqe_filler_strides: 0
rx31_buff_alloc_err: 0
rx31_cqe_compress_blks: 0
rx31_cqe_compress_pkts: 0
rx31_page_reuse: 0
rx31_cache_reuse: 0
rx31_cache_full: 0
rx31_cache_empty: 2560
rx31_cache_busy: 0
rx31_cache_waive: 0
rx31_congst_umr: 0
rx31_arfs_err: 0
rx31_xdp_tx_xmit: 0
rx31_xdp_tx_full: 0
rx31_xdp_tx_err: 0
rx31_xdp_tx_cqes: 0
rx32_packets: 0
rx32_bytes: 0
rx32_csum_complete: 0
rx32_csum_unnecessary: 0
rx32_csum_unnecessary_inner: 0
rx32_csum_none: 0
rx32_xdp_drop: 0
rx32_xdp_redirect: 0
rx32_lro_packets: 0
rx32_lro_bytes: 0
rx32_ecn_mark: 0
rx32_removed_vlan_packets: 0
rx32_wqe_err: 0
rx32_mpwqe_filler_cqes: 0
rx32_mpwqe_filler_strides: 0
rx32_buff_alloc_err: 0
rx32_cqe_compress_blks: 0
rx32_cqe_compress_pkts: 0
rx32_page_reuse: 0
rx32_cache_reuse: 0
rx32_cache_full: 0
rx32_cache_empty: 2560
rx32_cache_busy: 0
rx32_cache_waive: 0
rx32_congst_umr: 0
rx32_arfs_err: 0
rx32_xdp_tx_xmit: 0
rx32_xdp_tx_full: 0
rx32_xdp_tx_err: 0
rx32_xdp_tx_cqes: 0
rx33_packets: 0
rx33_bytes: 0
rx33_csum_complete: 0
rx33_csum_unnecessary: 0
rx33_csum_unnecessary_inner: 0
rx33_csum_none: 0
rx33_xdp_drop: 0
rx33_xdp_redirect: 0
rx33_lro_packets: 0
rx33_lro_bytes: 0
rx33_ecn_mark: 0
rx33_removed_vlan_packets: 0
rx33_wqe_err: 0
rx33_mpwqe_filler_cqes: 0
rx33_mpwqe_filler_strides: 0
rx33_buff_alloc_err: 0
rx33_cqe_compress_blks: 0
rx33_cqe_compress_pkts: 0
rx33_page_reuse: 0
rx33_cache_reuse: 0
rx33_cache_full: 0
rx33_cache_empty: 2560
rx33_cache_busy: 0
rx33_cache_waive: 0
rx33_congst_umr: 0
rx33_arfs_err: 0
rx33_xdp_tx_xmit: 0
rx33_xdp_tx_full: 0
rx33_xdp_tx_err: 0
rx33_xdp_tx_cqes: 0
rx34_packets: 0
rx34_bytes: 0
rx34_csum_complete: 0
rx34_csum_unnecessary: 0
rx34_csum_unnecessary_inner: 0
rx34_csum_none: 0
rx34_xdp_drop: 0
rx34_xdp_redirect: 0
rx34_lro_packets: 0
rx34_lro_bytes: 0
rx34_ecn_mark: 0
rx34_removed_vlan_packets: 0
rx34_wqe_err: 0
rx34_mpwqe_filler_cqes: 0
rx34_mpwqe_filler_strides: 0
rx34_buff_alloc_err: 0
rx34_cqe_compress_blks: 0
rx34_cqe_compress_pkts: 0
rx34_page_reuse: 0
rx34_cache_reuse: 0
rx34_cache_full: 0
rx34_cache_empty: 2560
rx34_cache_busy: 0
rx34_cache_waive: 0
rx34_congst_umr: 0
rx34_arfs_err: 0
rx34_xdp_tx_xmit: 0
rx34_xdp_tx_full: 0
rx34_xdp_tx_err: 0
rx34_xdp_tx_cqes: 0
rx35_packets: 0
rx35_bytes: 0
rx35_csum_complete: 0
rx35_csum_unnecessary: 0
rx35_csum_unnecessary_inner: 0
rx35_csum_none: 0
rx35_xdp_drop: 0
rx35_xdp_redirect: 0
rx35_lro_packets: 0
rx35_lro_bytes: 0
rx35_ecn_mark: 0
rx35_removed_vlan_packets: 0
rx35_wqe_err: 0
rx35_mpwqe_filler_cqes: 0
rx35_mpwqe_filler_strides: 0
rx35_buff_alloc_err: 0
rx35_cqe_compress_blks: 0
rx35_cqe_compress_pkts: 0
rx35_page_reuse: 0
rx35_cache_reuse: 0
rx35_cache_full: 0
rx35_cache_empty: 2560
rx35_cache_busy: 0
rx35_cache_waive: 0
rx35_congst_umr: 0
rx35_arfs_err: 0
rx35_xdp_tx_xmit: 0
rx35_xdp_tx_full: 0
rx35_xdp_tx_err: 0
rx35_xdp_tx_cqes: 0
rx36_packets: 0
rx36_bytes: 0
rx36_csum_complete: 0
rx36_csum_unnecessary: 0
rx36_csum_unnecessary_inner: 0
rx36_csum_none: 0
rx36_xdp_drop: 0
rx36_xdp_redirect: 0
rx36_lro_packets: 0
rx36_lro_bytes: 0
rx36_ecn_mark: 0
rx36_removed_vlan_packets: 0
rx36_wqe_err: 0
rx36_mpwqe_filler_cqes: 0
rx36_mpwqe_filler_strides: 0
rx36_buff_alloc_err: 0
rx36_cqe_compress_blks: 0
rx36_cqe_compress_pkts: 0
rx36_page_reuse: 0
rx36_cache_reuse: 0
rx36_cache_full: 0
rx36_cache_empty: 2560
rx36_cache_busy: 0
rx36_cache_waive: 0
rx36_congst_umr: 0
rx36_arfs_err: 0
rx36_xdp_tx_xmit: 0
rx36_xdp_tx_full: 0
rx36_xdp_tx_err: 0
rx36_xdp_tx_cqes: 0
rx37_packets: 0
rx37_bytes: 0
rx37_csum_complete: 0
rx37_csum_unnecessary: 0
rx37_csum_unnecessary_inner: 0
rx37_csum_none: 0
rx37_xdp_drop: 0
rx37_xdp_redirect: 0
rx37_lro_packets: 0
rx37_lro_bytes: 0
rx37_ecn_mark: 0
rx37_removed_vlan_packets: 0
rx37_wqe_err: 0
rx37_mpwqe_filler_cqes: 0
rx37_mpwqe_filler_strides: 0
rx37_buff_alloc_err: 0
rx37_cqe_compress_blks: 0
rx37_cqe_compress_pkts: 0
rx37_page_reuse: 0
rx37_cache_reuse: 0
rx37_cache_full: 0
rx37_cache_empty: 2560
rx37_cache_busy: 0
rx37_cache_waive: 0
rx37_congst_umr: 0
rx37_arfs_err: 0
rx37_xdp_tx_xmit: 0
rx37_xdp_tx_full: 0
rx37_xdp_tx_err: 0
rx37_xdp_tx_cqes: 0
rx38_packets: 0
rx38_bytes: 0
rx38_csum_complete: 0
rx38_csum_unnecessary: 0
rx38_csum_unnecessary_inner: 0
rx38_csum_none: 0
rx38_xdp_drop: 0
rx38_xdp_redirect: 0
rx38_lro_packets: 0
rx38_lro_bytes: 0
rx38_ecn_mark: 0
rx38_removed_vlan_packets: 0
rx38_wqe_err: 0
rx38_mpwqe_filler_cqes: 0
rx38_mpwqe_filler_strides: 0
rx38_buff_alloc_err: 0
rx38_cqe_compress_blks: 0
rx38_cqe_compress_pkts: 0
rx38_page_reuse: 0
rx38_cache_reuse: 0
rx38_cache_full: 0
rx38_cache_empty: 2560
rx38_cache_busy: 0
rx38_cache_waive: 0
rx38_congst_umr: 0
rx38_arfs_err: 0
rx38_xdp_tx_xmit: 0
rx38_xdp_tx_full: 0
rx38_xdp_tx_err: 0
rx38_xdp_tx_cqes: 0
rx39_packets: 0
rx39_bytes: 0
rx39_csum_complete: 0
rx39_csum_unnecessary: 0
rx39_csum_unnecessary_inner: 0
rx39_csum_none: 0
rx39_xdp_drop: 0
rx39_xdp_redirect: 0
rx39_lro_packets: 0
rx39_lro_bytes: 0
rx39_ecn_mark: 0
rx39_removed_vlan_packets: 0
rx39_wqe_err: 0
rx39_mpwqe_filler_cqes: 0
rx39_mpwqe_filler_strides: 0
rx39_buff_alloc_err: 0
rx39_cqe_compress_blks: 0
rx39_cqe_compress_pkts: 0
rx39_page_reuse: 0
rx39_cache_reuse: 0
rx39_cache_full: 0
rx39_cache_empty: 2560
rx39_cache_busy: 0
rx39_cache_waive: 0
rx39_congst_umr: 0
rx39_arfs_err: 0
rx39_xdp_tx_xmit: 0
rx39_xdp_tx_full: 0
rx39_xdp_tx_err: 0
rx39_xdp_tx_cqes: 0
rx40_packets: 0
rx40_bytes: 0
rx40_csum_complete: 0
rx40_csum_unnecessary: 0
rx40_csum_unnecessary_inner: 0
rx40_csum_none: 0
rx40_xdp_drop: 0
rx40_xdp_redirect: 0
rx40_lro_packets: 0
rx40_lro_bytes: 0
rx40_ecn_mark: 0
rx40_removed_vlan_packets: 0
rx40_wqe_err: 0
rx40_mpwqe_filler_cqes: 0
rx40_mpwqe_filler_strides: 0
rx40_buff_alloc_err: 0
rx40_cqe_compress_blks: 0
rx40_cqe_compress_pkts: 0
rx40_page_reuse: 0
rx40_cache_reuse: 0
rx40_cache_full: 0
rx40_cache_empty: 2560
rx40_cache_busy: 0
rx40_cache_waive: 0
rx40_congst_umr: 0
rx40_arfs_err: 0
rx40_xdp_tx_xmit: 0
rx40_xdp_tx_full: 0
rx40_xdp_tx_err: 0
rx40_xdp_tx_cqes: 0
rx41_packets: 0
rx41_bytes: 0
rx41_csum_complete: 0
rx41_csum_unnecessary: 0
rx41_csum_unnecessary_inner: 0
rx41_csum_none: 0
rx41_xdp_drop: 0
rx41_xdp_redirect: 0
rx41_lro_packets: 0
rx41_lro_bytes: 0
rx41_ecn_mark: 0
rx41_removed_vlan_packets: 0
rx41_wqe_err: 0
rx41_mpwqe_filler_cqes: 0
rx41_mpwqe_filler_strides: 0
rx41_buff_alloc_err: 0
rx41_cqe_compress_blks: 0
rx41_cqe_compress_pkts: 0
rx41_page_reuse: 0
rx41_cache_reuse: 0
rx41_cache_full: 0
rx41_cache_empty: 2560
rx41_cache_busy: 0
rx41_cache_waive: 0
rx41_congst_umr: 0
rx41_arfs_err: 0
rx41_xdp_tx_xmit: 0
rx41_xdp_tx_full: 0
rx41_xdp_tx_err: 0
rx41_xdp_tx_cqes: 0
rx42_packets: 0
rx42_bytes: 0
rx42_csum_complete: 0
rx42_csum_unnecessary: 0
rx42_csum_unnecessary_inner: 0
rx42_csum_none: 0
rx42_xdp_drop: 0
rx42_xdp_redirect: 0
rx42_lro_packets: 0
rx42_lro_bytes: 0
rx42_ecn_mark: 0
rx42_removed_vlan_packets: 0
rx42_wqe_err: 0
rx42_mpwqe_filler_cqes: 0
rx42_mpwqe_filler_strides: 0
rx42_buff_alloc_err: 0
rx42_cqe_compress_blks: 0
rx42_cqe_compress_pkts: 0
rx42_page_reuse: 0
rx42_cache_reuse: 0
rx42_cache_full: 0
rx42_cache_empty: 2560
rx42_cache_busy: 0
rx42_cache_waive: 0
rx42_congst_umr: 0
rx42_arfs_err: 0
rx42_xdp_tx_xmit: 0
rx42_xdp_tx_full: 0
rx42_xdp_tx_err: 0
rx42_xdp_tx_cqes: 0
rx43_packets: 0
rx43_bytes: 0
rx43_csum_complete: 0
rx43_csum_unnecessary: 0
rx43_csum_unnecessary_inner: 0
rx43_csum_none: 0
rx43_xdp_drop: 0
rx43_xdp_redirect: 0
rx43_lro_packets: 0
rx43_lro_bytes: 0
rx43_ecn_mark: 0
rx43_removed_vlan_packets: 0
rx43_wqe_err: 0
rx43_mpwqe_filler_cqes: 0
rx43_mpwqe_filler_strides: 0
rx43_buff_alloc_err: 0
rx43_cqe_compress_blks: 0
rx43_cqe_compress_pkts: 0
rx43_page_reuse: 0
rx43_cache_reuse: 0
rx43_cache_full: 0
rx43_cache_empty: 2560
rx43_cache_busy: 0
rx43_cache_waive: 0
rx43_congst_umr: 0
rx43_arfs_err: 0
rx43_xdp_tx_xmit: 0
rx43_xdp_tx_full: 0
rx43_xdp_tx_err: 0
rx43_xdp_tx_cqes: 0
rx44_packets: 0
rx44_bytes: 0
rx44_csum_complete: 0
rx44_csum_unnecessary: 0
rx44_csum_unnecessary_inner: 0
rx44_csum_none: 0
rx44_xdp_drop: 0
rx44_xdp_redirect: 0
rx44_lro_packets: 0
rx44_lro_bytes: 0
rx44_ecn_mark: 0
rx44_removed_vlan_packets: 0
rx44_wqe_err: 0
rx44_mpwqe_filler_cqes: 0
rx44_mpwqe_filler_strides: 0
rx44_buff_alloc_err: 0
rx44_cqe_compress_blks: 0
rx44_cqe_compress_pkts: 0
rx44_page_reuse: 0
rx44_cache_reuse: 0
rx44_cache_full: 0
rx44_cache_empty: 2560
rx44_cache_busy: 0
rx44_cache_waive: 0
rx44_congst_umr: 0
rx44_arfs_err: 0
rx44_xdp_tx_xmit: 0
rx44_xdp_tx_full: 0
rx44_xdp_tx_err: 0
rx44_xdp_tx_cqes: 0
rx45_packets: 0
rx45_bytes: 0
rx45_csum_complete: 0
rx45_csum_unnecessary: 0
rx45_csum_unnecessary_inner: 0
rx45_csum_none: 0
rx45_xdp_drop: 0
rx45_xdp_redirect: 0
rx45_lro_packets: 0
rx45_lro_bytes: 0
rx45_ecn_mark: 0
rx45_removed_vlan_packets: 0
rx45_wqe_err: 0
rx45_mpwqe_filler_cqes: 0
rx45_mpwqe_filler_strides: 0
rx45_buff_alloc_err: 0
rx45_cqe_compress_blks: 0
rx45_cqe_compress_pkts: 0
rx45_page_reuse: 0
rx45_cache_reuse: 0
rx45_cache_full: 0
rx45_cache_empty: 2560
rx45_cache_busy: 0
rx45_cache_waive: 0
rx45_congst_umr: 0
rx45_arfs_err: 0
rx45_xdp_tx_xmit: 0
rx45_xdp_tx_full: 0
rx45_xdp_tx_err: 0
rx45_xdp_tx_cqes: 0
rx46_packets: 0
rx46_bytes: 0
rx46_csum_complete: 0
rx46_csum_unnecessary: 0
rx46_csum_unnecessary_inner: 0
rx46_csum_none: 0
rx46_xdp_drop: 0
rx46_xdp_redirect: 0
rx46_lro_packets: 0
rx46_lro_bytes: 0
rx46_ecn_mark: 0
rx46_removed_vlan_packets: 0
rx46_wqe_err: 0
rx46_mpwqe_filler_cqes: 0
rx46_mpwqe_filler_strides: 0
rx46_buff_alloc_err: 0
rx46_cqe_compress_blks: 0
rx46_cqe_compress_pkts: 0
rx46_page_reuse: 0
rx46_cache_reuse: 0
rx46_cache_full: 0
rx46_cache_empty: 2560
rx46_cache_busy: 0
rx46_cache_waive: 0
rx46_congst_umr: 0
rx46_arfs_err: 0
rx46_xdp_tx_xmit: 0
rx46_xdp_tx_full: 0
rx46_xdp_tx_err: 0
rx46_xdp_tx_cqes: 0
rx47_packets: 0
rx47_bytes: 0
rx47_csum_complete: 0
rx47_csum_unnecessary: 0
rx47_csum_unnecessary_inner: 0
rx47_csum_none: 0
rx47_xdp_drop: 0
rx47_xdp_redirect: 0
rx47_lro_packets: 0
rx47_lro_bytes: 0
rx47_ecn_mark: 0
rx47_removed_vlan_packets: 0
rx47_wqe_err: 0
rx47_mpwqe_filler_cqes: 0
rx47_mpwqe_filler_strides: 0
rx47_buff_alloc_err: 0
rx47_cqe_compress_blks: 0
rx47_cqe_compress_pkts: 0
rx47_page_reuse: 0
rx47_cache_reuse: 0
rx47_cache_full: 0
rx47_cache_empty: 2560
rx47_cache_busy: 0
rx47_cache_waive: 0
rx47_congst_umr: 0
rx47_arfs_err: 0
rx47_xdp_tx_xmit: 0
rx47_xdp_tx_full: 0
rx47_xdp_tx_err: 0
rx47_xdp_tx_cqes: 0
rx48_packets: 0
rx48_bytes: 0
rx48_csum_complete: 0
rx48_csum_unnecessary: 0
rx48_csum_unnecessary_inner: 0
rx48_csum_none: 0
rx48_xdp_drop: 0
rx48_xdp_redirect: 0
rx48_lro_packets: 0
rx48_lro_bytes: 0
rx48_ecn_mark: 0
rx48_removed_vlan_packets: 0
rx48_wqe_err: 0
rx48_mpwqe_filler_cqes: 0
rx48_mpwqe_filler_strides: 0
rx48_buff_alloc_err: 0
rx48_cqe_compress_blks: 0
rx48_cqe_compress_pkts: 0
rx48_page_reuse: 0
rx48_cache_reuse: 0
rx48_cache_full: 0
rx48_cache_empty: 2560
rx48_cache_busy: 0
rx48_cache_waive: 0
rx48_congst_umr: 0
rx48_arfs_err: 0
rx48_xdp_tx_xmit: 0
rx48_xdp_tx_full: 0
rx48_xdp_tx_err: 0
rx48_xdp_tx_cqes: 0
rx49_packets: 0
rx49_bytes: 0
rx49_csum_complete: 0
rx49_csum_unnecessary: 0
rx49_csum_unnecessary_inner: 0
rx49_csum_none: 0
rx49_xdp_drop: 0
rx49_xdp_redirect: 0
rx49_lro_packets: 0
rx49_lro_bytes: 0
rx49_ecn_mark: 0
rx49_removed_vlan_packets: 0
rx49_wqe_err: 0
rx49_mpwqe_filler_cqes: 0
rx49_mpwqe_filler_strides: 0
rx49_buff_alloc_err: 0
rx49_cqe_compress_blks: 0
rx49_cqe_compress_pkts: 0
rx49_page_reuse: 0
rx49_cache_reuse: 0
rx49_cache_full: 0
rx49_cache_empty: 2560
rx49_cache_busy: 0
rx49_cache_waive: 0
rx49_congst_umr: 0
rx49_arfs_err: 0
rx49_xdp_tx_xmit: 0
rx49_xdp_tx_full: 0
rx49_xdp_tx_err: 0
rx49_xdp_tx_cqes: 0
rx50_packets: 0
rx50_bytes: 0
rx50_csum_complete: 0
rx50_csum_unnecessary: 0
rx50_csum_unnecessary_inner: 0
rx50_csum_none: 0
rx50_xdp_drop: 0
rx50_xdp_redirect: 0
rx50_lro_packets: 0
rx50_lro_bytes: 0
rx50_ecn_mark: 0
rx50_removed_vlan_packets: 0
rx50_wqe_err: 0
rx50_mpwqe_filler_cqes: 0
rx50_mpwqe_filler_strides: 0
rx50_buff_alloc_err: 0
rx50_cqe_compress_blks: 0
rx50_cqe_compress_pkts: 0
rx50_page_reuse: 0
rx50_cache_reuse: 0
rx50_cache_full: 0
rx50_cache_empty: 2560
rx50_cache_busy: 0
rx50_cache_waive: 0
rx50_congst_umr: 0
rx50_arfs_err: 0
rx50_xdp_tx_xmit: 0
rx50_xdp_tx_full: 0
rx50_xdp_tx_err: 0
rx50_xdp_tx_cqes: 0
rx51_packets: 0
rx51_bytes: 0
rx51_csum_complete: 0
rx51_csum_unnecessary: 0
rx51_csum_unnecessary_inner: 0
rx51_csum_none: 0
rx51_xdp_drop: 0
rx51_xdp_redirect: 0
rx51_lro_packets: 0
rx51_lro_bytes: 0
rx51_ecn_mark: 0
rx51_removed_vlan_packets: 0
rx51_wqe_err: 0
rx51_mpwqe_filler_cqes: 0
rx51_mpwqe_filler_strides: 0
rx51_buff_alloc_err: 0
rx51_cqe_compress_blks: 0
rx51_cqe_compress_pkts: 0
rx51_page_reuse: 0
rx51_cache_reuse: 0
rx51_cache_full: 0
rx51_cache_empty: 2560
rx51_cache_busy: 0
rx51_cache_waive: 0
rx51_congst_umr: 0
rx51_arfs_err: 0
rx51_xdp_tx_xmit: 0
rx51_xdp_tx_full: 0
rx51_xdp_tx_err: 0
rx51_xdp_tx_cqes: 0
rx52_packets: 0
rx52_bytes: 0
rx52_csum_complete: 0
rx52_csum_unnecessary: 0
rx52_csum_unnecessary_inner: 0
rx52_csum_none: 0
rx52_xdp_drop: 0
rx52_xdp_redirect: 0
rx52_lro_packets: 0
rx52_lro_bytes: 0
rx52_ecn_mark: 0
rx52_removed_vlan_packets: 0
rx52_wqe_err: 0
rx52_mpwqe_filler_cqes: 0
rx52_mpwqe_filler_strides: 0
rx52_buff_alloc_err: 0
rx52_cqe_compress_blks: 0
rx52_cqe_compress_pkts: 0
rx52_page_reuse: 0
rx52_cache_reuse: 0
rx52_cache_full: 0
rx52_cache_empty: 2560
rx52_cache_busy: 0
rx52_cache_waive: 0
rx52_congst_umr: 0
rx52_arfs_err: 0
rx52_xdp_tx_xmit: 0
rx52_xdp_tx_full: 0
rx52_xdp_tx_err: 0
rx52_xdp_tx_cqes: 0
rx53_packets: 0
rx53_bytes: 0
rx53_csum_complete: 0
rx53_csum_unnecessary: 0
rx53_csum_unnecessary_inner: 0
rx53_csum_none: 0
rx53_xdp_drop: 0
rx53_xdp_redirect: 0
rx53_lro_packets: 0
rx53_lro_bytes: 0
rx53_ecn_mark: 0
rx53_removed_vlan_packets: 0
rx53_wqe_err: 0
rx53_mpwqe_filler_cqes: 0
rx53_mpwqe_filler_strides: 0
rx53_buff_alloc_err: 0
rx53_cqe_compress_blks: 0
rx53_cqe_compress_pkts: 0
rx53_page_reuse: 0
rx53_cache_reuse: 0
rx53_cache_full: 0
rx53_cache_empty: 2560
rx53_cache_busy: 0
rx53_cache_waive: 0
rx53_congst_umr: 0
rx53_arfs_err: 0
rx53_xdp_tx_xmit: 0
rx53_xdp_tx_full: 0
rx53_xdp_tx_err: 0
rx53_xdp_tx_cqes: 0
rx54_packets: 0
rx54_bytes: 0
rx54_csum_complete: 0
rx54_csum_unnecessary: 0
rx54_csum_unnecessary_inner: 0
rx54_csum_none: 0
rx54_xdp_drop: 0
rx54_xdp_redirect: 0
rx54_lro_packets: 0
rx54_lro_bytes: 0
rx54_ecn_mark: 0
rx54_removed_vlan_packets: 0
rx54_wqe_err: 0
rx54_mpwqe_filler_cqes: 0
rx54_mpwqe_filler_strides: 0
rx54_buff_alloc_err: 0
rx54_cqe_compress_blks: 0
rx54_cqe_compress_pkts: 0
rx54_page_reuse: 0
rx54_cache_reuse: 0
rx54_cache_full: 0
rx54_cache_empty: 2560
rx54_cache_busy: 0
rx54_cache_waive: 0
rx54_congst_umr: 0
rx54_arfs_err: 0
rx54_xdp_tx_xmit: 0
rx54_xdp_tx_full: 0
rx54_xdp_tx_err: 0
rx54_xdp_tx_cqes: 0
rx55_packets: 0
rx55_bytes: 0
rx55_csum_complete: 0
rx55_csum_unnecessary: 0
rx55_csum_unnecessary_inner: 0
rx55_csum_none: 0
rx55_xdp_drop: 0
rx55_xdp_redirect: 0
rx55_lro_packets: 0
rx55_lro_bytes: 0
rx55_ecn_mark: 0
rx55_removed_vlan_packets: 0
rx55_wqe_err: 0
rx55_mpwqe_filler_cqes: 0
rx55_mpwqe_filler_strides: 0
rx55_buff_alloc_err: 0
rx55_cqe_compress_blks: 0
rx55_cqe_compress_pkts: 0
rx55_page_reuse: 0
rx55_cache_reuse: 0
rx55_cache_full: 0
rx55_cache_empty: 2560
rx55_cache_busy: 0
rx55_cache_waive: 0
rx55_congst_umr: 0
rx55_arfs_err: 0
rx55_xdp_tx_xmit: 0
rx55_xdp_tx_full: 0
rx55_xdp_tx_err: 0
rx55_xdp_tx_cqes: 0
tx0_packets: 6019477917
tx0_bytes: 3445238940825
tx0_tso_packets: 311304622
tx0_tso_bytes: 1897094773213
tx0_tso_inner_packets: 0
tx0_tso_inner_bytes: 0
tx0_csum_partial: 457981794
tx0_csum_partial_inner: 0
tx0_added_vlan_packets: 4965567654
tx0_nop: 72290329
tx0_csum_none: 4507585860
tx0_stopped: 9118
tx0_dropped: 0
tx0_xmit_more: 51651593
tx0_recover: 0
tx0_cqes: 4913918402
tx0_wake: 9118
tx0_cqe_err: 0
tx1_packets: 5700413414
tx1_bytes: 3340870662350
tx1_tso_packets: 318201557
tx1_tso_bytes: 1915233462303
tx1_tso_inner_packets: 0
tx1_tso_inner_bytes: 0
tx1_csum_partial: 461736722
tx1_csum_partial_inner: 0
tx1_added_vlan_packets: 4638708749
tx1_nop: 70061796
tx1_csum_none: 4176972027
tx1_stopped: 9248
tx1_dropped: 0
tx1_xmit_more: 39531959
tx1_recover: 0
tx1_cqes: 4599179178
tx1_wake: 9248
tx1_cqe_err: 0
tx2_packets: 5795960848
tx2_bytes: 3394876820271
tx2_tso_packets: 322935065
tx2_tso_bytes: 1910825901109
tx2_tso_inner_packets: 0
tx2_tso_inner_bytes: 0
tx2_csum_partial: 460747092
tx2_csum_partial_inner: 0
tx2_added_vlan_packets: 4743705654
tx2_nop: 72722430
tx2_csum_none: 4282958562
tx2_stopped: 8938
tx2_dropped: 0
tx2_xmit_more: 44084718
tx2_recover: 0
tx2_cqes: 4699623410
tx2_wake: 8938
tx2_cqe_err: 0
tx3_packets: 5580215878
tx3_bytes: 3191677257787
tx3_tso_packets: 305771141
tx3_tso_bytes: 1823265793476
tx3_tso_inner_packets: 0
tx3_tso_inner_bytes: 0
tx3_csum_partial: 434976070
tx3_csum_partial_inner: 0
tx3_added_vlan_packets: 4569899956
tx3_nop: 68184348
tx3_csum_none: 4134923886
tx3_stopped: 8383
tx3_dropped: 0
tx3_xmit_more: 41940375
tx3_recover: 0
tx3_cqes: 4527961924
tx3_wake: 8383
tx3_cqe_err: 0
tx4_packets: 6795007068
tx4_bytes: 3963890025270
tx4_tso_packets: 358437617
tx4_tso_bytes: 2154747995355
tx4_tso_inner_packets: 0
tx4_tso_inner_bytes: 0
tx4_csum_partial: 504764524
tx4_csum_partial_inner: 0
tx4_added_vlan_packets: 5602510191
tx4_nop: 81345604
tx4_csum_none: 5097745667
tx4_stopped: 10248
tx4_dropped: 0
tx4_xmit_more: 49068571
tx4_recover: 0
tx4_cqes: 5553444276
tx4_wake: 10248
tx4_cqe_err: 0
tx5_packets: 6408089261
tx5_bytes: 3676275848279
tx5_tso_packets: 345129329
tx5_tso_bytes: 2108447877473
tx5_tso_inner_packets: 0
tx5_tso_inner_bytes: 0
tx5_csum_partial: 494705894
tx5_csum_partial_inner: 0
tx5_added_vlan_packets: 5235998343
tx5_nop: 77694627
tx5_csum_none: 4741292449
tx5_stopped: 46
tx5_dropped: 0
tx5_xmit_more: 46675831
tx5_recover: 0
tx5_cqes: 5189323550
tx5_wake: 46
tx5_cqe_err: 0
tx6_packets: 6382289663
tx6_bytes: 3670991418150
tx6_tso_packets: 342927826
tx6_tso_bytes: 2075049679904
tx6_tso_inner_packets: 0
tx6_tso_inner_bytes: 0
tx6_csum_partial: 490369221
tx6_csum_partial_inner: 0
tx6_added_vlan_packets: 5232144528
tx6_nop: 77391246
tx6_csum_none: 4741775307
tx6_stopped: 10823
tx6_dropped: 0
tx6_xmit_more: 44487607
tx6_recover: 0
tx6_cqes: 5187659877
tx6_wake: 10823
tx6_cqe_err: 0
tx7_packets: 6456378284
tx7_bytes: 3758013320518
tx7_tso_packets: 350958294
tx7_tso_bytes: 2126833408524
tx7_tso_inner_packets: 0
tx7_tso_inner_bytes: 0
tx7_csum_partial: 501804109
tx7_csum_partial_inner: 0
tx7_added_vlan_packets: 5275635204
tx7_nop: 79010883
tx7_csum_none: 4773831096
tx7_stopped: 14684
tx7_dropped: 0
tx7_xmit_more: 44447469
tx7_recover: 0
tx7_cqes: 5231191770
tx7_wake: 14684
tx7_cqe_err: 0
tx8_packets: 6401799768
tx8_bytes: 3681210808766
tx8_tso_packets: 342878228
tx8_tso_bytes: 2089688012191
tx8_tso_inner_packets: 0
tx8_tso_inner_bytes: 0
tx8_csum_partial: 494865145
tx8_csum_partial_inner: 0
tx8_added_vlan_packets: 5242288908
tx8_nop: 77250910
tx8_csum_none: 4747423763
tx8_stopped: 2
tx8_dropped: 0
tx8_xmit_more: 44191737
tx8_recover: 0
tx8_cqes: 5198098454
tx8_wake: 2
tx8_cqe_err: 0
tx9_packets: 6632882888
tx9_bytes: 3820110338309
tx9_tso_packets: 354189056
tx9_tso_bytes: 2187883597128
tx9_tso_inner_packets: 0
tx9_tso_inner_bytes: 0
tx9_csum_partial: 511108218
tx9_csum_partial_inner: 0
tx9_added_vlan_packets: 5413836353
tx9_nop: 80560668
tx9_csum_none: 4902728135
tx9_stopped: 9091
tx9_dropped: 0
tx9_xmit_more: 54501293
tx9_recover: 0
tx9_cqes: 5359337638
tx9_wake: 9091
tx9_cqe_err: 0
tx10_packets: 6421786406
tx10_bytes: 3692798413429
tx10_tso_packets: 346878943
tx10_tso_bytes: 2111921062110
tx10_tso_inner_packets: 0
tx10_tso_inner_bytes: 0
tx10_csum_partial: 494356645
tx10_csum_partial_inner: 0
tx10_added_vlan_packets: 5248274374
tx10_nop: 77922624
tx10_csum_none: 4753917730
tx10_stopped: 9617
tx10_dropped: 0
tx10_xmit_more: 44473939
tx10_recover: 0
tx10_cqes: 5203802547
tx10_wake: 9617
tx10_cqe_err: 0
tx11_packets: 6406750938
tx11_bytes: 3660343565126
tx11_tso_packets: 355917271
tx11_tso_bytes: 2130812246956
tx11_tso_inner_packets: 0
tx11_tso_inner_bytes: 0
tx11_csum_partial: 500336369
tx11_csum_partial_inner: 0
tx11_added_vlan_packets: 5228267547
tx11_nop: 78906315
tx11_csum_none: 4727931178
tx11_stopped: 9607
tx11_dropped: 0
tx11_xmit_more: 40041492
tx11_recover: 0
tx11_cqes: 5188228290
tx11_wake: 9607
tx11_cqe_err: 0
tx12_packets: 6422347846
tx12_bytes: 3718772753227
tx12_tso_packets: 355397223
tx12_tso_bytes: 2162614059758
tx12_tso_inner_packets: 0
tx12_tso_inner_bytes: 0
tx12_csum_partial: 511437844
tx12_csum_partial_inner: 0
tx12_added_vlan_packets: 5221373746
tx12_nop: 78866779
tx12_csum_none: 4709935902
tx12_stopped: 10280
tx12_dropped: 0
tx12_xmit_more: 42189399
tx12_recover: 0
tx12_cqes: 5179187154
tx12_wake: 10280
tx12_cqe_err: 0
tx13_packets: 6429383816
tx13_bytes: 3725679445046
tx13_tso_packets: 360934759
tx13_tso_bytes: 2148016411436
tx13_tso_inner_packets: 0
tx13_tso_inner_bytes: 0
tx13_csum_partial: 505245849
tx13_csum_partial_inner: 0
tx13_added_vlan_packets: 5240267441
tx13_nop: 80295637
tx13_csum_none: 4735021592
tx13_stopped: 84
tx13_dropped: 0
tx13_xmit_more: 43118045
tx13_recover: 0
tx13_cqes: 5197150348
tx13_wake: 84
tx13_cqe_err: 0
tx14_packets: 6375279148
tx14_bytes: 3624267203336
tx14_tso_packets: 344388148
tx14_tso_bytes: 2094966273548
tx14_tso_inner_packets: 0
tx14_tso_inner_bytes: 0
tx14_csum_partial: 494129407
tx14_csum_partial_inner: 0
tx14_added_vlan_packets: 5210749337
tx14_nop: 77280615
tx14_csum_none: 4716619930
tx14_stopped: 13057
tx14_dropped: 0
tx14_xmit_more: 40849682
tx14_recover: 0
tx14_cqes: 5169902694
tx14_wake: 13057
tx14_cqe_err: 0
tx15_packets: 6489306520
tx15_bytes: 3775716194795
tx15_tso_packets: 368716406
tx15_tso_bytes: 2165876423354
tx15_tso_inner_packets: 0
tx15_tso_inner_bytes: 0
tx15_csum_partial: 509887864
tx15_csum_partial_inner: 0
tx15_added_vlan_packets: 5296767390
tx15_nop: 80803468
tx15_csum_none: 4786879529
tx15_stopped: 1
tx15_dropped: 0
tx15_xmit_more: 46979676
tx15_recover: 0
tx15_cqes: 5249789328
tx15_wake: 1
tx15_cqe_err: 0
tx16_packets: 6559857761
tx16_bytes: 3724080573905
tx16_tso_packets: 350864176
tx16_tso_bytes: 2099634006033
tx16_tso_inner_packets: 0
tx16_tso_inner_bytes: 0
tx16_csum_partial: 489397232
tx16_csum_partial_inner: 0
tx16_added_vlan_packets: 5398869334
tx16_nop: 79046075
tx16_csum_none: 4909472106
tx16_stopped: 4480
tx16_dropped: 0
tx16_xmit_more: 47273286
tx16_recover: 0
tx16_cqes: 5351598315
tx16_wake: 4480
tx16_cqe_err: 0
tx17_packets: 6358711533
tx17_bytes: 3650180865573
tx17_tso_packets: 350723136
tx17_tso_bytes: 2109426587128
tx17_tso_inner_packets: 0
tx17_tso_inner_bytes: 0
tx17_csum_partial: 494719487
tx17_csum_partial_inner: 0
tx17_added_vlan_packets: 5190068796
tx17_nop: 77285612
tx17_csum_none: 4695349309
tx17_stopped: 10443
tx17_dropped: 0
tx17_xmit_more: 45582108
tx17_recover: 0
tx17_cqes: 5144489363
tx17_wake: 10443
tx17_cqe_err: 0
tx18_packets: 6655328437
tx18_bytes: 3801768461807
tx18_tso_packets: 356516373
tx18_tso_bytes: 2164829247550
tx18_tso_inner_packets: 0
tx18_tso_inner_bytes: 0
tx18_csum_partial: 500508446
tx18_csum_partial_inner: 0
tx18_added_vlan_packets: 5454166840
tx18_nop: 80423007
tx18_csum_none: 4953658394
tx18_stopped: 14760
tx18_dropped: 0
tx18_xmit_more: 50837465
tx18_recover: 0
tx18_cqes: 5403332553
tx18_wake: 14760
tx18_cqe_err: 0
tx19_packets: 6408680611
tx19_bytes: 3644119934372
tx19_tso_packets: 350727530
tx19_tso_bytes: 2089896715365
tx19_tso_inner_packets: 0
tx19_tso_inner_bytes: 0
tx19_csum_partial: 486536490
tx19_csum_partial_inner: 0
tx19_added_vlan_packets: 5255839020
tx19_nop: 78525198
tx19_csum_none: 4769302530
tx19_stopped: 8614
tx19_dropped: 0
tx19_xmit_more: 43605232
tx19_recover: 0
tx19_cqes: 5212236833
tx19_wake: 8614
tx19_cqe_err: 0
tx20_packets: 5609275141
tx20_bytes: 3187279031581
tx20_tso_packets: 298609303
tx20_tso_bytes: 1794382229379
tx20_tso_inner_packets: 0
tx20_tso_inner_bytes: 0
tx20_csum_partial: 430691178
tx20_csum_partial_inner: 0
tx20_added_vlan_packets: 4616844286
tx20_nop: 67450040
tx20_csum_none: 4186153108
tx20_stopped: 9099
tx20_dropped: 0
tx20_xmit_more: 42040991
tx20_recover: 0
tx20_cqes: 4574805846
tx20_wake: 9099
tx20_cqe_err: 0
tx21_packets: 5641621183
tx21_bytes: 3279282331124
tx21_tso_packets: 311297057
tx21_tso_bytes: 1875735401012
tx21_tso_inner_packets: 0
tx21_tso_inner_bytes: 0
tx21_csum_partial: 444333894
tx21_csum_partial_inner: 0
tx21_added_vlan_packets: 4603527701
tx21_nop: 68857983
tx21_csum_none: 4159193807
tx21_stopped: 10082
tx21_dropped: 0
tx21_xmit_more: 43988081
tx21_recover: 0
tx21_cqes: 4559542410
tx21_wake: 10082
tx21_cqe_err: 0
tx22_packets: 5822168288
tx22_bytes: 3452026726862
tx22_tso_packets: 308230791
tx22_tso_bytes: 1859686450671
tx22_tso_inner_packets: 0
tx22_tso_inner_bytes: 0
tx22_csum_partial: 442751518
tx22_csum_partial_inner: 0
tx22_added_vlan_packets: 4792100335
tx22_nop: 70631706
tx22_csum_none: 4349348817
tx22_stopped: 9355
tx22_dropped: 0
tx22_xmit_more: 45165994
tx22_recover: 0
tx22_cqes: 4746936601
tx22_wake: 9355
tx22_cqe_err: 0
tx23_packets: 5664896066
tx23_bytes: 3207724186946
tx23_tso_packets: 300418757
tx23_tso_bytes: 1794180478679
tx23_tso_inner_packets: 0
tx23_tso_inner_bytes: 0
tx23_csum_partial: 429898848
tx23_csum_partial_inner: 0
tx23_added_vlan_packets: 4674317320
tx23_nop: 67899896
tx23_csum_none: 4244418472
tx23_stopped: 11684
tx23_dropped: 0
tx23_xmit_more: 43351132
tx23_recover: 0
tx23_cqes: 4630969028
tx23_wake: 11684
tx23_cqe_err: 0
tx24_packets: 5663326601
tx24_bytes: 3250127095110
tx24_tso_packets: 301327422
tx24_tso_bytes: 1831260534157
tx24_tso_inner_packets: 0
tx24_tso_inner_bytes: 0
tx24_csum_partial: 438757312
tx24_csum_partial_inner: 0
tx24_added_vlan_packets: 4646014986
tx24_nop: 68431153
tx24_csum_none: 4207257674
tx24_stopped: 9240
tx24_dropped: 0
tx24_xmit_more: 47699542
tx24_recover: 0
tx24_cqes: 4598317913
tx24_wake: 9240
tx24_cqe_err: 0
tx25_packets: 5703883962
tx25_bytes: 3291856915695
tx25_tso_packets: 308900318
tx25_tso_bytes: 1855516128386
tx25_tso_inner_packets: 0
tx25_tso_inner_bytes: 0
tx25_csum_partial: 444753744
tx25_csum_partial_inner: 0
tx25_added_vlan_packets: 4676528924
tx25_nop: 69230967
tx25_csum_none: 4231775180
tx25_stopped: 1140
tx25_dropped: 0
tx25_xmit_more: 40819195
tx25_recover: 0
tx25_cqes: 4635710966
tx25_wake: 1140
tx25_cqe_err: 0
tx26_packets: 5803495984
tx26_bytes: 3413564272139
tx26_tso_packets: 319986230
tx26_tso_bytes: 1929042839677
tx26_tso_inner_packets: 0
tx26_tso_inner_bytes: 0
tx26_csum_partial: 464771163
tx26_csum_partial_inner: 0
tx26_added_vlan_packets: 4734767280
tx26_nop: 71345080
tx26_csum_none: 4269996117
tx26_stopped: 10972
tx26_dropped: 0
tx26_xmit_more: 43793424
tx26_recover: 0
tx26_cqes: 4690976400
tx26_wake: 10972
tx26_cqe_err: 0
tx27_packets: 5960955343
tx27_bytes: 3444156164526
tx27_tso_packets: 325099639
tx27_tso_bytes: 1928378678784
tx27_tso_inner_packets: 0
tx27_tso_inner_bytes: 0
tx27_csum_partial: 467310289
tx27_csum_partial_inner: 0
tx27_added_vlan_packets: 4888651368
tx27_nop: 73201664
tx27_csum_none: 4421341079
tx27_stopped: 9465
tx27_dropped: 0
tx27_xmit_more: 53632121
tx27_recover: 0
tx27_cqes: 4835021398
tx27_wake: 9465
tx27_cqe_err: 0
tx28_packets: 0
tx28_bytes: 0
tx28_tso_packets: 0
tx28_tso_bytes: 0
tx28_tso_inner_packets: 0
tx28_tso_inner_bytes: 0
tx28_csum_partial: 0
tx28_csum_partial_inner: 0
tx28_added_vlan_packets: 0
tx28_nop: 0
tx28_csum_none: 0
tx28_stopped: 0
tx28_dropped: 0
tx28_xmit_more: 0
tx28_recover: 0
tx28_cqes: 0
tx28_wake: 0
tx28_cqe_err: 0
tx29_packets: 3
tx29_bytes: 266
tx29_tso_packets: 0
tx29_tso_bytes: 0
tx29_tso_inner_packets: 0
tx29_tso_inner_bytes: 0
tx29_csum_partial: 0
tx29_csum_partial_inner: 0
tx29_added_vlan_packets: 0
tx29_nop: 0
tx29_csum_none: 3
tx29_stopped: 0
tx29_dropped: 0
tx29_xmit_more: 1
tx29_recover: 0
tx29_cqes: 2
tx29_wake: 0
tx29_cqe_err: 0
tx30_packets: 0
tx30_bytes: 0
tx30_tso_packets: 0
tx30_tso_bytes: 0
tx30_tso_inner_packets: 0
tx30_tso_inner_bytes: 0
tx30_csum_partial: 0
tx30_csum_partial_inner: 0
tx30_added_vlan_packets: 0
tx30_nop: 0
tx30_csum_none: 0
tx30_stopped: 0
tx30_dropped: 0
tx30_xmit_more: 0
tx30_recover: 0
tx30_cqes: 0
tx30_wake: 0
tx30_cqe_err: 0
tx31_packets: 0
tx31_bytes: 0
tx31_tso_packets: 0
tx31_tso_bytes: 0
tx31_tso_inner_packets: 0
tx31_tso_inner_bytes: 0
tx31_csum_partial: 0
tx31_csum_partial_inner: 0
tx31_added_vlan_packets: 0
tx31_nop: 0
tx31_csum_none: 0
tx31_stopped: 0
tx31_dropped: 0
tx31_xmit_more: 0
tx31_recover: 0
tx31_cqes: 0
tx31_wake: 0
tx31_cqe_err: 0
tx32_packets: 0
tx32_bytes: 0
tx32_tso_packets: 0
tx32_tso_bytes: 0
tx32_tso_inner_packets: 0
tx32_tso_inner_bytes: 0
tx32_csum_partial: 0
tx32_csum_partial_inner: 0
tx32_added_vlan_packets: 0
tx32_nop: 0
tx32_csum_none: 0
tx32_stopped: 0
tx32_dropped: 0
tx32_xmit_more: 0
tx32_recover: 0
tx32_cqes: 0
tx32_wake: 0
tx32_cqe_err: 0
tx33_packets: 0
tx33_bytes: 0
tx33_tso_packets: 0
tx33_tso_bytes: 0
tx33_tso_inner_packets: 0
tx33_tso_inner_bytes: 0
tx33_csum_partial: 0
tx33_csum_partial_inner: 0
tx33_added_vlan_packets: 0
tx33_nop: 0
tx33_csum_none: 0
tx33_stopped: 0
tx33_dropped: 0
tx33_xmit_more: 0
tx33_recover: 0
tx33_cqes: 0
tx33_wake: 0
tx33_cqe_err: 0
tx34_packets: 0
tx34_bytes: 0
tx34_tso_packets: 0
tx34_tso_bytes: 0
tx34_tso_inner_packets: 0
tx34_tso_inner_bytes: 0
tx34_csum_partial: 0
tx34_csum_partial_inner: 0
tx34_added_vlan_packets: 0
tx34_nop: 0
tx34_csum_none: 0
tx34_stopped: 0
tx34_dropped: 0
tx34_xmit_more: 0
tx34_recover: 0
tx34_cqes: 0
tx34_wake: 0
tx34_cqe_err: 0
tx35_packets: 0
tx35_bytes: 0
tx35_tso_packets: 0
tx35_tso_bytes: 0
tx35_tso_inner_packets: 0
tx35_tso_inner_bytes: 0
tx35_csum_partial: 0
tx35_csum_partial_inner: 0
tx35_added_vlan_packets: 0
tx35_nop: 0
tx35_csum_none: 0
tx35_stopped: 0
tx35_dropped: 0
tx35_xmit_more: 0
tx35_recover: 0
tx35_cqes: 0
tx35_wake: 0
tx35_cqe_err: 0
tx36_packets: 0
tx36_bytes: 0
tx36_tso_packets: 0
tx36_tso_bytes: 0
tx36_tso_inner_packets: 0
tx36_tso_inner_bytes: 0
tx36_csum_partial: 0
tx36_csum_partial_inner: 0
tx36_added_vlan_packets: 0
tx36_nop: 0
tx36_csum_none: 0
tx36_stopped: 0
tx36_dropped: 0
tx36_xmit_more: 0
tx36_recover: 0
tx36_cqes: 0
tx36_wake: 0
tx36_cqe_err: 0
tx37_packets: 0
tx37_bytes: 0
tx37_tso_packets: 0
tx37_tso_bytes: 0
tx37_tso_inner_packets: 0
tx37_tso_inner_bytes: 0
tx37_csum_partial: 0
tx37_csum_partial_inner: 0
tx37_added_vlan_packets: 0
tx37_nop: 0
tx37_csum_none: 0
tx37_stopped: 0
tx37_dropped: 0
tx37_xmit_more: 0
tx37_recover: 0
tx37_cqes: 0
tx37_wake: 0
tx37_cqe_err: 0
tx38_packets: 0
tx38_bytes: 0
tx38_tso_packets: 0
tx38_tso_bytes: 0
tx38_tso_inner_packets: 0
tx38_tso_inner_bytes: 0
tx38_csum_partial: 0
tx38_csum_partial_inner: 0
tx38_added_vlan_packets: 0
tx38_nop: 0
tx38_csum_none: 0
tx38_stopped: 0
tx38_dropped: 0
tx38_xmit_more: 0
tx38_recover: 0
tx38_cqes: 0
tx38_wake: 0
tx38_cqe_err: 0
tx39_packets: 0
tx39_bytes: 0
tx39_tso_packets: 0
tx39_tso_bytes: 0
tx39_tso_inner_packets: 0
tx39_tso_inner_bytes: 0
tx39_csum_partial: 0
tx39_csum_partial_inner: 0
tx39_added_vlan_packets: 0
tx39_nop: 0
tx39_csum_none: 0
tx39_stopped: 0
tx39_dropped: 0
tx39_xmit_more: 0
tx39_recover: 0
tx39_cqes: 0
tx39_wake: 0
tx39_cqe_err: 0
tx40_packets: 0
tx40_bytes: 0
tx40_tso_packets: 0
tx40_tso_bytes: 0
tx40_tso_inner_packets: 0
tx40_tso_inner_bytes: 0
tx40_csum_partial: 0
tx40_csum_partial_inner: 0
tx40_added_vlan_packets: 0
tx40_nop: 0
tx40_csum_none: 0
tx40_stopped: 0
tx40_dropped: 0
tx40_xmit_more: 0
tx40_recover: 0
tx40_cqes: 0
tx40_wake: 0
tx40_cqe_err: 0
tx41_packets: 0
tx41_bytes: 0
tx41_tso_packets: 0
tx41_tso_bytes: 0
tx41_tso_inner_packets: 0
tx41_tso_inner_bytes: 0
tx41_csum_partial: 0
tx41_csum_partial_inner: 0
tx41_added_vlan_packets: 0
tx41_nop: 0
tx41_csum_none: 0
tx41_stopped: 0
tx41_dropped: 0
tx41_xmit_more: 0
tx41_recover: 0
tx41_cqes: 0
tx41_wake: 0
tx41_cqe_err: 0
tx42_packets: 0
tx42_bytes: 0
tx42_tso_packets: 0
tx42_tso_bytes: 0
tx42_tso_inner_packets: 0
tx42_tso_inner_bytes: 0
tx42_csum_partial: 0
tx42_csum_partial_inner: 0
tx42_added_vlan_packets: 0
tx42_nop: 0
tx42_csum_none: 0
tx42_stopped: 0
tx42_dropped: 0
tx42_xmit_more: 0
tx42_recover: 0
tx42_cqes: 0
tx42_wake: 0
tx42_cqe_err: 0
tx43_packets: 0
tx43_bytes: 0
tx43_tso_packets: 0
tx43_tso_bytes: 0
tx43_tso_inner_packets: 0
tx43_tso_inner_bytes: 0
tx43_csum_partial: 0
tx43_csum_partial_inner: 0
tx43_added_vlan_packets: 0
tx43_nop: 0
tx43_csum_none: 0
tx43_stopped: 0
tx43_dropped: 0
tx43_xmit_more: 0
tx43_recover: 0
tx43_cqes: 0
tx43_wake: 0
tx43_cqe_err: 0
tx44_packets: 0
tx44_bytes: 0
tx44_tso_packets: 0
tx44_tso_bytes: 0
tx44_tso_inner_packets: 0
tx44_tso_inner_bytes: 0
tx44_csum_partial: 0
tx44_csum_partial_inner: 0
tx44_added_vlan_packets: 0
tx44_nop: 0
tx44_csum_none: 0
tx44_stopped: 0
tx44_dropped: 0
tx44_xmit_more: 0
tx44_recover: 0
tx44_cqes: 0
tx44_wake: 0
tx44_cqe_err: 0
tx45_packets: 0
tx45_bytes: 0
tx45_tso_packets: 0
tx45_tso_bytes: 0
tx45_tso_inner_packets: 0
tx45_tso_inner_bytes: 0
tx45_csum_partial: 0
tx45_csum_partial_inner: 0
tx45_added_vlan_packets: 0
tx45_nop: 0
tx45_csum_none: 0
tx45_stopped: 0
tx45_dropped: 0
tx45_xmit_more: 0
tx45_recover: 0
tx45_cqes: 0
tx45_wake: 0
tx45_cqe_err: 0
tx46_packets: 0
tx46_bytes: 0
tx46_tso_packets: 0
tx46_tso_bytes: 0
tx46_tso_inner_packets: 0
tx46_tso_inner_bytes: 0
tx46_csum_partial: 0
tx46_csum_partial_inner: 0
tx46_added_vlan_packets: 0
tx46_nop: 0
tx46_csum_none: 0
tx46_stopped: 0
tx46_dropped: 0
tx46_xmit_more: 0
tx46_recover: 0
tx46_cqes: 0
tx46_wake: 0
tx46_cqe_err: 0
tx47_packets: 0
tx47_bytes: 0
tx47_tso_packets: 0
tx47_tso_bytes: 0
tx47_tso_inner_packets: 0
tx47_tso_inner_bytes: 0
tx47_csum_partial: 0
tx47_csum_partial_inner: 0
tx47_added_vlan_packets: 0
tx47_nop: 0
tx47_csum_none: 0
tx47_stopped: 0
tx47_dropped: 0
tx47_xmit_more: 0
tx47_recover: 0
tx47_cqes: 0
tx47_wake: 0
tx47_cqe_err: 0
tx48_packets: 0
tx48_bytes: 0
tx48_tso_packets: 0
tx48_tso_bytes: 0
tx48_tso_inner_packets: 0
tx48_tso_inner_bytes: 0
tx48_csum_partial: 0
tx48_csum_partial_inner: 0
tx48_added_vlan_packets: 0
tx48_nop: 0
tx48_csum_none: 0
tx48_stopped: 0
tx48_dropped: 0
tx48_xmit_more: 0
tx48_recover: 0
tx48_cqes: 0
tx48_wake: 0
tx48_cqe_err: 0
tx49_packets: 0
tx49_bytes: 0
tx49_tso_packets: 0
tx49_tso_bytes: 0
tx49_tso_inner_packets: 0
tx49_tso_inner_bytes: 0
tx49_csum_partial: 0
tx49_csum_partial_inner: 0
tx49_added_vlan_packets: 0
tx49_nop: 0
tx49_csum_none: 0
tx49_stopped: 0
tx49_dropped: 0
tx49_xmit_more: 0
tx49_recover: 0
tx49_cqes: 0
tx49_wake: 0
tx49_cqe_err: 0
tx50_packets: 0
tx50_bytes: 0
tx50_tso_packets: 0
tx50_tso_bytes: 0
tx50_tso_inner_packets: 0
tx50_tso_inner_bytes: 0
tx50_csum_partial: 0
tx50_csum_partial_inner: 0
tx50_added_vlan_packets: 0
tx50_nop: 0
tx50_csum_none: 0
tx50_stopped: 0
tx50_dropped: 0
tx50_xmit_more: 0
tx50_recover: 0
tx50_cqes: 0
tx50_wake: 0
tx50_cqe_err: 0
tx51_packets: 0
tx51_bytes: 0
tx51_tso_packets: 0
tx51_tso_bytes: 0
tx51_tso_inner_packets: 0
tx51_tso_inner_bytes: 0
tx51_csum_partial: 0
tx51_csum_partial_inner: 0
tx51_added_vlan_packets: 0
tx51_nop: 0
tx51_csum_none: 0
tx51_stopped: 0
tx51_dropped: 0
tx51_xmit_more: 0
tx51_recover: 0
tx51_cqes: 0
tx51_wake: 0
tx51_cqe_err: 0
tx52_packets: 0
tx52_bytes: 0
tx52_tso_packets: 0
tx52_tso_bytes: 0
tx52_tso_inner_packets: 0
tx52_tso_inner_bytes: 0
tx52_csum_partial: 0
tx52_csum_partial_inner: 0
tx52_added_vlan_packets: 0
tx52_nop: 0
tx52_csum_none: 0
tx52_stopped: 0
tx52_dropped: 0
tx52_xmit_more: 0
tx52_recover: 0
tx52_cqes: 0
tx52_wake: 0
tx52_cqe_err: 0
tx53_packets: 0
tx53_bytes: 0
tx53_tso_packets: 0
tx53_tso_bytes: 0
tx53_tso_inner_packets: 0
tx53_tso_inner_bytes: 0
tx53_csum_partial: 0
tx53_csum_partial_inner: 0
tx53_added_vlan_packets: 0
tx53_nop: 0
tx53_csum_none: 0
tx53_stopped: 0
tx53_dropped: 0
tx53_xmit_more: 0
tx53_recover: 0
tx53_cqes: 0
tx53_wake: 0
tx53_cqe_err: 0
tx54_packets: 0
tx54_bytes: 0
tx54_tso_packets: 0
tx54_tso_bytes: 0
tx54_tso_inner_packets: 0
tx54_tso_inner_bytes: 0
tx54_csum_partial: 0
tx54_csum_partial_inner: 0
tx54_added_vlan_packets: 0
tx54_nop: 0
tx54_csum_none: 0
tx54_stopped: 0
tx54_dropped: 0
tx54_xmit_more: 0
tx54_recover: 0
tx54_cqes: 0
tx54_wake: 0
tx54_cqe_err: 0
tx55_packets: 0
tx55_bytes: 0
tx55_tso_packets: 0
tx55_tso_bytes: 0
tx55_tso_inner_packets: 0
tx55_tso_inner_bytes: 0
tx55_csum_partial: 0
tx55_csum_partial_inner: 0
tx55_added_vlan_packets: 0
tx55_nop: 0
tx55_csum_none: 0
tx55_stopped: 0
tx55_dropped: 0
tx55_xmit_more: 0
tx55_recover: 0
tx55_cqes: 0
tx55_wake: 0
tx55_cqe_err: 0
tx0_xdp_xmit: 0
tx0_xdp_full: 0
tx0_xdp_err: 0
tx0_xdp_cqes: 0
tx1_xdp_xmit: 0
tx1_xdp_full: 0
tx1_xdp_err: 0
tx1_xdp_cqes: 0
tx2_xdp_xmit: 0
tx2_xdp_full: 0
tx2_xdp_err: 0
tx2_xdp_cqes: 0
tx3_xdp_xmit: 0
tx3_xdp_full: 0
tx3_xdp_err: 0
tx3_xdp_cqes: 0
tx4_xdp_xmit: 0
tx4_xdp_full: 0
tx4_xdp_err: 0
tx4_xdp_cqes: 0
tx5_xdp_xmit: 0
tx5_xdp_full: 0
tx5_xdp_err: 0
tx5_xdp_cqes: 0
tx6_xdp_xmit: 0
tx6_xdp_full: 0
tx6_xdp_err: 0
tx6_xdp_cqes: 0
tx7_xdp_xmit: 0
tx7_xdp_full: 0
tx7_xdp_err: 0
tx7_xdp_cqes: 0
tx8_xdp_xmit: 0
tx8_xdp_full: 0
tx8_xdp_err: 0
tx8_xdp_cqes: 0
tx9_xdp_xmit: 0
tx9_xdp_full: 0
tx9_xdp_err: 0
tx9_xdp_cqes: 0
tx10_xdp_xmit: 0
tx10_xdp_full: 0
tx10_xdp_err: 0
tx10_xdp_cqes: 0
tx11_xdp_xmit: 0
tx11_xdp_full: 0
tx11_xdp_err: 0
tx11_xdp_cqes: 0
tx12_xdp_xmit: 0
tx12_xdp_full: 0
tx12_xdp_err: 0
tx12_xdp_cqes: 0
tx13_xdp_xmit: 0
tx13_xdp_full: 0
tx13_xdp_err: 0
tx13_xdp_cqes: 0
tx14_xdp_xmit: 0
tx14_xdp_full: 0
tx14_xdp_err: 0
tx14_xdp_cqes: 0
tx15_xdp_xmit: 0
tx15_xdp_full: 0
tx15_xdp_err: 0
tx15_xdp_cqes: 0
tx16_xdp_xmit: 0
tx16_xdp_full: 0
tx16_xdp_err: 0
tx16_xdp_cqes: 0
tx17_xdp_xmit: 0
tx17_xdp_full: 0
tx17_xdp_err: 0
tx17_xdp_cqes: 0
tx18_xdp_xmit: 0
tx18_xdp_full: 0
tx18_xdp_err: 0
tx18_xdp_cqes: 0
tx19_xdp_xmit: 0
tx19_xdp_full: 0
tx19_xdp_err: 0
tx19_xdp_cqes: 0
tx20_xdp_xmit: 0
tx20_xdp_full: 0
tx20_xdp_err: 0
tx20_xdp_cqes: 0
tx21_xdp_xmit: 0
tx21_xdp_full: 0
tx21_xdp_err: 0
tx21_xdp_cqes: 0
tx22_xdp_xmit: 0
tx22_xdp_full: 0
tx22_xdp_err: 0
tx22_xdp_cqes: 0
tx23_xdp_xmit: 0
tx23_xdp_full: 0
tx23_xdp_err: 0
tx23_xdp_cqes: 0
tx24_xdp_xmit: 0
tx24_xdp_full: 0
tx24_xdp_err: 0
tx24_xdp_cqes: 0
tx25_xdp_xmit: 0
tx25_xdp_full: 0
tx25_xdp_err: 0
tx25_xdp_cqes: 0
tx26_xdp_xmit: 0
tx26_xdp_full: 0
tx26_xdp_err: 0
tx26_xdp_cqes: 0
tx27_xdp_xmit: 0
tx27_xdp_full: 0
tx27_xdp_err: 0
tx27_xdp_cqes: 0
tx28_xdp_xmit: 0
tx28_xdp_full: 0
tx28_xdp_err: 0
tx28_xdp_cqes: 0
tx29_xdp_xmit: 0
tx29_xdp_full: 0
tx29_xdp_err: 0
tx29_xdp_cqes: 0
tx30_xdp_xmit: 0
tx30_xdp_full: 0
tx30_xdp_err: 0
tx30_xdp_cqes: 0
tx31_xdp_xmit: 0
tx31_xdp_full: 0
tx31_xdp_err: 0
tx31_xdp_cqes: 0
tx32_xdp_xmit: 0
tx32_xdp_full: 0
tx32_xdp_err: 0
tx32_xdp_cqes: 0
tx33_xdp_xmit: 0
tx33_xdp_full: 0
tx33_xdp_err: 0
tx33_xdp_cqes: 0
tx34_xdp_xmit: 0
tx34_xdp_full: 0
tx34_xdp_err: 0
tx34_xdp_cqes: 0
tx35_xdp_xmit: 0
tx35_xdp_full: 0
tx35_xdp_err: 0
tx35_xdp_cqes: 0
tx36_xdp_xmit: 0
tx36_xdp_full: 0
tx36_xdp_err: 0
tx36_xdp_cqes: 0
tx37_xdp_xmit: 0
tx37_xdp_full: 0
tx37_xdp_err: 0
tx37_xdp_cqes: 0
tx38_xdp_xmit: 0
tx38_xdp_full: 0
tx38_xdp_err: 0
tx38_xdp_cqes: 0
tx39_xdp_xmit: 0
tx39_xdp_full: 0
tx39_xdp_err: 0
tx39_xdp_cqes: 0
tx40_xdp_xmit: 0
tx40_xdp_full: 0
tx40_xdp_err: 0
tx40_xdp_cqes: 0
tx41_xdp_xmit: 0
tx41_xdp_full: 0
tx41_xdp_err: 0
tx41_xdp_cqes: 0
tx42_xdp_xmit: 0
tx42_xdp_full: 0
tx42_xdp_err: 0
tx42_xdp_cqes: 0
tx43_xdp_xmit: 0
tx43_xdp_full: 0
tx43_xdp_err: 0
tx43_xdp_cqes: 0
tx44_xdp_xmit: 0
tx44_xdp_full: 0
tx44_xdp_err: 0
tx44_xdp_cqes: 0
tx45_xdp_xmit: 0
tx45_xdp_full: 0
tx45_xdp_err: 0
tx45_xdp_cqes: 0
tx46_xdp_xmit: 0
tx46_xdp_full: 0
tx46_xdp_err: 0
tx46_xdp_cqes: 0
tx47_xdp_xmit: 0
tx47_xdp_full: 0
tx47_xdp_err: 0
tx47_xdp_cqes: 0
tx48_xdp_xmit: 0
tx48_xdp_full: 0
tx48_xdp_err: 0
tx48_xdp_cqes: 0
tx49_xdp_xmit: 0
tx49_xdp_full: 0
tx49_xdp_err: 0
tx49_xdp_cqes: 0
tx50_xdp_xmit: 0
tx50_xdp_full: 0
tx50_xdp_err: 0
tx50_xdp_cqes: 0
tx51_xdp_xmit: 0
tx51_xdp_full: 0
tx51_xdp_err: 0
tx51_xdp_cqes: 0
tx52_xdp_xmit: 0
tx52_xdp_full: 0
tx52_xdp_err: 0
tx52_xdp_cqes: 0
tx53_xdp_xmit: 0
tx53_xdp_full: 0
tx53_xdp_err: 0
tx53_xdp_cqes: 0
tx54_xdp_xmit: 0
tx54_xdp_full: 0
tx54_xdp_err: 0
tx54_xdp_cqes: 0
tx55_xdp_xmit: 0
tx55_xdp_full: 0
tx55_xdp_err: 0
tx55_xdp_cqes: 0
mpstat -P ALL 1 10
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.04 0.00 6.94 0.02 0.00 32.00
0.00 0.00 0.00 61.00
Average: 0 0.00 0.00 1.20 0.00 0.00 0.00 0.00
0.00 0.00 98.80
Average: 1 0.00 0.00 2.30 0.00 0.00 0.00 0.00
0.00 0.00 97.70
Average: 2 0.10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 3 0.10 0.00 1.50 0.00 0.00 0.00 0.00
0.00 0.00 98.40
Average: 4 0.50 0.00 2.50 0.00 0.00 0.00 0.00
0.00 0.00 97.00
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 6 0.90 0.00 10.20 0.00 0.00 0.00 0.00
0.00 0.00 88.90
Average: 7 0.00 0.00 0.00 1.40 0.00 0.00 0.00
0.00 0.00 98.60
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 14 0.00 0.00 12.99 0.00 0.00 62.64
0.00 0.00 0.00 24.38
Average: 15 0.00 0.00 12.70 0.00 0.00 63.40
0.00 0.00 0.00 23.90
Average: 16 0.00 0.00 11.20 0.00 0.00 66.40
0.00 0.00 0.00 22.40
Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
0.00 0.00 0.00 31.30
Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
0.00 0.00 0.00 24.90
Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
0.00 0.00 0.00 19.68
Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
0.00 0.00 0.00 18.00
Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
0.00 0.00 0.00 17.40
Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
0.00 0.00 0.00 26.03
Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
0.00 0.00 0.00 17.52
Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
0.00 0.00 0.00 18.20
Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
0.00 0.00 0.00 17.68
Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
0.00 0.00 0.00 18.20
Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
0.00 0.00 0.00 20.42
Average: 28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 3.90 0.00 0.00 0.00 0.00
0.00 0.00 96.10
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 36 0.10 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.70
Average: 37 0.20 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.50
Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 39 0.00 0.00 2.60 0.00 0.00 0.00 0.00
0.00 0.00 97.40
Average: 40 0.00 0.00 0.90 0.00 0.00 0.00 0.00
0.00 0.00 99.10
Average: 41 0.10 0.00 0.50 0.00 0.00 0.00 0.00
0.00 0.00 99.40
Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
0.00 0.00 0.00 19.42
Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
0.00 0.00 0.00 26.60
Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
0.00 0.00 0.00 21.60
Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
0.00 0.00 0.00 20.50
Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
0.00 0.00 0.00 21.60
Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
0.00 0.00 0.00 24.58
Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
0.00 0.00 0.00 24.68
Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
0.00 0.00 0.00 28.34
Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
0.00 0.00 0.00 25.83
Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
0.00 0.00 0.00 27.20
Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
0.00 0.00 0.00 28.03
Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
0.00 0.00 0.00 24.70
Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
0.00 0.00 0.00 25.30
Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
0.00 0.00 0.00 26.77
ethtool -k enp175s0f0
Features for enp175s0f0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
ethtool -c enp175s0f0
Coalesce parameters for enp175s0f0:
Adaptive RX: off TX: on
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32703
rx-usecs: 256
rx-frames: 128
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 8
tx-frames: 128
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
ethtool -g enp175s0f0
Ring parameters for enp175s0f0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-10-31 21:57 Kernel 4.19 network performance - forwarding/routing normal users traffic Paweł Staszewski
@ 2018-10-31 22:09 ` Eric Dumazet
2018-10-31 22:20 ` Paweł Staszewski
2018-11-01 3:37 ` David Ahern
2018-11-01 9:50 ` Saeed Mahameed
2 siblings, 1 reply; 77+ messages in thread
From: Eric Dumazet @ 2018-10-31 22:09 UTC (permalink / raw)
To: Paweł Staszewski, netdev
On 10/31/2018 02:57 PM, Paweł Staszewski wrote:
> Hi
>
> So maybee someone will be interested how linux kernel handles normal traffic (not pktgen :) )
>
>
> Server HW configuration:
>
> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>
> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>
>
> Server software:
>
> FRR - as routing daemon
>
> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa node)
>
> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
>
>
> Maximum traffic that server can handle:
>
> Bandwidth
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> \ iface Rx Tx Total
> ==============================================================================
> enp175s0f1: 28.51 Gb/s 37.24 Gb/s 65.74 Gb/s
> enp175s0f0: 38.07 Gb/s 28.44 Gb/s 66.51 Gb/s
> ------------------------------------------------------------------------------
> total: 66.58 Gb/s 65.67 Gb/s 132.25 Gb/s
>
>
> Packets per second:
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> - iface Rx Tx Total
> ==============================================================================
> enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
> enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
> ------------------------------------------------------------------------------
> total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
>
>
> After reaching that limits nics on the upstream side (more RX traffic) start to drop packets
>
>
> I just dont understand that server can't handle more bandwidth (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX side are increasing.
>
> Was thinking that maybee reached some pcie x16 limit - but x16 8GT is 126Gbit - and also when testing with pktgen i can reach more bw and pps (like 4x more comparing to normal internet traffic)
>
> And wondering if there is something that can be improved here.
>
>
>
> Some more informations / counters / stats and perf top below:
>
> Perf top flame graph:
>
> https://uploadfiles.io/7zo6u
>
>
>
> System configuration(long):
>
>
> cat /sys/devices/system/node/node1/cpulist
> 14-27,42-55
> cat /sys/class/net/enp175s0f0/device/numa_node
> 1
> cat /sys/class/net/enp175s0f1/device/numa_node
> 1
>
>
>
>
>
> ip -s -d link ls dev enp175s0f0
> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
> RX: bytes packets errors dropped overrun mcast
> 184142375840858 141347715974 2 2806325 0 85050528
> TX: bytes packets errors dropped carrier collsns
> 99270697277430 172227994003 0 0 0 0
>
> ip -s -d link ls dev enp175s0f1
> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
> RX: bytes packets errors dropped overrun mcast
> 99686284170801 173507590134 61 669685 0 100304421
> TX: bytes packets errors dropped carrier collsns
> 184435107970545 142383178304 0 0 0 0
>
>
> ./softnet.sh
> cpu total dropped squeezed collision rps flow_limit
>
>
>
>
> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> 26.78% [kernel] [k] queued_spin_lock_slowpath
This is highly suspect.
A call graph (perf record -a -g sleep 1; perf report --stdio) would tell what is going on.
With that many TX/RX queues, I would expect you to not use RPS/RFS, and have a 1/1 RX/TX mapping,
so I do not know what could request a spinlock contention.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-10-31 22:09 ` Eric Dumazet
@ 2018-10-31 22:20 ` Paweł Staszewski
2018-10-31 22:45 ` Paweł Staszewski
2018-11-01 9:22 ` Jesper Dangaard Brouer
0 siblings, 2 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-10-31 22:20 UTC (permalink / raw)
To: Eric Dumazet, netdev
W dniu 31.10.2018 o 23:09, Eric Dumazet pisze:
>
> On 10/31/2018 02:57 PM, Paweł Staszewski wrote:
>> Hi
>>
>> So maybee someone will be interested how linux kernel handles normal traffic (not pktgen :) )
>>
>>
>> Server HW configuration:
>>
>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>
>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>
>>
>> Server software:
>>
>> FRR - as routing daemon
>>
>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa node)
>>
>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
>>
>>
>> Maximum traffic that server can handle:
>>
>> Bandwidth
>>
>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>> input: /proc/net/dev type: rate
>> \ iface Rx Tx Total
>> ==============================================================================
>> enp175s0f1: 28.51 Gb/s 37.24 Gb/s 65.74 Gb/s
>> enp175s0f0: 38.07 Gb/s 28.44 Gb/s 66.51 Gb/s
>> ------------------------------------------------------------------------------
>> total: 66.58 Gb/s 65.67 Gb/s 132.25 Gb/s
>>
>>
>> Packets per second:
>>
>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>> input: /proc/net/dev type: rate
>> - iface Rx Tx Total
>> ==============================================================================
>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
>> ------------------------------------------------------------------------------
>> total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
>>
>>
>> After reaching that limits nics on the upstream side (more RX traffic) start to drop packets
>>
>>
>> I just dont understand that server can't handle more bandwidth (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX side are increasing.
>>
>> Was thinking that maybee reached some pcie x16 limit - but x16 8GT is 126Gbit - and also when testing with pktgen i can reach more bw and pps (like 4x more comparing to normal internet traffic)
>>
>> And wondering if there is something that can be improved here.
>>
>>
>>
>> Some more informations / counters / stats and perf top below:
>>
>> Perf top flame graph:
>>
>> https://uploadfiles.io/7zo6u
>>
>>
>>
>> System configuration(long):
>>
>>
>> cat /sys/devices/system/node/node1/cpulist
>> 14-27,42-55
>> cat /sys/class/net/enp175s0f0/device/numa_node
>> 1
>> cat /sys/class/net/enp175s0f1/device/numa_node
>> 1
>>
>>
>>
>>
>>
>> ip -s -d link ls dev enp175s0f0
>> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
>> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
>> RX: bytes packets errors dropped overrun mcast
>> 184142375840858 141347715974 2 2806325 0 85050528
>> TX: bytes packets errors dropped carrier collsns
>> 99270697277430 172227994003 0 0 0 0
>>
>> ip -s -d link ls dev enp175s0f1
>> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
>> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
>> RX: bytes packets errors dropped overrun mcast
>> 99686284170801 173507590134 61 669685 0 100304421
>> TX: bytes packets errors dropped carrier collsns
>> 184435107970545 142383178304 0 0 0 0
>>
>>
>> ./softnet.sh
>> cpu total dropped squeezed collision rps flow_limit
>>
>>
>>
>>
>> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> 26.78% [kernel] [k] queued_spin_lock_slowpath
> This is highly suspect.
>
> A call graph (perf record -a -g sleep 1; perf report --stdio) would tell what is going on.
perf report:
https://ufile.io/rqp0h
>
> With that many TX/RX queues, I would expect you to not use RPS/RFS, and have a 1/1 RX/TX mapping,
> so I do not know what could request a spinlock contention.
>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-10-31 22:20 ` Paweł Staszewski
@ 2018-10-31 22:45 ` Paweł Staszewski
2018-11-01 9:22 ` Jesper Dangaard Brouer
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-10-31 22:45 UTC (permalink / raw)
To: Eric Dumazet, netdev
W dniu 31.10.2018 o 23:20, Paweł Staszewski pisze:
>
>
> W dniu 31.10.2018 o 23:09, Eric Dumazet pisze:
>>
>> On 10/31/2018 02:57 PM, Paweł Staszewski wrote:
>>> Hi
>>>
>>> So maybee someone will be interested how linux kernel handles normal
>>> traffic (not pktgen :) )
>>>
>>>
>>> Server HW configuration:
>>>
>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>
>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>
>>>
>>> Server software:
>>>
>>> FRR - as routing daemon
>>>
>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local
>>> numa node)
>>>
>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>> numa node)
>>>
>>>
>>> Maximum traffic that server can handle:
>>>
>>> Bandwidth
>>>
>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>> input: /proc/net/dev type: rate
>>> \ iface Rx Tx Total
>>> ==============================================================================
>>>
>>> enp175s0f1: 28.51 Gb/s 37.24
>>> Gb/s 65.74 Gb/s
>>> enp175s0f0: 38.07 Gb/s 28.44
>>> Gb/s 66.51 Gb/s
>>> ------------------------------------------------------------------------------
>>>
>>> total: 66.58 Gb/s 65.67
>>> Gb/s 132.25 Gb/s
>>>
>>>
>>> Packets per second:
>>>
>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>> input: /proc/net/dev type: rate
>>> - iface Rx Tx Total
>>> ==============================================================================
>>>
>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>> 8735207.00 P/s
>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>> 8790460.00 P/s
>>> ------------------------------------------------------------------------------
>>>
>>> total: 8806533.00 P/s 8719134.00 P/s
>>> 17525668.00 P/s
>>>
>>>
>>> After reaching that limits nics on the upstream side (more RX
>>> traffic) start to drop packets
>>>
>>>
>>> I just dont understand that server can't handle more bandwidth
>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
>>> side are increasing.
>>>
>>> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
>>> is 126Gbit - and also when testing with pktgen i can reach more bw
>>> and pps (like 4x more comparing to normal internet traffic)
>>>
>>> And wondering if there is something that can be improved here.
>>>
>>>
>>>
>>> Some more informations / counters / stats and perf top below:
>>>
>>> Perf top flame graph:
>>>
>>> https://uploadfiles.io/7zo6u
>>>
>>>
>>>
>>> System configuration(long):
>>>
>>>
>>> cat /sys/devices/system/node/node1/cpulist
>>> 14-27,42-55
>>> cat /sys/class/net/enp175s0f0/device/numa_node
>>> 1
>>> cat /sys/class/net/enp175s0f1/device/numa_node
>>> 1
>>>
>>>
>>>
>>>
>>>
>>> ip -s -d link ls dev enp175s0f0
>>> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>>> state UP mode DEFAULT group default qlen 8192
>>> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity
>>> 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size
>>> 65536 gso_max_segs 65535
>>> RX: bytes packets errors dropped overrun mcast
>>> 184142375840858 141347715974 2 2806325 0 85050528
>>> TX: bytes packets errors dropped carrier collsns
>>> 99270697277430 172227994003 0 0 0 0
>>>
>>> ip -s -d link ls dev enp175s0f1
>>> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>>> state UP mode DEFAULT group default qlen 8192
>>> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity
>>> 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size
>>> 65536 gso_max_segs 65535
>>> RX: bytes packets errors dropped overrun mcast
>>> 99686284170801 173507590134 61 669685 0 100304421
>>> TX: bytes packets errors dropped carrier collsns
>>> 184435107970545 142383178304 0 0 0 0
>>>
>>>
>>> ./softnet.sh
>>> cpu total dropped squeezed collision rps flow_limit
>>>
>>>
>>>
>>>
>>> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> 26.78% [kernel] [k] queued_spin_lock_slowpath
>> This is highly suspect.
>>
>> A call graph (perf record -a -g sleep 1; perf report --stdio) would
>> tell what is going on.
> perf report:
> https://ufile.io/rqp0h
>
>
>
>>
>> With that many TX/RX queues, I would expect you to not use RPS/RFS,
>> and have a 1/1 RX/TX mapping,
>> so I do not know what could request a spinlock contention.
>>
>>
>>
>
>
And yes there is no RPF/RFS - just 1/1 RX/TX and affinity mapping on
local cpu for the network controller for 28 RX+TX queues per nic .
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-10-31 21:57 Kernel 4.19 network performance - forwarding/routing normal users traffic Paweł Staszewski
2018-10-31 22:09 ` Eric Dumazet
@ 2018-11-01 3:37 ` David Ahern
2018-11-01 10:55 ` Jesper Dangaard Brouer
2018-11-01 9:50 ` Saeed Mahameed
2 siblings, 1 reply; 77+ messages in thread
From: David Ahern @ 2018-11-01 3:37 UTC (permalink / raw)
To: Paweł Staszewski, netdev
On 10/31/18 3:57 PM, Paweł Staszewski wrote:
> Hi
>
> So maybee someone will be interested how linux kernel handles normal
> traffic (not pktgen :) )
>
>
> Server HW configuration:
>
> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>
> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>
>
> Server software:
>
> FRR - as routing daemon
>
> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa
> node)
>
> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
>
>
> Maximum traffic that server can handle:
>
> Bandwidth
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> \ iface Rx Tx Total
> ==============================================================================
>
> enp175s0f1: 28.51 Gb/s 37.24 Gb/s
> 65.74 Gb/s
> enp175s0f0: 38.07 Gb/s 28.44 Gb/s
> 66.51 Gb/s
> ------------------------------------------------------------------------------
>
> total: 66.58 Gb/s 65.67 Gb/s
> 132.25 Gb/s
>
>
> Packets per second:
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> - iface Rx Tx Total
> ==============================================================================
>
> enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
> enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
> ------------------------------------------------------------------------------
>
> total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
>
>
> After reaching that limits nics on the upstream side (more RX traffic)
> start to drop packets
>
>
> I just dont understand that server can't handle more bandwidth
> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
> side are increasing.
>
> Was thinking that maybee reached some pcie x16 limit - but x16 8GT is
> 126Gbit - and also when testing with pktgen i can reach more bw and pps
> (like 4x more comparing to normal internet traffic)
>
> And wondering if there is something that can be improved here.
This is mainly a forwarding use case? Seems so based on the perf report.
I suspect forwarding with XDP would show pretty good improvement. You
need the vlan changes I have queued up though.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-10-31 22:20 ` Paweł Staszewski
2018-10-31 22:45 ` Paweł Staszewski
@ 2018-11-01 9:22 ` Jesper Dangaard Brouer
2018-11-01 10:34 ` Paweł Staszewski
2018-11-01 15:27 ` Aaron Lu
1 sibling, 2 replies; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-01 9:22 UTC (permalink / raw)
To: Paweł Staszewski
Cc: brouer, Eric Dumazet, netdev, Tariq Toukan, Ilias Apalodimas,
Yoel Caspersen, Mel Gorman, Aaron Lu
On Wed, 31 Oct 2018 23:20:01 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 31.10.2018 o 23:09, Eric Dumazet pisze:
> >
> > On 10/31/2018 02:57 PM, Paweł Staszewski wrote:
> >> Hi
> >>
> >> So maybee someone will be interested how linux kernel handles
> >> normal traffic (not pktgen :) )
Pawel is this live production traffic?
I know Yoel (Cc) is very interested to know the real-life limitation of
Linux as a router, especially with VLANs like you use.
> >>
> >> Server HW configuration:
> >>
> >> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
> >>
> >> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
> >>
> >>
> >> Server software:
> >>
> >> FRR - as routing daemon
> >>
> >> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa node)
> >>
> >> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
> >>
> >>
> >> Maximum traffic that server can handle:
> >>
> >> Bandwidth
> >>
> >> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> >> input: /proc/net/dev type: rate
> >> \ iface Rx Tx Total
> >> ==============================================================================
> >> enp175s0f1: 28.51 Gb/s 37.24 Gb/s 65.74 Gb/s
> >> enp175s0f0: 38.07 Gb/s 28.44 Gb/s 66.51 Gb/s
> >> ------------------------------------------------------------------------------
> >> total: 66.58 Gb/s 65.67 Gb/s 132.25 Gb/s
> >>
Actually rather impressive number for a Linux router.
> >>
> >> Packets per second:
> >>
> >> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> >> input: /proc/net/dev type: rate
> >> - iface Rx Tx Total
> >> ==============================================================================
> >> enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
> >> enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
> >> ------------------------------------------------------------------------------
> >> total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
> >>
Average packet size:
(28.51*10^9/8)/5248589 = 678.99 bytes
(38.07*10^9/8)/3557944 = 1337.49 bytes
> >> After reaching that limits nics on the upstream side (more RX
> >> traffic) start to drop packets
> >>
> >>
> >> I just dont understand that server can't handle more bandwidth
> >> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
> >> RX side are increasing.
> >>
> >> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
> >> is 126Gbit - and also when testing with pktgen i can reach more bw
> >> and pps (like 4x more comparing to normal internet traffic)
> >>
> >> And wondering if there is something that can be improved here.
> >>
> >>
> >>
> >> Some more informations / counters / stats and perf top below:
> >>
> >> Perf top flame graph:
> >>
> >> https://uploadfiles.io/7zo6u
Thanks a lot for the flame graph!
> >>
> >> System configuration(long):
> >>
> >>
> >> cat /sys/devices/system/node/node1/cpulist
> >> 14-27,42-55
> >> cat /sys/class/net/enp175s0f0/device/numa_node
> >> 1
> >> cat /sys/class/net/enp175s0f1/device/numa_node
> >> 1
> >>
Hint grep can give you nicer output that cat:
$ grep -H . /sys/class/net/*/device/numa_node
> >>
> >>
> >>
> >>
> >> ip -s -d link ls dev enp175s0f0
> >> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
> >> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
> >> RX: bytes packets errors dropped overrun mcast
> >> 184142375840858 141347715974 2 2806325 0 85050528
> >> TX: bytes packets errors dropped carrier collsns
> >> 99270697277430 172227994003 0 0 0 0
> >>
> >> ip -s -d link ls dev enp175s0f1
> >> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
> >> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
> >> RX: bytes packets errors dropped overrun mcast
> >> 99686284170801 173507590134 61 669685 0 100304421
> >> TX: bytes packets errors dropped carrier collsns
> >> 184435107970545 142383178304 0 0 0 0
> >>
You have increased the default (1000) qlen to 8192, why?
What default qdisc do you run?... looking through your very detail main
email report (I do love the details you give!). You run
pfifo_fast_dequeue, thus this 8192 qlen is actually having effect.
I would like to know if and how much qdisc_dequeue bulking is happening
in this setup? Can you run:
perf-stat-hist -m 8192 -P2 qdisc:qdisc_dequeue packets
The perf-stat-hist is from Brendan Gregg's git-tree:
https://github.com/brendangregg/perf-tools
https://github.com/brendangregg/perf-tools/blob/master/misc/perf-stat-hist
> >> ./softnet.sh
> >> cpu total dropped squeezed collision rps flow_limit
> >>
> >>
> >>
> >>
> >> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
> >> ------------------------------------------------------------------------------------------
> >>
> >> 26.78% [kernel] [k] queued_spin_lock_slowpath
> >
> > This is highly suspect.
> >
I agree! -- 26.78% spend in queued_spin_lock_slowpath. Hint if you see
_raw_spin_lock then it is likely not a contended lock, but if you see
queued_spin_lock_slowpath in a perf-report your workload is likely in
trouble.
> > A call graph (perf record -a -g sleep 1; perf report --stdio)
> > would tell what is going on.
>
> perf report:
> https://ufile.io/rqp0h
>
Thanks for the output (my 30" screen is just large enough to see the
full output). Together with the flame-graph, it is clear that this
lock happens in the page allocator code.
Section copied out:
mlx5e_poll_tx_cq
|
--16.34%--napi_consume_skb
|
|--12.65%--__free_pages_ok
| |
| --11.86%--free_one_page
| |
| |--10.10%--queued_spin_lock_slowpath
| |
| --0.65%--_raw_spin_lock
|
|--1.55%--page_frag_free
|
--1.44%--skb_release_data
Let me explain what (I think) happens. The mlx5 driver RX-page recycle
mechanism is not effective in this workload, and pages have to go
through the page allocator. The lock contention happens during mlx5
DMA TX completion cycle. And the page allocator cannot keep up at
these speeds.
One solution is extend page allocator with a bulk free API. (This have
been on my TODO list for a long time, but I don't have a
micro-benchmark that trick the driver page-recycle to fail). It should
fit nicely, as I can see that kmem_cache_free_bulk() does get
activated (bulk freeing SKBs), which means that DMA TX completion do
have a bulk of packets.
We can (and should) also improve the page recycle scheme in the driver.
After LPC, I have a project with Tariq and Ilias (Cc'ed) to improve the
page_pool, and we will (attempt) to generalize this, for both high-end
mlx5 and more low-end ARM64-boards (macchiatobin and espressobin).
The MM-people is in parallel working to improve the performance of
order-0 page returns. Thus, the explicit page bulk free API might
actually become less important. I actually think (Cc.) Aaron have a
patchset he would like you to test, which removes the (zone->)lock
you hit in free_one_page().
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-10-31 21:57 Kernel 4.19 network performance - forwarding/routing normal users traffic Paweł Staszewski
2018-10-31 22:09 ` Eric Dumazet
2018-11-01 3:37 ` David Ahern
@ 2018-11-01 9:50 ` Saeed Mahameed
2018-11-01 11:09 ` Paweł Staszewski
2 siblings, 1 reply; 77+ messages in thread
From: Saeed Mahameed @ 2018-11-01 9:50 UTC (permalink / raw)
To: pstaszewski, netdev
On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
> Hi
>
> So maybee someone will be interested how linux kernel handles normal
> traffic (not pktgen :) )
>
>
> Server HW configuration:
>
> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>
> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>
>
> Server software:
>
> FRR - as routing daemon
>
> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local
> numa
> node)
>
> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa
> node)
>
>
> Maximum traffic that server can handle:
>
> Bandwidth
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> \ iface Rx Tx Total
> =====================================================================
> =========
> enp175s0f1: 28.51 Gb/s 37.24
> Gb/s
> 65.74 Gb/s
> enp175s0f0: 38.07 Gb/s 28.44
> Gb/s
> 66.51 Gb/s
> -------------------------------------------------------------------
> -----------
> total: 66.58 Gb/s 65.67
> Gb/s
> 132.25 Gb/s
>
>
> Packets per second:
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> - iface Rx Tx Total
> =====================================================================
> =========
> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
> 8735207.00 P/s
> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
> 8790460.00 P/s
> -------------------------------------------------------------------
> -----------
> total: 8806533.00 P/s 8719134.00 P/s
> 17525668.00 P/s
>
>
> After reaching that limits nics on the upstream side (more RX
> traffic)
> start to drop packets
>
>
> I just dont understand that server can't handle more bandwidth
> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
> side are increasing.
>
Where do you see 40 Gb/s ? you showed that both ports on the same NIC (
same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) = 132.25
Gb/s which aligns with your pcie link limit, what am i missing ?
> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
> is
> 126Gbit - and also when testing with pktgen i can reach more bw and
> pps
> (like 4x more comparing to normal internet traffic)
>
Are you forwarding when using pktgen as well or you just testing the RX
side pps ?
> And wondering if there is something that can be improved here.
>
>
>
> Some more informations / counters / stats and perf top below:
>
> Perf top flame graph:
>
> https://uploadfiles.io/7zo6u
>
>
>
> System configuration(long):
>
>
> cat /sys/devices/system/node/node1/cpulist
> 14-27,42-55
> cat /sys/class/net/enp175s0f0/device/numa_node
> 1
> cat /sys/class/net/enp175s0f1/device/numa_node
> 1
>
>
>
>
>
> ip -s -d link ls dev enp175s0f0
> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state
> UP mode DEFAULT group default qlen 8192
> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity
> 0
> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
> gso_max_segs 65535
> RX: bytes packets errors dropped overrun mcast
> 184142375840858 141347715974 2 2806325 0 85050528
> TX: bytes packets errors dropped carrier collsns
> 99270697277430 172227994003 0 0 0 0
>
> ip -s -d link ls dev enp175s0f1
> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state
> UP mode DEFAULT group default qlen 8192
> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity
> 0
> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
> gso_max_segs 65535
> RX: bytes packets errors dropped overrun mcast
> 99686284170801 173507590134 61 669685 0 100304421
> TX: bytes packets errors dropped carrier collsns
> 184435107970545 142383178304 0 0 0 0
>
>
> ./softnet.sh
> cpu total dropped squeezed collision rps flow_limit
> 0 3961392822 0 1221478 0 0 0
> 1 3701952251 0 1258234 0 0 0
> 2 3879522030 0 1584282 0 0 0
> 3 3731349789 0 1529029 0 0 0
> 4 1323956701 0 2176371 0 0 0
> 5 420528963 0 1880146 0 0 0
> 6 348720322 0 1830142 0 0 0
> 7 372736328 0 1820891 0 0 0
> 8 567888751 0 1414763 0 0 0
> 9 476075775 0 1868150 0 0 0
> 10 468946725 0 1841428 0 0 0
> 11 676591958 0 1900160 0 0 0
> 12 346803472 0 1834600 0 0 0
> 13 457960872 0 1874529 0 0 0
> 14 1990279665 0 4699000 0 0 0
> 15 1211873601 0 4541281 0 0 0
> 16 1123871928 0 4544712 0 0 0
> 17 1014957263 0 4152355 0 0 0
> 18 2603779724 0 4593869 0 0 0
> 19 2181924054 0 4930618 0 0 0
> 20 2273502182 0 4894627 0 0 0
> 21 2232030947 0 4860048 0 0 0
> 22 2203555394 0 4603830 0 0 0
> 23 2194756800 0 4921294 0 0 0
> 24 2347158294 0 4818354 0 0 0
> 25 2291097883 0 4744469 0 0 0
> 26 2206945011 0 4836483 0 0 0
> 27 2318530217 0 4917617 0 0 0
> 28 512797543 0 1895200 0 0 0
> 29 597279474 0 1532134 0 0 0
> 30 475317503 0 1451523 0 0 0
> 31 499172796 0 1901207 0 0 0
> 32 493874745 0 1915382 0 0 0
> 33 296056288 0 1865535 0 0 0
> 34 3905097041 0 1580822 0 0 0
> 35 3905112345 0 1536105 0 0 0
> 36 3900358950 0 1166319 0 0 0
> 37 3940978093 0 1600219 0 0 0
> 38 3878632215 0 1180389 0 0 0
> 39 3814804736 0 1584925 0 0 0
> 40 4152934337 0 1663660 0 0 0
> 41 3855273904 0 1552219 0 0 0
> 42 2319538182 0 4884480 0 0 0
> 43 2448606991 0 4387456 0 0 0
> 44 1436136753 0 4485073 0 0 0
> 45 1200500141 0 4537284 0 0 0
> 46 1307799923 0 4534156 0 0 0
> 47 1586575293 0 4272997 0 0 0
> 48 3852574 0 4162653 0 0 0
> 49 391449390 0 3935202 0 0 0
> 50 791388200 0 4290738 0 0 0
> 51 127107573 0 3907750 0 0 0
> 52 115622148 0 4012843 0 0 0
> 53 71098871 0 4200625 0 0 0
> 54 305121466 0 4365614 0 0 0
> 55 10914257 0 4369426 0 0 0
>
>
>
>
> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> -------------------------------------------------------------------
> -------------------------------------------------------------------
> -------------------------------------------------------------------
> ------
>
> 26.78% [kernel] [k] queued_spin_lock_slowpath
> 9.09% [kernel] [k] mlx5e_skb_from_cqe_linear
> 4.94% [kernel] [k] mlx5e_sq_xmit
> 3.63% [kernel] [k] memcpy_erms
> 3.30% [kernel] [k] fib_table_lookup
> 3.26% [kernel] [k] build_skb
> 2.41% [kernel] [k] mlx5e_poll_tx_cq
> 2.11% [kernel] [k] get_page_from_freelist
> 1.51% [kernel] [k] vlan_do_receive
> 1.51% [kernel] [k] _raw_spin_lock
> 1.43% [kernel] [k] __dev_queue_xmit
> 1.41% [kernel] [k] dev_gro_receive
> 1.34% [kernel] [k] mlx5e_poll_rx_cq
> 1.26% [kernel] [k] tcp_gro_receive
> 1.21% [kernel] [k] free_one_page
> 1.13% [kernel] [k] swiotlb_map_page
> 1.13% [kernel] [k] mlx5e_post_rx_wqes
> 1.05% [kernel] [k] pfifo_fast_dequeue
> 1.05% [kernel] [k] mlx5e_handle_rx_cqe
> 1.03% [kernel] [k] ip_finish_output2
> 1.02% [kernel] [k] ipt_do_table
> 0.96% [kernel] [k] inet_gro_receive
> 0.91% [kernel] [k] mlx5_eq_int
> 0.88% [kernel] [k] __slab_free.isra.79
> 0.86% [kernel] [k] __build_skb
> 0.84% [kernel] [k] page_frag_free
> 0.76% [kernel] [k] skb_release_data
> 0.75% [kernel] [k] __netif_receive_skb_core
> 0.75% [kernel] [k] irq_entries_start
> 0.71% [kernel] [k] ip_route_input_rcu
> 0.65% [kernel] [k] vlan_dev_hard_start_xmit
> 0.56% [kernel] [k] ip_forward
> 0.56% [kernel] [k] __memcpy
> 0.52% [kernel] [k] kmem_cache_alloc
> 0.52% [kernel] [k] kmem_cache_free_bulk
> 0.49% [kernel] [k] mlx5e_page_release
> 0.47% [kernel] [k] netif_skb_features
> 0.47% [kernel] [k] mlx5e_build_rx_skb
> 0.47% [kernel] [k] dev_hard_start_xmit
> 0.43% [kernel] [k] __page_pool_put_page
> 0.43% [kernel] [k] __netif_schedule
> 0.43% [kernel] [k] mlx5e_xmit
> 0.41% [kernel] [k] __qdisc_run
> 0.41% [kernel] [k] validate_xmit_skb.isra.142
> 0.41% [kernel] [k] swiotlb_unmap_page
> 0.40% [kernel] [k] inet_lookup_ifaddr_rcu
> 0.34% [kernel] [k] ip_rcv_core.isra.20.constprop.25
> 0.34% [kernel] [k] tcp4_gro_receive
> 0.29% [kernel] [k] _raw_spin_lock_irqsave
> 0.29% [kernel] [k] napi_consume_skb
> 0.29% [kernel] [k] skb_gro_receive
> 0.29% [kernel] [k] ___slab_alloc.isra.80
> 0.27% [kernel] [k] eth_type_trans
> 0.26% [kernel] [k] __free_pages_ok
> 0.26% [kernel] [k] __get_xps_queue_idx
> 0.24% [kernel] [k] _raw_spin_trylock
> 0.23% [kernel] [k] __local_bh_enable_ip
> 0.22% [kernel] [k] pfifo_fast_enqueue
> 0.21% [kernel] [k] tasklet_action_common.isra.21
> 0.21% [kernel] [k] sch_direct_xmit
> 0.21% [kernel] [k] skb_network_protocol
> 0.21% [kernel] [k] kmem_cache_free
> 0.20% [kernel] [k] netdev_pick_tx
> 0.18% [kernel] [k] napi_gro_complete
> 0.18% [kernel] [k] __sched_text_start
> 0.18% [kernel] [k] mlx5e_xdp_handle
> 0.17% [kernel] [k] ip_finish_output
> 0.16% [kernel] [k] napi_gro_flush
> 0.16% [kernel] [k] vlan_passthru_hard_header
> 0.16% [kernel] [k] skb_segment
> 0.15% [kernel] [k] __alloc_pages_nodemask
> 0.15% [kernel] [k] mlx5e_features_check
> 0.15% [kernel] [k] mlx5e_napi_poll
> 0.15% [kernel] [k] napi_gro_receive
> 0.14% [kernel] [k] fib_validate_source
> 0.14% [kernel] [k] _raw_spin_lock_irq
> 0.14% [kernel] [k] inet_gro_complete
> 0.14% [kernel] [k] get_partial_node.isra.78
> 0.13% [kernel] [k] napi_complete_done
> 0.13% [kernel] [k] ip_rcv_finish_core.isra.17
> 0.13% [kernel] [k] cmd_exec
>
>
>
> ethtool -S enp175s0f1
> NIC statistics:
> rx_packets: 173730800927
> rx_bytes: 99827422751332
> tx_packets: 142532009512
> tx_bytes: 184633045911222
> tx_tso_packets: 25989113891
> tx_tso_bytes: 132933363384458
> tx_tso_inner_packets: 0
> tx_tso_inner_bytes: 0
> tx_added_vlan_packets: 74630239613
> tx_nop: 2029817748
> rx_lro_packets: 0
> rx_lro_bytes: 0
> rx_ecn_mark: 0
> rx_removed_vlan_packets: 173730800927
> rx_csum_unnecessary: 0
> rx_csum_none: 434357
> rx_csum_complete: 173730366570
> rx_csum_unnecessary_inner: 0
> rx_xdp_drop: 0
> rx_xdp_redirect: 0
> rx_xdp_tx_xmit: 0
> rx_xdp_tx_full: 0
> rx_xdp_tx_err: 0
> rx_xdp_tx_cqe: 0
> tx_csum_none: 38260960853
> tx_csum_partial: 36369278774
> tx_csum_partial_inner: 0
> tx_queue_stopped: 1
> tx_queue_dropped: 0
> tx_xmit_more: 748638099
> tx_recover: 0
> tx_cqes: 73881645031
> tx_queue_wake: 1
> tx_udp_seg_rem: 0
> tx_cqe_err: 0
> tx_xdp_xmit: 0
> tx_xdp_full: 0
> tx_xdp_err: 0
> tx_xdp_cqes: 0
> rx_wqe_err: 0
> rx_mpwqe_filler_cqes: 0
> rx_mpwqe_filler_strides: 0
> rx_buff_alloc_err: 0
> rx_cqe_compress_blks: 0
> rx_cqe_compress_pkts: 0
If this is a pcie bottleneck it might be useful to enable CQE
compression (to reduce PCIe completion descriptors transactions)
you should see the above rx_cqe_compress_pkts increasing when enabled.
$ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
$ ethtool --show-priv-flags enp175s0f1
Private flags for p6p1:
rx_cqe_moder : on
cqe_moder : off
rx_cqe_compress : on
...
try this on both interfaces.
> rx_page_reuse: 0
> rx_cache_reuse: 14441066823
> rx_cache_full: 51126004413
> rx_cache_empty: 21297344082
> rx_cache_busy: 51127247487
> rx_cache_waive: 21298322293
> rx_congst_umr: 0
> rx_arfs_err: 0
> ch_events: 24603119858
> ch_poll: 25180949074
> ch_arm: 24480437587
> ch_aff_change: 75
> ch_eq_rearm: 0
> rx_out_of_buffer: 669685
comparing this to rx_vport_unicast_packets, it is a very small
percentage of dropped packets due to stalled rx cpu, so rx cpu is not a
bottleneck, at least for the driver rx rings.
> rx_if_down_packets: 61
> rx_vport_unicast_packets: 173731641945
> rx_vport_unicast_bytes: 100522745036693
> tx_vport_unicast_packets: 142531901313
> tx_vport_unicast_bytes: 185189071776429
> rx_vport_multicast_packets: 100360886
> rx_vport_multicast_bytes: 6639236688
> tx_vport_multicast_packets: 32837
> tx_vport_multicast_bytes: 2978810
> rx_vport_broadcast_packets: 44854
> rx_vport_broadcast_bytes: 6313510
> tx_vport_broadcast_packets: 72258
> tx_vport_broadcast_bytes: 4335480
> rx_vport_rdma_unicast_packets: 0
> rx_vport_rdma_unicast_bytes: 0
> tx_vport_rdma_unicast_packets: 0
> tx_vport_rdma_unicast_bytes: 0
> rx_vport_rdma_multicast_packets: 0
> rx_vport_rdma_multicast_bytes: 0
> tx_vport_rdma_multicast_packets: 0
> tx_vport_rdma_multicast_bytes: 0
> tx_packets_phy: 142532004669
> rx_packets_phy: 173980375752
> rx_crc_errors_phy: 0
> tx_bytes_phy: 185759204762903
> rx_bytes_phy: 101326109361379
> tx_multicast_phy: 32837
> tx_broadcast_phy: 72258
> rx_multicast_phy: 100360885
> rx_broadcast_phy: 44854
> rx_in_range_len_errors_phy: 2
> rx_out_of_range_len_phy: 0
> rx_oversize_pkts_phy: 59
> rx_symbol_err_phy: 0
> tx_mac_control_phy: 0
> rx_mac_control_phy: 0
> rx_unsupported_op_phy: 0
> rx_pause_ctrl_phy: 0
> tx_pause_ctrl_phy: 0
> rx_discards_phy: 148328738
> tx_discards_phy: 0
> tx_errors_phy: 0
> rx_undersize_pkts_phy: 0
> rx_fragments_phy: 0
> rx_jabbers_phy: 0
> rx_64_bytes_phy: 36551843112
> rx_65_to_127_bytes_phy: 65102131735
> rx_128_to_255_bytes_phy: 5755731137
> rx_256_to_511_bytes_phy: 2475619839
> rx_512_to_1023_bytes_phy: 2826971156
> rx_1024_to_1518_bytes_phy: 42474023107
> rx_1519_to_2047_bytes_phy: 18794051270
> rx_2048_to_4095_bytes_phy: 0
> rx_4096_to_8191_bytes_phy: 0
> rx_8192_to_10239_bytes_phy: 0
> link_down_events_phy: 0
> rx_pcs_symbol_err_phy: 0
> rx_corrected_bits_phy: 0
> rx_pci_signal_integrity: 0
> tx_pci_signal_integrity: 48
> rx_prio0_bytes: 101316322498995
> rx_prio0_packets: 173711151686
> tx_prio0_bytes: 185759176566814
> tx_prio0_packets: 142531983704
> rx_prio1_bytes: 47062768
> rx_prio1_packets: 228932
> tx_prio1_bytes: 0
> tx_prio1_packets: 0
> rx_prio2_bytes: 12434759
> rx_prio2_packets: 83773
> tx_prio2_bytes: 0
> tx_prio2_packets: 0
> rx_prio3_bytes: 288843134
> rx_prio3_packets: 982102
> tx_prio3_bytes: 0
> tx_prio3_packets: 0
> rx_prio4_bytes: 699797236
> rx_prio4_packets: 8109231
> tx_prio4_bytes: 0
> tx_prio4_packets: 0
> rx_prio5_bytes: 1385386738
> rx_prio5_packets: 9661187
> tx_prio5_bytes: 0
> tx_prio5_packets: 0
> rx_prio6_bytes: 317092102
> rx_prio6_packets: 1951538
> tx_prio6_bytes: 0
> tx_prio6_packets: 0
> rx_prio7_bytes: 7015734695
> rx_prio7_packets: 99847456
> tx_prio7_bytes: 0
> tx_prio7_packets: 0
> module_unplug: 0
> module_bus_stuck: 0
> module_high_temp: 0
> module_bad_shorted: 0
> ch0_events: 936264703
> ch0_poll: 963766474
> ch0_arm: 930246079
> ch0_aff_change: 0
> ch0_eq_rearm: 0
> ch1_events: 869408429
> ch1_poll: 896099392
> ch1_arm: 864336861
> ch1_aff_change: 0
> ch1_eq_rearm: 0
> ch2_events: 843345698
> ch2_poll: 869749522
> ch2_arm: 838186113
> ch2_aff_change: 2
> ch2_eq_rearm: 0
> ch3_events: 850261340
> ch3_poll: 876721111
> ch3_arm: 845295235
> ch3_aff_change: 3
> ch3_eq_rearm: 0
> ch4_events: 974985780
> ch4_poll: 997781915
> ch4_arm: 969618250
> ch4_aff_change: 3
> ch4_eq_rearm: 0
> ch5_events: 888559089
> ch5_poll: 912783615
> ch5_arm: 883826078
> ch5_aff_change: 2
> ch5_eq_rearm: 0
> ch6_events: 873730730
> ch6_poll: 899635752
> ch6_arm: 868677574
> ch6_aff_change: 4
> ch6_eq_rearm: 0
> ch7_events: 873478411
> ch7_poll: 899216716
> ch7_arm: 868693645
> ch7_aff_change: 3
> ch7_eq_rearm: 0
> ch8_events: 871900967
> ch8_poll: 898575518
> ch8_arm: 866763693
> ch8_aff_change: 3
> ch8_eq_rearm: 0
> ch9_events: 880325565
> ch9_poll: 904983269
> ch9_arm: 875643922
> ch9_aff_change: 2
> ch9_eq_rearm: 0
> ch10_events: 889919775
> ch10_poll: 915335809
> ch10_arm: 885110225
> ch10_aff_change: 4
> ch10_eq_rearm: 0
> ch11_events: 962709175
> ch11_poll: 983963451
> ch11_arm: 958117526
> ch11_aff_change: 2
> ch11_eq_rearm: 0
> ch12_events: 941333837
> ch12_poll: 964625523
> ch12_arm: 936409706
> ch12_aff_change: 2
> ch12_eq_rearm: 0
> ch13_events: 914996974
> ch13_poll: 937441049
> ch13_arm: 910478393
> ch13_aff_change: 4
> ch13_eq_rearm: 0
> ch14_events: 888050001
> ch14_poll: 911818008
> ch14_arm: 883465035
> ch14_aff_change: 4
> ch14_eq_rearm: 0
> ch15_events: 947547704
> ch15_poll: 969073194
> ch15_arm: 942686515
> ch15_aff_change: 4
> ch15_eq_rearm: 0
> ch16_events: 825804904
> ch16_poll: 840630747
> ch16_arm: 822227488
> ch16_aff_change: 2
> ch16_eq_rearm: 0
> ch17_events: 861673823
> ch17_poll: 874754041
> ch17_arm: 858520448
> ch17_aff_change: 2
> ch17_eq_rearm: 0
> ch18_events: 879413440
> ch18_poll: 893962529
> ch18_arm: 875983204
> ch18_aff_change: 4
> ch18_eq_rearm: 0
> ch19_events: 896073709
> ch19_poll: 909216857
> ch19_arm: 893022121
> ch19_aff_change: 4
> ch19_eq_rearm: 0
> ch20_events: 865188535
> ch20_poll: 880692345
> ch20_arm: 861440265
> ch20_aff_change: 3
> ch20_eq_rearm: 0
> ch21_events: 862709303
> ch21_poll: 878104242
> ch21_arm: 859041767
> ch21_aff_change: 2
> ch21_eq_rearm: 0
> ch22_events: 887720551
> ch22_poll: 904122074
> ch22_arm: 883983794
> ch22_aff_change: 2
> ch22_eq_rearm: 0
> ch23_events: 813355027
> ch23_poll: 828074467
> ch23_arm: 809912398
> ch23_aff_change: 4
> ch23_eq_rearm: 0
> ch24_events: 822366675
> ch24_poll: 839917937
> ch24_arm: 818422754
> ch24_aff_change: 2
> ch24_eq_rearm: 0
> ch25_events: 826642292
> ch25_poll: 842630121
> ch25_arm: 822642618
> ch25_aff_change: 2
> ch25_eq_rearm: 0
> ch26_events: 826392584
> ch26_poll: 843406973
> ch26_arm: 822455000
> ch26_aff_change: 3
> ch26_eq_rearm: 0
> ch27_events: 828960899
> ch27_poll: 843866518
> ch27_arm: 825230937
> ch27_aff_change: 3
> ch27_eq_rearm: 0
> ch28_events: 7
> ch28_poll: 7
> ch28_arm: 7
> ch28_aff_change: 0
> ch28_eq_rearm: 0
> ch29_events: 4
> ch29_poll: 4
> ch29_arm: 4
> ch29_aff_change: 0
> ch29_eq_rearm: 0
> ch30_events: 4
> ch30_poll: 4
> ch30_arm: 4
> ch30_aff_change: 0
> ch30_eq_rearm: 0
> ch31_events: 4
> ch31_poll: 4
> ch31_arm: 4
> ch31_aff_change: 0
> ch31_eq_rearm: 0
> ch32_events: 4
> ch32_poll: 4
> ch32_arm: 4
> ch32_aff_change: 0
> ch32_eq_rearm: 0
> ch33_events: 4
> ch33_poll: 4
> ch33_arm: 4
> ch33_aff_change: 0
> ch33_eq_rearm: 0
> ch34_events: 4
> ch34_poll: 4
> ch34_arm: 4
> ch34_aff_change: 0
> ch34_eq_rearm: 0
> ch35_events: 4
> ch35_poll: 4
> ch35_arm: 4
> ch35_aff_change: 0
> ch35_eq_rearm: 0
> ch36_events: 4
> ch36_poll: 4
> ch36_arm: 4
> ch36_aff_change: 0
> ch36_eq_rearm: 0
> ch37_events: 4
> ch37_poll: 4
> ch37_arm: 4
> ch37_aff_change: 0
> ch37_eq_rearm: 0
> ch38_events: 4
> ch38_poll: 4
> ch38_arm: 4
> ch38_aff_change: 0
> ch38_eq_rearm: 0
> ch39_events: 4
> ch39_poll: 4
> ch39_arm: 4
> ch39_aff_change: 0
> ch39_eq_rearm: 0
> ch40_events: 4
> ch40_poll: 4
> ch40_arm: 4
> ch40_aff_change: 0
> ch40_eq_rearm: 0
> ch41_events: 4
> ch41_poll: 4
> ch41_arm: 4
> ch41_aff_change: 0
> ch41_eq_rearm: 0
> ch42_events: 4
> ch42_poll: 4
> ch42_arm: 4
> ch42_aff_change: 0
> ch42_eq_rearm: 0
> ch43_events: 4
> ch43_poll: 4
> ch43_arm: 4
> ch43_aff_change: 0
> ch43_eq_rearm: 0
> ch44_events: 4
> ch44_poll: 4
> ch44_arm: 4
> ch44_aff_change: 0
> ch44_eq_rearm: 0
> ch45_events: 4
> ch45_poll: 4
> ch45_arm: 4
> ch45_aff_change: 0
> ch45_eq_rearm: 0
> ch46_events: 4
> ch46_poll: 4
> ch46_arm: 4
> ch46_aff_change: 0
> ch46_eq_rearm: 0
> ch47_events: 4
> ch47_poll: 4
> ch47_arm: 4
> ch47_aff_change: 0
> ch47_eq_rearm: 0
> ch48_events: 4
> ch48_poll: 4
> ch48_arm: 4
> ch48_aff_change: 0
> ch48_eq_rearm: 0
> ch49_events: 4
> ch49_poll: 4
> ch49_arm: 4
> ch49_aff_change: 0
> ch49_eq_rearm: 0
> ch50_events: 4
> ch50_poll: 4
> ch50_arm: 4
> ch50_aff_change: 0
> ch50_eq_rearm: 0
> ch51_events: 4
> ch51_poll: 4
> ch51_arm: 4
> ch51_aff_change: 0
> ch51_eq_rearm: 0
> ch52_events: 4
> ch52_poll: 4
> ch52_arm: 4
> ch52_aff_change: 0
> ch52_eq_rearm: 0
> ch53_events: 4
> ch53_poll: 4
> ch53_arm: 4
> ch53_aff_change: 0
> ch53_eq_rearm: 0
> ch54_events: 4
> ch54_poll: 4
> ch54_arm: 4
> ch54_aff_change: 0
> ch54_eq_rearm: 0
> ch55_events: 4
> ch55_poll: 4
> ch55_arm: 4
> ch55_aff_change: 0
> ch55_eq_rearm: 0
> rx0_packets: 7284057433
> rx0_bytes: 4330611281319
> rx0_csum_complete: 7283623076
> rx0_csum_unnecessary: 0
> rx0_csum_unnecessary_inner: 0
> rx0_csum_none: 434357
> rx0_xdp_drop: 0
> rx0_xdp_redirect: 0
> rx0_lro_packets: 0
> rx0_lro_bytes: 0
> rx0_ecn_mark: 0
> rx0_removed_vlan_packets: 7284057433
> rx0_wqe_err: 0
> rx0_mpwqe_filler_cqes: 0
> rx0_mpwqe_filler_strides: 0
> rx0_buff_alloc_err: 0
> rx0_cqe_compress_blks: 0
> rx0_cqe_compress_pkts: 0
> rx0_page_reuse: 0
> rx0_cache_reuse: 1989731589
> rx0_cache_full: 28213297
> rx0_cache_empty: 1624089822
> rx0_cache_busy: 28213961
> rx0_cache_waive: 1624083610
> rx0_congst_umr: 0
> rx0_arfs_err: 0
> rx0_xdp_tx_xmit: 0
> rx0_xdp_tx_full: 0
> rx0_xdp_tx_err: 0
> rx0_xdp_tx_cqes: 0
> rx1_packets: 6691319211
> rx1_bytes: 3799580210608
> rx1_csum_complete: 6691319211
> rx1_csum_unnecessary: 0
> rx1_csum_unnecessary_inner: 0
> rx1_csum_none: 0
> rx1_xdp_drop: 0
> rx1_xdp_redirect: 0
> rx1_lro_packets: 0
> rx1_lro_bytes: 0
> rx1_ecn_mark: 0
> rx1_removed_vlan_packets: 6691319211
> rx1_wqe_err: 0
> rx1_mpwqe_filler_cqes: 0
> rx1_mpwqe_filler_strides: 0
> rx1_buff_alloc_err: 0
> rx1_cqe_compress_blks: 0
> rx1_cqe_compress_pkts: 0
> rx1_page_reuse: 0
> rx1_cache_reuse: 2270019
> rx1_cache_full: 3343389331
> rx1_cache_empty: 6656
> rx1_cache_busy: 3343389585
> rx1_cache_waive: 0
> rx1_congst_umr: 0
> rx1_arfs_err: 0
> rx1_xdp_tx_xmit: 0
> rx1_xdp_tx_full: 0
> rx1_xdp_tx_err: 0
> rx1_xdp_tx_cqes: 0
> rx2_packets: 6618370416
> rx2_bytes: 3762508364015
> rx2_csum_complete: 6618370416
> rx2_csum_unnecessary: 0
> rx2_csum_unnecessary_inner: 0
> rx2_csum_none: 0
> rx2_xdp_drop: 0
> rx2_xdp_redirect: 0
> rx2_lro_packets: 0
> rx2_lro_bytes: 0
> rx2_ecn_mark: 0
> rx2_removed_vlan_packets: 6618370416
> rx2_wqe_err: 0
> rx2_mpwqe_filler_cqes: 0
> rx2_mpwqe_filler_strides: 0
> rx2_buff_alloc_err: 0
> rx2_cqe_compress_blks: 0
> rx2_cqe_compress_pkts: 0
> rx2_page_reuse: 0
> rx2_cache_reuse: 111419328
> rx2_cache_full: 1807563903
> rx2_cache_empty: 1390208158
> rx2_cache_busy: 1807564378
> rx2_cache_waive: 1390201722
> rx2_congst_umr: 0
> rx2_arfs_err: 0
> rx2_xdp_tx_xmit: 0
> rx2_xdp_tx_full: 0
> rx2_xdp_tx_err: 0
> rx2_xdp_tx_cqes: 0
> rx3_packets: 6665308976
> rx3_bytes: 3828546206006
> rx3_csum_complete: 6665308976
> rx3_csum_unnecessary: 0
> rx3_csum_unnecessary_inner: 0
> rx3_csum_none: 0
> rx3_xdp_drop: 0
> rx3_xdp_redirect: 0
> rx3_lro_packets: 0
> rx3_lro_bytes: 0
> rx3_ecn_mark: 0
> rx3_removed_vlan_packets: 6665308976
> rx3_wqe_err: 0
> rx3_mpwqe_filler_cqes: 0
> rx3_mpwqe_filler_strides: 0
> rx3_buff_alloc_err: 0
> rx3_cqe_compress_blks: 0
> rx3_cqe_compress_pkts: 0
> rx3_page_reuse: 0
> rx3_cache_reuse: 215779091
> rx3_cache_full: 1720040649
> rx3_cache_empty: 1396840926
> rx3_cache_busy: 1720041127
> rx3_cache_waive: 1396834493
> rx3_congst_umr: 0
> rx3_arfs_err: 0
> rx3_xdp_tx_xmit: 0
> rx3_xdp_tx_full: 0
> rx3_xdp_tx_err: 0
> rx3_xdp_tx_cqes: 0
> rx4_packets: 6764448165
> rx4_bytes: 3883101339142
> rx4_csum_complete: 6764448165
> rx4_csum_unnecessary: 0
> rx4_csum_unnecessary_inner: 0
> rx4_csum_none: 0
> rx4_xdp_drop: 0
> rx4_xdp_redirect: 0
> rx4_lro_packets: 0
> rx4_lro_bytes: 0
> rx4_ecn_mark: 0
> rx4_removed_vlan_packets: 6764448165
> rx4_wqe_err: 0
> rx4_mpwqe_filler_cqes: 0
> rx4_mpwqe_filler_strides: 0
> rx4_buff_alloc_err: 0
> rx4_cqe_compress_blks: 0
> rx4_cqe_compress_pkts: 0
> rx4_page_reuse: 0
> rx4_cache_reuse: 1930710653
> rx4_cache_full: 6490815
> rx4_cache_empty: 1445028605
> rx4_cache_busy: 6491478
> rx4_cache_waive: 1445022392
> rx4_congst_umr: 0
> rx4_arfs_err: 0
> rx4_xdp_tx_xmit: 0
> rx4_xdp_tx_full: 0
> rx4_xdp_tx_err: 0
> rx4_xdp_tx_cqes: 0
> rx5_packets: 6736853264
> rx5_bytes: 3925186068552
> rx5_csum_complete: 6736853264
> rx5_csum_unnecessary: 0
> rx5_csum_unnecessary_inner: 0
> rx5_csum_none: 0
> rx5_xdp_drop: 0
> rx5_xdp_redirect: 0
> rx5_lro_packets: 0
> rx5_lro_bytes: 0
> rx5_ecn_mark: 0
> rx5_removed_vlan_packets: 6736853264
> rx5_wqe_err: 0
> rx5_mpwqe_filler_cqes: 0
> rx5_mpwqe_filler_strides: 0
> rx5_buff_alloc_err: 0
> rx5_cqe_compress_blks: 0
> rx5_cqe_compress_pkts: 0
> rx5_page_reuse: 0
> rx5_cache_reuse: 7283914
> rx5_cache_full: 3361142463
> rx5_cache_empty: 6656
> rx5_cache_busy: 3361142718
> rx5_cache_waive: 0
> rx5_congst_umr: 0
> rx5_arfs_err: 0
> rx5_xdp_tx_xmit: 0
> rx5_xdp_tx_full: 0
> rx5_xdp_tx_err: 0
> rx5_xdp_tx_cqes: 0
> rx6_packets: 6751588828
> rx6_bytes: 3860537598885
> rx6_csum_complete: 6751588828
> rx6_csum_unnecessary: 0
> rx6_csum_unnecessary_inner: 0
> rx6_csum_none: 0
> rx6_xdp_drop: 0
> rx6_xdp_redirect: 0
> rx6_lro_packets: 0
> rx6_lro_bytes: 0
> rx6_ecn_mark: 0
> rx6_removed_vlan_packets: 6751588828
> rx6_wqe_err: 0
> rx6_mpwqe_filler_cqes: 0
> rx6_mpwqe_filler_strides: 0
> rx6_buff_alloc_err: 0
> rx6_cqe_compress_blks: 0
> rx6_cqe_compress_pkts: 0
> rx6_page_reuse: 0
> rx6_cache_reuse: 96032126
> rx6_cache_full: 1857890923
> rx6_cache_empty: 1421877543
> rx6_cache_busy: 1857891399
> rx6_cache_waive: 1421871110
> rx6_congst_umr: 0
> rx6_arfs_err: 0
> rx6_xdp_tx_xmit: 0
> rx6_xdp_tx_full: 0
> rx6_xdp_tx_err: 0
> rx6_xdp_tx_cqes: 0
> rx7_packets: 6935300074
> rx7_bytes: 4004713524388
> rx7_csum_complete: 6935300074
> rx7_csum_unnecessary: 0
> rx7_csum_unnecessary_inner: 0
> rx7_csum_none: 0
> rx7_xdp_drop: 0
> rx7_xdp_redirect: 0
> rx7_lro_packets: 0
> rx7_lro_bytes: 0
> rx7_ecn_mark: 0
> rx7_removed_vlan_packets: 6935300074
> rx7_wqe_err: 0
> rx7_mpwqe_filler_cqes: 0
> rx7_mpwqe_filler_strides: 0
> rx7_buff_alloc_err: 0
> rx7_cqe_compress_blks: 0
> rx7_cqe_compress_pkts: 0
> rx7_page_reuse: 0
> rx7_cache_reuse: 17555187
> rx7_cache_full: 3450094595
> rx7_cache_empty: 6656
> rx7_cache_busy: 3450094849
> rx7_cache_waive: 0
> rx7_congst_umr: 0
> rx7_arfs_err: 0
> rx7_xdp_tx_xmit: 0
> rx7_xdp_tx_full: 0
> rx7_xdp_tx_err: 0
> rx7_xdp_tx_cqes: 0
> rx8_packets: 6678640094
> rx8_bytes: 3783722686028
> rx8_csum_complete: 6678640094
> rx8_csum_unnecessary: 0
> rx8_csum_unnecessary_inner: 0
> rx8_csum_none: 0
> rx8_xdp_drop: 0
> rx8_xdp_redirect: 0
> rx8_lro_packets: 0
> rx8_lro_bytes: 0
> rx8_ecn_mark: 0
> rx8_removed_vlan_packets: 6678640094
> rx8_wqe_err: 0
> rx8_mpwqe_filler_cqes: 0
> rx8_mpwqe_filler_strides: 0
> rx8_buff_alloc_err: 0
> rx8_cqe_compress_blks: 0
> rx8_cqe_compress_pkts: 0
> rx8_page_reuse: 0
> rx8_cache_reuse: 71006578
> rx8_cache_full: 1879380649
> rx8_cache_empty: 1388938999
> rx8_cache_busy: 1879381123
> rx8_cache_waive: 1388932565
> rx8_congst_umr: 0
> rx8_arfs_err: 0
> rx8_xdp_tx_xmit: 0
> rx8_xdp_tx_full: 0
> rx8_xdp_tx_err: 0
> rx8_xdp_tx_cqes: 0
> rx9_packets: 6709855557
> rx9_bytes: 3849522227880
> rx9_csum_complete: 6709855557
> rx9_csum_unnecessary: 0
> rx9_csum_unnecessary_inner: 0
> rx9_csum_none: 0
> rx9_xdp_drop: 0
> rx9_xdp_redirect: 0
> rx9_lro_packets: 0
> rx9_lro_bytes: 0
> rx9_ecn_mark: 0
> rx9_removed_vlan_packets: 6709855557
> rx9_wqe_err: 0
> rx9_mpwqe_filler_cqes: 0
> rx9_mpwqe_filler_strides: 0
> rx9_buff_alloc_err: 0
> rx9_cqe_compress_blks: 0
> rx9_cqe_compress_pkts: 0
> rx9_page_reuse: 0
> rx9_cache_reuse: 108980215
> rx9_cache_full: 1822730121
> rx9_cache_empty: 1423223623
> rx9_cache_busy: 1822730594
> rx9_cache_waive: 1423217187
> rx9_congst_umr: 0
> rx9_arfs_err: 0
> rx9_xdp_tx_xmit: 0
> rx9_xdp_tx_full: 0
> rx9_xdp_tx_err: 0
> rx9_xdp_tx_cqes: 0
> rx10_packets: 6761861066
> rx10_bytes: 3816266733385
> rx10_csum_complete: 6761861066
> rx10_csum_unnecessary: 0
> rx10_csum_unnecessary_inner: 0
> rx10_csum_none: 0
> rx10_xdp_drop: 0
> rx10_xdp_redirect: 0
> rx10_lro_packets: 0
> rx10_lro_bytes: 0
> rx10_ecn_mark: 0
> rx10_removed_vlan_packets: 6761861066
> rx10_wqe_err: 0
> rx10_mpwqe_filler_cqes: 0
> rx10_mpwqe_filler_strides: 0
> rx10_buff_alloc_err: 0
> rx10_cqe_compress_blks: 0
> rx10_cqe_compress_pkts: 0
> rx10_page_reuse: 0
> rx10_cache_reuse: 3489300
> rx10_cache_full: 3377440977
> rx10_cache_empty: 6656
> rx10_cache_busy: 3377441216
> rx10_cache_waive: 0
> rx10_congst_umr: 0
> rx10_arfs_err: 0
> rx10_xdp_tx_xmit: 0
> rx10_xdp_tx_full: 0
> rx10_xdp_tx_err: 0
> rx10_xdp_tx_cqes: 0
> rx11_packets: 6868113938
> rx11_bytes: 4048196300710
> rx11_csum_complete: 6868113938
> rx11_csum_unnecessary: 0
> rx11_csum_unnecessary_inner: 0
> rx11_csum_none: 0
> rx11_xdp_drop: 0
> rx11_xdp_redirect: 0
> rx11_lro_packets: 0
> rx11_lro_bytes: 0
> rx11_ecn_mark: 0
> rx11_removed_vlan_packets: 6868113938
> rx11_wqe_err: 0
> rx11_mpwqe_filler_cqes: 0
> rx11_mpwqe_filler_strides: 0
> rx11_buff_alloc_err: 0
> rx11_cqe_compress_blks: 0
> rx11_cqe_compress_pkts: 0
> rx11_page_reuse: 0
> rx11_cache_reuse: 1948516819
> rx11_cache_full: 17132157
> rx11_cache_empty: 1468413985
> rx11_cache_busy: 17132820
> rx11_cache_waive: 1468407772
> rx11_congst_umr: 0
> rx11_arfs_err: 0
> rx11_xdp_tx_xmit: 0
> rx11_xdp_tx_full: 0
> rx11_xdp_tx_err: 0
> rx11_xdp_tx_cqes: 0
> rx12_packets: 6742955386
> rx12_bytes: 3865747629271
> rx12_csum_complete: 6742955386
> rx12_csum_unnecessary: 0
> rx12_csum_unnecessary_inner: 0
> rx12_csum_none: 0
> rx12_xdp_drop: 0
> rx12_xdp_redirect: 0
> rx12_lro_packets: 0
> rx12_lro_bytes: 0
> rx12_ecn_mark: 0
> rx12_removed_vlan_packets: 6742955386
> rx12_wqe_err: 0
> rx12_mpwqe_filler_cqes: 0
> rx12_mpwqe_filler_strides: 0
> rx12_buff_alloc_err: 0
> rx12_cqe_compress_blks: 0
> rx12_cqe_compress_pkts: 0
> rx12_page_reuse: 0
> rx12_cache_reuse: 30809331
> rx12_cache_full: 3340668106
> rx12_cache_empty: 6656
> rx12_cache_busy: 3340668333
> rx12_cache_waive: 0
> rx12_congst_umr: 0
> rx12_arfs_err: 0
> rx12_xdp_tx_xmit: 0
> rx12_xdp_tx_full: 0
> rx12_xdp_tx_err: 0
> rx12_xdp_tx_cqes: 0
> rx13_packets: 6707028036
> rx13_bytes: 3813462190623
> rx13_csum_complete: 6707028036
> rx13_csum_unnecessary: 0
> rx13_csum_unnecessary_inner: 0
> rx13_csum_none: 0
> rx13_xdp_drop: 0
> rx13_xdp_redirect: 0
> rx13_lro_packets: 0
> rx13_lro_bytes: 0
> rx13_ecn_mark: 0
> rx13_removed_vlan_packets: 6707028036
> rx13_wqe_err: 0
> rx13_mpwqe_filler_cqes: 0
> rx13_mpwqe_filler_strides: 0
> rx13_buff_alloc_err: 0
> rx13_cqe_compress_blks: 0
> rx13_cqe_compress_pkts: 0
> rx13_page_reuse: 0
> rx13_cache_reuse: 14951053
> rx13_cache_full: 3338562710
> rx13_cache_empty: 6656
> rx13_cache_busy: 3338562963
> rx13_cache_waive: 0
> rx13_congst_umr: 0
> rx13_arfs_err: 0
> rx13_xdp_tx_xmit: 0
> rx13_xdp_tx_full: 0
> rx13_xdp_tx_err: 0
> rx13_xdp_tx_cqes: 0
> rx14_packets: 6737074410
> rx14_bytes: 3868905276119
> rx14_csum_complete: 6737074410
> rx14_csum_unnecessary: 0
> rx14_csum_unnecessary_inner: 0
> rx14_csum_none: 0
> rx14_xdp_drop: 0
> rx14_xdp_redirect: 0
> rx14_lro_packets: 0
> rx14_lro_bytes: 0
> rx14_ecn_mark: 0
> rx14_removed_vlan_packets: 6737074410
> rx14_wqe_err: 0
> rx14_mpwqe_filler_cqes: 0
> rx14_mpwqe_filler_strides: 0
> rx14_buff_alloc_err: 0
> rx14_cqe_compress_blks: 0
> rx14_cqe_compress_pkts: 0
> rx14_page_reuse: 0
> rx14_cache_reuse: 967799432
> rx14_cache_full: 982704312
> rx14_cache_empty: 1418039639
> rx14_cache_busy: 982704789
> rx14_cache_waive: 1418033206
> rx14_congst_umr: 0
> rx14_arfs_err: 0
> rx14_xdp_tx_xmit: 0
> rx14_xdp_tx_full: 0
> rx14_xdp_tx_err: 0
> rx14_xdp_tx_cqes: 0
> rx15_packets: 6641887441
> rx15_bytes: 3742874400402
> rx15_csum_complete: 6641887441
> rx15_csum_unnecessary: 0
> rx15_csum_unnecessary_inner: 0
> rx15_csum_none: 0
> rx15_xdp_drop: 0
> rx15_xdp_redirect: 0
> rx15_lro_packets: 0
> rx15_lro_bytes: 0
> rx15_ecn_mark: 0
> rx15_removed_vlan_packets: 6641887441
> rx15_wqe_err: 0
> rx15_mpwqe_filler_cqes: 0
> rx15_mpwqe_filler_strides: 0
> rx15_buff_alloc_err: 0
> rx15_cqe_compress_blks: 0
> rx15_cqe_compress_pkts: 0
> rx15_page_reuse: 0
> rx15_cache_reuse: 1920227538
> rx15_cache_full: 19386129
> rx15_cache_empty: 1381335137
> rx15_cache_busy: 19387693
> rx15_cache_waive: 1381329825
> rx15_congst_umr: 0
> rx15_arfs_err: 0
> rx15_xdp_tx_xmit: 0
> rx15_xdp_tx_full: 0
> rx15_xdp_tx_err: 0
> rx15_xdp_tx_cqes: 0
> rx16_packets: 5420472874
> rx16_bytes: 3079293332581
> rx16_csum_complete: 5420472874
> rx16_csum_unnecessary: 0
> rx16_csum_unnecessary_inner: 0
> rx16_csum_none: 0
> rx16_xdp_drop: 0
> rx16_xdp_redirect: 0
> rx16_lro_packets: 0
> rx16_lro_bytes: 0
> rx16_ecn_mark: 0
> rx16_removed_vlan_packets: 5420472874
> rx16_wqe_err: 0
> rx16_mpwqe_filler_cqes: 0
> rx16_mpwqe_filler_strides: 0
> rx16_buff_alloc_err: 0
> rx16_cqe_compress_blks: 0
> rx16_cqe_compress_pkts: 0
> rx16_page_reuse: 0
> rx16_cache_reuse: 2361079
> rx16_cache_full: 2707875103
> rx16_cache_empty: 6656
> rx16_cache_busy: 2707875349
> rx16_cache_waive: 0
> rx16_congst_umr: 0
> rx16_arfs_err: 0
> rx16_xdp_tx_xmit: 0
> rx16_xdp_tx_full: 0
> rx16_xdp_tx_err: 0
> rx16_xdp_tx_cqes: 0
> rx17_packets: 5428380986
> rx17_bytes: 3080981893118
> rx17_csum_complete: 5428380986
> rx17_csum_unnecessary: 0
> rx17_csum_unnecessary_inner: 0
> rx17_csum_none: 0
> rx17_xdp_drop: 0
> rx17_xdp_redirect: 0
> rx17_lro_packets: 0
> rx17_lro_bytes: 0
> rx17_ecn_mark: 0
> rx17_removed_vlan_packets: 5428380986
> rx17_wqe_err: 0
> rx17_mpwqe_filler_cqes: 0
> rx17_mpwqe_filler_strides: 0
> rx17_buff_alloc_err: 0
> rx17_cqe_compress_blks: 0
> rx17_cqe_compress_pkts: 0
> rx17_page_reuse: 0
> rx17_cache_reuse: 1552266402
> rx17_cache_full: 5947505
> rx17_cache_empty: 1155981856
> rx17_cache_busy: 5948870
> rx17_cache_waive: 1155976345
> rx17_congst_umr: 0
> rx17_arfs_err: 0
> rx17_xdp_tx_xmit: 0
> rx17_xdp_tx_full: 0
> rx17_xdp_tx_err: 0
> rx17_xdp_tx_cqes: 0
> rx18_packets: 5529118410
> rx18_bytes: 3254749573833
> rx18_csum_complete: 5529118410
> rx18_csum_unnecessary: 0
> rx18_csum_unnecessary_inner: 0
> rx18_csum_none: 0
> rx18_xdp_drop: 0
> rx18_xdp_redirect: 0
> rx18_lro_packets: 0
> rx18_lro_bytes: 0
> rx18_ecn_mark: 0
> rx18_removed_vlan_packets: 5529118410
> rx18_wqe_err: 0
> rx18_mpwqe_filler_cqes: 0
> rx18_mpwqe_filler_strides: 0
> rx18_buff_alloc_err: 0
> rx18_cqe_compress_blks: 0
> rx18_cqe_compress_pkts: 0
> rx18_page_reuse: 0
> rx18_cache_reuse: 67438840
> rx18_cache_full: 1536718472
> rx18_cache_empty: 1160408072
> rx18_cache_busy: 1536718932
> rx18_cache_waive: 1160401638
> rx18_congst_umr: 0
> rx18_arfs_err: 0
> rx18_xdp_tx_xmit: 0
> rx18_xdp_tx_full: 0
> rx18_xdp_tx_err: 0
> rx18_xdp_tx_cqes: 0
> rx19_packets: 5449932653
> rx19_bytes: 3148726579411
> rx19_csum_complete: 5449932653
> rx19_csum_unnecessary: 0
> rx19_csum_unnecessary_inner: 0
> rx19_csum_none: 0
> rx19_xdp_drop: 0
> rx19_xdp_redirect: 0
> rx19_lro_packets: 0
> rx19_lro_bytes: 0
> rx19_ecn_mark: 0
> rx19_removed_vlan_packets: 5449932653
> rx19_wqe_err: 0
> rx19_mpwqe_filler_cqes: 0
> rx19_mpwqe_filler_strides: 0
> rx19_buff_alloc_err: 0
> rx19_cqe_compress_blks: 0
> rx19_cqe_compress_pkts: 0
> rx19_page_reuse: 0
> rx19_cache_reuse: 1537841743
> rx19_cache_full: 9920960
> rx19_cache_empty: 1177208938
> rx19_cache_busy: 9922299
> rx19_cache_waive: 1177203401
> rx19_congst_umr: 0
> rx19_arfs_err: 0
> rx19_xdp_tx_xmit: 0
> rx19_xdp_tx_full: 0
> rx19_xdp_tx_err: 0
> rx19_xdp_tx_cqes: 0
> rx20_packets: 5407910071
> rx20_bytes: 3123560861922
> rx20_csum_complete: 5407910071
> rx20_csum_unnecessary: 0
> rx20_csum_unnecessary_inner: 0
> rx20_csum_none: 0
> rx20_xdp_drop: 0
> rx20_xdp_redirect: 0
> rx20_lro_packets: 0
> rx20_lro_bytes: 0
> rx20_ecn_mark: 0
> rx20_removed_vlan_packets: 5407910071
> rx20_wqe_err: 0
> rx20_mpwqe_filler_cqes: 0
> rx20_mpwqe_filler_strides: 0
> rx20_buff_alloc_err: 0
> rx20_cqe_compress_blks: 0
> rx20_cqe_compress_pkts: 0
> rx20_page_reuse: 0
> rx20_cache_reuse: 10255209
> rx20_cache_full: 2693699571
> rx20_cache_empty: 6656
> rx20_cache_busy: 2693699823
> rx20_cache_waive: 0
> rx20_congst_umr: 0
> rx20_arfs_err: 0
> rx20_xdp_tx_xmit: 0
> rx20_xdp_tx_full: 0
> rx20_xdp_tx_err: 0
> rx20_xdp_tx_cqes: 0
> rx21_packets: 5417498508
> rx21_bytes: 3131335892379
> rx21_csum_complete: 5417498508
> rx21_csum_unnecessary: 0
> rx21_csum_unnecessary_inner: 0
> rx21_csum_none: 0
> rx21_xdp_drop: 0
> rx21_xdp_redirect: 0
> rx21_lro_packets: 0
> rx21_lro_bytes: 0
> rx21_ecn_mark: 0
> rx21_removed_vlan_packets: 5417498508
> rx21_wqe_err: 0
> rx21_mpwqe_filler_cqes: 0
> rx21_mpwqe_filler_strides: 0
> rx21_buff_alloc_err: 0
> rx21_cqe_compress_blks: 0
> rx21_cqe_compress_pkts: 0
> rx21_page_reuse: 0
> rx21_cache_reuse: 192662917
> rx21_cache_full: 1374120417
> rx21_cache_empty: 1141972100
> rx21_cache_busy: 1374120891
> rx21_cache_waive: 1141965665
> rx21_congst_umr: 0
> rx21_arfs_err: 0
> rx21_xdp_tx_xmit: 0
> rx21_xdp_tx_full: 0
> rx21_xdp_tx_err: 0
> rx21_xdp_tx_cqes: 0
> rx22_packets: 5613634706
> rx22_bytes: 3240055099058
> rx22_csum_complete: 5613634706
> rx22_csum_unnecessary: 0
> rx22_csum_unnecessary_inner: 0
> rx22_csum_none: 0
> rx22_xdp_drop: 0
> rx22_xdp_redirect: 0
> rx22_lro_packets: 0
> rx22_lro_bytes: 0
> rx22_ecn_mark: 0
> rx22_removed_vlan_packets: 5613634706
> rx22_wqe_err: 0
> rx22_mpwqe_filler_cqes: 0
> rx22_mpwqe_filler_strides: 0
> rx22_buff_alloc_err: 0
> rx22_cqe_compress_blks: 0
> rx22_cqe_compress_pkts: 0
> rx22_page_reuse: 0
> rx22_cache_reuse: 12161531
> rx22_cache_full: 2794655567
> rx22_cache_empty: 6656
> rx22_cache_busy: 2794655821
> rx22_cache_waive: 0
> rx22_congst_umr: 0
> rx22_arfs_err: 0
> rx22_xdp_tx_xmit: 0
> rx22_xdp_tx_full: 0
> rx22_xdp_tx_err: 0
> rx22_xdp_tx_cqes: 0
> rx23_packets: 5389977167
> rx23_bytes: 3054270771559
> rx23_csum_complete: 5389977167
> rx23_csum_unnecessary: 0
> rx23_csum_unnecessary_inner: 0
> rx23_csum_none: 0
> rx23_xdp_drop: 0
> rx23_xdp_redirect: 0
> rx23_lro_packets: 0
> rx23_lro_bytes: 0
> rx23_ecn_mark: 0
> rx23_removed_vlan_packets: 5389977167
> rx23_wqe_err: 0
> rx23_mpwqe_filler_cqes: 0
> rx23_mpwqe_filler_strides: 0
> rx23_buff_alloc_err: 0
> rx23_cqe_compress_blks: 0
> rx23_cqe_compress_pkts: 0
> rx23_page_reuse: 0
> rx23_cache_reuse: 709328
> rx23_cache_full: 2694279000
> rx23_cache_empty: 6656
> rx23_cache_busy: 2694279252
> rx23_cache_waive: 0
> rx23_congst_umr: 0
> rx23_arfs_err: 0
> rx23_xdp_tx_xmit: 0
> rx23_xdp_tx_full: 0
> rx23_xdp_tx_err: 0
> rx23_xdp_tx_cqes: 0
> rx24_packets: 5547561932
> rx24_bytes: 3166602453443
> rx24_csum_complete: 5547561932
> rx24_csum_unnecessary: 0
> rx24_csum_unnecessary_inner: 0
> rx24_csum_none: 0
> rx24_xdp_drop: 0
> rx24_xdp_redirect: 0
> rx24_lro_packets: 0
> rx24_lro_bytes: 0
> rx24_ecn_mark: 0
> rx24_removed_vlan_packets: 5547561932
> rx24_wqe_err: 0
> rx24_mpwqe_filler_cqes: 0
> rx24_mpwqe_filler_strides: 0
> rx24_buff_alloc_err: 0
> rx24_cqe_compress_blks: 0
> rx24_cqe_compress_pkts: 0
> rx24_page_reuse: 0
> rx24_cache_reuse: 57885119
> rx24_cache_full: 1529450077
> rx24_cache_empty: 1186451948
> rx24_cache_busy: 1529450553
> rx24_cache_waive: 1186445515
> rx24_congst_umr: 0
> rx24_arfs_err: 0
> rx24_xdp_tx_xmit: 0
> rx24_xdp_tx_full: 0
> rx24_xdp_tx_err: 0
> rx24_xdp_tx_cqes: 0
> rx25_packets: 5414569326
> rx25_bytes: 3184757708091
> rx25_csum_complete: 5414569326
> rx25_csum_unnecessary: 0
> rx25_csum_unnecessary_inner: 0
> rx25_csum_none: 0
> rx25_xdp_drop: 0
> rx25_xdp_redirect: 0
> rx25_lro_packets: 0
> rx25_lro_bytes: 0
> rx25_ecn_mark: 0
> rx25_removed_vlan_packets: 5414569326
> rx25_wqe_err: 0
> rx25_mpwqe_filler_cqes: 0
> rx25_mpwqe_filler_strides: 0
> rx25_buff_alloc_err: 0
> rx25_cqe_compress_blks: 0
> rx25_cqe_compress_pkts: 0
> rx25_page_reuse: 0
> rx25_cache_reuse: 5080853
> rx25_cache_full: 2702203555
> rx25_cache_empty: 6656
> rx25_cache_busy: 2702203807
> rx25_cache_waive: 0
> rx25_congst_umr: 0
> rx25_arfs_err: 0
> rx25_xdp_tx_xmit: 0
> rx25_xdp_tx_full: 0
> rx25_xdp_tx_err: 0
> rx25_xdp_tx_cqes: 0
> rx26_packets: 5479972151
> rx26_bytes: 3110642276239
> rx26_csum_complete: 5479972151
> rx26_csum_unnecessary: 0
> rx26_csum_unnecessary_inner: 0
> rx26_csum_none: 0
> rx26_xdp_drop: 0
> rx26_xdp_redirect: 0
> rx26_lro_packets: 0
> rx26_lro_bytes: 0
> rx26_ecn_mark: 0
> rx26_removed_vlan_packets: 5479972151
> rx26_wqe_err: 0
> rx26_mpwqe_filler_cqes: 0
> rx26_mpwqe_filler_strides: 0
> rx26_buff_alloc_err: 0
> rx26_cqe_compress_blks: 0
> rx26_cqe_compress_pkts: 0
> rx26_page_reuse: 0
> rx26_cache_reuse: 26543335
> rx26_cache_full: 2713442485
> rx26_cache_empty: 6656
> rx26_cache_busy: 2713442737
> rx26_cache_waive: 0
> rx26_congst_umr: 0
> rx26_arfs_err: 0
> rx26_xdp_tx_xmit: 0
> rx26_xdp_tx_full: 0
> rx26_xdp_tx_err: 0
> rx26_xdp_tx_cqes: 0
> rx27_packets: 5337113900
> rx27_bytes: 3068966906075
> rx27_csum_complete: 5337113900
> rx27_csum_unnecessary: 0
> rx27_csum_unnecessary_inner: 0
> rx27_csum_none: 0
> rx27_xdp_drop: 0
> rx27_xdp_redirect: 0
> rx27_lro_packets: 0
> rx27_lro_bytes: 0
> rx27_ecn_mark: 0
> rx27_removed_vlan_packets: 5337113900
> rx27_wqe_err: 0
> rx27_mpwqe_filler_cqes: 0
> rx27_mpwqe_filler_strides: 0
> rx27_buff_alloc_err: 0
> rx27_cqe_compress_blks: 0
> rx27_cqe_compress_pkts: 0
> rx27_page_reuse: 0
> rx27_cache_reuse: 1539298962
> rx27_cache_full: 10861919
> rx27_cache_empty: 1117173179
> rx27_cache_busy: 12091463
> rx27_cache_waive: 1118395847
> rx27_congst_umr: 0
> rx27_arfs_err: 0
> rx27_xdp_tx_xmit: 0
> rx27_xdp_tx_full: 0
> rx27_xdp_tx_err: 0
> rx27_xdp_tx_cqes: 0
> rx28_packets: 0
> rx28_bytes: 0
> rx28_csum_complete: 0
> rx28_csum_unnecessary: 0
> rx28_csum_unnecessary_inner: 0
> rx28_csum_none: 0
> rx28_xdp_drop: 0
> rx28_xdp_redirect: 0
> rx28_lro_packets: 0
> rx28_lro_bytes: 0
> rx28_ecn_mark: 0
> rx28_removed_vlan_packets: 0
> rx28_wqe_err: 0
> rx28_mpwqe_filler_cqes: 0
> rx28_mpwqe_filler_strides: 0
> rx28_buff_alloc_err: 0
> rx28_cqe_compress_blks: 0
> rx28_cqe_compress_pkts: 0
> rx28_page_reuse: 0
> rx28_cache_reuse: 0
> rx28_cache_full: 0
> rx28_cache_empty: 2560
> rx28_cache_busy: 0
> rx28_cache_waive: 0
> rx28_congst_umr: 0
> rx28_arfs_err: 0
> rx28_xdp_tx_xmit: 0
> rx28_xdp_tx_full: 0
> rx28_xdp_tx_err: 0
> rx28_xdp_tx_cqes: 0
> rx29_packets: 0
> rx29_bytes: 0
> rx29_csum_complete: 0
> rx29_csum_unnecessary: 0
> rx29_csum_unnecessary_inner: 0
> rx29_csum_none: 0
> rx29_xdp_drop: 0
> rx29_xdp_redirect: 0
> rx29_lro_packets: 0
> rx29_lro_bytes: 0
> rx29_ecn_mark: 0
> rx29_removed_vlan_packets: 0
> rx29_wqe_err: 0
> rx29_mpwqe_filler_cqes: 0
> rx29_mpwqe_filler_strides: 0
> rx29_buff_alloc_err: 0
> rx29_cqe_compress_blks: 0
> rx29_cqe_compress_pkts: 0
> rx29_page_reuse: 0
> rx29_cache_reuse: 0
> rx29_cache_full: 0
> rx29_cache_empty: 2560
> rx29_cache_busy: 0
> rx29_cache_waive: 0
> rx29_congst_umr: 0
> rx29_arfs_err: 0
> rx29_xdp_tx_xmit: 0
> rx29_xdp_tx_full: 0
> rx29_xdp_tx_err: 0
> rx29_xdp_tx_cqes: 0
> rx30_packets: 0
> rx30_bytes: 0
> rx30_csum_complete: 0
> rx30_csum_unnecessary: 0
> rx30_csum_unnecessary_inner: 0
> rx30_csum_none: 0
> rx30_xdp_drop: 0
> rx30_xdp_redirect: 0
> rx30_lro_packets: 0
> rx30_lro_bytes: 0
> rx30_ecn_mark: 0
> rx30_removed_vlan_packets: 0
> rx30_wqe_err: 0
> rx30_mpwqe_filler_cqes: 0
> rx30_mpwqe_filler_strides: 0
> rx30_buff_alloc_err: 0
> rx30_cqe_compress_blks: 0
> rx30_cqe_compress_pkts: 0
> rx30_page_reuse: 0
> rx30_cache_reuse: 0
> rx30_cache_full: 0
> rx30_cache_empty: 2560
> rx30_cache_busy: 0
> rx30_cache_waive: 0
> rx30_congst_umr: 0
> rx30_arfs_err: 0
> rx30_xdp_tx_xmit: 0
> rx30_xdp_tx_full: 0
> rx30_xdp_tx_err: 0
> rx30_xdp_tx_cqes: 0
> rx31_packets: 0
> rx31_bytes: 0
> rx31_csum_complete: 0
> rx31_csum_unnecessary: 0
> rx31_csum_unnecessary_inner: 0
> rx31_csum_none: 0
> rx31_xdp_drop: 0
> rx31_xdp_redirect: 0
> rx31_lro_packets: 0
> rx31_lro_bytes: 0
> rx31_ecn_mark: 0
> rx31_removed_vlan_packets: 0
> rx31_wqe_err: 0
> rx31_mpwqe_filler_cqes: 0
> rx31_mpwqe_filler_strides: 0
> rx31_buff_alloc_err: 0
> rx31_cqe_compress_blks: 0
> rx31_cqe_compress_pkts: 0
> rx31_page_reuse: 0
> rx31_cache_reuse: 0
> rx31_cache_full: 0
> rx31_cache_empty: 2560
> rx31_cache_busy: 0
> rx31_cache_waive: 0
> rx31_congst_umr: 0
> rx31_arfs_err: 0
> rx31_xdp_tx_xmit: 0
> rx31_xdp_tx_full: 0
> rx31_xdp_tx_err: 0
> rx31_xdp_tx_cqes: 0
> rx32_packets: 0
> rx32_bytes: 0
> rx32_csum_complete: 0
> rx32_csum_unnecessary: 0
> rx32_csum_unnecessary_inner: 0
> rx32_csum_none: 0
> rx32_xdp_drop: 0
> rx32_xdp_redirect: 0
> rx32_lro_packets: 0
> rx32_lro_bytes: 0
> rx32_ecn_mark: 0
> rx32_removed_vlan_packets: 0
> rx32_wqe_err: 0
> rx32_mpwqe_filler_cqes: 0
> rx32_mpwqe_filler_strides: 0
> rx32_buff_alloc_err: 0
> rx32_cqe_compress_blks: 0
> rx32_cqe_compress_pkts: 0
> rx32_page_reuse: 0
> rx32_cache_reuse: 0
> rx32_cache_full: 0
> rx32_cache_empty: 2560
> rx32_cache_busy: 0
> rx32_cache_waive: 0
> rx32_congst_umr: 0
> rx32_arfs_err: 0
> rx32_xdp_tx_xmit: 0
> rx32_xdp_tx_full: 0
> rx32_xdp_tx_err: 0
> rx32_xdp_tx_cqes: 0
> rx33_packets: 0
> rx33_bytes: 0
> rx33_csum_complete: 0
> rx33_csum_unnecessary: 0
> rx33_csum_unnecessary_inner: 0
> rx33_csum_none: 0
> rx33_xdp_drop: 0
> rx33_xdp_redirect: 0
> rx33_lro_packets: 0
> rx33_lro_bytes: 0
> rx33_ecn_mark: 0
> rx33_removed_vlan_packets: 0
> rx33_wqe_err: 0
> rx33_mpwqe_filler_cqes: 0
> rx33_mpwqe_filler_strides: 0
> rx33_buff_alloc_err: 0
> rx33_cqe_compress_blks: 0
> rx33_cqe_compress_pkts: 0
> rx33_page_reuse: 0
> rx33_cache_reuse: 0
> rx33_cache_full: 0
> rx33_cache_empty: 2560
> rx33_cache_busy: 0
> rx33_cache_waive: 0
> rx33_congst_umr: 0
> rx33_arfs_err: 0
> rx33_xdp_tx_xmit: 0
> rx33_xdp_tx_full: 0
> rx33_xdp_tx_err: 0
> rx33_xdp_tx_cqes: 0
> rx34_packets: 0
> rx34_bytes: 0
> rx34_csum_complete: 0
> rx34_csum_unnecessary: 0
> rx34_csum_unnecessary_inner: 0
> rx34_csum_none: 0
> rx34_xdp_drop: 0
> rx34_xdp_redirect: 0
> rx34_lro_packets: 0
> rx34_lro_bytes: 0
> rx34_ecn_mark: 0
> rx34_removed_vlan_packets: 0
> rx34_wqe_err: 0
> rx34_mpwqe_filler_cqes: 0
> rx34_mpwqe_filler_strides: 0
> rx34_buff_alloc_err: 0
> rx34_cqe_compress_blks: 0
> rx34_cqe_compress_pkts: 0
> rx34_page_reuse: 0
> rx34_cache_reuse: 0
> rx34_cache_full: 0
> rx34_cache_empty: 2560
> rx34_cache_busy: 0
> rx34_cache_waive: 0
> rx34_congst_umr: 0
> rx34_arfs_err: 0
> rx34_xdp_tx_xmit: 0
> rx34_xdp_tx_full: 0
> rx34_xdp_tx_err: 0
> rx34_xdp_tx_cqes: 0
> rx35_packets: 0
> rx35_bytes: 0
> rx35_csum_complete: 0
> rx35_csum_unnecessary: 0
> rx35_csum_unnecessary_inner: 0
> rx35_csum_none: 0
> rx35_xdp_drop: 0
> rx35_xdp_redirect: 0
> rx35_lro_packets: 0
> rx35_lro_bytes: 0
> rx35_ecn_mark: 0
> rx35_removed_vlan_packets: 0
> rx35_wqe_err: 0
> rx35_mpwqe_filler_cqes: 0
> rx35_mpwqe_filler_strides: 0
> rx35_buff_alloc_err: 0
> rx35_cqe_compress_blks: 0
> rx35_cqe_compress_pkts: 0
> rx35_page_reuse: 0
> rx35_cache_reuse: 0
> rx35_cache_full: 0
> rx35_cache_empty: 2560
> rx35_cache_busy: 0
> rx35_cache_waive: 0
> rx35_congst_umr: 0
> rx35_arfs_err: 0
> rx35_xdp_tx_xmit: 0
> rx35_xdp_tx_full: 0
> rx35_xdp_tx_err: 0
> rx35_xdp_tx_cqes: 0
> rx36_packets: 0
> rx36_bytes: 0
> rx36_csum_complete: 0
> rx36_csum_unnecessary: 0
> rx36_csum_unnecessary_inner: 0
> rx36_csum_none: 0
> rx36_xdp_drop: 0
> rx36_xdp_redirect: 0
> rx36_lro_packets: 0
> rx36_lro_bytes: 0
> rx36_ecn_mark: 0
> rx36_removed_vlan_packets: 0
> rx36_wqe_err: 0
> rx36_mpwqe_filler_cqes: 0
> rx36_mpwqe_filler_strides: 0
> rx36_buff_alloc_err: 0
> rx36_cqe_compress_blks: 0
> rx36_cqe_compress_pkts: 0
> rx36_page_reuse: 0
> rx36_cache_reuse: 0
> rx36_cache_full: 0
> rx36_cache_empty: 2560
> rx36_cache_busy: 0
> rx36_cache_waive: 0
> rx36_congst_umr: 0
> rx36_arfs_err: 0
> rx36_xdp_tx_xmit: 0
> rx36_xdp_tx_full: 0
> rx36_xdp_tx_err: 0
> rx36_xdp_tx_cqes: 0
> rx37_packets: 0
> rx37_bytes: 0
> rx37_csum_complete: 0
> rx37_csum_unnecessary: 0
> rx37_csum_unnecessary_inner: 0
> rx37_csum_none: 0
> rx37_xdp_drop: 0
> rx37_xdp_redirect: 0
> rx37_lro_packets: 0
> rx37_lro_bytes: 0
> rx37_ecn_mark: 0
> rx37_removed_vlan_packets: 0
> rx37_wqe_err: 0
> rx37_mpwqe_filler_cqes: 0
> rx37_mpwqe_filler_strides: 0
> rx37_buff_alloc_err: 0
> rx37_cqe_compress_blks: 0
> rx37_cqe_compress_pkts: 0
> rx37_page_reuse: 0
> rx37_cache_reuse: 0
> rx37_cache_full: 0
> rx37_cache_empty: 2560
> rx37_cache_busy: 0
> rx37_cache_waive: 0
> rx37_congst_umr: 0
> rx37_arfs_err: 0
> rx37_xdp_tx_xmit: 0
> rx37_xdp_tx_full: 0
> rx37_xdp_tx_err: 0
> rx37_xdp_tx_cqes: 0
> rx38_packets: 0
> rx38_bytes: 0
> rx38_csum_complete: 0
> rx38_csum_unnecessary: 0
> rx38_csum_unnecessary_inner: 0
> rx38_csum_none: 0
> rx38_xdp_drop: 0
> rx38_xdp_redirect: 0
> rx38_lro_packets: 0
> rx38_lro_bytes: 0
> rx38_ecn_mark: 0
> rx38_removed_vlan_packets: 0
> rx38_wqe_err: 0
> rx38_mpwqe_filler_cqes: 0
> rx38_mpwqe_filler_strides: 0
> rx38_buff_alloc_err: 0
> rx38_cqe_compress_blks: 0
> rx38_cqe_compress_pkts: 0
> rx38_page_reuse: 0
> rx38_cache_reuse: 0
> rx38_cache_full: 0
> rx38_cache_empty: 2560
> rx38_cache_busy: 0
> rx38_cache_waive: 0
> rx38_congst_umr: 0
> rx38_arfs_err: 0
> rx38_xdp_tx_xmit: 0
> rx38_xdp_tx_full: 0
> rx38_xdp_tx_err: 0
> rx38_xdp_tx_cqes: 0
> rx39_packets: 0
> rx39_bytes: 0
> rx39_csum_complete: 0
> rx39_csum_unnecessary: 0
> rx39_csum_unnecessary_inner: 0
> rx39_csum_none: 0
> rx39_xdp_drop: 0
> rx39_xdp_redirect: 0
> rx39_lro_packets: 0
> rx39_lro_bytes: 0
> rx39_ecn_mark: 0
> rx39_removed_vlan_packets: 0
> rx39_wqe_err: 0
> rx39_mpwqe_filler_cqes: 0
> rx39_mpwqe_filler_strides: 0
> rx39_buff_alloc_err: 0
> rx39_cqe_compress_blks: 0
> rx39_cqe_compress_pkts: 0
> rx39_page_reuse: 0
> rx39_cache_reuse: 0
> rx39_cache_full: 0
> rx39_cache_empty: 2560
> rx39_cache_busy: 0
> rx39_cache_waive: 0
> rx39_congst_umr: 0
> rx39_arfs_err: 0
> rx39_xdp_tx_xmit: 0
> rx39_xdp_tx_full: 0
> rx39_xdp_tx_err: 0
> rx39_xdp_tx_cqes: 0
> rx40_packets: 0
> rx40_bytes: 0
> rx40_csum_complete: 0
> rx40_csum_unnecessary: 0
> rx40_csum_unnecessary_inner: 0
> rx40_csum_none: 0
> rx40_xdp_drop: 0
> rx40_xdp_redirect: 0
> rx40_lro_packets: 0
> rx40_lro_bytes: 0
> rx40_ecn_mark: 0
> rx40_removed_vlan_packets: 0
> rx40_wqe_err: 0
> rx40_mpwqe_filler_cqes: 0
> rx40_mpwqe_filler_strides: 0
> rx40_buff_alloc_err: 0
> rx40_cqe_compress_blks: 0
> rx40_cqe_compress_pkts: 0
> rx40_page_reuse: 0
> rx40_cache_reuse: 0
> rx40_cache_full: 0
> rx40_cache_empty: 2560
> rx40_cache_busy: 0
> rx40_cache_waive: 0
> rx40_congst_umr: 0
> rx40_arfs_err: 0
> rx40_xdp_tx_xmit: 0
> rx40_xdp_tx_full: 0
> rx40_xdp_tx_err: 0
> rx40_xdp_tx_cqes: 0
> rx41_packets: 0
> rx41_bytes: 0
> rx41_csum_complete: 0
> rx41_csum_unnecessary: 0
> rx41_csum_unnecessary_inner: 0
> rx41_csum_none: 0
> rx41_xdp_drop: 0
> rx41_xdp_redirect: 0
> rx41_lro_packets: 0
> rx41_lro_bytes: 0
> rx41_ecn_mark: 0
> rx41_removed_vlan_packets: 0
> rx41_wqe_err: 0
> rx41_mpwqe_filler_cqes: 0
> rx41_mpwqe_filler_strides: 0
> rx41_buff_alloc_err: 0
> rx41_cqe_compress_blks: 0
> rx41_cqe_compress_pkts: 0
> rx41_page_reuse: 0
> rx41_cache_reuse: 0
> rx41_cache_full: 0
> rx41_cache_empty: 2560
> rx41_cache_busy: 0
> rx41_cache_waive: 0
> rx41_congst_umr: 0
> rx41_arfs_err: 0
> rx41_xdp_tx_xmit: 0
> rx41_xdp_tx_full: 0
> rx41_xdp_tx_err: 0
> rx41_xdp_tx_cqes: 0
> rx42_packets: 0
> rx42_bytes: 0
> rx42_csum_complete: 0
> rx42_csum_unnecessary: 0
> rx42_csum_unnecessary_inner: 0
> rx42_csum_none: 0
> rx42_xdp_drop: 0
> rx42_xdp_redirect: 0
> rx42_lro_packets: 0
> rx42_lro_bytes: 0
> rx42_ecn_mark: 0
> rx42_removed_vlan_packets: 0
> rx42_wqe_err: 0
> rx42_mpwqe_filler_cqes: 0
> rx42_mpwqe_filler_strides: 0
> rx42_buff_alloc_err: 0
> rx42_cqe_compress_blks: 0
> rx42_cqe_compress_pkts: 0
> rx42_page_reuse: 0
> rx42_cache_reuse: 0
> rx42_cache_full: 0
> rx42_cache_empty: 2560
> rx42_cache_busy: 0
> rx42_cache_waive: 0
> rx42_congst_umr: 0
> rx42_arfs_err: 0
> rx42_xdp_tx_xmit: 0
> rx42_xdp_tx_full: 0
> rx42_xdp_tx_err: 0
> rx42_xdp_tx_cqes: 0
> rx43_packets: 0
> rx43_bytes: 0
> rx43_csum_complete: 0
> rx43_csum_unnecessary: 0
> rx43_csum_unnecessary_inner: 0
> rx43_csum_none: 0
> rx43_xdp_drop: 0
> rx43_xdp_redirect: 0
> rx43_lro_packets: 0
> rx43_lro_bytes: 0
> rx43_ecn_mark: 0
> rx43_removed_vlan_packets: 0
> rx43_wqe_err: 0
> rx43_mpwqe_filler_cqes: 0
> rx43_mpwqe_filler_strides: 0
> rx43_buff_alloc_err: 0
> rx43_cqe_compress_blks: 0
> rx43_cqe_compress_pkts: 0
> rx43_page_reuse: 0
> rx43_cache_reuse: 0
> rx43_cache_full: 0
> rx43_cache_empty: 2560
> rx43_cache_busy: 0
> rx43_cache_waive: 0
> rx43_congst_umr: 0
> rx43_arfs_err: 0
> rx43_xdp_tx_xmit: 0
> rx43_xdp_tx_full: 0
> rx43_xdp_tx_err: 0
> rx43_xdp_tx_cqes: 0
> rx44_packets: 0
> rx44_bytes: 0
> rx44_csum_complete: 0
> rx44_csum_unnecessary: 0
> rx44_csum_unnecessary_inner: 0
> rx44_csum_none: 0
> rx44_xdp_drop: 0
> rx44_xdp_redirect: 0
> rx44_lro_packets: 0
> rx44_lro_bytes: 0
> rx44_ecn_mark: 0
> rx44_removed_vlan_packets: 0
> rx44_wqe_err: 0
> rx44_mpwqe_filler_cqes: 0
> rx44_mpwqe_filler_strides: 0
> rx44_buff_alloc_err: 0
> rx44_cqe_compress_blks: 0
> rx44_cqe_compress_pkts: 0
> rx44_page_reuse: 0
> rx44_cache_reuse: 0
> rx44_cache_full: 0
> rx44_cache_empty: 2560
> rx44_cache_busy: 0
> rx44_cache_waive: 0
> rx44_congst_umr: 0
> rx44_arfs_err: 0
> rx44_xdp_tx_xmit: 0
> rx44_xdp_tx_full: 0
> rx44_xdp_tx_err: 0
> rx44_xdp_tx_cqes: 0
> rx45_packets: 0
> rx45_bytes: 0
> rx45_csum_complete: 0
> rx45_csum_unnecessary: 0
> rx45_csum_unnecessary_inner: 0
> rx45_csum_none: 0
> rx45_xdp_drop: 0
> rx45_xdp_redirect: 0
> rx45_lro_packets: 0
> rx45_lro_bytes: 0
> rx45_ecn_mark: 0
> rx45_removed_vlan_packets: 0
> rx45_wqe_err: 0
> rx45_mpwqe_filler_cqes: 0
> rx45_mpwqe_filler_strides: 0
> rx45_buff_alloc_err: 0
> rx45_cqe_compress_blks: 0
> rx45_cqe_compress_pkts: 0
> rx45_page_reuse: 0
> rx45_cache_reuse: 0
> rx45_cache_full: 0
> rx45_cache_empty: 2560
> rx45_cache_busy: 0
> rx45_cache_waive: 0
> rx45_congst_umr: 0
> rx45_arfs_err: 0
> rx45_xdp_tx_xmit: 0
> rx45_xdp_tx_full: 0
> rx45_xdp_tx_err: 0
> rx45_xdp_tx_cqes: 0
> rx46_packets: 0
> rx46_bytes: 0
> rx46_csum_complete: 0
> rx46_csum_unnecessary: 0
> rx46_csum_unnecessary_inner: 0
> rx46_csum_none: 0
> rx46_xdp_drop: 0
> rx46_xdp_redirect: 0
> rx46_lro_packets: 0
> rx46_lro_bytes: 0
> rx46_ecn_mark: 0
> rx46_removed_vlan_packets: 0
> rx46_wqe_err: 0
> rx46_mpwqe_filler_cqes: 0
> rx46_mpwqe_filler_strides: 0
> rx46_buff_alloc_err: 0
> rx46_cqe_compress_blks: 0
> rx46_cqe_compress_pkts: 0
> rx46_page_reuse: 0
> rx46_cache_reuse: 0
> rx46_cache_full: 0
> rx46_cache_empty: 2560
> rx46_cache_busy: 0
> rx46_cache_waive: 0
> rx46_congst_umr: 0
> rx46_arfs_err: 0
> rx46_xdp_tx_xmit: 0
> rx46_xdp_tx_full: 0
> rx46_xdp_tx_err: 0
> rx46_xdp_tx_cqes: 0
> rx47_packets: 0
> rx47_bytes: 0
> rx47_csum_complete: 0
> rx47_csum_unnecessary: 0
> rx47_csum_unnecessary_inner: 0
> rx47_csum_none: 0
> rx47_xdp_drop: 0
> rx47_xdp_redirect: 0
> rx47_lro_packets: 0
> rx47_lro_bytes: 0
> rx47_ecn_mark: 0
> rx47_removed_vlan_packets: 0
> rx47_wqe_err: 0
> rx47_mpwqe_filler_cqes: 0
> rx47_mpwqe_filler_strides: 0
> rx47_buff_alloc_err: 0
> rx47_cqe_compress_blks: 0
> rx47_cqe_compress_pkts: 0
> rx47_page_reuse: 0
> rx47_cache_reuse: 0
> rx47_cache_full: 0
> rx47_cache_empty: 2560
> rx47_cache_busy: 0
> rx47_cache_waive: 0
> rx47_congst_umr: 0
> rx47_arfs_err: 0
> rx47_xdp_tx_xmit: 0
> rx47_xdp_tx_full: 0
> rx47_xdp_tx_err: 0
> rx47_xdp_tx_cqes: 0
> rx48_packets: 0
> rx48_bytes: 0
> rx48_csum_complete: 0
> rx48_csum_unnecessary: 0
> rx48_csum_unnecessary_inner: 0
> rx48_csum_none: 0
> rx48_xdp_drop: 0
> rx48_xdp_redirect: 0
> rx48_lro_packets: 0
> rx48_lro_bytes: 0
> rx48_ecn_mark: 0
> rx48_removed_vlan_packets: 0
> rx48_wqe_err: 0
> rx48_mpwqe_filler_cqes: 0
> rx48_mpwqe_filler_strides: 0
> rx48_buff_alloc_err: 0
> rx48_cqe_compress_blks: 0
> rx48_cqe_compress_pkts: 0
> rx48_page_reuse: 0
> rx48_cache_reuse: 0
> rx48_cache_full: 0
> rx48_cache_empty: 2560
> rx48_cache_busy: 0
> rx48_cache_waive: 0
> rx48_congst_umr: 0
> rx48_arfs_err: 0
> rx48_xdp_tx_xmit: 0
> rx48_xdp_tx_full: 0
> rx48_xdp_tx_err: 0
> rx48_xdp_tx_cqes: 0
> rx49_packets: 0
> rx49_bytes: 0
> rx49_csum_complete: 0
> rx49_csum_unnecessary: 0
> rx49_csum_unnecessary_inner: 0
> rx49_csum_none: 0
> rx49_xdp_drop: 0
> rx49_xdp_redirect: 0
> rx49_lro_packets: 0
> rx49_lro_bytes: 0
> rx49_ecn_mark: 0
> rx49_removed_vlan_packets: 0
> rx49_wqe_err: 0
> rx49_mpwqe_filler_cqes: 0
> rx49_mpwqe_filler_strides: 0
> rx49_buff_alloc_err: 0
> rx49_cqe_compress_blks: 0
> rx49_cqe_compress_pkts: 0
> rx49_page_reuse: 0
> rx49_cache_reuse: 0
> rx49_cache_full: 0
> rx49_cache_empty: 2560
> rx49_cache_busy: 0
> rx49_cache_waive: 0
> rx49_congst_umr: 0
> rx49_arfs_err: 0
> rx49_xdp_tx_xmit: 0
> rx49_xdp_tx_full: 0
> rx49_xdp_tx_err: 0
> rx49_xdp_tx_cqes: 0
> rx50_packets: 0
> rx50_bytes: 0
> rx50_csum_complete: 0
> rx50_csum_unnecessary: 0
> rx50_csum_unnecessary_inner: 0
> rx50_csum_none: 0
> rx50_xdp_drop: 0
> rx50_xdp_redirect: 0
> rx50_lro_packets: 0
> rx50_lro_bytes: 0
> rx50_ecn_mark: 0
> rx50_removed_vlan_packets: 0
> rx50_wqe_err: 0
> rx50_mpwqe_filler_cqes: 0
> rx50_mpwqe_filler_strides: 0
> rx50_buff_alloc_err: 0
> rx50_cqe_compress_blks: 0
> rx50_cqe_compress_pkts: 0
> rx50_page_reuse: 0
> rx50_cache_reuse: 0
> rx50_cache_full: 0
> rx50_cache_empty: 2560
> rx50_cache_busy: 0
> rx50_cache_waive: 0
> rx50_congst_umr: 0
> rx50_arfs_err: 0
> rx50_xdp_tx_xmit: 0
> rx50_xdp_tx_full: 0
> rx50_xdp_tx_err: 0
> rx50_xdp_tx_cqes: 0
> rx51_packets: 0
> rx51_bytes: 0
> rx51_csum_complete: 0
> rx51_csum_unnecessary: 0
> rx51_csum_unnecessary_inner: 0
> rx51_csum_none: 0
> rx51_xdp_drop: 0
> rx51_xdp_redirect: 0
> rx51_lro_packets: 0
> rx51_lro_bytes: 0
> rx51_ecn_mark: 0
> rx51_removed_vlan_packets: 0
> rx51_wqe_err: 0
> rx51_mpwqe_filler_cqes: 0
> rx51_mpwqe_filler_strides: 0
> rx51_buff_alloc_err: 0
> rx51_cqe_compress_blks: 0
> rx51_cqe_compress_pkts: 0
> rx51_page_reuse: 0
> rx51_cache_reuse: 0
> rx51_cache_full: 0
> rx51_cache_empty: 2560
> rx51_cache_busy: 0
> rx51_cache_waive: 0
> rx51_congst_umr: 0
> rx51_arfs_err: 0
> rx51_xdp_tx_xmit: 0
> rx51_xdp_tx_full: 0
> rx51_xdp_tx_err: 0
> rx51_xdp_tx_cqes: 0
> rx52_packets: 0
> rx52_bytes: 0
> rx52_csum_complete: 0
> rx52_csum_unnecessary: 0
> rx52_csum_unnecessary_inner: 0
> rx52_csum_none: 0
> rx52_xdp_drop: 0
> rx52_xdp_redirect: 0
> rx52_lro_packets: 0
> rx52_lro_bytes: 0
> rx52_ecn_mark: 0
> rx52_removed_vlan_packets: 0
> rx52_wqe_err: 0
> rx52_mpwqe_filler_cqes: 0
> rx52_mpwqe_filler_strides: 0
> rx52_buff_alloc_err: 0
> rx52_cqe_compress_blks: 0
> rx52_cqe_compress_pkts: 0
> rx52_page_reuse: 0
> rx52_cache_reuse: 0
> rx52_cache_full: 0
> rx52_cache_empty: 2560
> rx52_cache_busy: 0
> rx52_cache_waive: 0
> rx52_congst_umr: 0
> rx52_arfs_err: 0
> rx52_xdp_tx_xmit: 0
> rx52_xdp_tx_full: 0
> rx52_xdp_tx_err: 0
> rx52_xdp_tx_cqes: 0
> rx53_packets: 0
> rx53_bytes: 0
> rx53_csum_complete: 0
> rx53_csum_unnecessary: 0
> rx53_csum_unnecessary_inner: 0
> rx53_csum_none: 0
> rx53_xdp_drop: 0
> rx53_xdp_redirect: 0
> rx53_lro_packets: 0
> rx53_lro_bytes: 0
> rx53_ecn_mark: 0
> rx53_removed_vlan_packets: 0
> rx53_wqe_err: 0
> rx53_mpwqe_filler_cqes: 0
> rx53_mpwqe_filler_strides: 0
> rx53_buff_alloc_err: 0
> rx53_cqe_compress_blks: 0
> rx53_cqe_compress_pkts: 0
> rx53_page_reuse: 0
> rx53_cache_reuse: 0
> rx53_cache_full: 0
> rx53_cache_empty: 2560
> rx53_cache_busy: 0
> rx53_cache_waive: 0
> rx53_congst_umr: 0
> rx53_arfs_err: 0
> rx53_xdp_tx_xmit: 0
> rx53_xdp_tx_full: 0
> rx53_xdp_tx_err: 0
> rx53_xdp_tx_cqes: 0
> rx54_packets: 0
> rx54_bytes: 0
> rx54_csum_complete: 0
> rx54_csum_unnecessary: 0
> rx54_csum_unnecessary_inner: 0
> rx54_csum_none: 0
> rx54_xdp_drop: 0
> rx54_xdp_redirect: 0
> rx54_lro_packets: 0
> rx54_lro_bytes: 0
> rx54_ecn_mark: 0
> rx54_removed_vlan_packets: 0
> rx54_wqe_err: 0
> rx54_mpwqe_filler_cqes: 0
> rx54_mpwqe_filler_strides: 0
> rx54_buff_alloc_err: 0
> rx54_cqe_compress_blks: 0
> rx54_cqe_compress_pkts: 0
> rx54_page_reuse: 0
> rx54_cache_reuse: 0
> rx54_cache_full: 0
> rx54_cache_empty: 2560
> rx54_cache_busy: 0
> rx54_cache_waive: 0
> rx54_congst_umr: 0
> rx54_arfs_err: 0
> rx54_xdp_tx_xmit: 0
> rx54_xdp_tx_full: 0
> rx54_xdp_tx_err: 0
> rx54_xdp_tx_cqes: 0
> rx55_packets: 0
> rx55_bytes: 0
> rx55_csum_complete: 0
> rx55_csum_unnecessary: 0
> rx55_csum_unnecessary_inner: 0
> rx55_csum_none: 0
> rx55_xdp_drop: 0
> rx55_xdp_redirect: 0
> rx55_lro_packets: 0
> rx55_lro_bytes: 0
> rx55_ecn_mark: 0
> rx55_removed_vlan_packets: 0
> rx55_wqe_err: 0
> rx55_mpwqe_filler_cqes: 0
> rx55_mpwqe_filler_strides: 0
> rx55_buff_alloc_err: 0
> rx55_cqe_compress_blks: 0
> rx55_cqe_compress_pkts: 0
> rx55_page_reuse: 0
> rx55_cache_reuse: 0
> rx55_cache_full: 0
> rx55_cache_empty: 2560
> rx55_cache_busy: 0
> rx55_cache_waive: 0
> rx55_congst_umr: 0
> rx55_arfs_err: 0
> rx55_xdp_tx_xmit: 0
> rx55_xdp_tx_full: 0
> rx55_xdp_tx_err: 0
> rx55_xdp_tx_cqes: 0
> tx0_packets: 5868971166
> tx0_bytes: 7384241881537
> tx0_tso_packets: 1005089669
> tx0_tso_bytes: 5138882499687
> tx0_tso_inner_packets: 0
> tx0_tso_inner_bytes: 0
> tx0_csum_partial: 1405330470
> tx0_csum_partial_inner: 0
> tx0_added_vlan_packets: 3247061022
> tx0_nop: 83925216
> tx0_csum_none: 1841730552
> tx0_stopped: 0
> tx0_dropped: 0
> tx0_xmit_more: 29664303
> tx0_recover: 0
> tx0_cqes: 3217398842
> tx0_wake: 0
> tx0_cqe_err: 0
> tx1_packets: 5599378674
> tx1_bytes: 7272236466962
> tx1_tso_packets: 1024612268
> tx1_tso_bytes: 5244192050917
> tx1_tso_inner_packets: 0
> tx1_tso_inner_bytes: 0
> tx1_csum_partial: 1438007932
> tx1_csum_partial_inner: 0
> tx1_added_vlan_packets: 2919765857
> tx1_nop: 79661231
> tx1_csum_none: 1481757925
> tx1_stopped: 0
> tx1_dropped: 0
> tx1_xmit_more: 29485355
> tx1_recover: 0
> tx1_cqes: 2890282176
> tx1_wake: 0
> tx1_cqe_err: 0
> tx2_packets: 5413821094
> tx2_bytes: 7033951631334
> tx2_tso_packets: 1002868589
> tx2_tso_bytes: 5089549008985
> tx2_tso_inner_packets: 0
> tx2_tso_inner_bytes: 0
> tx2_csum_partial: 1404186175
> tx2_csum_partial_inner: 0
> tx2_added_vlan_packets: 2822670460
> tx2_nop: 77115408
> tx2_csum_none: 1418484285
> tx2_stopped: 0
> tx2_dropped: 0
> tx2_xmit_more: 29321129
> tx2_recover: 0
> tx2_cqes: 2793351019
> tx2_wake: 0
> tx2_cqe_err: 0
> tx3_packets: 5479609727
> tx3_bytes: 7116904107659
> tx3_tso_packets: 1002992639
> tx3_tso_bytes: 5154225081979
> tx3_tso_inner_packets: 0
> tx3_tso_inner_bytes: 0
> tx3_csum_partial: 1415739849
> tx3_csum_partial_inner: 0
> tx3_added_vlan_packets: 2842823811
> tx3_nop: 78060813
> tx3_csum_none: 1427083971
> tx3_stopped: 0
> tx3_dropped: 0
> tx3_xmit_more: 28575040
> tx3_recover: 0
> tx3_cqes: 2814250785
> tx3_wake: 0
> tx3_cqe_err: 0
> tx4_packets: 5508297397
> tx4_bytes: 7127659369902
> tx4_tso_packets: 1007356432
> tx4_tso_bytes: 5145975736034
> tx4_tso_inner_packets: 0
> tx4_tso_inner_bytes: 0
> tx4_csum_partial: 1411271000
> tx4_csum_partial_inner: 0
> tx4_added_vlan_packets: 2882086825
> tx4_nop: 78433610
> tx4_csum_none: 1470815825
> tx4_stopped: 0
> tx4_dropped: 0
> tx4_xmit_more: 28632444
> tx4_recover: 0
> tx4_cqes: 2853456464
> tx4_wake: 0
> tx4_cqe_err: 0
> tx5_packets: 5513864156
> tx5_bytes: 7165864145517
> tx5_tso_packets: 1014046485
> tx5_tso_bytes: 5192635614477
> tx5_tso_inner_packets: 0
> tx5_tso_inner_bytes: 0
> tx5_csum_partial: 1420810473
> tx5_csum_partial_inner: 0
> tx5_added_vlan_packets: 2861370556
> tx5_nop: 78481355
> tx5_csum_none: 1440560083
> tx5_stopped: 0
> tx5_dropped: 0
> tx5_xmit_more: 28222467
> tx5_recover: 0
> tx5_cqes: 2833149758
> tx5_wake: 0
> tx5_cqe_err: 0
> tx6_packets: 5560724761
> tx6_bytes: 7210309972086
> tx6_tso_packets: 994050514
> tx6_tso_bytes: 5171393741595
> tx6_tso_inner_packets: 0
> tx6_tso_inner_bytes: 0
> tx6_csum_partial: 1414303265
> tx6_csum_partial_inner: 0
> tx6_added_vlan_packets: 2905794177
> tx6_nop: 79353318
> tx6_csum_none: 1491490912
> tx6_stopped: 0
> tx6_dropped: 0
> tx6_xmit_more: 31246664
> tx6_recover: 0
> tx6_cqes: 2874549217
> tx6_wake: 0
> tx6_cqe_err: 0
> tx7_packets: 5557594170
> tx7_bytes: 7223138778685
> tx7_tso_packets: 1013475396
> tx7_tso_bytes: 5241530065484
> tx7_tso_inner_packets: 0
> tx7_tso_inner_bytes: 0
> tx7_csum_partial: 1438604314
> tx7_csum_partial_inner: 0
> tx7_added_vlan_packets: 2873917552
> tx7_nop: 79057059
> tx7_csum_none: 1435313239
> tx7_stopped: 0
> tx7_dropped: 0
> tx7_xmit_more: 29258761
> tx7_recover: 0
> tx7_cqes: 2844660578
> tx7_wake: 0
> tx7_cqe_err: 0
> tx8_packets: 5521254733
> tx8_bytes: 7208043146297
> tx8_tso_packets: 1014670801
> tx8_tso_bytes: 5185842447246
> tx8_tso_inner_packets: 0
> tx8_tso_inner_bytes: 0
> tx8_csum_partial: 1431631562
> tx8_csum_partial_inner: 0
> tx8_added_vlan_packets: 2872641129
> tx8_nop: 78545776
> tx8_csum_none: 1441009567
> tx8_stopped: 0
> tx8_dropped: 0
> tx8_xmit_more: 29106291
> tx8_recover: 0
> tx8_cqes: 2843536748
> tx8_wake: 0
> tx8_cqe_err: 0
> tx9_packets: 5528889957
> tx9_bytes: 7191793816058
> tx9_tso_packets: 1015955476
> tx9_tso_bytes: 5207232047828
> tx9_tso_inner_packets: 0
> tx9_tso_inner_bytes: 0
> tx9_csum_partial: 1421266796
> tx9_csum_partial_inner: 0
> tx9_added_vlan_packets: 2869523921
> tx9_nop: 78586218
> tx9_csum_none: 1448257125
> tx9_stopped: 0
> tx9_dropped: 0
> tx9_xmit_more: 29483347
> tx9_recover: 0
> tx9_cqes: 2840042245
> tx9_wake: 0
> tx9_cqe_err: 0
> tx10_packets: 5556351222
> tx10_bytes: 7254798330757
> tx10_tso_packets: 1028554460
> tx10_tso_bytes: 5246179615774
> tx10_tso_inner_packets: 0
> tx10_tso_inner_bytes: 0
> tx10_csum_partial: 1430459021
> tx10_csum_partial_inner: 0
> tx10_added_vlan_packets: 2881683382
> tx10_nop: 79139584
> tx10_csum_none: 1451224361
> tx10_stopped: 0
> tx10_dropped: 0
> tx10_xmit_more: 29217190
> tx10_recover: 0
> tx10_cqes: 2852467898
> tx10_wake: 0
> tx10_cqe_err: 0
> tx11_packets: 5455631854
> tx11_bytes: 7061121713772
> tx11_tso_packets: 992133383
> tx11_tso_bytes: 5089419722682
> tx11_tso_inner_packets: 0
> tx11_tso_inner_bytes: 0
> tx11_csum_partial: 1395542033
> tx11_csum_partial_inner: 0
> tx11_added_vlan_packets: 2852589093
> tx11_nop: 77799857
> tx11_csum_none: 1457047060
> tx11_stopped: 0
> tx11_dropped: 0
> tx11_xmit_more: 29559927
> tx11_recover: 0
> tx11_cqes: 2823031110
> tx11_wake: 0
> tx11_cqe_err: 0
> tx12_packets: 5488286808
> tx12_bytes: 7137087569303
> tx12_tso_packets: 1006435537
> tx12_tso_bytes: 5163371416750
> tx12_tso_inner_packets: 0
> tx12_tso_inner_bytes: 0
> tx12_csum_partial: 1414799411
> tx12_csum_partial_inner: 0
> tx12_added_vlan_packets: 2841679543
> tx12_nop: 78387039
> tx12_csum_none: 1426880132
> tx12_stopped: 0
> tx12_dropped: 0
> tx12_xmit_more: 28607526
> tx12_recover: 0
> tx12_cqes: 2813073557
> tx12_wake: 0
> tx12_cqe_err: 0
> tx13_packets: 5594132290
> tx13_bytes: 7251106284829
> tx13_tso_packets: 1035172061
> tx13_tso_bytes: 5251200286298
> tx13_tso_inner_packets: 0
> tx13_tso_inner_bytes: 0
> tx13_csum_partial: 1443665981
> tx13_csum_partial_inner: 0
> tx13_added_vlan_packets: 2916604799
> tx13_nop: 79670465
> tx13_csum_none: 1472938818
> tx13_stopped: 0
> tx13_dropped: 0
> tx13_xmit_more: 27797067
> tx13_recover: 0
> tx13_cqes: 2888809352
> tx13_wake: 0
> tx13_cqe_err: 0
> tx14_packets: 5548790952
> tx14_bytes: 7194211868411
> tx14_tso_packets: 1021015561
> tx14_tso_bytes: 5231483708869
> tx14_tso_inner_packets: 0
> tx14_tso_inner_bytes: 0
> tx14_csum_partial: 1427711576
> tx14_csum_partial_inner: 0
> tx14_added_vlan_packets: 2875288572
> tx14_nop: 78900224
> tx14_csum_none: 1447576996
> tx14_stopped: 0
> tx14_dropped: 0
> tx14_xmit_more: 30003496
> tx14_recover: 0
> tx14_cqes: 2845286732
> tx14_wake: 0
> tx14_cqe_err: 0
> tx15_packets: 5609310963
> tx15_bytes: 7271380831798
> tx15_tso_packets: 1027830118
> tx15_tso_bytes: 5229697431506
> tx15_tso_inner_packets: 0
> tx15_tso_inner_bytes: 0
> tx15_csum_partial: 1429209941
> tx15_csum_partial_inner: 0
> tx15_added_vlan_packets: 2940315402
> tx15_nop: 79950883
> tx15_csum_none: 1511105462
> tx15_stopped: 0
> tx15_dropped: 0
> tx15_xmit_more: 28820740
> tx15_recover: 0
> tx15_cqes: 2911496633
> tx15_wake: 0
> tx15_cqe_err: 0
> tx16_packets: 4465363036
> tx16_bytes: 5769771803704
> tx16_tso_packets: 817101913
> tx16_tso_bytes: 4180172833814
> tx16_tso_inner_packets: 0
> tx16_tso_inner_bytes: 0
> tx16_csum_partial: 1136731404
> tx16_csum_partial_inner: 0
> tx16_added_vlan_packets: 2332178232
> tx16_nop: 63458573
> tx16_csum_none: 1195446828
> tx16_stopped: 0
> tx16_dropped: 0
> tx16_xmit_more: 23756254
> tx16_recover: 0
> tx16_cqes: 2308423025
> tx16_wake: 0
> tx16_cqe_err: 0
> tx17_packets: 4380386348
> tx17_bytes: 5708702994526
> tx17_tso_packets: 813638023
> tx17_tso_bytes: 4130806014947
> tx17_tso_inner_packets: 0
> tx17_tso_inner_bytes: 0
> tx17_csum_partial: 1133007164
> tx17_csum_partial_inner: 0
> tx17_added_vlan_packets: 2277314787
> tx17_nop: 62377372
> tx17_csum_none: 1144307623
> tx17_stopped: 0
> tx17_dropped: 0
> tx17_xmit_more: 23731361
> tx17_recover: 0
> tx17_cqes: 2253584638
> tx17_wake: 0
> tx17_cqe_err: 0
> tx18_packets: 4450359743
> tx18_bytes: 5758968674820
> tx18_tso_packets: 815791601
> tx18_tso_bytes: 4179942688909
> tx18_tso_inner_packets: 0
> tx18_tso_inner_bytes: 0
> tx18_csum_partial: 1137649257
> tx18_csum_partial_inner: 0
> tx18_added_vlan_packets: 2314556550
> tx18_nop: 63271085
> tx18_csum_none: 1176907293
> tx18_stopped: 0
> tx18_dropped: 0
> tx18_xmit_more: 23055770
> tx18_recover: 0
> tx18_cqes: 2291501928
> tx18_wake: 0
> tx18_cqe_err: 0
> tx19_packets: 4596064378
> tx19_bytes: 5916675706535
> tx19_tso_packets: 825788649
> tx19_tso_bytes: 4208046929921
> tx19_tso_inner_packets: 0
> tx19_tso_inner_bytes: 0
> tx19_csum_partial: 1150666569
> tx19_csum_partial_inner: 0
> tx19_added_vlan_packets: 2450567026
> tx19_nop: 65468504
> tx19_csum_none: 1299900457
> tx19_stopped: 0
> tx19_dropped: 0
> tx19_xmit_more: 23846250
> tx19_recover: 0
> tx19_cqes: 2426722127
> tx19_wake: 0
> tx19_cqe_err: 0
> tx20_packets: 4424935388
> tx20_bytes: 5757631205901
> tx20_tso_packets: 804875006
> tx20_tso_bytes: 4156262736109
> tx20_tso_inner_packets: 0
> tx20_tso_inner_bytes: 0
> tx20_csum_partial: 1134144916
> tx20_csum_partial_inner: 0
> tx20_added_vlan_packets: 2294839665
> tx20_nop: 63023986
> tx20_csum_none: 1160694749
> tx20_stopped: 0
> tx20_dropped: 0
> tx20_xmit_more: 23393201
> tx20_recover: 0
> tx20_cqes: 2271447623
> tx20_wake: 0
> tx20_cqe_err: 0
> tx21_packets: 4595062285
> tx21_bytes: 5958671993467
> tx21_tso_packets: 821936215
> tx21_tso_bytes: 4187977870684
> tx21_tso_inner_packets: 0
> tx21_tso_inner_bytes: 0
> tx21_csum_partial: 1143339787
> tx21_csum_partial_inner: 0
> tx21_added_vlan_packets: 2457167412
> tx21_nop: 65697763
> tx21_csum_none: 1313827625
> tx21_stopped: 0
> tx21_dropped: 0
> tx21_xmit_more: 23858345
> tx21_recover: 0
> tx21_cqes: 2433310348
> tx21_wake: 0
> tx21_cqe_err: 0
> tx22_packets: 4664446513
> tx22_bytes: 5931429292082
> tx22_tso_packets: 814457881
> tx22_tso_bytes: 4148607956533
> tx22_tso_inner_packets: 0
> tx22_tso_inner_bytes: 0
> tx22_csum_partial: 1127284783
> tx22_csum_partial_inner: 0
> tx22_added_vlan_packets: 2548650146
> tx22_nop: 66299909
> tx22_csum_none: 1421365363
> tx22_stopped: 0
> tx22_dropped: 0
> tx22_xmit_more: 23800911
> tx22_recover: 0
> tx22_cqes: 2524850415
> tx22_wake: 0
> tx22_cqe_err: 0
> tx23_packets: 4416221747
> tx23_bytes: 5721472587985
> tx23_tso_packets: 823538520
> tx23_tso_bytes: 4163520218617
> tx23_tso_inner_packets: 0
> tx23_tso_inner_bytes: 0
> tx23_csum_partial: 1135996006
> tx23_csum_partial_inner: 0
> tx23_added_vlan_packets: 2292404120
> tx23_nop: 62709432
> tx23_csum_none: 1156408114
> tx23_stopped: 0
> tx23_dropped: 0
> tx23_xmit_more: 22299889
> tx23_recover: 0
> tx23_cqes: 2270105487
> tx23_wake: 0
> tx23_cqe_err: 0
> tx24_packets: 4420014824
> tx24_bytes: 5740767318521
> tx24_tso_packets: 820838072
> tx24_tso_bytes: 4183722948422
> tx24_tso_inner_packets: 0
> tx24_tso_inner_bytes: 0
> tx24_csum_partial: 1138070059
> tx24_csum_partial_inner: 0
> tx24_added_vlan_packets: 2289043946
> tx24_nop: 62797341
> tx24_csum_none: 1150973887
> tx24_stopped: 0
> tx24_dropped: 0
> tx24_xmit_more: 22744690
> tx24_recover: 0
> tx24_cqes: 2266300568
> tx24_wake: 0
> tx24_cqe_err: 0
> tx25_packets: 4413225545
> tx25_bytes: 5716162617155
> tx25_tso_packets: 808274341
> tx25_tso_bytes: 4138408857714
> tx25_tso_inner_packets: 0
> tx25_tso_inner_bytes: 0
> tx25_csum_partial: 1134587898
> tx25_csum_partial_inner: 0
> tx25_added_vlan_packets: 2297149310
> tx25_nop: 62958238
> tx25_csum_none: 1162561412
> tx25_stopped: 0
> tx25_dropped: 0
> tx25_xmit_more: 24463552
> tx25_recover: 0
> tx25_cqes: 2272686971
> tx25_wake: 0
> tx25_cqe_err: 0
> tx26_packets: 4524907591
> tx26_bytes: 5865394280699
> tx26_tso_packets: 807270022
> tx26_tso_bytes: 4148754705317
> tx26_tso_inner_packets: 0
> tx26_tso_inner_bytes: 0
> tx26_csum_partial: 1130306933
> tx26_csum_partial_inner: 0
> tx26_added_vlan_packets: 2402682460
> tx26_nop: 64474322
> tx26_csum_none: 1272375527
> tx26_stopped: 1
> tx26_dropped: 0
> tx26_xmit_more: 23316186
> tx26_recover: 0
> tx26_cqes: 2379367502
> tx26_wake: 1
> tx26_cqe_err: 0
> tx27_packets: 4376114969
> tx27_bytes: 5683551238304
> tx27_tso_packets: 809344829
> tx27_tso_bytes: 4124331859270
> tx27_tso_inner_packets: 0
> tx27_tso_inner_bytes: 0
> tx27_csum_partial: 1124954937
> tx27_csum_partial_inner: 0
> tx27_added_vlan_packets: 2267871300
> tx27_nop: 62213214
> tx27_csum_none: 1142916363
> tx27_stopped: 0
> tx27_dropped: 0
> tx27_xmit_more: 23369974
> tx27_recover: 0
> tx27_cqes: 2244502686
> tx27_wake: 0
> tx27_cqe_err: 0
> tx28_packets: 3
> tx28_bytes: 266
> tx28_tso_packets: 0
> tx28_tso_bytes: 0
> tx28_tso_inner_packets: 0
> tx28_tso_inner_bytes: 0
> tx28_csum_partial: 0
> tx28_csum_partial_inner: 0
> tx28_added_vlan_packets: 0
> tx28_nop: 0
> tx28_csum_none: 3
> tx28_stopped: 0
> tx28_dropped: 0
> tx28_xmit_more: 0
> tx28_recover: 0
> tx28_cqes: 3
> tx28_wake: 0
> tx28_cqe_err: 0
> tx29_packets: 0
> tx29_bytes: 0
> tx29_tso_packets: 0
> tx29_tso_bytes: 0
> tx29_tso_inner_packets: 0
> tx29_tso_inner_bytes: 0
> tx29_csum_partial: 0
> tx29_csum_partial_inner: 0
> tx29_added_vlan_packets: 0
> tx29_nop: 0
> tx29_csum_none: 0
> tx29_stopped: 0
> tx29_dropped: 0
> tx29_xmit_more: 0
> tx29_recover: 0
> tx29_cqes: 0
> tx29_wake: 0
> tx29_cqe_err: 0
> tx30_packets: 0
> tx30_bytes: 0
> tx30_tso_packets: 0
> tx30_tso_bytes: 0
> tx30_tso_inner_packets: 0
> tx30_tso_inner_bytes: 0
> tx30_csum_partial: 0
> tx30_csum_partial_inner: 0
> tx30_added_vlan_packets: 0
> tx30_nop: 0
> tx30_csum_none: 0
> tx30_stopped: 0
> tx30_dropped: 0
> tx30_xmit_more: 0
> tx30_recover: 0
> tx30_cqes: 0
> tx30_wake: 0
> tx30_cqe_err: 0
> tx31_packets: 0
> tx31_bytes: 0
> tx31_tso_packets: 0
> tx31_tso_bytes: 0
> tx31_tso_inner_packets: 0
> tx31_tso_inner_bytes: 0
> tx31_csum_partial: 0
> tx31_csum_partial_inner: 0
> tx31_added_vlan_packets: 0
> tx31_nop: 0
> tx31_csum_none: 0
> tx31_stopped: 0
> tx31_dropped: 0
> tx31_xmit_more: 0
> tx31_recover: 0
> tx31_cqes: 0
> tx31_wake: 0
> tx31_cqe_err: 0
> tx32_packets: 0
> tx32_bytes: 0
> tx32_tso_packets: 0
> tx32_tso_bytes: 0
> tx32_tso_inner_packets: 0
> tx32_tso_inner_bytes: 0
> tx32_csum_partial: 0
> tx32_csum_partial_inner: 0
> tx32_added_vlan_packets: 0
> tx32_nop: 0
> tx32_csum_none: 0
> tx32_stopped: 0
> tx32_dropped: 0
> tx32_xmit_more: 0
> tx32_recover: 0
> tx32_cqes: 0
> tx32_wake: 0
> tx32_cqe_err: 0
> tx33_packets: 0
> tx33_bytes: 0
> tx33_tso_packets: 0
> tx33_tso_bytes: 0
> tx33_tso_inner_packets: 0
> tx33_tso_inner_bytes: 0
> tx33_csum_partial: 0
> tx33_csum_partial_inner: 0
> tx33_added_vlan_packets: 0
> tx33_nop: 0
> tx33_csum_none: 0
> tx33_stopped: 0
> tx33_dropped: 0
> tx33_xmit_more: 0
> tx33_recover: 0
> tx33_cqes: 0
> tx33_wake: 0
> tx33_cqe_err: 0
> tx34_packets: 0
> tx34_bytes: 0
> tx34_tso_packets: 0
> tx34_tso_bytes: 0
> tx34_tso_inner_packets: 0
> tx34_tso_inner_bytes: 0
> tx34_csum_partial: 0
> tx34_csum_partial_inner: 0
> tx34_added_vlan_packets: 0
> tx34_nop: 0
> tx34_csum_none: 0
> tx34_stopped: 0
> tx34_dropped: 0
> tx34_xmit_more: 0
> tx34_recover: 0
> tx34_cqes: 0
> tx34_wake: 0
> tx34_cqe_err: 0
> tx35_packets: 0
> tx35_bytes: 0
> tx35_tso_packets: 0
> tx35_tso_bytes: 0
> tx35_tso_inner_packets: 0
> tx35_tso_inner_bytes: 0
> tx35_csum_partial: 0
> tx35_csum_partial_inner: 0
> tx35_added_vlan_packets: 0
> tx35_nop: 0
> tx35_csum_none: 0
> tx35_stopped: 0
> tx35_dropped: 0
> tx35_xmit_more: 0
> tx35_recover: 0
> tx35_cqes: 0
> tx35_wake: 0
> tx35_cqe_err: 0
> tx36_packets: 0
> tx36_bytes: 0
> tx36_tso_packets: 0
> tx36_tso_bytes: 0
> tx36_tso_inner_packets: 0
> tx36_tso_inner_bytes: 0
> tx36_csum_partial: 0
> tx36_csum_partial_inner: 0
> tx36_added_vlan_packets: 0
> tx36_nop: 0
> tx36_csum_none: 0
> tx36_stopped: 0
> tx36_dropped: 0
> tx36_xmit_more: 0
> tx36_recover: 0
> tx36_cqes: 0
> tx36_wake: 0
> tx36_cqe_err: 0
> tx37_packets: 0
> tx37_bytes: 0
> tx37_tso_packets: 0
> tx37_tso_bytes: 0
> tx37_tso_inner_packets: 0
> tx37_tso_inner_bytes: 0
> tx37_csum_partial: 0
> tx37_csum_partial_inner: 0
> tx37_added_vlan_packets: 0
> tx37_nop: 0
> tx37_csum_none: 0
> tx37_stopped: 0
> tx37_dropped: 0
> tx37_xmit_more: 0
> tx37_recover: 0
> tx37_cqes: 0
> tx37_wake: 0
> tx37_cqe_err: 0
> tx38_packets: 0
> tx38_bytes: 0
> tx38_tso_packets: 0
> tx38_tso_bytes: 0
> tx38_tso_inner_packets: 0
> tx38_tso_inner_bytes: 0
> tx38_csum_partial: 0
> tx38_csum_partial_inner: 0
> tx38_added_vlan_packets: 0
> tx38_nop: 0
> tx38_csum_none: 0
> tx38_stopped: 0
> tx38_dropped: 0
> tx38_xmit_more: 0
> tx38_recover: 0
> tx38_cqes: 0
> tx38_wake: 0
> tx38_cqe_err: 0
> tx39_packets: 0
> tx39_bytes: 0
> tx39_tso_packets: 0
> tx39_tso_bytes: 0
> tx39_tso_inner_packets: 0
> tx39_tso_inner_bytes: 0
> tx39_csum_partial: 0
> tx39_csum_partial_inner: 0
> tx39_added_vlan_packets: 0
> tx39_nop: 0
> tx39_csum_none: 0
> tx39_stopped: 0
> tx39_dropped: 0
> tx39_xmit_more: 0
> tx39_recover: 0
> tx39_cqes: 0
> tx39_wake: 0
> tx39_cqe_err: 0
> tx40_packets: 0
> tx40_bytes: 0
> tx40_tso_packets: 0
> tx40_tso_bytes: 0
> tx40_tso_inner_packets: 0
> tx40_tso_inner_bytes: 0
> tx40_csum_partial: 0
> tx40_csum_partial_inner: 0
> tx40_added_vlan_packets: 0
> tx40_nop: 0
> tx40_csum_none: 0
> tx40_stopped: 0
> tx40_dropped: 0
> tx40_xmit_more: 0
> tx40_recover: 0
> tx40_cqes: 0
> tx40_wake: 0
> tx40_cqe_err: 0
> tx41_packets: 0
> tx41_bytes: 0
> tx41_tso_packets: 0
> tx41_tso_bytes: 0
> tx41_tso_inner_packets: 0
> tx41_tso_inner_bytes: 0
> tx41_csum_partial: 0
> tx41_csum_partial_inner: 0
> tx41_added_vlan_packets: 0
> tx41_nop: 0
> tx41_csum_none: 0
> tx41_stopped: 0
> tx41_dropped: 0
> tx41_xmit_more: 0
> tx41_recover: 0
> tx41_cqes: 0
> tx41_wake: 0
> tx41_cqe_err: 0
> tx42_packets: 0
> tx42_bytes: 0
> tx42_tso_packets: 0
> tx42_tso_bytes: 0
> tx42_tso_inner_packets: 0
> tx42_tso_inner_bytes: 0
> tx42_csum_partial: 0
> tx42_csum_partial_inner: 0
> tx42_added_vlan_packets: 0
> tx42_nop: 0
> tx42_csum_none: 0
> tx42_stopped: 0
> tx42_dropped: 0
> tx42_xmit_more: 0
> tx42_recover: 0
> tx42_cqes: 0
> tx42_wake: 0
> tx42_cqe_err: 0
> tx43_packets: 0
> tx43_bytes: 0
> tx43_tso_packets: 0
> tx43_tso_bytes: 0
> tx43_tso_inner_packets: 0
> tx43_tso_inner_bytes: 0
> tx43_csum_partial: 0
> tx43_csum_partial_inner: 0
> tx43_added_vlan_packets: 0
> tx43_nop: 0
> tx43_csum_none: 0
> tx43_stopped: 0
> tx43_dropped: 0
> tx43_xmit_more: 0
> tx43_recover: 0
> tx43_cqes: 0
> tx43_wake: 0
> tx43_cqe_err: 0
> tx44_packets: 0
> tx44_bytes: 0
> tx44_tso_packets: 0
> tx44_tso_bytes: 0
> tx44_tso_inner_packets: 0
> tx44_tso_inner_bytes: 0
> tx44_csum_partial: 0
> tx44_csum_partial_inner: 0
> tx44_added_vlan_packets: 0
> tx44_nop: 0
> tx44_csum_none: 0
> tx44_stopped: 0
> tx44_dropped: 0
> tx44_xmit_more: 0
> tx44_recover: 0
> tx44_cqes: 0
> tx44_wake: 0
> tx44_cqe_err: 0
> tx45_packets: 0
> tx45_bytes: 0
> tx45_tso_packets: 0
> tx45_tso_bytes: 0
> tx45_tso_inner_packets: 0
> tx45_tso_inner_bytes: 0
> tx45_csum_partial: 0
> tx45_csum_partial_inner: 0
> tx45_added_vlan_packets: 0
> tx45_nop: 0
> tx45_csum_none: 0
> tx45_stopped: 0
> tx45_dropped: 0
> tx45_xmit_more: 0
> tx45_recover: 0
> tx45_cqes: 0
> tx45_wake: 0
> tx45_cqe_err: 0
> tx46_packets: 0
> tx46_bytes: 0
> tx46_tso_packets: 0
> tx46_tso_bytes: 0
> tx46_tso_inner_packets: 0
> tx46_tso_inner_bytes: 0
> tx46_csum_partial: 0
> tx46_csum_partial_inner: 0
> tx46_added_vlan_packets: 0
> tx46_nop: 0
> tx46_csum_none: 0
> tx46_stopped: 0
> tx46_dropped: 0
> tx46_xmit_more: 0
> tx46_recover: 0
> tx46_cqes: 0
> tx46_wake: 0
> tx46_cqe_err: 0
> tx47_packets: 0
> tx47_bytes: 0
> tx47_tso_packets: 0
> tx47_tso_bytes: 0
> tx47_tso_inner_packets: 0
> tx47_tso_inner_bytes: 0
> tx47_csum_partial: 0
> tx47_csum_partial_inner: 0
> tx47_added_vlan_packets: 0
> tx47_nop: 0
> tx47_csum_none: 0
> tx47_stopped: 0
> tx47_dropped: 0
> tx47_xmit_more: 0
> tx47_recover: 0
> tx47_cqes: 0
> tx47_wake: 0
> tx47_cqe_err: 0
> tx48_packets: 0
> tx48_bytes: 0
> tx48_tso_packets: 0
> tx48_tso_bytes: 0
> tx48_tso_inner_packets: 0
> tx48_tso_inner_bytes: 0
> tx48_csum_partial: 0
> tx48_csum_partial_inner: 0
> tx48_added_vlan_packets: 0
> tx48_nop: 0
> tx48_csum_none: 0
> tx48_stopped: 0
> tx48_dropped: 0
> tx48_xmit_more: 0
> tx48_recover: 0
> tx48_cqes: 0
> tx48_wake: 0
> tx48_cqe_err: 0
> tx49_packets: 0
> tx49_bytes: 0
> tx49_tso_packets: 0
> tx49_tso_bytes: 0
> tx49_tso_inner_packets: 0
> tx49_tso_inner_bytes: 0
> tx49_csum_partial: 0
> tx49_csum_partial_inner: 0
> tx49_added_vlan_packets: 0
> tx49_nop: 0
> tx49_csum_none: 0
> tx49_stopped: 0
> tx49_dropped: 0
> tx49_xmit_more: 0
> tx49_recover: 0
> tx49_cqes: 0
> tx49_wake: 0
> tx49_cqe_err: 0
> tx50_packets: 0
> tx50_bytes: 0
> tx50_tso_packets: 0
> tx50_tso_bytes: 0
> tx50_tso_inner_packets: 0
> tx50_tso_inner_bytes: 0
> tx50_csum_partial: 0
> tx50_csum_partial_inner: 0
> tx50_added_vlan_packets: 0
> tx50_nop: 0
> tx50_csum_none: 0
> tx50_stopped: 0
> tx50_dropped: 0
> tx50_xmit_more: 0
> tx50_recover: 0
> tx50_cqes: 0
> tx50_wake: 0
> tx50_cqe_err: 0
> tx51_packets: 0
> tx51_bytes: 0
> tx51_tso_packets: 0
> tx51_tso_bytes: 0
> tx51_tso_inner_packets: 0
> tx51_tso_inner_bytes: 0
> tx51_csum_partial: 0
> tx51_csum_partial_inner: 0
> tx51_added_vlan_packets: 0
> tx51_nop: 0
> tx51_csum_none: 0
> tx51_stopped: 0
> tx51_dropped: 0
> tx51_xmit_more: 0
> tx51_recover: 0
> tx51_cqes: 0
> tx51_wake: 0
> tx51_cqe_err: 0
> tx52_packets: 0
> tx52_bytes: 0
> tx52_tso_packets: 0
> tx52_tso_bytes: 0
> tx52_tso_inner_packets: 0
> tx52_tso_inner_bytes: 0
> tx52_csum_partial: 0
> tx52_csum_partial_inner: 0
> tx52_added_vlan_packets: 0
> tx52_nop: 0
> tx52_csum_none: 0
> tx52_stopped: 0
> tx52_dropped: 0
> tx52_xmit_more: 0
> tx52_recover: 0
> tx52_cqes: 0
> tx52_wake: 0
> tx52_cqe_err: 0
> tx53_packets: 0
> tx53_bytes: 0
> tx53_tso_packets: 0
> tx53_tso_bytes: 0
> tx53_tso_inner_packets: 0
> tx53_tso_inner_bytes: 0
> tx53_csum_partial: 0
> tx53_csum_partial_inner: 0
> tx53_added_vlan_packets: 0
> tx53_nop: 0
> tx53_csum_none: 0
> tx53_stopped: 0
> tx53_dropped: 0
> tx53_xmit_more: 0
> tx53_recover: 0
> tx53_cqes: 0
> tx53_wake: 0
> tx53_cqe_err: 0
> tx54_packets: 0
> tx54_bytes: 0
> tx54_tso_packets: 0
> tx54_tso_bytes: 0
> tx54_tso_inner_packets: 0
> tx54_tso_inner_bytes: 0
> tx54_csum_partial: 0
> tx54_csum_partial_inner: 0
> tx54_added_vlan_packets: 0
> tx54_nop: 0
> tx54_csum_none: 0
> tx54_stopped: 0
> tx54_dropped: 0
> tx54_xmit_more: 0
> tx54_recover: 0
> tx54_cqes: 0
> tx54_wake: 0
> tx54_cqe_err: 0
> tx55_packets: 0
> tx55_bytes: 0
> tx55_tso_packets: 0
> tx55_tso_bytes: 0
> tx55_tso_inner_packets: 0
> tx55_tso_inner_bytes: 0
> tx55_csum_partial: 0
> tx55_csum_partial_inner: 0
> tx55_added_vlan_packets: 0
> tx55_nop: 0
> tx55_csum_none: 0
> tx55_stopped: 0
> tx55_dropped: 0
> tx55_xmit_more: 0
> tx55_recover: 0
> tx55_cqes: 0
> tx55_wake: 0
> tx55_cqe_err: 0
> tx0_xdp_xmit: 0
> tx0_xdp_full: 0
> tx0_xdp_err: 0
> tx0_xdp_cqes: 0
> tx1_xdp_xmit: 0
> tx1_xdp_full: 0
> tx1_xdp_err: 0
> tx1_xdp_cqes: 0
> tx2_xdp_xmit: 0
> tx2_xdp_full: 0
> tx2_xdp_err: 0
> tx2_xdp_cqes: 0
> tx3_xdp_xmit: 0
> tx3_xdp_full: 0
> tx3_xdp_err: 0
> tx3_xdp_cqes: 0
> tx4_xdp_xmit: 0
> tx4_xdp_full: 0
> tx4_xdp_err: 0
> tx4_xdp_cqes: 0
> tx5_xdp_xmit: 0
> tx5_xdp_full: 0
> tx5_xdp_err: 0
> tx5_xdp_cqes: 0
> tx6_xdp_xmit: 0
> tx6_xdp_full: 0
> tx6_xdp_err: 0
> tx6_xdp_cqes: 0
> tx7_xdp_xmit: 0
> tx7_xdp_full: 0
> tx7_xdp_err: 0
> tx7_xdp_cqes: 0
> tx8_xdp_xmit: 0
> tx8_xdp_full: 0
> tx8_xdp_err: 0
> tx8_xdp_cqes: 0
> tx9_xdp_xmit: 0
> tx9_xdp_full: 0
> tx9_xdp_err: 0
> tx9_xdp_cqes: 0
> tx10_xdp_xmit: 0
> tx10_xdp_full: 0
> tx10_xdp_err: 0
> tx10_xdp_cqes: 0
> tx11_xdp_xmit: 0
> tx11_xdp_full: 0
> tx11_xdp_err: 0
> tx11_xdp_cqes: 0
> tx12_xdp_xmit: 0
> tx12_xdp_full: 0
> tx12_xdp_err: 0
> tx12_xdp_cqes: 0
> tx13_xdp_xmit: 0
> tx13_xdp_full: 0
> tx13_xdp_err: 0
> tx13_xdp_cqes: 0
> tx14_xdp_xmit: 0
> tx14_xdp_full: 0
> tx14_xdp_err: 0
> tx14_xdp_cqes: 0
> tx15_xdp_xmit: 0
> tx15_xdp_full: 0
> tx15_xdp_err: 0
> tx15_xdp_cqes: 0
> tx16_xdp_xmit: 0
> tx16_xdp_full: 0
> tx16_xdp_err: 0
> tx16_xdp_cqes: 0
> tx17_xdp_xmit: 0
> tx17_xdp_full: 0
> tx17_xdp_err: 0
> tx17_xdp_cqes: 0
> tx18_xdp_xmit: 0
> tx18_xdp_full: 0
> tx18_xdp_err: 0
> tx18_xdp_cqes: 0
> tx19_xdp_xmit: 0
> tx19_xdp_full: 0
> tx19_xdp_err: 0
> tx19_xdp_cqes: 0
> tx20_xdp_xmit: 0
> tx20_xdp_full: 0
> tx20_xdp_err: 0
> tx20_xdp_cqes: 0
> tx21_xdp_xmit: 0
> tx21_xdp_full: 0
> tx21_xdp_err: 0
> tx21_xdp_cqes: 0
> tx22_xdp_xmit: 0
> tx22_xdp_full: 0
> tx22_xdp_err: 0
> tx22_xdp_cqes: 0
> tx23_xdp_xmit: 0
> tx23_xdp_full: 0
> tx23_xdp_err: 0
> tx23_xdp_cqes: 0
> tx24_xdp_xmit: 0
> tx24_xdp_full: 0
> tx24_xdp_err: 0
> tx24_xdp_cqes: 0
> tx25_xdp_xmit: 0
> tx25_xdp_full: 0
> tx25_xdp_err: 0
> tx25_xdp_cqes: 0
> tx26_xdp_xmit: 0
> tx26_xdp_full: 0
> tx26_xdp_err: 0
> tx26_xdp_cqes: 0
> tx27_xdp_xmit: 0
> tx27_xdp_full: 0
> tx27_xdp_err: 0
> tx27_xdp_cqes: 0
> tx28_xdp_xmit: 0
> tx28_xdp_full: 0
> tx28_xdp_err: 0
> tx28_xdp_cqes: 0
> tx29_xdp_xmit: 0
> tx29_xdp_full: 0
> tx29_xdp_err: 0
> tx29_xdp_cqes: 0
> tx30_xdp_xmit: 0
> tx30_xdp_full: 0
> tx30_xdp_err: 0
> tx30_xdp_cqes: 0
> tx31_xdp_xmit: 0
> tx31_xdp_full: 0
> tx31_xdp_err: 0
> tx31_xdp_cqes: 0
> tx32_xdp_xmit: 0
> tx32_xdp_full: 0
> tx32_xdp_err: 0
> tx32_xdp_cqes: 0
> tx33_xdp_xmit: 0
> tx33_xdp_full: 0
> tx33_xdp_err: 0
> tx33_xdp_cqes: 0
> tx34_xdp_xmit: 0
> tx34_xdp_full: 0
> tx34_xdp_err: 0
> tx34_xdp_cqes: 0
> tx35_xdp_xmit: 0
> tx35_xdp_full: 0
> tx35_xdp_err: 0
> tx35_xdp_cqes: 0
> tx36_xdp_xmit: 0
> tx36_xdp_full: 0
> tx36_xdp_err: 0
> tx36_xdp_cqes: 0
> tx37_xdp_xmit: 0
> tx37_xdp_full: 0
> tx37_xdp_err: 0
> tx37_xdp_cqes: 0
> tx38_xdp_xmit: 0
> tx38_xdp_full: 0
> tx38_xdp_err: 0
> tx38_xdp_cqes: 0
> tx39_xdp_xmit: 0
> tx39_xdp_full: 0
> tx39_xdp_err: 0
> tx39_xdp_cqes: 0
> tx40_xdp_xmit: 0
> tx40_xdp_full: 0
> tx40_xdp_err: 0
> tx40_xdp_cqes: 0
> tx41_xdp_xmit: 0
> tx41_xdp_full: 0
> tx41_xdp_err: 0
> tx41_xdp_cqes: 0
> tx42_xdp_xmit: 0
> tx42_xdp_full: 0
> tx42_xdp_err: 0
> tx42_xdp_cqes: 0
> tx43_xdp_xmit: 0
> tx43_xdp_full: 0
> tx43_xdp_err: 0
> tx43_xdp_cqes: 0
> tx44_xdp_xmit: 0
> tx44_xdp_full: 0
> tx44_xdp_err: 0
> tx44_xdp_cqes: 0
> tx45_xdp_xmit: 0
> tx45_xdp_full: 0
> tx45_xdp_err: 0
> tx45_xdp_cqes: 0
> tx46_xdp_xmit: 0
> tx46_xdp_full: 0
> tx46_xdp_err: 0
> tx46_xdp_cqes: 0
> tx47_xdp_xmit: 0
> tx47_xdp_full: 0
> tx47_xdp_err: 0
> tx47_xdp_cqes: 0
> tx48_xdp_xmit: 0
> tx48_xdp_full: 0
> tx48_xdp_err: 0
> tx48_xdp_cqes: 0
> tx49_xdp_xmit: 0
> tx49_xdp_full: 0
> tx49_xdp_err: 0
> tx49_xdp_cqes: 0
> tx50_xdp_xmit: 0
> tx50_xdp_full: 0
> tx50_xdp_err: 0
> tx50_xdp_cqes: 0
> tx51_xdp_xmit: 0
> tx51_xdp_full: 0
> tx51_xdp_err: 0
> tx51_xdp_cqes: 0
> tx52_xdp_xmit: 0
> tx52_xdp_full: 0
> tx52_xdp_err: 0
> tx52_xdp_cqes: 0
> tx53_xdp_xmit: 0
> tx53_xdp_full: 0
> tx53_xdp_err: 0
> tx53_xdp_cqes: 0
> tx54_xdp_xmit: 0
> tx54_xdp_full: 0
> tx54_xdp_err: 0
> tx54_xdp_cqes: 0
> tx55_xdp_xmit: 0
> tx55_xdp_full: 0
> tx55_xdp_err: 0
> tx55_xdp_cqes: 0
>
> ethtool -S enp175s0f0
> NIC statistics:
> rx_packets: 141574897253
> rx_bytes: 184445040406258
> tx_packets: 172569543894
> tx_bytes: 99486882076365
> tx_tso_packets: 9367664195
> tx_tso_bytes: 56435233992948
> tx_tso_inner_packets: 0
> tx_tso_inner_bytes: 0
> tx_added_vlan_packets: 141297671626
> tx_nop: 2102916272
> rx_lro_packets: 0
> rx_lro_bytes: 0
> rx_ecn_mark: 0
> rx_removed_vlan_packets: 141574897252
> rx_csum_unnecessary: 0
> rx_csum_none: 23135854
> rx_csum_complete: 141551761398
> rx_csum_unnecessary_inner: 0
> rx_xdp_drop: 0
> rx_xdp_redirect: 0
> rx_xdp_tx_xmit: 0
> rx_xdp_tx_full: 0
> rx_xdp_tx_err: 0
> rx_xdp_tx_cqe: 0
> tx_csum_none: 127934791664
It is a good idea to look into this, tx is not requesting hw tx
csumming for a lot of packets, maybe you are wasting a lot of cpu on
calculating csum, or maybe this is just the rx csum complete..
> tx_csum_partial: 13362879974
> tx_csum_partial_inner: 0
> tx_queue_stopped: 232561
TX queues are stalling, could be an indentation for the pcie
bottelneck.
> tx_queue_dropped: 0
> tx_xmit_more: 1266021946
> tx_recover: 0
> tx_cqes: 140031716469
> tx_queue_wake: 232561
> tx_udp_seg_rem: 0
> tx_cqe_err: 0
> tx_xdp_xmit: 0
> tx_xdp_full: 0
> tx_xdp_err: 0
> tx_xdp_cqes: 0
> rx_wqe_err: 0
> rx_mpwqe_filler_cqes: 0
> rx_mpwqe_filler_strides: 0
> rx_buff_alloc_err: 0
> rx_cqe_compress_blks: 0
> rx_cqe_compress_pkts: 0
> rx_page_reuse: 0
> rx_cache_reuse: 16625975793
> rx_cache_full: 54161465914
> rx_cache_empty: 258048
> rx_cache_busy: 54161472735
> rx_cache_waive: 0
> rx_congst_umr: 0
> rx_arfs_err: 0
> ch_events: 40572621887
> ch_poll: 40885650979
> ch_arm: 40429276692
> ch_aff_change: 0
> ch_eq_rearm: 0
> rx_out_of_buffer: 2791690
> rx_if_down_packets: 74
> rx_vport_unicast_packets: 141843476308
> rx_vport_unicast_bytes: 185421265403318
> tx_vport_unicast_packets: 172569484005
> tx_vport_unicast_bytes: 100019940094298
> rx_vport_multicast_packets: 85122935
> rx_vport_multicast_bytes: 5761316431
> tx_vport_multicast_packets: 6452
> tx_vport_multicast_bytes: 643540
> rx_vport_broadcast_packets: 22423624
> rx_vport_broadcast_bytes: 1390127090
> tx_vport_broadcast_packets: 22024
> tx_vport_broadcast_bytes: 1321440
> rx_vport_rdma_unicast_packets: 0
> rx_vport_rdma_unicast_bytes: 0
> tx_vport_rdma_unicast_packets: 0
> tx_vport_rdma_unicast_bytes: 0
> rx_vport_rdma_multicast_packets: 0
> rx_vport_rdma_multicast_bytes: 0
> tx_vport_rdma_multicast_packets: 0
> tx_vport_rdma_multicast_bytes: 0
> tx_packets_phy: 172569501577
> rx_packets_phy: 142871314588
> rx_crc_errors_phy: 0
> tx_bytes_phy: 100710212814151
> rx_bytes_phy: 187209224289564
> tx_multicast_phy: 6452
> tx_broadcast_phy: 22024
> rx_multicast_phy: 85122933
> rx_broadcast_phy: 22423623
> rx_in_range_len_errors_phy: 2
> rx_out_of_range_len_phy: 0
> rx_oversize_pkts_phy: 0
> rx_symbol_err_phy: 0
> tx_mac_control_phy: 0
> rx_mac_control_phy: 0
> rx_unsupported_op_phy: 0
> rx_pause_ctrl_phy: 0
> tx_pause_ctrl_phy: 0
> rx_discards_phy: 920161423
Ok, this port seem to be suffering more, RX is congested, maybe due to
the pcie bottleneck.
> tx_discards_phy: 0
> tx_errors_phy: 0
> rx_undersize_pkts_phy: 0
> rx_fragments_phy: 0
> rx_jabbers_phy: 0
> rx_64_bytes_phy: 412006326
> rx_65_to_127_bytes_phy: 11934371453
> rx_128_to_255_bytes_phy: 3415281165
> rx_256_to_511_bytes_phy: 2072955511
> rx_512_to_1023_bytes_phy: 2415393005
> rx_1024_to_1518_bytes_phy: 72182391608
> rx_1519_to_2047_bytes_phy: 50438902587
> rx_2048_to_4095_bytes_phy: 0
> rx_4096_to_8191_bytes_phy: 0
> rx_8192_to_10239_bytes_phy: 0
> link_down_events_phy: 0
> rx_pcs_symbol_err_phy: 0
> rx_corrected_bits_phy: 0
> rx_pci_signal_integrity: 0
> tx_pci_signal_integrity: 48
> rx_prio0_bytes: 186709842592642
> rx_prio0_packets: 141481966007
> tx_prio0_bytes: 100710171118138
> tx_prio0_packets: 172569437949
> rx_prio1_bytes: 492288152326
> rx_prio1_packets: 385996045
> tx_prio1_bytes: 0
> tx_prio1_packets: 0
> rx_prio2_bytes: 22119952
> rx_prio2_packets: 70788
> tx_prio2_bytes: 0
> tx_prio2_packets: 0
> rx_prio3_bytes: 546141102
> rx_prio3_packets: 681608
> tx_prio3_bytes: 0
> tx_prio3_packets: 0
> rx_prio4_bytes: 14665067
> rx_prio4_packets: 29486
> tx_prio4_bytes: 0
> tx_prio4_packets: 0
> rx_prio5_bytes: 158862504
> rx_prio5_packets: 965307
> tx_prio5_bytes: 0
> tx_prio5_packets: 0
> rx_prio6_bytes: 669337783
> rx_prio6_packets: 1475775
> tx_prio6_bytes: 0
> tx_prio6_packets: 0
> rx_prio7_bytes: 5623481349
> rx_prio7_packets: 79926412
> tx_prio7_bytes: 0
> tx_prio7_packets: 0
> module_unplug: 0
> module_bus_stuck: 0
> module_high_temp: 0
> module_bad_shorted: 0
> ch0_events: 1446162630
> ch0_poll: 1463312972
> ch0_arm: 1440728278
> ch0_aff_change: 0
> ch0_eq_rearm: 0
> ch1_events: 1384301405
> ch1_poll: 1399210915
> ch1_arm: 1378636486
> ch1_aff_change: 0
> ch1_eq_rearm: 0
> ch2_events: 1382788887
> ch2_poll: 1397231470
> ch2_arm: 1377058116
> ch2_aff_change: 0
> ch2_eq_rearm: 0
> ch3_events: 1461956995
> ch3_poll: 1475553146
> ch3_arm: 1456571625
> ch3_aff_change: 0
> ch3_eq_rearm: 0
> ch4_events: 1497359109
> ch4_poll: 1511021037
> ch4_arm: 1491733757
> ch4_aff_change: 0
> ch4_eq_rearm: 0
> ch5_events: 1387736262
> ch5_poll: 1400964615
> ch5_arm: 1382382834
> ch5_aff_change: 0
> ch5_eq_rearm: 0
> ch6_events: 1376772405
> ch6_poll: 1390851449
> ch6_arm: 1371551764
> ch6_aff_change: 0
> ch6_eq_rearm: 0
> ch7_events: 1431271514
> ch7_poll: 1445049729
> ch7_arm: 1425753718
> ch7_aff_change: 0
> ch7_eq_rearm: 0
> ch8_events: 1426976374
> ch8_poll: 1439938692
> ch8_arm: 1421392984
> ch8_aff_change: 0
> ch8_eq_rearm: 0
> ch9_events: 1456160031
> ch9_poll: 1468922870
> ch9_arm: 1450930446
> ch9_aff_change: 0
> ch9_eq_rearm: 0
> ch10_events: 1443640165
> ch10_poll: 1456812203
> ch10_arm: 1438425101
> ch10_aff_change: 0
> ch10_eq_rearm: 0
> ch11_events: 1381104776
> ch11_poll: 1393811057
> ch11_arm: 1376059326
> ch11_aff_change: 0
> ch11_eq_rearm: 0
> ch12_events: 1365223276
> ch12_poll: 1378406059
> ch12_arm: 1359950494
> ch12_aff_change: 0
> ch12_eq_rearm: 0
> ch13_events: 1421622259
> ch13_poll: 1434670996
> ch13_arm: 1416241801
> ch13_aff_change: 0
> ch13_eq_rearm: 0
> ch14_events: 1379084590
> ch14_poll: 1392425015
> ch14_arm: 1373675179
> ch14_aff_change: 0
> ch14_eq_rearm: 0
> ch15_events: 1531217338
> ch15_poll: 1543353833
> ch15_arm: 1526350453
> ch15_aff_change: 0
> ch15_eq_rearm: 0
> ch16_events: 1460469776
> ch16_poll: 1467995928
> ch16_arm: 1456010194
> ch16_aff_change: 0
> ch16_eq_rearm: 0
> ch17_events: 1494067670
> ch17_poll: 1500856680
> ch17_arm: 1489232674
> ch17_aff_change: 0
> ch17_eq_rearm: 0
> ch18_events: 1530126866
> ch18_poll: 1537293620
> ch18_arm: 1525476123
> ch18_aff_change: 0
> ch18_eq_rearm: 0
> ch19_events: 1499526149
> ch19_poll: 1506789309
> ch19_arm: 1495161602
> ch19_aff_change: 0
> ch19_eq_rearm: 0
> ch20_events: 1451479763
> ch20_poll: 1459767921
> ch20_arm: 1446360801
> ch20_aff_change: 0
> ch20_eq_rearm: 0
> ch21_events: 1521413613
> ch21_poll: 1529345146
> ch21_arm: 1517229314
> ch21_aff_change: 0
> ch21_eq_rearm: 0
> ch22_events: 1471950045
> ch22_poll: 1479746764
> ch22_arm: 1467681629
> ch22_aff_change: 0
> ch22_eq_rearm: 0
> ch23_events: 1502968393
> ch23_poll: 1510419909
> ch23_arm: 1498168438
> ch23_aff_change: 0
> ch23_eq_rearm: 0
> ch24_events: 1473451639
> ch24_poll: 1482606899
> ch24_arm: 1468212489
> ch24_aff_change: 0
> ch24_eq_rearm: 0
> ch25_events: 1440399182
> ch25_poll: 1448897475
> ch25_arm: 1435044786
> ch25_aff_change: 0
> ch25_eq_rearm: 0
> ch26_events: 1436831565
> ch26_poll: 1445485731
> ch26_arm: 1431827527
> ch26_aff_change: 0
> ch26_eq_rearm: 0
> ch27_events: 1516560621
> ch27_poll: 1524911010
> ch27_arm: 1511430164
> ch27_aff_change: 0
> ch27_eq_rearm: 0
> ch28_events: 4
> ch28_poll: 4
> ch28_arm: 4
> ch28_aff_change: 0
> ch28_eq_rearm: 0
> ch29_events: 6
> ch29_poll: 6
> ch29_arm: 6
> ch29_aff_change: 0
> ch29_eq_rearm: 0
> ch30_events: 4
> ch30_poll: 4
> ch30_arm: 4
> ch30_aff_change: 0
> ch30_eq_rearm: 0
> ch31_events: 4
> ch31_poll: 4
> ch31_arm: 4
> ch31_aff_change: 0
> ch31_eq_rearm: 0
> ch32_events: 4
> ch32_poll: 4
> ch32_arm: 4
> ch32_aff_change: 0
> ch32_eq_rearm: 0
> ch33_events: 4
> ch33_poll: 4
> ch33_arm: 4
> ch33_aff_change: 0
> ch33_eq_rearm: 0
> ch34_events: 4
> ch34_poll: 4
> ch34_arm: 4
> ch34_aff_change: 0
> ch34_eq_rearm: 0
> ch35_events: 4
> ch35_poll: 4
> ch35_arm: 4
> ch35_aff_change: 0
> ch35_eq_rearm: 0
> ch36_events: 4
> ch36_poll: 4
> ch36_arm: 4
> ch36_aff_change: 0
> ch36_eq_rearm: 0
> ch37_events: 4
> ch37_poll: 4
> ch37_arm: 4
> ch37_aff_change: 0
> ch37_eq_rearm: 0
> ch38_events: 4
> ch38_poll: 4
> ch38_arm: 4
> ch38_aff_change: 0
> ch38_eq_rearm: 0
> ch39_events: 4
> ch39_poll: 4
> ch39_arm: 4
> ch39_aff_change: 0
> ch39_eq_rearm: 0
> ch40_events: 4
> ch40_poll: 4
> ch40_arm: 4
> ch40_aff_change: 0
> ch40_eq_rearm: 0
> ch41_events: 4
> ch41_poll: 4
> ch41_arm: 4
> ch41_aff_change: 0
> ch41_eq_rearm: 0
> ch42_events: 4
> ch42_poll: 4
> ch42_arm: 4
> ch42_aff_change: 0
> ch42_eq_rearm: 0
> ch43_events: 4
> ch43_poll: 4
> ch43_arm: 4
> ch43_aff_change: 0
> ch43_eq_rearm: 0
> ch44_events: 4
> ch44_poll: 4
> ch44_arm: 4
> ch44_aff_change: 0
> ch44_eq_rearm: 0
> ch45_events: 4
> ch45_poll: 4
> ch45_arm: 4
> ch45_aff_change: 0
> ch45_eq_rearm: 0
> ch46_events: 4
> ch46_poll: 4
> ch46_arm: 4
> ch46_aff_change: 0
> ch46_eq_rearm: 0
> ch47_events: 4
> ch47_poll: 4
> ch47_arm: 4
> ch47_aff_change: 0
> ch47_eq_rearm: 0
> ch48_events: 4
> ch48_poll: 4
> ch48_arm: 4
> ch48_aff_change: 0
> ch48_eq_rearm: 0
> ch49_events: 4
> ch49_poll: 4
> ch49_arm: 4
> ch49_aff_change: 0
> ch49_eq_rearm: 0
> ch50_events: 4
> ch50_poll: 4
> ch50_arm: 4
> ch50_aff_change: 0
> ch50_eq_rearm: 0
> ch51_events: 4
> ch51_poll: 4
> ch51_arm: 4
> ch51_aff_change: 0
> ch51_eq_rearm: 0
> ch52_events: 4
> ch52_poll: 4
> ch52_arm: 4
> ch52_aff_change: 0
> ch52_eq_rearm: 0
> ch53_events: 4
> ch53_poll: 4
> ch53_arm: 4
> ch53_aff_change: 0
> ch53_eq_rearm: 0
> ch54_events: 4
> ch54_poll: 4
> ch54_arm: 4
> ch54_aff_change: 0
> ch54_eq_rearm: 0
> ch55_events: 4
> ch55_poll: 4
> ch55_arm: 4
> ch55_aff_change: 0
> ch55_eq_rearm: 0
> rx0_packets: 5861448653
> rx0_bytes: 7389128595728
> rx0_csum_complete: 5838312798
> rx0_csum_unnecessary: 0
> rx0_csum_unnecessary_inner: 0
> rx0_csum_none: 23135855
> rx0_xdp_drop: 0
> rx0_xdp_redirect: 0
> rx0_lro_packets: 0
> rx0_lro_bytes: 0
> rx0_ecn_mark: 0
> rx0_removed_vlan_packets: 5861448653
> rx0_wqe_err: 0
> rx0_mpwqe_filler_cqes: 0
> rx0_mpwqe_filler_strides: 0
> rx0_buff_alloc_err: 0
> rx0_cqe_compress_blks: 0
> rx0_cqe_compress_pkts: 0
> rx0_page_reuse: 0
> rx0_cache_reuse: 2559
> rx0_cache_full: 2930721512
> rx0_cache_empty: 6656
> rx0_cache_busy: 2930721765
> rx0_cache_waive: 0
> rx0_congst_umr: 0
> rx0_arfs_err: 0
> rx0_xdp_tx_xmit: 0
> rx0_xdp_tx_full: 0
> rx0_xdp_tx_err: 0
> rx0_xdp_tx_cqes: 0
> rx1_packets: 5550585106
> rx1_bytes: 7255635262803
> rx1_csum_complete: 5550585106
> rx1_csum_unnecessary: 0
> rx1_csum_unnecessary_inner: 0
> rx1_csum_none: 0
> rx1_xdp_drop: 0
> rx1_xdp_redirect: 0
> rx1_lro_packets: 0
> rx1_lro_bytes: 0
> rx1_ecn_mark: 0
> rx1_removed_vlan_packets: 5550585106
> rx1_wqe_err: 0
> rx1_mpwqe_filler_cqes: 0
> rx1_mpwqe_filler_strides: 0
> rx1_buff_alloc_err: 0
> rx1_cqe_compress_blks: 0
> rx1_cqe_compress_pkts: 0
> rx1_page_reuse: 0
> rx1_cache_reuse: 2918845
> rx1_cache_full: 2772373453
> rx1_cache_empty: 6656
> rx1_cache_busy: 2772373707
> rx1_cache_waive: 0
> rx1_congst_umr: 0
> rx1_arfs_err: 0
> rx1_xdp_tx_xmit: 0
> rx1_xdp_tx_full: 0
> rx1_xdp_tx_err: 0
> rx1_xdp_tx_cqes: 0
> rx2_packets: 5383874739
> rx2_bytes: 7031545423967
> rx2_csum_complete: 5383874739
> rx2_csum_unnecessary: 0
> rx2_csum_unnecessary_inner: 0
> rx2_csum_none: 0
> rx2_xdp_drop: 0
> rx2_xdp_redirect: 0
> rx2_lro_packets: 0
> rx2_lro_bytes: 0
> rx2_ecn_mark: 0
> rx2_removed_vlan_packets: 5383874739
> rx2_wqe_err: 0
> rx2_mpwqe_filler_cqes: 0
> rx2_mpwqe_filler_strides: 0
> rx2_buff_alloc_err: 0
> rx2_cqe_compress_blks: 0
> rx2_cqe_compress_pkts: 0
> rx2_page_reuse: 0
> rx2_cache_reuse: 2173370
> rx2_cache_full: 2689763744
> rx2_cache_empty: 6656
> rx2_cache_busy: 2689763998
> rx2_cache_waive: 0
> rx2_congst_umr: 0
> rx2_arfs_err: 0
> rx2_xdp_tx_xmit: 0
> rx2_xdp_tx_full: 0
> rx2_xdp_tx_err: 0
> rx2_xdp_tx_cqes: 0
> rx3_packets: 5456494012
> rx3_bytes: 7120241119485
> rx3_csum_complete: 5456494012
> rx3_csum_unnecessary: 0
> rx3_csum_unnecessary_inner: 0
> rx3_csum_none: 0
> rx3_xdp_drop: 0
> rx3_xdp_redirect: 0
> rx3_lro_packets: 0
> rx3_lro_bytes: 0
> rx3_ecn_mark: 0
> rx3_removed_vlan_packets: 5456494012
> rx3_wqe_err: 0
> rx3_mpwqe_filler_cqes: 0
> rx3_mpwqe_filler_strides: 0
> rx3_buff_alloc_err: 0
> rx3_cqe_compress_blks: 0
> rx3_cqe_compress_pkts: 0
> rx3_page_reuse: 0
> rx3_cache_reuse: 2120123
> rx3_cache_full: 2726126628
> rx3_cache_empty: 6656
> rx3_cache_busy: 2726126881
> rx3_cache_waive: 0
> rx3_congst_umr: 0
> rx3_arfs_err: 0
> rx3_xdp_tx_xmit: 0
> rx3_xdp_tx_full: 0
> rx3_xdp_tx_err: 0
> rx3_xdp_tx_cqes: 0
> rx4_packets: 5475216251
> rx4_bytes: 7123129170196
> rx4_csum_complete: 5475216251
> rx4_csum_unnecessary: 0
> rx4_csum_unnecessary_inner: 0
> rx4_csum_none: 0
> rx4_xdp_drop: 0
> rx4_xdp_redirect: 0
> rx4_lro_packets: 0
> rx4_lro_bytes: 0
> rx4_ecn_mark: 0
> rx4_removed_vlan_packets: 5475216251
> rx4_wqe_err: 0
> rx4_mpwqe_filler_cqes: 0
> rx4_mpwqe_filler_strides: 0
> rx4_buff_alloc_err: 0
> rx4_cqe_compress_blks: 0
> rx4_cqe_compress_pkts: 0
> rx4_page_reuse: 0
> rx4_cache_reuse: 2668296355
> rx4_cache_full: 69311549
> rx4_cache_empty: 6656
> rx4_cache_busy: 69311769
> rx4_cache_waive: 0
> rx4_congst_umr: 0
> rx4_arfs_err: 0
> rx4_xdp_tx_xmit: 0
> rx4_xdp_tx_full: 0
> rx4_xdp_tx_err: 0
> rx4_xdp_tx_cqes: 0
> rx5_packets: 5474372232
> rx5_bytes: 7159146801926
> rx5_csum_complete: 5474372232
> rx5_csum_unnecessary: 0
> rx5_csum_unnecessary_inner: 0
> rx5_csum_none: 0
> rx5_xdp_drop: 0
> rx5_xdp_redirect: 0
> rx5_lro_packets: 0
> rx5_lro_bytes: 0
> rx5_ecn_mark: 0
> rx5_removed_vlan_packets: 5474372232
> rx5_wqe_err: 0
> rx5_mpwqe_filler_cqes: 0
> rx5_mpwqe_filler_strides: 0
> rx5_buff_alloc_err: 0
> rx5_cqe_compress_blks: 0
> rx5_cqe_compress_pkts: 0
> rx5_page_reuse: 0
> rx5_cache_reuse: 626187
> rx5_cache_full: 2736559674
> rx5_cache_empty: 6656
> rx5_cache_busy: 2736559929
> rx5_cache_waive: 0
> rx5_congst_umr: 0
> rx5_arfs_err: 0
> rx5_xdp_tx_xmit: 0
> rx5_xdp_tx_full: 0
> rx5_xdp_tx_err: 0
> rx5_xdp_tx_cqes: 0
> rx6_packets: 5533622456
> rx6_bytes: 7207308809081
> rx6_csum_complete: 5533622456
> rx6_csum_unnecessary: 0
> rx6_csum_unnecessary_inner: 0
> rx6_csum_none: 0
> rx6_xdp_drop: 0
> rx6_xdp_redirect: 0
> rx6_lro_packets: 0
> rx6_lro_bytes: 0
> rx6_ecn_mark: 0
> rx6_removed_vlan_packets: 5533622456
> rx6_wqe_err: 0
> rx6_mpwqe_filler_cqes: 0
> rx6_mpwqe_filler_strides: 0
> rx6_buff_alloc_err: 0
> rx6_cqe_compress_blks: 0
> rx6_cqe_compress_pkts: 0
> rx6_page_reuse: 0
> rx6_cache_reuse: 2325217
> rx6_cache_full: 2764485756
> rx6_cache_empty: 6656
> rx6_cache_busy: 2764486011
> rx6_cache_waive: 0
> rx6_congst_umr: 0
> rx6_arfs_err: 0
> rx6_xdp_tx_xmit: 0
> rx6_xdp_tx_full: 0
> rx6_xdp_tx_err: 0
> rx6_xdp_tx_cqes: 0
> rx7_packets: 5533901822
> rx7_bytes: 7227441240536
> rx7_csum_complete: 5533901822
> rx7_csum_unnecessary: 0
> rx7_csum_unnecessary_inner: 0
> rx7_csum_none: 0
> rx7_xdp_drop: 0
> rx7_xdp_redirect: 0
> rx7_lro_packets: 0
> rx7_lro_bytes: 0
> rx7_ecn_mark: 0
> rx7_removed_vlan_packets: 5533901822
> rx7_wqe_err: 0
> rx7_mpwqe_filler_cqes: 0
> rx7_mpwqe_filler_strides: 0
> rx7_buff_alloc_err: 0
> rx7_cqe_compress_blks: 0
> rx7_cqe_compress_pkts: 0
> rx7_page_reuse: 0
> rx7_cache_reuse: 2372505
> rx7_cache_full: 2764578151
> rx7_cache_empty: 6656
> rx7_cache_busy: 2764578403
> rx7_cache_waive: 0
> rx7_congst_umr: 0
> rx7_arfs_err: 0
> rx7_xdp_tx_xmit: 0
> rx7_xdp_tx_full: 0
> rx7_xdp_tx_err: 0
> rx7_xdp_tx_cqes: 0
> rx8_packets: 5485670137
> rx8_bytes: 7203339989013
> rx8_csum_complete: 5485670137
> rx8_csum_unnecessary: 0
> rx8_csum_unnecessary_inner: 0
> rx8_csum_none: 0
> rx8_xdp_drop: 0
> rx8_xdp_redirect: 0
> rx8_lro_packets: 0
> rx8_lro_bytes: 0
> rx8_ecn_mark: 0
> rx8_removed_vlan_packets: 5485670137
> rx8_wqe_err: 0
> rx8_mpwqe_filler_cqes: 0
> rx8_mpwqe_filler_strides: 0
> rx8_buff_alloc_err: 0
> rx8_cqe_compress_blks: 0
> rx8_cqe_compress_pkts: 0
> rx8_page_reuse: 0
> rx8_cache_reuse: 7522232
> rx8_cache_full: 2735312581
> rx8_cache_empty: 6656
> rx8_cache_busy: 2735312836
> rx8_cache_waive: 0
> rx8_congst_umr: 0
> rx8_arfs_err: 0
> rx8_xdp_tx_xmit: 0
> rx8_xdp_tx_full: 0
> rx8_xdp_tx_err: 0
> rx8_xdp_tx_cqes: 0
> rx9_packets: 5482212354
> rx9_bytes: 7169663341718
> rx9_csum_complete: 5482212354
> rx9_csum_unnecessary: 0
> rx9_csum_unnecessary_inner: 0
> rx9_csum_none: 0
> rx9_xdp_drop: 0
> rx9_xdp_redirect: 0
> rx9_lro_packets: 0
> rx9_lro_bytes: 0
> rx9_ecn_mark: 0
> rx9_removed_vlan_packets: 5482212354
> rx9_wqe_err: 0
> rx9_mpwqe_filler_cqes: 0
> rx9_mpwqe_filler_strides: 0
> rx9_buff_alloc_err: 0
> rx9_cqe_compress_blks: 0
> rx9_cqe_compress_pkts: 0
> rx9_page_reuse: 0
> rx9_cache_reuse: 37279961
> rx9_cache_full: 2703825961
> rx9_cache_empty: 6656
> rx9_cache_busy: 2703826215
> rx9_cache_waive: 0
> rx9_congst_umr: 0
> rx9_arfs_err: 0
> rx9_xdp_tx_xmit: 0
> rx9_xdp_tx_full: 0
> rx9_xdp_tx_err: 0
> rx9_xdp_tx_cqes: 0
> rx10_packets: 5524679952
> rx10_bytes: 7248301275181
> rx10_csum_complete: 5524679952
> rx10_csum_unnecessary: 0
> rx10_csum_unnecessary_inner: 0
> rx10_csum_none: 0
> rx10_xdp_drop: 0
> rx10_xdp_redirect: 0
> rx10_lro_packets: 0
> rx10_lro_bytes: 0
> rx10_ecn_mark: 0
> rx10_removed_vlan_packets: 5524679952
> rx10_wqe_err: 0
> rx10_mpwqe_filler_cqes: 0
> rx10_mpwqe_filler_strides: 0
> rx10_buff_alloc_err: 0
> rx10_cqe_compress_blks: 0
> rx10_cqe_compress_pkts: 0
> rx10_page_reuse: 0
> rx10_cache_reuse: 2049666
> rx10_cache_full: 2760290055
> rx10_cache_empty: 6656
> rx10_cache_busy: 2760290310
> rx10_cache_waive: 0
> rx10_congst_umr: 0
> rx10_arfs_err: 0
> rx10_xdp_tx_xmit: 0
> rx10_xdp_tx_full: 0
> rx10_xdp_tx_err: 0
> rx10_xdp_tx_cqes: 0
> rx11_packets: 5394633545
> rx11_bytes: 7033509636092
> rx11_csum_complete: 5394633545
> rx11_csum_unnecessary: 0
> rx11_csum_unnecessary_inner: 0
> rx11_csum_none: 0
> rx11_xdp_drop: 0
> rx11_xdp_redirect: 0
> rx11_lro_packets: 0
> rx11_lro_bytes: 0
> rx11_ecn_mark: 0
> rx11_removed_vlan_packets: 5394633545
> rx11_wqe_err: 0
> rx11_mpwqe_filler_cqes: 0
> rx11_mpwqe_filler_strides: 0
> rx11_buff_alloc_err: 0
> rx11_cqe_compress_blks: 0
> rx11_cqe_compress_pkts: 0
> rx11_page_reuse: 0
> rx11_cache_reuse: 2617466268
> rx11_cache_full: 79850284
> rx11_cache_empty: 6656
> rx11_cache_busy: 79850504
> rx11_cache_waive: 0
> rx11_congst_umr: 0
> rx11_arfs_err: 0
> rx11_xdp_tx_xmit: 0
> rx11_xdp_tx_full: 0
> rx11_xdp_tx_err: 0
> rx11_xdp_tx_cqes: 0
> rx12_packets: 5458907385
> rx12_bytes: 7134867867515
> rx12_csum_complete: 5458907385
> rx12_csum_unnecessary: 0
> rx12_csum_unnecessary_inner: 0
> rx12_csum_none: 0
> rx12_xdp_drop: 0
> rx12_xdp_redirect: 0
> rx12_lro_packets: 0
> rx12_lro_bytes: 0
> rx12_ecn_mark: 0
> rx12_removed_vlan_packets: 5458907385
> rx12_wqe_err: 0
> rx12_mpwqe_filler_cqes: 0
> rx12_mpwqe_filler_strides: 0
> rx12_buff_alloc_err: 0
> rx12_cqe_compress_blks: 0
> rx12_cqe_compress_pkts: 0
> rx12_page_reuse: 0
> rx12_cache_reuse: 2650214169
> rx12_cache_full: 79239303
> rx12_cache_empty: 6656
> rx12_cache_busy: 79239523
> rx12_cache_waive: 0
> rx12_congst_umr: 0
> rx12_arfs_err: 0
> rx12_xdp_tx_xmit: 0
> rx12_xdp_tx_full: 0
> rx12_xdp_tx_err: 0
> rx12_xdp_tx_cqes: 0
> rx13_packets: 5549932912
> rx13_bytes: 7232548705586
> rx13_csum_complete: 5549932912
> rx13_csum_unnecessary: 0
> rx13_csum_unnecessary_inner: 0
> rx13_csum_none: 0
> rx13_xdp_drop: 0
> rx13_xdp_redirect: 0
> rx13_lro_packets: 0
> rx13_lro_bytes: 0
> rx13_ecn_mark: 0
> rx13_removed_vlan_packets: 5549932912
> rx13_wqe_err: 0
> rx13_mpwqe_filler_cqes: 0
> rx13_mpwqe_filler_strides: 0
> rx13_buff_alloc_err: 0
> rx13_cqe_compress_blks: 0
> rx13_cqe_compress_pkts: 0
> rx13_page_reuse: 0
> rx13_cache_reuse: 2417696
> rx13_cache_full: 2772548505
> rx13_cache_empty: 6656
> rx13_cache_busy: 2772548760
> rx13_cache_waive: 0
> rx13_congst_umr: 0
> rx13_arfs_err: 0
> rx13_xdp_tx_xmit: 0
> rx13_xdp_tx_full: 0
> rx13_xdp_tx_err: 0
> rx13_xdp_tx_cqes: 0
> rx14_packets: 5517712329
> rx14_bytes: 7192111965227
> rx14_csum_complete: 5517712329
> rx14_csum_unnecessary: 0
> rx14_csum_unnecessary_inner: 0
> rx14_csum_none: 0
> rx14_xdp_drop: 0
> rx14_xdp_redirect: 0
> rx14_lro_packets: 0
> rx14_lro_bytes: 0
> rx14_ecn_mark: 0
> rx14_removed_vlan_packets: 5517712329
> rx14_wqe_err: 0
> rx14_mpwqe_filler_cqes: 0
> rx14_mpwqe_filler_strides: 0
> rx14_buff_alloc_err: 0
> rx14_cqe_compress_blks: 0
> rx14_cqe_compress_pkts: 0
> rx14_page_reuse: 0
> rx14_cache_reuse: 1830206
> rx14_cache_full: 2757025703
> rx14_cache_empty: 6656
> rx14_cache_busy: 2757025958
> rx14_cache_waive: 0
> rx14_congst_umr: 0
> rx14_arfs_err: 0
> rx14_xdp_tx_xmit: 0
> rx14_xdp_tx_full: 0
> rx14_xdp_tx_err: 0
> rx14_xdp_tx_cqes: 0
> rx15_packets: 5578343373
> rx15_bytes: 7268484501219
> rx15_csum_complete: 5578343373
> rx15_csum_unnecessary: 0
> rx15_csum_unnecessary_inner: 0
> rx15_csum_none: 0
> rx15_xdp_drop: 0
> rx15_xdp_redirect: 0
> rx15_lro_packets: 0
> rx15_lro_bytes: 0
> rx15_ecn_mark: 0
> rx15_removed_vlan_packets: 5578343373
> rx15_wqe_err: 0
> rx15_mpwqe_filler_cqes: 0
> rx15_mpwqe_filler_strides: 0
> rx15_buff_alloc_err: 0
> rx15_cqe_compress_blks: 0
> rx15_cqe_compress_pkts: 0
> rx15_page_reuse: 0
> rx15_cache_reuse: 2317165
> rx15_cache_full: 2786854266
> rx15_cache_empty: 6656
> rx15_cache_busy: 2786854519
> rx15_cache_waive: 0
> rx15_congst_umr: 0
> rx15_arfs_err: 0
> rx15_xdp_tx_xmit: 0
> rx15_xdp_tx_full: 0
> rx15_xdp_tx_err: 0
> rx15_xdp_tx_cqes: 0
> rx16_packets: 4435773951
> rx16_bytes: 5766665272007
> rx16_csum_complete: 4435773951
> rx16_csum_unnecessary: 0
> rx16_csum_unnecessary_inner: 0
> rx16_csum_none: 0
> rx16_xdp_drop: 0
> rx16_xdp_redirect: 0
> rx16_lro_packets: 0
> rx16_lro_bytes: 0
> rx16_ecn_mark: 0
> rx16_removed_vlan_packets: 4435773951
> rx16_wqe_err: 0
> rx16_mpwqe_filler_cqes: 0
> rx16_mpwqe_filler_strides: 0
> rx16_buff_alloc_err: 0
> rx16_cqe_compress_blks: 0
> rx16_cqe_compress_pkts: 0
> rx16_page_reuse: 0
> rx16_cache_reuse: 2033793
> rx16_cache_full: 2215852927
> rx16_cache_empty: 6656
> rx16_cache_busy: 2215853179
> rx16_cache_waive: 0
> rx16_congst_umr: 0
> rx16_arfs_err: 0
> rx16_xdp_tx_xmit: 0
> rx16_xdp_tx_full: 0
> rx16_xdp_tx_err: 0
> rx16_xdp_tx_cqes: 0
> rx17_packets: 4344087587
> rx17_bytes: 5695006496323
> rx17_csum_complete: 4344087587
> rx17_csum_unnecessary: 0
> rx17_csum_unnecessary_inner: 0
> rx17_csum_none: 0
> rx17_xdp_drop: 0
> rx17_xdp_redirect: 0
> rx17_lro_packets: 0
> rx17_lro_bytes: 0
> rx17_ecn_mark: 0
> rx17_removed_vlan_packets: 4344087587
> rx17_wqe_err: 0
> rx17_mpwqe_filler_cqes: 0
> rx17_mpwqe_filler_strides: 0
> rx17_buff_alloc_err: 0
> rx17_cqe_compress_blks: 0
> rx17_cqe_compress_pkts: 0
> rx17_page_reuse: 0
> rx17_cache_reuse: 2652127
> rx17_cache_full: 2169391411
> rx17_cache_empty: 6656
> rx17_cache_busy: 2169391665
> rx17_cache_waive: 0
> rx17_congst_umr: 0
> rx17_arfs_err: 0
> rx17_xdp_tx_xmit: 0
> rx17_xdp_tx_full: 0
> rx17_xdp_tx_err: 0
> rx17_xdp_tx_cqes: 0
> rx18_packets: 4407422804
> rx18_bytes: 5741134634177
> rx18_csum_complete: 4407422804
> rx18_csum_unnecessary: 0
> rx18_csum_unnecessary_inner: 0
> rx18_csum_none: 0
> rx18_xdp_drop: 0
> rx18_xdp_redirect: 0
> rx18_lro_packets: 0
> rx18_lro_bytes: 0
> rx18_ecn_mark: 0
> rx18_removed_vlan_packets: 4407422804
> rx18_wqe_err: 0
> rx18_mpwqe_filler_cqes: 0
> rx18_mpwqe_filler_strides: 0
> rx18_buff_alloc_err: 0
> rx18_cqe_compress_blks: 0
> rx18_cqe_compress_pkts: 0
> rx18_page_reuse: 0
> rx18_cache_reuse: 2156080239
> rx18_cache_full: 47630941
> rx18_cache_empty: 6656
> rx18_cache_busy: 47631161
> rx18_cache_waive: 0
> rx18_congst_umr: 0
> rx18_arfs_err: 0
> rx18_xdp_tx_xmit: 0
> rx18_xdp_tx_full: 0
> rx18_xdp_tx_err: 0
> rx18_xdp_tx_cqes: 0
> rx19_packets: 4545554180
> rx19_bytes: 5905277503466
> rx19_csum_complete: 4545554180
> rx19_csum_unnecessary: 0
> rx19_csum_unnecessary_inner: 0
> rx19_csum_none: 0
> rx19_xdp_drop: 0
> rx19_xdp_redirect: 0
> rx19_lro_packets: 0
> rx19_lro_bytes: 0
> rx19_ecn_mark: 0
> rx19_removed_vlan_packets: 4545554180
> rx19_wqe_err: 0
> rx19_mpwqe_filler_cqes: 0
> rx19_mpwqe_filler_strides: 0
> rx19_buff_alloc_err: 0
> rx19_cqe_compress_blks: 0
> rx19_cqe_compress_pkts: 0
> rx19_page_reuse: 0
> rx19_cache_reuse: 11112455
> rx19_cache_full: 2261664379
> rx19_cache_empty: 6656
> rx19_cache_busy: 2261664601
> rx19_cache_waive: 0
> rx19_congst_umr: 0
> rx19_arfs_err: 0
> rx19_xdp_tx_xmit: 0
> rx19_xdp_tx_full: 0
> rx19_xdp_tx_err: 0
> rx19_xdp_tx_cqes: 0
> rx20_packets: 4397428553
> rx20_bytes: 5757329184301
> rx20_csum_complete: 4397428553
> rx20_csum_unnecessary: 0
> rx20_csum_unnecessary_inner: 0
> rx20_csum_none: 0
> rx20_xdp_drop: 0
> rx20_xdp_redirect: 0
> rx20_lro_packets: 0
> rx20_lro_bytes: 0
> rx20_ecn_mark: 0
> rx20_removed_vlan_packets: 4397428553
> rx20_wqe_err: 0
> rx20_mpwqe_filler_cqes: 0
> rx20_mpwqe_filler_strides: 0
> rx20_buff_alloc_err: 0
> rx20_cqe_compress_blks: 0
> rx20_cqe_compress_pkts: 0
> rx20_page_reuse: 0
> rx20_cache_reuse: 2168116995
> rx20_cache_full: 30597061
> rx20_cache_empty: 6656
> rx20_cache_busy: 30597281
> rx20_cache_waive: 0
> rx20_congst_umr: 0
> rx20_arfs_err: 0
> rx20_xdp_tx_xmit: 0
> rx20_xdp_tx_full: 0
> rx20_xdp_tx_err: 0
> rx20_xdp_tx_cqes: 0
> rx21_packets: 4552564821
> rx21_bytes: 5944840329249
> rx21_csum_complete: 4552564821
> rx21_csum_unnecessary: 0
> rx21_csum_unnecessary_inner: 0
> rx21_csum_none: 0
> rx21_xdp_drop: 0
> rx21_xdp_redirect: 0
> rx21_lro_packets: 0
> rx21_lro_bytes: 0
> rx21_ecn_mark: 0
> rx21_removed_vlan_packets: 4552564821
> rx21_wqe_err: 0
> rx21_mpwqe_filler_cqes: 0
> rx21_mpwqe_filler_strides: 0
> rx21_buff_alloc_err: 0
> rx21_cqe_compress_blks: 0
> rx21_cqe_compress_pkts: 0
> rx21_page_reuse: 0
> rx21_cache_reuse: 2295681
> rx21_cache_full: 2273986474
> rx21_cache_empty: 6656
> rx21_cache_busy: 2273986727
> rx21_cache_waive: 0
> rx21_congst_umr: 0
> rx21_arfs_err: 0
> rx21_xdp_tx_xmit: 0
> rx21_xdp_tx_full: 0
> rx21_xdp_tx_err: 0
> rx21_xdp_tx_cqes: 0
> rx22_packets: 4629499740
> rx22_bytes: 5924206566499
> rx22_csum_complete: 4629499740
> rx22_csum_unnecessary: 0
> rx22_csum_unnecessary_inner: 0
> rx22_csum_none: 0
> rx22_xdp_drop: 0
> rx22_xdp_redirect: 0
> rx22_lro_packets: 0
> rx22_lro_bytes: 0
> rx22_ecn_mark: 0
> rx22_removed_vlan_packets: 4629499740
> rx22_wqe_err: 0
> rx22_mpwqe_filler_cqes: 0
> rx22_mpwqe_filler_strides: 0
> rx22_buff_alloc_err: 0
> rx22_cqe_compress_blks: 0
> rx22_cqe_compress_pkts: 0
> rx22_page_reuse: 0
> rx22_cache_reuse: 1407527
> rx22_cache_full: 2313342088
> rx22_cache_empty: 6656
> rx22_cache_busy: 2313342341
> rx22_cache_waive: 0
> rx22_congst_umr: 0
> rx22_arfs_err: 0
> rx22_xdp_tx_xmit: 0
> rx22_xdp_tx_full: 0
> rx22_xdp_tx_err: 0
> rx22_xdp_tx_cqes: 0
> rx23_packets: 4387124505
> rx23_bytes: 5718118678470
> rx23_csum_complete: 4387124505
> rx23_csum_unnecessary: 0
> rx23_csum_unnecessary_inner: 0
> rx23_csum_none: 0
> rx23_xdp_drop: 0
> rx23_xdp_redirect: 0
> rx23_lro_packets: 0
> rx23_lro_bytes: 0
> rx23_ecn_mark: 0
> rx23_removed_vlan_packets: 4387124505
> rx23_wqe_err: 0
> rx23_mpwqe_filler_cqes: 0
> rx23_mpwqe_filler_strides: 0
> rx23_buff_alloc_err: 0
> rx23_cqe_compress_blks: 0
> rx23_cqe_compress_pkts: 0
> rx23_page_reuse: 0
> rx23_cache_reuse: 2013280
> rx23_cache_full: 2191548717
> rx23_cache_empty: 6656
> rx23_cache_busy: 2191548972
> rx23_cache_waive: 0
> rx23_congst_umr: 0
> rx23_arfs_err: 0
> rx23_xdp_tx_xmit: 0
> rx23_xdp_tx_full: 0
> rx23_xdp_tx_err: 0
> rx23_xdp_tx_cqes: 0
> rx24_packets: 4398791634
> rx24_bytes: 5744875564632
> rx24_csum_complete: 4398791634
> rx24_csum_unnecessary: 0
> rx24_csum_unnecessary_inner: 0
> rx24_csum_none: 0
> rx24_xdp_drop: 0
> rx24_xdp_redirect: 0
> rx24_lro_packets: 0
> rx24_lro_bytes: 0
> rx24_ecn_mark: 0
> rx24_removed_vlan_packets: 4398791634
> rx24_wqe_err: 0
> rx24_mpwqe_filler_cqes: 0
> rx24_mpwqe_filler_strides: 0
> rx24_buff_alloc_err: 0
> rx24_cqe_compress_blks: 0
> rx24_cqe_compress_pkts: 0
> rx24_page_reuse: 0
> rx24_cache_reuse: 2143926100
> rx24_cache_full: 55469496
> rx24_cache_empty: 6656
> rx24_cache_busy: 55469716
> rx24_cache_waive: 0
> rx24_congst_umr: 0
> rx24_arfs_err: 0
> rx24_xdp_tx_xmit: 0
> rx24_xdp_tx_full: 0
> rx24_xdp_tx_err: 0
> rx24_xdp_tx_cqes: 0
> rx25_packets: 4377204935
> rx25_bytes: 5710369124105
> rx25_csum_complete: 4377204935
> rx25_csum_unnecessary: 0
> rx25_csum_unnecessary_inner: 0
> rx25_csum_none: 0
> rx25_xdp_drop: 0
> rx25_xdp_redirect: 0
> rx25_lro_packets: 0
> rx25_lro_bytes: 0
> rx25_ecn_mark: 0
> rx25_removed_vlan_packets: 4377204935
> rx25_wqe_err: 0
> rx25_mpwqe_filler_cqes: 0
> rx25_mpwqe_filler_strides: 0
> rx25_buff_alloc_err: 0
> rx25_cqe_compress_blks: 0
> rx25_cqe_compress_pkts: 0
> rx25_page_reuse: 0
> rx25_cache_reuse: 2132658660
> rx25_cache_full: 55943584
> rx25_cache_empty: 6656
> rx25_cache_busy: 55943804
> rx25_cache_waive: 0
> rx25_congst_umr: 0
> rx25_arfs_err: 0
> rx25_xdp_tx_xmit: 0
> rx25_xdp_tx_full: 0
> rx25_xdp_tx_err: 0
> rx25_xdp_tx_cqes: 0
> rx26_packets: 4496003688
> rx26_bytes: 5862180715503
> rx26_csum_complete: 4496003688
> rx26_csum_unnecessary: 0
> rx26_csum_unnecessary_inner: 0
> rx26_csum_none: 0
> rx26_xdp_drop: 0
> rx26_xdp_redirect: 0
> rx26_lro_packets: 0
> rx26_lro_bytes: 0
> rx26_ecn_mark: 0
> rx26_removed_vlan_packets: 4496003688
> rx26_wqe_err: 0
> rx26_mpwqe_filler_cqes: 0
> rx26_mpwqe_filler_strides: 0
> rx26_buff_alloc_err: 0
> rx26_cqe_compress_blks: 0
> rx26_cqe_compress_pkts: 0
> rx26_page_reuse: 0
> rx26_cache_reuse: 8
> rx26_cache_full: 2248001581
> rx26_cache_empty: 6656
> rx26_cache_busy: 2248001836
> rx26_cache_waive: 0
> rx26_congst_umr: 0
> rx26_arfs_err: 0
> rx26_xdp_tx_xmit: 0
> rx26_xdp_tx_full: 0
> rx26_xdp_tx_err: 0
> rx26_xdp_tx_cqes: 0
> rx27_packets: 4341849333
> rx27_bytes: 5678653545018
> rx27_csum_complete: 4341849333
> rx27_csum_unnecessary: 0
> rx27_csum_unnecessary_inner: 0
> rx27_csum_none: 0
> rx27_xdp_drop: 0
> rx27_xdp_redirect: 0
> rx27_lro_packets: 0
> rx27_lro_bytes: 0
> rx27_ecn_mark: 0
> rx27_removed_vlan_packets: 4341849333
> rx27_wqe_err: 0
> rx27_mpwqe_filler_cqes: 0
> rx27_mpwqe_filler_strides: 0
> rx27_buff_alloc_err: 0
> rx27_cqe_compress_blks: 0
> rx27_cqe_compress_pkts: 0
> rx27_page_reuse: 0
> rx27_cache_reuse: 1748188
> rx27_cache_full: 2169176223
> rx27_cache_empty: 6656
> rx27_cache_busy: 2169176476
> rx27_cache_waive: 0
> rx27_congst_umr: 0
> rx27_arfs_err: 0
> rx27_xdp_tx_xmit: 0
> rx27_xdp_tx_full: 0
> rx27_xdp_tx_err: 0
> rx27_xdp_tx_cqes: 0
> rx28_packets: 0
> rx28_bytes: 0
> rx28_csum_complete: 0
> rx28_csum_unnecessary: 0
> rx28_csum_unnecessary_inner: 0
> rx28_csum_none: 0
> rx28_xdp_drop: 0
> rx28_xdp_redirect: 0
> rx28_lro_packets: 0
> rx28_lro_bytes: 0
> rx28_ecn_mark: 0
> rx28_removed_vlan_packets: 0
> rx28_wqe_err: 0
> rx28_mpwqe_filler_cqes: 0
> rx28_mpwqe_filler_strides: 0
> rx28_buff_alloc_err: 0
> rx28_cqe_compress_blks: 0
> rx28_cqe_compress_pkts: 0
> rx28_page_reuse: 0
> rx28_cache_reuse: 0
> rx28_cache_full: 0
> rx28_cache_empty: 2560
> rx28_cache_busy: 0
> rx28_cache_waive: 0
> rx28_congst_umr: 0
> rx28_arfs_err: 0
> rx28_xdp_tx_xmit: 0
> rx28_xdp_tx_full: 0
> rx28_xdp_tx_err: 0
> rx28_xdp_tx_cqes: 0
> rx29_packets: 0
> rx29_bytes: 0
> rx29_csum_complete: 0
> rx29_csum_unnecessary: 0
> rx29_csum_unnecessary_inner: 0
> rx29_csum_none: 0
> rx29_xdp_drop: 0
> rx29_xdp_redirect: 0
> rx29_lro_packets: 0
> rx29_lro_bytes: 0
> rx29_ecn_mark: 0
> rx29_removed_vlan_packets: 0
> rx29_wqe_err: 0
> rx29_mpwqe_filler_cqes: 0
> rx29_mpwqe_filler_strides: 0
> rx29_buff_alloc_err: 0
> rx29_cqe_compress_blks: 0
> rx29_cqe_compress_pkts: 0
> rx29_page_reuse: 0
> rx29_cache_reuse: 0
> rx29_cache_full: 0
> rx29_cache_empty: 2560
> rx29_cache_busy: 0
> rx29_cache_waive: 0
> rx29_congst_umr: 0
> rx29_arfs_err: 0
> rx29_xdp_tx_xmit: 0
> rx29_xdp_tx_full: 0
> rx29_xdp_tx_err: 0
> rx29_xdp_tx_cqes: 0
> rx30_packets: 0
> rx30_bytes: 0
> rx30_csum_complete: 0
> rx30_csum_unnecessary: 0
> rx30_csum_unnecessary_inner: 0
> rx30_csum_none: 0
> rx30_xdp_drop: 0
> rx30_xdp_redirect: 0
> rx30_lro_packets: 0
> rx30_lro_bytes: 0
> rx30_ecn_mark: 0
> rx30_removed_vlan_packets: 0
> rx30_wqe_err: 0
> rx30_mpwqe_filler_cqes: 0
> rx30_mpwqe_filler_strides: 0
> rx30_buff_alloc_err: 0
> rx30_cqe_compress_blks: 0
> rx30_cqe_compress_pkts: 0
> rx30_page_reuse: 0
> rx30_cache_reuse: 0
> rx30_cache_full: 0
> rx30_cache_empty: 2560
> rx30_cache_busy: 0
> rx30_cache_waive: 0
> rx30_congst_umr: 0
> rx30_arfs_err: 0
> rx30_xdp_tx_xmit: 0
> rx30_xdp_tx_full: 0
> rx30_xdp_tx_err: 0
> rx30_xdp_tx_cqes: 0
> rx31_packets: 0
> rx31_bytes: 0
> rx31_csum_complete: 0
> rx31_csum_unnecessary: 0
> rx31_csum_unnecessary_inner: 0
> rx31_csum_none: 0
> rx31_xdp_drop: 0
> rx31_xdp_redirect: 0
> rx31_lro_packets: 0
> rx31_lro_bytes: 0
> rx31_ecn_mark: 0
> rx31_removed_vlan_packets: 0
> rx31_wqe_err: 0
> rx31_mpwqe_filler_cqes: 0
> rx31_mpwqe_filler_strides: 0
> rx31_buff_alloc_err: 0
> rx31_cqe_compress_blks: 0
> rx31_cqe_compress_pkts: 0
> rx31_page_reuse: 0
> rx31_cache_reuse: 0
> rx31_cache_full: 0
> rx31_cache_empty: 2560
> rx31_cache_busy: 0
> rx31_cache_waive: 0
> rx31_congst_umr: 0
> rx31_arfs_err: 0
> rx31_xdp_tx_xmit: 0
> rx31_xdp_tx_full: 0
> rx31_xdp_tx_err: 0
> rx31_xdp_tx_cqes: 0
> rx32_packets: 0
> rx32_bytes: 0
> rx32_csum_complete: 0
> rx32_csum_unnecessary: 0
> rx32_csum_unnecessary_inner: 0
> rx32_csum_none: 0
> rx32_xdp_drop: 0
> rx32_xdp_redirect: 0
> rx32_lro_packets: 0
> rx32_lro_bytes: 0
> rx32_ecn_mark: 0
> rx32_removed_vlan_packets: 0
> rx32_wqe_err: 0
> rx32_mpwqe_filler_cqes: 0
> rx32_mpwqe_filler_strides: 0
> rx32_buff_alloc_err: 0
> rx32_cqe_compress_blks: 0
> rx32_cqe_compress_pkts: 0
> rx32_page_reuse: 0
> rx32_cache_reuse: 0
> rx32_cache_full: 0
> rx32_cache_empty: 2560
> rx32_cache_busy: 0
> rx32_cache_waive: 0
> rx32_congst_umr: 0
> rx32_arfs_err: 0
> rx32_xdp_tx_xmit: 0
> rx32_xdp_tx_full: 0
> rx32_xdp_tx_err: 0
> rx32_xdp_tx_cqes: 0
> rx33_packets: 0
> rx33_bytes: 0
> rx33_csum_complete: 0
> rx33_csum_unnecessary: 0
> rx33_csum_unnecessary_inner: 0
> rx33_csum_none: 0
> rx33_xdp_drop: 0
> rx33_xdp_redirect: 0
> rx33_lro_packets: 0
> rx33_lro_bytes: 0
> rx33_ecn_mark: 0
> rx33_removed_vlan_packets: 0
> rx33_wqe_err: 0
> rx33_mpwqe_filler_cqes: 0
> rx33_mpwqe_filler_strides: 0
> rx33_buff_alloc_err: 0
> rx33_cqe_compress_blks: 0
> rx33_cqe_compress_pkts: 0
> rx33_page_reuse: 0
> rx33_cache_reuse: 0
> rx33_cache_full: 0
> rx33_cache_empty: 2560
> rx33_cache_busy: 0
> rx33_cache_waive: 0
> rx33_congst_umr: 0
> rx33_arfs_err: 0
> rx33_xdp_tx_xmit: 0
> rx33_xdp_tx_full: 0
> rx33_xdp_tx_err: 0
> rx33_xdp_tx_cqes: 0
> rx34_packets: 0
> rx34_bytes: 0
> rx34_csum_complete: 0
> rx34_csum_unnecessary: 0
> rx34_csum_unnecessary_inner: 0
> rx34_csum_none: 0
> rx34_xdp_drop: 0
> rx34_xdp_redirect: 0
> rx34_lro_packets: 0
> rx34_lro_bytes: 0
> rx34_ecn_mark: 0
> rx34_removed_vlan_packets: 0
> rx34_wqe_err: 0
> rx34_mpwqe_filler_cqes: 0
> rx34_mpwqe_filler_strides: 0
> rx34_buff_alloc_err: 0
> rx34_cqe_compress_blks: 0
> rx34_cqe_compress_pkts: 0
> rx34_page_reuse: 0
> rx34_cache_reuse: 0
> rx34_cache_full: 0
> rx34_cache_empty: 2560
> rx34_cache_busy: 0
> rx34_cache_waive: 0
> rx34_congst_umr: 0
> rx34_arfs_err: 0
> rx34_xdp_tx_xmit: 0
> rx34_xdp_tx_full: 0
> rx34_xdp_tx_err: 0
> rx34_xdp_tx_cqes: 0
> rx35_packets: 0
> rx35_bytes: 0
> rx35_csum_complete: 0
> rx35_csum_unnecessary: 0
> rx35_csum_unnecessary_inner: 0
> rx35_csum_none: 0
> rx35_xdp_drop: 0
> rx35_xdp_redirect: 0
> rx35_lro_packets: 0
> rx35_lro_bytes: 0
> rx35_ecn_mark: 0
> rx35_removed_vlan_packets: 0
> rx35_wqe_err: 0
> rx35_mpwqe_filler_cqes: 0
> rx35_mpwqe_filler_strides: 0
> rx35_buff_alloc_err: 0
> rx35_cqe_compress_blks: 0
> rx35_cqe_compress_pkts: 0
> rx35_page_reuse: 0
> rx35_cache_reuse: 0
> rx35_cache_full: 0
> rx35_cache_empty: 2560
> rx35_cache_busy: 0
> rx35_cache_waive: 0
> rx35_congst_umr: 0
> rx35_arfs_err: 0
> rx35_xdp_tx_xmit: 0
> rx35_xdp_tx_full: 0
> rx35_xdp_tx_err: 0
> rx35_xdp_tx_cqes: 0
> rx36_packets: 0
> rx36_bytes: 0
> rx36_csum_complete: 0
> rx36_csum_unnecessary: 0
> rx36_csum_unnecessary_inner: 0
> rx36_csum_none: 0
> rx36_xdp_drop: 0
> rx36_xdp_redirect: 0
> rx36_lro_packets: 0
> rx36_lro_bytes: 0
> rx36_ecn_mark: 0
> rx36_removed_vlan_packets: 0
> rx36_wqe_err: 0
> rx36_mpwqe_filler_cqes: 0
> rx36_mpwqe_filler_strides: 0
> rx36_buff_alloc_err: 0
> rx36_cqe_compress_blks: 0
> rx36_cqe_compress_pkts: 0
> rx36_page_reuse: 0
> rx36_cache_reuse: 0
> rx36_cache_full: 0
> rx36_cache_empty: 2560
> rx36_cache_busy: 0
> rx36_cache_waive: 0
> rx36_congst_umr: 0
> rx36_arfs_err: 0
> rx36_xdp_tx_xmit: 0
> rx36_xdp_tx_full: 0
> rx36_xdp_tx_err: 0
> rx36_xdp_tx_cqes: 0
> rx37_packets: 0
> rx37_bytes: 0
> rx37_csum_complete: 0
> rx37_csum_unnecessary: 0
> rx37_csum_unnecessary_inner: 0
> rx37_csum_none: 0
> rx37_xdp_drop: 0
> rx37_xdp_redirect: 0
> rx37_lro_packets: 0
> rx37_lro_bytes: 0
> rx37_ecn_mark: 0
> rx37_removed_vlan_packets: 0
> rx37_wqe_err: 0
> rx37_mpwqe_filler_cqes: 0
> rx37_mpwqe_filler_strides: 0
> rx37_buff_alloc_err: 0
> rx37_cqe_compress_blks: 0
> rx37_cqe_compress_pkts: 0
> rx37_page_reuse: 0
> rx37_cache_reuse: 0
> rx37_cache_full: 0
> rx37_cache_empty: 2560
> rx37_cache_busy: 0
> rx37_cache_waive: 0
> rx37_congst_umr: 0
> rx37_arfs_err: 0
> rx37_xdp_tx_xmit: 0
> rx37_xdp_tx_full: 0
> rx37_xdp_tx_err: 0
> rx37_xdp_tx_cqes: 0
> rx38_packets: 0
> rx38_bytes: 0
> rx38_csum_complete: 0
> rx38_csum_unnecessary: 0
> rx38_csum_unnecessary_inner: 0
> rx38_csum_none: 0
> rx38_xdp_drop: 0
> rx38_xdp_redirect: 0
> rx38_lro_packets: 0
> rx38_lro_bytes: 0
> rx38_ecn_mark: 0
> rx38_removed_vlan_packets: 0
> rx38_wqe_err: 0
> rx38_mpwqe_filler_cqes: 0
> rx38_mpwqe_filler_strides: 0
> rx38_buff_alloc_err: 0
> rx38_cqe_compress_blks: 0
> rx38_cqe_compress_pkts: 0
> rx38_page_reuse: 0
> rx38_cache_reuse: 0
> rx38_cache_full: 0
> rx38_cache_empty: 2560
> rx38_cache_busy: 0
> rx38_cache_waive: 0
> rx38_congst_umr: 0
> rx38_arfs_err: 0
> rx38_xdp_tx_xmit: 0
> rx38_xdp_tx_full: 0
> rx38_xdp_tx_err: 0
> rx38_xdp_tx_cqes: 0
> rx39_packets: 0
> rx39_bytes: 0
> rx39_csum_complete: 0
> rx39_csum_unnecessary: 0
> rx39_csum_unnecessary_inner: 0
> rx39_csum_none: 0
> rx39_xdp_drop: 0
> rx39_xdp_redirect: 0
> rx39_lro_packets: 0
> rx39_lro_bytes: 0
> rx39_ecn_mark: 0
> rx39_removed_vlan_packets: 0
> rx39_wqe_err: 0
> rx39_mpwqe_filler_cqes: 0
> rx39_mpwqe_filler_strides: 0
> rx39_buff_alloc_err: 0
> rx39_cqe_compress_blks: 0
> rx39_cqe_compress_pkts: 0
> rx39_page_reuse: 0
> rx39_cache_reuse: 0
> rx39_cache_full: 0
> rx39_cache_empty: 2560
> rx39_cache_busy: 0
> rx39_cache_waive: 0
> rx39_congst_umr: 0
> rx39_arfs_err: 0
> rx39_xdp_tx_xmit: 0
> rx39_xdp_tx_full: 0
> rx39_xdp_tx_err: 0
> rx39_xdp_tx_cqes: 0
> rx40_packets: 0
> rx40_bytes: 0
> rx40_csum_complete: 0
> rx40_csum_unnecessary: 0
> rx40_csum_unnecessary_inner: 0
> rx40_csum_none: 0
> rx40_xdp_drop: 0
> rx40_xdp_redirect: 0
> rx40_lro_packets: 0
> rx40_lro_bytes: 0
> rx40_ecn_mark: 0
> rx40_removed_vlan_packets: 0
> rx40_wqe_err: 0
> rx40_mpwqe_filler_cqes: 0
> rx40_mpwqe_filler_strides: 0
> rx40_buff_alloc_err: 0
> rx40_cqe_compress_blks: 0
> rx40_cqe_compress_pkts: 0
> rx40_page_reuse: 0
> rx40_cache_reuse: 0
> rx40_cache_full: 0
> rx40_cache_empty: 2560
> rx40_cache_busy: 0
> rx40_cache_waive: 0
> rx40_congst_umr: 0
> rx40_arfs_err: 0
> rx40_xdp_tx_xmit: 0
> rx40_xdp_tx_full: 0
> rx40_xdp_tx_err: 0
> rx40_xdp_tx_cqes: 0
> rx41_packets: 0
> rx41_bytes: 0
> rx41_csum_complete: 0
> rx41_csum_unnecessary: 0
> rx41_csum_unnecessary_inner: 0
> rx41_csum_none: 0
> rx41_xdp_drop: 0
> rx41_xdp_redirect: 0
> rx41_lro_packets: 0
> rx41_lro_bytes: 0
> rx41_ecn_mark: 0
> rx41_removed_vlan_packets: 0
> rx41_wqe_err: 0
> rx41_mpwqe_filler_cqes: 0
> rx41_mpwqe_filler_strides: 0
> rx41_buff_alloc_err: 0
> rx41_cqe_compress_blks: 0
> rx41_cqe_compress_pkts: 0
> rx41_page_reuse: 0
> rx41_cache_reuse: 0
> rx41_cache_full: 0
> rx41_cache_empty: 2560
> rx41_cache_busy: 0
> rx41_cache_waive: 0
> rx41_congst_umr: 0
> rx41_arfs_err: 0
> rx41_xdp_tx_xmit: 0
> rx41_xdp_tx_full: 0
> rx41_xdp_tx_err: 0
> rx41_xdp_tx_cqes: 0
> rx42_packets: 0
> rx42_bytes: 0
> rx42_csum_complete: 0
> rx42_csum_unnecessary: 0
> rx42_csum_unnecessary_inner: 0
> rx42_csum_none: 0
> rx42_xdp_drop: 0
> rx42_xdp_redirect: 0
> rx42_lro_packets: 0
> rx42_lro_bytes: 0
> rx42_ecn_mark: 0
> rx42_removed_vlan_packets: 0
> rx42_wqe_err: 0
> rx42_mpwqe_filler_cqes: 0
> rx42_mpwqe_filler_strides: 0
> rx42_buff_alloc_err: 0
> rx42_cqe_compress_blks: 0
> rx42_cqe_compress_pkts: 0
> rx42_page_reuse: 0
> rx42_cache_reuse: 0
> rx42_cache_full: 0
> rx42_cache_empty: 2560
> rx42_cache_busy: 0
> rx42_cache_waive: 0
> rx42_congst_umr: 0
> rx42_arfs_err: 0
> rx42_xdp_tx_xmit: 0
> rx42_xdp_tx_full: 0
> rx42_xdp_tx_err: 0
> rx42_xdp_tx_cqes: 0
> rx43_packets: 0
> rx43_bytes: 0
> rx43_csum_complete: 0
> rx43_csum_unnecessary: 0
> rx43_csum_unnecessary_inner: 0
> rx43_csum_none: 0
> rx43_xdp_drop: 0
> rx43_xdp_redirect: 0
> rx43_lro_packets: 0
> rx43_lro_bytes: 0
> rx43_ecn_mark: 0
> rx43_removed_vlan_packets: 0
> rx43_wqe_err: 0
> rx43_mpwqe_filler_cqes: 0
> rx43_mpwqe_filler_strides: 0
> rx43_buff_alloc_err: 0
> rx43_cqe_compress_blks: 0
> rx43_cqe_compress_pkts: 0
> rx43_page_reuse: 0
> rx43_cache_reuse: 0
> rx43_cache_full: 0
> rx43_cache_empty: 2560
> rx43_cache_busy: 0
> rx43_cache_waive: 0
> rx43_congst_umr: 0
> rx43_arfs_err: 0
> rx43_xdp_tx_xmit: 0
> rx43_xdp_tx_full: 0
> rx43_xdp_tx_err: 0
> rx43_xdp_tx_cqes: 0
> rx44_packets: 0
> rx44_bytes: 0
> rx44_csum_complete: 0
> rx44_csum_unnecessary: 0
> rx44_csum_unnecessary_inner: 0
> rx44_csum_none: 0
> rx44_xdp_drop: 0
> rx44_xdp_redirect: 0
> rx44_lro_packets: 0
> rx44_lro_bytes: 0
> rx44_ecn_mark: 0
> rx44_removed_vlan_packets: 0
> rx44_wqe_err: 0
> rx44_mpwqe_filler_cqes: 0
> rx44_mpwqe_filler_strides: 0
> rx44_buff_alloc_err: 0
> rx44_cqe_compress_blks: 0
> rx44_cqe_compress_pkts: 0
> rx44_page_reuse: 0
> rx44_cache_reuse: 0
> rx44_cache_full: 0
> rx44_cache_empty: 2560
> rx44_cache_busy: 0
> rx44_cache_waive: 0
> rx44_congst_umr: 0
> rx44_arfs_err: 0
> rx44_xdp_tx_xmit: 0
> rx44_xdp_tx_full: 0
> rx44_xdp_tx_err: 0
> rx44_xdp_tx_cqes: 0
> rx45_packets: 0
> rx45_bytes: 0
> rx45_csum_complete: 0
> rx45_csum_unnecessary: 0
> rx45_csum_unnecessary_inner: 0
> rx45_csum_none: 0
> rx45_xdp_drop: 0
> rx45_xdp_redirect: 0
> rx45_lro_packets: 0
> rx45_lro_bytes: 0
> rx45_ecn_mark: 0
> rx45_removed_vlan_packets: 0
> rx45_wqe_err: 0
> rx45_mpwqe_filler_cqes: 0
> rx45_mpwqe_filler_strides: 0
> rx45_buff_alloc_err: 0
> rx45_cqe_compress_blks: 0
> rx45_cqe_compress_pkts: 0
> rx45_page_reuse: 0
> rx45_cache_reuse: 0
> rx45_cache_full: 0
> rx45_cache_empty: 2560
> rx45_cache_busy: 0
> rx45_cache_waive: 0
> rx45_congst_umr: 0
> rx45_arfs_err: 0
> rx45_xdp_tx_xmit: 0
> rx45_xdp_tx_full: 0
> rx45_xdp_tx_err: 0
> rx45_xdp_tx_cqes: 0
> rx46_packets: 0
> rx46_bytes: 0
> rx46_csum_complete: 0
> rx46_csum_unnecessary: 0
> rx46_csum_unnecessary_inner: 0
> rx46_csum_none: 0
> rx46_xdp_drop: 0
> rx46_xdp_redirect: 0
> rx46_lro_packets: 0
> rx46_lro_bytes: 0
> rx46_ecn_mark: 0
> rx46_removed_vlan_packets: 0
> rx46_wqe_err: 0
> rx46_mpwqe_filler_cqes: 0
> rx46_mpwqe_filler_strides: 0
> rx46_buff_alloc_err: 0
> rx46_cqe_compress_blks: 0
> rx46_cqe_compress_pkts: 0
> rx46_page_reuse: 0
> rx46_cache_reuse: 0
> rx46_cache_full: 0
> rx46_cache_empty: 2560
> rx46_cache_busy: 0
> rx46_cache_waive: 0
> rx46_congst_umr: 0
> rx46_arfs_err: 0
> rx46_xdp_tx_xmit: 0
> rx46_xdp_tx_full: 0
> rx46_xdp_tx_err: 0
> rx46_xdp_tx_cqes: 0
> rx47_packets: 0
> rx47_bytes: 0
> rx47_csum_complete: 0
> rx47_csum_unnecessary: 0
> rx47_csum_unnecessary_inner: 0
> rx47_csum_none: 0
> rx47_xdp_drop: 0
> rx47_xdp_redirect: 0
> rx47_lro_packets: 0
> rx47_lro_bytes: 0
> rx47_ecn_mark: 0
> rx47_removed_vlan_packets: 0
> rx47_wqe_err: 0
> rx47_mpwqe_filler_cqes: 0
> rx47_mpwqe_filler_strides: 0
> rx47_buff_alloc_err: 0
> rx47_cqe_compress_blks: 0
> rx47_cqe_compress_pkts: 0
> rx47_page_reuse: 0
> rx47_cache_reuse: 0
> rx47_cache_full: 0
> rx47_cache_empty: 2560
> rx47_cache_busy: 0
> rx47_cache_waive: 0
> rx47_congst_umr: 0
> rx47_arfs_err: 0
> rx47_xdp_tx_xmit: 0
> rx47_xdp_tx_full: 0
> rx47_xdp_tx_err: 0
> rx47_xdp_tx_cqes: 0
> rx48_packets: 0
> rx48_bytes: 0
> rx48_csum_complete: 0
> rx48_csum_unnecessary: 0
> rx48_csum_unnecessary_inner: 0
> rx48_csum_none: 0
> rx48_xdp_drop: 0
> rx48_xdp_redirect: 0
> rx48_lro_packets: 0
> rx48_lro_bytes: 0
> rx48_ecn_mark: 0
> rx48_removed_vlan_packets: 0
> rx48_wqe_err: 0
> rx48_mpwqe_filler_cqes: 0
> rx48_mpwqe_filler_strides: 0
> rx48_buff_alloc_err: 0
> rx48_cqe_compress_blks: 0
> rx48_cqe_compress_pkts: 0
> rx48_page_reuse: 0
> rx48_cache_reuse: 0
> rx48_cache_full: 0
> rx48_cache_empty: 2560
> rx48_cache_busy: 0
> rx48_cache_waive: 0
> rx48_congst_umr: 0
> rx48_arfs_err: 0
> rx48_xdp_tx_xmit: 0
> rx48_xdp_tx_full: 0
> rx48_xdp_tx_err: 0
> rx48_xdp_tx_cqes: 0
> rx49_packets: 0
> rx49_bytes: 0
> rx49_csum_complete: 0
> rx49_csum_unnecessary: 0
> rx49_csum_unnecessary_inner: 0
> rx49_csum_none: 0
> rx49_xdp_drop: 0
> rx49_xdp_redirect: 0
> rx49_lro_packets: 0
> rx49_lro_bytes: 0
> rx49_ecn_mark: 0
> rx49_removed_vlan_packets: 0
> rx49_wqe_err: 0
> rx49_mpwqe_filler_cqes: 0
> rx49_mpwqe_filler_strides: 0
> rx49_buff_alloc_err: 0
> rx49_cqe_compress_blks: 0
> rx49_cqe_compress_pkts: 0
> rx49_page_reuse: 0
> rx49_cache_reuse: 0
> rx49_cache_full: 0
> rx49_cache_empty: 2560
> rx49_cache_busy: 0
> rx49_cache_waive: 0
> rx49_congst_umr: 0
> rx49_arfs_err: 0
> rx49_xdp_tx_xmit: 0
> rx49_xdp_tx_full: 0
> rx49_xdp_tx_err: 0
> rx49_xdp_tx_cqes: 0
> rx50_packets: 0
> rx50_bytes: 0
> rx50_csum_complete: 0
> rx50_csum_unnecessary: 0
> rx50_csum_unnecessary_inner: 0
> rx50_csum_none: 0
> rx50_xdp_drop: 0
> rx50_xdp_redirect: 0
> rx50_lro_packets: 0
> rx50_lro_bytes: 0
> rx50_ecn_mark: 0
> rx50_removed_vlan_packets: 0
> rx50_wqe_err: 0
> rx50_mpwqe_filler_cqes: 0
> rx50_mpwqe_filler_strides: 0
> rx50_buff_alloc_err: 0
> rx50_cqe_compress_blks: 0
> rx50_cqe_compress_pkts: 0
> rx50_page_reuse: 0
> rx50_cache_reuse: 0
> rx50_cache_full: 0
> rx50_cache_empty: 2560
> rx50_cache_busy: 0
> rx50_cache_waive: 0
> rx50_congst_umr: 0
> rx50_arfs_err: 0
> rx50_xdp_tx_xmit: 0
> rx50_xdp_tx_full: 0
> rx50_xdp_tx_err: 0
> rx50_xdp_tx_cqes: 0
> rx51_packets: 0
> rx51_bytes: 0
> rx51_csum_complete: 0
> rx51_csum_unnecessary: 0
> rx51_csum_unnecessary_inner: 0
> rx51_csum_none: 0
> rx51_xdp_drop: 0
> rx51_xdp_redirect: 0
> rx51_lro_packets: 0
> rx51_lro_bytes: 0
> rx51_ecn_mark: 0
> rx51_removed_vlan_packets: 0
> rx51_wqe_err: 0
> rx51_mpwqe_filler_cqes: 0
> rx51_mpwqe_filler_strides: 0
> rx51_buff_alloc_err: 0
> rx51_cqe_compress_blks: 0
> rx51_cqe_compress_pkts: 0
> rx51_page_reuse: 0
> rx51_cache_reuse: 0
> rx51_cache_full: 0
> rx51_cache_empty: 2560
> rx51_cache_busy: 0
> rx51_cache_waive: 0
> rx51_congst_umr: 0
> rx51_arfs_err: 0
> rx51_xdp_tx_xmit: 0
> rx51_xdp_tx_full: 0
> rx51_xdp_tx_err: 0
> rx51_xdp_tx_cqes: 0
> rx52_packets: 0
> rx52_bytes: 0
> rx52_csum_complete: 0
> rx52_csum_unnecessary: 0
> rx52_csum_unnecessary_inner: 0
> rx52_csum_none: 0
> rx52_xdp_drop: 0
> rx52_xdp_redirect: 0
> rx52_lro_packets: 0
> rx52_lro_bytes: 0
> rx52_ecn_mark: 0
> rx52_removed_vlan_packets: 0
> rx52_wqe_err: 0
> rx52_mpwqe_filler_cqes: 0
> rx52_mpwqe_filler_strides: 0
> rx52_buff_alloc_err: 0
> rx52_cqe_compress_blks: 0
> rx52_cqe_compress_pkts: 0
> rx52_page_reuse: 0
> rx52_cache_reuse: 0
> rx52_cache_full: 0
> rx52_cache_empty: 2560
> rx52_cache_busy: 0
> rx52_cache_waive: 0
> rx52_congst_umr: 0
> rx52_arfs_err: 0
> rx52_xdp_tx_xmit: 0
> rx52_xdp_tx_full: 0
> rx52_xdp_tx_err: 0
> rx52_xdp_tx_cqes: 0
> rx53_packets: 0
> rx53_bytes: 0
> rx53_csum_complete: 0
> rx53_csum_unnecessary: 0
> rx53_csum_unnecessary_inner: 0
> rx53_csum_none: 0
> rx53_xdp_drop: 0
> rx53_xdp_redirect: 0
> rx53_lro_packets: 0
> rx53_lro_bytes: 0
> rx53_ecn_mark: 0
> rx53_removed_vlan_packets: 0
> rx53_wqe_err: 0
> rx53_mpwqe_filler_cqes: 0
> rx53_mpwqe_filler_strides: 0
> rx53_buff_alloc_err: 0
> rx53_cqe_compress_blks: 0
> rx53_cqe_compress_pkts: 0
> rx53_page_reuse: 0
> rx53_cache_reuse: 0
> rx53_cache_full: 0
> rx53_cache_empty: 2560
> rx53_cache_busy: 0
> rx53_cache_waive: 0
> rx53_congst_umr: 0
> rx53_arfs_err: 0
> rx53_xdp_tx_xmit: 0
> rx53_xdp_tx_full: 0
> rx53_xdp_tx_err: 0
> rx53_xdp_tx_cqes: 0
> rx54_packets: 0
> rx54_bytes: 0
> rx54_csum_complete: 0
> rx54_csum_unnecessary: 0
> rx54_csum_unnecessary_inner: 0
> rx54_csum_none: 0
> rx54_xdp_drop: 0
> rx54_xdp_redirect: 0
> rx54_lro_packets: 0
> rx54_lro_bytes: 0
> rx54_ecn_mark: 0
> rx54_removed_vlan_packets: 0
> rx54_wqe_err: 0
> rx54_mpwqe_filler_cqes: 0
> rx54_mpwqe_filler_strides: 0
> rx54_buff_alloc_err: 0
> rx54_cqe_compress_blks: 0
> rx54_cqe_compress_pkts: 0
> rx54_page_reuse: 0
> rx54_cache_reuse: 0
> rx54_cache_full: 0
> rx54_cache_empty: 2560
> rx54_cache_busy: 0
> rx54_cache_waive: 0
> rx54_congst_umr: 0
> rx54_arfs_err: 0
> rx54_xdp_tx_xmit: 0
> rx54_xdp_tx_full: 0
> rx54_xdp_tx_err: 0
> rx54_xdp_tx_cqes: 0
> rx55_packets: 0
> rx55_bytes: 0
> rx55_csum_complete: 0
> rx55_csum_unnecessary: 0
> rx55_csum_unnecessary_inner: 0
> rx55_csum_none: 0
> rx55_xdp_drop: 0
> rx55_xdp_redirect: 0
> rx55_lro_packets: 0
> rx55_lro_bytes: 0
> rx55_ecn_mark: 0
> rx55_removed_vlan_packets: 0
> rx55_wqe_err: 0
> rx55_mpwqe_filler_cqes: 0
> rx55_mpwqe_filler_strides: 0
> rx55_buff_alloc_err: 0
> rx55_cqe_compress_blks: 0
> rx55_cqe_compress_pkts: 0
> rx55_page_reuse: 0
> rx55_cache_reuse: 0
> rx55_cache_full: 0
> rx55_cache_empty: 2560
> rx55_cache_busy: 0
> rx55_cache_waive: 0
> rx55_congst_umr: 0
> rx55_arfs_err: 0
> rx55_xdp_tx_xmit: 0
> rx55_xdp_tx_full: 0
> rx55_xdp_tx_err: 0
> rx55_xdp_tx_cqes: 0
> tx0_packets: 6019477917
> tx0_bytes: 3445238940825
> tx0_tso_packets: 311304622
> tx0_tso_bytes: 1897094773213
> tx0_tso_inner_packets: 0
> tx0_tso_inner_bytes: 0
> tx0_csum_partial: 457981794
> tx0_csum_partial_inner: 0
> tx0_added_vlan_packets: 4965567654
> tx0_nop: 72290329
> tx0_csum_none: 4507585860
> tx0_stopped: 9118
> tx0_dropped: 0
> tx0_xmit_more: 51651593
> tx0_recover: 0
> tx0_cqes: 4913918402
> tx0_wake: 9118
> tx0_cqe_err: 0
> tx1_packets: 5700413414
> tx1_bytes: 3340870662350
> tx1_tso_packets: 318201557
> tx1_tso_bytes: 1915233462303
> tx1_tso_inner_packets: 0
> tx1_tso_inner_bytes: 0
> tx1_csum_partial: 461736722
> tx1_csum_partial_inner: 0
> tx1_added_vlan_packets: 4638708749
> tx1_nop: 70061796
> tx1_csum_none: 4176972027
> tx1_stopped: 9248
> tx1_dropped: 0
> tx1_xmit_more: 39531959
> tx1_recover: 0
> tx1_cqes: 4599179178
> tx1_wake: 9248
> tx1_cqe_err: 0
> tx2_packets: 5795960848
> tx2_bytes: 3394876820271
> tx2_tso_packets: 322935065
> tx2_tso_bytes: 1910825901109
> tx2_tso_inner_packets: 0
> tx2_tso_inner_bytes: 0
> tx2_csum_partial: 460747092
> tx2_csum_partial_inner: 0
> tx2_added_vlan_packets: 4743705654
> tx2_nop: 72722430
> tx2_csum_none: 4282958562
> tx2_stopped: 8938
> tx2_dropped: 0
> tx2_xmit_more: 44084718
> tx2_recover: 0
> tx2_cqes: 4699623410
> tx2_wake: 8938
> tx2_cqe_err: 0
> tx3_packets: 5580215878
> tx3_bytes: 3191677257787
> tx3_tso_packets: 305771141
> tx3_tso_bytes: 1823265793476
> tx3_tso_inner_packets: 0
> tx3_tso_inner_bytes: 0
> tx3_csum_partial: 434976070
> tx3_csum_partial_inner: 0
> tx3_added_vlan_packets: 4569899956
> tx3_nop: 68184348
> tx3_csum_none: 4134923886
> tx3_stopped: 8383
> tx3_dropped: 0
> tx3_xmit_more: 41940375
> tx3_recover: 0
> tx3_cqes: 4527961924
> tx3_wake: 8383
> tx3_cqe_err: 0
> tx4_packets: 6795007068
> tx4_bytes: 3963890025270
> tx4_tso_packets: 358437617
> tx4_tso_bytes: 2154747995355
> tx4_tso_inner_packets: 0
> tx4_tso_inner_bytes: 0
> tx4_csum_partial: 504764524
> tx4_csum_partial_inner: 0
> tx4_added_vlan_packets: 5602510191
> tx4_nop: 81345604
> tx4_csum_none: 5097745667
> tx4_stopped: 10248
> tx4_dropped: 0
> tx4_xmit_more: 49068571
> tx4_recover: 0
> tx4_cqes: 5553444276
> tx4_wake: 10248
> tx4_cqe_err: 0
> tx5_packets: 6408089261
> tx5_bytes: 3676275848279
> tx5_tso_packets: 345129329
> tx5_tso_bytes: 2108447877473
> tx5_tso_inner_packets: 0
> tx5_tso_inner_bytes: 0
> tx5_csum_partial: 494705894
> tx5_csum_partial_inner: 0
> tx5_added_vlan_packets: 5235998343
> tx5_nop: 77694627
> tx5_csum_none: 4741292449
> tx5_stopped: 46
> tx5_dropped: 0
> tx5_xmit_more: 46675831
> tx5_recover: 0
> tx5_cqes: 5189323550
> tx5_wake: 46
> tx5_cqe_err: 0
> tx6_packets: 6382289663
> tx6_bytes: 3670991418150
> tx6_tso_packets: 342927826
> tx6_tso_bytes: 2075049679904
> tx6_tso_inner_packets: 0
> tx6_tso_inner_bytes: 0
> tx6_csum_partial: 490369221
> tx6_csum_partial_inner: 0
> tx6_added_vlan_packets: 5232144528
> tx6_nop: 77391246
> tx6_csum_none: 4741775307
> tx6_stopped: 10823
> tx6_dropped: 0
> tx6_xmit_more: 44487607
> tx6_recover: 0
> tx6_cqes: 5187659877
> tx6_wake: 10823
> tx6_cqe_err: 0
> tx7_packets: 6456378284
> tx7_bytes: 3758013320518
> tx7_tso_packets: 350958294
> tx7_tso_bytes: 2126833408524
> tx7_tso_inner_packets: 0
> tx7_tso_inner_bytes: 0
> tx7_csum_partial: 501804109
> tx7_csum_partial_inner: 0
> tx7_added_vlan_packets: 5275635204
> tx7_nop: 79010883
> tx7_csum_none: 4773831096
> tx7_stopped: 14684
> tx7_dropped: 0
> tx7_xmit_more: 44447469
> tx7_recover: 0
> tx7_cqes: 5231191770
> tx7_wake: 14684
> tx7_cqe_err: 0
> tx8_packets: 6401799768
> tx8_bytes: 3681210808766
> tx8_tso_packets: 342878228
> tx8_tso_bytes: 2089688012191
> tx8_tso_inner_packets: 0
> tx8_tso_inner_bytes: 0
> tx8_csum_partial: 494865145
> tx8_csum_partial_inner: 0
> tx8_added_vlan_packets: 5242288908
> tx8_nop: 77250910
> tx8_csum_none: 4747423763
> tx8_stopped: 2
> tx8_dropped: 0
> tx8_xmit_more: 44191737
> tx8_recover: 0
> tx8_cqes: 5198098454
> tx8_wake: 2
> tx8_cqe_err: 0
> tx9_packets: 6632882888
> tx9_bytes: 3820110338309
> tx9_tso_packets: 354189056
> tx9_tso_bytes: 2187883597128
> tx9_tso_inner_packets: 0
> tx9_tso_inner_bytes: 0
> tx9_csum_partial: 511108218
> tx9_csum_partial_inner: 0
> tx9_added_vlan_packets: 5413836353
> tx9_nop: 80560668
> tx9_csum_none: 4902728135
> tx9_stopped: 9091
> tx9_dropped: 0
> tx9_xmit_more: 54501293
> tx9_recover: 0
> tx9_cqes: 5359337638
> tx9_wake: 9091
> tx9_cqe_err: 0
> tx10_packets: 6421786406
> tx10_bytes: 3692798413429
> tx10_tso_packets: 346878943
> tx10_tso_bytes: 2111921062110
> tx10_tso_inner_packets: 0
> tx10_tso_inner_bytes: 0
> tx10_csum_partial: 494356645
> tx10_csum_partial_inner: 0
> tx10_added_vlan_packets: 5248274374
> tx10_nop: 77922624
> tx10_csum_none: 4753917730
> tx10_stopped: 9617
> tx10_dropped: 0
> tx10_xmit_more: 44473939
> tx10_recover: 0
> tx10_cqes: 5203802547
> tx10_wake: 9617
> tx10_cqe_err: 0
> tx11_packets: 6406750938
> tx11_bytes: 3660343565126
> tx11_tso_packets: 355917271
> tx11_tso_bytes: 2130812246956
> tx11_tso_inner_packets: 0
> tx11_tso_inner_bytes: 0
> tx11_csum_partial: 500336369
> tx11_csum_partial_inner: 0
> tx11_added_vlan_packets: 5228267547
> tx11_nop: 78906315
> tx11_csum_none: 4727931178
> tx11_stopped: 9607
> tx11_dropped: 0
> tx11_xmit_more: 40041492
> tx11_recover: 0
> tx11_cqes: 5188228290
> tx11_wake: 9607
> tx11_cqe_err: 0
> tx12_packets: 6422347846
> tx12_bytes: 3718772753227
> tx12_tso_packets: 355397223
> tx12_tso_bytes: 2162614059758
> tx12_tso_inner_packets: 0
> tx12_tso_inner_bytes: 0
> tx12_csum_partial: 511437844
> tx12_csum_partial_inner: 0
> tx12_added_vlan_packets: 5221373746
> tx12_nop: 78866779
> tx12_csum_none: 4709935902
> tx12_stopped: 10280
> tx12_dropped: 0
> tx12_xmit_more: 42189399
> tx12_recover: 0
> tx12_cqes: 5179187154
> tx12_wake: 10280
> tx12_cqe_err: 0
> tx13_packets: 6429383816
> tx13_bytes: 3725679445046
> tx13_tso_packets: 360934759
> tx13_tso_bytes: 2148016411436
> tx13_tso_inner_packets: 0
> tx13_tso_inner_bytes: 0
> tx13_csum_partial: 505245849
> tx13_csum_partial_inner: 0
> tx13_added_vlan_packets: 5240267441
> tx13_nop: 80295637
> tx13_csum_none: 4735021592
> tx13_stopped: 84
> tx13_dropped: 0
> tx13_xmit_more: 43118045
> tx13_recover: 0
> tx13_cqes: 5197150348
> tx13_wake: 84
> tx13_cqe_err: 0
> tx14_packets: 6375279148
> tx14_bytes: 3624267203336
> tx14_tso_packets: 344388148
> tx14_tso_bytes: 2094966273548
> tx14_tso_inner_packets: 0
> tx14_tso_inner_bytes: 0
> tx14_csum_partial: 494129407
> tx14_csum_partial_inner: 0
> tx14_added_vlan_packets: 5210749337
> tx14_nop: 77280615
> tx14_csum_none: 4716619930
> tx14_stopped: 13057
> tx14_dropped: 0
> tx14_xmit_more: 40849682
> tx14_recover: 0
> tx14_cqes: 5169902694
> tx14_wake: 13057
> tx14_cqe_err: 0
> tx15_packets: 6489306520
> tx15_bytes: 3775716194795
> tx15_tso_packets: 368716406
> tx15_tso_bytes: 2165876423354
> tx15_tso_inner_packets: 0
> tx15_tso_inner_bytes: 0
> tx15_csum_partial: 509887864
> tx15_csum_partial_inner: 0
> tx15_added_vlan_packets: 5296767390
> tx15_nop: 80803468
> tx15_csum_none: 4786879529
> tx15_stopped: 1
> tx15_dropped: 0
> tx15_xmit_more: 46979676
> tx15_recover: 0
> tx15_cqes: 5249789328
> tx15_wake: 1
> tx15_cqe_err: 0
> tx16_packets: 6559857761
> tx16_bytes: 3724080573905
> tx16_tso_packets: 350864176
> tx16_tso_bytes: 2099634006033
> tx16_tso_inner_packets: 0
> tx16_tso_inner_bytes: 0
> tx16_csum_partial: 489397232
> tx16_csum_partial_inner: 0
> tx16_added_vlan_packets: 5398869334
> tx16_nop: 79046075
> tx16_csum_none: 4909472106
> tx16_stopped: 4480
> tx16_dropped: 0
> tx16_xmit_more: 47273286
> tx16_recover: 0
> tx16_cqes: 5351598315
> tx16_wake: 4480
> tx16_cqe_err: 0
> tx17_packets: 6358711533
> tx17_bytes: 3650180865573
> tx17_tso_packets: 350723136
> tx17_tso_bytes: 2109426587128
> tx17_tso_inner_packets: 0
> tx17_tso_inner_bytes: 0
> tx17_csum_partial: 494719487
> tx17_csum_partial_inner: 0
> tx17_added_vlan_packets: 5190068796
> tx17_nop: 77285612
> tx17_csum_none: 4695349309
> tx17_stopped: 10443
> tx17_dropped: 0
> tx17_xmit_more: 45582108
> tx17_recover: 0
> tx17_cqes: 5144489363
> tx17_wake: 10443
> tx17_cqe_err: 0
> tx18_packets: 6655328437
> tx18_bytes: 3801768461807
> tx18_tso_packets: 356516373
> tx18_tso_bytes: 2164829247550
> tx18_tso_inner_packets: 0
> tx18_tso_inner_bytes: 0
> tx18_csum_partial: 500508446
> tx18_csum_partial_inner: 0
> tx18_added_vlan_packets: 5454166840
> tx18_nop: 80423007
> tx18_csum_none: 4953658394
> tx18_stopped: 14760
> tx18_dropped: 0
> tx18_xmit_more: 50837465
> tx18_recover: 0
> tx18_cqes: 5403332553
> tx18_wake: 14760
> tx18_cqe_err: 0
> tx19_packets: 6408680611
> tx19_bytes: 3644119934372
> tx19_tso_packets: 350727530
> tx19_tso_bytes: 2089896715365
> tx19_tso_inner_packets: 0
> tx19_tso_inner_bytes: 0
> tx19_csum_partial: 486536490
> tx19_csum_partial_inner: 0
> tx19_added_vlan_packets: 5255839020
> tx19_nop: 78525198
> tx19_csum_none: 4769302530
> tx19_stopped: 8614
> tx19_dropped: 0
> tx19_xmit_more: 43605232
> tx19_recover: 0
> tx19_cqes: 5212236833
> tx19_wake: 8614
> tx19_cqe_err: 0
> tx20_packets: 5609275141
> tx20_bytes: 3187279031581
> tx20_tso_packets: 298609303
> tx20_tso_bytes: 1794382229379
> tx20_tso_inner_packets: 0
> tx20_tso_inner_bytes: 0
> tx20_csum_partial: 430691178
> tx20_csum_partial_inner: 0
> tx20_added_vlan_packets: 4616844286
> tx20_nop: 67450040
> tx20_csum_none: 4186153108
> tx20_stopped: 9099
> tx20_dropped: 0
> tx20_xmit_more: 42040991
> tx20_recover: 0
> tx20_cqes: 4574805846
> tx20_wake: 9099
> tx20_cqe_err: 0
> tx21_packets: 5641621183
> tx21_bytes: 3279282331124
> tx21_tso_packets: 311297057
> tx21_tso_bytes: 1875735401012
> tx21_tso_inner_packets: 0
> tx21_tso_inner_bytes: 0
> tx21_csum_partial: 444333894
> tx21_csum_partial_inner: 0
> tx21_added_vlan_packets: 4603527701
> tx21_nop: 68857983
> tx21_csum_none: 4159193807
> tx21_stopped: 10082
> tx21_dropped: 0
> tx21_xmit_more: 43988081
> tx21_recover: 0
> tx21_cqes: 4559542410
> tx21_wake: 10082
> tx21_cqe_err: 0
> tx22_packets: 5822168288
> tx22_bytes: 3452026726862
> tx22_tso_packets: 308230791
> tx22_tso_bytes: 1859686450671
> tx22_tso_inner_packets: 0
> tx22_tso_inner_bytes: 0
> tx22_csum_partial: 442751518
> tx22_csum_partial_inner: 0
> tx22_added_vlan_packets: 4792100335
> tx22_nop: 70631706
> tx22_csum_none: 4349348817
> tx22_stopped: 9355
> tx22_dropped: 0
> tx22_xmit_more: 45165994
> tx22_recover: 0
> tx22_cqes: 4746936601
> tx22_wake: 9355
> tx22_cqe_err: 0
> tx23_packets: 5664896066
> tx23_bytes: 3207724186946
> tx23_tso_packets: 300418757
> tx23_tso_bytes: 1794180478679
> tx23_tso_inner_packets: 0
> tx23_tso_inner_bytes: 0
> tx23_csum_partial: 429898848
> tx23_csum_partial_inner: 0
> tx23_added_vlan_packets: 4674317320
> tx23_nop: 67899896
> tx23_csum_none: 4244418472
> tx23_stopped: 11684
> tx23_dropped: 0
> tx23_xmit_more: 43351132
> tx23_recover: 0
> tx23_cqes: 4630969028
> tx23_wake: 11684
> tx23_cqe_err: 0
> tx24_packets: 5663326601
> tx24_bytes: 3250127095110
> tx24_tso_packets: 301327422
> tx24_tso_bytes: 1831260534157
> tx24_tso_inner_packets: 0
> tx24_tso_inner_bytes: 0
> tx24_csum_partial: 438757312
> tx24_csum_partial_inner: 0
> tx24_added_vlan_packets: 4646014986
> tx24_nop: 68431153
> tx24_csum_none: 4207257674
> tx24_stopped: 9240
> tx24_dropped: 0
> tx24_xmit_more: 47699542
> tx24_recover: 0
> tx24_cqes: 4598317913
> tx24_wake: 9240
> tx24_cqe_err: 0
> tx25_packets: 5703883962
> tx25_bytes: 3291856915695
> tx25_tso_packets: 308900318
> tx25_tso_bytes: 1855516128386
> tx25_tso_inner_packets: 0
> tx25_tso_inner_bytes: 0
> tx25_csum_partial: 444753744
> tx25_csum_partial_inner: 0
> tx25_added_vlan_packets: 4676528924
> tx25_nop: 69230967
> tx25_csum_none: 4231775180
> tx25_stopped: 1140
> tx25_dropped: 0
> tx25_xmit_more: 40819195
> tx25_recover: 0
> tx25_cqes: 4635710966
> tx25_wake: 1140
> tx25_cqe_err: 0
> tx26_packets: 5803495984
> tx26_bytes: 3413564272139
> tx26_tso_packets: 319986230
> tx26_tso_bytes: 1929042839677
> tx26_tso_inner_packets: 0
> tx26_tso_inner_bytes: 0
> tx26_csum_partial: 464771163
> tx26_csum_partial_inner: 0
> tx26_added_vlan_packets: 4734767280
> tx26_nop: 71345080
> tx26_csum_none: 4269996117
> tx26_stopped: 10972
> tx26_dropped: 0
> tx26_xmit_more: 43793424
> tx26_recover: 0
> tx26_cqes: 4690976400
> tx26_wake: 10972
> tx26_cqe_err: 0
> tx27_packets: 5960955343
> tx27_bytes: 3444156164526
> tx27_tso_packets: 325099639
> tx27_tso_bytes: 1928378678784
> tx27_tso_inner_packets: 0
> tx27_tso_inner_bytes: 0
> tx27_csum_partial: 467310289
> tx27_csum_partial_inner: 0
> tx27_added_vlan_packets: 4888651368
> tx27_nop: 73201664
> tx27_csum_none: 4421341079
> tx27_stopped: 9465
> tx27_dropped: 0
> tx27_xmit_more: 53632121
> tx27_recover: 0
> tx27_cqes: 4835021398
> tx27_wake: 9465
> tx27_cqe_err: 0
> tx28_packets: 0
> tx28_bytes: 0
> tx28_tso_packets: 0
> tx28_tso_bytes: 0
> tx28_tso_inner_packets: 0
> tx28_tso_inner_bytes: 0
> tx28_csum_partial: 0
> tx28_csum_partial_inner: 0
> tx28_added_vlan_packets: 0
> tx28_nop: 0
> tx28_csum_none: 0
> tx28_stopped: 0
> tx28_dropped: 0
> tx28_xmit_more: 0
> tx28_recover: 0
> tx28_cqes: 0
> tx28_wake: 0
> tx28_cqe_err: 0
> tx29_packets: 3
> tx29_bytes: 266
> tx29_tso_packets: 0
> tx29_tso_bytes: 0
> tx29_tso_inner_packets: 0
> tx29_tso_inner_bytes: 0
> tx29_csum_partial: 0
> tx29_csum_partial_inner: 0
> tx29_added_vlan_packets: 0
> tx29_nop: 0
> tx29_csum_none: 3
> tx29_stopped: 0
> tx29_dropped: 0
> tx29_xmit_more: 1
> tx29_recover: 0
> tx29_cqes: 2
> tx29_wake: 0
> tx29_cqe_err: 0
> tx30_packets: 0
> tx30_bytes: 0
> tx30_tso_packets: 0
> tx30_tso_bytes: 0
> tx30_tso_inner_packets: 0
> tx30_tso_inner_bytes: 0
> tx30_csum_partial: 0
> tx30_csum_partial_inner: 0
> tx30_added_vlan_packets: 0
> tx30_nop: 0
> tx30_csum_none: 0
> tx30_stopped: 0
> tx30_dropped: 0
> tx30_xmit_more: 0
> tx30_recover: 0
> tx30_cqes: 0
> tx30_wake: 0
> tx30_cqe_err: 0
> tx31_packets: 0
> tx31_bytes: 0
> tx31_tso_packets: 0
> tx31_tso_bytes: 0
> tx31_tso_inner_packets: 0
> tx31_tso_inner_bytes: 0
> tx31_csum_partial: 0
> tx31_csum_partial_inner: 0
> tx31_added_vlan_packets: 0
> tx31_nop: 0
> tx31_csum_none: 0
> tx31_stopped: 0
> tx31_dropped: 0
> tx31_xmit_more: 0
> tx31_recover: 0
> tx31_cqes: 0
> tx31_wake: 0
> tx31_cqe_err: 0
> tx32_packets: 0
> tx32_bytes: 0
> tx32_tso_packets: 0
> tx32_tso_bytes: 0
> tx32_tso_inner_packets: 0
> tx32_tso_inner_bytes: 0
> tx32_csum_partial: 0
> tx32_csum_partial_inner: 0
> tx32_added_vlan_packets: 0
> tx32_nop: 0
> tx32_csum_none: 0
> tx32_stopped: 0
> tx32_dropped: 0
> tx32_xmit_more: 0
> tx32_recover: 0
> tx32_cqes: 0
> tx32_wake: 0
> tx32_cqe_err: 0
> tx33_packets: 0
> tx33_bytes: 0
> tx33_tso_packets: 0
> tx33_tso_bytes: 0
> tx33_tso_inner_packets: 0
> tx33_tso_inner_bytes: 0
> tx33_csum_partial: 0
> tx33_csum_partial_inner: 0
> tx33_added_vlan_packets: 0
> tx33_nop: 0
> tx33_csum_none: 0
> tx33_stopped: 0
> tx33_dropped: 0
> tx33_xmit_more: 0
> tx33_recover: 0
> tx33_cqes: 0
> tx33_wake: 0
> tx33_cqe_err: 0
> tx34_packets: 0
> tx34_bytes: 0
> tx34_tso_packets: 0
> tx34_tso_bytes: 0
> tx34_tso_inner_packets: 0
> tx34_tso_inner_bytes: 0
> tx34_csum_partial: 0
> tx34_csum_partial_inner: 0
> tx34_added_vlan_packets: 0
> tx34_nop: 0
> tx34_csum_none: 0
> tx34_stopped: 0
> tx34_dropped: 0
> tx34_xmit_more: 0
> tx34_recover: 0
> tx34_cqes: 0
> tx34_wake: 0
> tx34_cqe_err: 0
> tx35_packets: 0
> tx35_bytes: 0
> tx35_tso_packets: 0
> tx35_tso_bytes: 0
> tx35_tso_inner_packets: 0
> tx35_tso_inner_bytes: 0
> tx35_csum_partial: 0
> tx35_csum_partial_inner: 0
> tx35_added_vlan_packets: 0
> tx35_nop: 0
> tx35_csum_none: 0
> tx35_stopped: 0
> tx35_dropped: 0
> tx35_xmit_more: 0
> tx35_recover: 0
> tx35_cqes: 0
> tx35_wake: 0
> tx35_cqe_err: 0
> tx36_packets: 0
> tx36_bytes: 0
> tx36_tso_packets: 0
> tx36_tso_bytes: 0
> tx36_tso_inner_packets: 0
> tx36_tso_inner_bytes: 0
> tx36_csum_partial: 0
> tx36_csum_partial_inner: 0
> tx36_added_vlan_packets: 0
> tx36_nop: 0
> tx36_csum_none: 0
> tx36_stopped: 0
> tx36_dropped: 0
> tx36_xmit_more: 0
> tx36_recover: 0
> tx36_cqes: 0
> tx36_wake: 0
> tx36_cqe_err: 0
> tx37_packets: 0
> tx37_bytes: 0
> tx37_tso_packets: 0
> tx37_tso_bytes: 0
> tx37_tso_inner_packets: 0
> tx37_tso_inner_bytes: 0
> tx37_csum_partial: 0
> tx37_csum_partial_inner: 0
> tx37_added_vlan_packets: 0
> tx37_nop: 0
> tx37_csum_none: 0
> tx37_stopped: 0
> tx37_dropped: 0
> tx37_xmit_more: 0
> tx37_recover: 0
> tx37_cqes: 0
> tx37_wake: 0
> tx37_cqe_err: 0
> tx38_packets: 0
> tx38_bytes: 0
> tx38_tso_packets: 0
> tx38_tso_bytes: 0
> tx38_tso_inner_packets: 0
> tx38_tso_inner_bytes: 0
> tx38_csum_partial: 0
> tx38_csum_partial_inner: 0
> tx38_added_vlan_packets: 0
> tx38_nop: 0
> tx38_csum_none: 0
> tx38_stopped: 0
> tx38_dropped: 0
> tx38_xmit_more: 0
> tx38_recover: 0
> tx38_cqes: 0
> tx38_wake: 0
> tx38_cqe_err: 0
> tx39_packets: 0
> tx39_bytes: 0
> tx39_tso_packets: 0
> tx39_tso_bytes: 0
> tx39_tso_inner_packets: 0
> tx39_tso_inner_bytes: 0
> tx39_csum_partial: 0
> tx39_csum_partial_inner: 0
> tx39_added_vlan_packets: 0
> tx39_nop: 0
> tx39_csum_none: 0
> tx39_stopped: 0
> tx39_dropped: 0
> tx39_xmit_more: 0
> tx39_recover: 0
> tx39_cqes: 0
> tx39_wake: 0
> tx39_cqe_err: 0
> tx40_packets: 0
> tx40_bytes: 0
> tx40_tso_packets: 0
> tx40_tso_bytes: 0
> tx40_tso_inner_packets: 0
> tx40_tso_inner_bytes: 0
> tx40_csum_partial: 0
> tx40_csum_partial_inner: 0
> tx40_added_vlan_packets: 0
> tx40_nop: 0
> tx40_csum_none: 0
> tx40_stopped: 0
> tx40_dropped: 0
> tx40_xmit_more: 0
> tx40_recover: 0
> tx40_cqes: 0
> tx40_wake: 0
> tx40_cqe_err: 0
> tx41_packets: 0
> tx41_bytes: 0
> tx41_tso_packets: 0
> tx41_tso_bytes: 0
> tx41_tso_inner_packets: 0
> tx41_tso_inner_bytes: 0
> tx41_csum_partial: 0
> tx41_csum_partial_inner: 0
> tx41_added_vlan_packets: 0
> tx41_nop: 0
> tx41_csum_none: 0
> tx41_stopped: 0
> tx41_dropped: 0
> tx41_xmit_more: 0
> tx41_recover: 0
> tx41_cqes: 0
> tx41_wake: 0
> tx41_cqe_err: 0
> tx42_packets: 0
> tx42_bytes: 0
> tx42_tso_packets: 0
> tx42_tso_bytes: 0
> tx42_tso_inner_packets: 0
> tx42_tso_inner_bytes: 0
> tx42_csum_partial: 0
> tx42_csum_partial_inner: 0
> tx42_added_vlan_packets: 0
> tx42_nop: 0
> tx42_csum_none: 0
> tx42_stopped: 0
> tx42_dropped: 0
> tx42_xmit_more: 0
> tx42_recover: 0
> tx42_cqes: 0
> tx42_wake: 0
> tx42_cqe_err: 0
> tx43_packets: 0
> tx43_bytes: 0
> tx43_tso_packets: 0
> tx43_tso_bytes: 0
> tx43_tso_inner_packets: 0
> tx43_tso_inner_bytes: 0
> tx43_csum_partial: 0
> tx43_csum_partial_inner: 0
> tx43_added_vlan_packets: 0
> tx43_nop: 0
> tx43_csum_none: 0
> tx43_stopped: 0
> tx43_dropped: 0
> tx43_xmit_more: 0
> tx43_recover: 0
> tx43_cqes: 0
> tx43_wake: 0
> tx43_cqe_err: 0
> tx44_packets: 0
> tx44_bytes: 0
> tx44_tso_packets: 0
> tx44_tso_bytes: 0
> tx44_tso_inner_packets: 0
> tx44_tso_inner_bytes: 0
> tx44_csum_partial: 0
> tx44_csum_partial_inner: 0
> tx44_added_vlan_packets: 0
> tx44_nop: 0
> tx44_csum_none: 0
> tx44_stopped: 0
> tx44_dropped: 0
> tx44_xmit_more: 0
> tx44_recover: 0
> tx44_cqes: 0
> tx44_wake: 0
> tx44_cqe_err: 0
> tx45_packets: 0
> tx45_bytes: 0
> tx45_tso_packets: 0
> tx45_tso_bytes: 0
> tx45_tso_inner_packets: 0
> tx45_tso_inner_bytes: 0
> tx45_csum_partial: 0
> tx45_csum_partial_inner: 0
> tx45_added_vlan_packets: 0
> tx45_nop: 0
> tx45_csum_none: 0
> tx45_stopped: 0
> tx45_dropped: 0
> tx45_xmit_more: 0
> tx45_recover: 0
> tx45_cqes: 0
> tx45_wake: 0
> tx45_cqe_err: 0
> tx46_packets: 0
> tx46_bytes: 0
> tx46_tso_packets: 0
> tx46_tso_bytes: 0
> tx46_tso_inner_packets: 0
> tx46_tso_inner_bytes: 0
> tx46_csum_partial: 0
> tx46_csum_partial_inner: 0
> tx46_added_vlan_packets: 0
> tx46_nop: 0
> tx46_csum_none: 0
> tx46_stopped: 0
> tx46_dropped: 0
> tx46_xmit_more: 0
> tx46_recover: 0
> tx46_cqes: 0
> tx46_wake: 0
> tx46_cqe_err: 0
> tx47_packets: 0
> tx47_bytes: 0
> tx47_tso_packets: 0
> tx47_tso_bytes: 0
> tx47_tso_inner_packets: 0
> tx47_tso_inner_bytes: 0
> tx47_csum_partial: 0
> tx47_csum_partial_inner: 0
> tx47_added_vlan_packets: 0
> tx47_nop: 0
> tx47_csum_none: 0
> tx47_stopped: 0
> tx47_dropped: 0
> tx47_xmit_more: 0
> tx47_recover: 0
> tx47_cqes: 0
> tx47_wake: 0
> tx47_cqe_err: 0
> tx48_packets: 0
> tx48_bytes: 0
> tx48_tso_packets: 0
> tx48_tso_bytes: 0
> tx48_tso_inner_packets: 0
> tx48_tso_inner_bytes: 0
> tx48_csum_partial: 0
> tx48_csum_partial_inner: 0
> tx48_added_vlan_packets: 0
> tx48_nop: 0
> tx48_csum_none: 0
> tx48_stopped: 0
> tx48_dropped: 0
> tx48_xmit_more: 0
> tx48_recover: 0
> tx48_cqes: 0
> tx48_wake: 0
> tx48_cqe_err: 0
> tx49_packets: 0
> tx49_bytes: 0
> tx49_tso_packets: 0
> tx49_tso_bytes: 0
> tx49_tso_inner_packets: 0
> tx49_tso_inner_bytes: 0
> tx49_csum_partial: 0
> tx49_csum_partial_inner: 0
> tx49_added_vlan_packets: 0
> tx49_nop: 0
> tx49_csum_none: 0
> tx49_stopped: 0
> tx49_dropped: 0
> tx49_xmit_more: 0
> tx49_recover: 0
> tx49_cqes: 0
> tx49_wake: 0
> tx49_cqe_err: 0
> tx50_packets: 0
> tx50_bytes: 0
> tx50_tso_packets: 0
> tx50_tso_bytes: 0
> tx50_tso_inner_packets: 0
> tx50_tso_inner_bytes: 0
> tx50_csum_partial: 0
> tx50_csum_partial_inner: 0
> tx50_added_vlan_packets: 0
> tx50_nop: 0
> tx50_csum_none: 0
> tx50_stopped: 0
> tx50_dropped: 0
> tx50_xmit_more: 0
> tx50_recover: 0
> tx50_cqes: 0
> tx50_wake: 0
> tx50_cqe_err: 0
> tx51_packets: 0
> tx51_bytes: 0
> tx51_tso_packets: 0
> tx51_tso_bytes: 0
> tx51_tso_inner_packets: 0
> tx51_tso_inner_bytes: 0
> tx51_csum_partial: 0
> tx51_csum_partial_inner: 0
> tx51_added_vlan_packets: 0
> tx51_nop: 0
> tx51_csum_none: 0
> tx51_stopped: 0
> tx51_dropped: 0
> tx51_xmit_more: 0
> tx51_recover: 0
> tx51_cqes: 0
> tx51_wake: 0
> tx51_cqe_err: 0
> tx52_packets: 0
> tx52_bytes: 0
> tx52_tso_packets: 0
> tx52_tso_bytes: 0
> tx52_tso_inner_packets: 0
> tx52_tso_inner_bytes: 0
> tx52_csum_partial: 0
> tx52_csum_partial_inner: 0
> tx52_added_vlan_packets: 0
> tx52_nop: 0
> tx52_csum_none: 0
> tx52_stopped: 0
> tx52_dropped: 0
> tx52_xmit_more: 0
> tx52_recover: 0
> tx52_cqes: 0
> tx52_wake: 0
> tx52_cqe_err: 0
> tx53_packets: 0
> tx53_bytes: 0
> tx53_tso_packets: 0
> tx53_tso_bytes: 0
> tx53_tso_inner_packets: 0
> tx53_tso_inner_bytes: 0
> tx53_csum_partial: 0
> tx53_csum_partial_inner: 0
> tx53_added_vlan_packets: 0
> tx53_nop: 0
> tx53_csum_none: 0
> tx53_stopped: 0
> tx53_dropped: 0
> tx53_xmit_more: 0
> tx53_recover: 0
> tx53_cqes: 0
> tx53_wake: 0
> tx53_cqe_err: 0
> tx54_packets: 0
> tx54_bytes: 0
> tx54_tso_packets: 0
> tx54_tso_bytes: 0
> tx54_tso_inner_packets: 0
> tx54_tso_inner_bytes: 0
> tx54_csum_partial: 0
> tx54_csum_partial_inner: 0
> tx54_added_vlan_packets: 0
> tx54_nop: 0
> tx54_csum_none: 0
> tx54_stopped: 0
> tx54_dropped: 0
> tx54_xmit_more: 0
> tx54_recover: 0
> tx54_cqes: 0
> tx54_wake: 0
> tx54_cqe_err: 0
> tx55_packets: 0
> tx55_bytes: 0
> tx55_tso_packets: 0
> tx55_tso_bytes: 0
> tx55_tso_inner_packets: 0
> tx55_tso_inner_bytes: 0
> tx55_csum_partial: 0
> tx55_csum_partial_inner: 0
> tx55_added_vlan_packets: 0
> tx55_nop: 0
> tx55_csum_none: 0
> tx55_stopped: 0
> tx55_dropped: 0
> tx55_xmit_more: 0
> tx55_recover: 0
> tx55_cqes: 0
> tx55_wake: 0
> tx55_cqe_err: 0
> tx0_xdp_xmit: 0
> tx0_xdp_full: 0
> tx0_xdp_err: 0
> tx0_xdp_cqes: 0
> tx1_xdp_xmit: 0
> tx1_xdp_full: 0
> tx1_xdp_err: 0
> tx1_xdp_cqes: 0
> tx2_xdp_xmit: 0
> tx2_xdp_full: 0
> tx2_xdp_err: 0
> tx2_xdp_cqes: 0
> tx3_xdp_xmit: 0
> tx3_xdp_full: 0
> tx3_xdp_err: 0
> tx3_xdp_cqes: 0
> tx4_xdp_xmit: 0
> tx4_xdp_full: 0
> tx4_xdp_err: 0
> tx4_xdp_cqes: 0
> tx5_xdp_xmit: 0
> tx5_xdp_full: 0
> tx5_xdp_err: 0
> tx5_xdp_cqes: 0
> tx6_xdp_xmit: 0
> tx6_xdp_full: 0
> tx6_xdp_err: 0
> tx6_xdp_cqes: 0
> tx7_xdp_xmit: 0
> tx7_xdp_full: 0
> tx7_xdp_err: 0
> tx7_xdp_cqes: 0
> tx8_xdp_xmit: 0
> tx8_xdp_full: 0
> tx8_xdp_err: 0
> tx8_xdp_cqes: 0
> tx9_xdp_xmit: 0
> tx9_xdp_full: 0
> tx9_xdp_err: 0
> tx9_xdp_cqes: 0
> tx10_xdp_xmit: 0
> tx10_xdp_full: 0
> tx10_xdp_err: 0
> tx10_xdp_cqes: 0
> tx11_xdp_xmit: 0
> tx11_xdp_full: 0
> tx11_xdp_err: 0
> tx11_xdp_cqes: 0
> tx12_xdp_xmit: 0
> tx12_xdp_full: 0
> tx12_xdp_err: 0
> tx12_xdp_cqes: 0
> tx13_xdp_xmit: 0
> tx13_xdp_full: 0
> tx13_xdp_err: 0
> tx13_xdp_cqes: 0
> tx14_xdp_xmit: 0
> tx14_xdp_full: 0
> tx14_xdp_err: 0
> tx14_xdp_cqes: 0
> tx15_xdp_xmit: 0
> tx15_xdp_full: 0
> tx15_xdp_err: 0
> tx15_xdp_cqes: 0
> tx16_xdp_xmit: 0
> tx16_xdp_full: 0
> tx16_xdp_err: 0
> tx16_xdp_cqes: 0
> tx17_xdp_xmit: 0
> tx17_xdp_full: 0
> tx17_xdp_err: 0
> tx17_xdp_cqes: 0
> tx18_xdp_xmit: 0
> tx18_xdp_full: 0
> tx18_xdp_err: 0
> tx18_xdp_cqes: 0
> tx19_xdp_xmit: 0
> tx19_xdp_full: 0
> tx19_xdp_err: 0
> tx19_xdp_cqes: 0
> tx20_xdp_xmit: 0
> tx20_xdp_full: 0
> tx20_xdp_err: 0
> tx20_xdp_cqes: 0
> tx21_xdp_xmit: 0
> tx21_xdp_full: 0
> tx21_xdp_err: 0
> tx21_xdp_cqes: 0
> tx22_xdp_xmit: 0
> tx22_xdp_full: 0
> tx22_xdp_err: 0
> tx22_xdp_cqes: 0
> tx23_xdp_xmit: 0
> tx23_xdp_full: 0
> tx23_xdp_err: 0
> tx23_xdp_cqes: 0
> tx24_xdp_xmit: 0
> tx24_xdp_full: 0
> tx24_xdp_err: 0
> tx24_xdp_cqes: 0
> tx25_xdp_xmit: 0
> tx25_xdp_full: 0
> tx25_xdp_err: 0
> tx25_xdp_cqes: 0
> tx26_xdp_xmit: 0
> tx26_xdp_full: 0
> tx26_xdp_err: 0
> tx26_xdp_cqes: 0
> tx27_xdp_xmit: 0
> tx27_xdp_full: 0
> tx27_xdp_err: 0
> tx27_xdp_cqes: 0
> tx28_xdp_xmit: 0
> tx28_xdp_full: 0
> tx28_xdp_err: 0
> tx28_xdp_cqes: 0
> tx29_xdp_xmit: 0
> tx29_xdp_full: 0
> tx29_xdp_err: 0
> tx29_xdp_cqes: 0
> tx30_xdp_xmit: 0
> tx30_xdp_full: 0
> tx30_xdp_err: 0
> tx30_xdp_cqes: 0
> tx31_xdp_xmit: 0
> tx31_xdp_full: 0
> tx31_xdp_err: 0
> tx31_xdp_cqes: 0
> tx32_xdp_xmit: 0
> tx32_xdp_full: 0
> tx32_xdp_err: 0
> tx32_xdp_cqes: 0
> tx33_xdp_xmit: 0
> tx33_xdp_full: 0
> tx33_xdp_err: 0
> tx33_xdp_cqes: 0
> tx34_xdp_xmit: 0
> tx34_xdp_full: 0
> tx34_xdp_err: 0
> tx34_xdp_cqes: 0
> tx35_xdp_xmit: 0
> tx35_xdp_full: 0
> tx35_xdp_err: 0
> tx35_xdp_cqes: 0
> tx36_xdp_xmit: 0
> tx36_xdp_full: 0
> tx36_xdp_err: 0
> tx36_xdp_cqes: 0
> tx37_xdp_xmit: 0
> tx37_xdp_full: 0
> tx37_xdp_err: 0
> tx37_xdp_cqes: 0
> tx38_xdp_xmit: 0
> tx38_xdp_full: 0
> tx38_xdp_err: 0
> tx38_xdp_cqes: 0
> tx39_xdp_xmit: 0
> tx39_xdp_full: 0
> tx39_xdp_err: 0
> tx39_xdp_cqes: 0
> tx40_xdp_xmit: 0
> tx40_xdp_full: 0
> tx40_xdp_err: 0
> tx40_xdp_cqes: 0
> tx41_xdp_xmit: 0
> tx41_xdp_full: 0
> tx41_xdp_err: 0
> tx41_xdp_cqes: 0
> tx42_xdp_xmit: 0
> tx42_xdp_full: 0
> tx42_xdp_err: 0
> tx42_xdp_cqes: 0
> tx43_xdp_xmit: 0
> tx43_xdp_full: 0
> tx43_xdp_err: 0
> tx43_xdp_cqes: 0
> tx44_xdp_xmit: 0
> tx44_xdp_full: 0
> tx44_xdp_err: 0
> tx44_xdp_cqes: 0
> tx45_xdp_xmit: 0
> tx45_xdp_full: 0
> tx45_xdp_err: 0
> tx45_xdp_cqes: 0
> tx46_xdp_xmit: 0
> tx46_xdp_full: 0
> tx46_xdp_err: 0
> tx46_xdp_cqes: 0
> tx47_xdp_xmit: 0
> tx47_xdp_full: 0
> tx47_xdp_err: 0
> tx47_xdp_cqes: 0
> tx48_xdp_xmit: 0
> tx48_xdp_full: 0
> tx48_xdp_err: 0
> tx48_xdp_cqes: 0
> tx49_xdp_xmit: 0
> tx49_xdp_full: 0
> tx49_xdp_err: 0
> tx49_xdp_cqes: 0
> tx50_xdp_xmit: 0
> tx50_xdp_full: 0
> tx50_xdp_err: 0
> tx50_xdp_cqes: 0
> tx51_xdp_xmit: 0
> tx51_xdp_full: 0
> tx51_xdp_err: 0
> tx51_xdp_cqes: 0
> tx52_xdp_xmit: 0
> tx52_xdp_full: 0
> tx52_xdp_err: 0
> tx52_xdp_cqes: 0
> tx53_xdp_xmit: 0
> tx53_xdp_full: 0
> tx53_xdp_err: 0
> tx53_xdp_cqes: 0
> tx54_xdp_xmit: 0
> tx54_xdp_full: 0
> tx54_xdp_err: 0
> tx54_xdp_cqes: 0
> tx55_xdp_xmit: 0
> tx55_xdp_full: 0
> tx55_xdp_err: 0
> tx55_xdp_cqes: 0
>
>
> mpstat -P ALL 1 10
> Average: CPU %usr %nice %sys %iowait %irq %soft
> %steal
> %guest %gnice %idle
> Average: all 0.04 0.00 6.94 0.02 0.00 32.00
> 0.00 0.00 0.00 61.00
> Average: 0 0.00 0.00 1.20 0.00 0.00 0.00
> 0.00
> 0.00 0.00 98.80
> Average: 1 0.00 0.00 2.30 0.00 0.00 0.00
> 0.00
> 0.00 0.00 97.70
> Average: 2 0.10 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.90
> Average: 3 0.10 0.00 1.50 0.00 0.00 0.00
> 0.00
> 0.00 0.00 98.40
> Average: 4 0.50 0.00 2.50 0.00 0.00 0.00
> 0.00
> 0.00 0.00 97.00
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 6 0.90 0.00 10.20 0.00 0.00 0.00
> 0.00
> 0.00 0.00 88.90
> Average: 7 0.00 0.00 0.00 1.40 0.00 0.00
> 0.00
> 0.00 0.00 98.60
> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 11 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 14 0.00 0.00 12.99 0.00 0.00 62.64
> 0.00 0.00 0.00 24.38
> Average: 15 0.00 0.00 12.70 0.00 0.00 63.40
> 0.00 0.00 0.00 23.90
> Average: 16 0.00 0.00 11.20 0.00 0.00 66.40
> 0.00 0.00 0.00 22.40
> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
> 0.00 0.00 0.00 31.30
> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
> 0.00 0.00 0.00 24.90
> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
> 0.00 0.00 0.00 19.68
> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
> 0.00 0.00 0.00 18.00
> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
> 0.00 0.00 0.00 17.40
> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
> 0.00 0.00 0.00 26.03
> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
> 0.00 0.00 0.00 17.52
> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
> 0.00 0.00 0.00 18.20
> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
> 0.00 0.00 0.00 17.68
> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
> 0.00 0.00 0.00 18.20
> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
> 0.00 0.00 0.00 20.42
The numa cores are not at 100% util, you have around 20% of idle on
each one.
> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
> 0.00
> 0.00 0.00 96.10
> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.70
> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.50
> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
> 0.00
> 0.00 0.00 97.40
> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.10
> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.40
> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
> 0.00 0.00 0.00 19.42
> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
> 0.00 0.00 0.00 26.60
> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
> 0.00 0.00 0.00 21.60
> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
> 0.00 0.00 0.00 20.50
> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
> 0.00 0.00 0.00 21.60
> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
> 0.00 0.00 0.00 24.58
> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
> 0.00 0.00 0.00 24.68
> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
> 0.00 0.00 0.00 28.34
> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
> 0.00 0.00 0.00 25.83
> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
> 0.00 0.00 0.00 27.20
> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
> 0.00 0.00 0.00 28.03
> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
> 0.00 0.00 0.00 24.70
> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
> 0.00 0.00 0.00 25.30
> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
> 0.00 0.00 0.00 26.77
>
>
> ethtool -k enp175s0f0
> Features for enp175s0f0:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: on
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: on
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: on
> highdma: on [fixed]
> rx-vlan-filter: on
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: on
> tx-gre-csum-segmentation: on
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: on
> tx-udp_tnl-csum-segmentation: on
> tx-gso-partial: on
> tx-sctp-segmentation: off [fixed]
> tx-esp-segmentation: off [fixed]
> tx-udp-segmentation: on
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off
> rx-all: off
> tx-vlan-stag-hw-insert: on
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: on [fixed]
> l2-fwd-offload: off [fixed]
> hw-tc-offload: off
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> rx-udp_tunnel-port-offload: on
> tls-hw-tx-offload: off [fixed]
> tls-hw-rx-offload: off [fixed]
> rx-gro-hw: off [fixed]
> tls-hw-record: off [fixed]
>
> ethtool -c enp175s0f0
> Coalesce parameters for enp175s0f0:
> Adaptive RX: off TX: on
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
> dmac: 32703
>
> rx-usecs: 256
> rx-frames: 128
> rx-usecs-irq: 0
> rx-frames-irq: 0
>
> tx-usecs: 8
> tx-frames: 128
> tx-usecs-irq: 0
> tx-frames-irq: 0
>
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
>
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
>
> ethtool -g enp175s0f0
> Ring parameters for enp175s0f0:
> Pre-set maximums:
> RX: 8192
> RX Mini: 0
> RX Jumbo: 0
> TX: 8192
> Current hardware settings:
> RX: 4096
> RX Mini: 0
> RX Jumbo: 0
> TX: 4096
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 9:22 ` Jesper Dangaard Brouer
@ 2018-11-01 10:34 ` Paweł Staszewski
2018-11-01 15:27 ` Aaron Lu
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 10:34 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Eric Dumazet, netdev, Tariq Toukan, Ilias Apalodimas,
Yoel Caspersen, Mel Gorman, Aaron Lu
W dniu 01.11.2018 o 10:22, Jesper Dangaard Brouer pisze:
> On Wed, 31 Oct 2018 23:20:01 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 31.10.2018 o 23:09, Eric Dumazet pisze:
>>> On 10/31/2018 02:57 PM, Paweł Staszewski wrote:
>>>> Hi
>>>>
>>>> So maybee someone will be interested how linux kernel handles
>>>> normal traffic (not pktgen :) )
> Pawel is this live production traffic?
Yes moved server from testlab to production to check (risking a little -
but this is traffic switched to backup router : ) )
>
> I know Yoel (Cc) is very interested to know the real-life limitation of
> Linux as a router, especially with VLANs like you use.
So yes this is real-life traffic , real users - normal mixed internet
traffic forwarded (including ddos-es :) )
>
>
>>>> Server HW configuration:
>>>>
>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>
>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>
>>>>
>>>> Server software:
>>>>
>>>> FRR - as routing daemon
>>>>
>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa node)
>>>>
>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
>>>>
>>>>
>>>> Maximum traffic that server can handle:
>>>>
>>>> Bandwidth
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> \ iface Rx Tx Total
>>>> ==============================================================================
>>>> enp175s0f1: 28.51 Gb/s 37.24 Gb/s 65.74 Gb/s
>>>> enp175s0f0: 38.07 Gb/s 28.44 Gb/s 66.51 Gb/s
>>>> ------------------------------------------------------------------------------
>>>> total: 66.58 Gb/s 65.67 Gb/s 132.25 Gb/s
>>>>
> Actually rather impressive number for a Linux router.
>
>>>> Packets per second:
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> ==============================================================================
>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
>>>> ------------------------------------------------------------------------------
>>>> total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
>>>>
> Average packet size:
> (28.51*10^9/8)/5248589 = 678.99 bytes
> (38.07*10^9/8)/3557944 = 1337.49 bytes
>
>
>>>> After reaching that limits nics on the upstream side (more RX
>>>> traffic) start to drop packets
>>>>
>>>>
>>>> I just dont understand that server can't handle more bandwidth
>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>> RX side are increasing.
>>>>
>>>> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
>>>> is 126Gbit - and also when testing with pktgen i can reach more bw
>>>> and pps (like 4x more comparing to normal internet traffic)
>>>>
>>>> And wondering if there is something that can be improved here.
>>>>
>>>>
>>>>
>>>> Some more informations / counters / stats and perf top below:
>>>>
>>>> Perf top flame graph:
>>>>
>>>> https://uploadfiles.io/7zo6u
> Thanks a lot for the flame graph!
>
>>>> System configuration(long):
>>>>
>>>>
>>>> cat /sys/devices/system/node/node1/cpulist
>>>> 14-27,42-55
>>>> cat /sys/class/net/enp175s0f0/device/numa_node
>>>> 1
>>>> cat /sys/class/net/enp175s0f1/device/numa_node
>>>> 1
>>>>
> Hint grep can give you nicer output that cat:
>
> $ grep -H . /sys/class/net/*/device/numa_node
Sure:
grep -H . /sys/class/net/*/device/numa_node
/sys/class/net/enp175s0f0/device/numa_node:1
/sys/class/net/enp175s0f1/device/numa_node:1
>
>>>>
>>>>
>>>>
>>>> ip -s -d link ls dev enp175s0f0
>>>> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
>>>> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
>>>> RX: bytes packets errors dropped overrun mcast
>>>> 184142375840858 141347715974 2 2806325 0 85050528
>>>> TX: bytes packets errors dropped carrier collsns
>>>> 99270697277430 172227994003 0 0 0 0
>>>>
>>>> ip -s -d link ls dev enp175s0f1
>>>> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
>>>> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
>>>> RX: bytes packets errors dropped overrun mcast
>>>> 99686284170801 173507590134 61 669685 0 100304421
>>>> TX: bytes packets errors dropped carrier collsns
>>>> 184435107970545 142383178304 0 0 0 0
>>>>
> You have increased the default (1000) qlen to 8192, why?
Was checking if higher txq will change anything
But no change for settings 1000,4096,8192
But yes i do not use there any traffic shaping like hfsc/hdb etc
- just default qdisc mq 0:
root pfifp_fast
tc qdisc show dev enp175s0f1
qdisc mq 0: root
qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
qdisc pfifo_fast 0: parent :36 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
...
...
And vlans are noqueue
tc -s -d qdisc show dev vlan1521
qdisc noqueue 0: root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
Weird is that no counters increasing but there is traffic in/out on that
vlans
ip -s -d link ls dev vlan1521
87: vlan1521@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0
vlan protocol 802.1Q id 1521 <REORDER_HDR> addrgenmode eui64
numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
RX: bytes packets errors dropped overrun mcast
562964218394 1639370761 0 0 0 0
TX: bytes packets errors dropped carrier collsns
1417648713052 618271312 0 0 0 0
>
> What default qdisc do you run?... looking through your very detail main
> email report (I do love the details you give!). You run
> pfifo_fast_dequeue, thus this 8192 qlen is actually having effect.
>
> I would like to know if and how much qdisc_dequeue bulking is happening
> in this setup? Can you run:
>
> perf-stat-hist -m 8192 -P2 qdisc:qdisc_dequeue packets
>
> The perf-stat-hist is from Brendan Gregg's git-tree:
> https://github.com/brendangregg/perf-tools
> https://github.com/brendangregg/perf-tools/blob/master/misc/perf-stat-hist
>
./perf-stat-hist -m 8192 -P2 qdisc:qdisc_dequeue packets
Tracing qdisc:qdisc_dequeue, power-of-2, max 8192, until Ctrl-C...
^C
Range : Count Distribution
-> -1 : 0 | |
0 -> 0 : 43768349
|######################################|
1 -> 1 : 43895249
|######################################|
2 -> 3 : 352 |# |
4 -> 7 : 228 |# |
8 -> 15 : 135 |# |
16 -> 31 : 73 |# |
32 -> 63 : 7 |# |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 0 | |
4096 -> 8191 : 0 | |
8192 -> : 0 | |
>>>> ./softnet.sh
>>>> cpu total dropped squeezed collision rps flow_limit
>>>>
>>>>
>>>>
>>>>
>>>> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
>>>> ------------------------------------------------------------------------------------------
>>>>
>>>> 26.78% [kernel] [k] queued_spin_lock_slowpath
>>> This is highly suspect.
>>>
> I agree! -- 26.78% spend in queued_spin_lock_slowpath. Hint if you see
> _raw_spin_lock then it is likely not a contended lock, but if you see
> queued_spin_lock_slowpath in a perf-report your workload is likely in
> trouble.
>
>
>>> A call graph (perf record -a -g sleep 1; perf report --stdio)
>>> would tell what is going on.
>> perf report:
>> https://ufile.io/rqp0h
>>
> Thanks for the output (my 30" screen is just large enough to see the
> full output). Together with the flame-graph, it is clear that this
> lock happens in the page allocator code.
>
> Section copied out:
>
> mlx5e_poll_tx_cq
> |
> --16.34%--napi_consume_skb
> |
> |--12.65%--__free_pages_ok
> | |
> | --11.86%--free_one_page
> | |
> | |--10.10%--queued_spin_lock_slowpath
> | |
> | --0.65%--_raw_spin_lock
> |
> |--1.55%--page_frag_free
> |
> --1.44%--skb_release_data
>
>
> Let me explain what (I think) happens. The mlx5 driver RX-page recycle
> mechanism is not effective in this workload, and pages have to go
> through the page allocator. The lock contention happens during mlx5
> DMA TX completion cycle. And the page allocator cannot keep up at
> these speeds.
>
> One solution is extend page allocator with a bulk free API. (This have
> been on my TODO list for a long time, but I don't have a
> micro-benchmark that trick the driver page-recycle to fail). It should
> fit nicely, as I can see that kmem_cache_free_bulk() does get
> activated (bulk freeing SKBs), which means that DMA TX completion do
> have a bulk of packets.
>
> We can (and should) also improve the page recycle scheme in the driver.
> After LPC, I have a project with Tariq and Ilias (Cc'ed) to improve the
> page_pool, and we will (attempt) to generalize this, for both high-end
> mlx5 and more low-end ARM64-boards (macchiatobin and espressobin).
>
> The MM-people is in parallel working to improve the performance of
> order-0 page returns. Thus, the explicit page bulk free API might
> actually become less important. I actually think (Cc.) Aaron have a
> patchset he would like you to test, which removes the (zone->)lock
> you hit in free_one_page().
>
Ok - Thank You Jesper
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 3:37 ` David Ahern
@ 2018-11-01 10:55 ` Jesper Dangaard Brouer
2018-11-01 13:52 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-01 10:55 UTC (permalink / raw)
To: David Ahern; +Cc: brouer, Paweł Staszewski, netdev, Yoel Caspersen
On Wed, 31 Oct 2018 21:37:16 -0600 David Ahern <dsahern@gmail.com> wrote:
> This is mainly a forwarding use case? Seems so based on the perf report.
> I suspect forwarding with XDP would show pretty good improvement.
Yes, significant performance improvements.
Notice Davids talk: "Leveraging Kernel Tables with XDP"
http://vger.kernel.org/lpc-networking2018.html#session-1
It looks like that you are doing "pure" IP-routing, without any
iptables conntrack stuff (from your perf report data). That will
actually be a really good use-case for accelerating this with XDP.
I want you to understand the philosophy behind how David and I want
people to leverage XDP. Think of XDP as a software offload layer for
the kernel network stack. Setup and use Linux kernel network stack, but
accelerate parts of it with XDP, e.g. the route FIB lookup.
Sample code avail here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/xdp_fwd_kern.c
(I do warn, what we just found a bug/crash in setup+tairdown for the
mlx5 driver you are using, that we/mlx _will_ fix soon)
> You need the vlan changes I have queued up though.
I know Yoel will be very interested in those changes too! I've
convinced Yoel to write an XDP program for his Border Network Gateway
(BNG) production system[1], and his is a heavy VLAN user. And the plan
is to Open Source this when he have-something-working.
[1] https://www.version2.dk/blog/software-router-del-5-linux-bng-1086060
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 9:50 ` Saeed Mahameed
@ 2018-11-01 11:09 ` Paweł Staszewski
2018-11-01 16:49 ` Paweł Staszewski
2018-11-01 20:37 ` Saeed Mahameed
0 siblings, 2 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 11:09 UTC (permalink / raw)
To: Saeed Mahameed, netdev
W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>> Hi
>>
>> So maybee someone will be interested how linux kernel handles normal
>> traffic (not pktgen :) )
>>
>>
>> Server HW configuration:
>>
>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>
>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>
>>
>> Server software:
>>
>> FRR - as routing daemon
>>
>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local
>> numa
>> node)
>>
>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa
>> node)
>>
>>
>> Maximum traffic that server can handle:
>>
>> Bandwidth
>>
>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>> input: /proc/net/dev type: rate
>> \ iface Rx Tx Total
>> =====================================================================
>> =========
>> enp175s0f1: 28.51 Gb/s 37.24
>> Gb/s
>> 65.74 Gb/s
>> enp175s0f0: 38.07 Gb/s 28.44
>> Gb/s
>> 66.51 Gb/s
>> -------------------------------------------------------------------
>> -----------
>> total: 66.58 Gb/s 65.67
>> Gb/s
>> 132.25 Gb/s
>>
>>
>> Packets per second:
>>
>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>> input: /proc/net/dev type: rate
>> - iface Rx Tx Total
>> =====================================================================
>> =========
>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>> 8735207.00 P/s
>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>> 8790460.00 P/s
>> -------------------------------------------------------------------
>> -----------
>> total: 8806533.00 P/s 8719134.00 P/s
>> 17525668.00 P/s
>>
>>
>> After reaching that limits nics on the upstream side (more RX
>> traffic)
>> start to drop packets
>>
>>
>> I just dont understand that server can't handle more bandwidth
>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
>> side are increasing.
>>
> Where do you see 40 Gb/s ? you showed that both ports on the same NIC (
> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) = 132.25
> Gb/s which aligns with your pcie link limit, what am i missing ?
hmm yes that was my concern also - cause cant find anywhere informations
about that bandwidth is uni or bidirectional - so if 126Gbit for x16 8GT
is unidir - then bidir will be 126/2 ~68Gbit - which will fit total bw
on both ports
This can explain maybee also why cpuload is rising rapidly from
120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net - so
there can be some error in reading them when offloading (gro/gso/tso) on
nic's is enabled that is why
>
>> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
>> is
>> 126Gbit - and also when testing with pktgen i can reach more bw and
>> pps
>> (like 4x more comparing to normal internet traffic)
>>
> Are you forwarding when using pktgen as well or you just testing the RX
> side pps ?
Yes pktgen was tested on single port RX
Can check also forwarding to eliminate pciex limits
>
>> And wondering if there is something that can be improved here.
>>
>>
>>
>> Some more informations / counters / stats and perf top below:
>>
>> Perf top flame graph:
>>
>> https://uploadfiles.io/7zo6u
>>
>>
>>
>> System configuration(long):
>>
>>
>> cat /sys/devices/system/node/node1/cpulist
>> 14-27,42-55
>> cat /sys/class/net/enp175s0f0/device/numa_node
>> 1
>> cat /sys/class/net/enp175s0f1/device/numa_node
>> 1
>>
>>
>>
>>
>>
>> ip -s -d link ls dev enp175s0f0
>> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>> state
>> UP mode DEFAULT group default qlen 8192
>> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity
>> 0
>> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
>> gso_max_segs 65535
>> RX: bytes packets errors dropped overrun mcast
>> 184142375840858 141347715974 2 2806325 0 85050528
>> TX: bytes packets errors dropped carrier collsns
>> 99270697277430 172227994003 0 0 0 0
>>
>> ip -s -d link ls dev enp175s0f1
>> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>> state
>> UP mode DEFAULT group default qlen 8192
>> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity
>> 0
>> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
>> gso_max_segs 65535
>> RX: bytes packets errors dropped overrun mcast
>> 99686284170801 173507590134 61 669685 0 100304421
>> TX: bytes packets errors dropped carrier collsns
>> 184435107970545 142383178304 0 0 0 0
>>
>>
>> ./softnet.sh
>> cpu total dropped squeezed collision rps flow_limit
>> 0 3961392822 0 1221478 0 0 0
>> 1 3701952251 0 1258234 0 0 0
>> 2 3879522030 0 1584282 0 0 0
>> 3 3731349789 0 1529029 0 0 0
>> 4 1323956701 0 2176371 0 0 0
>> 5 420528963 0 1880146 0 0 0
>> 6 348720322 0 1830142 0 0 0
>> 7 372736328 0 1820891 0 0 0
>> 8 567888751 0 1414763 0 0 0
>> 9 476075775 0 1868150 0 0 0
>> 10 468946725 0 1841428 0 0 0
>> 11 676591958 0 1900160 0 0 0
>> 12 346803472 0 1834600 0 0 0
>> 13 457960872 0 1874529 0 0 0
>> 14 1990279665 0 4699000 0 0 0
>> 15 1211873601 0 4541281 0 0 0
>> 16 1123871928 0 4544712 0 0 0
>> 17 1014957263 0 4152355 0 0 0
>> 18 2603779724 0 4593869 0 0 0
>> 19 2181924054 0 4930618 0 0 0
>> 20 2273502182 0 4894627 0 0 0
>> 21 2232030947 0 4860048 0 0 0
>> 22 2203555394 0 4603830 0 0 0
>> 23 2194756800 0 4921294 0 0 0
>> 24 2347158294 0 4818354 0 0 0
>> 25 2291097883 0 4744469 0 0 0
>> 26 2206945011 0 4836483 0 0 0
>> 27 2318530217 0 4917617 0 0 0
>> 28 512797543 0 1895200 0 0 0
>> 29 597279474 0 1532134 0 0 0
>> 30 475317503 0 1451523 0 0 0
>> 31 499172796 0 1901207 0 0 0
>> 32 493874745 0 1915382 0 0 0
>> 33 296056288 0 1865535 0 0 0
>> 34 3905097041 0 1580822 0 0 0
>> 35 3905112345 0 1536105 0 0 0
>> 36 3900358950 0 1166319 0 0 0
>> 37 3940978093 0 1600219 0 0 0
>> 38 3878632215 0 1180389 0 0 0
>> 39 3814804736 0 1584925 0 0 0
>> 40 4152934337 0 1663660 0 0 0
>> 41 3855273904 0 1552219 0 0 0
>> 42 2319538182 0 4884480 0 0 0
>> 43 2448606991 0 4387456 0 0 0
>> 44 1436136753 0 4485073 0 0 0
>> 45 1200500141 0 4537284 0 0 0
>> 46 1307799923 0 4534156 0 0 0
>> 47 1586575293 0 4272997 0 0 0
>> 48 3852574 0 4162653 0 0 0
>> 49 391449390 0 3935202 0 0 0
>> 50 791388200 0 4290738 0 0 0
>> 51 127107573 0 3907750 0 0 0
>> 52 115622148 0 4012843 0 0 0
>> 53 71098871 0 4200625 0 0 0
>> 54 305121466 0 4365614 0 0 0
>> 55 10914257 0 4369426 0 0 0
>>
>>
>>
>>
>> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> -------------------------------------------------------------------
>> -------------------------------------------------------------------
>> -------------------------------------------------------------------
>> ------
>>
>> 26.78% [kernel] [k] queued_spin_lock_slowpath
>> 9.09% [kernel] [k] mlx5e_skb_from_cqe_linear
>> 4.94% [kernel] [k] mlx5e_sq_xmit
>> 3.63% [kernel] [k] memcpy_erms
>> 3.30% [kernel] [k] fib_table_lookup
>> 3.26% [kernel] [k] build_skb
>> 2.41% [kernel] [k] mlx5e_poll_tx_cq
>> 2.11% [kernel] [k] get_page_from_freelist
>> 1.51% [kernel] [k] vlan_do_receive
>> 1.51% [kernel] [k] _raw_spin_lock
>> 1.43% [kernel] [k] __dev_queue_xmit
>> 1.41% [kernel] [k] dev_gro_receive
>> 1.34% [kernel] [k] mlx5e_poll_rx_cq
>> 1.26% [kernel] [k] tcp_gro_receive
>> 1.21% [kernel] [k] free_one_page
>> 1.13% [kernel] [k] swiotlb_map_page
>> 1.13% [kernel] [k] mlx5e_post_rx_wqes
>> 1.05% [kernel] [k] pfifo_fast_dequeue
>> 1.05% [kernel] [k] mlx5e_handle_rx_cqe
>> 1.03% [kernel] [k] ip_finish_output2
>> 1.02% [kernel] [k] ipt_do_table
>> 0.96% [kernel] [k] inet_gro_receive
>> 0.91% [kernel] [k] mlx5_eq_int
>> 0.88% [kernel] [k] __slab_free.isra.79
>> 0.86% [kernel] [k] __build_skb
>> 0.84% [kernel] [k] page_frag_free
>> 0.76% [kernel] [k] skb_release_data
>> 0.75% [kernel] [k] __netif_receive_skb_core
>> 0.75% [kernel] [k] irq_entries_start
>> 0.71% [kernel] [k] ip_route_input_rcu
>> 0.65% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.56% [kernel] [k] ip_forward
>> 0.56% [kernel] [k] __memcpy
>> 0.52% [kernel] [k] kmem_cache_alloc
>> 0.52% [kernel] [k] kmem_cache_free_bulk
>> 0.49% [kernel] [k] mlx5e_page_release
>> 0.47% [kernel] [k] netif_skb_features
>> 0.47% [kernel] [k] mlx5e_build_rx_skb
>> 0.47% [kernel] [k] dev_hard_start_xmit
>> 0.43% [kernel] [k] __page_pool_put_page
>> 0.43% [kernel] [k] __netif_schedule
>> 0.43% [kernel] [k] mlx5e_xmit
>> 0.41% [kernel] [k] __qdisc_run
>> 0.41% [kernel] [k] validate_xmit_skb.isra.142
>> 0.41% [kernel] [k] swiotlb_unmap_page
>> 0.40% [kernel] [k] inet_lookup_ifaddr_rcu
>> 0.34% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>> 0.34% [kernel] [k] tcp4_gro_receive
>> 0.29% [kernel] [k] _raw_spin_lock_irqsave
>> 0.29% [kernel] [k] napi_consume_skb
>> 0.29% [kernel] [k] skb_gro_receive
>> 0.29% [kernel] [k] ___slab_alloc.isra.80
>> 0.27% [kernel] [k] eth_type_trans
>> 0.26% [kernel] [k] __free_pages_ok
>> 0.26% [kernel] [k] __get_xps_queue_idx
>> 0.24% [kernel] [k] _raw_spin_trylock
>> 0.23% [kernel] [k] __local_bh_enable_ip
>> 0.22% [kernel] [k] pfifo_fast_enqueue
>> 0.21% [kernel] [k] tasklet_action_common.isra.21
>> 0.21% [kernel] [k] sch_direct_xmit
>> 0.21% [kernel] [k] skb_network_protocol
>> 0.21% [kernel] [k] kmem_cache_free
>> 0.20% [kernel] [k] netdev_pick_tx
>> 0.18% [kernel] [k] napi_gro_complete
>> 0.18% [kernel] [k] __sched_text_start
>> 0.18% [kernel] [k] mlx5e_xdp_handle
>> 0.17% [kernel] [k] ip_finish_output
>> 0.16% [kernel] [k] napi_gro_flush
>> 0.16% [kernel] [k] vlan_passthru_hard_header
>> 0.16% [kernel] [k] skb_segment
>> 0.15% [kernel] [k] __alloc_pages_nodemask
>> 0.15% [kernel] [k] mlx5e_features_check
>> 0.15% [kernel] [k] mlx5e_napi_poll
>> 0.15% [kernel] [k] napi_gro_receive
>> 0.14% [kernel] [k] fib_validate_source
>> 0.14% [kernel] [k] _raw_spin_lock_irq
>> 0.14% [kernel] [k] inet_gro_complete
>> 0.14% [kernel] [k] get_partial_node.isra.78
>> 0.13% [kernel] [k] napi_complete_done
>> 0.13% [kernel] [k] ip_rcv_finish_core.isra.17
>> 0.13% [kernel] [k] cmd_exec
>>
>>
>>
>> ethtool -S enp175s0f1
>> NIC statistics:
>> rx_packets: 173730800927
>> rx_bytes: 99827422751332
>> tx_packets: 142532009512
>> tx_bytes: 184633045911222
>> tx_tso_packets: 25989113891
>> tx_tso_bytes: 132933363384458
>> tx_tso_inner_packets: 0
>> tx_tso_inner_bytes: 0
>> tx_added_vlan_packets: 74630239613
>> tx_nop: 2029817748
>> rx_lro_packets: 0
>> rx_lro_bytes: 0
>> rx_ecn_mark: 0
>> rx_removed_vlan_packets: 173730800927
>> rx_csum_unnecessary: 0
>> rx_csum_none: 434357
>> rx_csum_complete: 173730366570
>> rx_csum_unnecessary_inner: 0
>> rx_xdp_drop: 0
>> rx_xdp_redirect: 0
>> rx_xdp_tx_xmit: 0
>> rx_xdp_tx_full: 0
>> rx_xdp_tx_err: 0
>> rx_xdp_tx_cqe: 0
>> tx_csum_none: 38260960853
>> tx_csum_partial: 36369278774
>> tx_csum_partial_inner: 0
>> tx_queue_stopped: 1
>> tx_queue_dropped: 0
>> tx_xmit_more: 748638099
>> tx_recover: 0
>> tx_cqes: 73881645031
>> tx_queue_wake: 1
>> tx_udp_seg_rem: 0
>> tx_cqe_err: 0
>> tx_xdp_xmit: 0
>> tx_xdp_full: 0
>> tx_xdp_err: 0
>> tx_xdp_cqes: 0
>> rx_wqe_err: 0
>> rx_mpwqe_filler_cqes: 0
>> rx_mpwqe_filler_strides: 0
>> rx_buff_alloc_err: 0
>> rx_cqe_compress_blks: 0
>> rx_cqe_compress_pkts: 0
> If this is a pcie bottleneck it might be useful to enable CQE
> compression (to reduce PCIe completion descriptors transactions)
> you should see the above rx_cqe_compress_pkts increasing when enabled.
>
> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
> $ ethtool --show-priv-flags enp175s0f1
> Private flags for p6p1:
> rx_cqe_moder : on
> cqe_moder : off
> rx_cqe_compress : on
> ...
>
> try this on both interfaces.
Done
ethtool --show-priv-flags enp175s0f1
Private flags for enp175s0f1:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : on
rx_striding_rq : off
rx_no_csum_complete: off
ethtool --show-priv-flags enp175s0f0
Private flags for enp175s0f0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : on
rx_striding_rq : off
rx_no_csum_complete: off
>
>> rx_page_reuse: 0
>> rx_cache_reuse: 14441066823
>> rx_cache_full: 51126004413
>> rx_cache_empty: 21297344082
>> rx_cache_busy: 51127247487
>> rx_cache_waive: 21298322293
>> rx_congst_umr: 0
>> rx_arfs_err: 0
>> ch_events: 24603119858
>> ch_poll: 25180949074
>> ch_arm: 24480437587
>> ch_aff_change: 75
>> ch_eq_rearm: 0
>> rx_out_of_buffer: 669685
> comparing this to rx_vport_unicast_packets, it is a very small
> percentage of dropped packets due to stalled rx cpu, so rx cpu is not a
> bottleneck, at least for the driver rx rings.
>
>> rx_if_down_packets: 61
>> rx_vport_unicast_packets: 173731641945
>> rx_vport_unicast_bytes: 100522745036693
>> tx_vport_unicast_packets: 142531901313
>> tx_vport_unicast_bytes: 185189071776429
>> rx_vport_multicast_packets: 100360886
>> rx_vport_multicast_bytes: 6639236688
>> tx_vport_multicast_packets: 32837
>> tx_vport_multicast_bytes: 2978810
>> rx_vport_broadcast_packets: 44854
>> rx_vport_broadcast_bytes: 6313510
>> tx_vport_broadcast_packets: 72258
>> tx_vport_broadcast_bytes: 4335480
>> rx_vport_rdma_unicast_packets: 0
>> rx_vport_rdma_unicast_bytes: 0
>> tx_vport_rdma_unicast_packets: 0
>> tx_vport_rdma_unicast_bytes: 0
>> rx_vport_rdma_multicast_packets: 0
>> rx_vport_rdma_multicast_bytes: 0
>> tx_vport_rdma_multicast_packets: 0
>> tx_vport_rdma_multicast_bytes: 0
>> tx_packets_phy: 142532004669
>> rx_packets_phy: 173980375752
>> rx_crc_errors_phy: 0
>> tx_bytes_phy: 185759204762903
>> rx_bytes_phy: 101326109361379
>> tx_multicast_phy: 32837
>> tx_broadcast_phy: 72258
>> rx_multicast_phy: 100360885
>> rx_broadcast_phy: 44854
>> rx_in_range_len_errors_phy: 2
>> rx_out_of_range_len_phy: 0
>> rx_oversize_pkts_phy: 59
>> rx_symbol_err_phy: 0
>> tx_mac_control_phy: 0
>> rx_mac_control_phy: 0
>> rx_unsupported_op_phy: 0
>> rx_pause_ctrl_phy: 0
>> tx_pause_ctrl_phy: 0
>> rx_discards_phy: 148328738
>> tx_discards_phy: 0
>> tx_errors_phy: 0
>> rx_undersize_pkts_phy: 0
>> rx_fragments_phy: 0
>> rx_jabbers_phy: 0
>> rx_64_bytes_phy: 36551843112
>> rx_65_to_127_bytes_phy: 65102131735
>> rx_128_to_255_bytes_phy: 5755731137
>> rx_256_to_511_bytes_phy: 2475619839
>> rx_512_to_1023_bytes_phy: 2826971156
>> rx_1024_to_1518_bytes_phy: 42474023107
>> rx_1519_to_2047_bytes_phy: 18794051270
>> rx_2048_to_4095_bytes_phy: 0
>> rx_4096_to_8191_bytes_phy: 0
>> rx_8192_to_10239_bytes_phy: 0
>> link_down_events_phy: 0
>> rx_pcs_symbol_err_phy: 0
>> rx_corrected_bits_phy: 0
>> rx_pci_signal_integrity: 0
>> tx_pci_signal_integrity: 48
>> rx_prio0_bytes: 101316322498995
>> rx_prio0_packets: 173711151686
>> tx_prio0_bytes: 185759176566814
>> tx_prio0_packets: 142531983704
>> rx_prio1_bytes: 47062768
>> rx_prio1_packets: 228932
>> tx_prio1_bytes: 0
>> tx_prio1_packets: 0
>> rx_prio2_bytes: 12434759
>> rx_prio2_packets: 83773
>> tx_prio2_bytes: 0
>> tx_prio2_packets: 0
>> rx_prio3_bytes: 288843134
>> rx_prio3_packets: 982102
>> tx_prio3_bytes: 0
>> tx_prio3_packets: 0
>> rx_prio4_bytes: 699797236
>> rx_prio4_packets: 8109231
>> tx_prio4_bytes: 0
>> tx_prio4_packets: 0
>> rx_prio5_bytes: 1385386738
>> rx_prio5_packets: 9661187
>> tx_prio5_bytes: 0
>> tx_prio5_packets: 0
>> rx_prio6_bytes: 317092102
>> rx_prio6_packets: 1951538
>> tx_prio6_bytes: 0
>> tx_prio6_packets: 0
>> rx_prio7_bytes: 7015734695
>> rx_prio7_packets: 99847456
>> tx_prio7_bytes: 0
>> tx_prio7_packets: 0
>> module_unplug: 0
>> module_bus_stuck: 0
>> module_high_temp: 0
>> module_bad_shorted: 0
>> ch0_events: 936264703
>> ch0_poll: 963766474
>> ch0_arm: 930246079
>> ch0_aff_change: 0
>> ch0_eq_rearm: 0
>> ch1_events: 869408429
>> ch1_poll: 896099392
>> ch1_arm: 864336861
>> ch1_aff_change: 0
>> ch1_eq_rearm: 0
>> ch2_events: 843345698
>> ch2_poll: 869749522
>> ch2_arm: 838186113
>> ch2_aff_change: 2
>> ch2_eq_rearm: 0
>> ch3_events: 850261340
>> ch3_poll: 876721111
>> ch3_arm: 845295235
>> ch3_aff_change: 3
>> ch3_eq_rearm: 0
>> ch4_events: 974985780
>> ch4_poll: 997781915
>> ch4_arm: 969618250
>> ch4_aff_change: 3
>> ch4_eq_rearm: 0
>> ch5_events: 888559089
>> ch5_poll: 912783615
>> ch5_arm: 883826078
>> ch5_aff_change: 2
>> ch5_eq_rearm: 0
>> ch6_events: 873730730
>> ch6_poll: 899635752
>> ch6_arm: 868677574
>> ch6_aff_change: 4
>> ch6_eq_rearm: 0
>> ch7_events: 873478411
>> ch7_poll: 899216716
>> ch7_arm: 868693645
>> ch7_aff_change: 3
>> ch7_eq_rearm: 0
>> ch8_events: 871900967
>> ch8_poll: 898575518
>> ch8_arm: 866763693
>> ch8_aff_change: 3
>> ch8_eq_rearm: 0
>> ch9_events: 880325565
>> ch9_poll: 904983269
>> ch9_arm: 875643922
>> ch9_aff_change: 2
>> ch9_eq_rearm: 0
>> ch10_events: 889919775
>> ch10_poll: 915335809
>> ch10_arm: 885110225
>> ch10_aff_change: 4
>> ch10_eq_rearm: 0
>> ch11_events: 962709175
>> ch11_poll: 983963451
>> ch11_arm: 958117526
>> ch11_aff_change: 2
>> ch11_eq_rearm: 0
>> ch12_events: 941333837
>> ch12_poll: 964625523
>> ch12_arm: 936409706
>> ch12_aff_change: 2
>> ch12_eq_rearm: 0
>> ch13_events: 914996974
>> ch13_poll: 937441049
>> ch13_arm: 910478393
>> ch13_aff_change: 4
>> ch13_eq_rearm: 0
>> ch14_events: 888050001
>> ch14_poll: 911818008
>> ch14_arm: 883465035
>> ch14_aff_change: 4
>> ch14_eq_rearm: 0
>> ch15_events: 947547704
>> ch15_poll: 969073194
>> ch15_arm: 942686515
>> ch15_aff_change: 4
>> ch15_eq_rearm: 0
>> ch16_events: 825804904
>> ch16_poll: 840630747
>> ch16_arm: 822227488
>> ch16_aff_change: 2
>> ch16_eq_rearm: 0
>> ch17_events: 861673823
>> ch17_poll: 874754041
>> ch17_arm: 858520448
>> ch17_aff_change: 2
>> ch17_eq_rearm: 0
>> ch18_events: 879413440
>> ch18_poll: 893962529
>> ch18_arm: 875983204
>> ch18_aff_change: 4
>> ch18_eq_rearm: 0
>> ch19_events: 896073709
>> ch19_poll: 909216857
>> ch19_arm: 893022121
>> ch19_aff_change: 4
>> ch19_eq_rearm: 0
>> ch20_events: 865188535
>> ch20_poll: 880692345
>> ch20_arm: 861440265
>> ch20_aff_change: 3
>> ch20_eq_rearm: 0
>> ch21_events: 862709303
>> ch21_poll: 878104242
>> ch21_arm: 859041767
>> ch21_aff_change: 2
>> ch21_eq_rearm: 0
>> ch22_events: 887720551
>> ch22_poll: 904122074
>> ch22_arm: 883983794
>> ch22_aff_change: 2
>> ch22_eq_rearm: 0
>> ch23_events: 813355027
>> ch23_poll: 828074467
>> ch23_arm: 809912398
>> ch23_aff_change: 4
>> ch23_eq_rearm: 0
>> ch24_events: 822366675
>> ch24_poll: 839917937
>> ch24_arm: 818422754
>> ch24_aff_change: 2
>> ch24_eq_rearm: 0
>> ch25_events: 826642292
>> ch25_poll: 842630121
>> ch25_arm: 822642618
>> ch25_aff_change: 2
>> ch25_eq_rearm: 0
>> ch26_events: 826392584
>> ch26_poll: 843406973
>> ch26_arm: 822455000
>> ch26_aff_change: 3
>> ch26_eq_rearm: 0
>> ch27_events: 828960899
>> ch27_poll: 843866518
>> ch27_arm: 825230937
>> ch27_aff_change: 3
>> ch27_eq_rearm: 0
>> ch28_events: 7
>> ch28_poll: 7
>> ch28_arm: 7
>> ch28_aff_change: 0
>> ch28_eq_rearm: 0
>> ch29_events: 4
>> ch29_poll: 4
>> ch29_arm: 4
>> ch29_aff_change: 0
>> ch29_eq_rearm: 0
>> ch30_events: 4
>> ch30_poll: 4
>> ch30_arm: 4
>> ch30_aff_change: 0
>> ch30_eq_rearm: 0
>> ch31_events: 4
>> ch31_poll: 4
>> ch31_arm: 4
>> ch31_aff_change: 0
>> ch31_eq_rearm: 0
>> ch32_events: 4
>> ch32_poll: 4
>> ch32_arm: 4
>> ch32_aff_change: 0
>> ch32_eq_rearm: 0
>> ch33_events: 4
>> ch33_poll: 4
>> ch33_arm: 4
>> ch33_aff_change: 0
>> ch33_eq_rearm: 0
>> ch34_events: 4
>> ch34_poll: 4
>> ch34_arm: 4
>> ch34_aff_change: 0
>> ch34_eq_rearm: 0
>> ch35_events: 4
>> ch35_poll: 4
>> ch35_arm: 4
>> ch35_aff_change: 0
>> ch35_eq_rearm: 0
>> ch36_events: 4
>> ch36_poll: 4
>> ch36_arm: 4
>> ch36_aff_change: 0
>> ch36_eq_rearm: 0
>> ch37_events: 4
>> ch37_poll: 4
>> ch37_arm: 4
>> ch37_aff_change: 0
>> ch37_eq_rearm: 0
>> ch38_events: 4
>> ch38_poll: 4
>> ch38_arm: 4
>> ch38_aff_change: 0
>> ch38_eq_rearm: 0
>> ch39_events: 4
>> ch39_poll: 4
>> ch39_arm: 4
>> ch39_aff_change: 0
>> ch39_eq_rearm: 0
>> ch40_events: 4
>> ch40_poll: 4
>> ch40_arm: 4
>> ch40_aff_change: 0
>> ch40_eq_rearm: 0
>> ch41_events: 4
>> ch41_poll: 4
>> ch41_arm: 4
>> ch41_aff_change: 0
>> ch41_eq_rearm: 0
>> ch42_events: 4
>> ch42_poll: 4
>> ch42_arm: 4
>> ch42_aff_change: 0
>> ch42_eq_rearm: 0
>> ch43_events: 4
>> ch43_poll: 4
>> ch43_arm: 4
>> ch43_aff_change: 0
>> ch43_eq_rearm: 0
>> ch44_events: 4
>> ch44_poll: 4
>> ch44_arm: 4
>> ch44_aff_change: 0
>> ch44_eq_rearm: 0
>> ch45_events: 4
>> ch45_poll: 4
>> ch45_arm: 4
>> ch45_aff_change: 0
>> ch45_eq_rearm: 0
>> ch46_events: 4
>> ch46_poll: 4
>> ch46_arm: 4
>> ch46_aff_change: 0
>> ch46_eq_rearm: 0
>> ch47_events: 4
>> ch47_poll: 4
>> ch47_arm: 4
>> ch47_aff_change: 0
>> ch47_eq_rearm: 0
>> ch48_events: 4
>> ch48_poll: 4
>> ch48_arm: 4
>> ch48_aff_change: 0
>> ch48_eq_rearm: 0
>> ch49_events: 4
>> ch49_poll: 4
>> ch49_arm: 4
>> ch49_aff_change: 0
>> ch49_eq_rearm: 0
>> ch50_events: 4
>> ch50_poll: 4
>> ch50_arm: 4
>> ch50_aff_change: 0
>> ch50_eq_rearm: 0
>> ch51_events: 4
>> ch51_poll: 4
>> ch51_arm: 4
>> ch51_aff_change: 0
>> ch51_eq_rearm: 0
>> ch52_events: 4
>> ch52_poll: 4
>> ch52_arm: 4
>> ch52_aff_change: 0
>> ch52_eq_rearm: 0
>> ch53_events: 4
>> ch53_poll: 4
>> ch53_arm: 4
>> ch53_aff_change: 0
>> ch53_eq_rearm: 0
>> ch54_events: 4
>> ch54_poll: 4
>> ch54_arm: 4
>> ch54_aff_change: 0
>> ch54_eq_rearm: 0
>> ch55_events: 4
>> ch55_poll: 4
>> ch55_arm: 4
>> ch55_aff_change: 0
>> ch55_eq_rearm: 0
>> rx0_packets: 7284057433
>> rx0_bytes: 4330611281319
>> rx0_csum_complete: 7283623076
>> rx0_csum_unnecessary: 0
>> rx0_csum_unnecessary_inner: 0
>> rx0_csum_none: 434357
>> rx0_xdp_drop: 0
>> rx0_xdp_redirect: 0
>> rx0_lro_packets: 0
>> rx0_lro_bytes: 0
>> rx0_ecn_mark: 0
>> rx0_removed_vlan_packets: 7284057433
>> rx0_wqe_err: 0
>> rx0_mpwqe_filler_cqes: 0
>> rx0_mpwqe_filler_strides: 0
>> rx0_buff_alloc_err: 0
>> rx0_cqe_compress_blks: 0
>> rx0_cqe_compress_pkts: 0
>> rx0_page_reuse: 0
>> rx0_cache_reuse: 1989731589
>> rx0_cache_full: 28213297
>> rx0_cache_empty: 1624089822
>> rx0_cache_busy: 28213961
>> rx0_cache_waive: 1624083610
>> rx0_congst_umr: 0
>> rx0_arfs_err: 0
>> rx0_xdp_tx_xmit: 0
>> rx0_xdp_tx_full: 0
>> rx0_xdp_tx_err: 0
>> rx0_xdp_tx_cqes: 0
>> rx1_packets: 6691319211
>> rx1_bytes: 3799580210608
>> rx1_csum_complete: 6691319211
>> rx1_csum_unnecessary: 0
>> rx1_csum_unnecessary_inner: 0
>> rx1_csum_none: 0
>> rx1_xdp_drop: 0
>> rx1_xdp_redirect: 0
>> rx1_lro_packets: 0
>> rx1_lro_bytes: 0
>> rx1_ecn_mark: 0
>> rx1_removed_vlan_packets: 6691319211
>> rx1_wqe_err: 0
>> rx1_mpwqe_filler_cqes: 0
>> rx1_mpwqe_filler_strides: 0
>> rx1_buff_alloc_err: 0
>> rx1_cqe_compress_blks: 0
>> rx1_cqe_compress_pkts: 0
>> rx1_page_reuse: 0
>> rx1_cache_reuse: 2270019
>> rx1_cache_full: 3343389331
>> rx1_cache_empty: 6656
>> rx1_cache_busy: 3343389585
>> rx1_cache_waive: 0
>> rx1_congst_umr: 0
>> rx1_arfs_err: 0
>> rx1_xdp_tx_xmit: 0
>> rx1_xdp_tx_full: 0
>> rx1_xdp_tx_err: 0
>> rx1_xdp_tx_cqes: 0
>> rx2_packets: 6618370416
>> rx2_bytes: 3762508364015
>> rx2_csum_complete: 6618370416
>> rx2_csum_unnecessary: 0
>> rx2_csum_unnecessary_inner: 0
>> rx2_csum_none: 0
>> rx2_xdp_drop: 0
>> rx2_xdp_redirect: 0
>> rx2_lro_packets: 0
>> rx2_lro_bytes: 0
>> rx2_ecn_mark: 0
>> rx2_removed_vlan_packets: 6618370416
>> rx2_wqe_err: 0
>> rx2_mpwqe_filler_cqes: 0
>> rx2_mpwqe_filler_strides: 0
>> rx2_buff_alloc_err: 0
>> rx2_cqe_compress_blks: 0
>> rx2_cqe_compress_pkts: 0
>> rx2_page_reuse: 0
>> rx2_cache_reuse: 111419328
>> rx2_cache_full: 1807563903
>> rx2_cache_empty: 1390208158
>> rx2_cache_busy: 1807564378
>> rx2_cache_waive: 1390201722
>> rx2_congst_umr: 0
>> rx2_arfs_err: 0
>> rx2_xdp_tx_xmit: 0
>> rx2_xdp_tx_full: 0
>> rx2_xdp_tx_err: 0
>> rx2_xdp_tx_cqes: 0
>> rx3_packets: 6665308976
>> rx3_bytes: 3828546206006
>> rx3_csum_complete: 6665308976
>> rx3_csum_unnecessary: 0
>> rx3_csum_unnecessary_inner: 0
>> rx3_csum_none: 0
>> rx3_xdp_drop: 0
>> rx3_xdp_redirect: 0
>> rx3_lro_packets: 0
>> rx3_lro_bytes: 0
>> rx3_ecn_mark: 0
>> rx3_removed_vlan_packets: 6665308976
>> rx3_wqe_err: 0
>> rx3_mpwqe_filler_cqes: 0
>> rx3_mpwqe_filler_strides: 0
>> rx3_buff_alloc_err: 0
>> rx3_cqe_compress_blks: 0
>> rx3_cqe_compress_pkts: 0
>> rx3_page_reuse: 0
>> rx3_cache_reuse: 215779091
>> rx3_cache_full: 1720040649
>> rx3_cache_empty: 1396840926
>> rx3_cache_busy: 1720041127
>> rx3_cache_waive: 1396834493
>> rx3_congst_umr: 0
>> rx3_arfs_err: 0
>> rx3_xdp_tx_xmit: 0
>> rx3_xdp_tx_full: 0
>> rx3_xdp_tx_err: 0
>> rx3_xdp_tx_cqes: 0
>> rx4_packets: 6764448165
>> rx4_bytes: 3883101339142
>> rx4_csum_complete: 6764448165
>> rx4_csum_unnecessary: 0
>> rx4_csum_unnecessary_inner: 0
>> rx4_csum_none: 0
>> rx4_xdp_drop: 0
>> rx4_xdp_redirect: 0
>> rx4_lro_packets: 0
>> rx4_lro_bytes: 0
>> rx4_ecn_mark: 0
>> rx4_removed_vlan_packets: 6764448165
>> rx4_wqe_err: 0
>> rx4_mpwqe_filler_cqes: 0
>> rx4_mpwqe_filler_strides: 0
>> rx4_buff_alloc_err: 0
>> rx4_cqe_compress_blks: 0
>> rx4_cqe_compress_pkts: 0
>> rx4_page_reuse: 0
>> rx4_cache_reuse: 1930710653
>> rx4_cache_full: 6490815
>> rx4_cache_empty: 1445028605
>> rx4_cache_busy: 6491478
>> rx4_cache_waive: 1445022392
>> rx4_congst_umr: 0
>> rx4_arfs_err: 0
>> rx4_xdp_tx_xmit: 0
>> rx4_xdp_tx_full: 0
>> rx4_xdp_tx_err: 0
>> rx4_xdp_tx_cqes: 0
>> rx5_packets: 6736853264
>> rx5_bytes: 3925186068552
>> rx5_csum_complete: 6736853264
>> rx5_csum_unnecessary: 0
>> rx5_csum_unnecessary_inner: 0
>> rx5_csum_none: 0
>> rx5_xdp_drop: 0
>> rx5_xdp_redirect: 0
>> rx5_lro_packets: 0
>> rx5_lro_bytes: 0
>> rx5_ecn_mark: 0
>> rx5_removed_vlan_packets: 6736853264
>> rx5_wqe_err: 0
>> rx5_mpwqe_filler_cqes: 0
>> rx5_mpwqe_filler_strides: 0
>> rx5_buff_alloc_err: 0
>> rx5_cqe_compress_blks: 0
>> rx5_cqe_compress_pkts: 0
>> rx5_page_reuse: 0
>> rx5_cache_reuse: 7283914
>> rx5_cache_full: 3361142463
>> rx5_cache_empty: 6656
>> rx5_cache_busy: 3361142718
>> rx5_cache_waive: 0
>> rx5_congst_umr: 0
>> rx5_arfs_err: 0
>> rx5_xdp_tx_xmit: 0
>> rx5_xdp_tx_full: 0
>> rx5_xdp_tx_err: 0
>> rx5_xdp_tx_cqes: 0
>> rx6_packets: 6751588828
>> rx6_bytes: 3860537598885
>> rx6_csum_complete: 6751588828
>> rx6_csum_unnecessary: 0
>> rx6_csum_unnecessary_inner: 0
>> rx6_csum_none: 0
>> rx6_xdp_drop: 0
>> rx6_xdp_redirect: 0
>> rx6_lro_packets: 0
>> rx6_lro_bytes: 0
>> rx6_ecn_mark: 0
>> rx6_removed_vlan_packets: 6751588828
>> rx6_wqe_err: 0
>> rx6_mpwqe_filler_cqes: 0
>> rx6_mpwqe_filler_strides: 0
>> rx6_buff_alloc_err: 0
>> rx6_cqe_compress_blks: 0
>> rx6_cqe_compress_pkts: 0
>> rx6_page_reuse: 0
>> rx6_cache_reuse: 96032126
>> rx6_cache_full: 1857890923
>> rx6_cache_empty: 1421877543
>> rx6_cache_busy: 1857891399
>> rx6_cache_waive: 1421871110
>> rx6_congst_umr: 0
>> rx6_arfs_err: 0
>> rx6_xdp_tx_xmit: 0
>> rx6_xdp_tx_full: 0
>> rx6_xdp_tx_err: 0
>> rx6_xdp_tx_cqes: 0
>> rx7_packets: 6935300074
>> rx7_bytes: 4004713524388
>> rx7_csum_complete: 6935300074
>> rx7_csum_unnecessary: 0
>> rx7_csum_unnecessary_inner: 0
>> rx7_csum_none: 0
>> rx7_xdp_drop: 0
>> rx7_xdp_redirect: 0
>> rx7_lro_packets: 0
>> rx7_lro_bytes: 0
>> rx7_ecn_mark: 0
>> rx7_removed_vlan_packets: 6935300074
>> rx7_wqe_err: 0
>> rx7_mpwqe_filler_cqes: 0
>> rx7_mpwqe_filler_strides: 0
>> rx7_buff_alloc_err: 0
>> rx7_cqe_compress_blks: 0
>> rx7_cqe_compress_pkts: 0
>> rx7_page_reuse: 0
>> rx7_cache_reuse: 17555187
>> rx7_cache_full: 3450094595
>> rx7_cache_empty: 6656
>> rx7_cache_busy: 3450094849
>> rx7_cache_waive: 0
>> rx7_congst_umr: 0
>> rx7_arfs_err: 0
>> rx7_xdp_tx_xmit: 0
>> rx7_xdp_tx_full: 0
>> rx7_xdp_tx_err: 0
>> rx7_xdp_tx_cqes: 0
>> rx8_packets: 6678640094
>> rx8_bytes: 3783722686028
>> rx8_csum_complete: 6678640094
>> rx8_csum_unnecessary: 0
>> rx8_csum_unnecessary_inner: 0
>> rx8_csum_none: 0
>> rx8_xdp_drop: 0
>> rx8_xdp_redirect: 0
>> rx8_lro_packets: 0
>> rx8_lro_bytes: 0
>> rx8_ecn_mark: 0
>> rx8_removed_vlan_packets: 6678640094
>> rx8_wqe_err: 0
>> rx8_mpwqe_filler_cqes: 0
>> rx8_mpwqe_filler_strides: 0
>> rx8_buff_alloc_err: 0
>> rx8_cqe_compress_blks: 0
>> rx8_cqe_compress_pkts: 0
>> rx8_page_reuse: 0
>> rx8_cache_reuse: 71006578
>> rx8_cache_full: 1879380649
>> rx8_cache_empty: 1388938999
>> rx8_cache_busy: 1879381123
>> rx8_cache_waive: 1388932565
>> rx8_congst_umr: 0
>> rx8_arfs_err: 0
>> rx8_xdp_tx_xmit: 0
>> rx8_xdp_tx_full: 0
>> rx8_xdp_tx_err: 0
>> rx8_xdp_tx_cqes: 0
>> rx9_packets: 6709855557
>> rx9_bytes: 3849522227880
>> rx9_csum_complete: 6709855557
>> rx9_csum_unnecessary: 0
>> rx9_csum_unnecessary_inner: 0
>> rx9_csum_none: 0
>> rx9_xdp_drop: 0
>> rx9_xdp_redirect: 0
>> rx9_lro_packets: 0
>> rx9_lro_bytes: 0
>> rx9_ecn_mark: 0
>> rx9_removed_vlan_packets: 6709855557
>> rx9_wqe_err: 0
>> rx9_mpwqe_filler_cqes: 0
>> rx9_mpwqe_filler_strides: 0
>> rx9_buff_alloc_err: 0
>> rx9_cqe_compress_blks: 0
>> rx9_cqe_compress_pkts: 0
>> rx9_page_reuse: 0
>> rx9_cache_reuse: 108980215
>> rx9_cache_full: 1822730121
>> rx9_cache_empty: 1423223623
>> rx9_cache_busy: 1822730594
>> rx9_cache_waive: 1423217187
>> rx9_congst_umr: 0
>> rx9_arfs_err: 0
>> rx9_xdp_tx_xmit: 0
>> rx9_xdp_tx_full: 0
>> rx9_xdp_tx_err: 0
>> rx9_xdp_tx_cqes: 0
>> rx10_packets: 6761861066
>> rx10_bytes: 3816266733385
>> rx10_csum_complete: 6761861066
>> rx10_csum_unnecessary: 0
>> rx10_csum_unnecessary_inner: 0
>> rx10_csum_none: 0
>> rx10_xdp_drop: 0
>> rx10_xdp_redirect: 0
>> rx10_lro_packets: 0
>> rx10_lro_bytes: 0
>> rx10_ecn_mark: 0
>> rx10_removed_vlan_packets: 6761861066
>> rx10_wqe_err: 0
>> rx10_mpwqe_filler_cqes: 0
>> rx10_mpwqe_filler_strides: 0
>> rx10_buff_alloc_err: 0
>> rx10_cqe_compress_blks: 0
>> rx10_cqe_compress_pkts: 0
>> rx10_page_reuse: 0
>> rx10_cache_reuse: 3489300
>> rx10_cache_full: 3377440977
>> rx10_cache_empty: 6656
>> rx10_cache_busy: 3377441216
>> rx10_cache_waive: 0
>> rx10_congst_umr: 0
>> rx10_arfs_err: 0
>> rx10_xdp_tx_xmit: 0
>> rx10_xdp_tx_full: 0
>> rx10_xdp_tx_err: 0
>> rx10_xdp_tx_cqes: 0
>> rx11_packets: 6868113938
>> rx11_bytes: 4048196300710
>> rx11_csum_complete: 6868113938
>> rx11_csum_unnecessary: 0
>> rx11_csum_unnecessary_inner: 0
>> rx11_csum_none: 0
>> rx11_xdp_drop: 0
>> rx11_xdp_redirect: 0
>> rx11_lro_packets: 0
>> rx11_lro_bytes: 0
>> rx11_ecn_mark: 0
>> rx11_removed_vlan_packets: 6868113938
>> rx11_wqe_err: 0
>> rx11_mpwqe_filler_cqes: 0
>> rx11_mpwqe_filler_strides: 0
>> rx11_buff_alloc_err: 0
>> rx11_cqe_compress_blks: 0
>> rx11_cqe_compress_pkts: 0
>> rx11_page_reuse: 0
>> rx11_cache_reuse: 1948516819
>> rx11_cache_full: 17132157
>> rx11_cache_empty: 1468413985
>> rx11_cache_busy: 17132820
>> rx11_cache_waive: 1468407772
>> rx11_congst_umr: 0
>> rx11_arfs_err: 0
>> rx11_xdp_tx_xmit: 0
>> rx11_xdp_tx_full: 0
>> rx11_xdp_tx_err: 0
>> rx11_xdp_tx_cqes: 0
>> rx12_packets: 6742955386
>> rx12_bytes: 3865747629271
>> rx12_csum_complete: 6742955386
>> rx12_csum_unnecessary: 0
>> rx12_csum_unnecessary_inner: 0
>> rx12_csum_none: 0
>> rx12_xdp_drop: 0
>> rx12_xdp_redirect: 0
>> rx12_lro_packets: 0
>> rx12_lro_bytes: 0
>> rx12_ecn_mark: 0
>> rx12_removed_vlan_packets: 6742955386
>> rx12_wqe_err: 0
>> rx12_mpwqe_filler_cqes: 0
>> rx12_mpwqe_filler_strides: 0
>> rx12_buff_alloc_err: 0
>> rx12_cqe_compress_blks: 0
>> rx12_cqe_compress_pkts: 0
>> rx12_page_reuse: 0
>> rx12_cache_reuse: 30809331
>> rx12_cache_full: 3340668106
>> rx12_cache_empty: 6656
>> rx12_cache_busy: 3340668333
>> rx12_cache_waive: 0
>> rx12_congst_umr: 0
>> rx12_arfs_err: 0
>> rx12_xdp_tx_xmit: 0
>> rx12_xdp_tx_full: 0
>> rx12_xdp_tx_err: 0
>> rx12_xdp_tx_cqes: 0
>> rx13_packets: 6707028036
>> rx13_bytes: 3813462190623
>> rx13_csum_complete: 6707028036
>> rx13_csum_unnecessary: 0
>> rx13_csum_unnecessary_inner: 0
>> rx13_csum_none: 0
>> rx13_xdp_drop: 0
>> rx13_xdp_redirect: 0
>> rx13_lro_packets: 0
>> rx13_lro_bytes: 0
>> rx13_ecn_mark: 0
>> rx13_removed_vlan_packets: 6707028036
>> rx13_wqe_err: 0
>> rx13_mpwqe_filler_cqes: 0
>> rx13_mpwqe_filler_strides: 0
>> rx13_buff_alloc_err: 0
>> rx13_cqe_compress_blks: 0
>> rx13_cqe_compress_pkts: 0
>> rx13_page_reuse: 0
>> rx13_cache_reuse: 14951053
>> rx13_cache_full: 3338562710
>> rx13_cache_empty: 6656
>> rx13_cache_busy: 3338562963
>> rx13_cache_waive: 0
>> rx13_congst_umr: 0
>> rx13_arfs_err: 0
>> rx13_xdp_tx_xmit: 0
>> rx13_xdp_tx_full: 0
>> rx13_xdp_tx_err: 0
>> rx13_xdp_tx_cqes: 0
>> rx14_packets: 6737074410
>> rx14_bytes: 3868905276119
>> rx14_csum_complete: 6737074410
>> rx14_csum_unnecessary: 0
>> rx14_csum_unnecessary_inner: 0
>> rx14_csum_none: 0
>> rx14_xdp_drop: 0
>> rx14_xdp_redirect: 0
>> rx14_lro_packets: 0
>> rx14_lro_bytes: 0
>> rx14_ecn_mark: 0
>> rx14_removed_vlan_packets: 6737074410
>> rx14_wqe_err: 0
>> rx14_mpwqe_filler_cqes: 0
>> rx14_mpwqe_filler_strides: 0
>> rx14_buff_alloc_err: 0
>> rx14_cqe_compress_blks: 0
>> rx14_cqe_compress_pkts: 0
>> rx14_page_reuse: 0
>> rx14_cache_reuse: 967799432
>> rx14_cache_full: 982704312
>> rx14_cache_empty: 1418039639
>> rx14_cache_busy: 982704789
>> rx14_cache_waive: 1418033206
>> rx14_congst_umr: 0
>> rx14_arfs_err: 0
>> rx14_xdp_tx_xmit: 0
>> rx14_xdp_tx_full: 0
>> rx14_xdp_tx_err: 0
>> rx14_xdp_tx_cqes: 0
>> rx15_packets: 6641887441
>> rx15_bytes: 3742874400402
>> rx15_csum_complete: 6641887441
>> rx15_csum_unnecessary: 0
>> rx15_csum_unnecessary_inner: 0
>> rx15_csum_none: 0
>> rx15_xdp_drop: 0
>> rx15_xdp_redirect: 0
>> rx15_lro_packets: 0
>> rx15_lro_bytes: 0
>> rx15_ecn_mark: 0
>> rx15_removed_vlan_packets: 6641887441
>> rx15_wqe_err: 0
>> rx15_mpwqe_filler_cqes: 0
>> rx15_mpwqe_filler_strides: 0
>> rx15_buff_alloc_err: 0
>> rx15_cqe_compress_blks: 0
>> rx15_cqe_compress_pkts: 0
>> rx15_page_reuse: 0
>> rx15_cache_reuse: 1920227538
>> rx15_cache_full: 19386129
>> rx15_cache_empty: 1381335137
>> rx15_cache_busy: 19387693
>> rx15_cache_waive: 1381329825
>> rx15_congst_umr: 0
>> rx15_arfs_err: 0
>> rx15_xdp_tx_xmit: 0
>> rx15_xdp_tx_full: 0
>> rx15_xdp_tx_err: 0
>> rx15_xdp_tx_cqes: 0
>> rx16_packets: 5420472874
>> rx16_bytes: 3079293332581
>> rx16_csum_complete: 5420472874
>> rx16_csum_unnecessary: 0
>> rx16_csum_unnecessary_inner: 0
>> rx16_csum_none: 0
>> rx16_xdp_drop: 0
>> rx16_xdp_redirect: 0
>> rx16_lro_packets: 0
>> rx16_lro_bytes: 0
>> rx16_ecn_mark: 0
>> rx16_removed_vlan_packets: 5420472874
>> rx16_wqe_err: 0
>> rx16_mpwqe_filler_cqes: 0
>> rx16_mpwqe_filler_strides: 0
>> rx16_buff_alloc_err: 0
>> rx16_cqe_compress_blks: 0
>> rx16_cqe_compress_pkts: 0
>> rx16_page_reuse: 0
>> rx16_cache_reuse: 2361079
>> rx16_cache_full: 2707875103
>> rx16_cache_empty: 6656
>> rx16_cache_busy: 2707875349
>> rx16_cache_waive: 0
>> rx16_congst_umr: 0
>> rx16_arfs_err: 0
>> rx16_xdp_tx_xmit: 0
>> rx16_xdp_tx_full: 0
>> rx16_xdp_tx_err: 0
>> rx16_xdp_tx_cqes: 0
>> rx17_packets: 5428380986
>> rx17_bytes: 3080981893118
>> rx17_csum_complete: 5428380986
>> rx17_csum_unnecessary: 0
>> rx17_csum_unnecessary_inner: 0
>> rx17_csum_none: 0
>> rx17_xdp_drop: 0
>> rx17_xdp_redirect: 0
>> rx17_lro_packets: 0
>> rx17_lro_bytes: 0
>> rx17_ecn_mark: 0
>> rx17_removed_vlan_packets: 5428380986
>> rx17_wqe_err: 0
>> rx17_mpwqe_filler_cqes: 0
>> rx17_mpwqe_filler_strides: 0
>> rx17_buff_alloc_err: 0
>> rx17_cqe_compress_blks: 0
>> rx17_cqe_compress_pkts: 0
>> rx17_page_reuse: 0
>> rx17_cache_reuse: 1552266402
>> rx17_cache_full: 5947505
>> rx17_cache_empty: 1155981856
>> rx17_cache_busy: 5948870
>> rx17_cache_waive: 1155976345
>> rx17_congst_umr: 0
>> rx17_arfs_err: 0
>> rx17_xdp_tx_xmit: 0
>> rx17_xdp_tx_full: 0
>> rx17_xdp_tx_err: 0
>> rx17_xdp_tx_cqes: 0
>> rx18_packets: 5529118410
>> rx18_bytes: 3254749573833
>> rx18_csum_complete: 5529118410
>> rx18_csum_unnecessary: 0
>> rx18_csum_unnecessary_inner: 0
>> rx18_csum_none: 0
>> rx18_xdp_drop: 0
>> rx18_xdp_redirect: 0
>> rx18_lro_packets: 0
>> rx18_lro_bytes: 0
>> rx18_ecn_mark: 0
>> rx18_removed_vlan_packets: 5529118410
>> rx18_wqe_err: 0
>> rx18_mpwqe_filler_cqes: 0
>> rx18_mpwqe_filler_strides: 0
>> rx18_buff_alloc_err: 0
>> rx18_cqe_compress_blks: 0
>> rx18_cqe_compress_pkts: 0
>> rx18_page_reuse: 0
>> rx18_cache_reuse: 67438840
>> rx18_cache_full: 1536718472
>> rx18_cache_empty: 1160408072
>> rx18_cache_busy: 1536718932
>> rx18_cache_waive: 1160401638
>> rx18_congst_umr: 0
>> rx18_arfs_err: 0
>> rx18_xdp_tx_xmit: 0
>> rx18_xdp_tx_full: 0
>> rx18_xdp_tx_err: 0
>> rx18_xdp_tx_cqes: 0
>> rx19_packets: 5449932653
>> rx19_bytes: 3148726579411
>> rx19_csum_complete: 5449932653
>> rx19_csum_unnecessary: 0
>> rx19_csum_unnecessary_inner: 0
>> rx19_csum_none: 0
>> rx19_xdp_drop: 0
>> rx19_xdp_redirect: 0
>> rx19_lro_packets: 0
>> rx19_lro_bytes: 0
>> rx19_ecn_mark: 0
>> rx19_removed_vlan_packets: 5449932653
>> rx19_wqe_err: 0
>> rx19_mpwqe_filler_cqes: 0
>> rx19_mpwqe_filler_strides: 0
>> rx19_buff_alloc_err: 0
>> rx19_cqe_compress_blks: 0
>> rx19_cqe_compress_pkts: 0
>> rx19_page_reuse: 0
>> rx19_cache_reuse: 1537841743
>> rx19_cache_full: 9920960
>> rx19_cache_empty: 1177208938
>> rx19_cache_busy: 9922299
>> rx19_cache_waive: 1177203401
>> rx19_congst_umr: 0
>> rx19_arfs_err: 0
>> rx19_xdp_tx_xmit: 0
>> rx19_xdp_tx_full: 0
>> rx19_xdp_tx_err: 0
>> rx19_xdp_tx_cqes: 0
>> rx20_packets: 5407910071
>> rx20_bytes: 3123560861922
>> rx20_csum_complete: 5407910071
>> rx20_csum_unnecessary: 0
>> rx20_csum_unnecessary_inner: 0
>> rx20_csum_none: 0
>> rx20_xdp_drop: 0
>> rx20_xdp_redirect: 0
>> rx20_lro_packets: 0
>> rx20_lro_bytes: 0
>> rx20_ecn_mark: 0
>> rx20_removed_vlan_packets: 5407910071
>> rx20_wqe_err: 0
>> rx20_mpwqe_filler_cqes: 0
>> rx20_mpwqe_filler_strides: 0
>> rx20_buff_alloc_err: 0
>> rx20_cqe_compress_blks: 0
>> rx20_cqe_compress_pkts: 0
>> rx20_page_reuse: 0
>> rx20_cache_reuse: 10255209
>> rx20_cache_full: 2693699571
>> rx20_cache_empty: 6656
>> rx20_cache_busy: 2693699823
>> rx20_cache_waive: 0
>> rx20_congst_umr: 0
>> rx20_arfs_err: 0
>> rx20_xdp_tx_xmit: 0
>> rx20_xdp_tx_full: 0
>> rx20_xdp_tx_err: 0
>> rx20_xdp_tx_cqes: 0
>> rx21_packets: 5417498508
>> rx21_bytes: 3131335892379
>> rx21_csum_complete: 5417498508
>> rx21_csum_unnecessary: 0
>> rx21_csum_unnecessary_inner: 0
>> rx21_csum_none: 0
>> rx21_xdp_drop: 0
>> rx21_xdp_redirect: 0
>> rx21_lro_packets: 0
>> rx21_lro_bytes: 0
>> rx21_ecn_mark: 0
>> rx21_removed_vlan_packets: 5417498508
>> rx21_wqe_err: 0
>> rx21_mpwqe_filler_cqes: 0
>> rx21_mpwqe_filler_strides: 0
>> rx21_buff_alloc_err: 0
>> rx21_cqe_compress_blks: 0
>> rx21_cqe_compress_pkts: 0
>> rx21_page_reuse: 0
>> rx21_cache_reuse: 192662917
>> rx21_cache_full: 1374120417
>> rx21_cache_empty: 1141972100
>> rx21_cache_busy: 1374120891
>> rx21_cache_waive: 1141965665
>> rx21_congst_umr: 0
>> rx21_arfs_err: 0
>> rx21_xdp_tx_xmit: 0
>> rx21_xdp_tx_full: 0
>> rx21_xdp_tx_err: 0
>> rx21_xdp_tx_cqes: 0
>> rx22_packets: 5613634706
>> rx22_bytes: 3240055099058
>> rx22_csum_complete: 5613634706
>> rx22_csum_unnecessary: 0
>> rx22_csum_unnecessary_inner: 0
>> rx22_csum_none: 0
>> rx22_xdp_drop: 0
>> rx22_xdp_redirect: 0
>> rx22_lro_packets: 0
>> rx22_lro_bytes: 0
>> rx22_ecn_mark: 0
>> rx22_removed_vlan_packets: 5613634706
>> rx22_wqe_err: 0
>> rx22_mpwqe_filler_cqes: 0
>> rx22_mpwqe_filler_strides: 0
>> rx22_buff_alloc_err: 0
>> rx22_cqe_compress_blks: 0
>> rx22_cqe_compress_pkts: 0
>> rx22_page_reuse: 0
>> rx22_cache_reuse: 12161531
>> rx22_cache_full: 2794655567
>> rx22_cache_empty: 6656
>> rx22_cache_busy: 2794655821
>> rx22_cache_waive: 0
>> rx22_congst_umr: 0
>> rx22_arfs_err: 0
>> rx22_xdp_tx_xmit: 0
>> rx22_xdp_tx_full: 0
>> rx22_xdp_tx_err: 0
>> rx22_xdp_tx_cqes: 0
>> rx23_packets: 5389977167
>> rx23_bytes: 3054270771559
>> rx23_csum_complete: 5389977167
>> rx23_csum_unnecessary: 0
>> rx23_csum_unnecessary_inner: 0
>> rx23_csum_none: 0
>> rx23_xdp_drop: 0
>> rx23_xdp_redirect: 0
>> rx23_lro_packets: 0
>> rx23_lro_bytes: 0
>> rx23_ecn_mark: 0
>> rx23_removed_vlan_packets: 5389977167
>> rx23_wqe_err: 0
>> rx23_mpwqe_filler_cqes: 0
>> rx23_mpwqe_filler_strides: 0
>> rx23_buff_alloc_err: 0
>> rx23_cqe_compress_blks: 0
>> rx23_cqe_compress_pkts: 0
>> rx23_page_reuse: 0
>> rx23_cache_reuse: 709328
>> rx23_cache_full: 2694279000
>> rx23_cache_empty: 6656
>> rx23_cache_busy: 2694279252
>> rx23_cache_waive: 0
>> rx23_congst_umr: 0
>> rx23_arfs_err: 0
>> rx23_xdp_tx_xmit: 0
>> rx23_xdp_tx_full: 0
>> rx23_xdp_tx_err: 0
>> rx23_xdp_tx_cqes: 0
>> rx24_packets: 5547561932
>> rx24_bytes: 3166602453443
>> rx24_csum_complete: 5547561932
>> rx24_csum_unnecessary: 0
>> rx24_csum_unnecessary_inner: 0
>> rx24_csum_none: 0
>> rx24_xdp_drop: 0
>> rx24_xdp_redirect: 0
>> rx24_lro_packets: 0
>> rx24_lro_bytes: 0
>> rx24_ecn_mark: 0
>> rx24_removed_vlan_packets: 5547561932
>> rx24_wqe_err: 0
>> rx24_mpwqe_filler_cqes: 0
>> rx24_mpwqe_filler_strides: 0
>> rx24_buff_alloc_err: 0
>> rx24_cqe_compress_blks: 0
>> rx24_cqe_compress_pkts: 0
>> rx24_page_reuse: 0
>> rx24_cache_reuse: 57885119
>> rx24_cache_full: 1529450077
>> rx24_cache_empty: 1186451948
>> rx24_cache_busy: 1529450553
>> rx24_cache_waive: 1186445515
>> rx24_congst_umr: 0
>> rx24_arfs_err: 0
>> rx24_xdp_tx_xmit: 0
>> rx24_xdp_tx_full: 0
>> rx24_xdp_tx_err: 0
>> rx24_xdp_tx_cqes: 0
>> rx25_packets: 5414569326
>> rx25_bytes: 3184757708091
>> rx25_csum_complete: 5414569326
>> rx25_csum_unnecessary: 0
>> rx25_csum_unnecessary_inner: 0
>> rx25_csum_none: 0
>> rx25_xdp_drop: 0
>> rx25_xdp_redirect: 0
>> rx25_lro_packets: 0
>> rx25_lro_bytes: 0
>> rx25_ecn_mark: 0
>> rx25_removed_vlan_packets: 5414569326
>> rx25_wqe_err: 0
>> rx25_mpwqe_filler_cqes: 0
>> rx25_mpwqe_filler_strides: 0
>> rx25_buff_alloc_err: 0
>> rx25_cqe_compress_blks: 0
>> rx25_cqe_compress_pkts: 0
>> rx25_page_reuse: 0
>> rx25_cache_reuse: 5080853
>> rx25_cache_full: 2702203555
>> rx25_cache_empty: 6656
>> rx25_cache_busy: 2702203807
>> rx25_cache_waive: 0
>> rx25_congst_umr: 0
>> rx25_arfs_err: 0
>> rx25_xdp_tx_xmit: 0
>> rx25_xdp_tx_full: 0
>> rx25_xdp_tx_err: 0
>> rx25_xdp_tx_cqes: 0
>> rx26_packets: 5479972151
>> rx26_bytes: 3110642276239
>> rx26_csum_complete: 5479972151
>> rx26_csum_unnecessary: 0
>> rx26_csum_unnecessary_inner: 0
>> rx26_csum_none: 0
>> rx26_xdp_drop: 0
>> rx26_xdp_redirect: 0
>> rx26_lro_packets: 0
>> rx26_lro_bytes: 0
>> rx26_ecn_mark: 0
>> rx26_removed_vlan_packets: 5479972151
>> rx26_wqe_err: 0
>> rx26_mpwqe_filler_cqes: 0
>> rx26_mpwqe_filler_strides: 0
>> rx26_buff_alloc_err: 0
>> rx26_cqe_compress_blks: 0
>> rx26_cqe_compress_pkts: 0
>> rx26_page_reuse: 0
>> rx26_cache_reuse: 26543335
>> rx26_cache_full: 2713442485
>> rx26_cache_empty: 6656
>> rx26_cache_busy: 2713442737
>> rx26_cache_waive: 0
>> rx26_congst_umr: 0
>> rx26_arfs_err: 0
>> rx26_xdp_tx_xmit: 0
>> rx26_xdp_tx_full: 0
>> rx26_xdp_tx_err: 0
>> rx26_xdp_tx_cqes: 0
>> rx27_packets: 5337113900
>> rx27_bytes: 3068966906075
>> rx27_csum_complete: 5337113900
>> rx27_csum_unnecessary: 0
>> rx27_csum_unnecessary_inner: 0
>> rx27_csum_none: 0
>> rx27_xdp_drop: 0
>> rx27_xdp_redirect: 0
>> rx27_lro_packets: 0
>> rx27_lro_bytes: 0
>> rx27_ecn_mark: 0
>> rx27_removed_vlan_packets: 5337113900
>> rx27_wqe_err: 0
>> rx27_mpwqe_filler_cqes: 0
>> rx27_mpwqe_filler_strides: 0
>> rx27_buff_alloc_err: 0
>> rx27_cqe_compress_blks: 0
>> rx27_cqe_compress_pkts: 0
>> rx27_page_reuse: 0
>> rx27_cache_reuse: 1539298962
>> rx27_cache_full: 10861919
>> rx27_cache_empty: 1117173179
>> rx27_cache_busy: 12091463
>> rx27_cache_waive: 1118395847
>> rx27_congst_umr: 0
>> rx27_arfs_err: 0
>> rx27_xdp_tx_xmit: 0
>> rx27_xdp_tx_full: 0
>> rx27_xdp_tx_err: 0
>> rx27_xdp_tx_cqes: 0
>> rx28_packets: 0
>> rx28_bytes: 0
>> rx28_csum_complete: 0
>> rx28_csum_unnecessary: 0
>> rx28_csum_unnecessary_inner: 0
>> rx28_csum_none: 0
>> rx28_xdp_drop: 0
>> rx28_xdp_redirect: 0
>> rx28_lro_packets: 0
>> rx28_lro_bytes: 0
>> rx28_ecn_mark: 0
>> rx28_removed_vlan_packets: 0
>> rx28_wqe_err: 0
>> rx28_mpwqe_filler_cqes: 0
>> rx28_mpwqe_filler_strides: 0
>> rx28_buff_alloc_err: 0
>> rx28_cqe_compress_blks: 0
>> rx28_cqe_compress_pkts: 0
>> rx28_page_reuse: 0
>> rx28_cache_reuse: 0
>> rx28_cache_full: 0
>> rx28_cache_empty: 2560
>> rx28_cache_busy: 0
>> rx28_cache_waive: 0
>> rx28_congst_umr: 0
>> rx28_arfs_err: 0
>> rx28_xdp_tx_xmit: 0
>> rx28_xdp_tx_full: 0
>> rx28_xdp_tx_err: 0
>> rx28_xdp_tx_cqes: 0
>> rx29_packets: 0
>> rx29_bytes: 0
>> rx29_csum_complete: 0
>> rx29_csum_unnecessary: 0
>> rx29_csum_unnecessary_inner: 0
>> rx29_csum_none: 0
>> rx29_xdp_drop: 0
>> rx29_xdp_redirect: 0
>> rx29_lro_packets: 0
>> rx29_lro_bytes: 0
>> rx29_ecn_mark: 0
>> rx29_removed_vlan_packets: 0
>> rx29_wqe_err: 0
>> rx29_mpwqe_filler_cqes: 0
>> rx29_mpwqe_filler_strides: 0
>> rx29_buff_alloc_err: 0
>> rx29_cqe_compress_blks: 0
>> rx29_cqe_compress_pkts: 0
>> rx29_page_reuse: 0
>> rx29_cache_reuse: 0
>> rx29_cache_full: 0
>> rx29_cache_empty: 2560
>> rx29_cache_busy: 0
>> rx29_cache_waive: 0
>> rx29_congst_umr: 0
>> rx29_arfs_err: 0
>> rx29_xdp_tx_xmit: 0
>> rx29_xdp_tx_full: 0
>> rx29_xdp_tx_err: 0
>> rx29_xdp_tx_cqes: 0
>> rx30_packets: 0
>> rx30_bytes: 0
>> rx30_csum_complete: 0
>> rx30_csum_unnecessary: 0
>> rx30_csum_unnecessary_inner: 0
>> rx30_csum_none: 0
>> rx30_xdp_drop: 0
>> rx30_xdp_redirect: 0
>> rx30_lro_packets: 0
>> rx30_lro_bytes: 0
>> rx30_ecn_mark: 0
>> rx30_removed_vlan_packets: 0
>> rx30_wqe_err: 0
>> rx30_mpwqe_filler_cqes: 0
>> rx30_mpwqe_filler_strides: 0
>> rx30_buff_alloc_err: 0
>> rx30_cqe_compress_blks: 0
>> rx30_cqe_compress_pkts: 0
>> rx30_page_reuse: 0
>> rx30_cache_reuse: 0
>> rx30_cache_full: 0
>> rx30_cache_empty: 2560
>> rx30_cache_busy: 0
>> rx30_cache_waive: 0
>> rx30_congst_umr: 0
>> rx30_arfs_err: 0
>> rx30_xdp_tx_xmit: 0
>> rx30_xdp_tx_full: 0
>> rx30_xdp_tx_err: 0
>> rx30_xdp_tx_cqes: 0
>> rx31_packets: 0
>> rx31_bytes: 0
>> rx31_csum_complete: 0
>> rx31_csum_unnecessary: 0
>> rx31_csum_unnecessary_inner: 0
>> rx31_csum_none: 0
>> rx31_xdp_drop: 0
>> rx31_xdp_redirect: 0
>> rx31_lro_packets: 0
>> rx31_lro_bytes: 0
>> rx31_ecn_mark: 0
>> rx31_removed_vlan_packets: 0
>> rx31_wqe_err: 0
>> rx31_mpwqe_filler_cqes: 0
>> rx31_mpwqe_filler_strides: 0
>> rx31_buff_alloc_err: 0
>> rx31_cqe_compress_blks: 0
>> rx31_cqe_compress_pkts: 0
>> rx31_page_reuse: 0
>> rx31_cache_reuse: 0
>> rx31_cache_full: 0
>> rx31_cache_empty: 2560
>> rx31_cache_busy: 0
>> rx31_cache_waive: 0
>> rx31_congst_umr: 0
>> rx31_arfs_err: 0
>> rx31_xdp_tx_xmit: 0
>> rx31_xdp_tx_full: 0
>> rx31_xdp_tx_err: 0
>> rx31_xdp_tx_cqes: 0
>> rx32_packets: 0
>> rx32_bytes: 0
>> rx32_csum_complete: 0
>> rx32_csum_unnecessary: 0
>> rx32_csum_unnecessary_inner: 0
>> rx32_csum_none: 0
>> rx32_xdp_drop: 0
>> rx32_xdp_redirect: 0
>> rx32_lro_packets: 0
>> rx32_lro_bytes: 0
>> rx32_ecn_mark: 0
>> rx32_removed_vlan_packets: 0
>> rx32_wqe_err: 0
>> rx32_mpwqe_filler_cqes: 0
>> rx32_mpwqe_filler_strides: 0
>> rx32_buff_alloc_err: 0
>> rx32_cqe_compress_blks: 0
>> rx32_cqe_compress_pkts: 0
>> rx32_page_reuse: 0
>> rx32_cache_reuse: 0
>> rx32_cache_full: 0
>> rx32_cache_empty: 2560
>> rx32_cache_busy: 0
>> rx32_cache_waive: 0
>> rx32_congst_umr: 0
>> rx32_arfs_err: 0
>> rx32_xdp_tx_xmit: 0
>> rx32_xdp_tx_full: 0
>> rx32_xdp_tx_err: 0
>> rx32_xdp_tx_cqes: 0
>> rx33_packets: 0
>> rx33_bytes: 0
>> rx33_csum_complete: 0
>> rx33_csum_unnecessary: 0
>> rx33_csum_unnecessary_inner: 0
>> rx33_csum_none: 0
>> rx33_xdp_drop: 0
>> rx33_xdp_redirect: 0
>> rx33_lro_packets: 0
>> rx33_lro_bytes: 0
>> rx33_ecn_mark: 0
>> rx33_removed_vlan_packets: 0
>> rx33_wqe_err: 0
>> rx33_mpwqe_filler_cqes: 0
>> rx33_mpwqe_filler_strides: 0
>> rx33_buff_alloc_err: 0
>> rx33_cqe_compress_blks: 0
>> rx33_cqe_compress_pkts: 0
>> rx33_page_reuse: 0
>> rx33_cache_reuse: 0
>> rx33_cache_full: 0
>> rx33_cache_empty: 2560
>> rx33_cache_busy: 0
>> rx33_cache_waive: 0
>> rx33_congst_umr: 0
>> rx33_arfs_err: 0
>> rx33_xdp_tx_xmit: 0
>> rx33_xdp_tx_full: 0
>> rx33_xdp_tx_err: 0
>> rx33_xdp_tx_cqes: 0
>> rx34_packets: 0
>> rx34_bytes: 0
>> rx34_csum_complete: 0
>> rx34_csum_unnecessary: 0
>> rx34_csum_unnecessary_inner: 0
>> rx34_csum_none: 0
>> rx34_xdp_drop: 0
>> rx34_xdp_redirect: 0
>> rx34_lro_packets: 0
>> rx34_lro_bytes: 0
>> rx34_ecn_mark: 0
>> rx34_removed_vlan_packets: 0
>> rx34_wqe_err: 0
>> rx34_mpwqe_filler_cqes: 0
>> rx34_mpwqe_filler_strides: 0
>> rx34_buff_alloc_err: 0
>> rx34_cqe_compress_blks: 0
>> rx34_cqe_compress_pkts: 0
>> rx34_page_reuse: 0
>> rx34_cache_reuse: 0
>> rx34_cache_full: 0
>> rx34_cache_empty: 2560
>> rx34_cache_busy: 0
>> rx34_cache_waive: 0
>> rx34_congst_umr: 0
>> rx34_arfs_err: 0
>> rx34_xdp_tx_xmit: 0
>> rx34_xdp_tx_full: 0
>> rx34_xdp_tx_err: 0
>> rx34_xdp_tx_cqes: 0
>> rx35_packets: 0
>> rx35_bytes: 0
>> rx35_csum_complete: 0
>> rx35_csum_unnecessary: 0
>> rx35_csum_unnecessary_inner: 0
>> rx35_csum_none: 0
>> rx35_xdp_drop: 0
>> rx35_xdp_redirect: 0
>> rx35_lro_packets: 0
>> rx35_lro_bytes: 0
>> rx35_ecn_mark: 0
>> rx35_removed_vlan_packets: 0
>> rx35_wqe_err: 0
>> rx35_mpwqe_filler_cqes: 0
>> rx35_mpwqe_filler_strides: 0
>> rx35_buff_alloc_err: 0
>> rx35_cqe_compress_blks: 0
>> rx35_cqe_compress_pkts: 0
>> rx35_page_reuse: 0
>> rx35_cache_reuse: 0
>> rx35_cache_full: 0
>> rx35_cache_empty: 2560
>> rx35_cache_busy: 0
>> rx35_cache_waive: 0
>> rx35_congst_umr: 0
>> rx35_arfs_err: 0
>> rx35_xdp_tx_xmit: 0
>> rx35_xdp_tx_full: 0
>> rx35_xdp_tx_err: 0
>> rx35_xdp_tx_cqes: 0
>> rx36_packets: 0
>> rx36_bytes: 0
>> rx36_csum_complete: 0
>> rx36_csum_unnecessary: 0
>> rx36_csum_unnecessary_inner: 0
>> rx36_csum_none: 0
>> rx36_xdp_drop: 0
>> rx36_xdp_redirect: 0
>> rx36_lro_packets: 0
>> rx36_lro_bytes: 0
>> rx36_ecn_mark: 0
>> rx36_removed_vlan_packets: 0
>> rx36_wqe_err: 0
>> rx36_mpwqe_filler_cqes: 0
>> rx36_mpwqe_filler_strides: 0
>> rx36_buff_alloc_err: 0
>> rx36_cqe_compress_blks: 0
>> rx36_cqe_compress_pkts: 0
>> rx36_page_reuse: 0
>> rx36_cache_reuse: 0
>> rx36_cache_full: 0
>> rx36_cache_empty: 2560
>> rx36_cache_busy: 0
>> rx36_cache_waive: 0
>> rx36_congst_umr: 0
>> rx36_arfs_err: 0
>> rx36_xdp_tx_xmit: 0
>> rx36_xdp_tx_full: 0
>> rx36_xdp_tx_err: 0
>> rx36_xdp_tx_cqes: 0
>> rx37_packets: 0
>> rx37_bytes: 0
>> rx37_csum_complete: 0
>> rx37_csum_unnecessary: 0
>> rx37_csum_unnecessary_inner: 0
>> rx37_csum_none: 0
>> rx37_xdp_drop: 0
>> rx37_xdp_redirect: 0
>> rx37_lro_packets: 0
>> rx37_lro_bytes: 0
>> rx37_ecn_mark: 0
>> rx37_removed_vlan_packets: 0
>> rx37_wqe_err: 0
>> rx37_mpwqe_filler_cqes: 0
>> rx37_mpwqe_filler_strides: 0
>> rx37_buff_alloc_err: 0
>> rx37_cqe_compress_blks: 0
>> rx37_cqe_compress_pkts: 0
>> rx37_page_reuse: 0
>> rx37_cache_reuse: 0
>> rx37_cache_full: 0
>> rx37_cache_empty: 2560
>> rx37_cache_busy: 0
>> rx37_cache_waive: 0
>> rx37_congst_umr: 0
>> rx37_arfs_err: 0
>> rx37_xdp_tx_xmit: 0
>> rx37_xdp_tx_full: 0
>> rx37_xdp_tx_err: 0
>> rx37_xdp_tx_cqes: 0
>> rx38_packets: 0
>> rx38_bytes: 0
>> rx38_csum_complete: 0
>> rx38_csum_unnecessary: 0
>> rx38_csum_unnecessary_inner: 0
>> rx38_csum_none: 0
>> rx38_xdp_drop: 0
>> rx38_xdp_redirect: 0
>> rx38_lro_packets: 0
>> rx38_lro_bytes: 0
>> rx38_ecn_mark: 0
>> rx38_removed_vlan_packets: 0
>> rx38_wqe_err: 0
>> rx38_mpwqe_filler_cqes: 0
>> rx38_mpwqe_filler_strides: 0
>> rx38_buff_alloc_err: 0
>> rx38_cqe_compress_blks: 0
>> rx38_cqe_compress_pkts: 0
>> rx38_page_reuse: 0
>> rx38_cache_reuse: 0
>> rx38_cache_full: 0
>> rx38_cache_empty: 2560
>> rx38_cache_busy: 0
>> rx38_cache_waive: 0
>> rx38_congst_umr: 0
>> rx38_arfs_err: 0
>> rx38_xdp_tx_xmit: 0
>> rx38_xdp_tx_full: 0
>> rx38_xdp_tx_err: 0
>> rx38_xdp_tx_cqes: 0
>> rx39_packets: 0
>> rx39_bytes: 0
>> rx39_csum_complete: 0
>> rx39_csum_unnecessary: 0
>> rx39_csum_unnecessary_inner: 0
>> rx39_csum_none: 0
>> rx39_xdp_drop: 0
>> rx39_xdp_redirect: 0
>> rx39_lro_packets: 0
>> rx39_lro_bytes: 0
>> rx39_ecn_mark: 0
>> rx39_removed_vlan_packets: 0
>> rx39_wqe_err: 0
>> rx39_mpwqe_filler_cqes: 0
>> rx39_mpwqe_filler_strides: 0
>> rx39_buff_alloc_err: 0
>> rx39_cqe_compress_blks: 0
>> rx39_cqe_compress_pkts: 0
>> rx39_page_reuse: 0
>> rx39_cache_reuse: 0
>> rx39_cache_full: 0
>> rx39_cache_empty: 2560
>> rx39_cache_busy: 0
>> rx39_cache_waive: 0
>> rx39_congst_umr: 0
>> rx39_arfs_err: 0
>> rx39_xdp_tx_xmit: 0
>> rx39_xdp_tx_full: 0
>> rx39_xdp_tx_err: 0
>> rx39_xdp_tx_cqes: 0
>> rx40_packets: 0
>> rx40_bytes: 0
>> rx40_csum_complete: 0
>> rx40_csum_unnecessary: 0
>> rx40_csum_unnecessary_inner: 0
>> rx40_csum_none: 0
>> rx40_xdp_drop: 0
>> rx40_xdp_redirect: 0
>> rx40_lro_packets: 0
>> rx40_lro_bytes: 0
>> rx40_ecn_mark: 0
>> rx40_removed_vlan_packets: 0
>> rx40_wqe_err: 0
>> rx40_mpwqe_filler_cqes: 0
>> rx40_mpwqe_filler_strides: 0
>> rx40_buff_alloc_err: 0
>> rx40_cqe_compress_blks: 0
>> rx40_cqe_compress_pkts: 0
>> rx40_page_reuse: 0
>> rx40_cache_reuse: 0
>> rx40_cache_full: 0
>> rx40_cache_empty: 2560
>> rx40_cache_busy: 0
>> rx40_cache_waive: 0
>> rx40_congst_umr: 0
>> rx40_arfs_err: 0
>> rx40_xdp_tx_xmit: 0
>> rx40_xdp_tx_full: 0
>> rx40_xdp_tx_err: 0
>> rx40_xdp_tx_cqes: 0
>> rx41_packets: 0
>> rx41_bytes: 0
>> rx41_csum_complete: 0
>> rx41_csum_unnecessary: 0
>> rx41_csum_unnecessary_inner: 0
>> rx41_csum_none: 0
>> rx41_xdp_drop: 0
>> rx41_xdp_redirect: 0
>> rx41_lro_packets: 0
>> rx41_lro_bytes: 0
>> rx41_ecn_mark: 0
>> rx41_removed_vlan_packets: 0
>> rx41_wqe_err: 0
>> rx41_mpwqe_filler_cqes: 0
>> rx41_mpwqe_filler_strides: 0
>> rx41_buff_alloc_err: 0
>> rx41_cqe_compress_blks: 0
>> rx41_cqe_compress_pkts: 0
>> rx41_page_reuse: 0
>> rx41_cache_reuse: 0
>> rx41_cache_full: 0
>> rx41_cache_empty: 2560
>> rx41_cache_busy: 0
>> rx41_cache_waive: 0
>> rx41_congst_umr: 0
>> rx41_arfs_err: 0
>> rx41_xdp_tx_xmit: 0
>> rx41_xdp_tx_full: 0
>> rx41_xdp_tx_err: 0
>> rx41_xdp_tx_cqes: 0
>> rx42_packets: 0
>> rx42_bytes: 0
>> rx42_csum_complete: 0
>> rx42_csum_unnecessary: 0
>> rx42_csum_unnecessary_inner: 0
>> rx42_csum_none: 0
>> rx42_xdp_drop: 0
>> rx42_xdp_redirect: 0
>> rx42_lro_packets: 0
>> rx42_lro_bytes: 0
>> rx42_ecn_mark: 0
>> rx42_removed_vlan_packets: 0
>> rx42_wqe_err: 0
>> rx42_mpwqe_filler_cqes: 0
>> rx42_mpwqe_filler_strides: 0
>> rx42_buff_alloc_err: 0
>> rx42_cqe_compress_blks: 0
>> rx42_cqe_compress_pkts: 0
>> rx42_page_reuse: 0
>> rx42_cache_reuse: 0
>> rx42_cache_full: 0
>> rx42_cache_empty: 2560
>> rx42_cache_busy: 0
>> rx42_cache_waive: 0
>> rx42_congst_umr: 0
>> rx42_arfs_err: 0
>> rx42_xdp_tx_xmit: 0
>> rx42_xdp_tx_full: 0
>> rx42_xdp_tx_err: 0
>> rx42_xdp_tx_cqes: 0
>> rx43_packets: 0
>> rx43_bytes: 0
>> rx43_csum_complete: 0
>> rx43_csum_unnecessary: 0
>> rx43_csum_unnecessary_inner: 0
>> rx43_csum_none: 0
>> rx43_xdp_drop: 0
>> rx43_xdp_redirect: 0
>> rx43_lro_packets: 0
>> rx43_lro_bytes: 0
>> rx43_ecn_mark: 0
>> rx43_removed_vlan_packets: 0
>> rx43_wqe_err: 0
>> rx43_mpwqe_filler_cqes: 0
>> rx43_mpwqe_filler_strides: 0
>> rx43_buff_alloc_err: 0
>> rx43_cqe_compress_blks: 0
>> rx43_cqe_compress_pkts: 0
>> rx43_page_reuse: 0
>> rx43_cache_reuse: 0
>> rx43_cache_full: 0
>> rx43_cache_empty: 2560
>> rx43_cache_busy: 0
>> rx43_cache_waive: 0
>> rx43_congst_umr: 0
>> rx43_arfs_err: 0
>> rx43_xdp_tx_xmit: 0
>> rx43_xdp_tx_full: 0
>> rx43_xdp_tx_err: 0
>> rx43_xdp_tx_cqes: 0
>> rx44_packets: 0
>> rx44_bytes: 0
>> rx44_csum_complete: 0
>> rx44_csum_unnecessary: 0
>> rx44_csum_unnecessary_inner: 0
>> rx44_csum_none: 0
>> rx44_xdp_drop: 0
>> rx44_xdp_redirect: 0
>> rx44_lro_packets: 0
>> rx44_lro_bytes: 0
>> rx44_ecn_mark: 0
>> rx44_removed_vlan_packets: 0
>> rx44_wqe_err: 0
>> rx44_mpwqe_filler_cqes: 0
>> rx44_mpwqe_filler_strides: 0
>> rx44_buff_alloc_err: 0
>> rx44_cqe_compress_blks: 0
>> rx44_cqe_compress_pkts: 0
>> rx44_page_reuse: 0
>> rx44_cache_reuse: 0
>> rx44_cache_full: 0
>> rx44_cache_empty: 2560
>> rx44_cache_busy: 0
>> rx44_cache_waive: 0
>> rx44_congst_umr: 0
>> rx44_arfs_err: 0
>> rx44_xdp_tx_xmit: 0
>> rx44_xdp_tx_full: 0
>> rx44_xdp_tx_err: 0
>> rx44_xdp_tx_cqes: 0
>> rx45_packets: 0
>> rx45_bytes: 0
>> rx45_csum_complete: 0
>> rx45_csum_unnecessary: 0
>> rx45_csum_unnecessary_inner: 0
>> rx45_csum_none: 0
>> rx45_xdp_drop: 0
>> rx45_xdp_redirect: 0
>> rx45_lro_packets: 0
>> rx45_lro_bytes: 0
>> rx45_ecn_mark: 0
>> rx45_removed_vlan_packets: 0
>> rx45_wqe_err: 0
>> rx45_mpwqe_filler_cqes: 0
>> rx45_mpwqe_filler_strides: 0
>> rx45_buff_alloc_err: 0
>> rx45_cqe_compress_blks: 0
>> rx45_cqe_compress_pkts: 0
>> rx45_page_reuse: 0
>> rx45_cache_reuse: 0
>> rx45_cache_full: 0
>> rx45_cache_empty: 2560
>> rx45_cache_busy: 0
>> rx45_cache_waive: 0
>> rx45_congst_umr: 0
>> rx45_arfs_err: 0
>> rx45_xdp_tx_xmit: 0
>> rx45_xdp_tx_full: 0
>> rx45_xdp_tx_err: 0
>> rx45_xdp_tx_cqes: 0
>> rx46_packets: 0
>> rx46_bytes: 0
>> rx46_csum_complete: 0
>> rx46_csum_unnecessary: 0
>> rx46_csum_unnecessary_inner: 0
>> rx46_csum_none: 0
>> rx46_xdp_drop: 0
>> rx46_xdp_redirect: 0
>> rx46_lro_packets: 0
>> rx46_lro_bytes: 0
>> rx46_ecn_mark: 0
>> rx46_removed_vlan_packets: 0
>> rx46_wqe_err: 0
>> rx46_mpwqe_filler_cqes: 0
>> rx46_mpwqe_filler_strides: 0
>> rx46_buff_alloc_err: 0
>> rx46_cqe_compress_blks: 0
>> rx46_cqe_compress_pkts: 0
>> rx46_page_reuse: 0
>> rx46_cache_reuse: 0
>> rx46_cache_full: 0
>> rx46_cache_empty: 2560
>> rx46_cache_busy: 0
>> rx46_cache_waive: 0
>> rx46_congst_umr: 0
>> rx46_arfs_err: 0
>> rx46_xdp_tx_xmit: 0
>> rx46_xdp_tx_full: 0
>> rx46_xdp_tx_err: 0
>> rx46_xdp_tx_cqes: 0
>> rx47_packets: 0
>> rx47_bytes: 0
>> rx47_csum_complete: 0
>> rx47_csum_unnecessary: 0
>> rx47_csum_unnecessary_inner: 0
>> rx47_csum_none: 0
>> rx47_xdp_drop: 0
>> rx47_xdp_redirect: 0
>> rx47_lro_packets: 0
>> rx47_lro_bytes: 0
>> rx47_ecn_mark: 0
>> rx47_removed_vlan_packets: 0
>> rx47_wqe_err: 0
>> rx47_mpwqe_filler_cqes: 0
>> rx47_mpwqe_filler_strides: 0
>> rx47_buff_alloc_err: 0
>> rx47_cqe_compress_blks: 0
>> rx47_cqe_compress_pkts: 0
>> rx47_page_reuse: 0
>> rx47_cache_reuse: 0
>> rx47_cache_full: 0
>> rx47_cache_empty: 2560
>> rx47_cache_busy: 0
>> rx47_cache_waive: 0
>> rx47_congst_umr: 0
>> rx47_arfs_err: 0
>> rx47_xdp_tx_xmit: 0
>> rx47_xdp_tx_full: 0
>> rx47_xdp_tx_err: 0
>> rx47_xdp_tx_cqes: 0
>> rx48_packets: 0
>> rx48_bytes: 0
>> rx48_csum_complete: 0
>> rx48_csum_unnecessary: 0
>> rx48_csum_unnecessary_inner: 0
>> rx48_csum_none: 0
>> rx48_xdp_drop: 0
>> rx48_xdp_redirect: 0
>> rx48_lro_packets: 0
>> rx48_lro_bytes: 0
>> rx48_ecn_mark: 0
>> rx48_removed_vlan_packets: 0
>> rx48_wqe_err: 0
>> rx48_mpwqe_filler_cqes: 0
>> rx48_mpwqe_filler_strides: 0
>> rx48_buff_alloc_err: 0
>> rx48_cqe_compress_blks: 0
>> rx48_cqe_compress_pkts: 0
>> rx48_page_reuse: 0
>> rx48_cache_reuse: 0
>> rx48_cache_full: 0
>> rx48_cache_empty: 2560
>> rx48_cache_busy: 0
>> rx48_cache_waive: 0
>> rx48_congst_umr: 0
>> rx48_arfs_err: 0
>> rx48_xdp_tx_xmit: 0
>> rx48_xdp_tx_full: 0
>> rx48_xdp_tx_err: 0
>> rx48_xdp_tx_cqes: 0
>> rx49_packets: 0
>> rx49_bytes: 0
>> rx49_csum_complete: 0
>> rx49_csum_unnecessary: 0
>> rx49_csum_unnecessary_inner: 0
>> rx49_csum_none: 0
>> rx49_xdp_drop: 0
>> rx49_xdp_redirect: 0
>> rx49_lro_packets: 0
>> rx49_lro_bytes: 0
>> rx49_ecn_mark: 0
>> rx49_removed_vlan_packets: 0
>> rx49_wqe_err: 0
>> rx49_mpwqe_filler_cqes: 0
>> rx49_mpwqe_filler_strides: 0
>> rx49_buff_alloc_err: 0
>> rx49_cqe_compress_blks: 0
>> rx49_cqe_compress_pkts: 0
>> rx49_page_reuse: 0
>> rx49_cache_reuse: 0
>> rx49_cache_full: 0
>> rx49_cache_empty: 2560
>> rx49_cache_busy: 0
>> rx49_cache_waive: 0
>> rx49_congst_umr: 0
>> rx49_arfs_err: 0
>> rx49_xdp_tx_xmit: 0
>> rx49_xdp_tx_full: 0
>> rx49_xdp_tx_err: 0
>> rx49_xdp_tx_cqes: 0
>> rx50_packets: 0
>> rx50_bytes: 0
>> rx50_csum_complete: 0
>> rx50_csum_unnecessary: 0
>> rx50_csum_unnecessary_inner: 0
>> rx50_csum_none: 0
>> rx50_xdp_drop: 0
>> rx50_xdp_redirect: 0
>> rx50_lro_packets: 0
>> rx50_lro_bytes: 0
>> rx50_ecn_mark: 0
>> rx50_removed_vlan_packets: 0
>> rx50_wqe_err: 0
>> rx50_mpwqe_filler_cqes: 0
>> rx50_mpwqe_filler_strides: 0
>> rx50_buff_alloc_err: 0
>> rx50_cqe_compress_blks: 0
>> rx50_cqe_compress_pkts: 0
>> rx50_page_reuse: 0
>> rx50_cache_reuse: 0
>> rx50_cache_full: 0
>> rx50_cache_empty: 2560
>> rx50_cache_busy: 0
>> rx50_cache_waive: 0
>> rx50_congst_umr: 0
>> rx50_arfs_err: 0
>> rx50_xdp_tx_xmit: 0
>> rx50_xdp_tx_full: 0
>> rx50_xdp_tx_err: 0
>> rx50_xdp_tx_cqes: 0
>> rx51_packets: 0
>> rx51_bytes: 0
>> rx51_csum_complete: 0
>> rx51_csum_unnecessary: 0
>> rx51_csum_unnecessary_inner: 0
>> rx51_csum_none: 0
>> rx51_xdp_drop: 0
>> rx51_xdp_redirect: 0
>> rx51_lro_packets: 0
>> rx51_lro_bytes: 0
>> rx51_ecn_mark: 0
>> rx51_removed_vlan_packets: 0
>> rx51_wqe_err: 0
>> rx51_mpwqe_filler_cqes: 0
>> rx51_mpwqe_filler_strides: 0
>> rx51_buff_alloc_err: 0
>> rx51_cqe_compress_blks: 0
>> rx51_cqe_compress_pkts: 0
>> rx51_page_reuse: 0
>> rx51_cache_reuse: 0
>> rx51_cache_full: 0
>> rx51_cache_empty: 2560
>> rx51_cache_busy: 0
>> rx51_cache_waive: 0
>> rx51_congst_umr: 0
>> rx51_arfs_err: 0
>> rx51_xdp_tx_xmit: 0
>> rx51_xdp_tx_full: 0
>> rx51_xdp_tx_err: 0
>> rx51_xdp_tx_cqes: 0
>> rx52_packets: 0
>> rx52_bytes: 0
>> rx52_csum_complete: 0
>> rx52_csum_unnecessary: 0
>> rx52_csum_unnecessary_inner: 0
>> rx52_csum_none: 0
>> rx52_xdp_drop: 0
>> rx52_xdp_redirect: 0
>> rx52_lro_packets: 0
>> rx52_lro_bytes: 0
>> rx52_ecn_mark: 0
>> rx52_removed_vlan_packets: 0
>> rx52_wqe_err: 0
>> rx52_mpwqe_filler_cqes: 0
>> rx52_mpwqe_filler_strides: 0
>> rx52_buff_alloc_err: 0
>> rx52_cqe_compress_blks: 0
>> rx52_cqe_compress_pkts: 0
>> rx52_page_reuse: 0
>> rx52_cache_reuse: 0
>> rx52_cache_full: 0
>> rx52_cache_empty: 2560
>> rx52_cache_busy: 0
>> rx52_cache_waive: 0
>> rx52_congst_umr: 0
>> rx52_arfs_err: 0
>> rx52_xdp_tx_xmit: 0
>> rx52_xdp_tx_full: 0
>> rx52_xdp_tx_err: 0
>> rx52_xdp_tx_cqes: 0
>> rx53_packets: 0
>> rx53_bytes: 0
>> rx53_csum_complete: 0
>> rx53_csum_unnecessary: 0
>> rx53_csum_unnecessary_inner: 0
>> rx53_csum_none: 0
>> rx53_xdp_drop: 0
>> rx53_xdp_redirect: 0
>> rx53_lro_packets: 0
>> rx53_lro_bytes: 0
>> rx53_ecn_mark: 0
>> rx53_removed_vlan_packets: 0
>> rx53_wqe_err: 0
>> rx53_mpwqe_filler_cqes: 0
>> rx53_mpwqe_filler_strides: 0
>> rx53_buff_alloc_err: 0
>> rx53_cqe_compress_blks: 0
>> rx53_cqe_compress_pkts: 0
>> rx53_page_reuse: 0
>> rx53_cache_reuse: 0
>> rx53_cache_full: 0
>> rx53_cache_empty: 2560
>> rx53_cache_busy: 0
>> rx53_cache_waive: 0
>> rx53_congst_umr: 0
>> rx53_arfs_err: 0
>> rx53_xdp_tx_xmit: 0
>> rx53_xdp_tx_full: 0
>> rx53_xdp_tx_err: 0
>> rx53_xdp_tx_cqes: 0
>> rx54_packets: 0
>> rx54_bytes: 0
>> rx54_csum_complete: 0
>> rx54_csum_unnecessary: 0
>> rx54_csum_unnecessary_inner: 0
>> rx54_csum_none: 0
>> rx54_xdp_drop: 0
>> rx54_xdp_redirect: 0
>> rx54_lro_packets: 0
>> rx54_lro_bytes: 0
>> rx54_ecn_mark: 0
>> rx54_removed_vlan_packets: 0
>> rx54_wqe_err: 0
>> rx54_mpwqe_filler_cqes: 0
>> rx54_mpwqe_filler_strides: 0
>> rx54_buff_alloc_err: 0
>> rx54_cqe_compress_blks: 0
>> rx54_cqe_compress_pkts: 0
>> rx54_page_reuse: 0
>> rx54_cache_reuse: 0
>> rx54_cache_full: 0
>> rx54_cache_empty: 2560
>> rx54_cache_busy: 0
>> rx54_cache_waive: 0
>> rx54_congst_umr: 0
>> rx54_arfs_err: 0
>> rx54_xdp_tx_xmit: 0
>> rx54_xdp_tx_full: 0
>> rx54_xdp_tx_err: 0
>> rx54_xdp_tx_cqes: 0
>> rx55_packets: 0
>> rx55_bytes: 0
>> rx55_csum_complete: 0
>> rx55_csum_unnecessary: 0
>> rx55_csum_unnecessary_inner: 0
>> rx55_csum_none: 0
>> rx55_xdp_drop: 0
>> rx55_xdp_redirect: 0
>> rx55_lro_packets: 0
>> rx55_lro_bytes: 0
>> rx55_ecn_mark: 0
>> rx55_removed_vlan_packets: 0
>> rx55_wqe_err: 0
>> rx55_mpwqe_filler_cqes: 0
>> rx55_mpwqe_filler_strides: 0
>> rx55_buff_alloc_err: 0
>> rx55_cqe_compress_blks: 0
>> rx55_cqe_compress_pkts: 0
>> rx55_page_reuse: 0
>> rx55_cache_reuse: 0
>> rx55_cache_full: 0
>> rx55_cache_empty: 2560
>> rx55_cache_busy: 0
>> rx55_cache_waive: 0
>> rx55_congst_umr: 0
>> rx55_arfs_err: 0
>> rx55_xdp_tx_xmit: 0
>> rx55_xdp_tx_full: 0
>> rx55_xdp_tx_err: 0
>> rx55_xdp_tx_cqes: 0
>> tx0_packets: 5868971166
>> tx0_bytes: 7384241881537
>> tx0_tso_packets: 1005089669
>> tx0_tso_bytes: 5138882499687
>> tx0_tso_inner_packets: 0
>> tx0_tso_inner_bytes: 0
>> tx0_csum_partial: 1405330470
>> tx0_csum_partial_inner: 0
>> tx0_added_vlan_packets: 3247061022
>> tx0_nop: 83925216
>> tx0_csum_none: 1841730552
>> tx0_stopped: 0
>> tx0_dropped: 0
>> tx0_xmit_more: 29664303
>> tx0_recover: 0
>> tx0_cqes: 3217398842
>> tx0_wake: 0
>> tx0_cqe_err: 0
>> tx1_packets: 5599378674
>> tx1_bytes: 7272236466962
>> tx1_tso_packets: 1024612268
>> tx1_tso_bytes: 5244192050917
>> tx1_tso_inner_packets: 0
>> tx1_tso_inner_bytes: 0
>> tx1_csum_partial: 1438007932
>> tx1_csum_partial_inner: 0
>> tx1_added_vlan_packets: 2919765857
>> tx1_nop: 79661231
>> tx1_csum_none: 1481757925
>> tx1_stopped: 0
>> tx1_dropped: 0
>> tx1_xmit_more: 29485355
>> tx1_recover: 0
>> tx1_cqes: 2890282176
>> tx1_wake: 0
>> tx1_cqe_err: 0
>> tx2_packets: 5413821094
>> tx2_bytes: 7033951631334
>> tx2_tso_packets: 1002868589
>> tx2_tso_bytes: 5089549008985
>> tx2_tso_inner_packets: 0
>> tx2_tso_inner_bytes: 0
>> tx2_csum_partial: 1404186175
>> tx2_csum_partial_inner: 0
>> tx2_added_vlan_packets: 2822670460
>> tx2_nop: 77115408
>> tx2_csum_none: 1418484285
>> tx2_stopped: 0
>> tx2_dropped: 0
>> tx2_xmit_more: 29321129
>> tx2_recover: 0
>> tx2_cqes: 2793351019
>> tx2_wake: 0
>> tx2_cqe_err: 0
>> tx3_packets: 5479609727
>> tx3_bytes: 7116904107659
>> tx3_tso_packets: 1002992639
>> tx3_tso_bytes: 5154225081979
>> tx3_tso_inner_packets: 0
>> tx3_tso_inner_bytes: 0
>> tx3_csum_partial: 1415739849
>> tx3_csum_partial_inner: 0
>> tx3_added_vlan_packets: 2842823811
>> tx3_nop: 78060813
>> tx3_csum_none: 1427083971
>> tx3_stopped: 0
>> tx3_dropped: 0
>> tx3_xmit_more: 28575040
>> tx3_recover: 0
>> tx3_cqes: 2814250785
>> tx3_wake: 0
>> tx3_cqe_err: 0
>> tx4_packets: 5508297397
>> tx4_bytes: 7127659369902
>> tx4_tso_packets: 1007356432
>> tx4_tso_bytes: 5145975736034
>> tx4_tso_inner_packets: 0
>> tx4_tso_inner_bytes: 0
>> tx4_csum_partial: 1411271000
>> tx4_csum_partial_inner: 0
>> tx4_added_vlan_packets: 2882086825
>> tx4_nop: 78433610
>> tx4_csum_none: 1470815825
>> tx4_stopped: 0
>> tx4_dropped: 0
>> tx4_xmit_more: 28632444
>> tx4_recover: 0
>> tx4_cqes: 2853456464
>> tx4_wake: 0
>> tx4_cqe_err: 0
>> tx5_packets: 5513864156
>> tx5_bytes: 7165864145517
>> tx5_tso_packets: 1014046485
>> tx5_tso_bytes: 5192635614477
>> tx5_tso_inner_packets: 0
>> tx5_tso_inner_bytes: 0
>> tx5_csum_partial: 1420810473
>> tx5_csum_partial_inner: 0
>> tx5_added_vlan_packets: 2861370556
>> tx5_nop: 78481355
>> tx5_csum_none: 1440560083
>> tx5_stopped: 0
>> tx5_dropped: 0
>> tx5_xmit_more: 28222467
>> tx5_recover: 0
>> tx5_cqes: 2833149758
>> tx5_wake: 0
>> tx5_cqe_err: 0
>> tx6_packets: 5560724761
>> tx6_bytes: 7210309972086
>> tx6_tso_packets: 994050514
>> tx6_tso_bytes: 5171393741595
>> tx6_tso_inner_packets: 0
>> tx6_tso_inner_bytes: 0
>> tx6_csum_partial: 1414303265
>> tx6_csum_partial_inner: 0
>> tx6_added_vlan_packets: 2905794177
>> tx6_nop: 79353318
>> tx6_csum_none: 1491490912
>> tx6_stopped: 0
>> tx6_dropped: 0
>> tx6_xmit_more: 31246664
>> tx6_recover: 0
>> tx6_cqes: 2874549217
>> tx6_wake: 0
>> tx6_cqe_err: 0
>> tx7_packets: 5557594170
>> tx7_bytes: 7223138778685
>> tx7_tso_packets: 1013475396
>> tx7_tso_bytes: 5241530065484
>> tx7_tso_inner_packets: 0
>> tx7_tso_inner_bytes: 0
>> tx7_csum_partial: 1438604314
>> tx7_csum_partial_inner: 0
>> tx7_added_vlan_packets: 2873917552
>> tx7_nop: 79057059
>> tx7_csum_none: 1435313239
>> tx7_stopped: 0
>> tx7_dropped: 0
>> tx7_xmit_more: 29258761
>> tx7_recover: 0
>> tx7_cqes: 2844660578
>> tx7_wake: 0
>> tx7_cqe_err: 0
>> tx8_packets: 5521254733
>> tx8_bytes: 7208043146297
>> tx8_tso_packets: 1014670801
>> tx8_tso_bytes: 5185842447246
>> tx8_tso_inner_packets: 0
>> tx8_tso_inner_bytes: 0
>> tx8_csum_partial: 1431631562
>> tx8_csum_partial_inner: 0
>> tx8_added_vlan_packets: 2872641129
>> tx8_nop: 78545776
>> tx8_csum_none: 1441009567
>> tx8_stopped: 0
>> tx8_dropped: 0
>> tx8_xmit_more: 29106291
>> tx8_recover: 0
>> tx8_cqes: 2843536748
>> tx8_wake: 0
>> tx8_cqe_err: 0
>> tx9_packets: 5528889957
>> tx9_bytes: 7191793816058
>> tx9_tso_packets: 1015955476
>> tx9_tso_bytes: 5207232047828
>> tx9_tso_inner_packets: 0
>> tx9_tso_inner_bytes: 0
>> tx9_csum_partial: 1421266796
>> tx9_csum_partial_inner: 0
>> tx9_added_vlan_packets: 2869523921
>> tx9_nop: 78586218
>> tx9_csum_none: 1448257125
>> tx9_stopped: 0
>> tx9_dropped: 0
>> tx9_xmit_more: 29483347
>> tx9_recover: 0
>> tx9_cqes: 2840042245
>> tx9_wake: 0
>> tx9_cqe_err: 0
>> tx10_packets: 5556351222
>> tx10_bytes: 7254798330757
>> tx10_tso_packets: 1028554460
>> tx10_tso_bytes: 5246179615774
>> tx10_tso_inner_packets: 0
>> tx10_tso_inner_bytes: 0
>> tx10_csum_partial: 1430459021
>> tx10_csum_partial_inner: 0
>> tx10_added_vlan_packets: 2881683382
>> tx10_nop: 79139584
>> tx10_csum_none: 1451224361
>> tx10_stopped: 0
>> tx10_dropped: 0
>> tx10_xmit_more: 29217190
>> tx10_recover: 0
>> tx10_cqes: 2852467898
>> tx10_wake: 0
>> tx10_cqe_err: 0
>> tx11_packets: 5455631854
>> tx11_bytes: 7061121713772
>> tx11_tso_packets: 992133383
>> tx11_tso_bytes: 5089419722682
>> tx11_tso_inner_packets: 0
>> tx11_tso_inner_bytes: 0
>> tx11_csum_partial: 1395542033
>> tx11_csum_partial_inner: 0
>> tx11_added_vlan_packets: 2852589093
>> tx11_nop: 77799857
>> tx11_csum_none: 1457047060
>> tx11_stopped: 0
>> tx11_dropped: 0
>> tx11_xmit_more: 29559927
>> tx11_recover: 0
>> tx11_cqes: 2823031110
>> tx11_wake: 0
>> tx11_cqe_err: 0
>> tx12_packets: 5488286808
>> tx12_bytes: 7137087569303
>> tx12_tso_packets: 1006435537
>> tx12_tso_bytes: 5163371416750
>> tx12_tso_inner_packets: 0
>> tx12_tso_inner_bytes: 0
>> tx12_csum_partial: 1414799411
>> tx12_csum_partial_inner: 0
>> tx12_added_vlan_packets: 2841679543
>> tx12_nop: 78387039
>> tx12_csum_none: 1426880132
>> tx12_stopped: 0
>> tx12_dropped: 0
>> tx12_xmit_more: 28607526
>> tx12_recover: 0
>> tx12_cqes: 2813073557
>> tx12_wake: 0
>> tx12_cqe_err: 0
>> tx13_packets: 5594132290
>> tx13_bytes: 7251106284829
>> tx13_tso_packets: 1035172061
>> tx13_tso_bytes: 5251200286298
>> tx13_tso_inner_packets: 0
>> tx13_tso_inner_bytes: 0
>> tx13_csum_partial: 1443665981
>> tx13_csum_partial_inner: 0
>> tx13_added_vlan_packets: 2916604799
>> tx13_nop: 79670465
>> tx13_csum_none: 1472938818
>> tx13_stopped: 0
>> tx13_dropped: 0
>> tx13_xmit_more: 27797067
>> tx13_recover: 0
>> tx13_cqes: 2888809352
>> tx13_wake: 0
>> tx13_cqe_err: 0
>> tx14_packets: 5548790952
>> tx14_bytes: 7194211868411
>> tx14_tso_packets: 1021015561
>> tx14_tso_bytes: 5231483708869
>> tx14_tso_inner_packets: 0
>> tx14_tso_inner_bytes: 0
>> tx14_csum_partial: 1427711576
>> tx14_csum_partial_inner: 0
>> tx14_added_vlan_packets: 2875288572
>> tx14_nop: 78900224
>> tx14_csum_none: 1447576996
>> tx14_stopped: 0
>> tx14_dropped: 0
>> tx14_xmit_more: 30003496
>> tx14_recover: 0
>> tx14_cqes: 2845286732
>> tx14_wake: 0
>> tx14_cqe_err: 0
>> tx15_packets: 5609310963
>> tx15_bytes: 7271380831798
>> tx15_tso_packets: 1027830118
>> tx15_tso_bytes: 5229697431506
>> tx15_tso_inner_packets: 0
>> tx15_tso_inner_bytes: 0
>> tx15_csum_partial: 1429209941
>> tx15_csum_partial_inner: 0
>> tx15_added_vlan_packets: 2940315402
>> tx15_nop: 79950883
>> tx15_csum_none: 1511105462
>> tx15_stopped: 0
>> tx15_dropped: 0
>> tx15_xmit_more: 28820740
>> tx15_recover: 0
>> tx15_cqes: 2911496633
>> tx15_wake: 0
>> tx15_cqe_err: 0
>> tx16_packets: 4465363036
>> tx16_bytes: 5769771803704
>> tx16_tso_packets: 817101913
>> tx16_tso_bytes: 4180172833814
>> tx16_tso_inner_packets: 0
>> tx16_tso_inner_bytes: 0
>> tx16_csum_partial: 1136731404
>> tx16_csum_partial_inner: 0
>> tx16_added_vlan_packets: 2332178232
>> tx16_nop: 63458573
>> tx16_csum_none: 1195446828
>> tx16_stopped: 0
>> tx16_dropped: 0
>> tx16_xmit_more: 23756254
>> tx16_recover: 0
>> tx16_cqes: 2308423025
>> tx16_wake: 0
>> tx16_cqe_err: 0
>> tx17_packets: 4380386348
>> tx17_bytes: 5708702994526
>> tx17_tso_packets: 813638023
>> tx17_tso_bytes: 4130806014947
>> tx17_tso_inner_packets: 0
>> tx17_tso_inner_bytes: 0
>> tx17_csum_partial: 1133007164
>> tx17_csum_partial_inner: 0
>> tx17_added_vlan_packets: 2277314787
>> tx17_nop: 62377372
>> tx17_csum_none: 1144307623
>> tx17_stopped: 0
>> tx17_dropped: 0
>> tx17_xmit_more: 23731361
>> tx17_recover: 0
>> tx17_cqes: 2253584638
>> tx17_wake: 0
>> tx17_cqe_err: 0
>> tx18_packets: 4450359743
>> tx18_bytes: 5758968674820
>> tx18_tso_packets: 815791601
>> tx18_tso_bytes: 4179942688909
>> tx18_tso_inner_packets: 0
>> tx18_tso_inner_bytes: 0
>> tx18_csum_partial: 1137649257
>> tx18_csum_partial_inner: 0
>> tx18_added_vlan_packets: 2314556550
>> tx18_nop: 63271085
>> tx18_csum_none: 1176907293
>> tx18_stopped: 0
>> tx18_dropped: 0
>> tx18_xmit_more: 23055770
>> tx18_recover: 0
>> tx18_cqes: 2291501928
>> tx18_wake: 0
>> tx18_cqe_err: 0
>> tx19_packets: 4596064378
>> tx19_bytes: 5916675706535
>> tx19_tso_packets: 825788649
>> tx19_tso_bytes: 4208046929921
>> tx19_tso_inner_packets: 0
>> tx19_tso_inner_bytes: 0
>> tx19_csum_partial: 1150666569
>> tx19_csum_partial_inner: 0
>> tx19_added_vlan_packets: 2450567026
>> tx19_nop: 65468504
>> tx19_csum_none: 1299900457
>> tx19_stopped: 0
>> tx19_dropped: 0
>> tx19_xmit_more: 23846250
>> tx19_recover: 0
>> tx19_cqes: 2426722127
>> tx19_wake: 0
>> tx19_cqe_err: 0
>> tx20_packets: 4424935388
>> tx20_bytes: 5757631205901
>> tx20_tso_packets: 804875006
>> tx20_tso_bytes: 4156262736109
>> tx20_tso_inner_packets: 0
>> tx20_tso_inner_bytes: 0
>> tx20_csum_partial: 1134144916
>> tx20_csum_partial_inner: 0
>> tx20_added_vlan_packets: 2294839665
>> tx20_nop: 63023986
>> tx20_csum_none: 1160694749
>> tx20_stopped: 0
>> tx20_dropped: 0
>> tx20_xmit_more: 23393201
>> tx20_recover: 0
>> tx20_cqes: 2271447623
>> tx20_wake: 0
>> tx20_cqe_err: 0
>> tx21_packets: 4595062285
>> tx21_bytes: 5958671993467
>> tx21_tso_packets: 821936215
>> tx21_tso_bytes: 4187977870684
>> tx21_tso_inner_packets: 0
>> tx21_tso_inner_bytes: 0
>> tx21_csum_partial: 1143339787
>> tx21_csum_partial_inner: 0
>> tx21_added_vlan_packets: 2457167412
>> tx21_nop: 65697763
>> tx21_csum_none: 1313827625
>> tx21_stopped: 0
>> tx21_dropped: 0
>> tx21_xmit_more: 23858345
>> tx21_recover: 0
>> tx21_cqes: 2433310348
>> tx21_wake: 0
>> tx21_cqe_err: 0
>> tx22_packets: 4664446513
>> tx22_bytes: 5931429292082
>> tx22_tso_packets: 814457881
>> tx22_tso_bytes: 4148607956533
>> tx22_tso_inner_packets: 0
>> tx22_tso_inner_bytes: 0
>> tx22_csum_partial: 1127284783
>> tx22_csum_partial_inner: 0
>> tx22_added_vlan_packets: 2548650146
>> tx22_nop: 66299909
>> tx22_csum_none: 1421365363
>> tx22_stopped: 0
>> tx22_dropped: 0
>> tx22_xmit_more: 23800911
>> tx22_recover: 0
>> tx22_cqes: 2524850415
>> tx22_wake: 0
>> tx22_cqe_err: 0
>> tx23_packets: 4416221747
>> tx23_bytes: 5721472587985
>> tx23_tso_packets: 823538520
>> tx23_tso_bytes: 4163520218617
>> tx23_tso_inner_packets: 0
>> tx23_tso_inner_bytes: 0
>> tx23_csum_partial: 1135996006
>> tx23_csum_partial_inner: 0
>> tx23_added_vlan_packets: 2292404120
>> tx23_nop: 62709432
>> tx23_csum_none: 1156408114
>> tx23_stopped: 0
>> tx23_dropped: 0
>> tx23_xmit_more: 22299889
>> tx23_recover: 0
>> tx23_cqes: 2270105487
>> tx23_wake: 0
>> tx23_cqe_err: 0
>> tx24_packets: 4420014824
>> tx24_bytes: 5740767318521
>> tx24_tso_packets: 820838072
>> tx24_tso_bytes: 4183722948422
>> tx24_tso_inner_packets: 0
>> tx24_tso_inner_bytes: 0
>> tx24_csum_partial: 1138070059
>> tx24_csum_partial_inner: 0
>> tx24_added_vlan_packets: 2289043946
>> tx24_nop: 62797341
>> tx24_csum_none: 1150973887
>> tx24_stopped: 0
>> tx24_dropped: 0
>> tx24_xmit_more: 22744690
>> tx24_recover: 0
>> tx24_cqes: 2266300568
>> tx24_wake: 0
>> tx24_cqe_err: 0
>> tx25_packets: 4413225545
>> tx25_bytes: 5716162617155
>> tx25_tso_packets: 808274341
>> tx25_tso_bytes: 4138408857714
>> tx25_tso_inner_packets: 0
>> tx25_tso_inner_bytes: 0
>> tx25_csum_partial: 1134587898
>> tx25_csum_partial_inner: 0
>> tx25_added_vlan_packets: 2297149310
>> tx25_nop: 62958238
>> tx25_csum_none: 1162561412
>> tx25_stopped: 0
>> tx25_dropped: 0
>> tx25_xmit_more: 24463552
>> tx25_recover: 0
>> tx25_cqes: 2272686971
>> tx25_wake: 0
>> tx25_cqe_err: 0
>> tx26_packets: 4524907591
>> tx26_bytes: 5865394280699
>> tx26_tso_packets: 807270022
>> tx26_tso_bytes: 4148754705317
>> tx26_tso_inner_packets: 0
>> tx26_tso_inner_bytes: 0
>> tx26_csum_partial: 1130306933
>> tx26_csum_partial_inner: 0
>> tx26_added_vlan_packets: 2402682460
>> tx26_nop: 64474322
>> tx26_csum_none: 1272375527
>> tx26_stopped: 1
>> tx26_dropped: 0
>> tx26_xmit_more: 23316186
>> tx26_recover: 0
>> tx26_cqes: 2379367502
>> tx26_wake: 1
>> tx26_cqe_err: 0
>> tx27_packets: 4376114969
>> tx27_bytes: 5683551238304
>> tx27_tso_packets: 809344829
>> tx27_tso_bytes: 4124331859270
>> tx27_tso_inner_packets: 0
>> tx27_tso_inner_bytes: 0
>> tx27_csum_partial: 1124954937
>> tx27_csum_partial_inner: 0
>> tx27_added_vlan_packets: 2267871300
>> tx27_nop: 62213214
>> tx27_csum_none: 1142916363
>> tx27_stopped: 0
>> tx27_dropped: 0
>> tx27_xmit_more: 23369974
>> tx27_recover: 0
>> tx27_cqes: 2244502686
>> tx27_wake: 0
>> tx27_cqe_err: 0
>> tx28_packets: 3
>> tx28_bytes: 266
>> tx28_tso_packets: 0
>> tx28_tso_bytes: 0
>> tx28_tso_inner_packets: 0
>> tx28_tso_inner_bytes: 0
>> tx28_csum_partial: 0
>> tx28_csum_partial_inner: 0
>> tx28_added_vlan_packets: 0
>> tx28_nop: 0
>> tx28_csum_none: 3
>> tx28_stopped: 0
>> tx28_dropped: 0
>> tx28_xmit_more: 0
>> tx28_recover: 0
>> tx28_cqes: 3
>> tx28_wake: 0
>> tx28_cqe_err: 0
>> tx29_packets: 0
>> tx29_bytes: 0
>> tx29_tso_packets: 0
>> tx29_tso_bytes: 0
>> tx29_tso_inner_packets: 0
>> tx29_tso_inner_bytes: 0
>> tx29_csum_partial: 0
>> tx29_csum_partial_inner: 0
>> tx29_added_vlan_packets: 0
>> tx29_nop: 0
>> tx29_csum_none: 0
>> tx29_stopped: 0
>> tx29_dropped: 0
>> tx29_xmit_more: 0
>> tx29_recover: 0
>> tx29_cqes: 0
>> tx29_wake: 0
>> tx29_cqe_err: 0
>> tx30_packets: 0
>> tx30_bytes: 0
>> tx30_tso_packets: 0
>> tx30_tso_bytes: 0
>> tx30_tso_inner_packets: 0
>> tx30_tso_inner_bytes: 0
>> tx30_csum_partial: 0
>> tx30_csum_partial_inner: 0
>> tx30_added_vlan_packets: 0
>> tx30_nop: 0
>> tx30_csum_none: 0
>> tx30_stopped: 0
>> tx30_dropped: 0
>> tx30_xmit_more: 0
>> tx30_recover: 0
>> tx30_cqes: 0
>> tx30_wake: 0
>> tx30_cqe_err: 0
>> tx31_packets: 0
>> tx31_bytes: 0
>> tx31_tso_packets: 0
>> tx31_tso_bytes: 0
>> tx31_tso_inner_packets: 0
>> tx31_tso_inner_bytes: 0
>> tx31_csum_partial: 0
>> tx31_csum_partial_inner: 0
>> tx31_added_vlan_packets: 0
>> tx31_nop: 0
>> tx31_csum_none: 0
>> tx31_stopped: 0
>> tx31_dropped: 0
>> tx31_xmit_more: 0
>> tx31_recover: 0
>> tx31_cqes: 0
>> tx31_wake: 0
>> tx31_cqe_err: 0
>> tx32_packets: 0
>> tx32_bytes: 0
>> tx32_tso_packets: 0
>> tx32_tso_bytes: 0
>> tx32_tso_inner_packets: 0
>> tx32_tso_inner_bytes: 0
>> tx32_csum_partial: 0
>> tx32_csum_partial_inner: 0
>> tx32_added_vlan_packets: 0
>> tx32_nop: 0
>> tx32_csum_none: 0
>> tx32_stopped: 0
>> tx32_dropped: 0
>> tx32_xmit_more: 0
>> tx32_recover: 0
>> tx32_cqes: 0
>> tx32_wake: 0
>> tx32_cqe_err: 0
>> tx33_packets: 0
>> tx33_bytes: 0
>> tx33_tso_packets: 0
>> tx33_tso_bytes: 0
>> tx33_tso_inner_packets: 0
>> tx33_tso_inner_bytes: 0
>> tx33_csum_partial: 0
>> tx33_csum_partial_inner: 0
>> tx33_added_vlan_packets: 0
>> tx33_nop: 0
>> tx33_csum_none: 0
>> tx33_stopped: 0
>> tx33_dropped: 0
>> tx33_xmit_more: 0
>> tx33_recover: 0
>> tx33_cqes: 0
>> tx33_wake: 0
>> tx33_cqe_err: 0
>> tx34_packets: 0
>> tx34_bytes: 0
>> tx34_tso_packets: 0
>> tx34_tso_bytes: 0
>> tx34_tso_inner_packets: 0
>> tx34_tso_inner_bytes: 0
>> tx34_csum_partial: 0
>> tx34_csum_partial_inner: 0
>> tx34_added_vlan_packets: 0
>> tx34_nop: 0
>> tx34_csum_none: 0
>> tx34_stopped: 0
>> tx34_dropped: 0
>> tx34_xmit_more: 0
>> tx34_recover: 0
>> tx34_cqes: 0
>> tx34_wake: 0
>> tx34_cqe_err: 0
>> tx35_packets: 0
>> tx35_bytes: 0
>> tx35_tso_packets: 0
>> tx35_tso_bytes: 0
>> tx35_tso_inner_packets: 0
>> tx35_tso_inner_bytes: 0
>> tx35_csum_partial: 0
>> tx35_csum_partial_inner: 0
>> tx35_added_vlan_packets: 0
>> tx35_nop: 0
>> tx35_csum_none: 0
>> tx35_stopped: 0
>> tx35_dropped: 0
>> tx35_xmit_more: 0
>> tx35_recover: 0
>> tx35_cqes: 0
>> tx35_wake: 0
>> tx35_cqe_err: 0
>> tx36_packets: 0
>> tx36_bytes: 0
>> tx36_tso_packets: 0
>> tx36_tso_bytes: 0
>> tx36_tso_inner_packets: 0
>> tx36_tso_inner_bytes: 0
>> tx36_csum_partial: 0
>> tx36_csum_partial_inner: 0
>> tx36_added_vlan_packets: 0
>> tx36_nop: 0
>> tx36_csum_none: 0
>> tx36_stopped: 0
>> tx36_dropped: 0
>> tx36_xmit_more: 0
>> tx36_recover: 0
>> tx36_cqes: 0
>> tx36_wake: 0
>> tx36_cqe_err: 0
>> tx37_packets: 0
>> tx37_bytes: 0
>> tx37_tso_packets: 0
>> tx37_tso_bytes: 0
>> tx37_tso_inner_packets: 0
>> tx37_tso_inner_bytes: 0
>> tx37_csum_partial: 0
>> tx37_csum_partial_inner: 0
>> tx37_added_vlan_packets: 0
>> tx37_nop: 0
>> tx37_csum_none: 0
>> tx37_stopped: 0
>> tx37_dropped: 0
>> tx37_xmit_more: 0
>> tx37_recover: 0
>> tx37_cqes: 0
>> tx37_wake: 0
>> tx37_cqe_err: 0
>> tx38_packets: 0
>> tx38_bytes: 0
>> tx38_tso_packets: 0
>> tx38_tso_bytes: 0
>> tx38_tso_inner_packets: 0
>> tx38_tso_inner_bytes: 0
>> tx38_csum_partial: 0
>> tx38_csum_partial_inner: 0
>> tx38_added_vlan_packets: 0
>> tx38_nop: 0
>> tx38_csum_none: 0
>> tx38_stopped: 0
>> tx38_dropped: 0
>> tx38_xmit_more: 0
>> tx38_recover: 0
>> tx38_cqes: 0
>> tx38_wake: 0
>> tx38_cqe_err: 0
>> tx39_packets: 0
>> tx39_bytes: 0
>> tx39_tso_packets: 0
>> tx39_tso_bytes: 0
>> tx39_tso_inner_packets: 0
>> tx39_tso_inner_bytes: 0
>> tx39_csum_partial: 0
>> tx39_csum_partial_inner: 0
>> tx39_added_vlan_packets: 0
>> tx39_nop: 0
>> tx39_csum_none: 0
>> tx39_stopped: 0
>> tx39_dropped: 0
>> tx39_xmit_more: 0
>> tx39_recover: 0
>> tx39_cqes: 0
>> tx39_wake: 0
>> tx39_cqe_err: 0
>> tx40_packets: 0
>> tx40_bytes: 0
>> tx40_tso_packets: 0
>> tx40_tso_bytes: 0
>> tx40_tso_inner_packets: 0
>> tx40_tso_inner_bytes: 0
>> tx40_csum_partial: 0
>> tx40_csum_partial_inner: 0
>> tx40_added_vlan_packets: 0
>> tx40_nop: 0
>> tx40_csum_none: 0
>> tx40_stopped: 0
>> tx40_dropped: 0
>> tx40_xmit_more: 0
>> tx40_recover: 0
>> tx40_cqes: 0
>> tx40_wake: 0
>> tx40_cqe_err: 0
>> tx41_packets: 0
>> tx41_bytes: 0
>> tx41_tso_packets: 0
>> tx41_tso_bytes: 0
>> tx41_tso_inner_packets: 0
>> tx41_tso_inner_bytes: 0
>> tx41_csum_partial: 0
>> tx41_csum_partial_inner: 0
>> tx41_added_vlan_packets: 0
>> tx41_nop: 0
>> tx41_csum_none: 0
>> tx41_stopped: 0
>> tx41_dropped: 0
>> tx41_xmit_more: 0
>> tx41_recover: 0
>> tx41_cqes: 0
>> tx41_wake: 0
>> tx41_cqe_err: 0
>> tx42_packets: 0
>> tx42_bytes: 0
>> tx42_tso_packets: 0
>> tx42_tso_bytes: 0
>> tx42_tso_inner_packets: 0
>> tx42_tso_inner_bytes: 0
>> tx42_csum_partial: 0
>> tx42_csum_partial_inner: 0
>> tx42_added_vlan_packets: 0
>> tx42_nop: 0
>> tx42_csum_none: 0
>> tx42_stopped: 0
>> tx42_dropped: 0
>> tx42_xmit_more: 0
>> tx42_recover: 0
>> tx42_cqes: 0
>> tx42_wake: 0
>> tx42_cqe_err: 0
>> tx43_packets: 0
>> tx43_bytes: 0
>> tx43_tso_packets: 0
>> tx43_tso_bytes: 0
>> tx43_tso_inner_packets: 0
>> tx43_tso_inner_bytes: 0
>> tx43_csum_partial: 0
>> tx43_csum_partial_inner: 0
>> tx43_added_vlan_packets: 0
>> tx43_nop: 0
>> tx43_csum_none: 0
>> tx43_stopped: 0
>> tx43_dropped: 0
>> tx43_xmit_more: 0
>> tx43_recover: 0
>> tx43_cqes: 0
>> tx43_wake: 0
>> tx43_cqe_err: 0
>> tx44_packets: 0
>> tx44_bytes: 0
>> tx44_tso_packets: 0
>> tx44_tso_bytes: 0
>> tx44_tso_inner_packets: 0
>> tx44_tso_inner_bytes: 0
>> tx44_csum_partial: 0
>> tx44_csum_partial_inner: 0
>> tx44_added_vlan_packets: 0
>> tx44_nop: 0
>> tx44_csum_none: 0
>> tx44_stopped: 0
>> tx44_dropped: 0
>> tx44_xmit_more: 0
>> tx44_recover: 0
>> tx44_cqes: 0
>> tx44_wake: 0
>> tx44_cqe_err: 0
>> tx45_packets: 0
>> tx45_bytes: 0
>> tx45_tso_packets: 0
>> tx45_tso_bytes: 0
>> tx45_tso_inner_packets: 0
>> tx45_tso_inner_bytes: 0
>> tx45_csum_partial: 0
>> tx45_csum_partial_inner: 0
>> tx45_added_vlan_packets: 0
>> tx45_nop: 0
>> tx45_csum_none: 0
>> tx45_stopped: 0
>> tx45_dropped: 0
>> tx45_xmit_more: 0
>> tx45_recover: 0
>> tx45_cqes: 0
>> tx45_wake: 0
>> tx45_cqe_err: 0
>> tx46_packets: 0
>> tx46_bytes: 0
>> tx46_tso_packets: 0
>> tx46_tso_bytes: 0
>> tx46_tso_inner_packets: 0
>> tx46_tso_inner_bytes: 0
>> tx46_csum_partial: 0
>> tx46_csum_partial_inner: 0
>> tx46_added_vlan_packets: 0
>> tx46_nop: 0
>> tx46_csum_none: 0
>> tx46_stopped: 0
>> tx46_dropped: 0
>> tx46_xmit_more: 0
>> tx46_recover: 0
>> tx46_cqes: 0
>> tx46_wake: 0
>> tx46_cqe_err: 0
>> tx47_packets: 0
>> tx47_bytes: 0
>> tx47_tso_packets: 0
>> tx47_tso_bytes: 0
>> tx47_tso_inner_packets: 0
>> tx47_tso_inner_bytes: 0
>> tx47_csum_partial: 0
>> tx47_csum_partial_inner: 0
>> tx47_added_vlan_packets: 0
>> tx47_nop: 0
>> tx47_csum_none: 0
>> tx47_stopped: 0
>> tx47_dropped: 0
>> tx47_xmit_more: 0
>> tx47_recover: 0
>> tx47_cqes: 0
>> tx47_wake: 0
>> tx47_cqe_err: 0
>> tx48_packets: 0
>> tx48_bytes: 0
>> tx48_tso_packets: 0
>> tx48_tso_bytes: 0
>> tx48_tso_inner_packets: 0
>> tx48_tso_inner_bytes: 0
>> tx48_csum_partial: 0
>> tx48_csum_partial_inner: 0
>> tx48_added_vlan_packets: 0
>> tx48_nop: 0
>> tx48_csum_none: 0
>> tx48_stopped: 0
>> tx48_dropped: 0
>> tx48_xmit_more: 0
>> tx48_recover: 0
>> tx48_cqes: 0
>> tx48_wake: 0
>> tx48_cqe_err: 0
>> tx49_packets: 0
>> tx49_bytes: 0
>> tx49_tso_packets: 0
>> tx49_tso_bytes: 0
>> tx49_tso_inner_packets: 0
>> tx49_tso_inner_bytes: 0
>> tx49_csum_partial: 0
>> tx49_csum_partial_inner: 0
>> tx49_added_vlan_packets: 0
>> tx49_nop: 0
>> tx49_csum_none: 0
>> tx49_stopped: 0
>> tx49_dropped: 0
>> tx49_xmit_more: 0
>> tx49_recover: 0
>> tx49_cqes: 0
>> tx49_wake: 0
>> tx49_cqe_err: 0
>> tx50_packets: 0
>> tx50_bytes: 0
>> tx50_tso_packets: 0
>> tx50_tso_bytes: 0
>> tx50_tso_inner_packets: 0
>> tx50_tso_inner_bytes: 0
>> tx50_csum_partial: 0
>> tx50_csum_partial_inner: 0
>> tx50_added_vlan_packets: 0
>> tx50_nop: 0
>> tx50_csum_none: 0
>> tx50_stopped: 0
>> tx50_dropped: 0
>> tx50_xmit_more: 0
>> tx50_recover: 0
>> tx50_cqes: 0
>> tx50_wake: 0
>> tx50_cqe_err: 0
>> tx51_packets: 0
>> tx51_bytes: 0
>> tx51_tso_packets: 0
>> tx51_tso_bytes: 0
>> tx51_tso_inner_packets: 0
>> tx51_tso_inner_bytes: 0
>> tx51_csum_partial: 0
>> tx51_csum_partial_inner: 0
>> tx51_added_vlan_packets: 0
>> tx51_nop: 0
>> tx51_csum_none: 0
>> tx51_stopped: 0
>> tx51_dropped: 0
>> tx51_xmit_more: 0
>> tx51_recover: 0
>> tx51_cqes: 0
>> tx51_wake: 0
>> tx51_cqe_err: 0
>> tx52_packets: 0
>> tx52_bytes: 0
>> tx52_tso_packets: 0
>> tx52_tso_bytes: 0
>> tx52_tso_inner_packets: 0
>> tx52_tso_inner_bytes: 0
>> tx52_csum_partial: 0
>> tx52_csum_partial_inner: 0
>> tx52_added_vlan_packets: 0
>> tx52_nop: 0
>> tx52_csum_none: 0
>> tx52_stopped: 0
>> tx52_dropped: 0
>> tx52_xmit_more: 0
>> tx52_recover: 0
>> tx52_cqes: 0
>> tx52_wake: 0
>> tx52_cqe_err: 0
>> tx53_packets: 0
>> tx53_bytes: 0
>> tx53_tso_packets: 0
>> tx53_tso_bytes: 0
>> tx53_tso_inner_packets: 0
>> tx53_tso_inner_bytes: 0
>> tx53_csum_partial: 0
>> tx53_csum_partial_inner: 0
>> tx53_added_vlan_packets: 0
>> tx53_nop: 0
>> tx53_csum_none: 0
>> tx53_stopped: 0
>> tx53_dropped: 0
>> tx53_xmit_more: 0
>> tx53_recover: 0
>> tx53_cqes: 0
>> tx53_wake: 0
>> tx53_cqe_err: 0
>> tx54_packets: 0
>> tx54_bytes: 0
>> tx54_tso_packets: 0
>> tx54_tso_bytes: 0
>> tx54_tso_inner_packets: 0
>> tx54_tso_inner_bytes: 0
>> tx54_csum_partial: 0
>> tx54_csum_partial_inner: 0
>> tx54_added_vlan_packets: 0
>> tx54_nop: 0
>> tx54_csum_none: 0
>> tx54_stopped: 0
>> tx54_dropped: 0
>> tx54_xmit_more: 0
>> tx54_recover: 0
>> tx54_cqes: 0
>> tx54_wake: 0
>> tx54_cqe_err: 0
>> tx55_packets: 0
>> tx55_bytes: 0
>> tx55_tso_packets: 0
>> tx55_tso_bytes: 0
>> tx55_tso_inner_packets: 0
>> tx55_tso_inner_bytes: 0
>> tx55_csum_partial: 0
>> tx55_csum_partial_inner: 0
>> tx55_added_vlan_packets: 0
>> tx55_nop: 0
>> tx55_csum_none: 0
>> tx55_stopped: 0
>> tx55_dropped: 0
>> tx55_xmit_more: 0
>> tx55_recover: 0
>> tx55_cqes: 0
>> tx55_wake: 0
>> tx55_cqe_err: 0
>> tx0_xdp_xmit: 0
>> tx0_xdp_full: 0
>> tx0_xdp_err: 0
>> tx0_xdp_cqes: 0
>> tx1_xdp_xmit: 0
>> tx1_xdp_full: 0
>> tx1_xdp_err: 0
>> tx1_xdp_cqes: 0
>> tx2_xdp_xmit: 0
>> tx2_xdp_full: 0
>> tx2_xdp_err: 0
>> tx2_xdp_cqes: 0
>> tx3_xdp_xmit: 0
>> tx3_xdp_full: 0
>> tx3_xdp_err: 0
>> tx3_xdp_cqes: 0
>> tx4_xdp_xmit: 0
>> tx4_xdp_full: 0
>> tx4_xdp_err: 0
>> tx4_xdp_cqes: 0
>> tx5_xdp_xmit: 0
>> tx5_xdp_full: 0
>> tx5_xdp_err: 0
>> tx5_xdp_cqes: 0
>> tx6_xdp_xmit: 0
>> tx6_xdp_full: 0
>> tx6_xdp_err: 0
>> tx6_xdp_cqes: 0
>> tx7_xdp_xmit: 0
>> tx7_xdp_full: 0
>> tx7_xdp_err: 0
>> tx7_xdp_cqes: 0
>> tx8_xdp_xmit: 0
>> tx8_xdp_full: 0
>> tx8_xdp_err: 0
>> tx8_xdp_cqes: 0
>> tx9_xdp_xmit: 0
>> tx9_xdp_full: 0
>> tx9_xdp_err: 0
>> tx9_xdp_cqes: 0
>> tx10_xdp_xmit: 0
>> tx10_xdp_full: 0
>> tx10_xdp_err: 0
>> tx10_xdp_cqes: 0
>> tx11_xdp_xmit: 0
>> tx11_xdp_full: 0
>> tx11_xdp_err: 0
>> tx11_xdp_cqes: 0
>> tx12_xdp_xmit: 0
>> tx12_xdp_full: 0
>> tx12_xdp_err: 0
>> tx12_xdp_cqes: 0
>> tx13_xdp_xmit: 0
>> tx13_xdp_full: 0
>> tx13_xdp_err: 0
>> tx13_xdp_cqes: 0
>> tx14_xdp_xmit: 0
>> tx14_xdp_full: 0
>> tx14_xdp_err: 0
>> tx14_xdp_cqes: 0
>> tx15_xdp_xmit: 0
>> tx15_xdp_full: 0
>> tx15_xdp_err: 0
>> tx15_xdp_cqes: 0
>> tx16_xdp_xmit: 0
>> tx16_xdp_full: 0
>> tx16_xdp_err: 0
>> tx16_xdp_cqes: 0
>> tx17_xdp_xmit: 0
>> tx17_xdp_full: 0
>> tx17_xdp_err: 0
>> tx17_xdp_cqes: 0
>> tx18_xdp_xmit: 0
>> tx18_xdp_full: 0
>> tx18_xdp_err: 0
>> tx18_xdp_cqes: 0
>> tx19_xdp_xmit: 0
>> tx19_xdp_full: 0
>> tx19_xdp_err: 0
>> tx19_xdp_cqes: 0
>> tx20_xdp_xmit: 0
>> tx20_xdp_full: 0
>> tx20_xdp_err: 0
>> tx20_xdp_cqes: 0
>> tx21_xdp_xmit: 0
>> tx21_xdp_full: 0
>> tx21_xdp_err: 0
>> tx21_xdp_cqes: 0
>> tx22_xdp_xmit: 0
>> tx22_xdp_full: 0
>> tx22_xdp_err: 0
>> tx22_xdp_cqes: 0
>> tx23_xdp_xmit: 0
>> tx23_xdp_full: 0
>> tx23_xdp_err: 0
>> tx23_xdp_cqes: 0
>> tx24_xdp_xmit: 0
>> tx24_xdp_full: 0
>> tx24_xdp_err: 0
>> tx24_xdp_cqes: 0
>> tx25_xdp_xmit: 0
>> tx25_xdp_full: 0
>> tx25_xdp_err: 0
>> tx25_xdp_cqes: 0
>> tx26_xdp_xmit: 0
>> tx26_xdp_full: 0
>> tx26_xdp_err: 0
>> tx26_xdp_cqes: 0
>> tx27_xdp_xmit: 0
>> tx27_xdp_full: 0
>> tx27_xdp_err: 0
>> tx27_xdp_cqes: 0
>> tx28_xdp_xmit: 0
>> tx28_xdp_full: 0
>> tx28_xdp_err: 0
>> tx28_xdp_cqes: 0
>> tx29_xdp_xmit: 0
>> tx29_xdp_full: 0
>> tx29_xdp_err: 0
>> tx29_xdp_cqes: 0
>> tx30_xdp_xmit: 0
>> tx30_xdp_full: 0
>> tx30_xdp_err: 0
>> tx30_xdp_cqes: 0
>> tx31_xdp_xmit: 0
>> tx31_xdp_full: 0
>> tx31_xdp_err: 0
>> tx31_xdp_cqes: 0
>> tx32_xdp_xmit: 0
>> tx32_xdp_full: 0
>> tx32_xdp_err: 0
>> tx32_xdp_cqes: 0
>> tx33_xdp_xmit: 0
>> tx33_xdp_full: 0
>> tx33_xdp_err: 0
>> tx33_xdp_cqes: 0
>> tx34_xdp_xmit: 0
>> tx34_xdp_full: 0
>> tx34_xdp_err: 0
>> tx34_xdp_cqes: 0
>> tx35_xdp_xmit: 0
>> tx35_xdp_full: 0
>> tx35_xdp_err: 0
>> tx35_xdp_cqes: 0
>> tx36_xdp_xmit: 0
>> tx36_xdp_full: 0
>> tx36_xdp_err: 0
>> tx36_xdp_cqes: 0
>> tx37_xdp_xmit: 0
>> tx37_xdp_full: 0
>> tx37_xdp_err: 0
>> tx37_xdp_cqes: 0
>> tx38_xdp_xmit: 0
>> tx38_xdp_full: 0
>> tx38_xdp_err: 0
>> tx38_xdp_cqes: 0
>> tx39_xdp_xmit: 0
>> tx39_xdp_full: 0
>> tx39_xdp_err: 0
>> tx39_xdp_cqes: 0
>> tx40_xdp_xmit: 0
>> tx40_xdp_full: 0
>> tx40_xdp_err: 0
>> tx40_xdp_cqes: 0
>> tx41_xdp_xmit: 0
>> tx41_xdp_full: 0
>> tx41_xdp_err: 0
>> tx41_xdp_cqes: 0
>> tx42_xdp_xmit: 0
>> tx42_xdp_full: 0
>> tx42_xdp_err: 0
>> tx42_xdp_cqes: 0
>> tx43_xdp_xmit: 0
>> tx43_xdp_full: 0
>> tx43_xdp_err: 0
>> tx43_xdp_cqes: 0
>> tx44_xdp_xmit: 0
>> tx44_xdp_full: 0
>> tx44_xdp_err: 0
>> tx44_xdp_cqes: 0
>> tx45_xdp_xmit: 0
>> tx45_xdp_full: 0
>> tx45_xdp_err: 0
>> tx45_xdp_cqes: 0
>> tx46_xdp_xmit: 0
>> tx46_xdp_full: 0
>> tx46_xdp_err: 0
>> tx46_xdp_cqes: 0
>> tx47_xdp_xmit: 0
>> tx47_xdp_full: 0
>> tx47_xdp_err: 0
>> tx47_xdp_cqes: 0
>> tx48_xdp_xmit: 0
>> tx48_xdp_full: 0
>> tx48_xdp_err: 0
>> tx48_xdp_cqes: 0
>> tx49_xdp_xmit: 0
>> tx49_xdp_full: 0
>> tx49_xdp_err: 0
>> tx49_xdp_cqes: 0
>> tx50_xdp_xmit: 0
>> tx50_xdp_full: 0
>> tx50_xdp_err: 0
>> tx50_xdp_cqes: 0
>> tx51_xdp_xmit: 0
>> tx51_xdp_full: 0
>> tx51_xdp_err: 0
>> tx51_xdp_cqes: 0
>> tx52_xdp_xmit: 0
>> tx52_xdp_full: 0
>> tx52_xdp_err: 0
>> tx52_xdp_cqes: 0
>> tx53_xdp_xmit: 0
>> tx53_xdp_full: 0
>> tx53_xdp_err: 0
>> tx53_xdp_cqes: 0
>> tx54_xdp_xmit: 0
>> tx54_xdp_full: 0
>> tx54_xdp_err: 0
>> tx54_xdp_cqes: 0
>> tx55_xdp_xmit: 0
>> tx55_xdp_full: 0
>> tx55_xdp_err: 0
>> tx55_xdp_cqes: 0
>>
>> ethtool -S enp175s0f0
>> NIC statistics:
>> rx_packets: 141574897253
>> rx_bytes: 184445040406258
>> tx_packets: 172569543894
>> tx_bytes: 99486882076365
>> tx_tso_packets: 9367664195
>> tx_tso_bytes: 56435233992948
>> tx_tso_inner_packets: 0
>> tx_tso_inner_bytes: 0
>> tx_added_vlan_packets: 141297671626
>> tx_nop: 2102916272
>> rx_lro_packets: 0
>> rx_lro_bytes: 0
>> rx_ecn_mark: 0
>> rx_removed_vlan_packets: 141574897252
>> rx_csum_unnecessary: 0
>> rx_csum_none: 23135854
>> rx_csum_complete: 141551761398
>> rx_csum_unnecessary_inner: 0
>> rx_xdp_drop: 0
>> rx_xdp_redirect: 0
>> rx_xdp_tx_xmit: 0
>> rx_xdp_tx_full: 0
>> rx_xdp_tx_err: 0
>> rx_xdp_tx_cqe: 0
>> tx_csum_none: 127934791664
> It is a good idea to look into this, tx is not requesting hw tx
> csumming for a lot of packets, maybe you are wasting a lot of cpu on
> calculating csum, or maybe this is just the rx csum complete..
>
>> tx_csum_partial: 13362879974
>> tx_csum_partial_inner: 0
>> tx_queue_stopped: 232561
> TX queues are stalling, could be an indentation for the pcie
> bottelneck.
>
>> tx_queue_dropped: 0
>> tx_xmit_more: 1266021946
>> tx_recover: 0
>> tx_cqes: 140031716469
>> tx_queue_wake: 232561
>> tx_udp_seg_rem: 0
>> tx_cqe_err: 0
>> tx_xdp_xmit: 0
>> tx_xdp_full: 0
>> tx_xdp_err: 0
>> tx_xdp_cqes: 0
>> rx_wqe_err: 0
>> rx_mpwqe_filler_cqes: 0
>> rx_mpwqe_filler_strides: 0
>> rx_buff_alloc_err: 0
>> rx_cqe_compress_blks: 0
>> rx_cqe_compress_pkts: 0
>> rx_page_reuse: 0
>> rx_cache_reuse: 16625975793
>> rx_cache_full: 54161465914
>> rx_cache_empty: 258048
>> rx_cache_busy: 54161472735
>> rx_cache_waive: 0
>> rx_congst_umr: 0
>> rx_arfs_err: 0
>> ch_events: 40572621887
>> ch_poll: 40885650979
>> ch_arm: 40429276692
>> ch_aff_change: 0
>> ch_eq_rearm: 0
>> rx_out_of_buffer: 2791690
>> rx_if_down_packets: 74
>> rx_vport_unicast_packets: 141843476308
>> rx_vport_unicast_bytes: 185421265403318
>> tx_vport_unicast_packets: 172569484005
>> tx_vport_unicast_bytes: 100019940094298
>> rx_vport_multicast_packets: 85122935
>> rx_vport_multicast_bytes: 5761316431
>> tx_vport_multicast_packets: 6452
>> tx_vport_multicast_bytes: 643540
>> rx_vport_broadcast_packets: 22423624
>> rx_vport_broadcast_bytes: 1390127090
>> tx_vport_broadcast_packets: 22024
>> tx_vport_broadcast_bytes: 1321440
>> rx_vport_rdma_unicast_packets: 0
>> rx_vport_rdma_unicast_bytes: 0
>> tx_vport_rdma_unicast_packets: 0
>> tx_vport_rdma_unicast_bytes: 0
>> rx_vport_rdma_multicast_packets: 0
>> rx_vport_rdma_multicast_bytes: 0
>> tx_vport_rdma_multicast_packets: 0
>> tx_vport_rdma_multicast_bytes: 0
>> tx_packets_phy: 172569501577
>> rx_packets_phy: 142871314588
>> rx_crc_errors_phy: 0
>> tx_bytes_phy: 100710212814151
>> rx_bytes_phy: 187209224289564
>> tx_multicast_phy: 6452
>> tx_broadcast_phy: 22024
>> rx_multicast_phy: 85122933
>> rx_broadcast_phy: 22423623
>> rx_in_range_len_errors_phy: 2
>> rx_out_of_range_len_phy: 0
>> rx_oversize_pkts_phy: 0
>> rx_symbol_err_phy: 0
>> tx_mac_control_phy: 0
>> rx_mac_control_phy: 0
>> rx_unsupported_op_phy: 0
>> rx_pause_ctrl_phy: 0
>> tx_pause_ctrl_phy: 0
>> rx_discards_phy: 920161423
> Ok, this port seem to be suffering more, RX is congested, maybe due to
> the pcie bottleneck.
Yes this side is receiving more traffic - second port is +10G more tx
>> tx_discards_phy: 0
>> tx_errors_phy: 0
>> rx_undersize_pkts_phy: 0
>> rx_fragments_phy: 0
>> rx_jabbers_phy: 0
>> rx_64_bytes_phy: 412006326
>> rx_65_to_127_bytes_phy: 11934371453
>> rx_128_to_255_bytes_phy: 3415281165
>> rx_256_to_511_bytes_phy: 2072955511
>> rx_512_to_1023_bytes_phy: 2415393005
>> rx_1024_to_1518_bytes_phy: 72182391608
>> rx_1519_to_2047_bytes_phy: 50438902587
>> rx_2048_to_4095_bytes_phy: 0
>> rx_4096_to_8191_bytes_phy: 0
>> rx_8192_to_10239_bytes_phy: 0
>> link_down_events_phy: 0
>> rx_pcs_symbol_err_phy: 0
>> rx_corrected_bits_phy: 0
>> rx_pci_signal_integrity: 0
>> tx_pci_signal_integrity: 48
>> rx_prio0_bytes: 186709842592642
>> rx_prio0_packets: 141481966007
>> tx_prio0_bytes: 100710171118138
>> tx_prio0_packets: 172569437949
>> rx_prio1_bytes: 492288152326
>> rx_prio1_packets: 385996045
>> tx_prio1_bytes: 0
>> tx_prio1_packets: 0
>> rx_prio2_bytes: 22119952
>> rx_prio2_packets: 70788
>> tx_prio2_bytes: 0
>> tx_prio2_packets: 0
>> rx_prio3_bytes: 546141102
>> rx_prio3_packets: 681608
>> tx_prio3_bytes: 0
>> tx_prio3_packets: 0
>> rx_prio4_bytes: 14665067
>> rx_prio4_packets: 29486
>> tx_prio4_bytes: 0
>> tx_prio4_packets: 0
>> rx_prio5_bytes: 158862504
>> rx_prio5_packets: 965307
>> tx_prio5_bytes: 0
>> tx_prio5_packets: 0
>> rx_prio6_bytes: 669337783
>> rx_prio6_packets: 1475775
>> tx_prio6_bytes: 0
>> tx_prio6_packets: 0
>> rx_prio7_bytes: 5623481349
>> rx_prio7_packets: 79926412
>> tx_prio7_bytes: 0
>> tx_prio7_packets: 0
>> module_unplug: 0
>> module_bus_stuck: 0
>> module_high_temp: 0
>> module_bad_shorted: 0
>> ch0_events: 1446162630
>> ch0_poll: 1463312972
>> ch0_arm: 1440728278
>> ch0_aff_change: 0
>> ch0_eq_rearm: 0
>> ch1_events: 1384301405
>> ch1_poll: 1399210915
>> ch1_arm: 1378636486
>> ch1_aff_change: 0
>> ch1_eq_rearm: 0
>> ch2_events: 1382788887
>> ch2_poll: 1397231470
>> ch2_arm: 1377058116
>> ch2_aff_change: 0
>> ch2_eq_rearm: 0
>> ch3_events: 1461956995
>> ch3_poll: 1475553146
>> ch3_arm: 1456571625
>> ch3_aff_change: 0
>> ch3_eq_rearm: 0
>> ch4_events: 1497359109
>> ch4_poll: 1511021037
>> ch4_arm: 1491733757
>> ch4_aff_change: 0
>> ch4_eq_rearm: 0
>> ch5_events: 1387736262
>> ch5_poll: 1400964615
>> ch5_arm: 1382382834
>> ch5_aff_change: 0
>> ch5_eq_rearm: 0
>> ch6_events: 1376772405
>> ch6_poll: 1390851449
>> ch6_arm: 1371551764
>> ch6_aff_change: 0
>> ch6_eq_rearm: 0
>> ch7_events: 1431271514
>> ch7_poll: 1445049729
>> ch7_arm: 1425753718
>> ch7_aff_change: 0
>> ch7_eq_rearm: 0
>> ch8_events: 1426976374
>> ch8_poll: 1439938692
>> ch8_arm: 1421392984
>> ch8_aff_change: 0
>> ch8_eq_rearm: 0
>> ch9_events: 1456160031
>> ch9_poll: 1468922870
>> ch9_arm: 1450930446
>> ch9_aff_change: 0
>> ch9_eq_rearm: 0
>> ch10_events: 1443640165
>> ch10_poll: 1456812203
>> ch10_arm: 1438425101
>> ch10_aff_change: 0
>> ch10_eq_rearm: 0
>> ch11_events: 1381104776
>> ch11_poll: 1393811057
>> ch11_arm: 1376059326
>> ch11_aff_change: 0
>> ch11_eq_rearm: 0
>> ch12_events: 1365223276
>> ch12_poll: 1378406059
>> ch12_arm: 1359950494
>> ch12_aff_change: 0
>> ch12_eq_rearm: 0
>> ch13_events: 1421622259
>> ch13_poll: 1434670996
>> ch13_arm: 1416241801
>> ch13_aff_change: 0
>> ch13_eq_rearm: 0
>> ch14_events: 1379084590
>> ch14_poll: 1392425015
>> ch14_arm: 1373675179
>> ch14_aff_change: 0
>> ch14_eq_rearm: 0
>> ch15_events: 1531217338
>> ch15_poll: 1543353833
>> ch15_arm: 1526350453
>> ch15_aff_change: 0
>> ch15_eq_rearm: 0
>> ch16_events: 1460469776
>> ch16_poll: 1467995928
>> ch16_arm: 1456010194
>> ch16_aff_change: 0
>> ch16_eq_rearm: 0
>> ch17_events: 1494067670
>> ch17_poll: 1500856680
>> ch17_arm: 1489232674
>> ch17_aff_change: 0
>> ch17_eq_rearm: 0
>> ch18_events: 1530126866
>> ch18_poll: 1537293620
>> ch18_arm: 1525476123
>> ch18_aff_change: 0
>> ch18_eq_rearm: 0
>> ch19_events: 1499526149
>> ch19_poll: 1506789309
>> ch19_arm: 1495161602
>> ch19_aff_change: 0
>> ch19_eq_rearm: 0
>> ch20_events: 1451479763
>> ch20_poll: 1459767921
>> ch20_arm: 1446360801
>> ch20_aff_change: 0
>> ch20_eq_rearm: 0
>> ch21_events: 1521413613
>> ch21_poll: 1529345146
>> ch21_arm: 1517229314
>> ch21_aff_change: 0
>> ch21_eq_rearm: 0
>> ch22_events: 1471950045
>> ch22_poll: 1479746764
>> ch22_arm: 1467681629
>> ch22_aff_change: 0
>> ch22_eq_rearm: 0
>> ch23_events: 1502968393
>> ch23_poll: 1510419909
>> ch23_arm: 1498168438
>> ch23_aff_change: 0
>> ch23_eq_rearm: 0
>> ch24_events: 1473451639
>> ch24_poll: 1482606899
>> ch24_arm: 1468212489
>> ch24_aff_change: 0
>> ch24_eq_rearm: 0
>> ch25_events: 1440399182
>> ch25_poll: 1448897475
>> ch25_arm: 1435044786
>> ch25_aff_change: 0
>> ch25_eq_rearm: 0
>> ch26_events: 1436831565
>> ch26_poll: 1445485731
>> ch26_arm: 1431827527
>> ch26_aff_change: 0
>> ch26_eq_rearm: 0
>> ch27_events: 1516560621
>> ch27_poll: 1524911010
>> ch27_arm: 1511430164
>> ch27_aff_change: 0
>> ch27_eq_rearm: 0
>> ch28_events: 4
>> ch28_poll: 4
>> ch28_arm: 4
>> ch28_aff_change: 0
>> ch28_eq_rearm: 0
>> ch29_events: 6
>> ch29_poll: 6
>> ch29_arm: 6
>> ch29_aff_change: 0
>> ch29_eq_rearm: 0
>> ch30_events: 4
>> ch30_poll: 4
>> ch30_arm: 4
>> ch30_aff_change: 0
>> ch30_eq_rearm: 0
>> ch31_events: 4
>> ch31_poll: 4
>> ch31_arm: 4
>> ch31_aff_change: 0
>> ch31_eq_rearm: 0
>> ch32_events: 4
>> ch32_poll: 4
>> ch32_arm: 4
>> ch32_aff_change: 0
>> ch32_eq_rearm: 0
>> ch33_events: 4
>> ch33_poll: 4
>> ch33_arm: 4
>> ch33_aff_change: 0
>> ch33_eq_rearm: 0
>> ch34_events: 4
>> ch34_poll: 4
>> ch34_arm: 4
>> ch34_aff_change: 0
>> ch34_eq_rearm: 0
>> ch35_events: 4
>> ch35_poll: 4
>> ch35_arm: 4
>> ch35_aff_change: 0
>> ch35_eq_rearm: 0
>> ch36_events: 4
>> ch36_poll: 4
>> ch36_arm: 4
>> ch36_aff_change: 0
>> ch36_eq_rearm: 0
>> ch37_events: 4
>> ch37_poll: 4
>> ch37_arm: 4
>> ch37_aff_change: 0
>> ch37_eq_rearm: 0
>> ch38_events: 4
>> ch38_poll: 4
>> ch38_arm: 4
>> ch38_aff_change: 0
>> ch38_eq_rearm: 0
>> ch39_events: 4
>> ch39_poll: 4
>> ch39_arm: 4
>> ch39_aff_change: 0
>> ch39_eq_rearm: 0
>> ch40_events: 4
>> ch40_poll: 4
>> ch40_arm: 4
>> ch40_aff_change: 0
>> ch40_eq_rearm: 0
>> ch41_events: 4
>> ch41_poll: 4
>> ch41_arm: 4
>> ch41_aff_change: 0
>> ch41_eq_rearm: 0
>> ch42_events: 4
>> ch42_poll: 4
>> ch42_arm: 4
>> ch42_aff_change: 0
>> ch42_eq_rearm: 0
>> ch43_events: 4
>> ch43_poll: 4
>> ch43_arm: 4
>> ch43_aff_change: 0
>> ch43_eq_rearm: 0
>> ch44_events: 4
>> ch44_poll: 4
>> ch44_arm: 4
>> ch44_aff_change: 0
>> ch44_eq_rearm: 0
>> ch45_events: 4
>> ch45_poll: 4
>> ch45_arm: 4
>> ch45_aff_change: 0
>> ch45_eq_rearm: 0
>> ch46_events: 4
>> ch46_poll: 4
>> ch46_arm: 4
>> ch46_aff_change: 0
>> ch46_eq_rearm: 0
>> ch47_events: 4
>> ch47_poll: 4
>> ch47_arm: 4
>> ch47_aff_change: 0
>> ch47_eq_rearm: 0
>> ch48_events: 4
>> ch48_poll: 4
>> ch48_arm: 4
>> ch48_aff_change: 0
>> ch48_eq_rearm: 0
>> ch49_events: 4
>> ch49_poll: 4
>> ch49_arm: 4
>> ch49_aff_change: 0
>> ch49_eq_rearm: 0
>> ch50_events: 4
>> ch50_poll: 4
>> ch50_arm: 4
>> ch50_aff_change: 0
>> ch50_eq_rearm: 0
>> ch51_events: 4
>> ch51_poll: 4
>> ch51_arm: 4
>> ch51_aff_change: 0
>> ch51_eq_rearm: 0
>> ch52_events: 4
>> ch52_poll: 4
>> ch52_arm: 4
>> ch52_aff_change: 0
>> ch52_eq_rearm: 0
>> ch53_events: 4
>> ch53_poll: 4
>> ch53_arm: 4
>> ch53_aff_change: 0
>> ch53_eq_rearm: 0
>> ch54_events: 4
>> ch54_poll: 4
>> ch54_arm: 4
>> ch54_aff_change: 0
>> ch54_eq_rearm: 0
>> ch55_events: 4
>> ch55_poll: 4
>> ch55_arm: 4
>> ch55_aff_change: 0
>> ch55_eq_rearm: 0
>> rx0_packets: 5861448653
>> rx0_bytes: 7389128595728
>> rx0_csum_complete: 5838312798
>> rx0_csum_unnecessary: 0
>> rx0_csum_unnecessary_inner: 0
>> rx0_csum_none: 23135855
>> rx0_xdp_drop: 0
>> rx0_xdp_redirect: 0
>> rx0_lro_packets: 0
>> rx0_lro_bytes: 0
>> rx0_ecn_mark: 0
>> rx0_removed_vlan_packets: 5861448653
>> rx0_wqe_err: 0
>> rx0_mpwqe_filler_cqes: 0
>> rx0_mpwqe_filler_strides: 0
>> rx0_buff_alloc_err: 0
>> rx0_cqe_compress_blks: 0
>> rx0_cqe_compress_pkts: 0
>> rx0_page_reuse: 0
>> rx0_cache_reuse: 2559
>> rx0_cache_full: 2930721512
>> rx0_cache_empty: 6656
>> rx0_cache_busy: 2930721765
>> rx0_cache_waive: 0
>> rx0_congst_umr: 0
>> rx0_arfs_err: 0
>> rx0_xdp_tx_xmit: 0
>> rx0_xdp_tx_full: 0
>> rx0_xdp_tx_err: 0
>> rx0_xdp_tx_cqes: 0
>> rx1_packets: 5550585106
>> rx1_bytes: 7255635262803
>> rx1_csum_complete: 5550585106
>> rx1_csum_unnecessary: 0
>> rx1_csum_unnecessary_inner: 0
>> rx1_csum_none: 0
>> rx1_xdp_drop: 0
>> rx1_xdp_redirect: 0
>> rx1_lro_packets: 0
>> rx1_lro_bytes: 0
>> rx1_ecn_mark: 0
>> rx1_removed_vlan_packets: 5550585106
>> rx1_wqe_err: 0
>> rx1_mpwqe_filler_cqes: 0
>> rx1_mpwqe_filler_strides: 0
>> rx1_buff_alloc_err: 0
>> rx1_cqe_compress_blks: 0
>> rx1_cqe_compress_pkts: 0
>> rx1_page_reuse: 0
>> rx1_cache_reuse: 2918845
>> rx1_cache_full: 2772373453
>> rx1_cache_empty: 6656
>> rx1_cache_busy: 2772373707
>> rx1_cache_waive: 0
>> rx1_congst_umr: 0
>> rx1_arfs_err: 0
>> rx1_xdp_tx_xmit: 0
>> rx1_xdp_tx_full: 0
>> rx1_xdp_tx_err: 0
>> rx1_xdp_tx_cqes: 0
>> rx2_packets: 5383874739
>> rx2_bytes: 7031545423967
>> rx2_csum_complete: 5383874739
>> rx2_csum_unnecessary: 0
>> rx2_csum_unnecessary_inner: 0
>> rx2_csum_none: 0
>> rx2_xdp_drop: 0
>> rx2_xdp_redirect: 0
>> rx2_lro_packets: 0
>> rx2_lro_bytes: 0
>> rx2_ecn_mark: 0
>> rx2_removed_vlan_packets: 5383874739
>> rx2_wqe_err: 0
>> rx2_mpwqe_filler_cqes: 0
>> rx2_mpwqe_filler_strides: 0
>> rx2_buff_alloc_err: 0
>> rx2_cqe_compress_blks: 0
>> rx2_cqe_compress_pkts: 0
>> rx2_page_reuse: 0
>> rx2_cache_reuse: 2173370
>> rx2_cache_full: 2689763744
>> rx2_cache_empty: 6656
>> rx2_cache_busy: 2689763998
>> rx2_cache_waive: 0
>> rx2_congst_umr: 0
>> rx2_arfs_err: 0
>> rx2_xdp_tx_xmit: 0
>> rx2_xdp_tx_full: 0
>> rx2_xdp_tx_err: 0
>> rx2_xdp_tx_cqes: 0
>> rx3_packets: 5456494012
>> rx3_bytes: 7120241119485
>> rx3_csum_complete: 5456494012
>> rx3_csum_unnecessary: 0
>> rx3_csum_unnecessary_inner: 0
>> rx3_csum_none: 0
>> rx3_xdp_drop: 0
>> rx3_xdp_redirect: 0
>> rx3_lro_packets: 0
>> rx3_lro_bytes: 0
>> rx3_ecn_mark: 0
>> rx3_removed_vlan_packets: 5456494012
>> rx3_wqe_err: 0
>> rx3_mpwqe_filler_cqes: 0
>> rx3_mpwqe_filler_strides: 0
>> rx3_buff_alloc_err: 0
>> rx3_cqe_compress_blks: 0
>> rx3_cqe_compress_pkts: 0
>> rx3_page_reuse: 0
>> rx3_cache_reuse: 2120123
>> rx3_cache_full: 2726126628
>> rx3_cache_empty: 6656
>> rx3_cache_busy: 2726126881
>> rx3_cache_waive: 0
>> rx3_congst_umr: 0
>> rx3_arfs_err: 0
>> rx3_xdp_tx_xmit: 0
>> rx3_xdp_tx_full: 0
>> rx3_xdp_tx_err: 0
>> rx3_xdp_tx_cqes: 0
>> rx4_packets: 5475216251
>> rx4_bytes: 7123129170196
>> rx4_csum_complete: 5475216251
>> rx4_csum_unnecessary: 0
>> rx4_csum_unnecessary_inner: 0
>> rx4_csum_none: 0
>> rx4_xdp_drop: 0
>> rx4_xdp_redirect: 0
>> rx4_lro_packets: 0
>> rx4_lro_bytes: 0
>> rx4_ecn_mark: 0
>> rx4_removed_vlan_packets: 5475216251
>> rx4_wqe_err: 0
>> rx4_mpwqe_filler_cqes: 0
>> rx4_mpwqe_filler_strides: 0
>> rx4_buff_alloc_err: 0
>> rx4_cqe_compress_blks: 0
>> rx4_cqe_compress_pkts: 0
>> rx4_page_reuse: 0
>> rx4_cache_reuse: 2668296355
>> rx4_cache_full: 69311549
>> rx4_cache_empty: 6656
>> rx4_cache_busy: 69311769
>> rx4_cache_waive: 0
>> rx4_congst_umr: 0
>> rx4_arfs_err: 0
>> rx4_xdp_tx_xmit: 0
>> rx4_xdp_tx_full: 0
>> rx4_xdp_tx_err: 0
>> rx4_xdp_tx_cqes: 0
>> rx5_packets: 5474372232
>> rx5_bytes: 7159146801926
>> rx5_csum_complete: 5474372232
>> rx5_csum_unnecessary: 0
>> rx5_csum_unnecessary_inner: 0
>> rx5_csum_none: 0
>> rx5_xdp_drop: 0
>> rx5_xdp_redirect: 0
>> rx5_lro_packets: 0
>> rx5_lro_bytes: 0
>> rx5_ecn_mark: 0
>> rx5_removed_vlan_packets: 5474372232
>> rx5_wqe_err: 0
>> rx5_mpwqe_filler_cqes: 0
>> rx5_mpwqe_filler_strides: 0
>> rx5_buff_alloc_err: 0
>> rx5_cqe_compress_blks: 0
>> rx5_cqe_compress_pkts: 0
>> rx5_page_reuse: 0
>> rx5_cache_reuse: 626187
>> rx5_cache_full: 2736559674
>> rx5_cache_empty: 6656
>> rx5_cache_busy: 2736559929
>> rx5_cache_waive: 0
>> rx5_congst_umr: 0
>> rx5_arfs_err: 0
>> rx5_xdp_tx_xmit: 0
>> rx5_xdp_tx_full: 0
>> rx5_xdp_tx_err: 0
>> rx5_xdp_tx_cqes: 0
>> rx6_packets: 5533622456
>> rx6_bytes: 7207308809081
>> rx6_csum_complete: 5533622456
>> rx6_csum_unnecessary: 0
>> rx6_csum_unnecessary_inner: 0
>> rx6_csum_none: 0
>> rx6_xdp_drop: 0
>> rx6_xdp_redirect: 0
>> rx6_lro_packets: 0
>> rx6_lro_bytes: 0
>> rx6_ecn_mark: 0
>> rx6_removed_vlan_packets: 5533622456
>> rx6_wqe_err: 0
>> rx6_mpwqe_filler_cqes: 0
>> rx6_mpwqe_filler_strides: 0
>> rx6_buff_alloc_err: 0
>> rx6_cqe_compress_blks: 0
>> rx6_cqe_compress_pkts: 0
>> rx6_page_reuse: 0
>> rx6_cache_reuse: 2325217
>> rx6_cache_full: 2764485756
>> rx6_cache_empty: 6656
>> rx6_cache_busy: 2764486011
>> rx6_cache_waive: 0
>> rx6_congst_umr: 0
>> rx6_arfs_err: 0
>> rx6_xdp_tx_xmit: 0
>> rx6_xdp_tx_full: 0
>> rx6_xdp_tx_err: 0
>> rx6_xdp_tx_cqes: 0
>> rx7_packets: 5533901822
>> rx7_bytes: 7227441240536
>> rx7_csum_complete: 5533901822
>> rx7_csum_unnecessary: 0
>> rx7_csum_unnecessary_inner: 0
>> rx7_csum_none: 0
>> rx7_xdp_drop: 0
>> rx7_xdp_redirect: 0
>> rx7_lro_packets: 0
>> rx7_lro_bytes: 0
>> rx7_ecn_mark: 0
>> rx7_removed_vlan_packets: 5533901822
>> rx7_wqe_err: 0
>> rx7_mpwqe_filler_cqes: 0
>> rx7_mpwqe_filler_strides: 0
>> rx7_buff_alloc_err: 0
>> rx7_cqe_compress_blks: 0
>> rx7_cqe_compress_pkts: 0
>> rx7_page_reuse: 0
>> rx7_cache_reuse: 2372505
>> rx7_cache_full: 2764578151
>> rx7_cache_empty: 6656
>> rx7_cache_busy: 2764578403
>> rx7_cache_waive: 0
>> rx7_congst_umr: 0
>> rx7_arfs_err: 0
>> rx7_xdp_tx_xmit: 0
>> rx7_xdp_tx_full: 0
>> rx7_xdp_tx_err: 0
>> rx7_xdp_tx_cqes: 0
>> rx8_packets: 5485670137
>> rx8_bytes: 7203339989013
>> rx8_csum_complete: 5485670137
>> rx8_csum_unnecessary: 0
>> rx8_csum_unnecessary_inner: 0
>> rx8_csum_none: 0
>> rx8_xdp_drop: 0
>> rx8_xdp_redirect: 0
>> rx8_lro_packets: 0
>> rx8_lro_bytes: 0
>> rx8_ecn_mark: 0
>> rx8_removed_vlan_packets: 5485670137
>> rx8_wqe_err: 0
>> rx8_mpwqe_filler_cqes: 0
>> rx8_mpwqe_filler_strides: 0
>> rx8_buff_alloc_err: 0
>> rx8_cqe_compress_blks: 0
>> rx8_cqe_compress_pkts: 0
>> rx8_page_reuse: 0
>> rx8_cache_reuse: 7522232
>> rx8_cache_full: 2735312581
>> rx8_cache_empty: 6656
>> rx8_cache_busy: 2735312836
>> rx8_cache_waive: 0
>> rx8_congst_umr: 0
>> rx8_arfs_err: 0
>> rx8_xdp_tx_xmit: 0
>> rx8_xdp_tx_full: 0
>> rx8_xdp_tx_err: 0
>> rx8_xdp_tx_cqes: 0
>> rx9_packets: 5482212354
>> rx9_bytes: 7169663341718
>> rx9_csum_complete: 5482212354
>> rx9_csum_unnecessary: 0
>> rx9_csum_unnecessary_inner: 0
>> rx9_csum_none: 0
>> rx9_xdp_drop: 0
>> rx9_xdp_redirect: 0
>> rx9_lro_packets: 0
>> rx9_lro_bytes: 0
>> rx9_ecn_mark: 0
>> rx9_removed_vlan_packets: 5482212354
>> rx9_wqe_err: 0
>> rx9_mpwqe_filler_cqes: 0
>> rx9_mpwqe_filler_strides: 0
>> rx9_buff_alloc_err: 0
>> rx9_cqe_compress_blks: 0
>> rx9_cqe_compress_pkts: 0
>> rx9_page_reuse: 0
>> rx9_cache_reuse: 37279961
>> rx9_cache_full: 2703825961
>> rx9_cache_empty: 6656
>> rx9_cache_busy: 2703826215
>> rx9_cache_waive: 0
>> rx9_congst_umr: 0
>> rx9_arfs_err: 0
>> rx9_xdp_tx_xmit: 0
>> rx9_xdp_tx_full: 0
>> rx9_xdp_tx_err: 0
>> rx9_xdp_tx_cqes: 0
>> rx10_packets: 5524679952
>> rx10_bytes: 7248301275181
>> rx10_csum_complete: 5524679952
>> rx10_csum_unnecessary: 0
>> rx10_csum_unnecessary_inner: 0
>> rx10_csum_none: 0
>> rx10_xdp_drop: 0
>> rx10_xdp_redirect: 0
>> rx10_lro_packets: 0
>> rx10_lro_bytes: 0
>> rx10_ecn_mark: 0
>> rx10_removed_vlan_packets: 5524679952
>> rx10_wqe_err: 0
>> rx10_mpwqe_filler_cqes: 0
>> rx10_mpwqe_filler_strides: 0
>> rx10_buff_alloc_err: 0
>> rx10_cqe_compress_blks: 0
>> rx10_cqe_compress_pkts: 0
>> rx10_page_reuse: 0
>> rx10_cache_reuse: 2049666
>> rx10_cache_full: 2760290055
>> rx10_cache_empty: 6656
>> rx10_cache_busy: 2760290310
>> rx10_cache_waive: 0
>> rx10_congst_umr: 0
>> rx10_arfs_err: 0
>> rx10_xdp_tx_xmit: 0
>> rx10_xdp_tx_full: 0
>> rx10_xdp_tx_err: 0
>> rx10_xdp_tx_cqes: 0
>> rx11_packets: 5394633545
>> rx11_bytes: 7033509636092
>> rx11_csum_complete: 5394633545
>> rx11_csum_unnecessary: 0
>> rx11_csum_unnecessary_inner: 0
>> rx11_csum_none: 0
>> rx11_xdp_drop: 0
>> rx11_xdp_redirect: 0
>> rx11_lro_packets: 0
>> rx11_lro_bytes: 0
>> rx11_ecn_mark: 0
>> rx11_removed_vlan_packets: 5394633545
>> rx11_wqe_err: 0
>> rx11_mpwqe_filler_cqes: 0
>> rx11_mpwqe_filler_strides: 0
>> rx11_buff_alloc_err: 0
>> rx11_cqe_compress_blks: 0
>> rx11_cqe_compress_pkts: 0
>> rx11_page_reuse: 0
>> rx11_cache_reuse: 2617466268
>> rx11_cache_full: 79850284
>> rx11_cache_empty: 6656
>> rx11_cache_busy: 79850504
>> rx11_cache_waive: 0
>> rx11_congst_umr: 0
>> rx11_arfs_err: 0
>> rx11_xdp_tx_xmit: 0
>> rx11_xdp_tx_full: 0
>> rx11_xdp_tx_err: 0
>> rx11_xdp_tx_cqes: 0
>> rx12_packets: 5458907385
>> rx12_bytes: 7134867867515
>> rx12_csum_complete: 5458907385
>> rx12_csum_unnecessary: 0
>> rx12_csum_unnecessary_inner: 0
>> rx12_csum_none: 0
>> rx12_xdp_drop: 0
>> rx12_xdp_redirect: 0
>> rx12_lro_packets: 0
>> rx12_lro_bytes: 0
>> rx12_ecn_mark: 0
>> rx12_removed_vlan_packets: 5458907385
>> rx12_wqe_err: 0
>> rx12_mpwqe_filler_cqes: 0
>> rx12_mpwqe_filler_strides: 0
>> rx12_buff_alloc_err: 0
>> rx12_cqe_compress_blks: 0
>> rx12_cqe_compress_pkts: 0
>> rx12_page_reuse: 0
>> rx12_cache_reuse: 2650214169
>> rx12_cache_full: 79239303
>> rx12_cache_empty: 6656
>> rx12_cache_busy: 79239523
>> rx12_cache_waive: 0
>> rx12_congst_umr: 0
>> rx12_arfs_err: 0
>> rx12_xdp_tx_xmit: 0
>> rx12_xdp_tx_full: 0
>> rx12_xdp_tx_err: 0
>> rx12_xdp_tx_cqes: 0
>> rx13_packets: 5549932912
>> rx13_bytes: 7232548705586
>> rx13_csum_complete: 5549932912
>> rx13_csum_unnecessary: 0
>> rx13_csum_unnecessary_inner: 0
>> rx13_csum_none: 0
>> rx13_xdp_drop: 0
>> rx13_xdp_redirect: 0
>> rx13_lro_packets: 0
>> rx13_lro_bytes: 0
>> rx13_ecn_mark: 0
>> rx13_removed_vlan_packets: 5549932912
>> rx13_wqe_err: 0
>> rx13_mpwqe_filler_cqes: 0
>> rx13_mpwqe_filler_strides: 0
>> rx13_buff_alloc_err: 0
>> rx13_cqe_compress_blks: 0
>> rx13_cqe_compress_pkts: 0
>> rx13_page_reuse: 0
>> rx13_cache_reuse: 2417696
>> rx13_cache_full: 2772548505
>> rx13_cache_empty: 6656
>> rx13_cache_busy: 2772548760
>> rx13_cache_waive: 0
>> rx13_congst_umr: 0
>> rx13_arfs_err: 0
>> rx13_xdp_tx_xmit: 0
>> rx13_xdp_tx_full: 0
>> rx13_xdp_tx_err: 0
>> rx13_xdp_tx_cqes: 0
>> rx14_packets: 5517712329
>> rx14_bytes: 7192111965227
>> rx14_csum_complete: 5517712329
>> rx14_csum_unnecessary: 0
>> rx14_csum_unnecessary_inner: 0
>> rx14_csum_none: 0
>> rx14_xdp_drop: 0
>> rx14_xdp_redirect: 0
>> rx14_lro_packets: 0
>> rx14_lro_bytes: 0
>> rx14_ecn_mark: 0
>> rx14_removed_vlan_packets: 5517712329
>> rx14_wqe_err: 0
>> rx14_mpwqe_filler_cqes: 0
>> rx14_mpwqe_filler_strides: 0
>> rx14_buff_alloc_err: 0
>> rx14_cqe_compress_blks: 0
>> rx14_cqe_compress_pkts: 0
>> rx14_page_reuse: 0
>> rx14_cache_reuse: 1830206
>> rx14_cache_full: 2757025703
>> rx14_cache_empty: 6656
>> rx14_cache_busy: 2757025958
>> rx14_cache_waive: 0
>> rx14_congst_umr: 0
>> rx14_arfs_err: 0
>> rx14_xdp_tx_xmit: 0
>> rx14_xdp_tx_full: 0
>> rx14_xdp_tx_err: 0
>> rx14_xdp_tx_cqes: 0
>> rx15_packets: 5578343373
>> rx15_bytes: 7268484501219
>> rx15_csum_complete: 5578343373
>> rx15_csum_unnecessary: 0
>> rx15_csum_unnecessary_inner: 0
>> rx15_csum_none: 0
>> rx15_xdp_drop: 0
>> rx15_xdp_redirect: 0
>> rx15_lro_packets: 0
>> rx15_lro_bytes: 0
>> rx15_ecn_mark: 0
>> rx15_removed_vlan_packets: 5578343373
>> rx15_wqe_err: 0
>> rx15_mpwqe_filler_cqes: 0
>> rx15_mpwqe_filler_strides: 0
>> rx15_buff_alloc_err: 0
>> rx15_cqe_compress_blks: 0
>> rx15_cqe_compress_pkts: 0
>> rx15_page_reuse: 0
>> rx15_cache_reuse: 2317165
>> rx15_cache_full: 2786854266
>> rx15_cache_empty: 6656
>> rx15_cache_busy: 2786854519
>> rx15_cache_waive: 0
>> rx15_congst_umr: 0
>> rx15_arfs_err: 0
>> rx15_xdp_tx_xmit: 0
>> rx15_xdp_tx_full: 0
>> rx15_xdp_tx_err: 0
>> rx15_xdp_tx_cqes: 0
>> rx16_packets: 4435773951
>> rx16_bytes: 5766665272007
>> rx16_csum_complete: 4435773951
>> rx16_csum_unnecessary: 0
>> rx16_csum_unnecessary_inner: 0
>> rx16_csum_none: 0
>> rx16_xdp_drop: 0
>> rx16_xdp_redirect: 0
>> rx16_lro_packets: 0
>> rx16_lro_bytes: 0
>> rx16_ecn_mark: 0
>> rx16_removed_vlan_packets: 4435773951
>> rx16_wqe_err: 0
>> rx16_mpwqe_filler_cqes: 0
>> rx16_mpwqe_filler_strides: 0
>> rx16_buff_alloc_err: 0
>> rx16_cqe_compress_blks: 0
>> rx16_cqe_compress_pkts: 0
>> rx16_page_reuse: 0
>> rx16_cache_reuse: 2033793
>> rx16_cache_full: 2215852927
>> rx16_cache_empty: 6656
>> rx16_cache_busy: 2215853179
>> rx16_cache_waive: 0
>> rx16_congst_umr: 0
>> rx16_arfs_err: 0
>> rx16_xdp_tx_xmit: 0
>> rx16_xdp_tx_full: 0
>> rx16_xdp_tx_err: 0
>> rx16_xdp_tx_cqes: 0
>> rx17_packets: 4344087587
>> rx17_bytes: 5695006496323
>> rx17_csum_complete: 4344087587
>> rx17_csum_unnecessary: 0
>> rx17_csum_unnecessary_inner: 0
>> rx17_csum_none: 0
>> rx17_xdp_drop: 0
>> rx17_xdp_redirect: 0
>> rx17_lro_packets: 0
>> rx17_lro_bytes: 0
>> rx17_ecn_mark: 0
>> rx17_removed_vlan_packets: 4344087587
>> rx17_wqe_err: 0
>> rx17_mpwqe_filler_cqes: 0
>> rx17_mpwqe_filler_strides: 0
>> rx17_buff_alloc_err: 0
>> rx17_cqe_compress_blks: 0
>> rx17_cqe_compress_pkts: 0
>> rx17_page_reuse: 0
>> rx17_cache_reuse: 2652127
>> rx17_cache_full: 2169391411
>> rx17_cache_empty: 6656
>> rx17_cache_busy: 2169391665
>> rx17_cache_waive: 0
>> rx17_congst_umr: 0
>> rx17_arfs_err: 0
>> rx17_xdp_tx_xmit: 0
>> rx17_xdp_tx_full: 0
>> rx17_xdp_tx_err: 0
>> rx17_xdp_tx_cqes: 0
>> rx18_packets: 4407422804
>> rx18_bytes: 5741134634177
>> rx18_csum_complete: 4407422804
>> rx18_csum_unnecessary: 0
>> rx18_csum_unnecessary_inner: 0
>> rx18_csum_none: 0
>> rx18_xdp_drop: 0
>> rx18_xdp_redirect: 0
>> rx18_lro_packets: 0
>> rx18_lro_bytes: 0
>> rx18_ecn_mark: 0
>> rx18_removed_vlan_packets: 4407422804
>> rx18_wqe_err: 0
>> rx18_mpwqe_filler_cqes: 0
>> rx18_mpwqe_filler_strides: 0
>> rx18_buff_alloc_err: 0
>> rx18_cqe_compress_blks: 0
>> rx18_cqe_compress_pkts: 0
>> rx18_page_reuse: 0
>> rx18_cache_reuse: 2156080239
>> rx18_cache_full: 47630941
>> rx18_cache_empty: 6656
>> rx18_cache_busy: 47631161
>> rx18_cache_waive: 0
>> rx18_congst_umr: 0
>> rx18_arfs_err: 0
>> rx18_xdp_tx_xmit: 0
>> rx18_xdp_tx_full: 0
>> rx18_xdp_tx_err: 0
>> rx18_xdp_tx_cqes: 0
>> rx19_packets: 4545554180
>> rx19_bytes: 5905277503466
>> rx19_csum_complete: 4545554180
>> rx19_csum_unnecessary: 0
>> rx19_csum_unnecessary_inner: 0
>> rx19_csum_none: 0
>> rx19_xdp_drop: 0
>> rx19_xdp_redirect: 0
>> rx19_lro_packets: 0
>> rx19_lro_bytes: 0
>> rx19_ecn_mark: 0
>> rx19_removed_vlan_packets: 4545554180
>> rx19_wqe_err: 0
>> rx19_mpwqe_filler_cqes: 0
>> rx19_mpwqe_filler_strides: 0
>> rx19_buff_alloc_err: 0
>> rx19_cqe_compress_blks: 0
>> rx19_cqe_compress_pkts: 0
>> rx19_page_reuse: 0
>> rx19_cache_reuse: 11112455
>> rx19_cache_full: 2261664379
>> rx19_cache_empty: 6656
>> rx19_cache_busy: 2261664601
>> rx19_cache_waive: 0
>> rx19_congst_umr: 0
>> rx19_arfs_err: 0
>> rx19_xdp_tx_xmit: 0
>> rx19_xdp_tx_full: 0
>> rx19_xdp_tx_err: 0
>> rx19_xdp_tx_cqes: 0
>> rx20_packets: 4397428553
>> rx20_bytes: 5757329184301
>> rx20_csum_complete: 4397428553
>> rx20_csum_unnecessary: 0
>> rx20_csum_unnecessary_inner: 0
>> rx20_csum_none: 0
>> rx20_xdp_drop: 0
>> rx20_xdp_redirect: 0
>> rx20_lro_packets: 0
>> rx20_lro_bytes: 0
>> rx20_ecn_mark: 0
>> rx20_removed_vlan_packets: 4397428553
>> rx20_wqe_err: 0
>> rx20_mpwqe_filler_cqes: 0
>> rx20_mpwqe_filler_strides: 0
>> rx20_buff_alloc_err: 0
>> rx20_cqe_compress_blks: 0
>> rx20_cqe_compress_pkts: 0
>> rx20_page_reuse: 0
>> rx20_cache_reuse: 2168116995
>> rx20_cache_full: 30597061
>> rx20_cache_empty: 6656
>> rx20_cache_busy: 30597281
>> rx20_cache_waive: 0
>> rx20_congst_umr: 0
>> rx20_arfs_err: 0
>> rx20_xdp_tx_xmit: 0
>> rx20_xdp_tx_full: 0
>> rx20_xdp_tx_err: 0
>> rx20_xdp_tx_cqes: 0
>> rx21_packets: 4552564821
>> rx21_bytes: 5944840329249
>> rx21_csum_complete: 4552564821
>> rx21_csum_unnecessary: 0
>> rx21_csum_unnecessary_inner: 0
>> rx21_csum_none: 0
>> rx21_xdp_drop: 0
>> rx21_xdp_redirect: 0
>> rx21_lro_packets: 0
>> rx21_lro_bytes: 0
>> rx21_ecn_mark: 0
>> rx21_removed_vlan_packets: 4552564821
>> rx21_wqe_err: 0
>> rx21_mpwqe_filler_cqes: 0
>> rx21_mpwqe_filler_strides: 0
>> rx21_buff_alloc_err: 0
>> rx21_cqe_compress_blks: 0
>> rx21_cqe_compress_pkts: 0
>> rx21_page_reuse: 0
>> rx21_cache_reuse: 2295681
>> rx21_cache_full: 2273986474
>> rx21_cache_empty: 6656
>> rx21_cache_busy: 2273986727
>> rx21_cache_waive: 0
>> rx21_congst_umr: 0
>> rx21_arfs_err: 0
>> rx21_xdp_tx_xmit: 0
>> rx21_xdp_tx_full: 0
>> rx21_xdp_tx_err: 0
>> rx21_xdp_tx_cqes: 0
>> rx22_packets: 4629499740
>> rx22_bytes: 5924206566499
>> rx22_csum_complete: 4629499740
>> rx22_csum_unnecessary: 0
>> rx22_csum_unnecessary_inner: 0
>> rx22_csum_none: 0
>> rx22_xdp_drop: 0
>> rx22_xdp_redirect: 0
>> rx22_lro_packets: 0
>> rx22_lro_bytes: 0
>> rx22_ecn_mark: 0
>> rx22_removed_vlan_packets: 4629499740
>> rx22_wqe_err: 0
>> rx22_mpwqe_filler_cqes: 0
>> rx22_mpwqe_filler_strides: 0
>> rx22_buff_alloc_err: 0
>> rx22_cqe_compress_blks: 0
>> rx22_cqe_compress_pkts: 0
>> rx22_page_reuse: 0
>> rx22_cache_reuse: 1407527
>> rx22_cache_full: 2313342088
>> rx22_cache_empty: 6656
>> rx22_cache_busy: 2313342341
>> rx22_cache_waive: 0
>> rx22_congst_umr: 0
>> rx22_arfs_err: 0
>> rx22_xdp_tx_xmit: 0
>> rx22_xdp_tx_full: 0
>> rx22_xdp_tx_err: 0
>> rx22_xdp_tx_cqes: 0
>> rx23_packets: 4387124505
>> rx23_bytes: 5718118678470
>> rx23_csum_complete: 4387124505
>> rx23_csum_unnecessary: 0
>> rx23_csum_unnecessary_inner: 0
>> rx23_csum_none: 0
>> rx23_xdp_drop: 0
>> rx23_xdp_redirect: 0
>> rx23_lro_packets: 0
>> rx23_lro_bytes: 0
>> rx23_ecn_mark: 0
>> rx23_removed_vlan_packets: 4387124505
>> rx23_wqe_err: 0
>> rx23_mpwqe_filler_cqes: 0
>> rx23_mpwqe_filler_strides: 0
>> rx23_buff_alloc_err: 0
>> rx23_cqe_compress_blks: 0
>> rx23_cqe_compress_pkts: 0
>> rx23_page_reuse: 0
>> rx23_cache_reuse: 2013280
>> rx23_cache_full: 2191548717
>> rx23_cache_empty: 6656
>> rx23_cache_busy: 2191548972
>> rx23_cache_waive: 0
>> rx23_congst_umr: 0
>> rx23_arfs_err: 0
>> rx23_xdp_tx_xmit: 0
>> rx23_xdp_tx_full: 0
>> rx23_xdp_tx_err: 0
>> rx23_xdp_tx_cqes: 0
>> rx24_packets: 4398791634
>> rx24_bytes: 5744875564632
>> rx24_csum_complete: 4398791634
>> rx24_csum_unnecessary: 0
>> rx24_csum_unnecessary_inner: 0
>> rx24_csum_none: 0
>> rx24_xdp_drop: 0
>> rx24_xdp_redirect: 0
>> rx24_lro_packets: 0
>> rx24_lro_bytes: 0
>> rx24_ecn_mark: 0
>> rx24_removed_vlan_packets: 4398791634
>> rx24_wqe_err: 0
>> rx24_mpwqe_filler_cqes: 0
>> rx24_mpwqe_filler_strides: 0
>> rx24_buff_alloc_err: 0
>> rx24_cqe_compress_blks: 0
>> rx24_cqe_compress_pkts: 0
>> rx24_page_reuse: 0
>> rx24_cache_reuse: 2143926100
>> rx24_cache_full: 55469496
>> rx24_cache_empty: 6656
>> rx24_cache_busy: 55469716
>> rx24_cache_waive: 0
>> rx24_congst_umr: 0
>> rx24_arfs_err: 0
>> rx24_xdp_tx_xmit: 0
>> rx24_xdp_tx_full: 0
>> rx24_xdp_tx_err: 0
>> rx24_xdp_tx_cqes: 0
>> rx25_packets: 4377204935
>> rx25_bytes: 5710369124105
>> rx25_csum_complete: 4377204935
>> rx25_csum_unnecessary: 0
>> rx25_csum_unnecessary_inner: 0
>> rx25_csum_none: 0
>> rx25_xdp_drop: 0
>> rx25_xdp_redirect: 0
>> rx25_lro_packets: 0
>> rx25_lro_bytes: 0
>> rx25_ecn_mark: 0
>> rx25_removed_vlan_packets: 4377204935
>> rx25_wqe_err: 0
>> rx25_mpwqe_filler_cqes: 0
>> rx25_mpwqe_filler_strides: 0
>> rx25_buff_alloc_err: 0
>> rx25_cqe_compress_blks: 0
>> rx25_cqe_compress_pkts: 0
>> rx25_page_reuse: 0
>> rx25_cache_reuse: 2132658660
>> rx25_cache_full: 55943584
>> rx25_cache_empty: 6656
>> rx25_cache_busy: 55943804
>> rx25_cache_waive: 0
>> rx25_congst_umr: 0
>> rx25_arfs_err: 0
>> rx25_xdp_tx_xmit: 0
>> rx25_xdp_tx_full: 0
>> rx25_xdp_tx_err: 0
>> rx25_xdp_tx_cqes: 0
>> rx26_packets: 4496003688
>> rx26_bytes: 5862180715503
>> rx26_csum_complete: 4496003688
>> rx26_csum_unnecessary: 0
>> rx26_csum_unnecessary_inner: 0
>> rx26_csum_none: 0
>> rx26_xdp_drop: 0
>> rx26_xdp_redirect: 0
>> rx26_lro_packets: 0
>> rx26_lro_bytes: 0
>> rx26_ecn_mark: 0
>> rx26_removed_vlan_packets: 4496003688
>> rx26_wqe_err: 0
>> rx26_mpwqe_filler_cqes: 0
>> rx26_mpwqe_filler_strides: 0
>> rx26_buff_alloc_err: 0
>> rx26_cqe_compress_blks: 0
>> rx26_cqe_compress_pkts: 0
>> rx26_page_reuse: 0
>> rx26_cache_reuse: 8
>> rx26_cache_full: 2248001581
>> rx26_cache_empty: 6656
>> rx26_cache_busy: 2248001836
>> rx26_cache_waive: 0
>> rx26_congst_umr: 0
>> rx26_arfs_err: 0
>> rx26_xdp_tx_xmit: 0
>> rx26_xdp_tx_full: 0
>> rx26_xdp_tx_err: 0
>> rx26_xdp_tx_cqes: 0
>> rx27_packets: 4341849333
>> rx27_bytes: 5678653545018
>> rx27_csum_complete: 4341849333
>> rx27_csum_unnecessary: 0
>> rx27_csum_unnecessary_inner: 0
>> rx27_csum_none: 0
>> rx27_xdp_drop: 0
>> rx27_xdp_redirect: 0
>> rx27_lro_packets: 0
>> rx27_lro_bytes: 0
>> rx27_ecn_mark: 0
>> rx27_removed_vlan_packets: 4341849333
>> rx27_wqe_err: 0
>> rx27_mpwqe_filler_cqes: 0
>> rx27_mpwqe_filler_strides: 0
>> rx27_buff_alloc_err: 0
>> rx27_cqe_compress_blks: 0
>> rx27_cqe_compress_pkts: 0
>> rx27_page_reuse: 0
>> rx27_cache_reuse: 1748188
>> rx27_cache_full: 2169176223
>> rx27_cache_empty: 6656
>> rx27_cache_busy: 2169176476
>> rx27_cache_waive: 0
>> rx27_congst_umr: 0
>> rx27_arfs_err: 0
>> rx27_xdp_tx_xmit: 0
>> rx27_xdp_tx_full: 0
>> rx27_xdp_tx_err: 0
>> rx27_xdp_tx_cqes: 0
>> rx28_packets: 0
>> rx28_bytes: 0
>> rx28_csum_complete: 0
>> rx28_csum_unnecessary: 0
>> rx28_csum_unnecessary_inner: 0
>> rx28_csum_none: 0
>> rx28_xdp_drop: 0
>> rx28_xdp_redirect: 0
>> rx28_lro_packets: 0
>> rx28_lro_bytes: 0
>> rx28_ecn_mark: 0
>> rx28_removed_vlan_packets: 0
>> rx28_wqe_err: 0
>> rx28_mpwqe_filler_cqes: 0
>> rx28_mpwqe_filler_strides: 0
>> rx28_buff_alloc_err: 0
>> rx28_cqe_compress_blks: 0
>> rx28_cqe_compress_pkts: 0
>> rx28_page_reuse: 0
>> rx28_cache_reuse: 0
>> rx28_cache_full: 0
>> rx28_cache_empty: 2560
>> rx28_cache_busy: 0
>> rx28_cache_waive: 0
>> rx28_congst_umr: 0
>> rx28_arfs_err: 0
>> rx28_xdp_tx_xmit: 0
>> rx28_xdp_tx_full: 0
>> rx28_xdp_tx_err: 0
>> rx28_xdp_tx_cqes: 0
>> rx29_packets: 0
>> rx29_bytes: 0
>> rx29_csum_complete: 0
>> rx29_csum_unnecessary: 0
>> rx29_csum_unnecessary_inner: 0
>> rx29_csum_none: 0
>> rx29_xdp_drop: 0
>> rx29_xdp_redirect: 0
>> rx29_lro_packets: 0
>> rx29_lro_bytes: 0
>> rx29_ecn_mark: 0
>> rx29_removed_vlan_packets: 0
>> rx29_wqe_err: 0
>> rx29_mpwqe_filler_cqes: 0
>> rx29_mpwqe_filler_strides: 0
>> rx29_buff_alloc_err: 0
>> rx29_cqe_compress_blks: 0
>> rx29_cqe_compress_pkts: 0
>> rx29_page_reuse: 0
>> rx29_cache_reuse: 0
>> rx29_cache_full: 0
>> rx29_cache_empty: 2560
>> rx29_cache_busy: 0
>> rx29_cache_waive: 0
>> rx29_congst_umr: 0
>> rx29_arfs_err: 0
>> rx29_xdp_tx_xmit: 0
>> rx29_xdp_tx_full: 0
>> rx29_xdp_tx_err: 0
>> rx29_xdp_tx_cqes: 0
>> rx30_packets: 0
>> rx30_bytes: 0
>> rx30_csum_complete: 0
>> rx30_csum_unnecessary: 0
>> rx30_csum_unnecessary_inner: 0
>> rx30_csum_none: 0
>> rx30_xdp_drop: 0
>> rx30_xdp_redirect: 0
>> rx30_lro_packets: 0
>> rx30_lro_bytes: 0
>> rx30_ecn_mark: 0
>> rx30_removed_vlan_packets: 0
>> rx30_wqe_err: 0
>> rx30_mpwqe_filler_cqes: 0
>> rx30_mpwqe_filler_strides: 0
>> rx30_buff_alloc_err: 0
>> rx30_cqe_compress_blks: 0
>> rx30_cqe_compress_pkts: 0
>> rx30_page_reuse: 0
>> rx30_cache_reuse: 0
>> rx30_cache_full: 0
>> rx30_cache_empty: 2560
>> rx30_cache_busy: 0
>> rx30_cache_waive: 0
>> rx30_congst_umr: 0
>> rx30_arfs_err: 0
>> rx30_xdp_tx_xmit: 0
>> rx30_xdp_tx_full: 0
>> rx30_xdp_tx_err: 0
>> rx30_xdp_tx_cqes: 0
>> rx31_packets: 0
>> rx31_bytes: 0
>> rx31_csum_complete: 0
>> rx31_csum_unnecessary: 0
>> rx31_csum_unnecessary_inner: 0
>> rx31_csum_none: 0
>> rx31_xdp_drop: 0
>> rx31_xdp_redirect: 0
>> rx31_lro_packets: 0
>> rx31_lro_bytes: 0
>> rx31_ecn_mark: 0
>> rx31_removed_vlan_packets: 0
>> rx31_wqe_err: 0
>> rx31_mpwqe_filler_cqes: 0
>> rx31_mpwqe_filler_strides: 0
>> rx31_buff_alloc_err: 0
>> rx31_cqe_compress_blks: 0
>> rx31_cqe_compress_pkts: 0
>> rx31_page_reuse: 0
>> rx31_cache_reuse: 0
>> rx31_cache_full: 0
>> rx31_cache_empty: 2560
>> rx31_cache_busy: 0
>> rx31_cache_waive: 0
>> rx31_congst_umr: 0
>> rx31_arfs_err: 0
>> rx31_xdp_tx_xmit: 0
>> rx31_xdp_tx_full: 0
>> rx31_xdp_tx_err: 0
>> rx31_xdp_tx_cqes: 0
>> rx32_packets: 0
>> rx32_bytes: 0
>> rx32_csum_complete: 0
>> rx32_csum_unnecessary: 0
>> rx32_csum_unnecessary_inner: 0
>> rx32_csum_none: 0
>> rx32_xdp_drop: 0
>> rx32_xdp_redirect: 0
>> rx32_lro_packets: 0
>> rx32_lro_bytes: 0
>> rx32_ecn_mark: 0
>> rx32_removed_vlan_packets: 0
>> rx32_wqe_err: 0
>> rx32_mpwqe_filler_cqes: 0
>> rx32_mpwqe_filler_strides: 0
>> rx32_buff_alloc_err: 0
>> rx32_cqe_compress_blks: 0
>> rx32_cqe_compress_pkts: 0
>> rx32_page_reuse: 0
>> rx32_cache_reuse: 0
>> rx32_cache_full: 0
>> rx32_cache_empty: 2560
>> rx32_cache_busy: 0
>> rx32_cache_waive: 0
>> rx32_congst_umr: 0
>> rx32_arfs_err: 0
>> rx32_xdp_tx_xmit: 0
>> rx32_xdp_tx_full: 0
>> rx32_xdp_tx_err: 0
>> rx32_xdp_tx_cqes: 0
>> rx33_packets: 0
>> rx33_bytes: 0
>> rx33_csum_complete: 0
>> rx33_csum_unnecessary: 0
>> rx33_csum_unnecessary_inner: 0
>> rx33_csum_none: 0
>> rx33_xdp_drop: 0
>> rx33_xdp_redirect: 0
>> rx33_lro_packets: 0
>> rx33_lro_bytes: 0
>> rx33_ecn_mark: 0
>> rx33_removed_vlan_packets: 0
>> rx33_wqe_err: 0
>> rx33_mpwqe_filler_cqes: 0
>> rx33_mpwqe_filler_strides: 0
>> rx33_buff_alloc_err: 0
>> rx33_cqe_compress_blks: 0
>> rx33_cqe_compress_pkts: 0
>> rx33_page_reuse: 0
>> rx33_cache_reuse: 0
>> rx33_cache_full: 0
>> rx33_cache_empty: 2560
>> rx33_cache_busy: 0
>> rx33_cache_waive: 0
>> rx33_congst_umr: 0
>> rx33_arfs_err: 0
>> rx33_xdp_tx_xmit: 0
>> rx33_xdp_tx_full: 0
>> rx33_xdp_tx_err: 0
>> rx33_xdp_tx_cqes: 0
>> rx34_packets: 0
>> rx34_bytes: 0
>> rx34_csum_complete: 0
>> rx34_csum_unnecessary: 0
>> rx34_csum_unnecessary_inner: 0
>> rx34_csum_none: 0
>> rx34_xdp_drop: 0
>> rx34_xdp_redirect: 0
>> rx34_lro_packets: 0
>> rx34_lro_bytes: 0
>> rx34_ecn_mark: 0
>> rx34_removed_vlan_packets: 0
>> rx34_wqe_err: 0
>> rx34_mpwqe_filler_cqes: 0
>> rx34_mpwqe_filler_strides: 0
>> rx34_buff_alloc_err: 0
>> rx34_cqe_compress_blks: 0
>> rx34_cqe_compress_pkts: 0
>> rx34_page_reuse: 0
>> rx34_cache_reuse: 0
>> rx34_cache_full: 0
>> rx34_cache_empty: 2560
>> rx34_cache_busy: 0
>> rx34_cache_waive: 0
>> rx34_congst_umr: 0
>> rx34_arfs_err: 0
>> rx34_xdp_tx_xmit: 0
>> rx34_xdp_tx_full: 0
>> rx34_xdp_tx_err: 0
>> rx34_xdp_tx_cqes: 0
>> rx35_packets: 0
>> rx35_bytes: 0
>> rx35_csum_complete: 0
>> rx35_csum_unnecessary: 0
>> rx35_csum_unnecessary_inner: 0
>> rx35_csum_none: 0
>> rx35_xdp_drop: 0
>> rx35_xdp_redirect: 0
>> rx35_lro_packets: 0
>> rx35_lro_bytes: 0
>> rx35_ecn_mark: 0
>> rx35_removed_vlan_packets: 0
>> rx35_wqe_err: 0
>> rx35_mpwqe_filler_cqes: 0
>> rx35_mpwqe_filler_strides: 0
>> rx35_buff_alloc_err: 0
>> rx35_cqe_compress_blks: 0
>> rx35_cqe_compress_pkts: 0
>> rx35_page_reuse: 0
>> rx35_cache_reuse: 0
>> rx35_cache_full: 0
>> rx35_cache_empty: 2560
>> rx35_cache_busy: 0
>> rx35_cache_waive: 0
>> rx35_congst_umr: 0
>> rx35_arfs_err: 0
>> rx35_xdp_tx_xmit: 0
>> rx35_xdp_tx_full: 0
>> rx35_xdp_tx_err: 0
>> rx35_xdp_tx_cqes: 0
>> rx36_packets: 0
>> rx36_bytes: 0
>> rx36_csum_complete: 0
>> rx36_csum_unnecessary: 0
>> rx36_csum_unnecessary_inner: 0
>> rx36_csum_none: 0
>> rx36_xdp_drop: 0
>> rx36_xdp_redirect: 0
>> rx36_lro_packets: 0
>> rx36_lro_bytes: 0
>> rx36_ecn_mark: 0
>> rx36_removed_vlan_packets: 0
>> rx36_wqe_err: 0
>> rx36_mpwqe_filler_cqes: 0
>> rx36_mpwqe_filler_strides: 0
>> rx36_buff_alloc_err: 0
>> rx36_cqe_compress_blks: 0
>> rx36_cqe_compress_pkts: 0
>> rx36_page_reuse: 0
>> rx36_cache_reuse: 0
>> rx36_cache_full: 0
>> rx36_cache_empty: 2560
>> rx36_cache_busy: 0
>> rx36_cache_waive: 0
>> rx36_congst_umr: 0
>> rx36_arfs_err: 0
>> rx36_xdp_tx_xmit: 0
>> rx36_xdp_tx_full: 0
>> rx36_xdp_tx_err: 0
>> rx36_xdp_tx_cqes: 0
>> rx37_packets: 0
>> rx37_bytes: 0
>> rx37_csum_complete: 0
>> rx37_csum_unnecessary: 0
>> rx37_csum_unnecessary_inner: 0
>> rx37_csum_none: 0
>> rx37_xdp_drop: 0
>> rx37_xdp_redirect: 0
>> rx37_lro_packets: 0
>> rx37_lro_bytes: 0
>> rx37_ecn_mark: 0
>> rx37_removed_vlan_packets: 0
>> rx37_wqe_err: 0
>> rx37_mpwqe_filler_cqes: 0
>> rx37_mpwqe_filler_strides: 0
>> rx37_buff_alloc_err: 0
>> rx37_cqe_compress_blks: 0
>> rx37_cqe_compress_pkts: 0
>> rx37_page_reuse: 0
>> rx37_cache_reuse: 0
>> rx37_cache_full: 0
>> rx37_cache_empty: 2560
>> rx37_cache_busy: 0
>> rx37_cache_waive: 0
>> rx37_congst_umr: 0
>> rx37_arfs_err: 0
>> rx37_xdp_tx_xmit: 0
>> rx37_xdp_tx_full: 0
>> rx37_xdp_tx_err: 0
>> rx37_xdp_tx_cqes: 0
>> rx38_packets: 0
>> rx38_bytes: 0
>> rx38_csum_complete: 0
>> rx38_csum_unnecessary: 0
>> rx38_csum_unnecessary_inner: 0
>> rx38_csum_none: 0
>> rx38_xdp_drop: 0
>> rx38_xdp_redirect: 0
>> rx38_lro_packets: 0
>> rx38_lro_bytes: 0
>> rx38_ecn_mark: 0
>> rx38_removed_vlan_packets: 0
>> rx38_wqe_err: 0
>> rx38_mpwqe_filler_cqes: 0
>> rx38_mpwqe_filler_strides: 0
>> rx38_buff_alloc_err: 0
>> rx38_cqe_compress_blks: 0
>> rx38_cqe_compress_pkts: 0
>> rx38_page_reuse: 0
>> rx38_cache_reuse: 0
>> rx38_cache_full: 0
>> rx38_cache_empty: 2560
>> rx38_cache_busy: 0
>> rx38_cache_waive: 0
>> rx38_congst_umr: 0
>> rx38_arfs_err: 0
>> rx38_xdp_tx_xmit: 0
>> rx38_xdp_tx_full: 0
>> rx38_xdp_tx_err: 0
>> rx38_xdp_tx_cqes: 0
>> rx39_packets: 0
>> rx39_bytes: 0
>> rx39_csum_complete: 0
>> rx39_csum_unnecessary: 0
>> rx39_csum_unnecessary_inner: 0
>> rx39_csum_none: 0
>> rx39_xdp_drop: 0
>> rx39_xdp_redirect: 0
>> rx39_lro_packets: 0
>> rx39_lro_bytes: 0
>> rx39_ecn_mark: 0
>> rx39_removed_vlan_packets: 0
>> rx39_wqe_err: 0
>> rx39_mpwqe_filler_cqes: 0
>> rx39_mpwqe_filler_strides: 0
>> rx39_buff_alloc_err: 0
>> rx39_cqe_compress_blks: 0
>> rx39_cqe_compress_pkts: 0
>> rx39_page_reuse: 0
>> rx39_cache_reuse: 0
>> rx39_cache_full: 0
>> rx39_cache_empty: 2560
>> rx39_cache_busy: 0
>> rx39_cache_waive: 0
>> rx39_congst_umr: 0
>> rx39_arfs_err: 0
>> rx39_xdp_tx_xmit: 0
>> rx39_xdp_tx_full: 0
>> rx39_xdp_tx_err: 0
>> rx39_xdp_tx_cqes: 0
>> rx40_packets: 0
>> rx40_bytes: 0
>> rx40_csum_complete: 0
>> rx40_csum_unnecessary: 0
>> rx40_csum_unnecessary_inner: 0
>> rx40_csum_none: 0
>> rx40_xdp_drop: 0
>> rx40_xdp_redirect: 0
>> rx40_lro_packets: 0
>> rx40_lro_bytes: 0
>> rx40_ecn_mark: 0
>> rx40_removed_vlan_packets: 0
>> rx40_wqe_err: 0
>> rx40_mpwqe_filler_cqes: 0
>> rx40_mpwqe_filler_strides: 0
>> rx40_buff_alloc_err: 0
>> rx40_cqe_compress_blks: 0
>> rx40_cqe_compress_pkts: 0
>> rx40_page_reuse: 0
>> rx40_cache_reuse: 0
>> rx40_cache_full: 0
>> rx40_cache_empty: 2560
>> rx40_cache_busy: 0
>> rx40_cache_waive: 0
>> rx40_congst_umr: 0
>> rx40_arfs_err: 0
>> rx40_xdp_tx_xmit: 0
>> rx40_xdp_tx_full: 0
>> rx40_xdp_tx_err: 0
>> rx40_xdp_tx_cqes: 0
>> rx41_packets: 0
>> rx41_bytes: 0
>> rx41_csum_complete: 0
>> rx41_csum_unnecessary: 0
>> rx41_csum_unnecessary_inner: 0
>> rx41_csum_none: 0
>> rx41_xdp_drop: 0
>> rx41_xdp_redirect: 0
>> rx41_lro_packets: 0
>> rx41_lro_bytes: 0
>> rx41_ecn_mark: 0
>> rx41_removed_vlan_packets: 0
>> rx41_wqe_err: 0
>> rx41_mpwqe_filler_cqes: 0
>> rx41_mpwqe_filler_strides: 0
>> rx41_buff_alloc_err: 0
>> rx41_cqe_compress_blks: 0
>> rx41_cqe_compress_pkts: 0
>> rx41_page_reuse: 0
>> rx41_cache_reuse: 0
>> rx41_cache_full: 0
>> rx41_cache_empty: 2560
>> rx41_cache_busy: 0
>> rx41_cache_waive: 0
>> rx41_congst_umr: 0
>> rx41_arfs_err: 0
>> rx41_xdp_tx_xmit: 0
>> rx41_xdp_tx_full: 0
>> rx41_xdp_tx_err: 0
>> rx41_xdp_tx_cqes: 0
>> rx42_packets: 0
>> rx42_bytes: 0
>> rx42_csum_complete: 0
>> rx42_csum_unnecessary: 0
>> rx42_csum_unnecessary_inner: 0
>> rx42_csum_none: 0
>> rx42_xdp_drop: 0
>> rx42_xdp_redirect: 0
>> rx42_lro_packets: 0
>> rx42_lro_bytes: 0
>> rx42_ecn_mark: 0
>> rx42_removed_vlan_packets: 0
>> rx42_wqe_err: 0
>> rx42_mpwqe_filler_cqes: 0
>> rx42_mpwqe_filler_strides: 0
>> rx42_buff_alloc_err: 0
>> rx42_cqe_compress_blks: 0
>> rx42_cqe_compress_pkts: 0
>> rx42_page_reuse: 0
>> rx42_cache_reuse: 0
>> rx42_cache_full: 0
>> rx42_cache_empty: 2560
>> rx42_cache_busy: 0
>> rx42_cache_waive: 0
>> rx42_congst_umr: 0
>> rx42_arfs_err: 0
>> rx42_xdp_tx_xmit: 0
>> rx42_xdp_tx_full: 0
>> rx42_xdp_tx_err: 0
>> rx42_xdp_tx_cqes: 0
>> rx43_packets: 0
>> rx43_bytes: 0
>> rx43_csum_complete: 0
>> rx43_csum_unnecessary: 0
>> rx43_csum_unnecessary_inner: 0
>> rx43_csum_none: 0
>> rx43_xdp_drop: 0
>> rx43_xdp_redirect: 0
>> rx43_lro_packets: 0
>> rx43_lro_bytes: 0
>> rx43_ecn_mark: 0
>> rx43_removed_vlan_packets: 0
>> rx43_wqe_err: 0
>> rx43_mpwqe_filler_cqes: 0
>> rx43_mpwqe_filler_strides: 0
>> rx43_buff_alloc_err: 0
>> rx43_cqe_compress_blks: 0
>> rx43_cqe_compress_pkts: 0
>> rx43_page_reuse: 0
>> rx43_cache_reuse: 0
>> rx43_cache_full: 0
>> rx43_cache_empty: 2560
>> rx43_cache_busy: 0
>> rx43_cache_waive: 0
>> rx43_congst_umr: 0
>> rx43_arfs_err: 0
>> rx43_xdp_tx_xmit: 0
>> rx43_xdp_tx_full: 0
>> rx43_xdp_tx_err: 0
>> rx43_xdp_tx_cqes: 0
>> rx44_packets: 0
>> rx44_bytes: 0
>> rx44_csum_complete: 0
>> rx44_csum_unnecessary: 0
>> rx44_csum_unnecessary_inner: 0
>> rx44_csum_none: 0
>> rx44_xdp_drop: 0
>> rx44_xdp_redirect: 0
>> rx44_lro_packets: 0
>> rx44_lro_bytes: 0
>> rx44_ecn_mark: 0
>> rx44_removed_vlan_packets: 0
>> rx44_wqe_err: 0
>> rx44_mpwqe_filler_cqes: 0
>> rx44_mpwqe_filler_strides: 0
>> rx44_buff_alloc_err: 0
>> rx44_cqe_compress_blks: 0
>> rx44_cqe_compress_pkts: 0
>> rx44_page_reuse: 0
>> rx44_cache_reuse: 0
>> rx44_cache_full: 0
>> rx44_cache_empty: 2560
>> rx44_cache_busy: 0
>> rx44_cache_waive: 0
>> rx44_congst_umr: 0
>> rx44_arfs_err: 0
>> rx44_xdp_tx_xmit: 0
>> rx44_xdp_tx_full: 0
>> rx44_xdp_tx_err: 0
>> rx44_xdp_tx_cqes: 0
>> rx45_packets: 0
>> rx45_bytes: 0
>> rx45_csum_complete: 0
>> rx45_csum_unnecessary: 0
>> rx45_csum_unnecessary_inner: 0
>> rx45_csum_none: 0
>> rx45_xdp_drop: 0
>> rx45_xdp_redirect: 0
>> rx45_lro_packets: 0
>> rx45_lro_bytes: 0
>> rx45_ecn_mark: 0
>> rx45_removed_vlan_packets: 0
>> rx45_wqe_err: 0
>> rx45_mpwqe_filler_cqes: 0
>> rx45_mpwqe_filler_strides: 0
>> rx45_buff_alloc_err: 0
>> rx45_cqe_compress_blks: 0
>> rx45_cqe_compress_pkts: 0
>> rx45_page_reuse: 0
>> rx45_cache_reuse: 0
>> rx45_cache_full: 0
>> rx45_cache_empty: 2560
>> rx45_cache_busy: 0
>> rx45_cache_waive: 0
>> rx45_congst_umr: 0
>> rx45_arfs_err: 0
>> rx45_xdp_tx_xmit: 0
>> rx45_xdp_tx_full: 0
>> rx45_xdp_tx_err: 0
>> rx45_xdp_tx_cqes: 0
>> rx46_packets: 0
>> rx46_bytes: 0
>> rx46_csum_complete: 0
>> rx46_csum_unnecessary: 0
>> rx46_csum_unnecessary_inner: 0
>> rx46_csum_none: 0
>> rx46_xdp_drop: 0
>> rx46_xdp_redirect: 0
>> rx46_lro_packets: 0
>> rx46_lro_bytes: 0
>> rx46_ecn_mark: 0
>> rx46_removed_vlan_packets: 0
>> rx46_wqe_err: 0
>> rx46_mpwqe_filler_cqes: 0
>> rx46_mpwqe_filler_strides: 0
>> rx46_buff_alloc_err: 0
>> rx46_cqe_compress_blks: 0
>> rx46_cqe_compress_pkts: 0
>> rx46_page_reuse: 0
>> rx46_cache_reuse: 0
>> rx46_cache_full: 0
>> rx46_cache_empty: 2560
>> rx46_cache_busy: 0
>> rx46_cache_waive: 0
>> rx46_congst_umr: 0
>> rx46_arfs_err: 0
>> rx46_xdp_tx_xmit: 0
>> rx46_xdp_tx_full: 0
>> rx46_xdp_tx_err: 0
>> rx46_xdp_tx_cqes: 0
>> rx47_packets: 0
>> rx47_bytes: 0
>> rx47_csum_complete: 0
>> rx47_csum_unnecessary: 0
>> rx47_csum_unnecessary_inner: 0
>> rx47_csum_none: 0
>> rx47_xdp_drop: 0
>> rx47_xdp_redirect: 0
>> rx47_lro_packets: 0
>> rx47_lro_bytes: 0
>> rx47_ecn_mark: 0
>> rx47_removed_vlan_packets: 0
>> rx47_wqe_err: 0
>> rx47_mpwqe_filler_cqes: 0
>> rx47_mpwqe_filler_strides: 0
>> rx47_buff_alloc_err: 0
>> rx47_cqe_compress_blks: 0
>> rx47_cqe_compress_pkts: 0
>> rx47_page_reuse: 0
>> rx47_cache_reuse: 0
>> rx47_cache_full: 0
>> rx47_cache_empty: 2560
>> rx47_cache_busy: 0
>> rx47_cache_waive: 0
>> rx47_congst_umr: 0
>> rx47_arfs_err: 0
>> rx47_xdp_tx_xmit: 0
>> rx47_xdp_tx_full: 0
>> rx47_xdp_tx_err: 0
>> rx47_xdp_tx_cqes: 0
>> rx48_packets: 0
>> rx48_bytes: 0
>> rx48_csum_complete: 0
>> rx48_csum_unnecessary: 0
>> rx48_csum_unnecessary_inner: 0
>> rx48_csum_none: 0
>> rx48_xdp_drop: 0
>> rx48_xdp_redirect: 0
>> rx48_lro_packets: 0
>> rx48_lro_bytes: 0
>> rx48_ecn_mark: 0
>> rx48_removed_vlan_packets: 0
>> rx48_wqe_err: 0
>> rx48_mpwqe_filler_cqes: 0
>> rx48_mpwqe_filler_strides: 0
>> rx48_buff_alloc_err: 0
>> rx48_cqe_compress_blks: 0
>> rx48_cqe_compress_pkts: 0
>> rx48_page_reuse: 0
>> rx48_cache_reuse: 0
>> rx48_cache_full: 0
>> rx48_cache_empty: 2560
>> rx48_cache_busy: 0
>> rx48_cache_waive: 0
>> rx48_congst_umr: 0
>> rx48_arfs_err: 0
>> rx48_xdp_tx_xmit: 0
>> rx48_xdp_tx_full: 0
>> rx48_xdp_tx_err: 0
>> rx48_xdp_tx_cqes: 0
>> rx49_packets: 0
>> rx49_bytes: 0
>> rx49_csum_complete: 0
>> rx49_csum_unnecessary: 0
>> rx49_csum_unnecessary_inner: 0
>> rx49_csum_none: 0
>> rx49_xdp_drop: 0
>> rx49_xdp_redirect: 0
>> rx49_lro_packets: 0
>> rx49_lro_bytes: 0
>> rx49_ecn_mark: 0
>> rx49_removed_vlan_packets: 0
>> rx49_wqe_err: 0
>> rx49_mpwqe_filler_cqes: 0
>> rx49_mpwqe_filler_strides: 0
>> rx49_buff_alloc_err: 0
>> rx49_cqe_compress_blks: 0
>> rx49_cqe_compress_pkts: 0
>> rx49_page_reuse: 0
>> rx49_cache_reuse: 0
>> rx49_cache_full: 0
>> rx49_cache_empty: 2560
>> rx49_cache_busy: 0
>> rx49_cache_waive: 0
>> rx49_congst_umr: 0
>> rx49_arfs_err: 0
>> rx49_xdp_tx_xmit: 0
>> rx49_xdp_tx_full: 0
>> rx49_xdp_tx_err: 0
>> rx49_xdp_tx_cqes: 0
>> rx50_packets: 0
>> rx50_bytes: 0
>> rx50_csum_complete: 0
>> rx50_csum_unnecessary: 0
>> rx50_csum_unnecessary_inner: 0
>> rx50_csum_none: 0
>> rx50_xdp_drop: 0
>> rx50_xdp_redirect: 0
>> rx50_lro_packets: 0
>> rx50_lro_bytes: 0
>> rx50_ecn_mark: 0
>> rx50_removed_vlan_packets: 0
>> rx50_wqe_err: 0
>> rx50_mpwqe_filler_cqes: 0
>> rx50_mpwqe_filler_strides: 0
>> rx50_buff_alloc_err: 0
>> rx50_cqe_compress_blks: 0
>> rx50_cqe_compress_pkts: 0
>> rx50_page_reuse: 0
>> rx50_cache_reuse: 0
>> rx50_cache_full: 0
>> rx50_cache_empty: 2560
>> rx50_cache_busy: 0
>> rx50_cache_waive: 0
>> rx50_congst_umr: 0
>> rx50_arfs_err: 0
>> rx50_xdp_tx_xmit: 0
>> rx50_xdp_tx_full: 0
>> rx50_xdp_tx_err: 0
>> rx50_xdp_tx_cqes: 0
>> rx51_packets: 0
>> rx51_bytes: 0
>> rx51_csum_complete: 0
>> rx51_csum_unnecessary: 0
>> rx51_csum_unnecessary_inner: 0
>> rx51_csum_none: 0
>> rx51_xdp_drop: 0
>> rx51_xdp_redirect: 0
>> rx51_lro_packets: 0
>> rx51_lro_bytes: 0
>> rx51_ecn_mark: 0
>> rx51_removed_vlan_packets: 0
>> rx51_wqe_err: 0
>> rx51_mpwqe_filler_cqes: 0
>> rx51_mpwqe_filler_strides: 0
>> rx51_buff_alloc_err: 0
>> rx51_cqe_compress_blks: 0
>> rx51_cqe_compress_pkts: 0
>> rx51_page_reuse: 0
>> rx51_cache_reuse: 0
>> rx51_cache_full: 0
>> rx51_cache_empty: 2560
>> rx51_cache_busy: 0
>> rx51_cache_waive: 0
>> rx51_congst_umr: 0
>> rx51_arfs_err: 0
>> rx51_xdp_tx_xmit: 0
>> rx51_xdp_tx_full: 0
>> rx51_xdp_tx_err: 0
>> rx51_xdp_tx_cqes: 0
>> rx52_packets: 0
>> rx52_bytes: 0
>> rx52_csum_complete: 0
>> rx52_csum_unnecessary: 0
>> rx52_csum_unnecessary_inner: 0
>> rx52_csum_none: 0
>> rx52_xdp_drop: 0
>> rx52_xdp_redirect: 0
>> rx52_lro_packets: 0
>> rx52_lro_bytes: 0
>> rx52_ecn_mark: 0
>> rx52_removed_vlan_packets: 0
>> rx52_wqe_err: 0
>> rx52_mpwqe_filler_cqes: 0
>> rx52_mpwqe_filler_strides: 0
>> rx52_buff_alloc_err: 0
>> rx52_cqe_compress_blks: 0
>> rx52_cqe_compress_pkts: 0
>> rx52_page_reuse: 0
>> rx52_cache_reuse: 0
>> rx52_cache_full: 0
>> rx52_cache_empty: 2560
>> rx52_cache_busy: 0
>> rx52_cache_waive: 0
>> rx52_congst_umr: 0
>> rx52_arfs_err: 0
>> rx52_xdp_tx_xmit: 0
>> rx52_xdp_tx_full: 0
>> rx52_xdp_tx_err: 0
>> rx52_xdp_tx_cqes: 0
>> rx53_packets: 0
>> rx53_bytes: 0
>> rx53_csum_complete: 0
>> rx53_csum_unnecessary: 0
>> rx53_csum_unnecessary_inner: 0
>> rx53_csum_none: 0
>> rx53_xdp_drop: 0
>> rx53_xdp_redirect: 0
>> rx53_lro_packets: 0
>> rx53_lro_bytes: 0
>> rx53_ecn_mark: 0
>> rx53_removed_vlan_packets: 0
>> rx53_wqe_err: 0
>> rx53_mpwqe_filler_cqes: 0
>> rx53_mpwqe_filler_strides: 0
>> rx53_buff_alloc_err: 0
>> rx53_cqe_compress_blks: 0
>> rx53_cqe_compress_pkts: 0
>> rx53_page_reuse: 0
>> rx53_cache_reuse: 0
>> rx53_cache_full: 0
>> rx53_cache_empty: 2560
>> rx53_cache_busy: 0
>> rx53_cache_waive: 0
>> rx53_congst_umr: 0
>> rx53_arfs_err: 0
>> rx53_xdp_tx_xmit: 0
>> rx53_xdp_tx_full: 0
>> rx53_xdp_tx_err: 0
>> rx53_xdp_tx_cqes: 0
>> rx54_packets: 0
>> rx54_bytes: 0
>> rx54_csum_complete: 0
>> rx54_csum_unnecessary: 0
>> rx54_csum_unnecessary_inner: 0
>> rx54_csum_none: 0
>> rx54_xdp_drop: 0
>> rx54_xdp_redirect: 0
>> rx54_lro_packets: 0
>> rx54_lro_bytes: 0
>> rx54_ecn_mark: 0
>> rx54_removed_vlan_packets: 0
>> rx54_wqe_err: 0
>> rx54_mpwqe_filler_cqes: 0
>> rx54_mpwqe_filler_strides: 0
>> rx54_buff_alloc_err: 0
>> rx54_cqe_compress_blks: 0
>> rx54_cqe_compress_pkts: 0
>> rx54_page_reuse: 0
>> rx54_cache_reuse: 0
>> rx54_cache_full: 0
>> rx54_cache_empty: 2560
>> rx54_cache_busy: 0
>> rx54_cache_waive: 0
>> rx54_congst_umr: 0
>> rx54_arfs_err: 0
>> rx54_xdp_tx_xmit: 0
>> rx54_xdp_tx_full: 0
>> rx54_xdp_tx_err: 0
>> rx54_xdp_tx_cqes: 0
>> rx55_packets: 0
>> rx55_bytes: 0
>> rx55_csum_complete: 0
>> rx55_csum_unnecessary: 0
>> rx55_csum_unnecessary_inner: 0
>> rx55_csum_none: 0
>> rx55_xdp_drop: 0
>> rx55_xdp_redirect: 0
>> rx55_lro_packets: 0
>> rx55_lro_bytes: 0
>> rx55_ecn_mark: 0
>> rx55_removed_vlan_packets: 0
>> rx55_wqe_err: 0
>> rx55_mpwqe_filler_cqes: 0
>> rx55_mpwqe_filler_strides: 0
>> rx55_buff_alloc_err: 0
>> rx55_cqe_compress_blks: 0
>> rx55_cqe_compress_pkts: 0
>> rx55_page_reuse: 0
>> rx55_cache_reuse: 0
>> rx55_cache_full: 0
>> rx55_cache_empty: 2560
>> rx55_cache_busy: 0
>> rx55_cache_waive: 0
>> rx55_congst_umr: 0
>> rx55_arfs_err: 0
>> rx55_xdp_tx_xmit: 0
>> rx55_xdp_tx_full: 0
>> rx55_xdp_tx_err: 0
>> rx55_xdp_tx_cqes: 0
>> tx0_packets: 6019477917
>> tx0_bytes: 3445238940825
>> tx0_tso_packets: 311304622
>> tx0_tso_bytes: 1897094773213
>> tx0_tso_inner_packets: 0
>> tx0_tso_inner_bytes: 0
>> tx0_csum_partial: 457981794
>> tx0_csum_partial_inner: 0
>> tx0_added_vlan_packets: 4965567654
>> tx0_nop: 72290329
>> tx0_csum_none: 4507585860
>> tx0_stopped: 9118
>> tx0_dropped: 0
>> tx0_xmit_more: 51651593
>> tx0_recover: 0
>> tx0_cqes: 4913918402
>> tx0_wake: 9118
>> tx0_cqe_err: 0
>> tx1_packets: 5700413414
>> tx1_bytes: 3340870662350
>> tx1_tso_packets: 318201557
>> tx1_tso_bytes: 1915233462303
>> tx1_tso_inner_packets: 0
>> tx1_tso_inner_bytes: 0
>> tx1_csum_partial: 461736722
>> tx1_csum_partial_inner: 0
>> tx1_added_vlan_packets: 4638708749
>> tx1_nop: 70061796
>> tx1_csum_none: 4176972027
>> tx1_stopped: 9248
>> tx1_dropped: 0
>> tx1_xmit_more: 39531959
>> tx1_recover: 0
>> tx1_cqes: 4599179178
>> tx1_wake: 9248
>> tx1_cqe_err: 0
>> tx2_packets: 5795960848
>> tx2_bytes: 3394876820271
>> tx2_tso_packets: 322935065
>> tx2_tso_bytes: 1910825901109
>> tx2_tso_inner_packets: 0
>> tx2_tso_inner_bytes: 0
>> tx2_csum_partial: 460747092
>> tx2_csum_partial_inner: 0
>> tx2_added_vlan_packets: 4743705654
>> tx2_nop: 72722430
>> tx2_csum_none: 4282958562
>> tx2_stopped: 8938
>> tx2_dropped: 0
>> tx2_xmit_more: 44084718
>> tx2_recover: 0
>> tx2_cqes: 4699623410
>> tx2_wake: 8938
>> tx2_cqe_err: 0
>> tx3_packets: 5580215878
>> tx3_bytes: 3191677257787
>> tx3_tso_packets: 305771141
>> tx3_tso_bytes: 1823265793476
>> tx3_tso_inner_packets: 0
>> tx3_tso_inner_bytes: 0
>> tx3_csum_partial: 434976070
>> tx3_csum_partial_inner: 0
>> tx3_added_vlan_packets: 4569899956
>> tx3_nop: 68184348
>> tx3_csum_none: 4134923886
>> tx3_stopped: 8383
>> tx3_dropped: 0
>> tx3_xmit_more: 41940375
>> tx3_recover: 0
>> tx3_cqes: 4527961924
>> tx3_wake: 8383
>> tx3_cqe_err: 0
>> tx4_packets: 6795007068
>> tx4_bytes: 3963890025270
>> tx4_tso_packets: 358437617
>> tx4_tso_bytes: 2154747995355
>> tx4_tso_inner_packets: 0
>> tx4_tso_inner_bytes: 0
>> tx4_csum_partial: 504764524
>> tx4_csum_partial_inner: 0
>> tx4_added_vlan_packets: 5602510191
>> tx4_nop: 81345604
>> tx4_csum_none: 5097745667
>> tx4_stopped: 10248
>> tx4_dropped: 0
>> tx4_xmit_more: 49068571
>> tx4_recover: 0
>> tx4_cqes: 5553444276
>> tx4_wake: 10248
>> tx4_cqe_err: 0
>> tx5_packets: 6408089261
>> tx5_bytes: 3676275848279
>> tx5_tso_packets: 345129329
>> tx5_tso_bytes: 2108447877473
>> tx5_tso_inner_packets: 0
>> tx5_tso_inner_bytes: 0
>> tx5_csum_partial: 494705894
>> tx5_csum_partial_inner: 0
>> tx5_added_vlan_packets: 5235998343
>> tx5_nop: 77694627
>> tx5_csum_none: 4741292449
>> tx5_stopped: 46
>> tx5_dropped: 0
>> tx5_xmit_more: 46675831
>> tx5_recover: 0
>> tx5_cqes: 5189323550
>> tx5_wake: 46
>> tx5_cqe_err: 0
>> tx6_packets: 6382289663
>> tx6_bytes: 3670991418150
>> tx6_tso_packets: 342927826
>> tx6_tso_bytes: 2075049679904
>> tx6_tso_inner_packets: 0
>> tx6_tso_inner_bytes: 0
>> tx6_csum_partial: 490369221
>> tx6_csum_partial_inner: 0
>> tx6_added_vlan_packets: 5232144528
>> tx6_nop: 77391246
>> tx6_csum_none: 4741775307
>> tx6_stopped: 10823
>> tx6_dropped: 0
>> tx6_xmit_more: 44487607
>> tx6_recover: 0
>> tx6_cqes: 5187659877
>> tx6_wake: 10823
>> tx6_cqe_err: 0
>> tx7_packets: 6456378284
>> tx7_bytes: 3758013320518
>> tx7_tso_packets: 350958294
>> tx7_tso_bytes: 2126833408524
>> tx7_tso_inner_packets: 0
>> tx7_tso_inner_bytes: 0
>> tx7_csum_partial: 501804109
>> tx7_csum_partial_inner: 0
>> tx7_added_vlan_packets: 5275635204
>> tx7_nop: 79010883
>> tx7_csum_none: 4773831096
>> tx7_stopped: 14684
>> tx7_dropped: 0
>> tx7_xmit_more: 44447469
>> tx7_recover: 0
>> tx7_cqes: 5231191770
>> tx7_wake: 14684
>> tx7_cqe_err: 0
>> tx8_packets: 6401799768
>> tx8_bytes: 3681210808766
>> tx8_tso_packets: 342878228
>> tx8_tso_bytes: 2089688012191
>> tx8_tso_inner_packets: 0
>> tx8_tso_inner_bytes: 0
>> tx8_csum_partial: 494865145
>> tx8_csum_partial_inner: 0
>> tx8_added_vlan_packets: 5242288908
>> tx8_nop: 77250910
>> tx8_csum_none: 4747423763
>> tx8_stopped: 2
>> tx8_dropped: 0
>> tx8_xmit_more: 44191737
>> tx8_recover: 0
>> tx8_cqes: 5198098454
>> tx8_wake: 2
>> tx8_cqe_err: 0
>> tx9_packets: 6632882888
>> tx9_bytes: 3820110338309
>> tx9_tso_packets: 354189056
>> tx9_tso_bytes: 2187883597128
>> tx9_tso_inner_packets: 0
>> tx9_tso_inner_bytes: 0
>> tx9_csum_partial: 511108218
>> tx9_csum_partial_inner: 0
>> tx9_added_vlan_packets: 5413836353
>> tx9_nop: 80560668
>> tx9_csum_none: 4902728135
>> tx9_stopped: 9091
>> tx9_dropped: 0
>> tx9_xmit_more: 54501293
>> tx9_recover: 0
>> tx9_cqes: 5359337638
>> tx9_wake: 9091
>> tx9_cqe_err: 0
>> tx10_packets: 6421786406
>> tx10_bytes: 3692798413429
>> tx10_tso_packets: 346878943
>> tx10_tso_bytes: 2111921062110
>> tx10_tso_inner_packets: 0
>> tx10_tso_inner_bytes: 0
>> tx10_csum_partial: 494356645
>> tx10_csum_partial_inner: 0
>> tx10_added_vlan_packets: 5248274374
>> tx10_nop: 77922624
>> tx10_csum_none: 4753917730
>> tx10_stopped: 9617
>> tx10_dropped: 0
>> tx10_xmit_more: 44473939
>> tx10_recover: 0
>> tx10_cqes: 5203802547
>> tx10_wake: 9617
>> tx10_cqe_err: 0
>> tx11_packets: 6406750938
>> tx11_bytes: 3660343565126
>> tx11_tso_packets: 355917271
>> tx11_tso_bytes: 2130812246956
>> tx11_tso_inner_packets: 0
>> tx11_tso_inner_bytes: 0
>> tx11_csum_partial: 500336369
>> tx11_csum_partial_inner: 0
>> tx11_added_vlan_packets: 5228267547
>> tx11_nop: 78906315
>> tx11_csum_none: 4727931178
>> tx11_stopped: 9607
>> tx11_dropped: 0
>> tx11_xmit_more: 40041492
>> tx11_recover: 0
>> tx11_cqes: 5188228290
>> tx11_wake: 9607
>> tx11_cqe_err: 0
>> tx12_packets: 6422347846
>> tx12_bytes: 3718772753227
>> tx12_tso_packets: 355397223
>> tx12_tso_bytes: 2162614059758
>> tx12_tso_inner_packets: 0
>> tx12_tso_inner_bytes: 0
>> tx12_csum_partial: 511437844
>> tx12_csum_partial_inner: 0
>> tx12_added_vlan_packets: 5221373746
>> tx12_nop: 78866779
>> tx12_csum_none: 4709935902
>> tx12_stopped: 10280
>> tx12_dropped: 0
>> tx12_xmit_more: 42189399
>> tx12_recover: 0
>> tx12_cqes: 5179187154
>> tx12_wake: 10280
>> tx12_cqe_err: 0
>> tx13_packets: 6429383816
>> tx13_bytes: 3725679445046
>> tx13_tso_packets: 360934759
>> tx13_tso_bytes: 2148016411436
>> tx13_tso_inner_packets: 0
>> tx13_tso_inner_bytes: 0
>> tx13_csum_partial: 505245849
>> tx13_csum_partial_inner: 0
>> tx13_added_vlan_packets: 5240267441
>> tx13_nop: 80295637
>> tx13_csum_none: 4735021592
>> tx13_stopped: 84
>> tx13_dropped: 0
>> tx13_xmit_more: 43118045
>> tx13_recover: 0
>> tx13_cqes: 5197150348
>> tx13_wake: 84
>> tx13_cqe_err: 0
>> tx14_packets: 6375279148
>> tx14_bytes: 3624267203336
>> tx14_tso_packets: 344388148
>> tx14_tso_bytes: 2094966273548
>> tx14_tso_inner_packets: 0
>> tx14_tso_inner_bytes: 0
>> tx14_csum_partial: 494129407
>> tx14_csum_partial_inner: 0
>> tx14_added_vlan_packets: 5210749337
>> tx14_nop: 77280615
>> tx14_csum_none: 4716619930
>> tx14_stopped: 13057
>> tx14_dropped: 0
>> tx14_xmit_more: 40849682
>> tx14_recover: 0
>> tx14_cqes: 5169902694
>> tx14_wake: 13057
>> tx14_cqe_err: 0
>> tx15_packets: 6489306520
>> tx15_bytes: 3775716194795
>> tx15_tso_packets: 368716406
>> tx15_tso_bytes: 2165876423354
>> tx15_tso_inner_packets: 0
>> tx15_tso_inner_bytes: 0
>> tx15_csum_partial: 509887864
>> tx15_csum_partial_inner: 0
>> tx15_added_vlan_packets: 5296767390
>> tx15_nop: 80803468
>> tx15_csum_none: 4786879529
>> tx15_stopped: 1
>> tx15_dropped: 0
>> tx15_xmit_more: 46979676
>> tx15_recover: 0
>> tx15_cqes: 5249789328
>> tx15_wake: 1
>> tx15_cqe_err: 0
>> tx16_packets: 6559857761
>> tx16_bytes: 3724080573905
>> tx16_tso_packets: 350864176
>> tx16_tso_bytes: 2099634006033
>> tx16_tso_inner_packets: 0
>> tx16_tso_inner_bytes: 0
>> tx16_csum_partial: 489397232
>> tx16_csum_partial_inner: 0
>> tx16_added_vlan_packets: 5398869334
>> tx16_nop: 79046075
>> tx16_csum_none: 4909472106
>> tx16_stopped: 4480
>> tx16_dropped: 0
>> tx16_xmit_more: 47273286
>> tx16_recover: 0
>> tx16_cqes: 5351598315
>> tx16_wake: 4480
>> tx16_cqe_err: 0
>> tx17_packets: 6358711533
>> tx17_bytes: 3650180865573
>> tx17_tso_packets: 350723136
>> tx17_tso_bytes: 2109426587128
>> tx17_tso_inner_packets: 0
>> tx17_tso_inner_bytes: 0
>> tx17_csum_partial: 494719487
>> tx17_csum_partial_inner: 0
>> tx17_added_vlan_packets: 5190068796
>> tx17_nop: 77285612
>> tx17_csum_none: 4695349309
>> tx17_stopped: 10443
>> tx17_dropped: 0
>> tx17_xmit_more: 45582108
>> tx17_recover: 0
>> tx17_cqes: 5144489363
>> tx17_wake: 10443
>> tx17_cqe_err: 0
>> tx18_packets: 6655328437
>> tx18_bytes: 3801768461807
>> tx18_tso_packets: 356516373
>> tx18_tso_bytes: 2164829247550
>> tx18_tso_inner_packets: 0
>> tx18_tso_inner_bytes: 0
>> tx18_csum_partial: 500508446
>> tx18_csum_partial_inner: 0
>> tx18_added_vlan_packets: 5454166840
>> tx18_nop: 80423007
>> tx18_csum_none: 4953658394
>> tx18_stopped: 14760
>> tx18_dropped: 0
>> tx18_xmit_more: 50837465
>> tx18_recover: 0
>> tx18_cqes: 5403332553
>> tx18_wake: 14760
>> tx18_cqe_err: 0
>> tx19_packets: 6408680611
>> tx19_bytes: 3644119934372
>> tx19_tso_packets: 350727530
>> tx19_tso_bytes: 2089896715365
>> tx19_tso_inner_packets: 0
>> tx19_tso_inner_bytes: 0
>> tx19_csum_partial: 486536490
>> tx19_csum_partial_inner: 0
>> tx19_added_vlan_packets: 5255839020
>> tx19_nop: 78525198
>> tx19_csum_none: 4769302530
>> tx19_stopped: 8614
>> tx19_dropped: 0
>> tx19_xmit_more: 43605232
>> tx19_recover: 0
>> tx19_cqes: 5212236833
>> tx19_wake: 8614
>> tx19_cqe_err: 0
>> tx20_packets: 5609275141
>> tx20_bytes: 3187279031581
>> tx20_tso_packets: 298609303
>> tx20_tso_bytes: 1794382229379
>> tx20_tso_inner_packets: 0
>> tx20_tso_inner_bytes: 0
>> tx20_csum_partial: 430691178
>> tx20_csum_partial_inner: 0
>> tx20_added_vlan_packets: 4616844286
>> tx20_nop: 67450040
>> tx20_csum_none: 4186153108
>> tx20_stopped: 9099
>> tx20_dropped: 0
>> tx20_xmit_more: 42040991
>> tx20_recover: 0
>> tx20_cqes: 4574805846
>> tx20_wake: 9099
>> tx20_cqe_err: 0
>> tx21_packets: 5641621183
>> tx21_bytes: 3279282331124
>> tx21_tso_packets: 311297057
>> tx21_tso_bytes: 1875735401012
>> tx21_tso_inner_packets: 0
>> tx21_tso_inner_bytes: 0
>> tx21_csum_partial: 444333894
>> tx21_csum_partial_inner: 0
>> tx21_added_vlan_packets: 4603527701
>> tx21_nop: 68857983
>> tx21_csum_none: 4159193807
>> tx21_stopped: 10082
>> tx21_dropped: 0
>> tx21_xmit_more: 43988081
>> tx21_recover: 0
>> tx21_cqes: 4559542410
>> tx21_wake: 10082
>> tx21_cqe_err: 0
>> tx22_packets: 5822168288
>> tx22_bytes: 3452026726862
>> tx22_tso_packets: 308230791
>> tx22_tso_bytes: 1859686450671
>> tx22_tso_inner_packets: 0
>> tx22_tso_inner_bytes: 0
>> tx22_csum_partial: 442751518
>> tx22_csum_partial_inner: 0
>> tx22_added_vlan_packets: 4792100335
>> tx22_nop: 70631706
>> tx22_csum_none: 4349348817
>> tx22_stopped: 9355
>> tx22_dropped: 0
>> tx22_xmit_more: 45165994
>> tx22_recover: 0
>> tx22_cqes: 4746936601
>> tx22_wake: 9355
>> tx22_cqe_err: 0
>> tx23_packets: 5664896066
>> tx23_bytes: 3207724186946
>> tx23_tso_packets: 300418757
>> tx23_tso_bytes: 1794180478679
>> tx23_tso_inner_packets: 0
>> tx23_tso_inner_bytes: 0
>> tx23_csum_partial: 429898848
>> tx23_csum_partial_inner: 0
>> tx23_added_vlan_packets: 4674317320
>> tx23_nop: 67899896
>> tx23_csum_none: 4244418472
>> tx23_stopped: 11684
>> tx23_dropped: 0
>> tx23_xmit_more: 43351132
>> tx23_recover: 0
>> tx23_cqes: 4630969028
>> tx23_wake: 11684
>> tx23_cqe_err: 0
>> tx24_packets: 5663326601
>> tx24_bytes: 3250127095110
>> tx24_tso_packets: 301327422
>> tx24_tso_bytes: 1831260534157
>> tx24_tso_inner_packets: 0
>> tx24_tso_inner_bytes: 0
>> tx24_csum_partial: 438757312
>> tx24_csum_partial_inner: 0
>> tx24_added_vlan_packets: 4646014986
>> tx24_nop: 68431153
>> tx24_csum_none: 4207257674
>> tx24_stopped: 9240
>> tx24_dropped: 0
>> tx24_xmit_more: 47699542
>> tx24_recover: 0
>> tx24_cqes: 4598317913
>> tx24_wake: 9240
>> tx24_cqe_err: 0
>> tx25_packets: 5703883962
>> tx25_bytes: 3291856915695
>> tx25_tso_packets: 308900318
>> tx25_tso_bytes: 1855516128386
>> tx25_tso_inner_packets: 0
>> tx25_tso_inner_bytes: 0
>> tx25_csum_partial: 444753744
>> tx25_csum_partial_inner: 0
>> tx25_added_vlan_packets: 4676528924
>> tx25_nop: 69230967
>> tx25_csum_none: 4231775180
>> tx25_stopped: 1140
>> tx25_dropped: 0
>> tx25_xmit_more: 40819195
>> tx25_recover: 0
>> tx25_cqes: 4635710966
>> tx25_wake: 1140
>> tx25_cqe_err: 0
>> tx26_packets: 5803495984
>> tx26_bytes: 3413564272139
>> tx26_tso_packets: 319986230
>> tx26_tso_bytes: 1929042839677
>> tx26_tso_inner_packets: 0
>> tx26_tso_inner_bytes: 0
>> tx26_csum_partial: 464771163
>> tx26_csum_partial_inner: 0
>> tx26_added_vlan_packets: 4734767280
>> tx26_nop: 71345080
>> tx26_csum_none: 4269996117
>> tx26_stopped: 10972
>> tx26_dropped: 0
>> tx26_xmit_more: 43793424
>> tx26_recover: 0
>> tx26_cqes: 4690976400
>> tx26_wake: 10972
>> tx26_cqe_err: 0
>> tx27_packets: 5960955343
>> tx27_bytes: 3444156164526
>> tx27_tso_packets: 325099639
>> tx27_tso_bytes: 1928378678784
>> tx27_tso_inner_packets: 0
>> tx27_tso_inner_bytes: 0
>> tx27_csum_partial: 467310289
>> tx27_csum_partial_inner: 0
>> tx27_added_vlan_packets: 4888651368
>> tx27_nop: 73201664
>> tx27_csum_none: 4421341079
>> tx27_stopped: 9465
>> tx27_dropped: 0
>> tx27_xmit_more: 53632121
>> tx27_recover: 0
>> tx27_cqes: 4835021398
>> tx27_wake: 9465
>> tx27_cqe_err: 0
>> tx28_packets: 0
>> tx28_bytes: 0
>> tx28_tso_packets: 0
>> tx28_tso_bytes: 0
>> tx28_tso_inner_packets: 0
>> tx28_tso_inner_bytes: 0
>> tx28_csum_partial: 0
>> tx28_csum_partial_inner: 0
>> tx28_added_vlan_packets: 0
>> tx28_nop: 0
>> tx28_csum_none: 0
>> tx28_stopped: 0
>> tx28_dropped: 0
>> tx28_xmit_more: 0
>> tx28_recover: 0
>> tx28_cqes: 0
>> tx28_wake: 0
>> tx28_cqe_err: 0
>> tx29_packets: 3
>> tx29_bytes: 266
>> tx29_tso_packets: 0
>> tx29_tso_bytes: 0
>> tx29_tso_inner_packets: 0
>> tx29_tso_inner_bytes: 0
>> tx29_csum_partial: 0
>> tx29_csum_partial_inner: 0
>> tx29_added_vlan_packets: 0
>> tx29_nop: 0
>> tx29_csum_none: 3
>> tx29_stopped: 0
>> tx29_dropped: 0
>> tx29_xmit_more: 1
>> tx29_recover: 0
>> tx29_cqes: 2
>> tx29_wake: 0
>> tx29_cqe_err: 0
>> tx30_packets: 0
>> tx30_bytes: 0
>> tx30_tso_packets: 0
>> tx30_tso_bytes: 0
>> tx30_tso_inner_packets: 0
>> tx30_tso_inner_bytes: 0
>> tx30_csum_partial: 0
>> tx30_csum_partial_inner: 0
>> tx30_added_vlan_packets: 0
>> tx30_nop: 0
>> tx30_csum_none: 0
>> tx30_stopped: 0
>> tx30_dropped: 0
>> tx30_xmit_more: 0
>> tx30_recover: 0
>> tx30_cqes: 0
>> tx30_wake: 0
>> tx30_cqe_err: 0
>> tx31_packets: 0
>> tx31_bytes: 0
>> tx31_tso_packets: 0
>> tx31_tso_bytes: 0
>> tx31_tso_inner_packets: 0
>> tx31_tso_inner_bytes: 0
>> tx31_csum_partial: 0
>> tx31_csum_partial_inner: 0
>> tx31_added_vlan_packets: 0
>> tx31_nop: 0
>> tx31_csum_none: 0
>> tx31_stopped: 0
>> tx31_dropped: 0
>> tx31_xmit_more: 0
>> tx31_recover: 0
>> tx31_cqes: 0
>> tx31_wake: 0
>> tx31_cqe_err: 0
>> tx32_packets: 0
>> tx32_bytes: 0
>> tx32_tso_packets: 0
>> tx32_tso_bytes: 0
>> tx32_tso_inner_packets: 0
>> tx32_tso_inner_bytes: 0
>> tx32_csum_partial: 0
>> tx32_csum_partial_inner: 0
>> tx32_added_vlan_packets: 0
>> tx32_nop: 0
>> tx32_csum_none: 0
>> tx32_stopped: 0
>> tx32_dropped: 0
>> tx32_xmit_more: 0
>> tx32_recover: 0
>> tx32_cqes: 0
>> tx32_wake: 0
>> tx32_cqe_err: 0
>> tx33_packets: 0
>> tx33_bytes: 0
>> tx33_tso_packets: 0
>> tx33_tso_bytes: 0
>> tx33_tso_inner_packets: 0
>> tx33_tso_inner_bytes: 0
>> tx33_csum_partial: 0
>> tx33_csum_partial_inner: 0
>> tx33_added_vlan_packets: 0
>> tx33_nop: 0
>> tx33_csum_none: 0
>> tx33_stopped: 0
>> tx33_dropped: 0
>> tx33_xmit_more: 0
>> tx33_recover: 0
>> tx33_cqes: 0
>> tx33_wake: 0
>> tx33_cqe_err: 0
>> tx34_packets: 0
>> tx34_bytes: 0
>> tx34_tso_packets: 0
>> tx34_tso_bytes: 0
>> tx34_tso_inner_packets: 0
>> tx34_tso_inner_bytes: 0
>> tx34_csum_partial: 0
>> tx34_csum_partial_inner: 0
>> tx34_added_vlan_packets: 0
>> tx34_nop: 0
>> tx34_csum_none: 0
>> tx34_stopped: 0
>> tx34_dropped: 0
>> tx34_xmit_more: 0
>> tx34_recover: 0
>> tx34_cqes: 0
>> tx34_wake: 0
>> tx34_cqe_err: 0
>> tx35_packets: 0
>> tx35_bytes: 0
>> tx35_tso_packets: 0
>> tx35_tso_bytes: 0
>> tx35_tso_inner_packets: 0
>> tx35_tso_inner_bytes: 0
>> tx35_csum_partial: 0
>> tx35_csum_partial_inner: 0
>> tx35_added_vlan_packets: 0
>> tx35_nop: 0
>> tx35_csum_none: 0
>> tx35_stopped: 0
>> tx35_dropped: 0
>> tx35_xmit_more: 0
>> tx35_recover: 0
>> tx35_cqes: 0
>> tx35_wake: 0
>> tx35_cqe_err: 0
>> tx36_packets: 0
>> tx36_bytes: 0
>> tx36_tso_packets: 0
>> tx36_tso_bytes: 0
>> tx36_tso_inner_packets: 0
>> tx36_tso_inner_bytes: 0
>> tx36_csum_partial: 0
>> tx36_csum_partial_inner: 0
>> tx36_added_vlan_packets: 0
>> tx36_nop: 0
>> tx36_csum_none: 0
>> tx36_stopped: 0
>> tx36_dropped: 0
>> tx36_xmit_more: 0
>> tx36_recover: 0
>> tx36_cqes: 0
>> tx36_wake: 0
>> tx36_cqe_err: 0
>> tx37_packets: 0
>> tx37_bytes: 0
>> tx37_tso_packets: 0
>> tx37_tso_bytes: 0
>> tx37_tso_inner_packets: 0
>> tx37_tso_inner_bytes: 0
>> tx37_csum_partial: 0
>> tx37_csum_partial_inner: 0
>> tx37_added_vlan_packets: 0
>> tx37_nop: 0
>> tx37_csum_none: 0
>> tx37_stopped: 0
>> tx37_dropped: 0
>> tx37_xmit_more: 0
>> tx37_recover: 0
>> tx37_cqes: 0
>> tx37_wake: 0
>> tx37_cqe_err: 0
>> tx38_packets: 0
>> tx38_bytes: 0
>> tx38_tso_packets: 0
>> tx38_tso_bytes: 0
>> tx38_tso_inner_packets: 0
>> tx38_tso_inner_bytes: 0
>> tx38_csum_partial: 0
>> tx38_csum_partial_inner: 0
>> tx38_added_vlan_packets: 0
>> tx38_nop: 0
>> tx38_csum_none: 0
>> tx38_stopped: 0
>> tx38_dropped: 0
>> tx38_xmit_more: 0
>> tx38_recover: 0
>> tx38_cqes: 0
>> tx38_wake: 0
>> tx38_cqe_err: 0
>> tx39_packets: 0
>> tx39_bytes: 0
>> tx39_tso_packets: 0
>> tx39_tso_bytes: 0
>> tx39_tso_inner_packets: 0
>> tx39_tso_inner_bytes: 0
>> tx39_csum_partial: 0
>> tx39_csum_partial_inner: 0
>> tx39_added_vlan_packets: 0
>> tx39_nop: 0
>> tx39_csum_none: 0
>> tx39_stopped: 0
>> tx39_dropped: 0
>> tx39_xmit_more: 0
>> tx39_recover: 0
>> tx39_cqes: 0
>> tx39_wake: 0
>> tx39_cqe_err: 0
>> tx40_packets: 0
>> tx40_bytes: 0
>> tx40_tso_packets: 0
>> tx40_tso_bytes: 0
>> tx40_tso_inner_packets: 0
>> tx40_tso_inner_bytes: 0
>> tx40_csum_partial: 0
>> tx40_csum_partial_inner: 0
>> tx40_added_vlan_packets: 0
>> tx40_nop: 0
>> tx40_csum_none: 0
>> tx40_stopped: 0
>> tx40_dropped: 0
>> tx40_xmit_more: 0
>> tx40_recover: 0
>> tx40_cqes: 0
>> tx40_wake: 0
>> tx40_cqe_err: 0
>> tx41_packets: 0
>> tx41_bytes: 0
>> tx41_tso_packets: 0
>> tx41_tso_bytes: 0
>> tx41_tso_inner_packets: 0
>> tx41_tso_inner_bytes: 0
>> tx41_csum_partial: 0
>> tx41_csum_partial_inner: 0
>> tx41_added_vlan_packets: 0
>> tx41_nop: 0
>> tx41_csum_none: 0
>> tx41_stopped: 0
>> tx41_dropped: 0
>> tx41_xmit_more: 0
>> tx41_recover: 0
>> tx41_cqes: 0
>> tx41_wake: 0
>> tx41_cqe_err: 0
>> tx42_packets: 0
>> tx42_bytes: 0
>> tx42_tso_packets: 0
>> tx42_tso_bytes: 0
>> tx42_tso_inner_packets: 0
>> tx42_tso_inner_bytes: 0
>> tx42_csum_partial: 0
>> tx42_csum_partial_inner: 0
>> tx42_added_vlan_packets: 0
>> tx42_nop: 0
>> tx42_csum_none: 0
>> tx42_stopped: 0
>> tx42_dropped: 0
>> tx42_xmit_more: 0
>> tx42_recover: 0
>> tx42_cqes: 0
>> tx42_wake: 0
>> tx42_cqe_err: 0
>> tx43_packets: 0
>> tx43_bytes: 0
>> tx43_tso_packets: 0
>> tx43_tso_bytes: 0
>> tx43_tso_inner_packets: 0
>> tx43_tso_inner_bytes: 0
>> tx43_csum_partial: 0
>> tx43_csum_partial_inner: 0
>> tx43_added_vlan_packets: 0
>> tx43_nop: 0
>> tx43_csum_none: 0
>> tx43_stopped: 0
>> tx43_dropped: 0
>> tx43_xmit_more: 0
>> tx43_recover: 0
>> tx43_cqes: 0
>> tx43_wake: 0
>> tx43_cqe_err: 0
>> tx44_packets: 0
>> tx44_bytes: 0
>> tx44_tso_packets: 0
>> tx44_tso_bytes: 0
>> tx44_tso_inner_packets: 0
>> tx44_tso_inner_bytes: 0
>> tx44_csum_partial: 0
>> tx44_csum_partial_inner: 0
>> tx44_added_vlan_packets: 0
>> tx44_nop: 0
>> tx44_csum_none: 0
>> tx44_stopped: 0
>> tx44_dropped: 0
>> tx44_xmit_more: 0
>> tx44_recover: 0
>> tx44_cqes: 0
>> tx44_wake: 0
>> tx44_cqe_err: 0
>> tx45_packets: 0
>> tx45_bytes: 0
>> tx45_tso_packets: 0
>> tx45_tso_bytes: 0
>> tx45_tso_inner_packets: 0
>> tx45_tso_inner_bytes: 0
>> tx45_csum_partial: 0
>> tx45_csum_partial_inner: 0
>> tx45_added_vlan_packets: 0
>> tx45_nop: 0
>> tx45_csum_none: 0
>> tx45_stopped: 0
>> tx45_dropped: 0
>> tx45_xmit_more: 0
>> tx45_recover: 0
>> tx45_cqes: 0
>> tx45_wake: 0
>> tx45_cqe_err: 0
>> tx46_packets: 0
>> tx46_bytes: 0
>> tx46_tso_packets: 0
>> tx46_tso_bytes: 0
>> tx46_tso_inner_packets: 0
>> tx46_tso_inner_bytes: 0
>> tx46_csum_partial: 0
>> tx46_csum_partial_inner: 0
>> tx46_added_vlan_packets: 0
>> tx46_nop: 0
>> tx46_csum_none: 0
>> tx46_stopped: 0
>> tx46_dropped: 0
>> tx46_xmit_more: 0
>> tx46_recover: 0
>> tx46_cqes: 0
>> tx46_wake: 0
>> tx46_cqe_err: 0
>> tx47_packets: 0
>> tx47_bytes: 0
>> tx47_tso_packets: 0
>> tx47_tso_bytes: 0
>> tx47_tso_inner_packets: 0
>> tx47_tso_inner_bytes: 0
>> tx47_csum_partial: 0
>> tx47_csum_partial_inner: 0
>> tx47_added_vlan_packets: 0
>> tx47_nop: 0
>> tx47_csum_none: 0
>> tx47_stopped: 0
>> tx47_dropped: 0
>> tx47_xmit_more: 0
>> tx47_recover: 0
>> tx47_cqes: 0
>> tx47_wake: 0
>> tx47_cqe_err: 0
>> tx48_packets: 0
>> tx48_bytes: 0
>> tx48_tso_packets: 0
>> tx48_tso_bytes: 0
>> tx48_tso_inner_packets: 0
>> tx48_tso_inner_bytes: 0
>> tx48_csum_partial: 0
>> tx48_csum_partial_inner: 0
>> tx48_added_vlan_packets: 0
>> tx48_nop: 0
>> tx48_csum_none: 0
>> tx48_stopped: 0
>> tx48_dropped: 0
>> tx48_xmit_more: 0
>> tx48_recover: 0
>> tx48_cqes: 0
>> tx48_wake: 0
>> tx48_cqe_err: 0
>> tx49_packets: 0
>> tx49_bytes: 0
>> tx49_tso_packets: 0
>> tx49_tso_bytes: 0
>> tx49_tso_inner_packets: 0
>> tx49_tso_inner_bytes: 0
>> tx49_csum_partial: 0
>> tx49_csum_partial_inner: 0
>> tx49_added_vlan_packets: 0
>> tx49_nop: 0
>> tx49_csum_none: 0
>> tx49_stopped: 0
>> tx49_dropped: 0
>> tx49_xmit_more: 0
>> tx49_recover: 0
>> tx49_cqes: 0
>> tx49_wake: 0
>> tx49_cqe_err: 0
>> tx50_packets: 0
>> tx50_bytes: 0
>> tx50_tso_packets: 0
>> tx50_tso_bytes: 0
>> tx50_tso_inner_packets: 0
>> tx50_tso_inner_bytes: 0
>> tx50_csum_partial: 0
>> tx50_csum_partial_inner: 0
>> tx50_added_vlan_packets: 0
>> tx50_nop: 0
>> tx50_csum_none: 0
>> tx50_stopped: 0
>> tx50_dropped: 0
>> tx50_xmit_more: 0
>> tx50_recover: 0
>> tx50_cqes: 0
>> tx50_wake: 0
>> tx50_cqe_err: 0
>> tx51_packets: 0
>> tx51_bytes: 0
>> tx51_tso_packets: 0
>> tx51_tso_bytes: 0
>> tx51_tso_inner_packets: 0
>> tx51_tso_inner_bytes: 0
>> tx51_csum_partial: 0
>> tx51_csum_partial_inner: 0
>> tx51_added_vlan_packets: 0
>> tx51_nop: 0
>> tx51_csum_none: 0
>> tx51_stopped: 0
>> tx51_dropped: 0
>> tx51_xmit_more: 0
>> tx51_recover: 0
>> tx51_cqes: 0
>> tx51_wake: 0
>> tx51_cqe_err: 0
>> tx52_packets: 0
>> tx52_bytes: 0
>> tx52_tso_packets: 0
>> tx52_tso_bytes: 0
>> tx52_tso_inner_packets: 0
>> tx52_tso_inner_bytes: 0
>> tx52_csum_partial: 0
>> tx52_csum_partial_inner: 0
>> tx52_added_vlan_packets: 0
>> tx52_nop: 0
>> tx52_csum_none: 0
>> tx52_stopped: 0
>> tx52_dropped: 0
>> tx52_xmit_more: 0
>> tx52_recover: 0
>> tx52_cqes: 0
>> tx52_wake: 0
>> tx52_cqe_err: 0
>> tx53_packets: 0
>> tx53_bytes: 0
>> tx53_tso_packets: 0
>> tx53_tso_bytes: 0
>> tx53_tso_inner_packets: 0
>> tx53_tso_inner_bytes: 0
>> tx53_csum_partial: 0
>> tx53_csum_partial_inner: 0
>> tx53_added_vlan_packets: 0
>> tx53_nop: 0
>> tx53_csum_none: 0
>> tx53_stopped: 0
>> tx53_dropped: 0
>> tx53_xmit_more: 0
>> tx53_recover: 0
>> tx53_cqes: 0
>> tx53_wake: 0
>> tx53_cqe_err: 0
>> tx54_packets: 0
>> tx54_bytes: 0
>> tx54_tso_packets: 0
>> tx54_tso_bytes: 0
>> tx54_tso_inner_packets: 0
>> tx54_tso_inner_bytes: 0
>> tx54_csum_partial: 0
>> tx54_csum_partial_inner: 0
>> tx54_added_vlan_packets: 0
>> tx54_nop: 0
>> tx54_csum_none: 0
>> tx54_stopped: 0
>> tx54_dropped: 0
>> tx54_xmit_more: 0
>> tx54_recover: 0
>> tx54_cqes: 0
>> tx54_wake: 0
>> tx54_cqe_err: 0
>> tx55_packets: 0
>> tx55_bytes: 0
>> tx55_tso_packets: 0
>> tx55_tso_bytes: 0
>> tx55_tso_inner_packets: 0
>> tx55_tso_inner_bytes: 0
>> tx55_csum_partial: 0
>> tx55_csum_partial_inner: 0
>> tx55_added_vlan_packets: 0
>> tx55_nop: 0
>> tx55_csum_none: 0
>> tx55_stopped: 0
>> tx55_dropped: 0
>> tx55_xmit_more: 0
>> tx55_recover: 0
>> tx55_cqes: 0
>> tx55_wake: 0
>> tx55_cqe_err: 0
>> tx0_xdp_xmit: 0
>> tx0_xdp_full: 0
>> tx0_xdp_err: 0
>> tx0_xdp_cqes: 0
>> tx1_xdp_xmit: 0
>> tx1_xdp_full: 0
>> tx1_xdp_err: 0
>> tx1_xdp_cqes: 0
>> tx2_xdp_xmit: 0
>> tx2_xdp_full: 0
>> tx2_xdp_err: 0
>> tx2_xdp_cqes: 0
>> tx3_xdp_xmit: 0
>> tx3_xdp_full: 0
>> tx3_xdp_err: 0
>> tx3_xdp_cqes: 0
>> tx4_xdp_xmit: 0
>> tx4_xdp_full: 0
>> tx4_xdp_err: 0
>> tx4_xdp_cqes: 0
>> tx5_xdp_xmit: 0
>> tx5_xdp_full: 0
>> tx5_xdp_err: 0
>> tx5_xdp_cqes: 0
>> tx6_xdp_xmit: 0
>> tx6_xdp_full: 0
>> tx6_xdp_err: 0
>> tx6_xdp_cqes: 0
>> tx7_xdp_xmit: 0
>> tx7_xdp_full: 0
>> tx7_xdp_err: 0
>> tx7_xdp_cqes: 0
>> tx8_xdp_xmit: 0
>> tx8_xdp_full: 0
>> tx8_xdp_err: 0
>> tx8_xdp_cqes: 0
>> tx9_xdp_xmit: 0
>> tx9_xdp_full: 0
>> tx9_xdp_err: 0
>> tx9_xdp_cqes: 0
>> tx10_xdp_xmit: 0
>> tx10_xdp_full: 0
>> tx10_xdp_err: 0
>> tx10_xdp_cqes: 0
>> tx11_xdp_xmit: 0
>> tx11_xdp_full: 0
>> tx11_xdp_err: 0
>> tx11_xdp_cqes: 0
>> tx12_xdp_xmit: 0
>> tx12_xdp_full: 0
>> tx12_xdp_err: 0
>> tx12_xdp_cqes: 0
>> tx13_xdp_xmit: 0
>> tx13_xdp_full: 0
>> tx13_xdp_err: 0
>> tx13_xdp_cqes: 0
>> tx14_xdp_xmit: 0
>> tx14_xdp_full: 0
>> tx14_xdp_err: 0
>> tx14_xdp_cqes: 0
>> tx15_xdp_xmit: 0
>> tx15_xdp_full: 0
>> tx15_xdp_err: 0
>> tx15_xdp_cqes: 0
>> tx16_xdp_xmit: 0
>> tx16_xdp_full: 0
>> tx16_xdp_err: 0
>> tx16_xdp_cqes: 0
>> tx17_xdp_xmit: 0
>> tx17_xdp_full: 0
>> tx17_xdp_err: 0
>> tx17_xdp_cqes: 0
>> tx18_xdp_xmit: 0
>> tx18_xdp_full: 0
>> tx18_xdp_err: 0
>> tx18_xdp_cqes: 0
>> tx19_xdp_xmit: 0
>> tx19_xdp_full: 0
>> tx19_xdp_err: 0
>> tx19_xdp_cqes: 0
>> tx20_xdp_xmit: 0
>> tx20_xdp_full: 0
>> tx20_xdp_err: 0
>> tx20_xdp_cqes: 0
>> tx21_xdp_xmit: 0
>> tx21_xdp_full: 0
>> tx21_xdp_err: 0
>> tx21_xdp_cqes: 0
>> tx22_xdp_xmit: 0
>> tx22_xdp_full: 0
>> tx22_xdp_err: 0
>> tx22_xdp_cqes: 0
>> tx23_xdp_xmit: 0
>> tx23_xdp_full: 0
>> tx23_xdp_err: 0
>> tx23_xdp_cqes: 0
>> tx24_xdp_xmit: 0
>> tx24_xdp_full: 0
>> tx24_xdp_err: 0
>> tx24_xdp_cqes: 0
>> tx25_xdp_xmit: 0
>> tx25_xdp_full: 0
>> tx25_xdp_err: 0
>> tx25_xdp_cqes: 0
>> tx26_xdp_xmit: 0
>> tx26_xdp_full: 0
>> tx26_xdp_err: 0
>> tx26_xdp_cqes: 0
>> tx27_xdp_xmit: 0
>> tx27_xdp_full: 0
>> tx27_xdp_err: 0
>> tx27_xdp_cqes: 0
>> tx28_xdp_xmit: 0
>> tx28_xdp_full: 0
>> tx28_xdp_err: 0
>> tx28_xdp_cqes: 0
>> tx29_xdp_xmit: 0
>> tx29_xdp_full: 0
>> tx29_xdp_err: 0
>> tx29_xdp_cqes: 0
>> tx30_xdp_xmit: 0
>> tx30_xdp_full: 0
>> tx30_xdp_err: 0
>> tx30_xdp_cqes: 0
>> tx31_xdp_xmit: 0
>> tx31_xdp_full: 0
>> tx31_xdp_err: 0
>> tx31_xdp_cqes: 0
>> tx32_xdp_xmit: 0
>> tx32_xdp_full: 0
>> tx32_xdp_err: 0
>> tx32_xdp_cqes: 0
>> tx33_xdp_xmit: 0
>> tx33_xdp_full: 0
>> tx33_xdp_err: 0
>> tx33_xdp_cqes: 0
>> tx34_xdp_xmit: 0
>> tx34_xdp_full: 0
>> tx34_xdp_err: 0
>> tx34_xdp_cqes: 0
>> tx35_xdp_xmit: 0
>> tx35_xdp_full: 0
>> tx35_xdp_err: 0
>> tx35_xdp_cqes: 0
>> tx36_xdp_xmit: 0
>> tx36_xdp_full: 0
>> tx36_xdp_err: 0
>> tx36_xdp_cqes: 0
>> tx37_xdp_xmit: 0
>> tx37_xdp_full: 0
>> tx37_xdp_err: 0
>> tx37_xdp_cqes: 0
>> tx38_xdp_xmit: 0
>> tx38_xdp_full: 0
>> tx38_xdp_err: 0
>> tx38_xdp_cqes: 0
>> tx39_xdp_xmit: 0
>> tx39_xdp_full: 0
>> tx39_xdp_err: 0
>> tx39_xdp_cqes: 0
>> tx40_xdp_xmit: 0
>> tx40_xdp_full: 0
>> tx40_xdp_err: 0
>> tx40_xdp_cqes: 0
>> tx41_xdp_xmit: 0
>> tx41_xdp_full: 0
>> tx41_xdp_err: 0
>> tx41_xdp_cqes: 0
>> tx42_xdp_xmit: 0
>> tx42_xdp_full: 0
>> tx42_xdp_err: 0
>> tx42_xdp_cqes: 0
>> tx43_xdp_xmit: 0
>> tx43_xdp_full: 0
>> tx43_xdp_err: 0
>> tx43_xdp_cqes: 0
>> tx44_xdp_xmit: 0
>> tx44_xdp_full: 0
>> tx44_xdp_err: 0
>> tx44_xdp_cqes: 0
>> tx45_xdp_xmit: 0
>> tx45_xdp_full: 0
>> tx45_xdp_err: 0
>> tx45_xdp_cqes: 0
>> tx46_xdp_xmit: 0
>> tx46_xdp_full: 0
>> tx46_xdp_err: 0
>> tx46_xdp_cqes: 0
>> tx47_xdp_xmit: 0
>> tx47_xdp_full: 0
>> tx47_xdp_err: 0
>> tx47_xdp_cqes: 0
>> tx48_xdp_xmit: 0
>> tx48_xdp_full: 0
>> tx48_xdp_err: 0
>> tx48_xdp_cqes: 0
>> tx49_xdp_xmit: 0
>> tx49_xdp_full: 0
>> tx49_xdp_err: 0
>> tx49_xdp_cqes: 0
>> tx50_xdp_xmit: 0
>> tx50_xdp_full: 0
>> tx50_xdp_err: 0
>> tx50_xdp_cqes: 0
>> tx51_xdp_xmit: 0
>> tx51_xdp_full: 0
>> tx51_xdp_err: 0
>> tx51_xdp_cqes: 0
>> tx52_xdp_xmit: 0
>> tx52_xdp_full: 0
>> tx52_xdp_err: 0
>> tx52_xdp_cqes: 0
>> tx53_xdp_xmit: 0
>> tx53_xdp_full: 0
>> tx53_xdp_err: 0
>> tx53_xdp_cqes: 0
>> tx54_xdp_xmit: 0
>> tx54_xdp_full: 0
>> tx54_xdp_err: 0
>> tx54_xdp_cqes: 0
>> tx55_xdp_xmit: 0
>> tx55_xdp_full: 0
>> tx55_xdp_err: 0
>> tx55_xdp_cqes: 0
>>
>>
>> mpstat -P ALL 1 10
>> Average: CPU %usr %nice %sys %iowait %irq %soft
>> %steal
>> %guest %gnice %idle
>> Average: all 0.04 0.00 6.94 0.02 0.00 32.00
>> 0.00 0.00 0.00 61.00
>> Average: 0 0.00 0.00 1.20 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 98.80
>> Average: 1 0.00 0.00 2.30 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 97.70
>> Average: 2 0.10 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.90
>> Average: 3 0.10 0.00 1.50 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 98.40
>> Average: 4 0.50 0.00 2.50 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 97.00
>> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 6 0.90 0.00 10.20 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 88.90
>> Average: 7 0.00 0.00 0.00 1.40 0.00 0.00
>> 0.00
>> 0.00 0.00 98.60
>> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 11 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 14 0.00 0.00 12.99 0.00 0.00 62.64
>> 0.00 0.00 0.00 24.38
>> Average: 15 0.00 0.00 12.70 0.00 0.00 63.40
>> 0.00 0.00 0.00 23.90
>> Average: 16 0.00 0.00 11.20 0.00 0.00 66.40
>> 0.00 0.00 0.00 22.40
>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>> 0.00 0.00 0.00 31.30
>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>> 0.00 0.00 0.00 24.90
>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>> 0.00 0.00 0.00 19.68
>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>> 0.00 0.00 0.00 18.00
>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>> 0.00 0.00 0.00 17.40
>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>> 0.00 0.00 0.00 26.03
>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>> 0.00 0.00 0.00 17.52
>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>> 0.00 0.00 0.00 18.20
>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>> 0.00 0.00 0.00 17.68
>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>> 0.00 0.00 0.00 18.20
>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>> 0.00 0.00 0.00 20.42
> The numa cores are not at 100% util, you have around 20% of idle on
> each one.
Yes - no 100% cpu - but the difference between 80% and 100% is like push
aditional 1-2Gbit/s
>
>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 96.10
>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.70
>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.50
>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 97.40
>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.10
>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.40
>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>> 0.00 0.00 0.00 19.42
>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>> 0.00 0.00 0.00 26.60
>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>> 0.00 0.00 0.00 21.60
>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>> 0.00 0.00 0.00 20.50
>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>> 0.00 0.00 0.00 21.60
>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>> 0.00 0.00 0.00 24.58
>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>> 0.00 0.00 0.00 24.68
>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>> 0.00 0.00 0.00 28.34
>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>> 0.00 0.00 0.00 25.83
>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>> 0.00 0.00 0.00 27.20
>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>> 0.00 0.00 0.00 28.03
>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>> 0.00 0.00 0.00 24.70
>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>> 0.00 0.00 0.00 25.30
>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>> 0.00 0.00 0.00 26.77
>>
>>
>> ethtool -k enp175s0f0
>> Features for enp175s0f0:
>> rx-checksumming: on
>> tx-checksumming: on
>> tx-checksum-ipv4: on
>> tx-checksum-ip-generic: off [fixed]
>> tx-checksum-ipv6: on
>> tx-checksum-fcoe-crc: off [fixed]
>> tx-checksum-sctp: off [fixed]
>> scatter-gather: on
>> tx-scatter-gather: on
>> tx-scatter-gather-fraglist: off [fixed]
>> tcp-segmentation-offload: on
>> tx-tcp-segmentation: on
>> tx-tcp-ecn-segmentation: off [fixed]
>> tx-tcp-mangleid-segmentation: off
>> tx-tcp6-segmentation: on
>> udp-fragmentation-offload: off
>> generic-segmentation-offload: on
>> generic-receive-offload: on
>> large-receive-offload: off [fixed]
>> rx-vlan-offload: on
>> tx-vlan-offload: on
>> ntuple-filters: off
>> receive-hashing: on
>> highdma: on [fixed]
>> rx-vlan-filter: on
>> vlan-challenged: off [fixed]
>> tx-lockless: off [fixed]
>> netns-local: off [fixed]
>> tx-gso-robust: off [fixed]
>> tx-fcoe-segmentation: off [fixed]
>> tx-gre-segmentation: on
>> tx-gre-csum-segmentation: on
>> tx-ipxip4-segmentation: off [fixed]
>> tx-ipxip6-segmentation: off [fixed]
>> tx-udp_tnl-segmentation: on
>> tx-udp_tnl-csum-segmentation: on
>> tx-gso-partial: on
>> tx-sctp-segmentation: off [fixed]
>> tx-esp-segmentation: off [fixed]
>> tx-udp-segmentation: on
>> fcoe-mtu: off [fixed]
>> tx-nocache-copy: off
>> loopback: off [fixed]
>> rx-fcs: off
>> rx-all: off
>> tx-vlan-stag-hw-insert: on
>> rx-vlan-stag-hw-parse: off [fixed]
>> rx-vlan-stag-filter: on [fixed]
>> l2-fwd-offload: off [fixed]
>> hw-tc-offload: off
>> esp-hw-offload: off [fixed]
>> esp-tx-csum-hw-offload: off [fixed]
>> rx-udp_tunnel-port-offload: on
>> tls-hw-tx-offload: off [fixed]
>> tls-hw-rx-offload: off [fixed]
>> rx-gro-hw: off [fixed]
>> tls-hw-record: off [fixed]
>>
>> ethtool -c enp175s0f0
>> Coalesce parameters for enp175s0f0:
>> Adaptive RX: off TX: on
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>> dmac: 32703
>>
>> rx-usecs: 256
>> rx-frames: 128
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 8
>> tx-frames: 128
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
>> rx-usecs-low: 0
>> rx-frame-low: 0
>> tx-usecs-low: 0
>> tx-frame-low: 0
>>
>> rx-usecs-high: 0
>> rx-frame-high: 0
>> tx-usecs-high: 0
>> tx-frame-high: 0
>>
>> ethtool -g enp175s0f0
>> Ring parameters for enp175s0f0:
>> Pre-set maximums:
>> RX: 8192
>> RX Mini: 0
>> RX Jumbo: 0
>> TX: 8192
>> Current hardware settings:
>> RX: 4096
>> RX Mini: 0
>> RX Jumbo: 0
>> TX: 4096
>>
>>
>>
>>
>>
>>
Also changed a little coalesce params - and best for this config are:
ethtool -c enp175s0f0
Coalesce parameters for enp175s0f0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32573
rx-usecs: 40
rx-frames: 128
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 8
tx-frames: 8
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
Less drops on RX side - and more pps in overall forwarded.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 10:55 ` Jesper Dangaard Brouer
@ 2018-11-01 13:52 ` Paweł Staszewski
2018-11-01 17:23 ` David Ahern
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 13:52 UTC (permalink / raw)
To: Jesper Dangaard Brouer, David Ahern; +Cc: netdev, Yoel Caspersen
W dniu 01.11.2018 o 11:55, Jesper Dangaard Brouer pisze:
> On Wed, 31 Oct 2018 21:37:16 -0600 David Ahern <dsahern@gmail.com> wrote:
>
>> This is mainly a forwarding use case? Seems so based on the perf report.
>> I suspect forwarding with XDP would show pretty good improvement.
> Yes, significant performance improvements.
>
> Notice Davids talk: "Leveraging Kernel Tables with XDP"
> http://vger.kernel.org/lpc-networking2018.html#session-1
It will be rly interesting
> It looks like that you are doing "pure" IP-routing, without any
> iptables conntrack stuff (from your perf report data). That will
> actually be a really good use-case for accelerating this with XDP.
Yes pure IP routing
iptables used only for some local input filtering.
>
> I want you to understand the philosophy behind how David and I want
> people to leverage XDP. Think of XDP as a software offload layer for
> the kernel network stack. Setup and use Linux kernel network stack, but
> accelerate parts of it with XDP, e.g. the route FIB lookup.
>
> Sample code avail here:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/xdp_fwd_kern.c
I can try some tests on same hw but testlab configuration - will give it
a try :)
> (I do warn, what we just found a bug/crash in setup+tairdown for the
> mlx5 driver you are using, that we/mlx _will_ fix soon)
Ok
>
>
>> You need the vlan changes I have queued up though.
> I know Yoel will be very interested in those changes too! I've
> convinced Yoel to write an XDP program for his Border Network Gateway
> (BNG) production system[1], and his is a heavy VLAN user. And the plan
> is to Open Source this when he have-something-working.
>
> [1] https://www.version2.dk/blog/software-router-del-5-linux-bng-1086060
Ok - for now i need to split traffic into two separate 100G ports placed
in two different x16 pciexpress slots to check if the problem is mainly
caused by no more pciex x16 bandwidth available.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 9:22 ` Jesper Dangaard Brouer
2018-11-01 10:34 ` Paweł Staszewski
@ 2018-11-01 15:27 ` Aaron Lu
2018-11-01 20:23 ` Saeed Mahameed
1 sibling, 1 reply; 77+ messages in thread
From: Aaron Lu @ 2018-11-01 15:27 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Paweł Staszewski, Eric Dumazet, netdev, Tariq Toukan,
Ilias Apalodimas, Yoel Caspersen, Mel Gorman
On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer wrote:
... ...
> Section copied out:
>
> mlx5e_poll_tx_cq
> |
> --16.34%--napi_consume_skb
> |
> |--12.65%--__free_pages_ok
> | |
> | --11.86%--free_one_page
> | |
> | |--10.10%--queued_spin_lock_slowpath
> | |
> | --0.65%--_raw_spin_lock
This callchain looks like it is freeing higher order pages than order 0:
__free_pages_ok is only called for pages whose order are bigger than 0.
> |
> |--1.55%--page_frag_free
> |
> --1.44%--skb_release_data
>
>
> Let me explain what (I think) happens. The mlx5 driver RX-page recycle
> mechanism is not effective in this workload, and pages have to go
> through the page allocator. The lock contention happens during mlx5
> DMA TX completion cycle. And the page allocator cannot keep up at
> these speeds.
>
> One solution is extend page allocator with a bulk free API. (This have
> been on my TODO list for a long time, but I don't have a
> micro-benchmark that trick the driver page-recycle to fail). It should
> fit nicely, as I can see that kmem_cache_free_bulk() does get
> activated (bulk freeing SKBs), which means that DMA TX completion do
> have a bulk of packets.
>
> We can (and should) also improve the page recycle scheme in the driver.
> After LPC, I have a project with Tariq and Ilias (Cc'ed) to improve the
> page_pool, and we will (attempt) to generalize this, for both high-end
> mlx5 and more low-end ARM64-boards (macchiatobin and espressobin).
>
> The MM-people is in parallel working to improve the performance of
> order-0 page returns. Thus, the explicit page bulk free API might
> actually become less important. I actually think (Cc.) Aaron have a
> patchset he would like you to test, which removes the (zone->)lock
> you hit in free_one_page().
Thanks Jesper.
Yes, the said patchset is in this branch:
https://github.com/aaronlu/linux no_merge_cluster_alloc_4.19-rc5
But as I said above, I think the lock contention here is for
order > 0 pages so my current patchset will not work here, unfortunately.
BTW, Mel Gorman has suggested an alternative way to improve page
allocator's scalability and I'm working on it right now, it will
improve page allocator's scalability for all order pages. I might be
able to post it some time next week, will CC all of you when it's ready.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 11:09 ` Paweł Staszewski
@ 2018-11-01 16:49 ` Paweł Staszewski
2018-11-01 20:37 ` Saeed Mahameed
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 16:49 UTC (permalink / raw)
To: Saeed Mahameed, netdev
W dniu 01.11.2018 o 12:09, Paweł Staszewski pisze:
>>> rx_cqe_compress_pkts: 0
>> If this is a pcie bottleneck it might be useful to enable CQE
>> compression (to reduce PCIe completion descriptors transactions)
>> you should see the above rx_cqe_compress_pkts increasing when enabled.
>>
>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>> $ ethtool --show-priv-flags enp175s0f1
>> Private flags for p6p1:
>> rx_cqe_moder : on
>> cqe_moder : off
>> rx_cqe_compress : on
>> ...
>>
>> try this on both interfaces.
> Done
> ethtool --show-priv-flags enp175s0f1
> Private flags for enp175s0f1:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : on
> rx_striding_rq : off
> rx_no_csum_complete: off
>
> ethtool --show-priv-flags enp175s0f0
> Private flags for enp175s0f0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : on
> rx_striding_rq : off
> rx_no_csum_complete: off
Enabling cqe compress changes nothing after reaching 64Gbit RX /
64Gbit/s TX on interfaces cpu's are saturated at 100%
ethtool -S enp175s0f1 | grep rx_cqe_compress
rx_cqe_compress_blks: 5657836379
rx_cqe_compress_pkts: 13153761080
ethtool -S enp175s0f0 | grep rx_cqe_compress
rx_cqe_compress_blks: 5994612500
rx_cqe_compress_pkts: 13579014869
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
- iface Rx Tx Total
==============================================================================
enp175s0f1: 27.03 Gb/s 37.09 Gb/s
64.12 Gb/s
enp175s0f0: 36.84 Gb/s 26.82 Gb/s
63.66 Gb/s
------------------------------------------------------------------------------
total: 63.85 Gb/s 63.87 Gb/s 127.72 Gb/s
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
/ iface Rx Tx Total
==============================================================================
enp175s0f1: 3.22 GB/s 4.26 GB/s
7.48 GB/s
enp175s0f0: 4.24 GB/s 3.21 GB/s
7.45 GB/s
------------------------------------------------------------------------------
total: 7.46 GB/s 7.47 GB/s
14.93 GB/s
mpstat
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.05 0.00 0.19 0.02 0.00 42.74 0.00
0.00 0.00 56.99
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 1 0.00 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.70
Average: 2 0.00 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.80
Average: 3 0.00 0.00 0.20 1.20 0.00 0.00 0.00
0.00 0.00 98.60
Average: 4 0.10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 5 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 6 0.10 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.70
Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 12 1.40 0.00 4.50 0.00 0.00 0.00 0.00
0.00 0.00 94.10
Average: 13 0.00 0.00 1.60 0.00 0.00 0.00 0.00
0.00 0.00 98.40
Average: 14 0.00 0.00 0.00 0.00 0.00 84.10 0.00
0.00 0.00 15.90
Average: 15 0.00 0.00 0.10 0.00 0.00 93.70 0.00
0.00 0.00 6.20
Average: 16 0.00 0.00 0.10 0.00 0.00 94.31 0.00
0.00 0.00 5.59
Average: 17 0.00 0.00 0.00 0.00 0.00 95.30 0.00
0.00 0.00 4.70
Average: 18 0.00 0.00 0.00 0.00 0.00 62.80 0.00
0.00 0.00 37.20
Average: 19 0.00 0.00 0.10 0.00 0.00 98.90 0.00
0.00 0.00 1.00
Average: 20 0.00 0.00 0.00 0.00 0.00 99.30 0.00
0.00 0.00 0.70
Average: 21 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 22 0.00 0.00 0.00 0.00 0.00 99.90 0.00
0.00 0.00 0.10
Average: 23 0.00 0.00 0.10 0.00 0.00 99.90 0.00
0.00 0.00 0.00
Average: 24 0.00 0.00 0.10 0.00 0.00 97.10 0.00
0.00 0.00 2.80
Average: 25 0.00 0.00 0.00 0.00 0.00 64.06 0.00
0.00 0.00 35.94
Average: 26 0.00 0.00 0.10 0.00 0.00 88.50 0.00
0.00 0.00 11.40
Average: 27 0.00 0.00 0.00 0.00 0.00 94.10 0.00
0.00 0.00 5.90
Average: 28 0.80 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.20
Average: 29 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 31 0.20 0.00 0.80 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 36 0.20 0.00 0.40 0.00 0.00 0.00 0.00
0.00 0.00 99.40
Average: 37 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 39 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 40 0.10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 41 0.10 0.00 1.20 0.00 0.00 0.00 0.00
0.00 0.00 98.70
Average: 42 0.00 0.00 0.10 0.00 0.00 78.92 0.00
0.00 0.00 20.98
Average: 43 0.00 0.00 0.00 0.00 0.00 81.00 0.00
0.00 0.00 19.00
Average: 44 0.00 0.00 0.00 0.00 0.00 82.58 0.00
0.00 0.00 17.42
Average: 45 0.00 0.00 0.00 0.00 0.00 68.97 0.00
0.00 0.00 31.03
Average: 46 0.00 0.00 0.10 0.00 0.00 79.20 0.00
0.00 0.00 20.70
Average: 47 0.00 0.00 0.00 0.00 0.00 71.33 0.00
0.00 0.00 28.67
Average: 48 0.00 0.00 0.10 0.00 0.00 72.40 0.00
0.00 0.00 27.50
Average: 49 0.00 0.00 0.00 0.00 0.00 90.79 0.00
0.00 0.00 9.21
Average: 50 0.00 0.00 0.10 0.00 0.00 93.20 0.00
0.00 0.00 6.70
Average: 51 0.00 0.00 0.00 0.00 0.00 91.70 0.00
0.00 0.00 8.30
Average: 52 0.00 0.00 0.10 0.00 0.00 79.90 0.00
0.00 0.00 20.00
Average: 53 0.00 0.00 0.00 0.00 0.00 76.20 0.00
0.00 0.00 23.80
Average: 54 0.00 0.00 0.00 0.00 0.00 89.59 0.00
0.00 0.00 10.41
Average: 55 0.00 0.00 0.10 0.00 0.00 65.97 0.00
0.00 0.00 33.93
So yes it looks like pcie x16 limit and pcie is 8/8GB/s both directions
- 16GB/s one direction
So 100Gbit/s network controller need to have pcie x32 :)
I understand in normal server host scenario (no forwarding) most traffic
is outbound or inbound - not many situations where we have 100G input
and 100G output.
I will replace then this 2 port 100G nic with two connect-x 5 100G nic's
installed in two different pcie x16
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 13:52 ` Paweł Staszewski
@ 2018-11-01 17:23 ` David Ahern
2018-11-01 17:30 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: David Ahern @ 2018-11-01 17:23 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer, David Ahern
Cc: netdev, Yoel Caspersen
On 11/1/18 7:52 AM, Paweł Staszewski wrote:
>
>
> W dniu 01.11.2018 o 11:55, Jesper Dangaard Brouer pisze:
>> On Wed, 31 Oct 2018 21:37:16 -0600 David Ahern <dsahern@gmail.com> wrote:
>>
>>> This is mainly a forwarding use case? Seems so based on the perf report.
>>> I suspect forwarding with XDP would show pretty good improvement.
>> Yes, significant performance improvements.
>>
>> Notice Davids talk: "Leveraging Kernel Tables with XDP"
>> http://vger.kernel.org/lpc-networking2018.html#session-1
> It will be rly interesting
It's pushing the exact use case you have: FRR manages the FIB, XDP
programs get access to updates as they happen for fast path forwarding.
>
>> It looks like that you are doing "pure" IP-routing, without any
>> iptables conntrack stuff (from your perf report data). That will
>> actually be a really good use-case for accelerating this with XDP.
> Yes pure IP routing
> iptables used only for some local input filtering.
>
>
>>
>> I want you to understand the philosophy behind how David and I want
>> people to leverage XDP. Think of XDP as a software offload layer for
>> the kernel network stack. Setup and use Linux kernel network stack, but
>> accelerate parts of it with XDP, e.g. the route FIB lookup.
>>
>> Sample code avail here:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/xdp_fwd_kern.c
>>
> I can try some tests on same hw but testlab configuration - will give it
> a try :)
>
That version does not work with VLANs. I have patches for it but it
needs a bit more work before sending out. Perhaps I can get back to it
next week.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 17:23 ` David Ahern
@ 2018-11-01 17:30 ` Paweł Staszewski
2018-11-03 17:32 ` David Ahern
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 17:30 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 01.11.2018 o 18:23, David Ahern pisze:
> On 11/1/18 7:52 AM, Paweł Staszewski wrote:
>>
>> W dniu 01.11.2018 o 11:55, Jesper Dangaard Brouer pisze:
>>> On Wed, 31 Oct 2018 21:37:16 -0600 David Ahern <dsahern@gmail.com> wrote:
>>>
>>>> This is mainly a forwarding use case? Seems so based on the perf report.
>>>> I suspect forwarding with XDP would show pretty good improvement.
>>> Yes, significant performance improvements.
>>>
>>> Notice Davids talk: "Leveraging Kernel Tables with XDP"
>>> http://vger.kernel.org/lpc-networking2018.html#session-1
>> It will be rly interesting
> It's pushing the exact use case you have: FRR manages the FIB, XDP
> programs get access to updates as they happen for fast path forwarding.
Cant wait then :)
>>> It looks like that you are doing "pure" IP-routing, without any
>>> iptables conntrack stuff (from your perf report data). That will
>>> actually be a really good use-case for accelerating this with XDP.
>> Yes pure IP routing
>> iptables used only for some local input filtering.
>>
>>
>>> I want you to understand the philosophy behind how David and I want
>>> people to leverage XDP. Think of XDP as a software offload layer for
>>> the kernel network stack. Setup and use Linux kernel network stack, but
>>> accelerate parts of it with XDP, e.g. the route FIB lookup.
>>>
>>> Sample code avail here:
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/xdp_fwd_kern.c
>>>
>> I can try some tests on same hw but testlab configuration - will give it
>> a try :)
>>
> That version does not work with VLANs. I have patches for it but it
> needs a bit more work before sending out. Perhaps I can get back to it
> next week.
>
Will be nice - next week i will be able to replace network controller
and install separate two 100Gbit nics into two pciex x16 slots - so can
test without hitting pcie bandwidth limits.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 15:27 ` Aaron Lu
@ 2018-11-01 20:23 ` Saeed Mahameed
2018-11-02 5:23 ` Aaron Lu
0 siblings, 1 reply; 77+ messages in thread
From: Saeed Mahameed @ 2018-11-01 20:23 UTC (permalink / raw)
To: aaron.lu, brouer
Cc: pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> wrote:
> ... ...
> > Section copied out:
> >
> > mlx5e_poll_tx_cq
> > |
> > --16.34%--napi_consume_skb
> > |
> > |--12.65%--__free_pages_ok
> > | |
> > | --11.86%--free_one_page
> > | |
> > | |--10.10%
> > --queued_spin_lock_slowpath
> > | |
> > | --0.65%--_raw_spin_lock
>
> This callchain looks like it is freeing higher order pages than order
> 0:
> __free_pages_ok is only called for pages whose order are bigger than
> 0.
mlx5 rx uses only order 0 pages, so i don't know where these high order
tx SKBs are coming from..
>
> > |
> > |--1.55%--page_frag_free
> > |
> > --1.44%--skb_release_data
> >
> >
> > Let me explain what (I think) happens. The mlx5 driver RX-page
> > recycle
> > mechanism is not effective in this workload, and pages have to go
> > through the page allocator. The lock contention happens during
> > mlx5
> > DMA TX completion cycle. And the page allocator cannot keep up at
> > these speeds.
> >
> > One solution is extend page allocator with a bulk free API. (This
> > have
> > been on my TODO list for a long time, but I don't have a
> > micro-benchmark that trick the driver page-recycle to fail). It
> > should
> > fit nicely, as I can see that kmem_cache_free_bulk() does get
> > activated (bulk freeing SKBs), which means that DMA TX completion
> > do
> > have a bulk of packets.
> >
> > We can (and should) also improve the page recycle scheme in the
> > driver.
> > After LPC, I have a project with Tariq and Ilias (Cc'ed) to improve
> > the
> > page_pool, and we will (attempt) to generalize this, for both high-
> > end
> > mlx5 and more low-end ARM64-boards (macchiatobin and espressobin).
> >
> > The MM-people is in parallel working to improve the performance of
> > order-0 page returns. Thus, the explicit page bulk free API might
> > actually become less important. I actually think (Cc.) Aaron have
> > a
> > patchset he would like you to test, which removes the (zone->)lock
> > you hit in free_one_page().
>
> Thanks Jesper.
>
> Yes, the said patchset is in this branch:
> https://github.com/aaronlu/linux no_merge_cluster_alloc_4.19-rc5
>
> But as I said above, I think the lock contention here is for
> order > 0 pages so my current patchset will not work here,
> unfortunately.
>
> BTW, Mel Gorman has suggested an alternative way to improve page
> allocator's scalability and I'm working on it right now, it will
> improve page allocator's scalability for all order pages. I might be
> able to post it some time next week, will CC all of you when it's
> ready.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 11:09 ` Paweł Staszewski
2018-11-01 16:49 ` Paweł Staszewski
@ 2018-11-01 20:37 ` Saeed Mahameed
2018-11-01 21:18 ` Paweł Staszewski
2018-11-03 0:18 ` Paweł Staszewski
1 sibling, 2 replies; 77+ messages in thread
From: Saeed Mahameed @ 2018-11-01 20:37 UTC (permalink / raw)
To: pstaszewski, netdev
On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>
> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
> > On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
> > > Hi
> > >
> > > So maybee someone will be interested how linux kernel handles
> > > normal
> > > traffic (not pktgen :) )
> > >
> > >
> > > Server HW configuration:
> > >
> > > CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
> > >
> > > NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
> > >
> > >
> > > Server software:
> > >
> > > FRR - as routing daemon
> > >
> > > enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
> > > local
> > > numa
> > > node)
> > >
> > > enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
> > > numa
> > > node)
> > >
> > >
> > > Maximum traffic that server can handle:
> > >
> > > Bandwidth
> > >
> > > bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> > > input: /proc/net/dev type: rate
> > > \ iface Rx Tx Total
> > > =================================================================
> > > ====
> > > =========
> > > enp175s0f1: 28.51 Gb/s 37.24
> > > Gb/s
> > > 65.74 Gb/s
> > > enp175s0f0: 38.07 Gb/s 28.44
> > > Gb/s
> > > 66.51 Gb/s
> > > ---------------------------------------------------------------
> > > ----
> > > -----------
> > > total: 66.58 Gb/s 65.67
> > > Gb/s
> > > 132.25 Gb/s
> > >
> > >
> > > Packets per second:
> > >
> > > bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> > > input: /proc/net/dev type: rate
> > > - iface Rx Tx Total
> > > =================================================================
> > > ====
> > > =========
> > > enp175s0f1: 5248589.00 P/s 3486617.75 P/s
> > > 8735207.00 P/s
> > > enp175s0f0: 3557944.25 P/s 5232516.00 P/s
> > > 8790460.00 P/s
> > > ---------------------------------------------------------------
> > > ----
> > > -----------
> > > total: 8806533.00 P/s 8719134.00 P/s
> > > 17525668.00 P/s
> > >
> > >
> > > After reaching that limits nics on the upstream side (more RX
> > > traffic)
> > > start to drop packets
> > >
> > >
> > > I just dont understand that server can't handle more bandwidth
> > > (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
> > > RX
> > > side are increasing.
> > >
> >
> > Where do you see 40 Gb/s ? you showed that both ports on the same
> > NIC (
> > same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
> > 132.25
> > Gb/s which aligns with your pcie link limit, what am i missing ?
>
> hmm yes that was my concern also - cause cant find anywhere
> informations
> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
> 8GT
> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
> bw
> on both ports
i think it is bidir
> This can explain maybee also why cpuload is rising rapidly from
> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
> so
> there can be some error in reading them when offloading (gro/gso/tso)
> on
> nic's is enabled that is why
>
> >
> > > Was thinking that maybee reached some pcie x16 limit - but x16
> > > 8GT
> > > is
> > > 126Gbit - and also when testing with pktgen i can reach more bw
> > > and
> > > pps
> > > (like 4x more comparing to normal internet traffic)
> > >
> >
> > Are you forwarding when using pktgen as well or you just testing
> > the RX
> > side pps ?
>
> Yes pktgen was tested on single port RX
> Can check also forwarding to eliminate pciex limits
>
So this explains why you have more RX pps, since tx is idle and pcie
will be free to do only rx.
[...]
> >
> > > ethtool -S enp175s0f1
> > > NIC statistics:
> > > rx_packets: 173730800927
> > > rx_bytes: 99827422751332
> > > tx_packets: 142532009512
> > > tx_bytes: 184633045911222
> > > tx_tso_packets: 25989113891
> > > tx_tso_bytes: 132933363384458
> > > tx_tso_inner_packets: 0
> > > tx_tso_inner_bytes: 0
> > > tx_added_vlan_packets: 74630239613
> > > tx_nop: 2029817748
> > > rx_lro_packets: 0
> > > rx_lro_bytes: 0
> > > rx_ecn_mark: 0
> > > rx_removed_vlan_packets: 173730800927
> > > rx_csum_unnecessary: 0
> > > rx_csum_none: 434357
> > > rx_csum_complete: 173730366570
> > > rx_csum_unnecessary_inner: 0
> > > rx_xdp_drop: 0
> > > rx_xdp_redirect: 0
> > > rx_xdp_tx_xmit: 0
> > > rx_xdp_tx_full: 0
> > > rx_xdp_tx_err: 0
> > > rx_xdp_tx_cqe: 0
> > > tx_csum_none: 38260960853
> > > tx_csum_partial: 36369278774
> > > tx_csum_partial_inner: 0
> > > tx_queue_stopped: 1
> > > tx_queue_dropped: 0
> > > tx_xmit_more: 748638099
> > > tx_recover: 0
> > > tx_cqes: 73881645031
> > > tx_queue_wake: 1
> > > tx_udp_seg_rem: 0
> > > tx_cqe_err: 0
> > > tx_xdp_xmit: 0
> > > tx_xdp_full: 0
> > > tx_xdp_err: 0
> > > tx_xdp_cqes: 0
> > > rx_wqe_err: 0
> > > rx_mpwqe_filler_cqes: 0
> > > rx_mpwqe_filler_strides: 0
> > > rx_buff_alloc_err: 0
> > > rx_cqe_compress_blks: 0
> > > rx_cqe_compress_pkts: 0
> >
> > If this is a pcie bottleneck it might be useful to enable CQE
> > compression (to reduce PCIe completion descriptors transactions)
> > you should see the above rx_cqe_compress_pkts increasing when
> > enabled.
> >
> > $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
> > $ ethtool --show-priv-flags enp175s0f1
> > Private flags for p6p1:
> > rx_cqe_moder : on
> > cqe_moder : off
> > rx_cqe_compress : on
> > ...
> >
> > try this on both interfaces.
>
> Done
> ethtool --show-priv-flags enp175s0f1
> Private flags for enp175s0f1:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : on
> rx_striding_rq : off
> rx_no_csum_complete: off
>
> ethtool --show-priv-flags enp175s0f0
> Private flags for enp175s0f0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : on
> rx_striding_rq : off
> rx_no_csum_complete: off
>
did it help reduce the load on the pcie ? do you see more pps ?
what is the ratio between rx_cqe_compress_pkts and over all rx packets
?
[...]
> > > ethtool -S enp175s0f0
> > > NIC statistics:
> > > rx_packets: 141574897253
> > > rx_bytes: 184445040406258
> > > tx_packets: 172569543894
> > > tx_bytes: 99486882076365
> > > tx_tso_packets: 9367664195
> > > tx_tso_bytes: 56435233992948
> > > tx_tso_inner_packets: 0
> > > tx_tso_inner_bytes: 0
> > > tx_added_vlan_packets: 141297671626
> > > tx_nop: 2102916272
> > > rx_lro_packets: 0
> > > rx_lro_bytes: 0
> > > rx_ecn_mark: 0
> > > rx_removed_vlan_packets: 141574897252
> > > rx_csum_unnecessary: 0
> > > rx_csum_none: 23135854
> > > rx_csum_complete: 141551761398
> > > rx_csum_unnecessary_inner: 0
> > > rx_xdp_drop: 0
> > > rx_xdp_redirect: 0
> > > rx_xdp_tx_xmit: 0
> > > rx_xdp_tx_full: 0
> > > rx_xdp_tx_err: 0
> > > rx_xdp_tx_cqe: 0
> > > tx_csum_none: 127934791664
> >
> > It is a good idea to look into this, tx is not requesting hw tx
> > csumming for a lot of packets, maybe you are wasting a lot of cpu
> > on
> > calculating csum, or maybe this is just the rx csum complete..
> >
> > > tx_csum_partial: 13362879974
> > > tx_csum_partial_inner: 0
> > > tx_queue_stopped: 232561
> >
> > TX queues are stalling, could be an indentation for the pcie
> > bottelneck.
> >
> > > tx_queue_dropped: 0
> > > tx_xmit_more: 1266021946
> > > tx_recover: 0
> > > tx_cqes: 140031716469
> > > tx_queue_wake: 232561
> > > tx_udp_seg_rem: 0
> > > tx_cqe_err: 0
> > > tx_xdp_xmit: 0
> > > tx_xdp_full: 0
> > > tx_xdp_err: 0
> > > tx_xdp_cqes: 0
> > > rx_wqe_err: 0
> > > rx_mpwqe_filler_cqes: 0
> > > rx_mpwqe_filler_strides: 0
> > > rx_buff_alloc_err: 0
> > > rx_cqe_compress_blks: 0
> > > rx_cqe_compress_pkts: 0
> > > rx_page_reuse: 0
> > > rx_cache_reuse: 16625975793
> > > rx_cache_full: 54161465914
> > > rx_cache_empty: 258048
> > > rx_cache_busy: 54161472735
> > > rx_cache_waive: 0
> > > rx_congst_umr: 0
> > > rx_arfs_err: 0
> > > ch_events: 40572621887
> > > ch_poll: 40885650979
> > > ch_arm: 40429276692
> > > ch_aff_change: 0
> > > ch_eq_rearm: 0
> > > rx_out_of_buffer: 2791690
> > > rx_if_down_packets: 74
> > > rx_vport_unicast_packets: 141843476308
> > > rx_vport_unicast_bytes: 185421265403318
> > > tx_vport_unicast_packets: 172569484005
> > > tx_vport_unicast_bytes: 100019940094298
> > > rx_vport_multicast_packets: 85122935
> > > rx_vport_multicast_bytes: 5761316431
> > > tx_vport_multicast_packets: 6452
> > > tx_vport_multicast_bytes: 643540
> > > rx_vport_broadcast_packets: 22423624
> > > rx_vport_broadcast_bytes: 1390127090
> > > tx_vport_broadcast_packets: 22024
> > > tx_vport_broadcast_bytes: 1321440
> > > rx_vport_rdma_unicast_packets: 0
> > > rx_vport_rdma_unicast_bytes: 0
> > > tx_vport_rdma_unicast_packets: 0
> > > tx_vport_rdma_unicast_bytes: 0
> > > rx_vport_rdma_multicast_packets: 0
> > > rx_vport_rdma_multicast_bytes: 0
> > > tx_vport_rdma_multicast_packets: 0
> > > tx_vport_rdma_multicast_bytes: 0
> > > tx_packets_phy: 172569501577
> > > rx_packets_phy: 142871314588
> > > rx_crc_errors_phy: 0
> > > tx_bytes_phy: 100710212814151
> > > rx_bytes_phy: 187209224289564
> > > tx_multicast_phy: 6452
> > > tx_broadcast_phy: 22024
> > > rx_multicast_phy: 85122933
> > > rx_broadcast_phy: 22423623
> > > rx_in_range_len_errors_phy: 2
> > > rx_out_of_range_len_phy: 0
> > > rx_oversize_pkts_phy: 0
> > > rx_symbol_err_phy: 0
> > > tx_mac_control_phy: 0
> > > rx_mac_control_phy: 0
> > > rx_unsupported_op_phy: 0
> > > rx_pause_ctrl_phy: 0
> > > tx_pause_ctrl_phy: 0
> > > rx_discards_phy: 920161423
> >
> > Ok, this port seem to be suffering more, RX is congested, maybe due
> > to
> > the pcie bottleneck.
>
> Yes this side is receiving more traffic - second port is +10G more tx
>
[...]
> > > Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
> > > 0.00 0.00 0.00 31.30
> > > Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
> > > 0.00 0.00 0.00 24.90
> > > Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
> > > 0.00 0.00 0.00 19.68
> > > Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
> > > 0.00 0.00 0.00 18.00
> > > Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
> > > 0.00 0.00 0.00 17.40
> > > Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
> > > 0.00 0.00 0.00 26.03
> > > Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
> > > 0.00 0.00 0.00 17.52
> > > Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
> > > 0.00 0.00 0.00 18.20
> > > Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
> > > 0.00 0.00 0.00 17.68
> > > Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
> > > 0.00 0.00 0.00 18.20
> > > Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
> > > 0.00 0.00 0.00 20.42
> >
> > The numa cores are not at 100% util, you have around 20% of idle on
> > each one.
>
> Yes - no 100% cpu - but the difference between 80% and 100% is like
> push
> aditional 1-2Gbit/s
>
yes but, it doens't look like the bottleneck is the cpu, although it is
close to be :)..
> >
> > > Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 96.10
> > > Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.70
> > > Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.50
> > > Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 97.40
> > > Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.10
> > > Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.40
> > > Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
> > > 0.00 0.00 0.00 19.42
> > > Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
> > > 0.00 0.00 0.00 26.60
> > > Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
> > > 0.00 0.00 0.00 21.60
> > > Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
> > > 0.00 0.00 0.00 20.50
> > > Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
> > > 0.00 0.00 0.00 21.60
> > > Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
> > > 0.00 0.00 0.00 24.58
> > > Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
> > > 0.00 0.00 0.00 24.68
> > > Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
> > > 0.00 0.00 0.00 28.34
> > > Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
> > > 0.00 0.00 0.00 25.83
> > > Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
> > > 0.00 0.00 0.00 27.20
> > > Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
> > > 0.00 0.00 0.00 28.03
> > > Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
> > > 0.00 0.00 0.00 24.70
> > > Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
> > > 0.00 0.00 0.00 25.30
> > > Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
> > > 0.00 0.00 0.00 26.77
> > >
> > >
> > > ethtool -k enp175s0f0
> > > Features for enp175s0f0:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > tx-checksum-ipv4: on
> > > tx-checksum-ip-generic: off [fixed]
> > > tx-checksum-ipv6: on
> > > tx-checksum-fcoe-crc: off [fixed]
> > > tx-checksum-sctp: off [fixed]
> > > scatter-gather: on
> > > tx-scatter-gather: on
> > > tx-scatter-gather-fraglist: off [fixed]
> > > tcp-segmentation-offload: on
> > > tx-tcp-segmentation: on
> > > tx-tcp-ecn-segmentation: off [fixed]
> > > tx-tcp-mangleid-segmentation: off
> > > tx-tcp6-segmentation: on
> > > udp-fragmentation-offload: off
> > > generic-segmentation-offload: on
> > > generic-receive-offload: on
> > > large-receive-offload: off [fixed]
> > > rx-vlan-offload: on
> > > tx-vlan-offload: on
> > > ntuple-filters: off
> > > receive-hashing: on
> > > highdma: on [fixed]
> > > rx-vlan-filter: on
> > > vlan-challenged: off [fixed]
> > > tx-lockless: off [fixed]
> > > netns-local: off [fixed]
> > > tx-gso-robust: off [fixed]
> > > tx-fcoe-segmentation: off [fixed]
> > > tx-gre-segmentation: on
> > > tx-gre-csum-segmentation: on
> > > tx-ipxip4-segmentation: off [fixed]
> > > tx-ipxip6-segmentation: off [fixed]
> > > tx-udp_tnl-segmentation: on
> > > tx-udp_tnl-csum-segmentation: on
> > > tx-gso-partial: on
> > > tx-sctp-segmentation: off [fixed]
> > > tx-esp-segmentation: off [fixed]
> > > tx-udp-segmentation: on
> > > fcoe-mtu: off [fixed]
> > > tx-nocache-copy: off
> > > loopback: off [fixed]
> > > rx-fcs: off
> > > rx-all: off
> > > tx-vlan-stag-hw-insert: on
> > > rx-vlan-stag-hw-parse: off [fixed]
> > > rx-vlan-stag-filter: on [fixed]
> > > l2-fwd-offload: off [fixed]
> > > hw-tc-offload: off
> > > esp-hw-offload: off [fixed]
> > > esp-tx-csum-hw-offload: off [fixed]
> > > rx-udp_tunnel-port-offload: on
> > > tls-hw-tx-offload: off [fixed]
> > > tls-hw-rx-offload: off [fixed]
> > > rx-gro-hw: off [fixed]
> > > tls-hw-record: off [fixed]
> > >
> > > ethtool -c enp175s0f0
> > > Coalesce parameters for enp175s0f0:
> > > Adaptive RX: off TX: on
> > > stats-block-usecs: 0
> > > sample-interval: 0
> > > pkt-rate-low: 0
> > > pkt-rate-high: 0
> > > dmac: 32703
> > >
> > > rx-usecs: 256
> > > rx-frames: 128
> > > rx-usecs-irq: 0
> > > rx-frames-irq: 0
> > >
> > > tx-usecs: 8
> > > tx-frames: 128
> > > tx-usecs-irq: 0
> > > tx-frames-irq: 0
> > >
> > > rx-usecs-low: 0
> > > rx-frame-low: 0
> > > tx-usecs-low: 0
> > > tx-frame-low: 0
> > >
> > > rx-usecs-high: 0
> > > rx-frame-high: 0
> > > tx-usecs-high: 0
> > > tx-frame-high: 0
> > >
> > > ethtool -g enp175s0f0
> > > Ring parameters for enp175s0f0:
> > > Pre-set maximums:
> > > RX: 8192
> > > RX Mini: 0
> > > RX Jumbo: 0
> > > TX: 8192
> > > Current hardware settings:
> > > RX: 4096
> > > RX Mini: 0
> > > RX Jumbo: 0
> > > TX: 4096
> > >
> > >
> > >
> > >
> > >
> > >
>
> Also changed a little coalesce params - and best for this config are:
> ethtool -c enp175s0f0
> Coalesce parameters for enp175s0f0:
> Adaptive RX: off TX: off
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
> dmac: 32573
>
> rx-usecs: 40
> rx-frames: 128
> rx-usecs-irq: 0
> rx-frames-irq: 0
>
> tx-usecs: 8
> tx-frames: 8
> tx-usecs-irq: 0
> tx-frames-irq: 0
>
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
>
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
>
>
> Less drops on RX side - and more pps in overall forwarded.
>
how much improvement ? maybe we can improve our adaptive rx coal to be
efficient for this work load.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 20:37 ` Saeed Mahameed
@ 2018-11-01 21:18 ` Paweł Staszewski
2018-11-01 21:24 ` Paweł Staszewski
2018-11-03 0:18 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 21:18 UTC (permalink / raw)
To: Saeed Mahameed, netdev
W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>> Hi
>>>>
>>>> So maybee someone will be interested how linux kernel handles
>>>> normal
>>>> traffic (not pktgen :) )
>>>>
>>>>
>>>> Server HW configuration:
>>>>
>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>
>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>
>>>>
>>>> Server software:
>>>>
>>>> FRR - as routing daemon
>>>>
>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>> local
>>>> numa
>>>> node)
>>>>
>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>> numa
>>>> node)
>>>>
>>>>
>>>> Maximum traffic that server can handle:
>>>>
>>>> Bandwidth
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> \ iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>> Gb/s
>>>> 65.74 Gb/s
>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>> Gb/s
>>>> 66.51 Gb/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 66.58 Gb/s 65.67
>>>> Gb/s
>>>> 132.25 Gb/s
>>>>
>>>>
>>>> Packets per second:
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>> 8735207.00 P/s
>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>> 8790460.00 P/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>> 17525668.00 P/s
>>>>
>>>>
>>>> After reaching that limits nics on the upstream side (more RX
>>>> traffic)
>>>> start to drop packets
>>>>
>>>>
>>>> I just dont understand that server can't handle more bandwidth
>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>> RX
>>>> side are increasing.
>>>>
>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>> NIC (
>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>> 132.25
>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>> hmm yes that was my concern also - cause cant find anywhere
>> informations
>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>> 8GT
>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>> bw
>> on both ports
> i think it is bidir
>
>> This can explain maybee also why cpuload is rising rapidly from
>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>> so
>> there can be some error in reading them when offloading (gro/gso/tso)
>> on
>> nic's is enabled that is why
>>
>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>> 8GT
>>>> is
>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>> and
>>>> pps
>>>> (like 4x more comparing to normal internet traffic)
>>>>
>>> Are you forwarding when using pktgen as well or you just testing
>>> the RX
>>> side pps ?
>> Yes pktgen was tested on single port RX
>> Can check also forwarding to eliminate pciex limits
>>
> So this explains why you have more RX pps, since tx is idle and pcie
> will be free to do only rx.
>
> [...]
>
>
>>>> ethtool -S enp175s0f1
>>>> NIC statistics:
>>>> rx_packets: 173730800927
>>>> rx_bytes: 99827422751332
>>>> tx_packets: 142532009512
>>>> tx_bytes: 184633045911222
>>>> tx_tso_packets: 25989113891
>>>> tx_tso_bytes: 132933363384458
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 74630239613
>>>> tx_nop: 2029817748
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 173730800927
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 434357
>>>> rx_csum_complete: 173730366570
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 38260960853
>>>> tx_csum_partial: 36369278774
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 1
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 748638099
>>>> tx_recover: 0
>>>> tx_cqes: 73881645031
>>>> tx_queue_wake: 1
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>> If this is a pcie bottleneck it might be useful to enable CQE
>>> compression (to reduce PCIe completion descriptors transactions)
>>> you should see the above rx_cqe_compress_pkts increasing when
>>> enabled.
>>>
>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>> $ ethtool --show-priv-flags enp175s0f1
>>> Private flags for p6p1:
>>> rx_cqe_moder : on
>>> cqe_moder : off
>>> rx_cqe_compress : on
>>> ...
>>>
>>> try this on both interfaces.
>> Done
>> ethtool --show-priv-flags enp175s0f1
>> Private flags for enp175s0f1:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
>> ethtool --show-priv-flags enp175s0f0
>> Private flags for enp175s0f0:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
> did it help reduce the load on the pcie ? do you see more pps ?
> what is the ratio between rx_cqe_compress_pkts and over all rx packets
> ?
So - a little more pps
Before change top - graph / after bottom -> image with graph stats from
proc/net/dev
cqe_compress enabled at 11:55
Sorry - but for real life traffic it is hard to do any counter
differences - cause traffic just rising alone from minute to minute :)
But for that time the change is visible on graph - cause was almost same
for past 20minutes before change.
full ethtool below:
NIC statistics:
rx_packets: 516522465438
rx_bytes: 680052911258729
tx_packets: 677697545586
tx_bytes: 413647643141709
tx_tso_packets: 42530913279
tx_tso_bytes: 235655668554142
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 551156530885
tx_nop: 8536823558
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 516522465438
rx_csum_unnecessary: 0
rx_csum_none: 50382868
rx_csum_complete: 516472082570
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 494075047017
tx_csum_partial: 57081483898
tx_csum_partial_inner: 0
tx_queue_stopped: 518624
tx_queue_dropped: 0
tx_xmit_more: 1717880628
tx_recover: 0
tx_cqes: 549438869029
tx_queue_wake: 518627
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 11483228712
rx_cqe_compress_pkts: 25794213324
rx_page_reuse: 0
rx_cache_reuse: 63610249810
rx_cache_full: 194650916511
rx_cache_empty: 1118208
rx_cache_busy: 194650982430
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 119556002196
ch_poll: 121107424977
ch_arm: 115856746008
ch_aff_change: 31
ch_eq_rearm: 0
rx_out_of_buffer: 6880325
rx_if_down_packets: 2062529
rx_vport_unicast_packets: 517433716795
rx_vport_unicast_bytes: 683464347301443
tx_vport_unicast_packets: 677697453738
tx_vport_unicast_bytes: 415788589663315
rx_vport_multicast_packets: 208258309
rx_vport_multicast_bytes: 14224046052
tx_vport_multicast_packets: 21689
tx_vport_multicast_bytes: 2158334
rx_vport_broadcast_packets: 75838646
rx_vport_broadcast_bytes: 4697944695
tx_vport_broadcast_packets: 68730
tx_vport_broadcast_bytes: 4123800
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 677697543252
rx_packets_phy: 521319491878
rx_crc_errors_phy: 0
tx_bytes_phy: 418499385791411
rx_bytes_phy: 690322537017274
tx_multicast_phy: 21689
tx_broadcast_phy: 68730
rx_multicast_phy: 208258305
rx_broadcast_phy: 75838646
rx_in_range_len_errors_phy: 4
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 3601449265
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 1416456771
rx_65_to_127_bytes_phy: 40750434737
rx_128_to_255_bytes_phy: 11518110310
rx_256_to_511_bytes_phy: 7055850637
rx_512_to_1023_bytes_phy: 7811550424
rx_1024_to_1518_bytes_phy: 265547564845
rx_1519_to_2047_bytes_phy: 187219522899
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 48
rx_prio0_bytes: 688807632117485
rx_prio0_packets: 516310309931
tx_prio0_bytes: 418499382756025
tx_prio0_packets: 677697534982
rx_prio1_bytes: 1497701612877
rx_prio1_packets: 1206768094
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 112271227
rx_prio2_packets: 337295
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 1165455555
rx_prio3_packets: 1544310
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 161857240
rx_prio4_packets: 341392
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 455031612
rx_prio5_packets: 2861469
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 1873928697
rx_prio6_packets: 5146981
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 13423452430
rx_prio7_packets: 190724796
tx_prio7_bytes: 0
tx_prio7_packets: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
ch0_events: 4252266777
ch0_poll: 4330804273
ch0_arm: 4120233182
ch0_aff_change: 2
ch0_eq_rearm: 0
ch1_events: 3938415938
ch1_poll: 4012621322
ch1_arm: 3810131188
ch1_aff_change: 2
ch1_eq_rearm: 0
ch2_events: 3897428860
ch2_poll: 3973886848
ch2_arm: 3773019397
ch2_aff_change: 1
ch2_eq_rearm: 0
ch3_events: 4108000541
ch3_poll: 4180139872
ch3_arm: 3982093366
ch3_aff_change: 1
ch3_eq_rearm: 0
ch4_events: 4652570079
ch4_poll: 4720541090
ch4_arm: 4524475054
ch4_aff_change: 2
ch4_eq_rearm: 0
ch5_events: 3899177385
ch5_poll: 3974274186
ch5_arm: 3772299186
ch5_aff_change: 2
ch5_eq_rearm: 0
ch6_events: 3915161350
ch6_poll: 3992338199
ch6_arm: 3794710989
ch6_aff_change: 0
ch6_eq_rearm: 0
ch7_events: 4008175631
ch7_poll: 4081321248
ch7_arm: 3882263723
ch7_aff_change: 0
ch7_eq_rearm: 0
ch8_events: 4207422352
ch8_poll: 4276465449
ch8_arm: 4077650366
ch8_aff_change: 0
ch8_eq_rearm: 0
ch9_events: 4036491879
ch9_poll: 4108975987
ch9_arm: 3914493694
ch9_aff_change: 0
ch9_eq_rearm: 0
ch10_events: 4066261595
ch10_poll: 4134419606
ch10_arm: 3936637711
ch10_aff_change: 1
ch10_eq_rearm: 0
ch11_events: 4440494043
ch11_poll: 4507578730
ch11_arm: 4318629438
ch11_aff_change: 0
ch11_eq_rearm: 0
ch12_events: 4066958252
ch12_poll: 4130191506
ch12_arm: 3934337782
ch12_aff_change: 0
ch12_eq_rearm: 0
ch13_events: 4051309159
ch13_poll: 4118864120
ch13_arm: 3921011919
ch13_aff_change: 0
ch13_eq_rearm: 0
ch14_events: 4321664800
ch14_poll: 4382433680
ch14_arm: 4186130552
ch14_aff_change: 0
ch14_eq_rearm: 0
ch15_events: 4701102075
ch15_poll: 4760373932
ch15_arm: 4570151468
ch15_aff_change: 0
ch15_eq_rearm: 0
ch16_events: 4311052687
ch16_poll: 4345937129
ch16_arm: 4170883819
ch16_aff_change: 0
ch16_eq_rearm: 0
ch17_events: 4647570931
ch17_poll: 4680218533
ch17_arm: 4509426288
ch17_aff_change: 0
ch17_eq_rearm: 0
ch18_events: 4598195702
ch18_poll: 4631314898
ch18_arm: 4457267084
ch18_aff_change: 0
ch18_eq_rearm: 0
ch19_events: 4808094560
ch19_poll: 4841368340
ch19_arm: 4670604358
ch19_aff_change: 0
ch19_eq_rearm: 0
ch20_events: 4240910605
ch20_poll: 4276531502
ch20_arm: 4101767278
ch20_aff_change: 1
ch20_eq_rearm: 0
ch21_events: 4389371472
ch21_poll: 4426870311
ch21_arm: 4249339045
ch21_aff_change: 2
ch21_eq_rearm: 0
ch22_events: 4282958754
ch22_poll: 4319228073
ch22_arm: 4145102991
ch22_aff_change: 2
ch22_eq_rearm: 0
ch23_events: 4440196528
ch23_poll: 4474090188
ch23_arm: 4300837147
ch23_aff_change: 2
ch23_eq_rearm: 0
ch24_events: 4326875785
ch24_poll: 4364971263
ch24_arm: 4186404526
ch24_aff_change: 2
ch24_eq_rearm: 0
ch25_events: 4286528453
ch25_poll: 4324089445
ch25_arm: 4147222616
ch25_aff_change: 3
ch25_eq_rearm: 0
ch26_events: 4098043104
ch26_poll: 4138133745
ch26_arm: 3967438971
ch26_aff_change: 4
ch26_eq_rearm: 0
ch27_events: 4563302840
ch27_poll: 4599441446
ch27_arm: 4432182806
ch27_aff_change: 4
ch27_eq_rearm: 0
ch28_events: 4
ch28_poll: 4
ch28_arm: 4
ch28_aff_change: 0
ch28_eq_rearm: 0
ch29_events: 6
ch29_poll: 6
ch29_arm: 6
ch29_aff_change: 0
ch29_eq_rearm: 0
ch30_events: 4
ch30_poll: 4
ch30_arm: 4
ch30_aff_change: 0
ch30_eq_rearm: 0
ch31_events: 4
ch31_poll: 4
ch31_arm: 4
ch31_aff_change: 0
ch31_eq_rearm: 0
ch32_events: 4
ch32_poll: 4
ch32_arm: 4
ch32_aff_change: 0
ch32_eq_rearm: 0
ch33_events: 4
ch33_poll: 4
ch33_arm: 4
ch33_aff_change: 0
ch33_eq_rearm: 0
ch34_events: 4
ch34_poll: 4
ch34_arm: 4
ch34_aff_change: 0
ch34_eq_rearm: 0
ch35_events: 4
ch35_poll: 4
ch35_arm: 4
ch35_aff_change: 0
ch35_eq_rearm: 0
ch36_events: 4
ch36_poll: 4
ch36_arm: 4
ch36_aff_change: 0
ch36_eq_rearm: 0
ch37_events: 4
ch37_poll: 4
ch37_arm: 4
ch37_aff_change: 0
ch37_eq_rearm: 0
ch38_events: 4
ch38_poll: 4
ch38_arm: 4
ch38_aff_change: 0
ch38_eq_rearm: 0
ch39_events: 4
ch39_poll: 4
ch39_arm: 4
ch39_aff_change: 0
ch39_eq_rearm: 0
ch40_events: 4
ch40_poll: 4
ch40_arm: 4
ch40_aff_change: 0
ch40_eq_rearm: 0
ch41_events: 4
ch41_poll: 4
ch41_arm: 4
ch41_aff_change: 0
ch41_eq_rearm: 0
ch42_events: 4
ch42_poll: 4
ch42_arm: 4
ch42_aff_change: 0
ch42_eq_rearm: 0
ch43_events: 4
ch43_poll: 4
ch43_arm: 4
ch43_aff_change: 0
ch43_eq_rearm: 0
ch44_events: 4
ch44_poll: 4
ch44_arm: 4
ch44_aff_change: 0
ch44_eq_rearm: 0
ch45_events: 4
ch45_poll: 4
ch45_arm: 4
ch45_aff_change: 0
ch45_eq_rearm: 0
ch46_events: 4
ch46_poll: 4
ch46_arm: 4
ch46_aff_change: 0
ch46_eq_rearm: 0
ch47_events: 4
ch47_poll: 4
ch47_arm: 4
ch47_aff_change: 0
ch47_eq_rearm: 0
ch48_events: 4
ch48_poll: 4
ch48_arm: 4
ch48_aff_change: 0
ch48_eq_rearm: 0
ch49_events: 4
ch49_poll: 4
ch49_arm: 4
ch49_aff_change: 0
ch49_eq_rearm: 0
ch50_events: 4
ch50_poll: 4
ch50_arm: 4
ch50_aff_change: 0
ch50_eq_rearm: 0
ch51_events: 4
ch51_poll: 4
ch51_arm: 4
ch51_aff_change: 0
ch51_eq_rearm: 0
ch52_events: 4
ch52_poll: 4
ch52_arm: 4
ch52_aff_change: 0
ch52_eq_rearm: 0
ch53_events: 4
ch53_poll: 4
ch53_arm: 4
ch53_aff_change: 0
ch53_eq_rearm: 0
ch54_events: 4
ch54_poll: 4
ch54_arm: 4
ch54_aff_change: 0
ch54_eq_rearm: 0
ch55_events: 4
ch55_poll: 4
ch55_arm: 4
ch55_aff_change: 0
ch55_eq_rearm: 0
rx0_packets: 21390033774
rx0_bytes: 27326856299122
rx0_csum_complete: 21339650906
rx0_csum_unnecessary: 0
rx0_csum_unnecessary_inner: 0
rx0_csum_none: 50382868
rx0_xdp_drop: 0
rx0_xdp_redirect: 0
rx0_lro_packets: 0
rx0_lro_bytes: 0
rx0_ecn_mark: 0
rx0_removed_vlan_packets: 21390033774
rx0_wqe_err: 0
rx0_mpwqe_filler_cqes: 0
rx0_mpwqe_filler_strides: 0
rx0_buff_alloc_err: 0
rx0_cqe_compress_blks: 481077641
rx0_cqe_compress_pkts: 1085647489
rx0_page_reuse: 0
rx0_cache_reuse: 19050049
rx0_cache_full: 10675964285
rx0_cache_empty: 37376
rx0_cache_busy: 10675966819
rx0_cache_waive: 0
rx0_congst_umr: 0
rx0_arfs_err: 0
rx0_xdp_tx_xmit: 0
rx0_xdp_tx_full: 0
rx0_xdp_tx_err: 0
rx0_xdp_tx_cqes: 0
rx1_packets: 19868919527
rx1_bytes: 26149716991561
rx1_csum_complete: 19868919527
rx1_csum_unnecessary: 0
rx1_csum_unnecessary_inner: 0
rx1_csum_none: 0
rx1_xdp_drop: 0
rx1_xdp_redirect: 0
rx1_lro_packets: 0
rx1_lro_bytes: 0
rx1_ecn_mark: 0
rx1_removed_vlan_packets: 19868919527
rx1_wqe_err: 0
rx1_mpwqe_filler_cqes: 0
rx1_mpwqe_filler_strides: 0
rx1_buff_alloc_err: 0
rx1_cqe_compress_blks: 420210560
rx1_cqe_compress_pkts: 941233388
rx1_page_reuse: 0
rx1_cache_reuse: 46200002
rx1_cache_full: 9888257242
rx1_cache_empty: 37376
rx1_cache_busy: 9888259746
rx1_cache_waive: 0
rx1_congst_umr: 0
rx1_arfs_err: 0
rx1_xdp_tx_xmit: 0
rx1_xdp_tx_full: 0
rx1_xdp_tx_err: 0
rx1_xdp_tx_cqes: 0
rx2_packets: 19575013662
rx2_bytes: 25759818417945
rx2_csum_complete: 19575013662
rx2_csum_unnecessary: 0
rx2_csum_unnecessary_inner: 0
rx2_csum_none: 0
rx2_xdp_drop: 0
rx2_xdp_redirect: 0
rx2_lro_packets: 0
rx2_lro_bytes: 0
rx2_ecn_mark: 0
rx2_removed_vlan_packets: 19575013662
rx2_wqe_err: 0
rx2_mpwqe_filler_cqes: 0
rx2_mpwqe_filler_strides: 0
rx2_buff_alloc_err: 0
rx2_cqe_compress_blks: 412345511
rx2_cqe_compress_pkts: 923376167
rx2_page_reuse: 0
rx2_cache_reuse: 38837731
rx2_cache_full: 9748666548
rx2_cache_empty: 37376
rx2_cache_busy: 9748669093
rx2_cache_waive: 0
rx2_congst_umr: 0
rx2_arfs_err: 0
rx2_xdp_tx_xmit: 0
rx2_xdp_tx_full: 0
rx2_xdp_tx_err: 0
rx2_xdp_tx_cqes: 0
rx3_packets: 19795911749
rx3_bytes: 25969475566905
rx3_csum_complete: 19795911749
rx3_csum_unnecessary: 0
rx3_csum_unnecessary_inner: 0
rx3_csum_none: 0
rx3_xdp_drop: 0
rx3_xdp_redirect: 0
rx3_lro_packets: 0
rx3_lro_bytes: 0
rx3_ecn_mark: 0
rx3_removed_vlan_packets: 19795911749
rx3_wqe_err: 0
rx3_mpwqe_filler_cqes: 0
rx3_mpwqe_filler_strides: 0
rx3_buff_alloc_err: 0
rx3_cqe_compress_blks: 416658765
rx3_cqe_compress_pkts: 934986266
rx3_page_reuse: 0
rx3_cache_reuse: 34542124
rx3_cache_full: 9863411232
rx3_cache_empty: 37376
rx3_cache_busy: 9863413732
rx3_cache_waive: 0
rx3_congst_umr: 0
rx3_arfs_err: 0
rx3_xdp_tx_xmit: 0
rx3_xdp_tx_full: 0
rx3_xdp_tx_err: 0
rx3_xdp_tx_cqes: 0
rx4_packets: 20445652378
rx4_bytes: 26949065110265
rx4_csum_complete: 20445652378
rx4_csum_unnecessary: 0
rx4_csum_unnecessary_inner: 0
rx4_csum_none: 0
rx4_xdp_drop: 0
rx4_xdp_redirect: 0
rx4_lro_packets: 0
rx4_lro_bytes: 0
rx4_ecn_mark: 0
rx4_removed_vlan_packets: 20445652378
rx4_wqe_err: 0
rx4_mpwqe_filler_cqes: 0
rx4_mpwqe_filler_strides: 0
rx4_buff_alloc_err: 0
rx4_cqe_compress_blks: 506085858
rx4_cqe_compress_pkts: 1147860328
rx4_page_reuse: 0
rx4_cache_reuse: 10122542864
rx4_cache_full: 100281206
rx4_cache_empty: 37376
rx4_cache_busy: 100283304
rx4_cache_waive: 0
rx4_congst_umr: 0
rx4_arfs_err: 0
rx4_xdp_tx_xmit: 0
rx4_xdp_tx_full: 0
rx4_xdp_tx_err: 0
rx4_xdp_tx_cqes: 0
rx5_packets: 19622362246
rx5_bytes: 25843450982982
rx5_csum_complete: 19622362246
rx5_csum_unnecessary: 0
rx5_csum_unnecessary_inner: 0
rx5_csum_none: 0
rx5_xdp_drop: 0
rx5_xdp_redirect: 0
rx5_lro_packets: 0
rx5_lro_bytes: 0
rx5_ecn_mark: 0
rx5_removed_vlan_packets: 19622362246
rx5_wqe_err: 0
rx5_mpwqe_filler_cqes: 0
rx5_mpwqe_filler_strides: 0
rx5_buff_alloc_err: 0
rx5_cqe_compress_blks: 422840924
rx5_cqe_compress_pkts: 948005878
rx5_page_reuse: 0
rx5_cache_reuse: 31285453
rx5_cache_full: 9779893117
rx5_cache_empty: 37376
rx5_cache_busy: 9779895647
rx5_cache_waive: 0
rx5_congst_umr: 0
rx5_arfs_err: 0
rx5_xdp_tx_xmit: 0
rx5_xdp_tx_full: 0
rx5_xdp_tx_err: 0
rx5_xdp_tx_cqes: 0
rx6_packets: 19788231278
rx6_bytes: 25985783006486
rx6_csum_complete: 19788231278
rx6_csum_unnecessary: 0
rx6_csum_unnecessary_inner: 0
rx6_csum_none: 0
rx6_xdp_drop: 0
rx6_xdp_redirect: 0
rx6_lro_packets: 0
rx6_lro_bytes: 0
rx6_ecn_mark: 0
rx6_removed_vlan_packets: 19788231278
rx6_wqe_err: 0
rx6_mpwqe_filler_cqes: 0
rx6_mpwqe_filler_strides: 0
rx6_buff_alloc_err: 0
rx6_cqe_compress_blks: 418799056
rx6_cqe_compress_pkts: 938282685
rx6_page_reuse: 0
rx6_cache_reuse: 18114793
rx6_cache_full: 9875998295
rx6_cache_empty: 37376
rx6_cache_busy: 9876000831
rx6_cache_waive: 0
rx6_congst_umr: 0
rx6_arfs_err: 0
rx6_xdp_tx_xmit: 0
rx6_xdp_tx_full: 0
rx6_xdp_tx_err: 0
rx6_xdp_tx_cqes: 0
rx7_packets: 19795759168
rx7_bytes: 26085056586860
rx7_csum_complete: 19795759168
rx7_csum_unnecessary: 0
rx7_csum_unnecessary_inner: 0
rx7_csum_none: 0
rx7_xdp_drop: 0
rx7_xdp_redirect: 0
rx7_lro_packets: 0
rx7_lro_bytes: 0
rx7_ecn_mark: 0
rx7_removed_vlan_packets: 19795759168
rx7_wqe_err: 0
rx7_mpwqe_filler_cqes: 0
rx7_mpwqe_filler_strides: 0
rx7_buff_alloc_err: 0
rx7_cqe_compress_blks: 413959224
rx7_cqe_compress_pkts: 927675936
rx7_page_reuse: 0
rx7_cache_reuse: 23902990
rx7_cache_full: 9873974042
rx7_cache_empty: 37376
rx7_cache_busy: 9873976574
rx7_cache_waive: 0
rx7_congst_umr: 0
rx7_arfs_err: 0
rx7_xdp_tx_xmit: 0
rx7_xdp_tx_full: 0
rx7_xdp_tx_err: 0
rx7_xdp_tx_cqes: 0
rx8_packets: 19963477439
rx8_bytes: 26384640501789
rx8_csum_complete: 19963477439
rx8_csum_unnecessary: 0
rx8_csum_unnecessary_inner: 0
rx8_csum_none: 0
rx8_xdp_drop: 0
rx8_xdp_redirect: 0
rx8_lro_packets: 0
rx8_lro_bytes: 0
rx8_ecn_mark: 0
rx8_removed_vlan_packets: 19963477439
rx8_wqe_err: 0
rx8_mpwqe_filler_cqes: 0
rx8_mpwqe_filler_strides: 0
rx8_buff_alloc_err: 0
rx8_cqe_compress_blks: 420422857
rx8_cqe_compress_pkts: 942720292
rx8_page_reuse: 0
rx8_cache_reuse: 88181713
rx8_cache_full: 9893554525
rx8_cache_empty: 37376
rx8_cache_busy: 9893556983
rx8_cache_waive: 0
rx8_congst_umr: 0
rx8_arfs_err: 0
rx8_xdp_tx_xmit: 0
rx8_xdp_tx_full: 0
rx8_xdp_tx_err: 0
rx8_xdp_tx_cqes: 0
rx9_packets: 19726642138
rx9_bytes: 26063924286499
rx9_csum_complete: 19726642138
rx9_csum_unnecessary: 0
rx9_csum_unnecessary_inner: 0
rx9_csum_none: 0
rx9_xdp_drop: 0
rx9_xdp_redirect: 0
rx9_lro_packets: 0
rx9_lro_bytes: 0
rx9_ecn_mark: 0
rx9_removed_vlan_packets: 19726642138
rx9_wqe_err: 0
rx9_mpwqe_filler_cqes: 0
rx9_mpwqe_filler_strides: 0
rx9_buff_alloc_err: 0
rx9_cqe_compress_blks: 424227411
rx9_cqe_compress_pkts: 951534873
rx9_page_reuse: 0
rx9_cache_reuse: 482901440
rx9_cache_full: 9380417487
rx9_cache_empty: 37376
rx9_cache_busy: 9380419608
rx9_cache_waive: 0
rx9_congst_umr: 0
rx9_arfs_err: 0
rx9_xdp_tx_xmit: 0
rx9_xdp_tx_full: 0
rx9_xdp_tx_err: 0
rx9_xdp_tx_cqes: 0
rx10_packets: 19901229170
rx10_bytes: 26300854495044
rx10_csum_complete: 19901229170
rx10_csum_unnecessary: 0
rx10_csum_unnecessary_inner: 0
rx10_csum_none: 0
rx10_xdp_drop: 0
rx10_xdp_redirect: 0
rx10_lro_packets: 0
rx10_lro_bytes: 0
rx10_ecn_mark: 0
rx10_removed_vlan_packets: 19901229170
rx10_wqe_err: 0
rx10_mpwqe_filler_cqes: 0
rx10_mpwqe_filler_strides: 0
rx10_buff_alloc_err: 0
rx10_cqe_compress_blks: 419082938
rx10_cqe_compress_pkts: 940791347
rx10_page_reuse: 0
rx10_cache_reuse: 14896055
rx10_cache_full: 9935715977
rx10_cache_empty: 37376
rx10_cache_busy: 9935718513
rx10_cache_waive: 0
rx10_congst_umr: 0
rx10_arfs_err: 0
rx10_xdp_tx_xmit: 0
rx10_xdp_tx_full: 0
rx10_xdp_tx_err: 0
rx10_xdp_tx_cqes: 0
rx11_packets: 20352190494
rx11_bytes: 26851034425372
rx11_csum_complete: 20352190494
rx11_csum_unnecessary: 0
rx11_csum_unnecessary_inner: 0
rx11_csum_none: 0
rx11_xdp_drop: 0
rx11_xdp_redirect: 0
rx11_lro_packets: 0
rx11_lro_bytes: 0
rx11_ecn_mark: 0
rx11_removed_vlan_packets: 20352190494
rx11_wqe_err: 0
rx11_mpwqe_filler_cqes: 0
rx11_mpwqe_filler_strides: 0
rx11_buff_alloc_err: 0
rx11_cqe_compress_blks: 501992147
rx11_cqe_compress_pkts: 1140398610
rx11_page_reuse: 0
rx11_cache_reuse: 10071721531
rx11_cache_full: 104371621
rx11_cache_empty: 37376
rx11_cache_busy: 104373697
rx11_cache_waive: 0
rx11_congst_umr: 0
rx11_arfs_err: 0
rx11_xdp_tx_xmit: 0
rx11_xdp_tx_full: 0
rx11_xdp_tx_err: 0
rx11_xdp_tx_cqes: 0
rx12_packets: 19934747149
rx12_bytes: 26296478787829
rx12_csum_complete: 19934747149
rx12_csum_unnecessary: 0
rx12_csum_unnecessary_inner: 0
rx12_csum_none: 0
rx12_xdp_drop: 0
rx12_xdp_redirect: 0
rx12_lro_packets: 0
rx12_lro_bytes: 0
rx12_ecn_mark: 0
rx12_removed_vlan_packets: 19934747149
rx12_wqe_err: 0
rx12_mpwqe_filler_cqes: 0
rx12_mpwqe_filler_strides: 0
rx12_buff_alloc_err: 0
rx12_cqe_compress_blks: 443350570
rx12_cqe_compress_pkts: 995997220
rx12_page_reuse: 0
rx12_cache_reuse: 9864934174
rx12_cache_full: 102437428
rx12_cache_empty: 37376
rx12_cache_busy: 102439382
rx12_cache_waive: 0
rx12_congst_umr: 0
rx12_arfs_err: 0
rx12_xdp_tx_xmit: 0
rx12_xdp_tx_full: 0
rx12_xdp_tx_err: 0
rx12_xdp_tx_cqes: 0
rx13_packets: 19866908096
rx13_bytes: 26160931936186
rx13_csum_complete: 19866908096
rx13_csum_unnecessary: 0
rx13_csum_unnecessary_inner: 0
rx13_csum_none: 0
rx13_xdp_drop: 0
rx13_xdp_redirect: 0
rx13_lro_packets: 0
rx13_lro_bytes: 0
rx13_ecn_mark: 0
rx13_removed_vlan_packets: 19866908096
rx13_wqe_err: 0
rx13_mpwqe_filler_cqes: 0
rx13_mpwqe_filler_strides: 0
rx13_buff_alloc_err: 0
rx13_cqe_compress_blks: 413640141
rx13_cqe_compress_pkts: 926175066
rx13_page_reuse: 0
rx13_cache_reuse: 36358610
rx13_cache_full: 9897092921
rx13_cache_empty: 37376
rx13_cache_busy: 9897095422
rx13_cache_waive: 0
rx13_congst_umr: 0
rx13_arfs_err: 0
rx13_xdp_tx_xmit: 0
rx13_xdp_tx_full: 0
rx13_xdp_tx_err: 0
rx13_xdp_tx_cqes: 0
rx14_packets: 20229035746
rx14_bytes: 26655092809172
rx14_csum_complete: 20229035746
rx14_csum_unnecessary: 0
rx14_csum_unnecessary_inner: 0
rx14_csum_none: 0
rx14_xdp_drop: 0
rx14_xdp_redirect: 0
rx14_lro_packets: 0
rx14_lro_bytes: 0
rx14_ecn_mark: 0
rx14_removed_vlan_packets: 20229035746
rx14_wqe_err: 0
rx14_mpwqe_filler_cqes: 0
rx14_mpwqe_filler_strides: 0
rx14_buff_alloc_err: 0
rx14_cqe_compress_blks: 460990337
rx14_cqe_compress_pkts: 1041287948
rx14_page_reuse: 0
rx14_cache_reuse: 25649275
rx14_cache_full: 10088866045
rx14_cache_empty: 37376
rx14_cache_busy: 10088868574
rx14_cache_waive: 0
rx14_congst_umr: 0
rx14_arfs_err: 0
rx14_xdp_tx_xmit: 0
rx14_xdp_tx_full: 0
rx14_xdp_tx_err: 0
rx14_xdp_tx_cqes: 0
rx15_packets: 20528177154
rx15_bytes: 27029263893264
rx15_csum_complete: 20528177154
rx15_csum_unnecessary: 0
rx15_csum_unnecessary_inner: 0
rx15_csum_none: 0
rx15_xdp_drop: 0
rx15_xdp_redirect: 0
rx15_lro_packets: 0
rx15_lro_bytes: 0
rx15_ecn_mark: 0
rx15_removed_vlan_packets: 20528177154
rx15_wqe_err: 0
rx15_mpwqe_filler_cqes: 0
rx15_mpwqe_filler_strides: 0
rx15_buff_alloc_err: 0
rx15_cqe_compress_blks: 476776176
rx15_cqe_compress_pkts: 1076153263
rx15_page_reuse: 0
rx15_cache_reuse: 48426735
rx15_cache_full: 10215659289
rx15_cache_empty: 37376
rx15_cache_busy: 10215661817
rx15_cache_waive: 0
rx15_congst_umr: 0
rx15_arfs_err: 0
rx15_xdp_tx_xmit: 0
rx15_xdp_tx_full: 0
rx15_xdp_tx_err: 0
rx15_xdp_tx_cqes: 0
rx16_packets: 16104078098
rx16_bytes: 21256361789679
rx16_csum_complete: 16104078098
rx16_csum_unnecessary: 0
rx16_csum_unnecessary_inner: 0
rx16_csum_none: 0
rx16_xdp_drop: 0
rx16_xdp_redirect: 0
rx16_lro_packets: 0
rx16_lro_bytes: 0
rx16_ecn_mark: 0
rx16_removed_vlan_packets: 16104078098
rx16_wqe_err: 0
rx16_mpwqe_filler_cqes: 0
rx16_mpwqe_filler_strides: 0
rx16_buff_alloc_err: 0
rx16_cqe_compress_blks: 352082054
rx16_cqe_compress_pkts: 787161670
rx16_page_reuse: 0
rx16_cache_reuse: 25912567
rx16_cache_full: 8026124051
rx16_cache_empty: 37376
rx16_cache_busy: 8026126465
rx16_cache_waive: 0
rx16_congst_umr: 0
rx16_arfs_err: 0
rx16_xdp_tx_xmit: 0
rx16_xdp_tx_full: 0
rx16_xdp_tx_err: 0
rx16_xdp_tx_cqes: 0
rx17_packets: 16314055017
rx17_bytes: 21589139030173
rx17_csum_complete: 16314055017
rx17_csum_unnecessary: 0
rx17_csum_unnecessary_inner: 0
rx17_csum_none: 0
rx17_xdp_drop: 0
rx17_xdp_redirect: 0
rx17_lro_packets: 0
rx17_lro_bytes: 0
rx17_ecn_mark: 0
rx17_removed_vlan_packets: 16314055017
rx17_wqe_err: 0
rx17_mpwqe_filler_cqes: 0
rx17_mpwqe_filler_strides: 0
rx17_buff_alloc_err: 0
rx17_cqe_compress_blks: 387834541
rx17_cqe_compress_pkts: 871851081
rx17_page_reuse: 0
rx17_cache_reuse: 24021313
rx17_cache_full: 8133003829
rx17_cache_empty: 37376
rx17_cache_busy: 8133006175
rx17_cache_waive: 0
rx17_congst_umr: 0
rx17_arfs_err: 0
rx17_xdp_tx_xmit: 0
rx17_xdp_tx_full: 0
rx17_xdp_tx_err: 0
rx17_xdp_tx_cqes: 0
rx18_packets: 16439016814
rx18_bytes: 21648651917475
rx18_csum_complete: 16439016814
rx18_csum_unnecessary: 0
rx18_csum_unnecessary_inner: 0
rx18_csum_none: 0
rx18_xdp_drop: 0
rx18_xdp_redirect: 0
rx18_lro_packets: 0
rx18_lro_bytes: 0
rx18_ecn_mark: 0
rx18_removed_vlan_packets: 16439016814
rx18_wqe_err: 0
rx18_mpwqe_filler_cqes: 0
rx18_mpwqe_filler_strides: 0
rx18_buff_alloc_err: 0
rx18_cqe_compress_blks: 375066666
rx18_cqe_compress_pkts: 843563974
rx18_page_reuse: 0
rx18_cache_reuse: 8151064266
rx18_cache_full: 68442025
rx18_cache_empty: 37376
rx18_cache_busy: 68444122
rx18_cache_waive: 0
rx18_congst_umr: 0
rx18_arfs_err: 0
rx18_xdp_tx_xmit: 0
rx18_xdp_tx_full: 0
rx18_xdp_tx_err: 0
rx18_xdp_tx_cqes: 0
rx19_packets: 16641223506
rx19_bytes: 21964749940935
rx19_csum_complete: 16641223506
rx19_csum_unnecessary: 0
rx19_csum_unnecessary_inner: 0
rx19_csum_none: 0
rx19_xdp_drop: 0
rx19_xdp_redirect: 0
rx19_lro_packets: 0
rx19_lro_bytes: 0
rx19_ecn_mark: 0
rx19_removed_vlan_packets: 16641223506
rx19_wqe_err: 0
rx19_mpwqe_filler_cqes: 0
rx19_mpwqe_filler_strides: 0
rx19_buff_alloc_err: 0
rx19_cqe_compress_blks: 387825932
rx19_cqe_compress_pkts: 872266355
rx19_page_reuse: 0
rx19_cache_reuse: 116433620
rx19_cache_full: 8204175954
rx19_cache_empty: 37376
rx19_cache_busy: 8204178120
rx19_cache_waive: 0
rx19_congst_umr: 0
rx19_arfs_err: 0
rx19_xdp_tx_xmit: 0
rx19_xdp_tx_full: 0
rx19_xdp_tx_err: 0
rx19_xdp_tx_cqes: 0
rx20_packets: 16206927741
rx20_bytes: 21387447038430
rx20_csum_complete: 16206927741
rx20_csum_unnecessary: 0
rx20_csum_unnecessary_inner: 0
rx20_csum_none: 0
rx20_xdp_drop: 0
rx20_xdp_redirect: 0
rx20_lro_packets: 0
rx20_lro_bytes: 0
rx20_ecn_mark: 0
rx20_removed_vlan_packets: 16206927741
rx20_wqe_err: 0
rx20_mpwqe_filler_cqes: 0
rx20_mpwqe_filler_strides: 0
rx20_buff_alloc_err: 0
rx20_cqe_compress_blks: 370144620
rx20_cqe_compress_pkts: 829122671
rx20_page_reuse: 0
rx20_cache_reuse: 8053733744
rx20_cache_full: 49728026
rx20_cache_empty: 37376
rx20_cache_busy: 49730116
rx20_cache_waive: 0
rx20_congst_umr: 0
rx20_arfs_err: 0
rx20_xdp_tx_xmit: 0
rx20_xdp_tx_full: 0
rx20_xdp_tx_err: 0
rx20_xdp_tx_cqes: 0
rx21_packets: 16562361314
rx21_bytes: 21856653284356
rx21_csum_complete: 16562361314
rx21_csum_unnecessary: 0
rx21_csum_unnecessary_inner: 0
rx21_csum_none: 0
rx21_xdp_drop: 0
rx21_xdp_redirect: 0
rx21_lro_packets: 0
rx21_lro_bytes: 0
rx21_ecn_mark: 0
rx21_removed_vlan_packets: 16562361314
rx21_wqe_err: 0
rx21_mpwqe_filler_cqes: 0
rx21_mpwqe_filler_strides: 0
rx21_buff_alloc_err: 0
rx21_cqe_compress_blks: 350790425
rx21_cqe_compress_pkts: 783850729
rx21_page_reuse: 0
rx21_cache_reuse: 28077493
rx21_cache_full: 8253100706
rx21_cache_empty: 37376
rx21_cache_busy: 8253103147
rx21_cache_waive: 0
rx21_congst_umr: 0
rx21_arfs_err: 0
rx21_xdp_tx_xmit: 0
rx21_xdp_tx_full: 0
rx21_xdp_tx_err: 0
rx21_xdp_tx_cqes: 0
rx22_packets: 16350307571
rx22_bytes: 21408575325592
rx22_csum_complete: 16350307571
rx22_csum_unnecessary: 0
rx22_csum_unnecessary_inner: 0
rx22_csum_none: 0
rx22_xdp_drop: 0
rx22_xdp_redirect: 0
rx22_lro_packets: 0
rx22_lro_bytes: 0
rx22_ecn_mark: 0
rx22_removed_vlan_packets: 16350307571
rx22_wqe_err: 0
rx22_mpwqe_filler_cqes: 0
rx22_mpwqe_filler_strides: 0
rx22_buff_alloc_err: 0
rx22_cqe_compress_blks: 353531065
rx22_cqe_compress_pkts: 790814415
rx22_page_reuse: 0
rx22_cache_reuse: 16934343
rx22_cache_full: 8158216889
rx22_cache_empty: 37376
rx22_cache_busy: 8158219417
rx22_cache_waive: 0
rx22_congst_umr: 0
rx22_arfs_err: 0
rx22_xdp_tx_xmit: 0
rx22_xdp_tx_full: 0
rx22_xdp_tx_err: 0
rx22_xdp_tx_cqes: 0
rx23_packets: 16019811764
rx23_bytes: 21137182570985
rx23_csum_complete: 16019811764
rx23_csum_unnecessary: 0
rx23_csum_unnecessary_inner: 0
rx23_csum_none: 0
rx23_xdp_drop: 0
rx23_xdp_redirect: 0
rx23_lro_packets: 0
rx23_lro_bytes: 0
rx23_ecn_mark: 0
rx23_removed_vlan_packets: 16019811764
rx23_wqe_err: 0
rx23_mpwqe_filler_cqes: 0
rx23_mpwqe_filler_strides: 0
rx23_buff_alloc_err: 0
rx23_cqe_compress_blks: 349733033
rx23_cqe_compress_pkts: 781248862
rx23_page_reuse: 0
rx23_cache_reuse: 33422343
rx23_cache_full: 7976481152
rx23_cache_empty: 37376
rx23_cache_busy: 7976483525
rx23_cache_waive: 0
rx23_congst_umr: 0
rx23_arfs_err: 0
rx23_xdp_tx_xmit: 0
rx23_xdp_tx_full: 0
rx23_xdp_tx_err: 0
rx23_xdp_tx_cqes: 0
rx24_packets: 16212040646
rx24_bytes: 21393399325700
rx24_csum_complete: 16212040646
rx24_csum_unnecessary: 0
rx24_csum_unnecessary_inner: 0
rx24_csum_none: 0
rx24_xdp_drop: 0
rx24_xdp_redirect: 0
rx24_lro_packets: 0
rx24_lro_bytes: 0
rx24_ecn_mark: 0
rx24_removed_vlan_packets: 16212040646
rx24_wqe_err: 0
rx24_mpwqe_filler_cqes: 0
rx24_mpwqe_filler_strides: 0
rx24_buff_alloc_err: 0
rx24_cqe_compress_blks: 379833752
rx24_cqe_compress_pkts: 852020179
rx24_page_reuse: 0
rx24_cache_reuse: 8033552512
rx24_cache_full: 72465843
rx24_cache_empty: 37376
rx24_cache_busy: 72467789
rx24_cache_waive: 0
rx24_congst_umr: 0
rx24_arfs_err: 0
rx24_xdp_tx_xmit: 0
rx24_xdp_tx_full: 0
rx24_xdp_tx_err: 0
rx24_xdp_tx_cqes: 0
rx25_packets: 16412186257
rx25_bytes: 21651198388407
rx25_csum_complete: 16412186257
rx25_csum_unnecessary: 0
rx25_csum_unnecessary_inner: 0
rx25_csum_none: 0
rx25_xdp_drop: 0
rx25_xdp_redirect: 0
rx25_lro_packets: 0
rx25_lro_bytes: 0
rx25_ecn_mark: 0
rx25_removed_vlan_packets: 16412186257
rx25_wqe_err: 0
rx25_mpwqe_filler_cqes: 0
rx25_mpwqe_filler_strides: 0
rx25_buff_alloc_err: 0
rx25_cqe_compress_blks: 383979685
rx25_cqe_compress_pkts: 861985772
rx25_page_reuse: 0
rx25_cache_reuse: 8129807841
rx25_cache_full: 76283342
rx25_cache_empty: 37376
rx25_cache_busy: 76285271
rx25_cache_waive: 0
rx25_congst_umr: 0
rx25_arfs_err: 0
rx25_xdp_tx_xmit: 0
rx25_xdp_tx_full: 0
rx25_xdp_tx_err: 0
rx25_xdp_tx_cqes: 0
rx26_packets: 16304310003
rx26_bytes: 21571217538721
rx26_csum_complete: 16304310003
rx26_csum_unnecessary: 0
rx26_csum_unnecessary_inner: 0
rx26_csum_none: 0
rx26_xdp_drop: 0
rx26_xdp_redirect: 0
rx26_lro_packets: 0
rx26_lro_bytes: 0
rx26_ecn_mark: 0
rx26_removed_vlan_packets: 16304310003
rx26_wqe_err: 0
rx26_mpwqe_filler_cqes: 0
rx26_mpwqe_filler_strides: 0
rx26_buff_alloc_err: 0
rx26_cqe_compress_blks: 353314041
rx26_cqe_compress_pkts: 788838424
rx26_page_reuse: 0
rx26_cache_reuse: 19673790
rx26_cache_full: 8132478659
rx26_cache_empty: 37376
rx26_cache_busy: 8132481198
rx26_cache_waive: 0
rx26_congst_umr: 0
rx26_arfs_err: 0
rx26_xdp_tx_xmit: 0
rx26_xdp_tx_full: 0
rx26_xdp_tx_err: 0
rx26_xdp_tx_cqes: 0
rx27_packets: 16171856079
rx27_bytes: 21376891736540
rx27_csum_complete: 16171856079
rx27_csum_unnecessary: 0
rx27_csum_unnecessary_inner: 0
rx27_csum_none: 0
rx27_xdp_drop: 0
rx27_xdp_redirect: 0
rx27_lro_packets: 0
rx27_lro_bytes: 0
rx27_ecn_mark: 0
rx27_removed_vlan_packets: 16171856079
rx27_wqe_err: 0
rx27_mpwqe_filler_cqes: 0
rx27_mpwqe_filler_strides: 0
rx27_buff_alloc_err: 0
rx27_cqe_compress_blks: 386632845
rx27_cqe_compress_pkts: 869362576
rx27_page_reuse: 0
rx27_cache_reuse: 10070560
rx27_cache_full: 8075854928
rx27_cache_empty: 37376
rx27_cache_busy: 8075857468
rx27_cache_waive: 0
rx27_congst_umr: 0
rx27_arfs_err: 0
rx27_xdp_tx_xmit: 0
rx27_xdp_tx_full: 0
rx27_xdp_tx_err: 0
rx27_xdp_tx_cqes: 0
rx28_packets: 0
rx28_bytes: 0
rx28_csum_complete: 0
rx28_csum_unnecessary: 0
rx28_csum_unnecessary_inner: 0
rx28_csum_none: 0
rx28_xdp_drop: 0
rx28_xdp_redirect: 0
rx28_lro_packets: 0
rx28_lro_bytes: 0
rx28_ecn_mark: 0
rx28_removed_vlan_packets: 0
rx28_wqe_err: 0
rx28_mpwqe_filler_cqes: 0
rx28_mpwqe_filler_strides: 0
rx28_buff_alloc_err: 0
rx28_cqe_compress_blks: 0
rx28_cqe_compress_pkts: 0
rx28_page_reuse: 0
rx28_cache_reuse: 0
rx28_cache_full: 0
rx28_cache_empty: 2560
rx28_cache_busy: 0
rx28_cache_waive: 0
rx28_congst_umr: 0
rx28_arfs_err: 0
rx28_xdp_tx_xmit: 0
rx28_xdp_tx_full: 0
rx28_xdp_tx_err: 0
rx28_xdp_tx_cqes: 0
rx29_packets: 0
rx29_bytes: 0
rx29_csum_complete: 0
rx29_csum_unnecessary: 0
rx29_csum_unnecessary_inner: 0
rx29_csum_none: 0
rx29_xdp_drop: 0
rx29_xdp_redirect: 0
rx29_lro_packets: 0
rx29_lro_bytes: 0
rx29_ecn_mark: 0
rx29_removed_vlan_packets: 0
rx29_wqe_err: 0
rx29_mpwqe_filler_cqes: 0
rx29_mpwqe_filler_strides: 0
rx29_buff_alloc_err: 0
rx29_cqe_compress_blks: 0
rx29_cqe_compress_pkts: 0
rx29_page_reuse: 0
rx29_cache_reuse: 0
rx29_cache_full: 0
rx29_cache_empty: 2560
rx29_cache_busy: 0
rx29_cache_waive: 0
rx29_congst_umr: 0
rx29_arfs_err: 0
rx29_xdp_tx_xmit: 0
rx29_xdp_tx_full: 0
rx29_xdp_tx_err: 0
rx29_xdp_tx_cqes: 0
rx30_packets: 0
rx30_bytes: 0
rx30_csum_complete: 0
rx30_csum_unnecessary: 0
rx30_csum_unnecessary_inner: 0
rx30_csum_none: 0
rx30_xdp_drop: 0
rx30_xdp_redirect: 0
rx30_lro_packets: 0
rx30_lro_bytes: 0
rx30_ecn_mark: 0
rx30_removed_vlan_packets: 0
rx30_wqe_err: 0
rx30_mpwqe_filler_cqes: 0
rx30_mpwqe_filler_strides: 0
rx30_buff_alloc_err: 0
rx30_cqe_compress_blks: 0
rx30_cqe_compress_pkts: 0
rx30_page_reuse: 0
rx30_cache_reuse: 0
rx30_cache_full: 0
rx30_cache_empty: 2560
rx30_cache_busy: 0
rx30_cache_waive: 0
rx30_congst_umr: 0
rx30_arfs_err: 0
rx30_xdp_tx_xmit: 0
rx30_xdp_tx_full: 0
rx30_xdp_tx_err: 0
rx30_xdp_tx_cqes: 0
rx31_packets: 0
rx31_bytes: 0
rx31_csum_complete: 0
rx31_csum_unnecessary: 0
rx31_csum_unnecessary_inner: 0
rx31_csum_none: 0
rx31_xdp_drop: 0
rx31_xdp_redirect: 0
rx31_lro_packets: 0
rx31_lro_bytes: 0
rx31_ecn_mark: 0
rx31_removed_vlan_packets: 0
rx31_wqe_err: 0
rx31_mpwqe_filler_cqes: 0
rx31_mpwqe_filler_strides: 0
rx31_buff_alloc_err: 0
rx31_cqe_compress_blks: 0
rx31_cqe_compress_pkts: 0
rx31_page_reuse: 0
rx31_cache_reuse: 0
rx31_cache_full: 0
rx31_cache_empty: 2560
rx31_cache_busy: 0
rx31_cache_waive: 0
rx31_congst_umr: 0
rx31_arfs_err: 0
rx31_xdp_tx_xmit: 0
rx31_xdp_tx_full: 0
rx31_xdp_tx_err: 0
rx31_xdp_tx_cqes: 0
rx32_packets: 0
rx32_bytes: 0
rx32_csum_complete: 0
rx32_csum_unnecessary: 0
rx32_csum_unnecessary_inner: 0
rx32_csum_none: 0
rx32_xdp_drop: 0
rx32_xdp_redirect: 0
rx32_lro_packets: 0
rx32_lro_bytes: 0
rx32_ecn_mark: 0
rx32_removed_vlan_packets: 0
rx32_wqe_err: 0
rx32_mpwqe_filler_cqes: 0
rx32_mpwqe_filler_strides: 0
rx32_buff_alloc_err: 0
rx32_cqe_compress_blks: 0
rx32_cqe_compress_pkts: 0
rx32_page_reuse: 0
rx32_cache_reuse: 0
rx32_cache_full: 0
rx32_cache_empty: 2560
rx32_cache_busy: 0
rx32_cache_waive: 0
rx32_congst_umr: 0
rx32_arfs_err: 0
rx32_xdp_tx_xmit: 0
rx32_xdp_tx_full: 0
rx32_xdp_tx_err: 0
rx32_xdp_tx_cqes: 0
rx33_packets: 0
rx33_bytes: 0
rx33_csum_complete: 0
rx33_csum_unnecessary: 0
rx33_csum_unnecessary_inner: 0
rx33_csum_none: 0
rx33_xdp_drop: 0
rx33_xdp_redirect: 0
rx33_lro_packets: 0
rx33_lro_bytes: 0
rx33_ecn_mark: 0
rx33_removed_vlan_packets: 0
rx33_wqe_err: 0
rx33_mpwqe_filler_cqes: 0
rx33_mpwqe_filler_strides: 0
rx33_buff_alloc_err: 0
rx33_cqe_compress_blks: 0
rx33_cqe_compress_pkts: 0
rx33_page_reuse: 0
rx33_cache_reuse: 0
rx33_cache_full: 0
rx33_cache_empty: 2560
rx33_cache_busy: 0
rx33_cache_waive: 0
rx33_congst_umr: 0
rx33_arfs_err: 0
rx33_xdp_tx_xmit: 0
rx33_xdp_tx_full: 0
rx33_xdp_tx_err: 0
rx33_xdp_tx_cqes: 0
rx34_packets: 0
rx34_bytes: 0
rx34_csum_complete: 0
rx34_csum_unnecessary: 0
rx34_csum_unnecessary_inner: 0
rx34_csum_none: 0
rx34_xdp_drop: 0
rx34_xdp_redirect: 0
rx34_lro_packets: 0
rx34_lro_bytes: 0
rx34_ecn_mark: 0
rx34_removed_vlan_packets: 0
rx34_wqe_err: 0
rx34_mpwqe_filler_cqes: 0
rx34_mpwqe_filler_strides: 0
rx34_buff_alloc_err: 0
rx34_cqe_compress_blks: 0
rx34_cqe_compress_pkts: 0
rx34_page_reuse: 0
rx34_cache_reuse: 0
rx34_cache_full: 0
rx34_cache_empty: 2560
rx34_cache_busy: 0
rx34_cache_waive: 0
rx34_congst_umr: 0
rx34_arfs_err: 0
rx34_xdp_tx_xmit: 0
rx34_xdp_tx_full: 0
rx34_xdp_tx_err: 0
rx34_xdp_tx_cqes: 0
rx35_packets: 0
rx35_bytes: 0
rx35_csum_complete: 0
rx35_csum_unnecessary: 0
rx35_csum_unnecessary_inner: 0
rx35_csum_none: 0
rx35_xdp_drop: 0
rx35_xdp_redirect: 0
rx35_lro_packets: 0
rx35_lro_bytes: 0
rx35_ecn_mark: 0
rx35_removed_vlan_packets: 0
rx35_wqe_err: 0
rx35_mpwqe_filler_cqes: 0
rx35_mpwqe_filler_strides: 0
rx35_buff_alloc_err: 0
rx35_cqe_compress_blks: 0
rx35_cqe_compress_pkts: 0
rx35_page_reuse: 0
rx35_cache_reuse: 0
rx35_cache_full: 0
rx35_cache_empty: 2560
rx35_cache_busy: 0
rx35_cache_waive: 0
rx35_congst_umr: 0
rx35_arfs_err: 0
rx35_xdp_tx_xmit: 0
rx35_xdp_tx_full: 0
rx35_xdp_tx_err: 0
rx35_xdp_tx_cqes: 0
rx36_packets: 0
rx36_bytes: 0
rx36_csum_complete: 0
rx36_csum_unnecessary: 0
rx36_csum_unnecessary_inner: 0
rx36_csum_none: 0
rx36_xdp_drop: 0
rx36_xdp_redirect: 0
rx36_lro_packets: 0
rx36_lro_bytes: 0
rx36_ecn_mark: 0
rx36_removed_vlan_packets: 0
rx36_wqe_err: 0
rx36_mpwqe_filler_cqes: 0
rx36_mpwqe_filler_strides: 0
rx36_buff_alloc_err: 0
rx36_cqe_compress_blks: 0
rx36_cqe_compress_pkts: 0
rx36_page_reuse: 0
rx36_cache_reuse: 0
rx36_cache_full: 0
rx36_cache_empty: 2560
rx36_cache_busy: 0
rx36_cache_waive: 0
rx36_congst_umr: 0
rx36_arfs_err: 0
rx36_xdp_tx_xmit: 0
rx36_xdp_tx_full: 0
rx36_xdp_tx_err: 0
rx36_xdp_tx_cqes: 0
rx37_packets: 0
rx37_bytes: 0
rx37_csum_complete: 0
rx37_csum_unnecessary: 0
rx37_csum_unnecessary_inner: 0
rx37_csum_none: 0
rx37_xdp_drop: 0
rx37_xdp_redirect: 0
rx37_lro_packets: 0
rx37_lro_bytes: 0
rx37_ecn_mark: 0
rx37_removed_vlan_packets: 0
rx37_wqe_err: 0
rx37_mpwqe_filler_cqes: 0
rx37_mpwqe_filler_strides: 0
rx37_buff_alloc_err: 0
rx37_cqe_compress_blks: 0
rx37_cqe_compress_pkts: 0
rx37_page_reuse: 0
rx37_cache_reuse: 0
rx37_cache_full: 0
rx37_cache_empty: 2560
rx37_cache_busy: 0
rx37_cache_waive: 0
rx37_congst_umr: 0
rx37_arfs_err: 0
rx37_xdp_tx_xmit: 0
rx37_xdp_tx_full: 0
rx37_xdp_tx_err: 0
rx37_xdp_tx_cqes: 0
rx38_packets: 0
rx38_bytes: 0
rx38_csum_complete: 0
rx38_csum_unnecessary: 0
rx38_csum_unnecessary_inner: 0
rx38_csum_none: 0
rx38_xdp_drop: 0
rx38_xdp_redirect: 0
rx38_lro_packets: 0
rx38_lro_bytes: 0
rx38_ecn_mark: 0
rx38_removed_vlan_packets: 0
rx38_wqe_err: 0
rx38_mpwqe_filler_cqes: 0
rx38_mpwqe_filler_strides: 0
rx38_buff_alloc_err: 0
rx38_cqe_compress_blks: 0
rx38_cqe_compress_pkts: 0
rx38_page_reuse: 0
rx38_cache_reuse: 0
rx38_cache_full: 0
rx38_cache_empty: 2560
rx38_cache_busy: 0
rx38_cache_waive: 0
rx38_congst_umr: 0
rx38_arfs_err: 0
rx38_xdp_tx_xmit: 0
rx38_xdp_tx_full: 0
rx38_xdp_tx_err: 0
rx38_xdp_tx_cqes: 0
rx39_packets: 0
rx39_bytes: 0
rx39_csum_complete: 0
rx39_csum_unnecessary: 0
rx39_csum_unnecessary_inner: 0
rx39_csum_none: 0
rx39_xdp_drop: 0
rx39_xdp_redirect: 0
rx39_lro_packets: 0
rx39_lro_bytes: 0
rx39_ecn_mark: 0
rx39_removed_vlan_packets: 0
rx39_wqe_err: 0
rx39_mpwqe_filler_cqes: 0
rx39_mpwqe_filler_strides: 0
rx39_buff_alloc_err: 0
rx39_cqe_compress_blks: 0
rx39_cqe_compress_pkts: 0
rx39_page_reuse: 0
rx39_cache_reuse: 0
rx39_cache_full: 0
rx39_cache_empty: 2560
rx39_cache_busy: 0
rx39_cache_waive: 0
rx39_congst_umr: 0
rx39_arfs_err: 0
rx39_xdp_tx_xmit: 0
rx39_xdp_tx_full: 0
rx39_xdp_tx_err: 0
rx39_xdp_tx_cqes: 0
rx40_packets: 0
rx40_bytes: 0
rx40_csum_complete: 0
rx40_csum_unnecessary: 0
rx40_csum_unnecessary_inner: 0
rx40_csum_none: 0
rx40_xdp_drop: 0
rx40_xdp_redirect: 0
rx40_lro_packets: 0
rx40_lro_bytes: 0
rx40_ecn_mark: 0
rx40_removed_vlan_packets: 0
rx40_wqe_err: 0
rx40_mpwqe_filler_cqes: 0
rx40_mpwqe_filler_strides: 0
rx40_buff_alloc_err: 0
rx40_cqe_compress_blks: 0
rx40_cqe_compress_pkts: 0
rx40_page_reuse: 0
rx40_cache_reuse: 0
rx40_cache_full: 0
rx40_cache_empty: 2560
rx40_cache_busy: 0
rx40_cache_waive: 0
rx40_congst_umr: 0
rx40_arfs_err: 0
rx40_xdp_tx_xmit: 0
rx40_xdp_tx_full: 0
rx40_xdp_tx_err: 0
rx40_xdp_tx_cqes: 0
rx41_packets: 0
rx41_bytes: 0
rx41_csum_complete: 0
rx41_csum_unnecessary: 0
rx41_csum_unnecessary_inner: 0
rx41_csum_none: 0
rx41_xdp_drop: 0
rx41_xdp_redirect: 0
rx41_lro_packets: 0
rx41_lro_bytes: 0
rx41_ecn_mark: 0
rx41_removed_vlan_packets: 0
rx41_wqe_err: 0
rx41_mpwqe_filler_cqes: 0
rx41_mpwqe_filler_strides: 0
rx41_buff_alloc_err: 0
rx41_cqe_compress_blks: 0
rx41_cqe_compress_pkts: 0
rx41_page_reuse: 0
rx41_cache_reuse: 0
rx41_cache_full: 0
rx41_cache_empty: 2560
rx41_cache_busy: 0
rx41_cache_waive: 0
rx41_congst_umr: 0
rx41_arfs_err: 0
rx41_xdp_tx_xmit: 0
rx41_xdp_tx_full: 0
rx41_xdp_tx_err: 0
rx41_xdp_tx_cqes: 0
rx42_packets: 0
rx42_bytes: 0
rx42_csum_complete: 0
rx42_csum_unnecessary: 0
rx42_csum_unnecessary_inner: 0
rx42_csum_none: 0
rx42_xdp_drop: 0
rx42_xdp_redirect: 0
rx42_lro_packets: 0
rx42_lro_bytes: 0
rx42_ecn_mark: 0
rx42_removed_vlan_packets: 0
rx42_wqe_err: 0
rx42_mpwqe_filler_cqes: 0
rx42_mpwqe_filler_strides: 0
rx42_buff_alloc_err: 0
rx42_cqe_compress_blks: 0
rx42_cqe_compress_pkts: 0
rx42_page_reuse: 0
rx42_cache_reuse: 0
rx42_cache_full: 0
rx42_cache_empty: 2560
rx42_cache_busy: 0
rx42_cache_waive: 0
rx42_congst_umr: 0
rx42_arfs_err: 0
rx42_xdp_tx_xmit: 0
rx42_xdp_tx_full: 0
rx42_xdp_tx_err: 0
rx42_xdp_tx_cqes: 0
rx43_packets: 0
rx43_bytes: 0
rx43_csum_complete: 0
rx43_csum_unnecessary: 0
rx43_csum_unnecessary_inner: 0
rx43_csum_none: 0
rx43_xdp_drop: 0
rx43_xdp_redirect: 0
rx43_lro_packets: 0
rx43_lro_bytes: 0
rx43_ecn_mark: 0
rx43_removed_vlan_packets: 0
rx43_wqe_err: 0
rx43_mpwqe_filler_cqes: 0
rx43_mpwqe_filler_strides: 0
rx43_buff_alloc_err: 0
rx43_cqe_compress_blks: 0
rx43_cqe_compress_pkts: 0
rx43_page_reuse: 0
rx43_cache_reuse: 0
rx43_cache_full: 0
rx43_cache_empty: 2560
rx43_cache_busy: 0
rx43_cache_waive: 0
rx43_congst_umr: 0
rx43_arfs_err: 0
rx43_xdp_tx_xmit: 0
rx43_xdp_tx_full: 0
rx43_xdp_tx_err: 0
rx43_xdp_tx_cqes: 0
rx44_packets: 0
rx44_bytes: 0
rx44_csum_complete: 0
rx44_csum_unnecessary: 0
rx44_csum_unnecessary_inner: 0
rx44_csum_none: 0
rx44_xdp_drop: 0
rx44_xdp_redirect: 0
rx44_lro_packets: 0
rx44_lro_bytes: 0
rx44_ecn_mark: 0
rx44_removed_vlan_packets: 0
rx44_wqe_err: 0
rx44_mpwqe_filler_cqes: 0
rx44_mpwqe_filler_strides: 0
rx44_buff_alloc_err: 0
rx44_cqe_compress_blks: 0
rx44_cqe_compress_pkts: 0
rx44_page_reuse: 0
rx44_cache_reuse: 0
rx44_cache_full: 0
rx44_cache_empty: 2560
rx44_cache_busy: 0
rx44_cache_waive: 0
rx44_congst_umr: 0
rx44_arfs_err: 0
rx44_xdp_tx_xmit: 0
rx44_xdp_tx_full: 0
rx44_xdp_tx_err: 0
rx44_xdp_tx_cqes: 0
rx45_packets: 0
rx45_bytes: 0
rx45_csum_complete: 0
rx45_csum_unnecessary: 0
rx45_csum_unnecessary_inner: 0
rx45_csum_none: 0
rx45_xdp_drop: 0
rx45_xdp_redirect: 0
rx45_lro_packets: 0
rx45_lro_bytes: 0
rx45_ecn_mark: 0
rx45_removed_vlan_packets: 0
rx45_wqe_err: 0
rx45_mpwqe_filler_cqes: 0
rx45_mpwqe_filler_strides: 0
rx45_buff_alloc_err: 0
rx45_cqe_compress_blks: 0
rx45_cqe_compress_pkts: 0
rx45_page_reuse: 0
rx45_cache_reuse: 0
rx45_cache_full: 0
rx45_cache_empty: 2560
rx45_cache_busy: 0
rx45_cache_waive: 0
rx45_congst_umr: 0
rx45_arfs_err: 0
rx45_xdp_tx_xmit: 0
rx45_xdp_tx_full: 0
rx45_xdp_tx_err: 0
rx45_xdp_tx_cqes: 0
rx46_packets: 0
rx46_bytes: 0
rx46_csum_complete: 0
rx46_csum_unnecessary: 0
rx46_csum_unnecessary_inner: 0
rx46_csum_none: 0
rx46_xdp_drop: 0
rx46_xdp_redirect: 0
rx46_lro_packets: 0
rx46_lro_bytes: 0
rx46_ecn_mark: 0
rx46_removed_vlan_packets: 0
rx46_wqe_err: 0
rx46_mpwqe_filler_cqes: 0
rx46_mpwqe_filler_strides: 0
rx46_buff_alloc_err: 0
rx46_cqe_compress_blks: 0
rx46_cqe_compress_pkts: 0
rx46_page_reuse: 0
rx46_cache_reuse: 0
rx46_cache_full: 0
rx46_cache_empty: 2560
rx46_cache_busy: 0
rx46_cache_waive: 0
rx46_congst_umr: 0
rx46_arfs_err: 0
rx46_xdp_tx_xmit: 0
rx46_xdp_tx_full: 0
rx46_xdp_tx_err: 0
rx46_xdp_tx_cqes: 0
rx47_packets: 0
rx47_bytes: 0
rx47_csum_complete: 0
rx47_csum_unnecessary: 0
rx47_csum_unnecessary_inner: 0
rx47_csum_none: 0
rx47_xdp_drop: 0
rx47_xdp_redirect: 0
rx47_lro_packets: 0
rx47_lro_bytes: 0
rx47_ecn_mark: 0
rx47_removed_vlan_packets: 0
rx47_wqe_err: 0
rx47_mpwqe_filler_cqes: 0
rx47_mpwqe_filler_strides: 0
rx47_buff_alloc_err: 0
rx47_cqe_compress_blks: 0
rx47_cqe_compress_pkts: 0
rx47_page_reuse: 0
rx47_cache_reuse: 0
rx47_cache_full: 0
rx47_cache_empty: 2560
rx47_cache_busy: 0
rx47_cache_waive: 0
rx47_congst_umr: 0
rx47_arfs_err: 0
rx47_xdp_tx_xmit: 0
rx47_xdp_tx_full: 0
rx47_xdp_tx_err: 0
rx47_xdp_tx_cqes: 0
rx48_packets: 0
rx48_bytes: 0
rx48_csum_complete: 0
rx48_csum_unnecessary: 0
rx48_csum_unnecessary_inner: 0
rx48_csum_none: 0
rx48_xdp_drop: 0
rx48_xdp_redirect: 0
rx48_lro_packets: 0
rx48_lro_bytes: 0
rx48_ecn_mark: 0
rx48_removed_vlan_packets: 0
rx48_wqe_err: 0
rx48_mpwqe_filler_cqes: 0
rx48_mpwqe_filler_strides: 0
rx48_buff_alloc_err: 0
rx48_cqe_compress_blks: 0
rx48_cqe_compress_pkts: 0
rx48_page_reuse: 0
rx48_cache_reuse: 0
rx48_cache_full: 0
rx48_cache_empty: 2560
rx48_cache_busy: 0
rx48_cache_waive: 0
rx48_congst_umr: 0
rx48_arfs_err: 0
rx48_xdp_tx_xmit: 0
rx48_xdp_tx_full: 0
rx48_xdp_tx_err: 0
rx48_xdp_tx_cqes: 0
rx49_packets: 0
rx49_bytes: 0
rx49_csum_complete: 0
rx49_csum_unnecessary: 0
rx49_csum_unnecessary_inner: 0
rx49_csum_none: 0
rx49_xdp_drop: 0
rx49_xdp_redirect: 0
rx49_lro_packets: 0
rx49_lro_bytes: 0
rx49_ecn_mark: 0
rx49_removed_vlan_packets: 0
rx49_wqe_err: 0
rx49_mpwqe_filler_cqes: 0
rx49_mpwqe_filler_strides: 0
rx49_buff_alloc_err: 0
rx49_cqe_compress_blks: 0
rx49_cqe_compress_pkts: 0
rx49_page_reuse: 0
rx49_cache_reuse: 0
rx49_cache_full: 0
rx49_cache_empty: 2560
rx49_cache_busy: 0
rx49_cache_waive: 0
rx49_congst_umr: 0
rx49_arfs_err: 0
rx49_xdp_tx_xmit: 0
rx49_xdp_tx_full: 0
rx49_xdp_tx_err: 0
rx49_xdp_tx_cqes: 0
rx50_packets: 0
rx50_bytes: 0
rx50_csum_complete: 0
rx50_csum_unnecessary: 0
rx50_csum_unnecessary_inner: 0
rx50_csum_none: 0
rx50_xdp_drop: 0
rx50_xdp_redirect: 0
rx50_lro_packets: 0
rx50_lro_bytes: 0
rx50_ecn_mark: 0
rx50_removed_vlan_packets: 0
rx50_wqe_err: 0
rx50_mpwqe_filler_cqes: 0
rx50_mpwqe_filler_strides: 0
rx50_buff_alloc_err: 0
rx50_cqe_compress_blks: 0
rx50_cqe_compress_pkts: 0
rx50_page_reuse: 0
rx50_cache_reuse: 0
rx50_cache_full: 0
rx50_cache_empty: 2560
rx50_cache_busy: 0
rx50_cache_waive: 0
rx50_congst_umr: 0
rx50_arfs_err: 0
rx50_xdp_tx_xmit: 0
rx50_xdp_tx_full: 0
rx50_xdp_tx_err: 0
rx50_xdp_tx_cqes: 0
rx51_packets: 0
rx51_bytes: 0
rx51_csum_complete: 0
rx51_csum_unnecessary: 0
rx51_csum_unnecessary_inner: 0
rx51_csum_none: 0
rx51_xdp_drop: 0
rx51_xdp_redirect: 0
rx51_lro_packets: 0
rx51_lro_bytes: 0
rx51_ecn_mark: 0
rx51_removed_vlan_packets: 0
rx51_wqe_err: 0
rx51_mpwqe_filler_cqes: 0
rx51_mpwqe_filler_strides: 0
rx51_buff_alloc_err: 0
rx51_cqe_compress_blks: 0
rx51_cqe_compress_pkts: 0
rx51_page_reuse: 0
rx51_cache_reuse: 0
rx51_cache_full: 0
rx51_cache_empty: 2560
rx51_cache_busy: 0
rx51_cache_waive: 0
rx51_congst_umr: 0
rx51_arfs_err: 0
rx51_xdp_tx_xmit: 0
rx51_xdp_tx_full: 0
rx51_xdp_tx_err: 0
rx51_xdp_tx_cqes: 0
rx52_packets: 0
rx52_bytes: 0
rx52_csum_complete: 0
rx52_csum_unnecessary: 0
rx52_csum_unnecessary_inner: 0
rx52_csum_none: 0
rx52_xdp_drop: 0
rx52_xdp_redirect: 0
rx52_lro_packets: 0
rx52_lro_bytes: 0
rx52_ecn_mark: 0
rx52_removed_vlan_packets: 0
rx52_wqe_err: 0
rx52_mpwqe_filler_cqes: 0
rx52_mpwqe_filler_strides: 0
rx52_buff_alloc_err: 0
rx52_cqe_compress_blks: 0
rx52_cqe_compress_pkts: 0
rx52_page_reuse: 0
rx52_cache_reuse: 0
rx52_cache_full: 0
rx52_cache_empty: 2560
rx52_cache_busy: 0
rx52_cache_waive: 0
rx52_congst_umr: 0
rx52_arfs_err: 0
rx52_xdp_tx_xmit: 0
rx52_xdp_tx_full: 0
rx52_xdp_tx_err: 0
rx52_xdp_tx_cqes: 0
rx53_packets: 0
rx53_bytes: 0
rx53_csum_complete: 0
rx53_csum_unnecessary: 0
rx53_csum_unnecessary_inner: 0
rx53_csum_none: 0
rx53_xdp_drop: 0
rx53_xdp_redirect: 0
rx53_lro_packets: 0
rx53_lro_bytes: 0
rx53_ecn_mark: 0
rx53_removed_vlan_packets: 0
rx53_wqe_err: 0
rx53_mpwqe_filler_cqes: 0
rx53_mpwqe_filler_strides: 0
rx53_buff_alloc_err: 0
rx53_cqe_compress_blks: 0
rx53_cqe_compress_pkts: 0
rx53_page_reuse: 0
rx53_cache_reuse: 0
rx53_cache_full: 0
rx53_cache_empty: 2560
rx53_cache_busy: 0
rx53_cache_waive: 0
rx53_congst_umr: 0
rx53_arfs_err: 0
rx53_xdp_tx_xmit: 0
rx53_xdp_tx_full: 0
rx53_xdp_tx_err: 0
rx53_xdp_tx_cqes: 0
rx54_packets: 0
rx54_bytes: 0
rx54_csum_complete: 0
rx54_csum_unnecessary: 0
rx54_csum_unnecessary_inner: 0
rx54_csum_none: 0
rx54_xdp_drop: 0
rx54_xdp_redirect: 0
rx54_lro_packets: 0
rx54_lro_bytes: 0
rx54_ecn_mark: 0
rx54_removed_vlan_packets: 0
rx54_wqe_err: 0
rx54_mpwqe_filler_cqes: 0
rx54_mpwqe_filler_strides: 0
rx54_buff_alloc_err: 0
rx54_cqe_compress_blks: 0
rx54_cqe_compress_pkts: 0
rx54_page_reuse: 0
rx54_cache_reuse: 0
rx54_cache_full: 0
rx54_cache_empty: 2560
rx54_cache_busy: 0
rx54_cache_waive: 0
rx54_congst_umr: 0
rx54_arfs_err: 0
rx54_xdp_tx_xmit: 0
rx54_xdp_tx_full: 0
rx54_xdp_tx_err: 0
rx54_xdp_tx_cqes: 0
rx55_packets: 0
rx55_bytes: 0
rx55_csum_complete: 0
rx55_csum_unnecessary: 0
rx55_csum_unnecessary_inner: 0
rx55_csum_none: 0
rx55_xdp_drop: 0
rx55_xdp_redirect: 0
rx55_lro_packets: 0
rx55_lro_bytes: 0
rx55_ecn_mark: 0
rx55_removed_vlan_packets: 0
rx55_wqe_err: 0
rx55_mpwqe_filler_cqes: 0
rx55_mpwqe_filler_strides: 0
rx55_buff_alloc_err: 0
rx55_cqe_compress_blks: 0
rx55_cqe_compress_pkts: 0
rx55_page_reuse: 0
rx55_cache_reuse: 0
rx55_cache_full: 0
rx55_cache_empty: 2560
rx55_cache_busy: 0
rx55_cache_waive: 0
rx55_congst_umr: 0
rx55_arfs_err: 0
rx55_xdp_tx_xmit: 0
rx55_xdp_tx_full: 0
rx55_xdp_tx_err: 0
rx55_xdp_tx_cqes: 0
tx0_packets: 24512439668
tx0_bytes: 15287569052791
tx0_tso_packets: 1536157106
tx0_tso_bytes: 8571753637944
tx0_tso_inner_packets: 0
tx0_tso_inner_bytes: 0
tx0_csum_partial: 2132156117
tx0_csum_partial_inner: 0
tx0_added_vlan_packets: 19906601448
tx0_nop: 308098536
tx0_csum_none: 17774445331
tx0_stopped: 19625
tx0_dropped: 0
tx0_xmit_more: 67864870
tx0_recover: 0
tx0_cqes: 19838744246
tx0_wake: 19624
tx0_cqe_err: 0
tx1_packets: 22598557053
tx1_bytes: 13568850145010
tx1_tso_packets: 1369529475
tx1_tso_bytes: 7661777265382
tx1_tso_inner_packets: 0
tx1_tso_inner_bytes: 0
tx1_csum_partial: 1884639496
tx1_csum_partial_inner: 0
tx1_added_vlan_packets: 18468333696
tx1_nop: 281301783
tx1_csum_none: 16583694200
tx1_stopped: 19457
tx1_dropped: 0
tx1_xmit_more: 55170875
tx1_recover: 0
tx1_cqes: 18413169824
tx1_wake: 19455
tx1_cqe_err: 0
tx2_packets: 22821611433
tx2_bytes: 13752535163683
tx2_tso_packets: 1396978825
tx2_tso_bytes: 7774704508463
tx2_tso_inner_packets: 0
tx2_tso_inner_bytes: 0
tx2_csum_partial: 1897834558
tx2_csum_partial_inner: 0
tx2_added_vlan_packets: 18641958085
tx2_nop: 286934891
tx2_csum_none: 16744123527
tx2_stopped: 13214
tx2_dropped: 0
tx2_xmit_more: 61749446
tx2_recover: 0
tx2_cqes: 18580215654
tx2_wake: 13214
tx2_cqe_err: 0
tx3_packets: 22580809948
tx3_bytes: 13730542936609
tx3_tso_packets: 1370434579
tx3_tso_bytes: 7605636711455
tx3_tso_inner_packets: 0
tx3_tso_inner_bytes: 0
tx3_csum_partial: 1865573748
tx3_csum_partial_inner: 0
tx3_added_vlan_packets: 18491873644
tx3_nop: 281195875
tx3_csum_none: 16626299896
tx3_stopped: 12542
tx3_dropped: 0
tx3_xmit_more: 57681647
tx3_recover: 0
tx3_cqes: 18434198757
tx3_wake: 12540
tx3_cqe_err: 0
tx4_packets: 27801801208
tx4_bytes: 17058453171137
tx4_tso_packets: 1740500105
tx4_tso_bytes: 9474905622036
tx4_tso_inner_packets: 0
tx4_tso_inner_bytes: 0
tx4_csum_partial: 2279225376
tx4_csum_partial_inner: 0
tx4_added_vlan_packets: 22744081633
tx4_nop: 349753979
tx4_csum_none: 20464856257
tx4_stopped: 14816
tx4_dropped: 0
tx4_xmit_more: 65469322
tx4_recover: 0
tx4_cqes: 22678618972
tx4_wake: 14816
tx4_cqe_err: 0
tx5_packets: 25099783024
tx5_bytes: 14917740698381
tx5_tso_packets: 1512988013
tx5_tso_bytes: 8571208921023
tx5_tso_inner_packets: 0
tx5_tso_inner_bytes: 0
tx5_csum_partial: 2078498561
tx5_csum_partial_inner: 0
tx5_added_vlan_packets: 20465533760
tx5_nop: 312614719
tx5_csum_none: 18387035199
tx5_stopped: 4605
tx5_dropped: 0
tx5_xmit_more: 64188936
tx5_recover: 0
tx5_cqes: 20401350718
tx5_wake: 4604
tx5_cqe_err: 0
tx6_packets: 25025504896
tx6_bytes: 14908021946070
tx6_tso_packets: 1515718977
tx6_tso_bytes: 8511442522461
tx6_tso_inner_packets: 0
tx6_tso_inner_bytes: 0
tx6_csum_partial: 2056378610
tx6_csum_partial_inner: 0
tx6_added_vlan_packets: 20434066400
tx6_nop: 310594020
tx6_csum_none: 18377687790
tx6_stopped: 15234
tx6_dropped: 0
tx6_xmit_more: 61130422
tx6_recover: 0
tx6_cqes: 20372943611
tx6_wake: 15234
tx6_cqe_err: 0
tx7_packets: 25457096169
tx7_bytes: 15456289446172
tx7_tso_packets: 1553342799
tx7_tso_bytes: 8764550988105
tx7_tso_inner_packets: 0
tx7_tso_inner_bytes: 0
tx7_csum_partial: 2105765233
tx7_csum_partial_inner: 0
tx7_added_vlan_packets: 20720382377
tx7_nop: 319044853
tx7_csum_none: 18614617145
tx7_stopped: 18745
tx7_dropped: 0
tx7_xmit_more: 57050107
tx7_recover: 0
tx7_cqes: 20663340775
tx7_wake: 18746
tx7_cqe_err: 0
tx8_packets: 25389771649
tx8_bytes: 15225503883962
tx8_tso_packets: 1563367648
tx8_tso_bytes: 8710384514258
tx8_tso_inner_packets: 0
tx8_tso_inner_bytes: 0
tx8_csum_partial: 2106586634
tx8_csum_partial_inner: 0
tx8_added_vlan_packets: 20704676274
tx8_nop: 318149261
tx8_csum_none: 18598089640
tx8_stopped: 4733
tx8_dropped: 0
tx8_xmit_more: 61014317
tx8_recover: 0
tx8_cqes: 20643667301
tx8_wake: 4735
tx8_cqe_err: 0
tx9_packets: 25521500166
tx9_bytes: 15302293145755
tx9_tso_packets: 1546316697
tx9_tso_bytes: 8770688145926
tx9_tso_inner_packets: 0
tx9_tso_inner_bytes: 0
tx9_csum_partial: 2097652880
tx9_csum_partial_inner: 0
tx9_added_vlan_packets: 20778408432
tx9_nop: 318538543
tx9_csum_none: 18680755556
tx9_stopped: 16118
tx9_dropped: 0
tx9_xmit_more: 68509728
tx9_recover: 0
tx9_cqes: 20709906498
tx9_wake: 16118
tx9_cqe_err: 0
tx10_packets: 25451605829
tx10_bytes: 15386896170792
tx10_tso_packets: 1576473520
tx10_tso_bytes: 8880888676383
tx10_tso_inner_packets: 0
tx10_tso_inner_bytes: 0
tx10_csum_partial: 2129796141
tx10_csum_partial_inner: 0
tx10_added_vlan_packets: 20659622590
tx10_nop: 319117433
tx10_csum_none: 18529826450
tx10_stopped: 20187
tx10_dropped: 0
tx10_xmit_more: 58892184
tx10_recover: 0
tx10_cqes: 20600737739
tx10_wake: 20188
tx10_cqe_err: 0
tx11_packets: 27008919793
tx11_bytes: 16587719213058
tx11_tso_packets: 1734884654
tx11_tso_bytes: 9475681471870
tx11_tso_inner_packets: 0
tx11_tso_inner_bytes: 0
tx11_csum_partial: 2296162292
tx11_csum_partial_inner: 0
tx11_added_vlan_packets: 21943096263
tx11_nop: 344188182
tx11_csum_none: 19646933971
tx11_stopped: 9703
tx11_dropped: 0
tx11_xmit_more: 66530718
tx11_recover: 0
tx11_cqes: 21876571667
tx11_wake: 9704
tx11_cqe_err: 0
tx12_packets: 25969493269
tx12_bytes: 15980767963416
tx12_tso_packets: 1671396456
tx12_tso_bytes: 9268973672821
tx12_tso_inner_packets: 0
tx12_tso_inner_bytes: 0
tx12_csum_partial: 2243809182
tx12_csum_partial_inner: 0
tx12_added_vlan_packets: 20980642456
tx12_nop: 330241007
tx12_csum_none: 18736833276
tx12_stopped: 10341
tx12_dropped: 0
tx12_xmit_more: 57834100
tx12_recover: 0
tx12_cqes: 20922815079
tx12_wake: 10342
tx12_cqe_err: 0
tx13_packets: 25332762261
tx13_bytes: 15353213283280
tx13_tso_packets: 1577433599
tx13_tso_bytes: 8785240284281
tx13_tso_inner_packets: 0
tx13_tso_inner_bytes: 0
tx13_csum_partial: 2110640515
tx13_csum_partial_inner: 0
tx13_added_vlan_packets: 20605670910
tx13_nop: 319805741
tx13_csum_none: 18495030395
tx13_stopped: 7006
tx13_dropped: 0
tx13_xmit_more: 58314402
tx13_recover: 0
tx13_cqes: 20547362770
tx13_wake: 7008
tx13_cqe_err: 0
tx14_packets: 26333743548
tx14_bytes: 16070719060573
tx14_tso_packets: 1677922970
tx14_tso_bytes: 9240299765487
tx14_tso_inner_packets: 0
tx14_tso_inner_bytes: 0
tx14_csum_partial: 2215668906
tx14_csum_partial_inner: 0
tx14_added_vlan_packets: 21384410786
tx14_nop: 332734939
tx14_csum_none: 19168741880
tx14_stopped: 13160
tx14_dropped: 0
tx14_xmit_more: 57650391
tx14_recover: 0
tx14_cqes: 21326767783
tx14_wake: 13161
tx14_cqe_err: 0
tx15_packets: 26824968971
tx15_bytes: 16687994233452
tx15_tso_packets: 1755745052
tx15_tso_bytes: 9533814012441
tx15_tso_inner_packets: 0
tx15_tso_inner_bytes: 0
tx15_csum_partial: 2304778064
tx15_csum_partial_inner: 0
tx15_added_vlan_packets: 21740906107
tx15_nop: 344143287
tx15_csum_none: 19436128058
tx15_stopped: 75
tx15_dropped: 0
tx15_xmit_more: 63325832
tx15_recover: 0
tx15_cqes: 21677585345
tx15_wake: 74
tx15_cqe_err: 0
tx16_packets: 24488158946
tx16_bytes: 15027415004570
tx16_tso_packets: 1559127391
tx16_tso_bytes: 8658691917845
tx16_tso_inner_packets: 0
tx16_tso_inner_bytes: 0
tx16_csum_partial: 2075856395
tx16_csum_partial_inner: 0
tx16_added_vlan_packets: 19835695731
tx16_nop: 308464189
tx16_csum_none: 17759839340
tx16_stopped: 4567
tx16_dropped: 0
tx16_xmit_more: 62631422
tx16_recover: 0
tx16_cqes: 19773070012
tx16_wake: 4568
tx16_cqe_err: 0
tx17_packets: 24700413784
tx17_bytes: 15216529713715
tx17_tso_packets: 1597555108
tx17_tso_bytes: 8773728661243
tx17_tso_inner_packets: 0
tx17_tso_inner_bytes: 0
tx17_csum_partial: 2127177297
tx17_csum_partial_inner: 0
tx17_added_vlan_packets: 20003144561
tx17_nop: 313356918
tx17_csum_none: 17875967264
tx17_stopped: 12572
tx17_dropped: 0
tx17_xmit_more: 62742980
tx17_recover: 0
tx17_cqes: 19940407615
tx17_wake: 12573
tx17_cqe_err: 0
tx18_packets: 24887710046
tx18_bytes: 15245034034664
tx18_tso_packets: 1582550520
tx18_tso_bytes: 8782692335483
tx18_tso_inner_packets: 0
tx18_tso_inner_bytes: 0
tx18_csum_partial: 2084514331
tx18_csum_partial_inner: 0
tx18_added_vlan_packets: 20173879181
tx18_nop: 314818702
tx18_csum_none: 18089364850
tx18_stopped: 21366
tx18_dropped: 0
tx18_xmit_more: 62485819
tx18_recover: 0
tx18_cqes: 20111400935
tx18_wake: 21366
tx18_cqe_err: 0
tx19_packets: 24831057648
tx19_bytes: 15164663890576
tx19_tso_packets: 1599135489
tx19_tso_bytes: 8756045449746
tx19_tso_inner_packets: 0
tx19_tso_inner_bytes: 0
tx19_csum_partial: 2119746608
tx19_csum_partial_inner: 0
tx19_added_vlan_packets: 20143573903
tx19_nop: 316966450
tx19_csum_none: 18023827295
tx19_stopped: 11431
tx19_dropped: 0
tx19_xmit_more: 57535904
tx19_recover: 0
tx19_cqes: 20086045325
tx19_wake: 11431
tx19_cqe_err: 0
tx20_packets: 21943735263
tx20_bytes: 13528749492187
tx20_tso_packets: 1390048103
tx20_tso_bytes: 7629058809637
tx20_tso_inner_packets: 0
tx20_tso_inner_bytes: 0
tx20_csum_partial: 1848533941
tx20_csum_partial_inner: 0
tx20_added_vlan_packets: 17861417651
tx20_nop: 276840365
tx20_csum_none: 16012883710
tx20_stopped: 38457
tx20_dropped: 0
tx20_xmit_more: 57042753
tx20_recover: 0
tx20_cqes: 17804384839
tx20_wake: 38457
tx20_cqe_err: 0
tx21_packets: 21476926958
tx21_bytes: 13096410597896
tx21_tso_packets: 1367724090
tx21_tso_bytes: 7568364585127
tx21_tso_inner_packets: 0
tx21_tso_inner_bytes: 0
tx21_csum_partial: 1830570727
tx21_csum_partial_inner: 0
tx21_added_vlan_packets: 17421087814
tx21_nop: 270611519
tx21_csum_none: 15590517087
tx21_stopped: 31213
tx21_dropped: 0
tx21_xmit_more: 60305389
tx21_recover: 0
tx21_cqes: 17360791205
tx21_wake: 31213
tx21_cqe_err: 0
tx22_packets: 21819106444
tx22_bytes: 13492871887100
tx22_tso_packets: 1387002018
tx22_tso_bytes: 7617705727669
tx22_tso_inner_packets: 0
tx22_tso_inner_bytes: 0
tx22_csum_partial: 1853632107
tx22_csum_partial_inner: 0
tx22_added_vlan_packets: 17743255447
tx22_nop: 274820992
tx22_csum_none: 15889623340
tx22_stopped: 24814
tx22_dropped: 0
tx22_xmit_more: 60811304
tx22_recover: 0
tx22_cqes: 17682451111
tx22_wake: 24815
tx22_cqe_err: 0
tx23_packets: 21830455800
tx23_bytes: 13427551902532
tx23_tso_packets: 1388556038
tx23_tso_bytes: 7604040587125
tx23_tso_inner_packets: 0
tx23_tso_inner_bytes: 0
tx23_csum_partial: 1850819694
tx23_csum_partial_inner: 0
tx23_added_vlan_packets: 17761271122
tx23_nop: 275142775
tx23_csum_none: 15910451428
tx23_stopped: 29899
tx23_dropped: 0
tx23_xmit_more: 58924909
tx23_recover: 0
tx23_cqes: 17702355187
tx23_wake: 29898
tx23_cqe_err: 0
tx24_packets: 21961484213
tx24_bytes: 13531373062497
tx24_tso_packets: 1394697504
tx24_tso_bytes: 7663866609308
tx24_tso_inner_packets: 0
tx24_tso_inner_bytes: 0
tx24_csum_partial: 1857072074
tx24_csum_partial_inner: 0
tx24_added_vlan_packets: 17856887568
tx24_nop: 276352855
tx24_csum_none: 15999815494
tx24_stopped: 33924
tx24_dropped: 0
tx24_xmit_more: 63992426
tx24_recover: 0
tx24_cqes: 17792905243
tx24_wake: 33923
tx24_cqe_err: 0
tx25_packets: 21853593838
tx25_bytes: 13357487830519
tx25_tso_packets: 1398822411
tx25_tso_bytes: 7691191518838
tx25_tso_inner_packets: 0
tx25_tso_inner_bytes: 0
tx25_csum_partial: 1869483109
tx25_csum_partial_inner: 0
tx25_added_vlan_packets: 17734634614
tx25_nop: 276327643
tx25_csum_none: 15865151505
tx25_stopped: 38651
tx25_dropped: 0
tx25_xmit_more: 56410535
tx25_recover: 0
tx25_cqes: 17678234537
tx25_wake: 38650
tx25_cqe_err: 0
tx26_packets: 21480261205
tx26_bytes: 13148973015935
tx26_tso_packets: 1348132284
tx26_tso_bytes: 7523489481775
tx26_tso_inner_packets: 0
tx26_tso_inner_bytes: 0
tx26_csum_partial: 1839740745
tx26_csum_partial_inner: 0
tx26_added_vlan_packets: 17430592911
tx26_nop: 270367836
tx26_csum_none: 15590852166
tx26_stopped: 34044
tx26_dropped: 0
tx26_xmit_more: 59870114
tx26_recover: 0
tx26_cqes: 17370736612
tx26_wake: 34043
tx26_cqe_err: 0
tx27_packets: 22694273108
tx27_bytes: 14135473431004
tx27_tso_packets: 1418371875
tx27_tso_bytes: 7784842263038
tx27_tso_inner_packets: 0
tx27_tso_inner_bytes: 0
tx27_csum_partial: 1919170584
tx27_csum_partial_inner: 0
tx27_added_vlan_packets: 18520826023
tx27_nop: 286296272
tx27_csum_none: 16601655439
tx27_stopped: 38125
tx27_dropped: 0
tx27_xmit_more: 72749775
tx27_recover: 0
tx27_cqes: 18448090270
tx27_wake: 38127
tx27_cqe_err: 0
tx28_packets: 0
tx28_bytes: 0
tx28_tso_packets: 0
tx28_tso_bytes: 0
tx28_tso_inner_packets: 0
tx28_tso_inner_bytes: 0
tx28_csum_partial: 0
tx28_csum_partial_inner: 0
tx28_added_vlan_packets: 0
tx28_nop: 0
tx28_csum_none: 0
tx28_stopped: 0
tx28_dropped: 0
tx28_xmit_more: 0
tx28_recover: 0
tx28_cqes: 0
tx28_wake: 0
tx28_cqe_err: 0
tx29_packets: 3
tx29_bytes: 266
tx29_tso_packets: 0
tx29_tso_bytes: 0
tx29_tso_inner_packets: 0
tx29_tso_inner_bytes: 0
tx29_csum_partial: 0
tx29_csum_partial_inner: 0
tx29_added_vlan_packets: 0
tx29_nop: 0
tx29_csum_none: 3
tx29_stopped: 0
tx29_dropped: 0
tx29_xmit_more: 1
tx29_recover: 0
tx29_cqes: 2
tx29_wake: 0
tx29_cqe_err: 0
tx30_packets: 0
tx30_bytes: 0
tx30_tso_packets: 0
tx30_tso_bytes: 0
tx30_tso_inner_packets: 0
tx30_tso_inner_bytes: 0
tx30_csum_partial: 0
tx30_csum_partial_inner: 0
tx30_added_vlan_packets: 0
tx30_nop: 0
tx30_csum_none: 0
tx30_stopped: 0
tx30_dropped: 0
tx30_xmit_more: 0
tx30_recover: 0
tx30_cqes: 0
tx30_wake: 0
tx30_cqe_err: 0
tx31_packets: 0
tx31_bytes: 0
tx31_tso_packets: 0
tx31_tso_bytes: 0
tx31_tso_inner_packets: 0
tx31_tso_inner_bytes: 0
tx31_csum_partial: 0
tx31_csum_partial_inner: 0
tx31_added_vlan_packets: 0
tx31_nop: 0
tx31_csum_none: 0
tx31_stopped: 0
tx31_dropped: 0
tx31_xmit_more: 0
tx31_recover: 0
tx31_cqes: 0
tx31_wake: 0
tx31_cqe_err: 0
tx32_packets: 0
tx32_bytes: 0
tx32_tso_packets: 0
tx32_tso_bytes: 0
tx32_tso_inner_packets: 0
tx32_tso_inner_bytes: 0
tx32_csum_partial: 0
tx32_csum_partial_inner: 0
tx32_added_vlan_packets: 0
tx32_nop: 0
tx32_csum_none: 0
tx32_stopped: 0
tx32_dropped: 0
tx32_xmit_more: 0
tx32_recover: 0
tx32_cqes: 0
tx32_wake: 0
tx32_cqe_err: 0
tx33_packets: 0
tx33_bytes: 0
tx33_tso_packets: 0
tx33_tso_bytes: 0
tx33_tso_inner_packets: 0
tx33_tso_inner_bytes: 0
tx33_csum_partial: 0
tx33_csum_partial_inner: 0
tx33_added_vlan_packets: 0
tx33_nop: 0
tx33_csum_none: 0
tx33_stopped: 0
tx33_dropped: 0
tx33_xmit_more: 0
tx33_recover: 0
tx33_cqes: 0
tx33_wake: 0
tx33_cqe_err: 0
tx34_packets: 0
tx34_bytes: 0
tx34_tso_packets: 0
tx34_tso_bytes: 0
tx34_tso_inner_packets: 0
tx34_tso_inner_bytes: 0
tx34_csum_partial: 0
tx34_csum_partial_inner: 0
tx34_added_vlan_packets: 0
tx34_nop: 0
tx34_csum_none: 0
tx34_stopped: 0
tx34_dropped: 0
tx34_xmit_more: 0
tx34_recover: 0
tx34_cqes: 0
tx34_wake: 0
tx34_cqe_err: 0
tx35_packets: 0
tx35_bytes: 0
tx35_tso_packets: 0
tx35_tso_bytes: 0
tx35_tso_inner_packets: 0
tx35_tso_inner_bytes: 0
tx35_csum_partial: 0
tx35_csum_partial_inner: 0
tx35_added_vlan_packets: 0
tx35_nop: 0
tx35_csum_none: 0
tx35_stopped: 0
tx35_dropped: 0
tx35_xmit_more: 0
tx35_recover: 0
tx35_cqes: 0
tx35_wake: 0
tx35_cqe_err: 0
tx36_packets: 0
tx36_bytes: 0
tx36_tso_packets: 0
tx36_tso_bytes: 0
tx36_tso_inner_packets: 0
tx36_tso_inner_bytes: 0
tx36_csum_partial: 0
tx36_csum_partial_inner: 0
tx36_added_vlan_packets: 0
tx36_nop: 0
tx36_csum_none: 0
tx36_stopped: 0
tx36_dropped: 0
tx36_xmit_more: 0
tx36_recover: 0
tx36_cqes: 0
tx36_wake: 0
tx36_cqe_err: 0
tx37_packets: 0
tx37_bytes: 0
tx37_tso_packets: 0
tx37_tso_bytes: 0
tx37_tso_inner_packets: 0
tx37_tso_inner_bytes: 0
tx37_csum_partial: 0
tx37_csum_partial_inner: 0
tx37_added_vlan_packets: 0
tx37_nop: 0
tx37_csum_none: 0
tx37_stopped: 0
tx37_dropped: 0
tx37_xmit_more: 0
tx37_recover: 0
tx37_cqes: 0
tx37_wake: 0
tx37_cqe_err: 0
tx38_packets: 0
tx38_bytes: 0
tx38_tso_packets: 0
tx38_tso_bytes: 0
tx38_tso_inner_packets: 0
tx38_tso_inner_bytes: 0
tx38_csum_partial: 0
tx38_csum_partial_inner: 0
tx38_added_vlan_packets: 0
tx38_nop: 0
tx38_csum_none: 0
tx38_stopped: 0
tx38_dropped: 0
tx38_xmit_more: 0
tx38_recover: 0
tx38_cqes: 0
tx38_wake: 0
tx38_cqe_err: 0
tx39_packets: 0
tx39_bytes: 0
tx39_tso_packets: 0
tx39_tso_bytes: 0
tx39_tso_inner_packets: 0
tx39_tso_inner_bytes: 0
tx39_csum_partial: 0
tx39_csum_partial_inner: 0
tx39_added_vlan_packets: 0
tx39_nop: 0
tx39_csum_none: 0
tx39_stopped: 0
tx39_dropped: 0
tx39_xmit_more: 0
tx39_recover: 0
tx39_cqes: 0
tx39_wake: 0
tx39_cqe_err: 0
tx40_packets: 0
tx40_bytes: 0
tx40_tso_packets: 0
tx40_tso_bytes: 0
tx40_tso_inner_packets: 0
tx40_tso_inner_bytes: 0
tx40_csum_partial: 0
tx40_csum_partial_inner: 0
tx40_added_vlan_packets: 0
tx40_nop: 0
tx40_csum_none: 0
tx40_stopped: 0
tx40_dropped: 0
tx40_xmit_more: 0
tx40_recover: 0
tx40_cqes: 0
tx40_wake: 0
tx40_cqe_err: 0
tx41_packets: 0
tx41_bytes: 0
tx41_tso_packets: 0
tx41_tso_bytes: 0
tx41_tso_inner_packets: 0
tx41_tso_inner_bytes: 0
tx41_csum_partial: 0
tx41_csum_partial_inner: 0
tx41_added_vlan_packets: 0
tx41_nop: 0
tx41_csum_none: 0
tx41_stopped: 0
tx41_dropped: 0
tx41_xmit_more: 0
tx41_recover: 0
tx41_cqes: 0
tx41_wake: 0
tx41_cqe_err: 0
tx42_packets: 0
tx42_bytes: 0
tx42_tso_packets: 0
tx42_tso_bytes: 0
tx42_tso_inner_packets: 0
tx42_tso_inner_bytes: 0
tx42_csum_partial: 0
tx42_csum_partial_inner: 0
tx42_added_vlan_packets: 0
tx42_nop: 0
tx42_csum_none: 0
tx42_stopped: 0
tx42_dropped: 0
tx42_xmit_more: 0
tx42_recover: 0
tx42_cqes: 0
tx42_wake: 0
tx42_cqe_err: 0
tx43_packets: 0
tx43_bytes: 0
tx43_tso_packets: 0
tx43_tso_bytes: 0
tx43_tso_inner_packets: 0
tx43_tso_inner_bytes: 0
tx43_csum_partial: 0
tx43_csum_partial_inner: 0
tx43_added_vlan_packets: 0
tx43_nop: 0
tx43_csum_none: 0
tx43_stopped: 0
tx43_dropped: 0
tx43_xmit_more: 0
tx43_recover: 0
tx43_cqes: 0
tx43_wake: 0
tx43_cqe_err: 0
tx44_packets: 0
tx44_bytes: 0
tx44_tso_packets: 0
tx44_tso_bytes: 0
tx44_tso_inner_packets: 0
tx44_tso_inner_bytes: 0
tx44_csum_partial: 0
tx44_csum_partial_inner: 0
tx44_added_vlan_packets: 0
tx44_nop: 0
tx44_csum_none: 0
tx44_stopped: 0
tx44_dropped: 0
tx44_xmit_more: 0
tx44_recover: 0
tx44_cqes: 0
tx44_wake: 0
tx44_cqe_err: 0
tx45_packets: 0
tx45_bytes: 0
tx45_tso_packets: 0
tx45_tso_bytes: 0
tx45_tso_inner_packets: 0
tx45_tso_inner_bytes: 0
tx45_csum_partial: 0
tx45_csum_partial_inner: 0
tx45_added_vlan_packets: 0
tx45_nop: 0
tx45_csum_none: 0
tx45_stopped: 0
tx45_dropped: 0
tx45_xmit_more: 0
tx45_recover: 0
tx45_cqes: 0
tx45_wake: 0
tx45_cqe_err: 0
tx46_packets: 0
tx46_bytes: 0
tx46_tso_packets: 0
tx46_tso_bytes: 0
tx46_tso_inner_packets: 0
tx46_tso_inner_bytes: 0
tx46_csum_partial: 0
tx46_csum_partial_inner: 0
tx46_added_vlan_packets: 0
tx46_nop: 0
tx46_csum_none: 0
tx46_stopped: 0
tx46_dropped: 0
tx46_xmit_more: 0
tx46_recover: 0
tx46_cqes: 0
tx46_wake: 0
tx46_cqe_err: 0
tx47_packets: 0
tx47_bytes: 0
tx47_tso_packets: 0
tx47_tso_bytes: 0
tx47_tso_inner_packets: 0
tx47_tso_inner_bytes: 0
tx47_csum_partial: 0
tx47_csum_partial_inner: 0
tx47_added_vlan_packets: 0
tx47_nop: 0
tx47_csum_none: 0
tx47_stopped: 0
tx47_dropped: 0
tx47_xmit_more: 0
tx47_recover: 0
tx47_cqes: 0
tx47_wake: 0
tx47_cqe_err: 0
tx48_packets: 0
tx48_bytes: 0
tx48_tso_packets: 0
tx48_tso_bytes: 0
tx48_tso_inner_packets: 0
tx48_tso_inner_bytes: 0
tx48_csum_partial: 0
tx48_csum_partial_inner: 0
tx48_added_vlan_packets: 0
tx48_nop: 0
tx48_csum_none: 0
tx48_stopped: 0
tx48_dropped: 0
tx48_xmit_more: 0
tx48_recover: 0
tx48_cqes: 0
tx48_wake: 0
tx48_cqe_err: 0
tx49_packets: 0
tx49_bytes: 0
tx49_tso_packets: 0
tx49_tso_bytes: 0
tx49_tso_inner_packets: 0
tx49_tso_inner_bytes: 0
tx49_csum_partial: 0
tx49_csum_partial_inner: 0
tx49_added_vlan_packets: 0
tx49_nop: 0
tx49_csum_none: 0
tx49_stopped: 0
tx49_dropped: 0
tx49_xmit_more: 0
tx49_recover: 0
tx49_cqes: 0
tx49_wake: 0
tx49_cqe_err: 0
tx50_packets: 0
tx50_bytes: 0
tx50_tso_packets: 0
tx50_tso_bytes: 0
tx50_tso_inner_packets: 0
tx50_tso_inner_bytes: 0
tx50_csum_partial: 0
tx50_csum_partial_inner: 0
tx50_added_vlan_packets: 0
tx50_nop: 0
tx50_csum_none: 0
tx50_stopped: 0
tx50_dropped: 0
tx50_xmit_more: 0
tx50_recover: 0
tx50_cqes: 0
tx50_wake: 0
tx50_cqe_err: 0
tx51_packets: 0
tx51_bytes: 0
tx51_tso_packets: 0
tx51_tso_bytes: 0
tx51_tso_inner_packets: 0
tx51_tso_inner_bytes: 0
tx51_csum_partial: 0
tx51_csum_partial_inner: 0
tx51_added_vlan_packets: 0
tx51_nop: 0
tx51_csum_none: 0
tx51_stopped: 0
tx51_dropped: 0
tx51_xmit_more: 0
tx51_recover: 0
tx51_cqes: 0
tx51_wake: 0
tx51_cqe_err: 0
tx52_packets: 0
tx52_bytes: 0
tx52_tso_packets: 0
tx52_tso_bytes: 0
tx52_tso_inner_packets: 0
tx52_tso_inner_bytes: 0
tx52_csum_partial: 0
tx52_csum_partial_inner: 0
tx52_added_vlan_packets: 0
tx52_nop: 0
tx52_csum_none: 0
tx52_stopped: 0
tx52_dropped: 0
tx52_xmit_more: 0
tx52_recover: 0
tx52_cqes: 0
tx52_wake: 0
tx52_cqe_err: 0
tx53_packets: 0
tx53_bytes: 0
tx53_tso_packets: 0
tx53_tso_bytes: 0
tx53_tso_inner_packets: 0
tx53_tso_inner_bytes: 0
tx53_csum_partial: 0
tx53_csum_partial_inner: 0
tx53_added_vlan_packets: 0
tx53_nop: 0
tx53_csum_none: 0
tx53_stopped: 0
tx53_dropped: 0
tx53_xmit_more: 0
tx53_recover: 0
tx53_cqes: 0
tx53_wake: 0
tx53_cqe_err: 0
tx54_packets: 0
tx54_bytes: 0
tx54_tso_packets: 0
tx54_tso_bytes: 0
tx54_tso_inner_packets: 0
tx54_tso_inner_bytes: 0
tx54_csum_partial: 0
tx54_csum_partial_inner: 0
tx54_added_vlan_packets: 0
tx54_nop: 0
tx54_csum_none: 0
tx54_stopped: 0
tx54_dropped: 0
tx54_xmit_more: 0
tx54_recover: 0
tx54_cqes: 0
tx54_wake: 0
tx54_cqe_err: 0
tx55_packets: 0
tx55_bytes: 0
tx55_tso_packets: 0
tx55_tso_bytes: 0
tx55_tso_inner_packets: 0
tx55_tso_inner_bytes: 0
tx55_csum_partial: 0
tx55_csum_partial_inner: 0
tx55_added_vlan_packets: 0
tx55_nop: 0
tx55_csum_none: 0
tx55_stopped: 0
tx55_dropped: 0
tx55_xmit_more: 0
tx55_recover: 0
tx55_cqes: 0
tx55_wake: 0
tx55_cqe_err: 0
tx0_xdp_xmit: 0
tx0_xdp_full: 0
tx0_xdp_err: 0
tx0_xdp_cqes: 0
tx1_xdp_xmit: 0
tx1_xdp_full: 0
tx1_xdp_err: 0
tx1_xdp_cqes: 0
tx2_xdp_xmit: 0
tx2_xdp_full: 0
tx2_xdp_err: 0
tx2_xdp_cqes: 0
tx3_xdp_xmit: 0
tx3_xdp_full: 0
tx3_xdp_err: 0
tx3_xdp_cqes: 0
tx4_xdp_xmit: 0
tx4_xdp_full: 0
tx4_xdp_err: 0
tx4_xdp_cqes: 0
tx5_xdp_xmit: 0
tx5_xdp_full: 0
tx5_xdp_err: 0
tx5_xdp_cqes: 0
tx6_xdp_xmit: 0
tx6_xdp_full: 0
tx6_xdp_err: 0
tx6_xdp_cqes: 0
tx7_xdp_xmit: 0
tx7_xdp_full: 0
tx7_xdp_err: 0
tx7_xdp_cqes: 0
tx8_xdp_xmit: 0
tx8_xdp_full: 0
tx8_xdp_err: 0
tx8_xdp_cqes: 0
tx9_xdp_xmit: 0
tx9_xdp_full: 0
tx9_xdp_err: 0
tx9_xdp_cqes: 0
tx10_xdp_xmit: 0
tx10_xdp_full: 0
tx10_xdp_err: 0
tx10_xdp_cqes: 0
tx11_xdp_xmit: 0
tx11_xdp_full: 0
tx11_xdp_err: 0
tx11_xdp_cqes: 0
tx12_xdp_xmit: 0
tx12_xdp_full: 0
tx12_xdp_err: 0
tx12_xdp_cqes: 0
tx13_xdp_xmit: 0
tx13_xdp_full: 0
tx13_xdp_err: 0
tx13_xdp_cqes: 0
tx14_xdp_xmit: 0
tx14_xdp_full: 0
tx14_xdp_err: 0
tx14_xdp_cqes: 0
tx15_xdp_xmit: 0
tx15_xdp_full: 0
tx15_xdp_err: 0
tx15_xdp_cqes: 0
tx16_xdp_xmit: 0
tx16_xdp_full: 0
tx16_xdp_err: 0
tx16_xdp_cqes: 0
tx17_xdp_xmit: 0
tx17_xdp_full: 0
tx17_xdp_err: 0
tx17_xdp_cqes: 0
tx18_xdp_xmit: 0
tx18_xdp_full: 0
tx18_xdp_err: 0
tx18_xdp_cqes: 0
tx19_xdp_xmit: 0
tx19_xdp_full: 0
tx19_xdp_err: 0
tx19_xdp_cqes: 0
tx20_xdp_xmit: 0
tx20_xdp_full: 0
tx20_xdp_err: 0
tx20_xdp_cqes: 0
tx21_xdp_xmit: 0
tx21_xdp_full: 0
tx21_xdp_err: 0
tx21_xdp_cqes: 0
tx22_xdp_xmit: 0
tx22_xdp_full: 0
tx22_xdp_err: 0
tx22_xdp_cqes: 0
tx23_xdp_xmit: 0
tx23_xdp_full: 0
tx23_xdp_err: 0
tx23_xdp_cqes: 0
tx24_xdp_xmit: 0
tx24_xdp_full: 0
tx24_xdp_err: 0
tx24_xdp_cqes: 0
tx25_xdp_xmit: 0
tx25_xdp_full: 0
tx25_xdp_err: 0
tx25_xdp_cqes: 0
tx26_xdp_xmit: 0
tx26_xdp_full: 0
tx26_xdp_err: 0
tx26_xdp_cqes: 0
tx27_xdp_xmit: 0
tx27_xdp_full: 0
tx27_xdp_err: 0
tx27_xdp_cqes: 0
tx28_xdp_xmit: 0
tx28_xdp_full: 0
tx28_xdp_err: 0
tx28_xdp_cqes: 0
tx29_xdp_xmit: 0
tx29_xdp_full: 0
tx29_xdp_err: 0
tx29_xdp_cqes: 0
tx30_xdp_xmit: 0
tx30_xdp_full: 0
tx30_xdp_err: 0
tx30_xdp_cqes: 0
tx31_xdp_xmit: 0
tx31_xdp_full: 0
tx31_xdp_err: 0
tx31_xdp_cqes: 0
tx32_xdp_xmit: 0
tx32_xdp_full: 0
tx32_xdp_err: 0
tx32_xdp_cqes: 0
tx33_xdp_xmit: 0
tx33_xdp_full: 0
tx33_xdp_err: 0
tx33_xdp_cqes: 0
tx34_xdp_xmit: 0
tx34_xdp_full: 0
tx34_xdp_err: 0
tx34_xdp_cqes: 0
tx35_xdp_xmit: 0
tx35_xdp_full: 0
tx35_xdp_err: 0
tx35_xdp_cqes: 0
tx36_xdp_xmit: 0
tx36_xdp_full: 0
tx36_xdp_err: 0
tx36_xdp_cqes: 0
tx37_xdp_xmit: 0
tx37_xdp_full: 0
tx37_xdp_err: 0
tx37_xdp_cqes: 0
tx38_xdp_xmit: 0
tx38_xdp_full: 0
tx38_xdp_err: 0
tx38_xdp_cqes: 0
tx39_xdp_xmit: 0
tx39_xdp_full: 0
tx39_xdp_err: 0
tx39_xdp_cqes: 0
tx40_xdp_xmit: 0
tx40_xdp_full: 0
tx40_xdp_err: 0
tx40_xdp_cqes: 0
tx41_xdp_xmit: 0
tx41_xdp_full: 0
tx41_xdp_err: 0
tx41_xdp_cqes: 0
tx42_xdp_xmit: 0
tx42_xdp_full: 0
tx42_xdp_err: 0
tx42_xdp_cqes: 0
tx43_xdp_xmit: 0
tx43_xdp_full: 0
tx43_xdp_err: 0
tx43_xdp_cqes: 0
tx44_xdp_xmit: 0
tx44_xdp_full: 0
tx44_xdp_err: 0
tx44_xdp_cqes: 0
tx45_xdp_xmit: 0
tx45_xdp_full: 0
tx45_xdp_err: 0
tx45_xdp_cqes: 0
tx46_xdp_xmit: 0
tx46_xdp_full: 0
tx46_xdp_err: 0
tx46_xdp_cqes: 0
tx47_xdp_xmit: 0
tx47_xdp_full: 0
tx47_xdp_err: 0
tx47_xdp_cqes: 0
tx48_xdp_xmit: 0
tx48_xdp_full: 0
tx48_xdp_err: 0
tx48_xdp_cqes: 0
tx49_xdp_xmit: 0
tx49_xdp_full: 0
tx49_xdp_err: 0
tx49_xdp_cqes: 0
tx50_xdp_xmit: 0
tx50_xdp_full: 0
tx50_xdp_err: 0
tx50_xdp_cqes: 0
tx51_xdp_xmit: 0
tx51_xdp_full: 0
tx51_xdp_err: 0
tx51_xdp_cqes: 0
tx52_xdp_xmit: 0
tx52_xdp_full: 0
tx52_xdp_err: 0
tx52_xdp_cqes: 0
tx53_xdp_xmit: 0
tx53_xdp_full: 0
tx53_xdp_err: 0
tx53_xdp_cqes: 0
tx54_xdp_xmit: 0
tx54_xdp_full: 0
tx54_xdp_err: 0
tx54_xdp_cqes: 0
tx55_xdp_xmit: 0
tx55_xdp_full: 0
tx55_xdp_err: 0
tx55_xdp_cqes: 0
> [...]
>
>>>> ethtool -S enp175s0f0
>>>> NIC statistics:
>>>> rx_packets: 141574897253
>>>> rx_bytes: 184445040406258
>>>> tx_packets: 172569543894
>>>> tx_bytes: 99486882076365
>>>> tx_tso_packets: 9367664195
>>>> tx_tso_bytes: 56435233992948
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 141297671626
>>>> tx_nop: 2102916272
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 141574897252
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 23135854
>>>> rx_csum_complete: 141551761398
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 127934791664
>>> It is a good idea to look into this, tx is not requesting hw tx
>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>> on
>>> calculating csum, or maybe this is just the rx csum complete..
>>>
>>>> tx_csum_partial: 13362879974
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 232561
>>> TX queues are stalling, could be an indentation for the pcie
>>> bottelneck.
>>>
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 1266021946
>>>> tx_recover: 0
>>>> tx_cqes: 140031716469
>>>> tx_queue_wake: 232561
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>>> rx_page_reuse: 0
>>>> rx_cache_reuse: 16625975793
>>>> rx_cache_full: 54161465914
>>>> rx_cache_empty: 258048
>>>> rx_cache_busy: 54161472735
>>>> rx_cache_waive: 0
>>>> rx_congst_umr: 0
>>>> rx_arfs_err: 0
>>>> ch_events: 40572621887
>>>> ch_poll: 40885650979
>>>> ch_arm: 40429276692
>>>> ch_aff_change: 0
>>>> ch_eq_rearm: 0
>>>> rx_out_of_buffer: 2791690
>>>> rx_if_down_packets: 74
>>>> rx_vport_unicast_packets: 141843476308
>>>> rx_vport_unicast_bytes: 185421265403318
>>>> tx_vport_unicast_packets: 172569484005
>>>> tx_vport_unicast_bytes: 100019940094298
>>>> rx_vport_multicast_packets: 85122935
>>>> rx_vport_multicast_bytes: 5761316431
>>>> tx_vport_multicast_packets: 6452
>>>> tx_vport_multicast_bytes: 643540
>>>> rx_vport_broadcast_packets: 22423624
>>>> rx_vport_broadcast_bytes: 1390127090
>>>> tx_vport_broadcast_packets: 22024
>>>> tx_vport_broadcast_bytes: 1321440
>>>> rx_vport_rdma_unicast_packets: 0
>>>> rx_vport_rdma_unicast_bytes: 0
>>>> tx_vport_rdma_unicast_packets: 0
>>>> tx_vport_rdma_unicast_bytes: 0
>>>> rx_vport_rdma_multicast_packets: 0
>>>> rx_vport_rdma_multicast_bytes: 0
>>>> tx_vport_rdma_multicast_packets: 0
>>>> tx_vport_rdma_multicast_bytes: 0
>>>> tx_packets_phy: 172569501577
>>>> rx_packets_phy: 142871314588
>>>> rx_crc_errors_phy: 0
>>>> tx_bytes_phy: 100710212814151
>>>> rx_bytes_phy: 187209224289564
>>>> tx_multicast_phy: 6452
>>>> tx_broadcast_phy: 22024
>>>> rx_multicast_phy: 85122933
>>>> rx_broadcast_phy: 22423623
>>>> rx_in_range_len_errors_phy: 2
>>>> rx_out_of_range_len_phy: 0
>>>> rx_oversize_pkts_phy: 0
>>>> rx_symbol_err_phy: 0
>>>> tx_mac_control_phy: 0
>>>> rx_mac_control_phy: 0
>>>> rx_unsupported_op_phy: 0
>>>> rx_pause_ctrl_phy: 0
>>>> tx_pause_ctrl_phy: 0
>>>> rx_discards_phy: 920161423
>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>> to
>>> the pcie bottleneck.
>> Yes this side is receiving more traffic - second port is +10G more tx
>>
> [...]
>
>
>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>> 0.00 0.00 0.00 31.30
>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>> 0.00 0.00 0.00 24.90
>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>> 0.00 0.00 0.00 19.68
>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>> 0.00 0.00 0.00 18.00
>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>> 0.00 0.00 0.00 17.40
>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>> 0.00 0.00 0.00 26.03
>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>> 0.00 0.00 0.00 17.52
>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>> 0.00 0.00 0.00 17.68
>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>> 0.00 0.00 0.00 20.42
>>> The numa cores are not at 100% util, you have around 20% of idle on
>>> each one.
>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>> push
>> aditional 1-2Gbit/s
>>
> yes but, it doens't look like the bottleneck is the cpu, although it is
> close to be :)..
>
>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 96.10
>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.70
>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.50
>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 97.40
>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.10
>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.40
>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>> 0.00 0.00 0.00 19.42
>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>> 0.00 0.00 0.00 26.60
>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>> 0.00 0.00 0.00 20.50
>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>> 0.00 0.00 0.00 24.58
>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>> 0.00 0.00 0.00 24.68
>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>> 0.00 0.00 0.00 28.34
>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>> 0.00 0.00 0.00 25.83
>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>> 0.00 0.00 0.00 27.20
>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>> 0.00 0.00 0.00 28.03
>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>> 0.00 0.00 0.00 24.70
>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>> 0.00 0.00 0.00 25.30
>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>> 0.00 0.00 0.00 26.77
>>>>
>>>>
>>>> ethtool -k enp175s0f0
>>>> Features for enp175s0f0:
>>>> rx-checksumming: on
>>>> tx-checksumming: on
>>>> tx-checksum-ipv4: on
>>>> tx-checksum-ip-generic: off [fixed]
>>>> tx-checksum-ipv6: on
>>>> tx-checksum-fcoe-crc: off [fixed]
>>>> tx-checksum-sctp: off [fixed]
>>>> scatter-gather: on
>>>> tx-scatter-gather: on
>>>> tx-scatter-gather-fraglist: off [fixed]
>>>> tcp-segmentation-offload: on
>>>> tx-tcp-segmentation: on
>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>> tx-tcp-mangleid-segmentation: off
>>>> tx-tcp6-segmentation: on
>>>> udp-fragmentation-offload: off
>>>> generic-segmentation-offload: on
>>>> generic-receive-offload: on
>>>> large-receive-offload: off [fixed]
>>>> rx-vlan-offload: on
>>>> tx-vlan-offload: on
>>>> ntuple-filters: off
>>>> receive-hashing: on
>>>> highdma: on [fixed]
>>>> rx-vlan-filter: on
>>>> vlan-challenged: off [fixed]
>>>> tx-lockless: off [fixed]
>>>> netns-local: off [fixed]
>>>> tx-gso-robust: off [fixed]
>>>> tx-fcoe-segmentation: off [fixed]
>>>> tx-gre-segmentation: on
>>>> tx-gre-csum-segmentation: on
>>>> tx-ipxip4-segmentation: off [fixed]
>>>> tx-ipxip6-segmentation: off [fixed]
>>>> tx-udp_tnl-segmentation: on
>>>> tx-udp_tnl-csum-segmentation: on
>>>> tx-gso-partial: on
>>>> tx-sctp-segmentation: off [fixed]
>>>> tx-esp-segmentation: off [fixed]
>>>> tx-udp-segmentation: on
>>>> fcoe-mtu: off [fixed]
>>>> tx-nocache-copy: off
>>>> loopback: off [fixed]
>>>> rx-fcs: off
>>>> rx-all: off
>>>> tx-vlan-stag-hw-insert: on
>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>> rx-vlan-stag-filter: on [fixed]
>>>> l2-fwd-offload: off [fixed]
>>>> hw-tc-offload: off
>>>> esp-hw-offload: off [fixed]
>>>> esp-tx-csum-hw-offload: off [fixed]
>>>> rx-udp_tunnel-port-offload: on
>>>> tls-hw-tx-offload: off [fixed]
>>>> tls-hw-rx-offload: off [fixed]
>>>> rx-gro-hw: off [fixed]
>>>> tls-hw-record: off [fixed]
>>>>
>>>> ethtool -c enp175s0f0
>>>> Coalesce parameters for enp175s0f0:
>>>> Adaptive RX: off TX: on
>>>> stats-block-usecs: 0
>>>> sample-interval: 0
>>>> pkt-rate-low: 0
>>>> pkt-rate-high: 0
>>>> dmac: 32703
>>>>
>>>> rx-usecs: 256
>>>> rx-frames: 128
>>>> rx-usecs-irq: 0
>>>> rx-frames-irq: 0
>>>>
>>>> tx-usecs: 8
>>>> tx-frames: 128
>>>> tx-usecs-irq: 0
>>>> tx-frames-irq: 0
>>>>
>>>> rx-usecs-low: 0
>>>> rx-frame-low: 0
>>>> tx-usecs-low: 0
>>>> tx-frame-low: 0
>>>>
>>>> rx-usecs-high: 0
>>>> rx-frame-high: 0
>>>> tx-usecs-high: 0
>>>> tx-frame-high: 0
>>>>
>>>> ethtool -g enp175s0f0
>>>> Ring parameters for enp175s0f0:
>>>> Pre-set maximums:
>>>> RX: 8192
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 8192
>>>> Current hardware settings:
>>>> RX: 4096
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 4096
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Also changed a little coalesce params - and best for this config are:
>> ethtool -c enp175s0f0
>> Coalesce parameters for enp175s0f0:
>> Adaptive RX: off TX: off
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>> dmac: 32573
>>
>> rx-usecs: 40
>> rx-frames: 128
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 8
>> tx-frames: 8
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
>> rx-usecs-low: 0
>> rx-frame-low: 0
>> tx-usecs-low: 0
>> tx-frame-low: 0
>>
>> rx-usecs-high: 0
>> rx-frame-high: 0
>> tx-usecs-high: 0
>> tx-frame-high: 0
>>
>>
>> Less drops on RX side - and more pps in overall forwarded.
>>
> how much improvement ? maybe we can improve our adaptive rx coal to be
> efficient for this work load.
>
>
I can prepare more stats with ethtool maybee to compare - but normally
tested with simple icmp forwarded from interface to interface
- before change coalescence params:
adaptive-rx off rx-usecs 384 rx-frames 128
3% loss for icmp
- after change to:
adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8
tx-frames 8
2% loss for icmp
But yes - to know better will need to compare rx/tx counters from
ethtool + /proc/net/dev
Was trying to turn on adaptative-tx+rx - but 100% saturation at 43Gbit/s
RX / 43Gbit/s TX
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 21:18 ` Paweł Staszewski
@ 2018-11-01 21:24 ` Paweł Staszewski
2018-11-01 21:34 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 21:24 UTC (permalink / raw)
To: Saeed Mahameed, netdev
W dniu 01.11.2018 o 22:18, Paweł Staszewski pisze:
>
>
> W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
>> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>>> Hi
>>>>>
>>>>> So maybee someone will be interested how linux kernel handles
>>>>> normal
>>>>> traffic (not pktgen :) )
>>>>>
>>>>>
>>>>> Server HW configuration:
>>>>>
>>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>>
>>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>>
>>>>>
>>>>> Server software:
>>>>>
>>>>> FRR - as routing daemon
>>>>>
>>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>>> local
>>>>> numa
>>>>> node)
>>>>>
>>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>>> numa
>>>>> node)
>>>>>
>>>>>
>>>>> Maximum traffic that server can handle:
>>>>>
>>>>> Bandwidth
>>>>>
>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>> input: /proc/net/dev type: rate
>>>>> \ iface Rx Tx Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>>> Gb/s
>>>>> 65.74 Gb/s
>>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>>> Gb/s
>>>>> 66.51 Gb/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>> total: 66.58 Gb/s 65.67
>>>>> Gb/s
>>>>> 132.25 Gb/s
>>>>>
>>>>>
>>>>> Packets per second:
>>>>>
>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>> input: /proc/net/dev type: rate
>>>>> - iface Rx Tx Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>>> 8735207.00 P/s
>>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>>> 8790460.00 P/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>>> 17525668.00 P/s
>>>>>
>>>>>
>>>>> After reaching that limits nics on the upstream side (more RX
>>>>> traffic)
>>>>> start to drop packets
>>>>>
>>>>>
>>>>> I just dont understand that server can't handle more bandwidth
>>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>>> RX
>>>>> side are increasing.
>>>>>
>>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>>> NIC (
>>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>>> 132.25
>>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>>> hmm yes that was my concern also - cause cant find anywhere
>>> informations
>>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>>> 8GT
>>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>>> bw
>>> on both ports
>> i think it is bidir
>>
>>> This can explain maybee also why cpuload is rising rapidly from
>>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>>> so
>>> there can be some error in reading them when offloading (gro/gso/tso)
>>> on
>>> nic's is enabled that is why
>>>
>>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>>> 8GT
>>>>> is
>>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>>> and
>>>>> pps
>>>>> (like 4x more comparing to normal internet traffic)
>>>>>
>>>> Are you forwarding when using pktgen as well or you just testing
>>>> the RX
>>>> side pps ?
>>> Yes pktgen was tested on single port RX
>>> Can check also forwarding to eliminate pciex limits
>>>
>> So this explains why you have more RX pps, since tx is idle and pcie
>> will be free to do only rx.
>>
>> [...]
>>
>>
>>>>> ethtool -S enp175s0f1
>>>>> NIC statistics:
>>>>> rx_packets: 173730800927
>>>>> rx_bytes: 99827422751332
>>>>> tx_packets: 142532009512
>>>>> tx_bytes: 184633045911222
>>>>> tx_tso_packets: 25989113891
>>>>> tx_tso_bytes: 132933363384458
>>>>> tx_tso_inner_packets: 0
>>>>> tx_tso_inner_bytes: 0
>>>>> tx_added_vlan_packets: 74630239613
>>>>> tx_nop: 2029817748
>>>>> rx_lro_packets: 0
>>>>> rx_lro_bytes: 0
>>>>> rx_ecn_mark: 0
>>>>> rx_removed_vlan_packets: 173730800927
>>>>> rx_csum_unnecessary: 0
>>>>> rx_csum_none: 434357
>>>>> rx_csum_complete: 173730366570
>>>>> rx_csum_unnecessary_inner: 0
>>>>> rx_xdp_drop: 0
>>>>> rx_xdp_redirect: 0
>>>>> rx_xdp_tx_xmit: 0
>>>>> rx_xdp_tx_full: 0
>>>>> rx_xdp_tx_err: 0
>>>>> rx_xdp_tx_cqe: 0
>>>>> tx_csum_none: 38260960853
>>>>> tx_csum_partial: 36369278774
>>>>> tx_csum_partial_inner: 0
>>>>> tx_queue_stopped: 1
>>>>> tx_queue_dropped: 0
>>>>> tx_xmit_more: 748638099
>>>>> tx_recover: 0
>>>>> tx_cqes: 73881645031
>>>>> tx_queue_wake: 1
>>>>> tx_udp_seg_rem: 0
>>>>> tx_cqe_err: 0
>>>>> tx_xdp_xmit: 0
>>>>> tx_xdp_full: 0
>>>>> tx_xdp_err: 0
>>>>> tx_xdp_cqes: 0
>>>>> rx_wqe_err: 0
>>>>> rx_mpwqe_filler_cqes: 0
>>>>> rx_mpwqe_filler_strides: 0
>>>>> rx_buff_alloc_err: 0
>>>>> rx_cqe_compress_blks: 0
>>>>> rx_cqe_compress_pkts: 0
>>>> If this is a pcie bottleneck it might be useful to enable CQE
>>>> compression (to reduce PCIe completion descriptors transactions)
>>>> you should see the above rx_cqe_compress_pkts increasing when
>>>> enabled.
>>>>
>>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>>> $ ethtool --show-priv-flags enp175s0f1
>>>> Private flags for p6p1:
>>>> rx_cqe_moder : on
>>>> cqe_moder : off
>>>> rx_cqe_compress : on
>>>> ...
>>>>
>>>> try this on both interfaces.
>>> Done
>>> ethtool --show-priv-flags enp175s0f1
>>> Private flags for enp175s0f1:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : on
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>>> ethtool --show-priv-flags enp175s0f0
>>> Private flags for enp175s0f0:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : on
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>> did it help reduce the load on the pcie ? do you see more pps ?
>> what is the ratio between rx_cqe_compress_pkts and over all rx packets
>> ?
> So - a little more pps
> Before change top - graph / after bottom -> image with graph stats
> from proc/net/dev
Attached link to graph
https://uploadfiles.io/5vgbh
> cqe_compress enabled at 11:55
>
> Sorry - but for real life traffic it is hard to do any counter
> differences - cause traffic just rising alone from minute to minute :)
> But for that time the change is visible on graph - cause was almost
> same for past 20minutes before change.
>
>
> full ethtool below:
> NIC statistics:
> rx_packets: 516522465438
> rx_bytes: 680052911258729
> tx_packets: 677697545586
> tx_bytes: 413647643141709
> tx_tso_packets: 42530913279
> tx_tso_bytes: 235655668554142
> tx_tso_inner_packets: 0
> tx_tso_inner_bytes: 0
> tx_added_vlan_packets: 551156530885
> tx_nop: 8536823558
> rx_lro_packets: 0
> rx_lro_bytes: 0
> rx_ecn_mark: 0
> rx_removed_vlan_packets: 516522465438
> rx_csum_unnecessary: 0
> rx_csum_none: 50382868
> rx_csum_complete: 516472082570
> rx_csum_unnecessary_inner: 0
> rx_xdp_drop: 0
> rx_xdp_redirect: 0
> rx_xdp_tx_xmit: 0
> rx_xdp_tx_full: 0
> rx_xdp_tx_err: 0
> rx_xdp_tx_cqe: 0
> tx_csum_none: 494075047017
> tx_csum_partial: 57081483898
> tx_csum_partial_inner: 0
> tx_queue_stopped: 518624
> tx_queue_dropped: 0
> tx_xmit_more: 1717880628
> tx_recover: 0
> tx_cqes: 549438869029
> tx_queue_wake: 518627
> tx_udp_seg_rem: 0
> tx_cqe_err: 0
> tx_xdp_xmit: 0
> tx_xdp_full: 0
> tx_xdp_err: 0
> tx_xdp_cqes: 0
> rx_wqe_err: 0
> rx_mpwqe_filler_cqes: 0
> rx_mpwqe_filler_strides: 0
> rx_buff_alloc_err: 0
> rx_cqe_compress_blks: 11483228712
> rx_cqe_compress_pkts: 25794213324
> rx_page_reuse: 0
> rx_cache_reuse: 63610249810
> rx_cache_full: 194650916511
> rx_cache_empty: 1118208
> rx_cache_busy: 194650982430
> rx_cache_waive: 0
> rx_congst_umr: 0
> rx_arfs_err: 0
> ch_events: 119556002196
> ch_poll: 121107424977
> ch_arm: 115856746008
> ch_aff_change: 31
> ch_eq_rearm: 0
> rx_out_of_buffer: 6880325
> rx_if_down_packets: 2062529
> rx_vport_unicast_packets: 517433716795
> rx_vport_unicast_bytes: 683464347301443
> tx_vport_unicast_packets: 677697453738
> tx_vport_unicast_bytes: 415788589663315
> rx_vport_multicast_packets: 208258309
> rx_vport_multicast_bytes: 14224046052
> tx_vport_multicast_packets: 21689
> tx_vport_multicast_bytes: 2158334
> rx_vport_broadcast_packets: 75838646
> rx_vport_broadcast_bytes: 4697944695
> tx_vport_broadcast_packets: 68730
> tx_vport_broadcast_bytes: 4123800
> rx_vport_rdma_unicast_packets: 0
> rx_vport_rdma_unicast_bytes: 0
> tx_vport_rdma_unicast_packets: 0
> tx_vport_rdma_unicast_bytes: 0
> rx_vport_rdma_multicast_packets: 0
> rx_vport_rdma_multicast_bytes: 0
> tx_vport_rdma_multicast_packets: 0
> tx_vport_rdma_multicast_bytes: 0
> tx_packets_phy: 677697543252
> rx_packets_phy: 521319491878
> rx_crc_errors_phy: 0
> tx_bytes_phy: 418499385791411
> rx_bytes_phy: 690322537017274
> tx_multicast_phy: 21689
> tx_broadcast_phy: 68730
> rx_multicast_phy: 208258305
> rx_broadcast_phy: 75838646
> rx_in_range_len_errors_phy: 4
> rx_out_of_range_len_phy: 0
> rx_oversize_pkts_phy: 0
> rx_symbol_err_phy: 0
> tx_mac_control_phy: 0
> rx_mac_control_phy: 0
> rx_unsupported_op_phy: 0
> rx_pause_ctrl_phy: 0
> tx_pause_ctrl_phy: 0
> rx_discards_phy: 3601449265
> tx_discards_phy: 0
> tx_errors_phy: 0
> rx_undersize_pkts_phy: 0
> rx_fragments_phy: 0
> rx_jabbers_phy: 0
> rx_64_bytes_phy: 1416456771
> rx_65_to_127_bytes_phy: 40750434737
> rx_128_to_255_bytes_phy: 11518110310
> rx_256_to_511_bytes_phy: 7055850637
> rx_512_to_1023_bytes_phy: 7811550424
> rx_1024_to_1518_bytes_phy: 265547564845
> rx_1519_to_2047_bytes_phy: 187219522899
> rx_2048_to_4095_bytes_phy: 0
> rx_4096_to_8191_bytes_phy: 0
> rx_8192_to_10239_bytes_phy: 0
> link_down_events_phy: 0
> rx_pcs_symbol_err_phy: 0
> rx_corrected_bits_phy: 0
> rx_pci_signal_integrity: 0
> tx_pci_signal_integrity: 48
> rx_prio0_bytes: 688807632117485
> rx_prio0_packets: 516310309931
> tx_prio0_bytes: 418499382756025
> tx_prio0_packets: 677697534982
> rx_prio1_bytes: 1497701612877
> rx_prio1_packets: 1206768094
> tx_prio1_bytes: 0
> tx_prio1_packets: 0
> rx_prio2_bytes: 112271227
> rx_prio2_packets: 337295
> tx_prio2_bytes: 0
> tx_prio2_packets: 0
> rx_prio3_bytes: 1165455555
> rx_prio3_packets: 1544310
> tx_prio3_bytes: 0
> tx_prio3_packets: 0
> rx_prio4_bytes: 161857240
> rx_prio4_packets: 341392
> tx_prio4_bytes: 0
> tx_prio4_packets: 0
> rx_prio5_bytes: 455031612
> rx_prio5_packets: 2861469
> tx_prio5_bytes: 0
> tx_prio5_packets: 0
> rx_prio6_bytes: 1873928697
> rx_prio6_packets: 5146981
> tx_prio6_bytes: 0
> tx_prio6_packets: 0
> rx_prio7_bytes: 13423452430
> rx_prio7_packets: 190724796
> tx_prio7_bytes: 0
> tx_prio7_packets: 0
> module_unplug: 0
> module_bus_stuck: 0
> module_high_temp: 0
> module_bad_shorted: 0
> ch0_events: 4252266777
> ch0_poll: 4330804273
> ch0_arm: 4120233182
> ch0_aff_change: 2
> ch0_eq_rearm: 0
> ch1_events: 3938415938
> ch1_poll: 4012621322
> ch1_arm: 3810131188
> ch1_aff_change: 2
> ch1_eq_rearm: 0
> ch2_events: 3897428860
> ch2_poll: 3973886848
> ch2_arm: 3773019397
> ch2_aff_change: 1
> ch2_eq_rearm: 0
> ch3_events: 4108000541
> ch3_poll: 4180139872
> ch3_arm: 3982093366
> ch3_aff_change: 1
> ch3_eq_rearm: 0
> ch4_events: 4652570079
> ch4_poll: 4720541090
> ch4_arm: 4524475054
> ch4_aff_change: 2
> ch4_eq_rearm: 0
> ch5_events: 3899177385
> ch5_poll: 3974274186
> ch5_arm: 3772299186
> ch5_aff_change: 2
> ch5_eq_rearm: 0
> ch6_events: 3915161350
> ch6_poll: 3992338199
> ch6_arm: 3794710989
> ch6_aff_change: 0
> ch6_eq_rearm: 0
> ch7_events: 4008175631
> ch7_poll: 4081321248
> ch7_arm: 3882263723
> ch7_aff_change: 0
> ch7_eq_rearm: 0
> ch8_events: 4207422352
> ch8_poll: 4276465449
> ch8_arm: 4077650366
> ch8_aff_change: 0
> ch8_eq_rearm: 0
> ch9_events: 4036491879
> ch9_poll: 4108975987
> ch9_arm: 3914493694
> ch9_aff_change: 0
> ch9_eq_rearm: 0
> ch10_events: 4066261595
> ch10_poll: 4134419606
> ch10_arm: 3936637711
> ch10_aff_change: 1
> ch10_eq_rearm: 0
> ch11_events: 4440494043
> ch11_poll: 4507578730
> ch11_arm: 4318629438
> ch11_aff_change: 0
> ch11_eq_rearm: 0
> ch12_events: 4066958252
> ch12_poll: 4130191506
> ch12_arm: 3934337782
> ch12_aff_change: 0
> ch12_eq_rearm: 0
> ch13_events: 4051309159
> ch13_poll: 4118864120
> ch13_arm: 3921011919
> ch13_aff_change: 0
> ch13_eq_rearm: 0
> ch14_events: 4321664800
> ch14_poll: 4382433680
> ch14_arm: 4186130552
> ch14_aff_change: 0
> ch14_eq_rearm: 0
> ch15_events: 4701102075
> ch15_poll: 4760373932
> ch15_arm: 4570151468
> ch15_aff_change: 0
> ch15_eq_rearm: 0
> ch16_events: 4311052687
> ch16_poll: 4345937129
> ch16_arm: 4170883819
> ch16_aff_change: 0
> ch16_eq_rearm: 0
> ch17_events: 4647570931
> ch17_poll: 4680218533
> ch17_arm: 4509426288
> ch17_aff_change: 0
> ch17_eq_rearm: 0
> ch18_events: 4598195702
> ch18_poll: 4631314898
> ch18_arm: 4457267084
> ch18_aff_change: 0
> ch18_eq_rearm: 0
> ch19_events: 4808094560
> ch19_poll: 4841368340
> ch19_arm: 4670604358
> ch19_aff_change: 0
> ch19_eq_rearm: 0
> ch20_events: 4240910605
> ch20_poll: 4276531502
> ch20_arm: 4101767278
> ch20_aff_change: 1
> ch20_eq_rearm: 0
> ch21_events: 4389371472
> ch21_poll: 4426870311
> ch21_arm: 4249339045
> ch21_aff_change: 2
> ch21_eq_rearm: 0
> ch22_events: 4282958754
> ch22_poll: 4319228073
> ch22_arm: 4145102991
> ch22_aff_change: 2
> ch22_eq_rearm: 0
> ch23_events: 4440196528
> ch23_poll: 4474090188
> ch23_arm: 4300837147
> ch23_aff_change: 2
> ch23_eq_rearm: 0
> ch24_events: 4326875785
> ch24_poll: 4364971263
> ch24_arm: 4186404526
> ch24_aff_change: 2
> ch24_eq_rearm: 0
> ch25_events: 4286528453
> ch25_poll: 4324089445
> ch25_arm: 4147222616
> ch25_aff_change: 3
> ch25_eq_rearm: 0
> ch26_events: 4098043104
> ch26_poll: 4138133745
> ch26_arm: 3967438971
> ch26_aff_change: 4
> ch26_eq_rearm: 0
> ch27_events: 4563302840
> ch27_poll: 4599441446
> ch27_arm: 4432182806
> ch27_aff_change: 4
> ch27_eq_rearm: 0
> ch28_events: 4
> ch28_poll: 4
> ch28_arm: 4
> ch28_aff_change: 0
> ch28_eq_rearm: 0
> ch29_events: 6
> ch29_poll: 6
> ch29_arm: 6
> ch29_aff_change: 0
> ch29_eq_rearm: 0
> ch30_events: 4
> ch30_poll: 4
> ch30_arm: 4
> ch30_aff_change: 0
> ch30_eq_rearm: 0
> ch31_events: 4
> ch31_poll: 4
> ch31_arm: 4
> ch31_aff_change: 0
> ch31_eq_rearm: 0
> ch32_events: 4
> ch32_poll: 4
> ch32_arm: 4
> ch32_aff_change: 0
> ch32_eq_rearm: 0
> ch33_events: 4
> ch33_poll: 4
> ch33_arm: 4
> ch33_aff_change: 0
> ch33_eq_rearm: 0
> ch34_events: 4
> ch34_poll: 4
> ch34_arm: 4
> ch34_aff_change: 0
> ch34_eq_rearm: 0
> ch35_events: 4
> ch35_poll: 4
> ch35_arm: 4
> ch35_aff_change: 0
> ch35_eq_rearm: 0
> ch36_events: 4
> ch36_poll: 4
> ch36_arm: 4
> ch36_aff_change: 0
> ch36_eq_rearm: 0
> ch37_events: 4
> ch37_poll: 4
> ch37_arm: 4
> ch37_aff_change: 0
> ch37_eq_rearm: 0
> ch38_events: 4
> ch38_poll: 4
> ch38_arm: 4
> ch38_aff_change: 0
> ch38_eq_rearm: 0
> ch39_events: 4
> ch39_poll: 4
> ch39_arm: 4
> ch39_aff_change: 0
> ch39_eq_rearm: 0
> ch40_events: 4
> ch40_poll: 4
> ch40_arm: 4
> ch40_aff_change: 0
> ch40_eq_rearm: 0
> ch41_events: 4
> ch41_poll: 4
> ch41_arm: 4
> ch41_aff_change: 0
> ch41_eq_rearm: 0
> ch42_events: 4
> ch42_poll: 4
> ch42_arm: 4
> ch42_aff_change: 0
> ch42_eq_rearm: 0
> ch43_events: 4
> ch43_poll: 4
> ch43_arm: 4
> ch43_aff_change: 0
> ch43_eq_rearm: 0
> ch44_events: 4
> ch44_poll: 4
> ch44_arm: 4
> ch44_aff_change: 0
> ch44_eq_rearm: 0
> ch45_events: 4
> ch45_poll: 4
> ch45_arm: 4
> ch45_aff_change: 0
> ch45_eq_rearm: 0
> ch46_events: 4
> ch46_poll: 4
> ch46_arm: 4
> ch46_aff_change: 0
> ch46_eq_rearm: 0
> ch47_events: 4
> ch47_poll: 4
> ch47_arm: 4
> ch47_aff_change: 0
> ch47_eq_rearm: 0
> ch48_events: 4
> ch48_poll: 4
> ch48_arm: 4
> ch48_aff_change: 0
> ch48_eq_rearm: 0
> ch49_events: 4
> ch49_poll: 4
> ch49_arm: 4
> ch49_aff_change: 0
> ch49_eq_rearm: 0
> ch50_events: 4
> ch50_poll: 4
> ch50_arm: 4
> ch50_aff_change: 0
> ch50_eq_rearm: 0
> ch51_events: 4
> ch51_poll: 4
> ch51_arm: 4
> ch51_aff_change: 0
> ch51_eq_rearm: 0
> ch52_events: 4
> ch52_poll: 4
> ch52_arm: 4
> ch52_aff_change: 0
> ch52_eq_rearm: 0
> ch53_events: 4
> ch53_poll: 4
> ch53_arm: 4
> ch53_aff_change: 0
> ch53_eq_rearm: 0
> ch54_events: 4
> ch54_poll: 4
> ch54_arm: 4
> ch54_aff_change: 0
> ch54_eq_rearm: 0
> ch55_events: 4
> ch55_poll: 4
> ch55_arm: 4
> ch55_aff_change: 0
> ch55_eq_rearm: 0
> rx0_packets: 21390033774
> rx0_bytes: 27326856299122
> rx0_csum_complete: 21339650906
> rx0_csum_unnecessary: 0
> rx0_csum_unnecessary_inner: 0
> rx0_csum_none: 50382868
> rx0_xdp_drop: 0
> rx0_xdp_redirect: 0
> rx0_lro_packets: 0
> rx0_lro_bytes: 0
> rx0_ecn_mark: 0
> rx0_removed_vlan_packets: 21390033774
> rx0_wqe_err: 0
> rx0_mpwqe_filler_cqes: 0
> rx0_mpwqe_filler_strides: 0
> rx0_buff_alloc_err: 0
> rx0_cqe_compress_blks: 481077641
> rx0_cqe_compress_pkts: 1085647489
> rx0_page_reuse: 0
> rx0_cache_reuse: 19050049
> rx0_cache_full: 10675964285
> rx0_cache_empty: 37376
> rx0_cache_busy: 10675966819
> rx0_cache_waive: 0
> rx0_congst_umr: 0
> rx0_arfs_err: 0
> rx0_xdp_tx_xmit: 0
> rx0_xdp_tx_full: 0
> rx0_xdp_tx_err: 0
> rx0_xdp_tx_cqes: 0
> rx1_packets: 19868919527
> rx1_bytes: 26149716991561
> rx1_csum_complete: 19868919527
> rx1_csum_unnecessary: 0
> rx1_csum_unnecessary_inner: 0
> rx1_csum_none: 0
> rx1_xdp_drop: 0
> rx1_xdp_redirect: 0
> rx1_lro_packets: 0
> rx1_lro_bytes: 0
> rx1_ecn_mark: 0
> rx1_removed_vlan_packets: 19868919527
> rx1_wqe_err: 0
> rx1_mpwqe_filler_cqes: 0
> rx1_mpwqe_filler_strides: 0
> rx1_buff_alloc_err: 0
> rx1_cqe_compress_blks: 420210560
> rx1_cqe_compress_pkts: 941233388
> rx1_page_reuse: 0
> rx1_cache_reuse: 46200002
> rx1_cache_full: 9888257242
> rx1_cache_empty: 37376
> rx1_cache_busy: 9888259746
> rx1_cache_waive: 0
> rx1_congst_umr: 0
> rx1_arfs_err: 0
> rx1_xdp_tx_xmit: 0
> rx1_xdp_tx_full: 0
> rx1_xdp_tx_err: 0
> rx1_xdp_tx_cqes: 0
> rx2_packets: 19575013662
> rx2_bytes: 25759818417945
> rx2_csum_complete: 19575013662
> rx2_csum_unnecessary: 0
> rx2_csum_unnecessary_inner: 0
> rx2_csum_none: 0
> rx2_xdp_drop: 0
> rx2_xdp_redirect: 0
> rx2_lro_packets: 0
> rx2_lro_bytes: 0
> rx2_ecn_mark: 0
> rx2_removed_vlan_packets: 19575013662
> rx2_wqe_err: 0
> rx2_mpwqe_filler_cqes: 0
> rx2_mpwqe_filler_strides: 0
> rx2_buff_alloc_err: 0
> rx2_cqe_compress_blks: 412345511
> rx2_cqe_compress_pkts: 923376167
> rx2_page_reuse: 0
> rx2_cache_reuse: 38837731
> rx2_cache_full: 9748666548
> rx2_cache_empty: 37376
> rx2_cache_busy: 9748669093
> rx2_cache_waive: 0
> rx2_congst_umr: 0
> rx2_arfs_err: 0
> rx2_xdp_tx_xmit: 0
> rx2_xdp_tx_full: 0
> rx2_xdp_tx_err: 0
> rx2_xdp_tx_cqes: 0
> rx3_packets: 19795911749
> rx3_bytes: 25969475566905
> rx3_csum_complete: 19795911749
> rx3_csum_unnecessary: 0
> rx3_csum_unnecessary_inner: 0
> rx3_csum_none: 0
> rx3_xdp_drop: 0
> rx3_xdp_redirect: 0
> rx3_lro_packets: 0
> rx3_lro_bytes: 0
> rx3_ecn_mark: 0
> rx3_removed_vlan_packets: 19795911749
> rx3_wqe_err: 0
> rx3_mpwqe_filler_cqes: 0
> rx3_mpwqe_filler_strides: 0
> rx3_buff_alloc_err: 0
> rx3_cqe_compress_blks: 416658765
> rx3_cqe_compress_pkts: 934986266
> rx3_page_reuse: 0
> rx3_cache_reuse: 34542124
> rx3_cache_full: 9863411232
> rx3_cache_empty: 37376
> rx3_cache_busy: 9863413732
> rx3_cache_waive: 0
> rx3_congst_umr: 0
> rx3_arfs_err: 0
> rx3_xdp_tx_xmit: 0
> rx3_xdp_tx_full: 0
> rx3_xdp_tx_err: 0
> rx3_xdp_tx_cqes: 0
> rx4_packets: 20445652378
> rx4_bytes: 26949065110265
> rx4_csum_complete: 20445652378
> rx4_csum_unnecessary: 0
> rx4_csum_unnecessary_inner: 0
> rx4_csum_none: 0
> rx4_xdp_drop: 0
> rx4_xdp_redirect: 0
> rx4_lro_packets: 0
> rx4_lro_bytes: 0
> rx4_ecn_mark: 0
> rx4_removed_vlan_packets: 20445652378
> rx4_wqe_err: 0
> rx4_mpwqe_filler_cqes: 0
> rx4_mpwqe_filler_strides: 0
> rx4_buff_alloc_err: 0
> rx4_cqe_compress_blks: 506085858
> rx4_cqe_compress_pkts: 1147860328
> rx4_page_reuse: 0
> rx4_cache_reuse: 10122542864
> rx4_cache_full: 100281206
> rx4_cache_empty: 37376
> rx4_cache_busy: 100283304
> rx4_cache_waive: 0
> rx4_congst_umr: 0
> rx4_arfs_err: 0
> rx4_xdp_tx_xmit: 0
> rx4_xdp_tx_full: 0
> rx4_xdp_tx_err: 0
> rx4_xdp_tx_cqes: 0
> rx5_packets: 19622362246
> rx5_bytes: 25843450982982
> rx5_csum_complete: 19622362246
> rx5_csum_unnecessary: 0
> rx5_csum_unnecessary_inner: 0
> rx5_csum_none: 0
> rx5_xdp_drop: 0
> rx5_xdp_redirect: 0
> rx5_lro_packets: 0
> rx5_lro_bytes: 0
> rx5_ecn_mark: 0
> rx5_removed_vlan_packets: 19622362246
> rx5_wqe_err: 0
> rx5_mpwqe_filler_cqes: 0
> rx5_mpwqe_filler_strides: 0
> rx5_buff_alloc_err: 0
> rx5_cqe_compress_blks: 422840924
> rx5_cqe_compress_pkts: 948005878
> rx5_page_reuse: 0
> rx5_cache_reuse: 31285453
> rx5_cache_full: 9779893117
> rx5_cache_empty: 37376
> rx5_cache_busy: 9779895647
> rx5_cache_waive: 0
> rx5_congst_umr: 0
> rx5_arfs_err: 0
> rx5_xdp_tx_xmit: 0
> rx5_xdp_tx_full: 0
> rx5_xdp_tx_err: 0
> rx5_xdp_tx_cqes: 0
> rx6_packets: 19788231278
> rx6_bytes: 25985783006486
> rx6_csum_complete: 19788231278
> rx6_csum_unnecessary: 0
> rx6_csum_unnecessary_inner: 0
> rx6_csum_none: 0
> rx6_xdp_drop: 0
> rx6_xdp_redirect: 0
> rx6_lro_packets: 0
> rx6_lro_bytes: 0
> rx6_ecn_mark: 0
> rx6_removed_vlan_packets: 19788231278
> rx6_wqe_err: 0
> rx6_mpwqe_filler_cqes: 0
> rx6_mpwqe_filler_strides: 0
> rx6_buff_alloc_err: 0
> rx6_cqe_compress_blks: 418799056
> rx6_cqe_compress_pkts: 938282685
> rx6_page_reuse: 0
> rx6_cache_reuse: 18114793
> rx6_cache_full: 9875998295
> rx6_cache_empty: 37376
> rx6_cache_busy: 9876000831
> rx6_cache_waive: 0
> rx6_congst_umr: 0
> rx6_arfs_err: 0
> rx6_xdp_tx_xmit: 0
> rx6_xdp_tx_full: 0
> rx6_xdp_tx_err: 0
> rx6_xdp_tx_cqes: 0
> rx7_packets: 19795759168
> rx7_bytes: 26085056586860
> rx7_csum_complete: 19795759168
> rx7_csum_unnecessary: 0
> rx7_csum_unnecessary_inner: 0
> rx7_csum_none: 0
> rx7_xdp_drop: 0
> rx7_xdp_redirect: 0
> rx7_lro_packets: 0
> rx7_lro_bytes: 0
> rx7_ecn_mark: 0
> rx7_removed_vlan_packets: 19795759168
> rx7_wqe_err: 0
> rx7_mpwqe_filler_cqes: 0
> rx7_mpwqe_filler_strides: 0
> rx7_buff_alloc_err: 0
> rx7_cqe_compress_blks: 413959224
> rx7_cqe_compress_pkts: 927675936
> rx7_page_reuse: 0
> rx7_cache_reuse: 23902990
> rx7_cache_full: 9873974042
> rx7_cache_empty: 37376
> rx7_cache_busy: 9873976574
> rx7_cache_waive: 0
> rx7_congst_umr: 0
> rx7_arfs_err: 0
> rx7_xdp_tx_xmit: 0
> rx7_xdp_tx_full: 0
> rx7_xdp_tx_err: 0
> rx7_xdp_tx_cqes: 0
> rx8_packets: 19963477439
> rx8_bytes: 26384640501789
> rx8_csum_complete: 19963477439
> rx8_csum_unnecessary: 0
> rx8_csum_unnecessary_inner: 0
> rx8_csum_none: 0
> rx8_xdp_drop: 0
> rx8_xdp_redirect: 0
> rx8_lro_packets: 0
> rx8_lro_bytes: 0
> rx8_ecn_mark: 0
> rx8_removed_vlan_packets: 19963477439
> rx8_wqe_err: 0
> rx8_mpwqe_filler_cqes: 0
> rx8_mpwqe_filler_strides: 0
> rx8_buff_alloc_err: 0
> rx8_cqe_compress_blks: 420422857
> rx8_cqe_compress_pkts: 942720292
> rx8_page_reuse: 0
> rx8_cache_reuse: 88181713
> rx8_cache_full: 9893554525
> rx8_cache_empty: 37376
> rx8_cache_busy: 9893556983
> rx8_cache_waive: 0
> rx8_congst_umr: 0
> rx8_arfs_err: 0
> rx8_xdp_tx_xmit: 0
> rx8_xdp_tx_full: 0
> rx8_xdp_tx_err: 0
> rx8_xdp_tx_cqes: 0
> rx9_packets: 19726642138
> rx9_bytes: 26063924286499
> rx9_csum_complete: 19726642138
> rx9_csum_unnecessary: 0
> rx9_csum_unnecessary_inner: 0
> rx9_csum_none: 0
> rx9_xdp_drop: 0
> rx9_xdp_redirect: 0
> rx9_lro_packets: 0
> rx9_lro_bytes: 0
> rx9_ecn_mark: 0
> rx9_removed_vlan_packets: 19726642138
> rx9_wqe_err: 0
> rx9_mpwqe_filler_cqes: 0
> rx9_mpwqe_filler_strides: 0
> rx9_buff_alloc_err: 0
> rx9_cqe_compress_blks: 424227411
> rx9_cqe_compress_pkts: 951534873
> rx9_page_reuse: 0
> rx9_cache_reuse: 482901440
> rx9_cache_full: 9380417487
> rx9_cache_empty: 37376
> rx9_cache_busy: 9380419608
> rx9_cache_waive: 0
> rx9_congst_umr: 0
> rx9_arfs_err: 0
> rx9_xdp_tx_xmit: 0
> rx9_xdp_tx_full: 0
> rx9_xdp_tx_err: 0
> rx9_xdp_tx_cqes: 0
> rx10_packets: 19901229170
> rx10_bytes: 26300854495044
> rx10_csum_complete: 19901229170
> rx10_csum_unnecessary: 0
> rx10_csum_unnecessary_inner: 0
> rx10_csum_none: 0
> rx10_xdp_drop: 0
> rx10_xdp_redirect: 0
> rx10_lro_packets: 0
> rx10_lro_bytes: 0
> rx10_ecn_mark: 0
> rx10_removed_vlan_packets: 19901229170
> rx10_wqe_err: 0
> rx10_mpwqe_filler_cqes: 0
> rx10_mpwqe_filler_strides: 0
> rx10_buff_alloc_err: 0
> rx10_cqe_compress_blks: 419082938
> rx10_cqe_compress_pkts: 940791347
> rx10_page_reuse: 0
> rx10_cache_reuse: 14896055
> rx10_cache_full: 9935715977
> rx10_cache_empty: 37376
> rx10_cache_busy: 9935718513
> rx10_cache_waive: 0
> rx10_congst_umr: 0
> rx10_arfs_err: 0
> rx10_xdp_tx_xmit: 0
> rx10_xdp_tx_full: 0
> rx10_xdp_tx_err: 0
> rx10_xdp_tx_cqes: 0
> rx11_packets: 20352190494
> rx11_bytes: 26851034425372
> rx11_csum_complete: 20352190494
> rx11_csum_unnecessary: 0
> rx11_csum_unnecessary_inner: 0
> rx11_csum_none: 0
> rx11_xdp_drop: 0
> rx11_xdp_redirect: 0
> rx11_lro_packets: 0
> rx11_lro_bytes: 0
> rx11_ecn_mark: 0
> rx11_removed_vlan_packets: 20352190494
> rx11_wqe_err: 0
> rx11_mpwqe_filler_cqes: 0
> rx11_mpwqe_filler_strides: 0
> rx11_buff_alloc_err: 0
> rx11_cqe_compress_blks: 501992147
> rx11_cqe_compress_pkts: 1140398610
> rx11_page_reuse: 0
> rx11_cache_reuse: 10071721531
> rx11_cache_full: 104371621
> rx11_cache_empty: 37376
> rx11_cache_busy: 104373697
> rx11_cache_waive: 0
> rx11_congst_umr: 0
> rx11_arfs_err: 0
> rx11_xdp_tx_xmit: 0
> rx11_xdp_tx_full: 0
> rx11_xdp_tx_err: 0
> rx11_xdp_tx_cqes: 0
> rx12_packets: 19934747149
> rx12_bytes: 26296478787829
> rx12_csum_complete: 19934747149
> rx12_csum_unnecessary: 0
> rx12_csum_unnecessary_inner: 0
> rx12_csum_none: 0
> rx12_xdp_drop: 0
> rx12_xdp_redirect: 0
> rx12_lro_packets: 0
> rx12_lro_bytes: 0
> rx12_ecn_mark: 0
> rx12_removed_vlan_packets: 19934747149
> rx12_wqe_err: 0
> rx12_mpwqe_filler_cqes: 0
> rx12_mpwqe_filler_strides: 0
> rx12_buff_alloc_err: 0
> rx12_cqe_compress_blks: 443350570
> rx12_cqe_compress_pkts: 995997220
> rx12_page_reuse: 0
> rx12_cache_reuse: 9864934174
> rx12_cache_full: 102437428
> rx12_cache_empty: 37376
> rx12_cache_busy: 102439382
> rx12_cache_waive: 0
> rx12_congst_umr: 0
> rx12_arfs_err: 0
> rx12_xdp_tx_xmit: 0
> rx12_xdp_tx_full: 0
> rx12_xdp_tx_err: 0
> rx12_xdp_tx_cqes: 0
> rx13_packets: 19866908096
> rx13_bytes: 26160931936186
> rx13_csum_complete: 19866908096
> rx13_csum_unnecessary: 0
> rx13_csum_unnecessary_inner: 0
> rx13_csum_none: 0
> rx13_xdp_drop: 0
> rx13_xdp_redirect: 0
> rx13_lro_packets: 0
> rx13_lro_bytes: 0
> rx13_ecn_mark: 0
> rx13_removed_vlan_packets: 19866908096
> rx13_wqe_err: 0
> rx13_mpwqe_filler_cqes: 0
> rx13_mpwqe_filler_strides: 0
> rx13_buff_alloc_err: 0
> rx13_cqe_compress_blks: 413640141
> rx13_cqe_compress_pkts: 926175066
> rx13_page_reuse: 0
> rx13_cache_reuse: 36358610
> rx13_cache_full: 9897092921
> rx13_cache_empty: 37376
> rx13_cache_busy: 9897095422
> rx13_cache_waive: 0
> rx13_congst_umr: 0
> rx13_arfs_err: 0
> rx13_xdp_tx_xmit: 0
> rx13_xdp_tx_full: 0
> rx13_xdp_tx_err: 0
> rx13_xdp_tx_cqes: 0
> rx14_packets: 20229035746
> rx14_bytes: 26655092809172
> rx14_csum_complete: 20229035746
> rx14_csum_unnecessary: 0
> rx14_csum_unnecessary_inner: 0
> rx14_csum_none: 0
> rx14_xdp_drop: 0
> rx14_xdp_redirect: 0
> rx14_lro_packets: 0
> rx14_lro_bytes: 0
> rx14_ecn_mark: 0
> rx14_removed_vlan_packets: 20229035746
> rx14_wqe_err: 0
> rx14_mpwqe_filler_cqes: 0
> rx14_mpwqe_filler_strides: 0
> rx14_buff_alloc_err: 0
> rx14_cqe_compress_blks: 460990337
> rx14_cqe_compress_pkts: 1041287948
> rx14_page_reuse: 0
> rx14_cache_reuse: 25649275
> rx14_cache_full: 10088866045
> rx14_cache_empty: 37376
> rx14_cache_busy: 10088868574
> rx14_cache_waive: 0
> rx14_congst_umr: 0
> rx14_arfs_err: 0
> rx14_xdp_tx_xmit: 0
> rx14_xdp_tx_full: 0
> rx14_xdp_tx_err: 0
> rx14_xdp_tx_cqes: 0
> rx15_packets: 20528177154
> rx15_bytes: 27029263893264
> rx15_csum_complete: 20528177154
> rx15_csum_unnecessary: 0
> rx15_csum_unnecessary_inner: 0
> rx15_csum_none: 0
> rx15_xdp_drop: 0
> rx15_xdp_redirect: 0
> rx15_lro_packets: 0
> rx15_lro_bytes: 0
> rx15_ecn_mark: 0
> rx15_removed_vlan_packets: 20528177154
> rx15_wqe_err: 0
> rx15_mpwqe_filler_cqes: 0
> rx15_mpwqe_filler_strides: 0
> rx15_buff_alloc_err: 0
> rx15_cqe_compress_blks: 476776176
> rx15_cqe_compress_pkts: 1076153263
> rx15_page_reuse: 0
> rx15_cache_reuse: 48426735
> rx15_cache_full: 10215659289
> rx15_cache_empty: 37376
> rx15_cache_busy: 10215661817
> rx15_cache_waive: 0
> rx15_congst_umr: 0
> rx15_arfs_err: 0
> rx15_xdp_tx_xmit: 0
> rx15_xdp_tx_full: 0
> rx15_xdp_tx_err: 0
> rx15_xdp_tx_cqes: 0
> rx16_packets: 16104078098
> rx16_bytes: 21256361789679
> rx16_csum_complete: 16104078098
> rx16_csum_unnecessary: 0
> rx16_csum_unnecessary_inner: 0
> rx16_csum_none: 0
> rx16_xdp_drop: 0
> rx16_xdp_redirect: 0
> rx16_lro_packets: 0
> rx16_lro_bytes: 0
> rx16_ecn_mark: 0
> rx16_removed_vlan_packets: 16104078098
> rx16_wqe_err: 0
> rx16_mpwqe_filler_cqes: 0
> rx16_mpwqe_filler_strides: 0
> rx16_buff_alloc_err: 0
> rx16_cqe_compress_blks: 352082054
> rx16_cqe_compress_pkts: 787161670
> rx16_page_reuse: 0
> rx16_cache_reuse: 25912567
> rx16_cache_full: 8026124051
> rx16_cache_empty: 37376
> rx16_cache_busy: 8026126465
> rx16_cache_waive: 0
> rx16_congst_umr: 0
> rx16_arfs_err: 0
> rx16_xdp_tx_xmit: 0
> rx16_xdp_tx_full: 0
> rx16_xdp_tx_err: 0
> rx16_xdp_tx_cqes: 0
> rx17_packets: 16314055017
> rx17_bytes: 21589139030173
> rx17_csum_complete: 16314055017
> rx17_csum_unnecessary: 0
> rx17_csum_unnecessary_inner: 0
> rx17_csum_none: 0
> rx17_xdp_drop: 0
> rx17_xdp_redirect: 0
> rx17_lro_packets: 0
> rx17_lro_bytes: 0
> rx17_ecn_mark: 0
> rx17_removed_vlan_packets: 16314055017
> rx17_wqe_err: 0
> rx17_mpwqe_filler_cqes: 0
> rx17_mpwqe_filler_strides: 0
> rx17_buff_alloc_err: 0
> rx17_cqe_compress_blks: 387834541
> rx17_cqe_compress_pkts: 871851081
> rx17_page_reuse: 0
> rx17_cache_reuse: 24021313
> rx17_cache_full: 8133003829
> rx17_cache_empty: 37376
> rx17_cache_busy: 8133006175
> rx17_cache_waive: 0
> rx17_congst_umr: 0
> rx17_arfs_err: 0
> rx17_xdp_tx_xmit: 0
> rx17_xdp_tx_full: 0
> rx17_xdp_tx_err: 0
> rx17_xdp_tx_cqes: 0
> rx18_packets: 16439016814
> rx18_bytes: 21648651917475
> rx18_csum_complete: 16439016814
> rx18_csum_unnecessary: 0
> rx18_csum_unnecessary_inner: 0
> rx18_csum_none: 0
> rx18_xdp_drop: 0
> rx18_xdp_redirect: 0
> rx18_lro_packets: 0
> rx18_lro_bytes: 0
> rx18_ecn_mark: 0
> rx18_removed_vlan_packets: 16439016814
> rx18_wqe_err: 0
> rx18_mpwqe_filler_cqes: 0
> rx18_mpwqe_filler_strides: 0
> rx18_buff_alloc_err: 0
> rx18_cqe_compress_blks: 375066666
> rx18_cqe_compress_pkts: 843563974
> rx18_page_reuse: 0
> rx18_cache_reuse: 8151064266
> rx18_cache_full: 68442025
> rx18_cache_empty: 37376
> rx18_cache_busy: 68444122
> rx18_cache_waive: 0
> rx18_congst_umr: 0
> rx18_arfs_err: 0
> rx18_xdp_tx_xmit: 0
> rx18_xdp_tx_full: 0
> rx18_xdp_tx_err: 0
> rx18_xdp_tx_cqes: 0
> rx19_packets: 16641223506
> rx19_bytes: 21964749940935
> rx19_csum_complete: 16641223506
> rx19_csum_unnecessary: 0
> rx19_csum_unnecessary_inner: 0
> rx19_csum_none: 0
> rx19_xdp_drop: 0
> rx19_xdp_redirect: 0
> rx19_lro_packets: 0
> rx19_lro_bytes: 0
> rx19_ecn_mark: 0
> rx19_removed_vlan_packets: 16641223506
> rx19_wqe_err: 0
> rx19_mpwqe_filler_cqes: 0
> rx19_mpwqe_filler_strides: 0
> rx19_buff_alloc_err: 0
> rx19_cqe_compress_blks: 387825932
> rx19_cqe_compress_pkts: 872266355
> rx19_page_reuse: 0
> rx19_cache_reuse: 116433620
> rx19_cache_full: 8204175954
> rx19_cache_empty: 37376
> rx19_cache_busy: 8204178120
> rx19_cache_waive: 0
> rx19_congst_umr: 0
> rx19_arfs_err: 0
> rx19_xdp_tx_xmit: 0
> rx19_xdp_tx_full: 0
> rx19_xdp_tx_err: 0
> rx19_xdp_tx_cqes: 0
> rx20_packets: 16206927741
> rx20_bytes: 21387447038430
> rx20_csum_complete: 16206927741
> rx20_csum_unnecessary: 0
> rx20_csum_unnecessary_inner: 0
> rx20_csum_none: 0
> rx20_xdp_drop: 0
> rx20_xdp_redirect: 0
> rx20_lro_packets: 0
> rx20_lro_bytes: 0
> rx20_ecn_mark: 0
> rx20_removed_vlan_packets: 16206927741
> rx20_wqe_err: 0
> rx20_mpwqe_filler_cqes: 0
> rx20_mpwqe_filler_strides: 0
> rx20_buff_alloc_err: 0
> rx20_cqe_compress_blks: 370144620
> rx20_cqe_compress_pkts: 829122671
> rx20_page_reuse: 0
> rx20_cache_reuse: 8053733744
> rx20_cache_full: 49728026
> rx20_cache_empty: 37376
> rx20_cache_busy: 49730116
> rx20_cache_waive: 0
> rx20_congst_umr: 0
> rx20_arfs_err: 0
> rx20_xdp_tx_xmit: 0
> rx20_xdp_tx_full: 0
> rx20_xdp_tx_err: 0
> rx20_xdp_tx_cqes: 0
> rx21_packets: 16562361314
> rx21_bytes: 21856653284356
> rx21_csum_complete: 16562361314
> rx21_csum_unnecessary: 0
> rx21_csum_unnecessary_inner: 0
> rx21_csum_none: 0
> rx21_xdp_drop: 0
> rx21_xdp_redirect: 0
> rx21_lro_packets: 0
> rx21_lro_bytes: 0
> rx21_ecn_mark: 0
> rx21_removed_vlan_packets: 16562361314
> rx21_wqe_err: 0
> rx21_mpwqe_filler_cqes: 0
> rx21_mpwqe_filler_strides: 0
> rx21_buff_alloc_err: 0
> rx21_cqe_compress_blks: 350790425
> rx21_cqe_compress_pkts: 783850729
> rx21_page_reuse: 0
> rx21_cache_reuse: 28077493
> rx21_cache_full: 8253100706
> rx21_cache_empty: 37376
> rx21_cache_busy: 8253103147
> rx21_cache_waive: 0
> rx21_congst_umr: 0
> rx21_arfs_err: 0
> rx21_xdp_tx_xmit: 0
> rx21_xdp_tx_full: 0
> rx21_xdp_tx_err: 0
> rx21_xdp_tx_cqes: 0
> rx22_packets: 16350307571
> rx22_bytes: 21408575325592
> rx22_csum_complete: 16350307571
> rx22_csum_unnecessary: 0
> rx22_csum_unnecessary_inner: 0
> rx22_csum_none: 0
> rx22_xdp_drop: 0
> rx22_xdp_redirect: 0
> rx22_lro_packets: 0
> rx22_lro_bytes: 0
> rx22_ecn_mark: 0
> rx22_removed_vlan_packets: 16350307571
> rx22_wqe_err: 0
> rx22_mpwqe_filler_cqes: 0
> rx22_mpwqe_filler_strides: 0
> rx22_buff_alloc_err: 0
> rx22_cqe_compress_blks: 353531065
> rx22_cqe_compress_pkts: 790814415
> rx22_page_reuse: 0
> rx22_cache_reuse: 16934343
> rx22_cache_full: 8158216889
> rx22_cache_empty: 37376
> rx22_cache_busy: 8158219417
> rx22_cache_waive: 0
> rx22_congst_umr: 0
> rx22_arfs_err: 0
> rx22_xdp_tx_xmit: 0
> rx22_xdp_tx_full: 0
> rx22_xdp_tx_err: 0
> rx22_xdp_tx_cqes: 0
> rx23_packets: 16019811764
> rx23_bytes: 21137182570985
> rx23_csum_complete: 16019811764
> rx23_csum_unnecessary: 0
> rx23_csum_unnecessary_inner: 0
> rx23_csum_none: 0
> rx23_xdp_drop: 0
> rx23_xdp_redirect: 0
> rx23_lro_packets: 0
> rx23_lro_bytes: 0
> rx23_ecn_mark: 0
> rx23_removed_vlan_packets: 16019811764
> rx23_wqe_err: 0
> rx23_mpwqe_filler_cqes: 0
> rx23_mpwqe_filler_strides: 0
> rx23_buff_alloc_err: 0
> rx23_cqe_compress_blks: 349733033
> rx23_cqe_compress_pkts: 781248862
> rx23_page_reuse: 0
> rx23_cache_reuse: 33422343
> rx23_cache_full: 7976481152
> rx23_cache_empty: 37376
> rx23_cache_busy: 7976483525
> rx23_cache_waive: 0
> rx23_congst_umr: 0
> rx23_arfs_err: 0
> rx23_xdp_tx_xmit: 0
> rx23_xdp_tx_full: 0
> rx23_xdp_tx_err: 0
> rx23_xdp_tx_cqes: 0
> rx24_packets: 16212040646
> rx24_bytes: 21393399325700
> rx24_csum_complete: 16212040646
> rx24_csum_unnecessary: 0
> rx24_csum_unnecessary_inner: 0
> rx24_csum_none: 0
> rx24_xdp_drop: 0
> rx24_xdp_redirect: 0
> rx24_lro_packets: 0
> rx24_lro_bytes: 0
> rx24_ecn_mark: 0
> rx24_removed_vlan_packets: 16212040646
> rx24_wqe_err: 0
> rx24_mpwqe_filler_cqes: 0
> rx24_mpwqe_filler_strides: 0
> rx24_buff_alloc_err: 0
> rx24_cqe_compress_blks: 379833752
> rx24_cqe_compress_pkts: 852020179
> rx24_page_reuse: 0
> rx24_cache_reuse: 8033552512
> rx24_cache_full: 72465843
> rx24_cache_empty: 37376
> rx24_cache_busy: 72467789
> rx24_cache_waive: 0
> rx24_congst_umr: 0
> rx24_arfs_err: 0
> rx24_xdp_tx_xmit: 0
> rx24_xdp_tx_full: 0
> rx24_xdp_tx_err: 0
> rx24_xdp_tx_cqes: 0
> rx25_packets: 16412186257
> rx25_bytes: 21651198388407
> rx25_csum_complete: 16412186257
> rx25_csum_unnecessary: 0
> rx25_csum_unnecessary_inner: 0
> rx25_csum_none: 0
> rx25_xdp_drop: 0
> rx25_xdp_redirect: 0
> rx25_lro_packets: 0
> rx25_lro_bytes: 0
> rx25_ecn_mark: 0
> rx25_removed_vlan_packets: 16412186257
> rx25_wqe_err: 0
> rx25_mpwqe_filler_cqes: 0
> rx25_mpwqe_filler_strides: 0
> rx25_buff_alloc_err: 0
> rx25_cqe_compress_blks: 383979685
> rx25_cqe_compress_pkts: 861985772
> rx25_page_reuse: 0
> rx25_cache_reuse: 8129807841
> rx25_cache_full: 76283342
> rx25_cache_empty: 37376
> rx25_cache_busy: 76285271
> rx25_cache_waive: 0
> rx25_congst_umr: 0
> rx25_arfs_err: 0
> rx25_xdp_tx_xmit: 0
> rx25_xdp_tx_full: 0
> rx25_xdp_tx_err: 0
> rx25_xdp_tx_cqes: 0
> rx26_packets: 16304310003
> rx26_bytes: 21571217538721
> rx26_csum_complete: 16304310003
> rx26_csum_unnecessary: 0
> rx26_csum_unnecessary_inner: 0
> rx26_csum_none: 0
> rx26_xdp_drop: 0
> rx26_xdp_redirect: 0
> rx26_lro_packets: 0
> rx26_lro_bytes: 0
> rx26_ecn_mark: 0
> rx26_removed_vlan_packets: 16304310003
> rx26_wqe_err: 0
> rx26_mpwqe_filler_cqes: 0
> rx26_mpwqe_filler_strides: 0
> rx26_buff_alloc_err: 0
> rx26_cqe_compress_blks: 353314041
> rx26_cqe_compress_pkts: 788838424
> rx26_page_reuse: 0
> rx26_cache_reuse: 19673790
> rx26_cache_full: 8132478659
> rx26_cache_empty: 37376
> rx26_cache_busy: 8132481198
> rx26_cache_waive: 0
> rx26_congst_umr: 0
> rx26_arfs_err: 0
> rx26_xdp_tx_xmit: 0
> rx26_xdp_tx_full: 0
> rx26_xdp_tx_err: 0
> rx26_xdp_tx_cqes: 0
> rx27_packets: 16171856079
> rx27_bytes: 21376891736540
> rx27_csum_complete: 16171856079
> rx27_csum_unnecessary: 0
> rx27_csum_unnecessary_inner: 0
> rx27_csum_none: 0
> rx27_xdp_drop: 0
> rx27_xdp_redirect: 0
> rx27_lro_packets: 0
> rx27_lro_bytes: 0
> rx27_ecn_mark: 0
> rx27_removed_vlan_packets: 16171856079
> rx27_wqe_err: 0
> rx27_mpwqe_filler_cqes: 0
> rx27_mpwqe_filler_strides: 0
> rx27_buff_alloc_err: 0
> rx27_cqe_compress_blks: 386632845
> rx27_cqe_compress_pkts: 869362576
> rx27_page_reuse: 0
> rx27_cache_reuse: 10070560
> rx27_cache_full: 8075854928
> rx27_cache_empty: 37376
> rx27_cache_busy: 8075857468
> rx27_cache_waive: 0
> rx27_congst_umr: 0
> rx27_arfs_err: 0
> rx27_xdp_tx_xmit: 0
> rx27_xdp_tx_full: 0
> rx27_xdp_tx_err: 0
> rx27_xdp_tx_cqes: 0
> rx28_packets: 0
> rx28_bytes: 0
> rx28_csum_complete: 0
> rx28_csum_unnecessary: 0
> rx28_csum_unnecessary_inner: 0
> rx28_csum_none: 0
> rx28_xdp_drop: 0
> rx28_xdp_redirect: 0
> rx28_lro_packets: 0
> rx28_lro_bytes: 0
> rx28_ecn_mark: 0
> rx28_removed_vlan_packets: 0
> rx28_wqe_err: 0
> rx28_mpwqe_filler_cqes: 0
> rx28_mpwqe_filler_strides: 0
> rx28_buff_alloc_err: 0
> rx28_cqe_compress_blks: 0
> rx28_cqe_compress_pkts: 0
> rx28_page_reuse: 0
> rx28_cache_reuse: 0
> rx28_cache_full: 0
> rx28_cache_empty: 2560
> rx28_cache_busy: 0
> rx28_cache_waive: 0
> rx28_congst_umr: 0
> rx28_arfs_err: 0
> rx28_xdp_tx_xmit: 0
> rx28_xdp_tx_full: 0
> rx28_xdp_tx_err: 0
> rx28_xdp_tx_cqes: 0
> rx29_packets: 0
> rx29_bytes: 0
> rx29_csum_complete: 0
> rx29_csum_unnecessary: 0
> rx29_csum_unnecessary_inner: 0
> rx29_csum_none: 0
> rx29_xdp_drop: 0
> rx29_xdp_redirect: 0
> rx29_lro_packets: 0
> rx29_lro_bytes: 0
> rx29_ecn_mark: 0
> rx29_removed_vlan_packets: 0
> rx29_wqe_err: 0
> rx29_mpwqe_filler_cqes: 0
> rx29_mpwqe_filler_strides: 0
> rx29_buff_alloc_err: 0
> rx29_cqe_compress_blks: 0
> rx29_cqe_compress_pkts: 0
> rx29_page_reuse: 0
> rx29_cache_reuse: 0
> rx29_cache_full: 0
> rx29_cache_empty: 2560
> rx29_cache_busy: 0
> rx29_cache_waive: 0
> rx29_congst_umr: 0
> rx29_arfs_err: 0
> rx29_xdp_tx_xmit: 0
> rx29_xdp_tx_full: 0
> rx29_xdp_tx_err: 0
> rx29_xdp_tx_cqes: 0
> rx30_packets: 0
> rx30_bytes: 0
> rx30_csum_complete: 0
> rx30_csum_unnecessary: 0
> rx30_csum_unnecessary_inner: 0
> rx30_csum_none: 0
> rx30_xdp_drop: 0
> rx30_xdp_redirect: 0
> rx30_lro_packets: 0
> rx30_lro_bytes: 0
> rx30_ecn_mark: 0
> rx30_removed_vlan_packets: 0
> rx30_wqe_err: 0
> rx30_mpwqe_filler_cqes: 0
> rx30_mpwqe_filler_strides: 0
> rx30_buff_alloc_err: 0
> rx30_cqe_compress_blks: 0
> rx30_cqe_compress_pkts: 0
> rx30_page_reuse: 0
> rx30_cache_reuse: 0
> rx30_cache_full: 0
> rx30_cache_empty: 2560
> rx30_cache_busy: 0
> rx30_cache_waive: 0
> rx30_congst_umr: 0
> rx30_arfs_err: 0
> rx30_xdp_tx_xmit: 0
> rx30_xdp_tx_full: 0
> rx30_xdp_tx_err: 0
> rx30_xdp_tx_cqes: 0
> rx31_packets: 0
> rx31_bytes: 0
> rx31_csum_complete: 0
> rx31_csum_unnecessary: 0
> rx31_csum_unnecessary_inner: 0
> rx31_csum_none: 0
> rx31_xdp_drop: 0
> rx31_xdp_redirect: 0
> rx31_lro_packets: 0
> rx31_lro_bytes: 0
> rx31_ecn_mark: 0
> rx31_removed_vlan_packets: 0
> rx31_wqe_err: 0
> rx31_mpwqe_filler_cqes: 0
> rx31_mpwqe_filler_strides: 0
> rx31_buff_alloc_err: 0
> rx31_cqe_compress_blks: 0
> rx31_cqe_compress_pkts: 0
> rx31_page_reuse: 0
> rx31_cache_reuse: 0
> rx31_cache_full: 0
> rx31_cache_empty: 2560
> rx31_cache_busy: 0
> rx31_cache_waive: 0
> rx31_congst_umr: 0
> rx31_arfs_err: 0
> rx31_xdp_tx_xmit: 0
> rx31_xdp_tx_full: 0
> rx31_xdp_tx_err: 0
> rx31_xdp_tx_cqes: 0
> rx32_packets: 0
> rx32_bytes: 0
> rx32_csum_complete: 0
> rx32_csum_unnecessary: 0
> rx32_csum_unnecessary_inner: 0
> rx32_csum_none: 0
> rx32_xdp_drop: 0
> rx32_xdp_redirect: 0
> rx32_lro_packets: 0
> rx32_lro_bytes: 0
> rx32_ecn_mark: 0
> rx32_removed_vlan_packets: 0
> rx32_wqe_err: 0
> rx32_mpwqe_filler_cqes: 0
> rx32_mpwqe_filler_strides: 0
> rx32_buff_alloc_err: 0
> rx32_cqe_compress_blks: 0
> rx32_cqe_compress_pkts: 0
> rx32_page_reuse: 0
> rx32_cache_reuse: 0
> rx32_cache_full: 0
> rx32_cache_empty: 2560
> rx32_cache_busy: 0
> rx32_cache_waive: 0
> rx32_congst_umr: 0
> rx32_arfs_err: 0
> rx32_xdp_tx_xmit: 0
> rx32_xdp_tx_full: 0
> rx32_xdp_tx_err: 0
> rx32_xdp_tx_cqes: 0
> rx33_packets: 0
> rx33_bytes: 0
> rx33_csum_complete: 0
> rx33_csum_unnecessary: 0
> rx33_csum_unnecessary_inner: 0
> rx33_csum_none: 0
> rx33_xdp_drop: 0
> rx33_xdp_redirect: 0
> rx33_lro_packets: 0
> rx33_lro_bytes: 0
> rx33_ecn_mark: 0
> rx33_removed_vlan_packets: 0
> rx33_wqe_err: 0
> rx33_mpwqe_filler_cqes: 0
> rx33_mpwqe_filler_strides: 0
> rx33_buff_alloc_err: 0
> rx33_cqe_compress_blks: 0
> rx33_cqe_compress_pkts: 0
> rx33_page_reuse: 0
> rx33_cache_reuse: 0
> rx33_cache_full: 0
> rx33_cache_empty: 2560
> rx33_cache_busy: 0
> rx33_cache_waive: 0
> rx33_congst_umr: 0
> rx33_arfs_err: 0
> rx33_xdp_tx_xmit: 0
> rx33_xdp_tx_full: 0
> rx33_xdp_tx_err: 0
> rx33_xdp_tx_cqes: 0
> rx34_packets: 0
> rx34_bytes: 0
> rx34_csum_complete: 0
> rx34_csum_unnecessary: 0
> rx34_csum_unnecessary_inner: 0
> rx34_csum_none: 0
> rx34_xdp_drop: 0
> rx34_xdp_redirect: 0
> rx34_lro_packets: 0
> rx34_lro_bytes: 0
> rx34_ecn_mark: 0
> rx34_removed_vlan_packets: 0
> rx34_wqe_err: 0
> rx34_mpwqe_filler_cqes: 0
> rx34_mpwqe_filler_strides: 0
> rx34_buff_alloc_err: 0
> rx34_cqe_compress_blks: 0
> rx34_cqe_compress_pkts: 0
> rx34_page_reuse: 0
> rx34_cache_reuse: 0
> rx34_cache_full: 0
> rx34_cache_empty: 2560
> rx34_cache_busy: 0
> rx34_cache_waive: 0
> rx34_congst_umr: 0
> rx34_arfs_err: 0
> rx34_xdp_tx_xmit: 0
> rx34_xdp_tx_full: 0
> rx34_xdp_tx_err: 0
> rx34_xdp_tx_cqes: 0
> rx35_packets: 0
> rx35_bytes: 0
> rx35_csum_complete: 0
> rx35_csum_unnecessary: 0
> rx35_csum_unnecessary_inner: 0
> rx35_csum_none: 0
> rx35_xdp_drop: 0
> rx35_xdp_redirect: 0
> rx35_lro_packets: 0
> rx35_lro_bytes: 0
> rx35_ecn_mark: 0
> rx35_removed_vlan_packets: 0
> rx35_wqe_err: 0
> rx35_mpwqe_filler_cqes: 0
> rx35_mpwqe_filler_strides: 0
> rx35_buff_alloc_err: 0
> rx35_cqe_compress_blks: 0
> rx35_cqe_compress_pkts: 0
> rx35_page_reuse: 0
> rx35_cache_reuse: 0
> rx35_cache_full: 0
> rx35_cache_empty: 2560
> rx35_cache_busy: 0
> rx35_cache_waive: 0
> rx35_congst_umr: 0
> rx35_arfs_err: 0
> rx35_xdp_tx_xmit: 0
> rx35_xdp_tx_full: 0
> rx35_xdp_tx_err: 0
> rx35_xdp_tx_cqes: 0
> rx36_packets: 0
> rx36_bytes: 0
> rx36_csum_complete: 0
> rx36_csum_unnecessary: 0
> rx36_csum_unnecessary_inner: 0
> rx36_csum_none: 0
> rx36_xdp_drop: 0
> rx36_xdp_redirect: 0
> rx36_lro_packets: 0
> rx36_lro_bytes: 0
> rx36_ecn_mark: 0
> rx36_removed_vlan_packets: 0
> rx36_wqe_err: 0
> rx36_mpwqe_filler_cqes: 0
> rx36_mpwqe_filler_strides: 0
> rx36_buff_alloc_err: 0
> rx36_cqe_compress_blks: 0
> rx36_cqe_compress_pkts: 0
> rx36_page_reuse: 0
> rx36_cache_reuse: 0
> rx36_cache_full: 0
> rx36_cache_empty: 2560
> rx36_cache_busy: 0
> rx36_cache_waive: 0
> rx36_congst_umr: 0
> rx36_arfs_err: 0
> rx36_xdp_tx_xmit: 0
> rx36_xdp_tx_full: 0
> rx36_xdp_tx_err: 0
> rx36_xdp_tx_cqes: 0
> rx37_packets: 0
> rx37_bytes: 0
> rx37_csum_complete: 0
> rx37_csum_unnecessary: 0
> rx37_csum_unnecessary_inner: 0
> rx37_csum_none: 0
> rx37_xdp_drop: 0
> rx37_xdp_redirect: 0
> rx37_lro_packets: 0
> rx37_lro_bytes: 0
> rx37_ecn_mark: 0
> rx37_removed_vlan_packets: 0
> rx37_wqe_err: 0
> rx37_mpwqe_filler_cqes: 0
> rx37_mpwqe_filler_strides: 0
> rx37_buff_alloc_err: 0
> rx37_cqe_compress_blks: 0
> rx37_cqe_compress_pkts: 0
> rx37_page_reuse: 0
> rx37_cache_reuse: 0
> rx37_cache_full: 0
> rx37_cache_empty: 2560
> rx37_cache_busy: 0
> rx37_cache_waive: 0
> rx37_congst_umr: 0
> rx37_arfs_err: 0
> rx37_xdp_tx_xmit: 0
> rx37_xdp_tx_full: 0
> rx37_xdp_tx_err: 0
> rx37_xdp_tx_cqes: 0
> rx38_packets: 0
> rx38_bytes: 0
> rx38_csum_complete: 0
> rx38_csum_unnecessary: 0
> rx38_csum_unnecessary_inner: 0
> rx38_csum_none: 0
> rx38_xdp_drop: 0
> rx38_xdp_redirect: 0
> rx38_lro_packets: 0
> rx38_lro_bytes: 0
> rx38_ecn_mark: 0
> rx38_removed_vlan_packets: 0
> rx38_wqe_err: 0
> rx38_mpwqe_filler_cqes: 0
> rx38_mpwqe_filler_strides: 0
> rx38_buff_alloc_err: 0
> rx38_cqe_compress_blks: 0
> rx38_cqe_compress_pkts: 0
> rx38_page_reuse: 0
> rx38_cache_reuse: 0
> rx38_cache_full: 0
> rx38_cache_empty: 2560
> rx38_cache_busy: 0
> rx38_cache_waive: 0
> rx38_congst_umr: 0
> rx38_arfs_err: 0
> rx38_xdp_tx_xmit: 0
> rx38_xdp_tx_full: 0
> rx38_xdp_tx_err: 0
> rx38_xdp_tx_cqes: 0
> rx39_packets: 0
> rx39_bytes: 0
> rx39_csum_complete: 0
> rx39_csum_unnecessary: 0
> rx39_csum_unnecessary_inner: 0
> rx39_csum_none: 0
> rx39_xdp_drop: 0
> rx39_xdp_redirect: 0
> rx39_lro_packets: 0
> rx39_lro_bytes: 0
> rx39_ecn_mark: 0
> rx39_removed_vlan_packets: 0
> rx39_wqe_err: 0
> rx39_mpwqe_filler_cqes: 0
> rx39_mpwqe_filler_strides: 0
> rx39_buff_alloc_err: 0
> rx39_cqe_compress_blks: 0
> rx39_cqe_compress_pkts: 0
> rx39_page_reuse: 0
> rx39_cache_reuse: 0
> rx39_cache_full: 0
> rx39_cache_empty: 2560
> rx39_cache_busy: 0
> rx39_cache_waive: 0
> rx39_congst_umr: 0
> rx39_arfs_err: 0
> rx39_xdp_tx_xmit: 0
> rx39_xdp_tx_full: 0
> rx39_xdp_tx_err: 0
> rx39_xdp_tx_cqes: 0
> rx40_packets: 0
> rx40_bytes: 0
> rx40_csum_complete: 0
> rx40_csum_unnecessary: 0
> rx40_csum_unnecessary_inner: 0
> rx40_csum_none: 0
> rx40_xdp_drop: 0
> rx40_xdp_redirect: 0
> rx40_lro_packets: 0
> rx40_lro_bytes: 0
> rx40_ecn_mark: 0
> rx40_removed_vlan_packets: 0
> rx40_wqe_err: 0
> rx40_mpwqe_filler_cqes: 0
> rx40_mpwqe_filler_strides: 0
> rx40_buff_alloc_err: 0
> rx40_cqe_compress_blks: 0
> rx40_cqe_compress_pkts: 0
> rx40_page_reuse: 0
> rx40_cache_reuse: 0
> rx40_cache_full: 0
> rx40_cache_empty: 2560
> rx40_cache_busy: 0
> rx40_cache_waive: 0
> rx40_congst_umr: 0
> rx40_arfs_err: 0
> rx40_xdp_tx_xmit: 0
> rx40_xdp_tx_full: 0
> rx40_xdp_tx_err: 0
> rx40_xdp_tx_cqes: 0
> rx41_packets: 0
> rx41_bytes: 0
> rx41_csum_complete: 0
> rx41_csum_unnecessary: 0
> rx41_csum_unnecessary_inner: 0
> rx41_csum_none: 0
> rx41_xdp_drop: 0
> rx41_xdp_redirect: 0
> rx41_lro_packets: 0
> rx41_lro_bytes: 0
> rx41_ecn_mark: 0
> rx41_removed_vlan_packets: 0
> rx41_wqe_err: 0
> rx41_mpwqe_filler_cqes: 0
> rx41_mpwqe_filler_strides: 0
> rx41_buff_alloc_err: 0
> rx41_cqe_compress_blks: 0
> rx41_cqe_compress_pkts: 0
> rx41_page_reuse: 0
> rx41_cache_reuse: 0
> rx41_cache_full: 0
> rx41_cache_empty: 2560
> rx41_cache_busy: 0
> rx41_cache_waive: 0
> rx41_congst_umr: 0
> rx41_arfs_err: 0
> rx41_xdp_tx_xmit: 0
> rx41_xdp_tx_full: 0
> rx41_xdp_tx_err: 0
> rx41_xdp_tx_cqes: 0
> rx42_packets: 0
> rx42_bytes: 0
> rx42_csum_complete: 0
> rx42_csum_unnecessary: 0
> rx42_csum_unnecessary_inner: 0
> rx42_csum_none: 0
> rx42_xdp_drop: 0
> rx42_xdp_redirect: 0
> rx42_lro_packets: 0
> rx42_lro_bytes: 0
> rx42_ecn_mark: 0
> rx42_removed_vlan_packets: 0
> rx42_wqe_err: 0
> rx42_mpwqe_filler_cqes: 0
> rx42_mpwqe_filler_strides: 0
> rx42_buff_alloc_err: 0
> rx42_cqe_compress_blks: 0
> rx42_cqe_compress_pkts: 0
> rx42_page_reuse: 0
> rx42_cache_reuse: 0
> rx42_cache_full: 0
> rx42_cache_empty: 2560
> rx42_cache_busy: 0
> rx42_cache_waive: 0
> rx42_congst_umr: 0
> rx42_arfs_err: 0
> rx42_xdp_tx_xmit: 0
> rx42_xdp_tx_full: 0
> rx42_xdp_tx_err: 0
> rx42_xdp_tx_cqes: 0
> rx43_packets: 0
> rx43_bytes: 0
> rx43_csum_complete: 0
> rx43_csum_unnecessary: 0
> rx43_csum_unnecessary_inner: 0
> rx43_csum_none: 0
> rx43_xdp_drop: 0
> rx43_xdp_redirect: 0
> rx43_lro_packets: 0
> rx43_lro_bytes: 0
> rx43_ecn_mark: 0
> rx43_removed_vlan_packets: 0
> rx43_wqe_err: 0
> rx43_mpwqe_filler_cqes: 0
> rx43_mpwqe_filler_strides: 0
> rx43_buff_alloc_err: 0
> rx43_cqe_compress_blks: 0
> rx43_cqe_compress_pkts: 0
> rx43_page_reuse: 0
> rx43_cache_reuse: 0
> rx43_cache_full: 0
> rx43_cache_empty: 2560
> rx43_cache_busy: 0
> rx43_cache_waive: 0
> rx43_congst_umr: 0
> rx43_arfs_err: 0
> rx43_xdp_tx_xmit: 0
> rx43_xdp_tx_full: 0
> rx43_xdp_tx_err: 0
> rx43_xdp_tx_cqes: 0
> rx44_packets: 0
> rx44_bytes: 0
> rx44_csum_complete: 0
> rx44_csum_unnecessary: 0
> rx44_csum_unnecessary_inner: 0
> rx44_csum_none: 0
> rx44_xdp_drop: 0
> rx44_xdp_redirect: 0
> rx44_lro_packets: 0
> rx44_lro_bytes: 0
> rx44_ecn_mark: 0
> rx44_removed_vlan_packets: 0
> rx44_wqe_err: 0
> rx44_mpwqe_filler_cqes: 0
> rx44_mpwqe_filler_strides: 0
> rx44_buff_alloc_err: 0
> rx44_cqe_compress_blks: 0
> rx44_cqe_compress_pkts: 0
> rx44_page_reuse: 0
> rx44_cache_reuse: 0
> rx44_cache_full: 0
> rx44_cache_empty: 2560
> rx44_cache_busy: 0
> rx44_cache_waive: 0
> rx44_congst_umr: 0
> rx44_arfs_err: 0
> rx44_xdp_tx_xmit: 0
> rx44_xdp_tx_full: 0
> rx44_xdp_tx_err: 0
> rx44_xdp_tx_cqes: 0
> rx45_packets: 0
> rx45_bytes: 0
> rx45_csum_complete: 0
> rx45_csum_unnecessary: 0
> rx45_csum_unnecessary_inner: 0
> rx45_csum_none: 0
> rx45_xdp_drop: 0
> rx45_xdp_redirect: 0
> rx45_lro_packets: 0
> rx45_lro_bytes: 0
> rx45_ecn_mark: 0
> rx45_removed_vlan_packets: 0
> rx45_wqe_err: 0
> rx45_mpwqe_filler_cqes: 0
> rx45_mpwqe_filler_strides: 0
> rx45_buff_alloc_err: 0
> rx45_cqe_compress_blks: 0
> rx45_cqe_compress_pkts: 0
> rx45_page_reuse: 0
> rx45_cache_reuse: 0
> rx45_cache_full: 0
> rx45_cache_empty: 2560
> rx45_cache_busy: 0
> rx45_cache_waive: 0
> rx45_congst_umr: 0
> rx45_arfs_err: 0
> rx45_xdp_tx_xmit: 0
> rx45_xdp_tx_full: 0
> rx45_xdp_tx_err: 0
> rx45_xdp_tx_cqes: 0
> rx46_packets: 0
> rx46_bytes: 0
> rx46_csum_complete: 0
> rx46_csum_unnecessary: 0
> rx46_csum_unnecessary_inner: 0
> rx46_csum_none: 0
> rx46_xdp_drop: 0
> rx46_xdp_redirect: 0
> rx46_lro_packets: 0
> rx46_lro_bytes: 0
> rx46_ecn_mark: 0
> rx46_removed_vlan_packets: 0
> rx46_wqe_err: 0
> rx46_mpwqe_filler_cqes: 0
> rx46_mpwqe_filler_strides: 0
> rx46_buff_alloc_err: 0
> rx46_cqe_compress_blks: 0
> rx46_cqe_compress_pkts: 0
> rx46_page_reuse: 0
> rx46_cache_reuse: 0
> rx46_cache_full: 0
> rx46_cache_empty: 2560
> rx46_cache_busy: 0
> rx46_cache_waive: 0
> rx46_congst_umr: 0
> rx46_arfs_err: 0
> rx46_xdp_tx_xmit: 0
> rx46_xdp_tx_full: 0
> rx46_xdp_tx_err: 0
> rx46_xdp_tx_cqes: 0
> rx47_packets: 0
> rx47_bytes: 0
> rx47_csum_complete: 0
> rx47_csum_unnecessary: 0
> rx47_csum_unnecessary_inner: 0
> rx47_csum_none: 0
> rx47_xdp_drop: 0
> rx47_xdp_redirect: 0
> rx47_lro_packets: 0
> rx47_lro_bytes: 0
> rx47_ecn_mark: 0
> rx47_removed_vlan_packets: 0
> rx47_wqe_err: 0
> rx47_mpwqe_filler_cqes: 0
> rx47_mpwqe_filler_strides: 0
> rx47_buff_alloc_err: 0
> rx47_cqe_compress_blks: 0
> rx47_cqe_compress_pkts: 0
> rx47_page_reuse: 0
> rx47_cache_reuse: 0
> rx47_cache_full: 0
> rx47_cache_empty: 2560
> rx47_cache_busy: 0
> rx47_cache_waive: 0
> rx47_congst_umr: 0
> rx47_arfs_err: 0
> rx47_xdp_tx_xmit: 0
> rx47_xdp_tx_full: 0
> rx47_xdp_tx_err: 0
> rx47_xdp_tx_cqes: 0
> rx48_packets: 0
> rx48_bytes: 0
> rx48_csum_complete: 0
> rx48_csum_unnecessary: 0
> rx48_csum_unnecessary_inner: 0
> rx48_csum_none: 0
> rx48_xdp_drop: 0
> rx48_xdp_redirect: 0
> rx48_lro_packets: 0
> rx48_lro_bytes: 0
> rx48_ecn_mark: 0
> rx48_removed_vlan_packets: 0
> rx48_wqe_err: 0
> rx48_mpwqe_filler_cqes: 0
> rx48_mpwqe_filler_strides: 0
> rx48_buff_alloc_err: 0
> rx48_cqe_compress_blks: 0
> rx48_cqe_compress_pkts: 0
> rx48_page_reuse: 0
> rx48_cache_reuse: 0
> rx48_cache_full: 0
> rx48_cache_empty: 2560
> rx48_cache_busy: 0
> rx48_cache_waive: 0
> rx48_congst_umr: 0
> rx48_arfs_err: 0
> rx48_xdp_tx_xmit: 0
> rx48_xdp_tx_full: 0
> rx48_xdp_tx_err: 0
> rx48_xdp_tx_cqes: 0
> rx49_packets: 0
> rx49_bytes: 0
> rx49_csum_complete: 0
> rx49_csum_unnecessary: 0
> rx49_csum_unnecessary_inner: 0
> rx49_csum_none: 0
> rx49_xdp_drop: 0
> rx49_xdp_redirect: 0
> rx49_lro_packets: 0
> rx49_lro_bytes: 0
> rx49_ecn_mark: 0
> rx49_removed_vlan_packets: 0
> rx49_wqe_err: 0
> rx49_mpwqe_filler_cqes: 0
> rx49_mpwqe_filler_strides: 0
> rx49_buff_alloc_err: 0
> rx49_cqe_compress_blks: 0
> rx49_cqe_compress_pkts: 0
> rx49_page_reuse: 0
> rx49_cache_reuse: 0
> rx49_cache_full: 0
> rx49_cache_empty: 2560
> rx49_cache_busy: 0
> rx49_cache_waive: 0
> rx49_congst_umr: 0
> rx49_arfs_err: 0
> rx49_xdp_tx_xmit: 0
> rx49_xdp_tx_full: 0
> rx49_xdp_tx_err: 0
> rx49_xdp_tx_cqes: 0
> rx50_packets: 0
> rx50_bytes: 0
> rx50_csum_complete: 0
> rx50_csum_unnecessary: 0
> rx50_csum_unnecessary_inner: 0
> rx50_csum_none: 0
> rx50_xdp_drop: 0
> rx50_xdp_redirect: 0
> rx50_lro_packets: 0
> rx50_lro_bytes: 0
> rx50_ecn_mark: 0
> rx50_removed_vlan_packets: 0
> rx50_wqe_err: 0
> rx50_mpwqe_filler_cqes: 0
> rx50_mpwqe_filler_strides: 0
> rx50_buff_alloc_err: 0
> rx50_cqe_compress_blks: 0
> rx50_cqe_compress_pkts: 0
> rx50_page_reuse: 0
> rx50_cache_reuse: 0
> rx50_cache_full: 0
> rx50_cache_empty: 2560
> rx50_cache_busy: 0
> rx50_cache_waive: 0
> rx50_congst_umr: 0
> rx50_arfs_err: 0
> rx50_xdp_tx_xmit: 0
> rx50_xdp_tx_full: 0
> rx50_xdp_tx_err: 0
> rx50_xdp_tx_cqes: 0
> rx51_packets: 0
> rx51_bytes: 0
> rx51_csum_complete: 0
> rx51_csum_unnecessary: 0
> rx51_csum_unnecessary_inner: 0
> rx51_csum_none: 0
> rx51_xdp_drop: 0
> rx51_xdp_redirect: 0
> rx51_lro_packets: 0
> rx51_lro_bytes: 0
> rx51_ecn_mark: 0
> rx51_removed_vlan_packets: 0
> rx51_wqe_err: 0
> rx51_mpwqe_filler_cqes: 0
> rx51_mpwqe_filler_strides: 0
> rx51_buff_alloc_err: 0
> rx51_cqe_compress_blks: 0
> rx51_cqe_compress_pkts: 0
> rx51_page_reuse: 0
> rx51_cache_reuse: 0
> rx51_cache_full: 0
> rx51_cache_empty: 2560
> rx51_cache_busy: 0
> rx51_cache_waive: 0
> rx51_congst_umr: 0
> rx51_arfs_err: 0
> rx51_xdp_tx_xmit: 0
> rx51_xdp_tx_full: 0
> rx51_xdp_tx_err: 0
> rx51_xdp_tx_cqes: 0
> rx52_packets: 0
> rx52_bytes: 0
> rx52_csum_complete: 0
> rx52_csum_unnecessary: 0
> rx52_csum_unnecessary_inner: 0
> rx52_csum_none: 0
> rx52_xdp_drop: 0
> rx52_xdp_redirect: 0
> rx52_lro_packets: 0
> rx52_lro_bytes: 0
> rx52_ecn_mark: 0
> rx52_removed_vlan_packets: 0
> rx52_wqe_err: 0
> rx52_mpwqe_filler_cqes: 0
> rx52_mpwqe_filler_strides: 0
> rx52_buff_alloc_err: 0
> rx52_cqe_compress_blks: 0
> rx52_cqe_compress_pkts: 0
> rx52_page_reuse: 0
> rx52_cache_reuse: 0
> rx52_cache_full: 0
> rx52_cache_empty: 2560
> rx52_cache_busy: 0
> rx52_cache_waive: 0
> rx52_congst_umr: 0
> rx52_arfs_err: 0
> rx52_xdp_tx_xmit: 0
> rx52_xdp_tx_full: 0
> rx52_xdp_tx_err: 0
> rx52_xdp_tx_cqes: 0
> rx53_packets: 0
> rx53_bytes: 0
> rx53_csum_complete: 0
> rx53_csum_unnecessary: 0
> rx53_csum_unnecessary_inner: 0
> rx53_csum_none: 0
> rx53_xdp_drop: 0
> rx53_xdp_redirect: 0
> rx53_lro_packets: 0
> rx53_lro_bytes: 0
> rx53_ecn_mark: 0
> rx53_removed_vlan_packets: 0
> rx53_wqe_err: 0
> rx53_mpwqe_filler_cqes: 0
> rx53_mpwqe_filler_strides: 0
> rx53_buff_alloc_err: 0
> rx53_cqe_compress_blks: 0
> rx53_cqe_compress_pkts: 0
> rx53_page_reuse: 0
> rx53_cache_reuse: 0
> rx53_cache_full: 0
> rx53_cache_empty: 2560
> rx53_cache_busy: 0
> rx53_cache_waive: 0
> rx53_congst_umr: 0
> rx53_arfs_err: 0
> rx53_xdp_tx_xmit: 0
> rx53_xdp_tx_full: 0
> rx53_xdp_tx_err: 0
> rx53_xdp_tx_cqes: 0
> rx54_packets: 0
> rx54_bytes: 0
> rx54_csum_complete: 0
> rx54_csum_unnecessary: 0
> rx54_csum_unnecessary_inner: 0
> rx54_csum_none: 0
> rx54_xdp_drop: 0
> rx54_xdp_redirect: 0
> rx54_lro_packets: 0
> rx54_lro_bytes: 0
> rx54_ecn_mark: 0
> rx54_removed_vlan_packets: 0
> rx54_wqe_err: 0
> rx54_mpwqe_filler_cqes: 0
> rx54_mpwqe_filler_strides: 0
> rx54_buff_alloc_err: 0
> rx54_cqe_compress_blks: 0
> rx54_cqe_compress_pkts: 0
> rx54_page_reuse: 0
> rx54_cache_reuse: 0
> rx54_cache_full: 0
> rx54_cache_empty: 2560
> rx54_cache_busy: 0
> rx54_cache_waive: 0
> rx54_congst_umr: 0
> rx54_arfs_err: 0
> rx54_xdp_tx_xmit: 0
> rx54_xdp_tx_full: 0
> rx54_xdp_tx_err: 0
> rx54_xdp_tx_cqes: 0
> rx55_packets: 0
> rx55_bytes: 0
> rx55_csum_complete: 0
> rx55_csum_unnecessary: 0
> rx55_csum_unnecessary_inner: 0
> rx55_csum_none: 0
> rx55_xdp_drop: 0
> rx55_xdp_redirect: 0
> rx55_lro_packets: 0
> rx55_lro_bytes: 0
> rx55_ecn_mark: 0
> rx55_removed_vlan_packets: 0
> rx55_wqe_err: 0
> rx55_mpwqe_filler_cqes: 0
> rx55_mpwqe_filler_strides: 0
> rx55_buff_alloc_err: 0
> rx55_cqe_compress_blks: 0
> rx55_cqe_compress_pkts: 0
> rx55_page_reuse: 0
> rx55_cache_reuse: 0
> rx55_cache_full: 0
> rx55_cache_empty: 2560
> rx55_cache_busy: 0
> rx55_cache_waive: 0
> rx55_congst_umr: 0
> rx55_arfs_err: 0
> rx55_xdp_tx_xmit: 0
> rx55_xdp_tx_full: 0
> rx55_xdp_tx_err: 0
> rx55_xdp_tx_cqes: 0
> tx0_packets: 24512439668
> tx0_bytes: 15287569052791
> tx0_tso_packets: 1536157106
> tx0_tso_bytes: 8571753637944
> tx0_tso_inner_packets: 0
> tx0_tso_inner_bytes: 0
> tx0_csum_partial: 2132156117
> tx0_csum_partial_inner: 0
> tx0_added_vlan_packets: 19906601448
> tx0_nop: 308098536
> tx0_csum_none: 17774445331
> tx0_stopped: 19625
> tx0_dropped: 0
> tx0_xmit_more: 67864870
> tx0_recover: 0
> tx0_cqes: 19838744246
> tx0_wake: 19624
> tx0_cqe_err: 0
> tx1_packets: 22598557053
> tx1_bytes: 13568850145010
> tx1_tso_packets: 1369529475
> tx1_tso_bytes: 7661777265382
> tx1_tso_inner_packets: 0
> tx1_tso_inner_bytes: 0
> tx1_csum_partial: 1884639496
> tx1_csum_partial_inner: 0
> tx1_added_vlan_packets: 18468333696
> tx1_nop: 281301783
> tx1_csum_none: 16583694200
> tx1_stopped: 19457
> tx1_dropped: 0
> tx1_xmit_more: 55170875
> tx1_recover: 0
> tx1_cqes: 18413169824
> tx1_wake: 19455
> tx1_cqe_err: 0
> tx2_packets: 22821611433
> tx2_bytes: 13752535163683
> tx2_tso_packets: 1396978825
> tx2_tso_bytes: 7774704508463
> tx2_tso_inner_packets: 0
> tx2_tso_inner_bytes: 0
> tx2_csum_partial: 1897834558
> tx2_csum_partial_inner: 0
> tx2_added_vlan_packets: 18641958085
> tx2_nop: 286934891
> tx2_csum_none: 16744123527
> tx2_stopped: 13214
> tx2_dropped: 0
> tx2_xmit_more: 61749446
> tx2_recover: 0
> tx2_cqes: 18580215654
> tx2_wake: 13214
> tx2_cqe_err: 0
> tx3_packets: 22580809948
> tx3_bytes: 13730542936609
> tx3_tso_packets: 1370434579
> tx3_tso_bytes: 7605636711455
> tx3_tso_inner_packets: 0
> tx3_tso_inner_bytes: 0
> tx3_csum_partial: 1865573748
> tx3_csum_partial_inner: 0
> tx3_added_vlan_packets: 18491873644
> tx3_nop: 281195875
> tx3_csum_none: 16626299896
> tx3_stopped: 12542
> tx3_dropped: 0
> tx3_xmit_more: 57681647
> tx3_recover: 0
> tx3_cqes: 18434198757
> tx3_wake: 12540
> tx3_cqe_err: 0
> tx4_packets: 27801801208
> tx4_bytes: 17058453171137
> tx4_tso_packets: 1740500105
> tx4_tso_bytes: 9474905622036
> tx4_tso_inner_packets: 0
> tx4_tso_inner_bytes: 0
> tx4_csum_partial: 2279225376
> tx4_csum_partial_inner: 0
> tx4_added_vlan_packets: 22744081633
> tx4_nop: 349753979
> tx4_csum_none: 20464856257
> tx4_stopped: 14816
> tx4_dropped: 0
> tx4_xmit_more: 65469322
> tx4_recover: 0
> tx4_cqes: 22678618972
> tx4_wake: 14816
> tx4_cqe_err: 0
> tx5_packets: 25099783024
> tx5_bytes: 14917740698381
> tx5_tso_packets: 1512988013
> tx5_tso_bytes: 8571208921023
> tx5_tso_inner_packets: 0
> tx5_tso_inner_bytes: 0
> tx5_csum_partial: 2078498561
> tx5_csum_partial_inner: 0
> tx5_added_vlan_packets: 20465533760
> tx5_nop: 312614719
> tx5_csum_none: 18387035199
> tx5_stopped: 4605
> tx5_dropped: 0
> tx5_xmit_more: 64188936
> tx5_recover: 0
> tx5_cqes: 20401350718
> tx5_wake: 4604
> tx5_cqe_err: 0
> tx6_packets: 25025504896
> tx6_bytes: 14908021946070
> tx6_tso_packets: 1515718977
> tx6_tso_bytes: 8511442522461
> tx6_tso_inner_packets: 0
> tx6_tso_inner_bytes: 0
> tx6_csum_partial: 2056378610
> tx6_csum_partial_inner: 0
> tx6_added_vlan_packets: 20434066400
> tx6_nop: 310594020
> tx6_csum_none: 18377687790
> tx6_stopped: 15234
> tx6_dropped: 0
> tx6_xmit_more: 61130422
> tx6_recover: 0
> tx6_cqes: 20372943611
> tx6_wake: 15234
> tx6_cqe_err: 0
> tx7_packets: 25457096169
> tx7_bytes: 15456289446172
> tx7_tso_packets: 1553342799
> tx7_tso_bytes: 8764550988105
> tx7_tso_inner_packets: 0
> tx7_tso_inner_bytes: 0
> tx7_csum_partial: 2105765233
> tx7_csum_partial_inner: 0
> tx7_added_vlan_packets: 20720382377
> tx7_nop: 319044853
> tx7_csum_none: 18614617145
> tx7_stopped: 18745
> tx7_dropped: 0
> tx7_xmit_more: 57050107
> tx7_recover: 0
> tx7_cqes: 20663340775
> tx7_wake: 18746
> tx7_cqe_err: 0
> tx8_packets: 25389771649
> tx8_bytes: 15225503883962
> tx8_tso_packets: 1563367648
> tx8_tso_bytes: 8710384514258
> tx8_tso_inner_packets: 0
> tx8_tso_inner_bytes: 0
> tx8_csum_partial: 2106586634
> tx8_csum_partial_inner: 0
> tx8_added_vlan_packets: 20704676274
> tx8_nop: 318149261
> tx8_csum_none: 18598089640
> tx8_stopped: 4733
> tx8_dropped: 0
> tx8_xmit_more: 61014317
> tx8_recover: 0
> tx8_cqes: 20643667301
> tx8_wake: 4735
> tx8_cqe_err: 0
> tx9_packets: 25521500166
> tx9_bytes: 15302293145755
> tx9_tso_packets: 1546316697
> tx9_tso_bytes: 8770688145926
> tx9_tso_inner_packets: 0
> tx9_tso_inner_bytes: 0
> tx9_csum_partial: 2097652880
> tx9_csum_partial_inner: 0
> tx9_added_vlan_packets: 20778408432
> tx9_nop: 318538543
> tx9_csum_none: 18680755556
> tx9_stopped: 16118
> tx9_dropped: 0
> tx9_xmit_more: 68509728
> tx9_recover: 0
> tx9_cqes: 20709906498
> tx9_wake: 16118
> tx9_cqe_err: 0
> tx10_packets: 25451605829
> tx10_bytes: 15386896170792
> tx10_tso_packets: 1576473520
> tx10_tso_bytes: 8880888676383
> tx10_tso_inner_packets: 0
> tx10_tso_inner_bytes: 0
> tx10_csum_partial: 2129796141
> tx10_csum_partial_inner: 0
> tx10_added_vlan_packets: 20659622590
> tx10_nop: 319117433
> tx10_csum_none: 18529826450
> tx10_stopped: 20187
> tx10_dropped: 0
> tx10_xmit_more: 58892184
> tx10_recover: 0
> tx10_cqes: 20600737739
> tx10_wake: 20188
> tx10_cqe_err: 0
> tx11_packets: 27008919793
> tx11_bytes: 16587719213058
> tx11_tso_packets: 1734884654
> tx11_tso_bytes: 9475681471870
> tx11_tso_inner_packets: 0
> tx11_tso_inner_bytes: 0
> tx11_csum_partial: 2296162292
> tx11_csum_partial_inner: 0
> tx11_added_vlan_packets: 21943096263
> tx11_nop: 344188182
> tx11_csum_none: 19646933971
> tx11_stopped: 9703
> tx11_dropped: 0
> tx11_xmit_more: 66530718
> tx11_recover: 0
> tx11_cqes: 21876571667
> tx11_wake: 9704
> tx11_cqe_err: 0
> tx12_packets: 25969493269
> tx12_bytes: 15980767963416
> tx12_tso_packets: 1671396456
> tx12_tso_bytes: 9268973672821
> tx12_tso_inner_packets: 0
> tx12_tso_inner_bytes: 0
> tx12_csum_partial: 2243809182
> tx12_csum_partial_inner: 0
> tx12_added_vlan_packets: 20980642456
> tx12_nop: 330241007
> tx12_csum_none: 18736833276
> tx12_stopped: 10341
> tx12_dropped: 0
> tx12_xmit_more: 57834100
> tx12_recover: 0
> tx12_cqes: 20922815079
> tx12_wake: 10342
> tx12_cqe_err: 0
> tx13_packets: 25332762261
> tx13_bytes: 15353213283280
> tx13_tso_packets: 1577433599
> tx13_tso_bytes: 8785240284281
> tx13_tso_inner_packets: 0
> tx13_tso_inner_bytes: 0
> tx13_csum_partial: 2110640515
> tx13_csum_partial_inner: 0
> tx13_added_vlan_packets: 20605670910
> tx13_nop: 319805741
> tx13_csum_none: 18495030395
> tx13_stopped: 7006
> tx13_dropped: 0
> tx13_xmit_more: 58314402
> tx13_recover: 0
> tx13_cqes: 20547362770
> tx13_wake: 7008
> tx13_cqe_err: 0
> tx14_packets: 26333743548
> tx14_bytes: 16070719060573
> tx14_tso_packets: 1677922970
> tx14_tso_bytes: 9240299765487
> tx14_tso_inner_packets: 0
> tx14_tso_inner_bytes: 0
> tx14_csum_partial: 2215668906
> tx14_csum_partial_inner: 0
> tx14_added_vlan_packets: 21384410786
> tx14_nop: 332734939
> tx14_csum_none: 19168741880
> tx14_stopped: 13160
> tx14_dropped: 0
> tx14_xmit_more: 57650391
> tx14_recover: 0
> tx14_cqes: 21326767783
> tx14_wake: 13161
> tx14_cqe_err: 0
> tx15_packets: 26824968971
> tx15_bytes: 16687994233452
> tx15_tso_packets: 1755745052
> tx15_tso_bytes: 9533814012441
> tx15_tso_inner_packets: 0
> tx15_tso_inner_bytes: 0
> tx15_csum_partial: 2304778064
> tx15_csum_partial_inner: 0
> tx15_added_vlan_packets: 21740906107
> tx15_nop: 344143287
> tx15_csum_none: 19436128058
> tx15_stopped: 75
> tx15_dropped: 0
> tx15_xmit_more: 63325832
> tx15_recover: 0
> tx15_cqes: 21677585345
> tx15_wake: 74
> tx15_cqe_err: 0
> tx16_packets: 24488158946
> tx16_bytes: 15027415004570
> tx16_tso_packets: 1559127391
> tx16_tso_bytes: 8658691917845
> tx16_tso_inner_packets: 0
> tx16_tso_inner_bytes: 0
> tx16_csum_partial: 2075856395
> tx16_csum_partial_inner: 0
> tx16_added_vlan_packets: 19835695731
> tx16_nop: 308464189
> tx16_csum_none: 17759839340
> tx16_stopped: 4567
> tx16_dropped: 0
> tx16_xmit_more: 62631422
> tx16_recover: 0
> tx16_cqes: 19773070012
> tx16_wake: 4568
> tx16_cqe_err: 0
> tx17_packets: 24700413784
> tx17_bytes: 15216529713715
> tx17_tso_packets: 1597555108
> tx17_tso_bytes: 8773728661243
> tx17_tso_inner_packets: 0
> tx17_tso_inner_bytes: 0
> tx17_csum_partial: 2127177297
> tx17_csum_partial_inner: 0
> tx17_added_vlan_packets: 20003144561
> tx17_nop: 313356918
> tx17_csum_none: 17875967264
> tx17_stopped: 12572
> tx17_dropped: 0
> tx17_xmit_more: 62742980
> tx17_recover: 0
> tx17_cqes: 19940407615
> tx17_wake: 12573
> tx17_cqe_err: 0
> tx18_packets: 24887710046
> tx18_bytes: 15245034034664
> tx18_tso_packets: 1582550520
> tx18_tso_bytes: 8782692335483
> tx18_tso_inner_packets: 0
> tx18_tso_inner_bytes: 0
> tx18_csum_partial: 2084514331
> tx18_csum_partial_inner: 0
> tx18_added_vlan_packets: 20173879181
> tx18_nop: 314818702
> tx18_csum_none: 18089364850
> tx18_stopped: 21366
> tx18_dropped: 0
> tx18_xmit_more: 62485819
> tx18_recover: 0
> tx18_cqes: 20111400935
> tx18_wake: 21366
> tx18_cqe_err: 0
> tx19_packets: 24831057648
> tx19_bytes: 15164663890576
> tx19_tso_packets: 1599135489
> tx19_tso_bytes: 8756045449746
> tx19_tso_inner_packets: 0
> tx19_tso_inner_bytes: 0
> tx19_csum_partial: 2119746608
> tx19_csum_partial_inner: 0
> tx19_added_vlan_packets: 20143573903
> tx19_nop: 316966450
> tx19_csum_none: 18023827295
> tx19_stopped: 11431
> tx19_dropped: 0
> tx19_xmit_more: 57535904
> tx19_recover: 0
> tx19_cqes: 20086045325
> tx19_wake: 11431
> tx19_cqe_err: 0
> tx20_packets: 21943735263
> tx20_bytes: 13528749492187
> tx20_tso_packets: 1390048103
> tx20_tso_bytes: 7629058809637
> tx20_tso_inner_packets: 0
> tx20_tso_inner_bytes: 0
> tx20_csum_partial: 1848533941
> tx20_csum_partial_inner: 0
> tx20_added_vlan_packets: 17861417651
> tx20_nop: 276840365
> tx20_csum_none: 16012883710
> tx20_stopped: 38457
> tx20_dropped: 0
> tx20_xmit_more: 57042753
> tx20_recover: 0
> tx20_cqes: 17804384839
> tx20_wake: 38457
> tx20_cqe_err: 0
> tx21_packets: 21476926958
> tx21_bytes: 13096410597896
> tx21_tso_packets: 1367724090
> tx21_tso_bytes: 7568364585127
> tx21_tso_inner_packets: 0
> tx21_tso_inner_bytes: 0
> tx21_csum_partial: 1830570727
> tx21_csum_partial_inner: 0
> tx21_added_vlan_packets: 17421087814
> tx21_nop: 270611519
> tx21_csum_none: 15590517087
> tx21_stopped: 31213
> tx21_dropped: 0
> tx21_xmit_more: 60305389
> tx21_recover: 0
> tx21_cqes: 17360791205
> tx21_wake: 31213
> tx21_cqe_err: 0
> tx22_packets: 21819106444
> tx22_bytes: 13492871887100
> tx22_tso_packets: 1387002018
> tx22_tso_bytes: 7617705727669
> tx22_tso_inner_packets: 0
> tx22_tso_inner_bytes: 0
> tx22_csum_partial: 1853632107
> tx22_csum_partial_inner: 0
> tx22_added_vlan_packets: 17743255447
> tx22_nop: 274820992
> tx22_csum_none: 15889623340
> tx22_stopped: 24814
> tx22_dropped: 0
> tx22_xmit_more: 60811304
> tx22_recover: 0
> tx22_cqes: 17682451111
> tx22_wake: 24815
> tx22_cqe_err: 0
> tx23_packets: 21830455800
> tx23_bytes: 13427551902532
> tx23_tso_packets: 1388556038
> tx23_tso_bytes: 7604040587125
> tx23_tso_inner_packets: 0
> tx23_tso_inner_bytes: 0
> tx23_csum_partial: 1850819694
> tx23_csum_partial_inner: 0
> tx23_added_vlan_packets: 17761271122
> tx23_nop: 275142775
> tx23_csum_none: 15910451428
> tx23_stopped: 29899
> tx23_dropped: 0
> tx23_xmit_more: 58924909
> tx23_recover: 0
> tx23_cqes: 17702355187
> tx23_wake: 29898
> tx23_cqe_err: 0
> tx24_packets: 21961484213
> tx24_bytes: 13531373062497
> tx24_tso_packets: 1394697504
> tx24_tso_bytes: 7663866609308
> tx24_tso_inner_packets: 0
> tx24_tso_inner_bytes: 0
> tx24_csum_partial: 1857072074
> tx24_csum_partial_inner: 0
> tx24_added_vlan_packets: 17856887568
> tx24_nop: 276352855
> tx24_csum_none: 15999815494
> tx24_stopped: 33924
> tx24_dropped: 0
> tx24_xmit_more: 63992426
> tx24_recover: 0
> tx24_cqes: 17792905243
> tx24_wake: 33923
> tx24_cqe_err: 0
> tx25_packets: 21853593838
> tx25_bytes: 13357487830519
> tx25_tso_packets: 1398822411
> tx25_tso_bytes: 7691191518838
> tx25_tso_inner_packets: 0
> tx25_tso_inner_bytes: 0
> tx25_csum_partial: 1869483109
> tx25_csum_partial_inner: 0
> tx25_added_vlan_packets: 17734634614
> tx25_nop: 276327643
> tx25_csum_none: 15865151505
> tx25_stopped: 38651
> tx25_dropped: 0
> tx25_xmit_more: 56410535
> tx25_recover: 0
> tx25_cqes: 17678234537
> tx25_wake: 38650
> tx25_cqe_err: 0
> tx26_packets: 21480261205
> tx26_bytes: 13148973015935
> tx26_tso_packets: 1348132284
> tx26_tso_bytes: 7523489481775
> tx26_tso_inner_packets: 0
> tx26_tso_inner_bytes: 0
> tx26_csum_partial: 1839740745
> tx26_csum_partial_inner: 0
> tx26_added_vlan_packets: 17430592911
> tx26_nop: 270367836
> tx26_csum_none: 15590852166
> tx26_stopped: 34044
> tx26_dropped: 0
> tx26_xmit_more: 59870114
> tx26_recover: 0
> tx26_cqes: 17370736612
> tx26_wake: 34043
> tx26_cqe_err: 0
> tx27_packets: 22694273108
> tx27_bytes: 14135473431004
> tx27_tso_packets: 1418371875
> tx27_tso_bytes: 7784842263038
> tx27_tso_inner_packets: 0
> tx27_tso_inner_bytes: 0
> tx27_csum_partial: 1919170584
> tx27_csum_partial_inner: 0
> tx27_added_vlan_packets: 18520826023
> tx27_nop: 286296272
> tx27_csum_none: 16601655439
> tx27_stopped: 38125
> tx27_dropped: 0
> tx27_xmit_more: 72749775
> tx27_recover: 0
> tx27_cqes: 18448090270
> tx27_wake: 38127
> tx27_cqe_err: 0
> tx28_packets: 0
> tx28_bytes: 0
> tx28_tso_packets: 0
> tx28_tso_bytes: 0
> tx28_tso_inner_packets: 0
> tx28_tso_inner_bytes: 0
> tx28_csum_partial: 0
> tx28_csum_partial_inner: 0
> tx28_added_vlan_packets: 0
> tx28_nop: 0
> tx28_csum_none: 0
> tx28_stopped: 0
> tx28_dropped: 0
> tx28_xmit_more: 0
> tx28_recover: 0
> tx28_cqes: 0
> tx28_wake: 0
> tx28_cqe_err: 0
> tx29_packets: 3
> tx29_bytes: 266
> tx29_tso_packets: 0
> tx29_tso_bytes: 0
> tx29_tso_inner_packets: 0
> tx29_tso_inner_bytes: 0
> tx29_csum_partial: 0
> tx29_csum_partial_inner: 0
> tx29_added_vlan_packets: 0
> tx29_nop: 0
> tx29_csum_none: 3
> tx29_stopped: 0
> tx29_dropped: 0
> tx29_xmit_more: 1
> tx29_recover: 0
> tx29_cqes: 2
> tx29_wake: 0
> tx29_cqe_err: 0
> tx30_packets: 0
> tx30_bytes: 0
> tx30_tso_packets: 0
> tx30_tso_bytes: 0
> tx30_tso_inner_packets: 0
> tx30_tso_inner_bytes: 0
> tx30_csum_partial: 0
> tx30_csum_partial_inner: 0
> tx30_added_vlan_packets: 0
> tx30_nop: 0
> tx30_csum_none: 0
> tx30_stopped: 0
> tx30_dropped: 0
> tx30_xmit_more: 0
> tx30_recover: 0
> tx30_cqes: 0
> tx30_wake: 0
> tx30_cqe_err: 0
> tx31_packets: 0
> tx31_bytes: 0
> tx31_tso_packets: 0
> tx31_tso_bytes: 0
> tx31_tso_inner_packets: 0
> tx31_tso_inner_bytes: 0
> tx31_csum_partial: 0
> tx31_csum_partial_inner: 0
> tx31_added_vlan_packets: 0
> tx31_nop: 0
> tx31_csum_none: 0
> tx31_stopped: 0
> tx31_dropped: 0
> tx31_xmit_more: 0
> tx31_recover: 0
> tx31_cqes: 0
> tx31_wake: 0
> tx31_cqe_err: 0
> tx32_packets: 0
> tx32_bytes: 0
> tx32_tso_packets: 0
> tx32_tso_bytes: 0
> tx32_tso_inner_packets: 0
> tx32_tso_inner_bytes: 0
> tx32_csum_partial: 0
> tx32_csum_partial_inner: 0
> tx32_added_vlan_packets: 0
> tx32_nop: 0
> tx32_csum_none: 0
> tx32_stopped: 0
> tx32_dropped: 0
> tx32_xmit_more: 0
> tx32_recover: 0
> tx32_cqes: 0
> tx32_wake: 0
> tx32_cqe_err: 0
> tx33_packets: 0
> tx33_bytes: 0
> tx33_tso_packets: 0
> tx33_tso_bytes: 0
> tx33_tso_inner_packets: 0
> tx33_tso_inner_bytes: 0
> tx33_csum_partial: 0
> tx33_csum_partial_inner: 0
> tx33_added_vlan_packets: 0
> tx33_nop: 0
> tx33_csum_none: 0
> tx33_stopped: 0
> tx33_dropped: 0
> tx33_xmit_more: 0
> tx33_recover: 0
> tx33_cqes: 0
> tx33_wake: 0
> tx33_cqe_err: 0
> tx34_packets: 0
> tx34_bytes: 0
> tx34_tso_packets: 0
> tx34_tso_bytes: 0
> tx34_tso_inner_packets: 0
> tx34_tso_inner_bytes: 0
> tx34_csum_partial: 0
> tx34_csum_partial_inner: 0
> tx34_added_vlan_packets: 0
> tx34_nop: 0
> tx34_csum_none: 0
> tx34_stopped: 0
> tx34_dropped: 0
> tx34_xmit_more: 0
> tx34_recover: 0
> tx34_cqes: 0
> tx34_wake: 0
> tx34_cqe_err: 0
> tx35_packets: 0
> tx35_bytes: 0
> tx35_tso_packets: 0
> tx35_tso_bytes: 0
> tx35_tso_inner_packets: 0
> tx35_tso_inner_bytes: 0
> tx35_csum_partial: 0
> tx35_csum_partial_inner: 0
> tx35_added_vlan_packets: 0
> tx35_nop: 0
> tx35_csum_none: 0
> tx35_stopped: 0
> tx35_dropped: 0
> tx35_xmit_more: 0
> tx35_recover: 0
> tx35_cqes: 0
> tx35_wake: 0
> tx35_cqe_err: 0
> tx36_packets: 0
> tx36_bytes: 0
> tx36_tso_packets: 0
> tx36_tso_bytes: 0
> tx36_tso_inner_packets: 0
> tx36_tso_inner_bytes: 0
> tx36_csum_partial: 0
> tx36_csum_partial_inner: 0
> tx36_added_vlan_packets: 0
> tx36_nop: 0
> tx36_csum_none: 0
> tx36_stopped: 0
> tx36_dropped: 0
> tx36_xmit_more: 0
> tx36_recover: 0
> tx36_cqes: 0
> tx36_wake: 0
> tx36_cqe_err: 0
> tx37_packets: 0
> tx37_bytes: 0
> tx37_tso_packets: 0
> tx37_tso_bytes: 0
> tx37_tso_inner_packets: 0
> tx37_tso_inner_bytes: 0
> tx37_csum_partial: 0
> tx37_csum_partial_inner: 0
> tx37_added_vlan_packets: 0
> tx37_nop: 0
> tx37_csum_none: 0
> tx37_stopped: 0
> tx37_dropped: 0
> tx37_xmit_more: 0
> tx37_recover: 0
> tx37_cqes: 0
> tx37_wake: 0
> tx37_cqe_err: 0
> tx38_packets: 0
> tx38_bytes: 0
> tx38_tso_packets: 0
> tx38_tso_bytes: 0
> tx38_tso_inner_packets: 0
> tx38_tso_inner_bytes: 0
> tx38_csum_partial: 0
> tx38_csum_partial_inner: 0
> tx38_added_vlan_packets: 0
> tx38_nop: 0
> tx38_csum_none: 0
> tx38_stopped: 0
> tx38_dropped: 0
> tx38_xmit_more: 0
> tx38_recover: 0
> tx38_cqes: 0
> tx38_wake: 0
> tx38_cqe_err: 0
> tx39_packets: 0
> tx39_bytes: 0
> tx39_tso_packets: 0
> tx39_tso_bytes: 0
> tx39_tso_inner_packets: 0
> tx39_tso_inner_bytes: 0
> tx39_csum_partial: 0
> tx39_csum_partial_inner: 0
> tx39_added_vlan_packets: 0
> tx39_nop: 0
> tx39_csum_none: 0
> tx39_stopped: 0
> tx39_dropped: 0
> tx39_xmit_more: 0
> tx39_recover: 0
> tx39_cqes: 0
> tx39_wake: 0
> tx39_cqe_err: 0
> tx40_packets: 0
> tx40_bytes: 0
> tx40_tso_packets: 0
> tx40_tso_bytes: 0
> tx40_tso_inner_packets: 0
> tx40_tso_inner_bytes: 0
> tx40_csum_partial: 0
> tx40_csum_partial_inner: 0
> tx40_added_vlan_packets: 0
> tx40_nop: 0
> tx40_csum_none: 0
> tx40_stopped: 0
> tx40_dropped: 0
> tx40_xmit_more: 0
> tx40_recover: 0
> tx40_cqes: 0
> tx40_wake: 0
> tx40_cqe_err: 0
> tx41_packets: 0
> tx41_bytes: 0
> tx41_tso_packets: 0
> tx41_tso_bytes: 0
> tx41_tso_inner_packets: 0
> tx41_tso_inner_bytes: 0
> tx41_csum_partial: 0
> tx41_csum_partial_inner: 0
> tx41_added_vlan_packets: 0
> tx41_nop: 0
> tx41_csum_none: 0
> tx41_stopped: 0
> tx41_dropped: 0
> tx41_xmit_more: 0
> tx41_recover: 0
> tx41_cqes: 0
> tx41_wake: 0
> tx41_cqe_err: 0
> tx42_packets: 0
> tx42_bytes: 0
> tx42_tso_packets: 0
> tx42_tso_bytes: 0
> tx42_tso_inner_packets: 0
> tx42_tso_inner_bytes: 0
> tx42_csum_partial: 0
> tx42_csum_partial_inner: 0
> tx42_added_vlan_packets: 0
> tx42_nop: 0
> tx42_csum_none: 0
> tx42_stopped: 0
> tx42_dropped: 0
> tx42_xmit_more: 0
> tx42_recover: 0
> tx42_cqes: 0
> tx42_wake: 0
> tx42_cqe_err: 0
> tx43_packets: 0
> tx43_bytes: 0
> tx43_tso_packets: 0
> tx43_tso_bytes: 0
> tx43_tso_inner_packets: 0
> tx43_tso_inner_bytes: 0
> tx43_csum_partial: 0
> tx43_csum_partial_inner: 0
> tx43_added_vlan_packets: 0
> tx43_nop: 0
> tx43_csum_none: 0
> tx43_stopped: 0
> tx43_dropped: 0
> tx43_xmit_more: 0
> tx43_recover: 0
> tx43_cqes: 0
> tx43_wake: 0
> tx43_cqe_err: 0
> tx44_packets: 0
> tx44_bytes: 0
> tx44_tso_packets: 0
> tx44_tso_bytes: 0
> tx44_tso_inner_packets: 0
> tx44_tso_inner_bytes: 0
> tx44_csum_partial: 0
> tx44_csum_partial_inner: 0
> tx44_added_vlan_packets: 0
> tx44_nop: 0
> tx44_csum_none: 0
> tx44_stopped: 0
> tx44_dropped: 0
> tx44_xmit_more: 0
> tx44_recover: 0
> tx44_cqes: 0
> tx44_wake: 0
> tx44_cqe_err: 0
> tx45_packets: 0
> tx45_bytes: 0
> tx45_tso_packets: 0
> tx45_tso_bytes: 0
> tx45_tso_inner_packets: 0
> tx45_tso_inner_bytes: 0
> tx45_csum_partial: 0
> tx45_csum_partial_inner: 0
> tx45_added_vlan_packets: 0
> tx45_nop: 0
> tx45_csum_none: 0
> tx45_stopped: 0
> tx45_dropped: 0
> tx45_xmit_more: 0
> tx45_recover: 0
> tx45_cqes: 0
> tx45_wake: 0
> tx45_cqe_err: 0
> tx46_packets: 0
> tx46_bytes: 0
> tx46_tso_packets: 0
> tx46_tso_bytes: 0
> tx46_tso_inner_packets: 0
> tx46_tso_inner_bytes: 0
> tx46_csum_partial: 0
> tx46_csum_partial_inner: 0
> tx46_added_vlan_packets: 0
> tx46_nop: 0
> tx46_csum_none: 0
> tx46_stopped: 0
> tx46_dropped: 0
> tx46_xmit_more: 0
> tx46_recover: 0
> tx46_cqes: 0
> tx46_wake: 0
> tx46_cqe_err: 0
> tx47_packets: 0
> tx47_bytes: 0
> tx47_tso_packets: 0
> tx47_tso_bytes: 0
> tx47_tso_inner_packets: 0
> tx47_tso_inner_bytes: 0
> tx47_csum_partial: 0
> tx47_csum_partial_inner: 0
> tx47_added_vlan_packets: 0
> tx47_nop: 0
> tx47_csum_none: 0
> tx47_stopped: 0
> tx47_dropped: 0
> tx47_xmit_more: 0
> tx47_recover: 0
> tx47_cqes: 0
> tx47_wake: 0
> tx47_cqe_err: 0
> tx48_packets: 0
> tx48_bytes: 0
> tx48_tso_packets: 0
> tx48_tso_bytes: 0
> tx48_tso_inner_packets: 0
> tx48_tso_inner_bytes: 0
> tx48_csum_partial: 0
> tx48_csum_partial_inner: 0
> tx48_added_vlan_packets: 0
> tx48_nop: 0
> tx48_csum_none: 0
> tx48_stopped: 0
> tx48_dropped: 0
> tx48_xmit_more: 0
> tx48_recover: 0
> tx48_cqes: 0
> tx48_wake: 0
> tx48_cqe_err: 0
> tx49_packets: 0
> tx49_bytes: 0
> tx49_tso_packets: 0
> tx49_tso_bytes: 0
> tx49_tso_inner_packets: 0
> tx49_tso_inner_bytes: 0
> tx49_csum_partial: 0
> tx49_csum_partial_inner: 0
> tx49_added_vlan_packets: 0
> tx49_nop: 0
> tx49_csum_none: 0
> tx49_stopped: 0
> tx49_dropped: 0
> tx49_xmit_more: 0
> tx49_recover: 0
> tx49_cqes: 0
> tx49_wake: 0
> tx49_cqe_err: 0
> tx50_packets: 0
> tx50_bytes: 0
> tx50_tso_packets: 0
> tx50_tso_bytes: 0
> tx50_tso_inner_packets: 0
> tx50_tso_inner_bytes: 0
> tx50_csum_partial: 0
> tx50_csum_partial_inner: 0
> tx50_added_vlan_packets: 0
> tx50_nop: 0
> tx50_csum_none: 0
> tx50_stopped: 0
> tx50_dropped: 0
> tx50_xmit_more: 0
> tx50_recover: 0
> tx50_cqes: 0
> tx50_wake: 0
> tx50_cqe_err: 0
> tx51_packets: 0
> tx51_bytes: 0
> tx51_tso_packets: 0
> tx51_tso_bytes: 0
> tx51_tso_inner_packets: 0
> tx51_tso_inner_bytes: 0
> tx51_csum_partial: 0
> tx51_csum_partial_inner: 0
> tx51_added_vlan_packets: 0
> tx51_nop: 0
> tx51_csum_none: 0
> tx51_stopped: 0
> tx51_dropped: 0
> tx51_xmit_more: 0
> tx51_recover: 0
> tx51_cqes: 0
> tx51_wake: 0
> tx51_cqe_err: 0
> tx52_packets: 0
> tx52_bytes: 0
> tx52_tso_packets: 0
> tx52_tso_bytes: 0
> tx52_tso_inner_packets: 0
> tx52_tso_inner_bytes: 0
> tx52_csum_partial: 0
> tx52_csum_partial_inner: 0
> tx52_added_vlan_packets: 0
> tx52_nop: 0
> tx52_csum_none: 0
> tx52_stopped: 0
> tx52_dropped: 0
> tx52_xmit_more: 0
> tx52_recover: 0
> tx52_cqes: 0
> tx52_wake: 0
> tx52_cqe_err: 0
> tx53_packets: 0
> tx53_bytes: 0
> tx53_tso_packets: 0
> tx53_tso_bytes: 0
> tx53_tso_inner_packets: 0
> tx53_tso_inner_bytes: 0
> tx53_csum_partial: 0
> tx53_csum_partial_inner: 0
> tx53_added_vlan_packets: 0
> tx53_nop: 0
> tx53_csum_none: 0
> tx53_stopped: 0
> tx53_dropped: 0
> tx53_xmit_more: 0
> tx53_recover: 0
> tx53_cqes: 0
> tx53_wake: 0
> tx53_cqe_err: 0
> tx54_packets: 0
> tx54_bytes: 0
> tx54_tso_packets: 0
> tx54_tso_bytes: 0
> tx54_tso_inner_packets: 0
> tx54_tso_inner_bytes: 0
> tx54_csum_partial: 0
> tx54_csum_partial_inner: 0
> tx54_added_vlan_packets: 0
> tx54_nop: 0
> tx54_csum_none: 0
> tx54_stopped: 0
> tx54_dropped: 0
> tx54_xmit_more: 0
> tx54_recover: 0
> tx54_cqes: 0
> tx54_wake: 0
> tx54_cqe_err: 0
> tx55_packets: 0
> tx55_bytes: 0
> tx55_tso_packets: 0
> tx55_tso_bytes: 0
> tx55_tso_inner_packets: 0
> tx55_tso_inner_bytes: 0
> tx55_csum_partial: 0
> tx55_csum_partial_inner: 0
> tx55_added_vlan_packets: 0
> tx55_nop: 0
> tx55_csum_none: 0
> tx55_stopped: 0
> tx55_dropped: 0
> tx55_xmit_more: 0
> tx55_recover: 0
> tx55_cqes: 0
> tx55_wake: 0
> tx55_cqe_err: 0
> tx0_xdp_xmit: 0
> tx0_xdp_full: 0
> tx0_xdp_err: 0
> tx0_xdp_cqes: 0
> tx1_xdp_xmit: 0
> tx1_xdp_full: 0
> tx1_xdp_err: 0
> tx1_xdp_cqes: 0
> tx2_xdp_xmit: 0
> tx2_xdp_full: 0
> tx2_xdp_err: 0
> tx2_xdp_cqes: 0
> tx3_xdp_xmit: 0
> tx3_xdp_full: 0
> tx3_xdp_err: 0
> tx3_xdp_cqes: 0
> tx4_xdp_xmit: 0
> tx4_xdp_full: 0
> tx4_xdp_err: 0
> tx4_xdp_cqes: 0
> tx5_xdp_xmit: 0
> tx5_xdp_full: 0
> tx5_xdp_err: 0
> tx5_xdp_cqes: 0
> tx6_xdp_xmit: 0
> tx6_xdp_full: 0
> tx6_xdp_err: 0
> tx6_xdp_cqes: 0
> tx7_xdp_xmit: 0
> tx7_xdp_full: 0
> tx7_xdp_err: 0
> tx7_xdp_cqes: 0
> tx8_xdp_xmit: 0
> tx8_xdp_full: 0
> tx8_xdp_err: 0
> tx8_xdp_cqes: 0
> tx9_xdp_xmit: 0
> tx9_xdp_full: 0
> tx9_xdp_err: 0
> tx9_xdp_cqes: 0
> tx10_xdp_xmit: 0
> tx10_xdp_full: 0
> tx10_xdp_err: 0
> tx10_xdp_cqes: 0
> tx11_xdp_xmit: 0
> tx11_xdp_full: 0
> tx11_xdp_err: 0
> tx11_xdp_cqes: 0
> tx12_xdp_xmit: 0
> tx12_xdp_full: 0
> tx12_xdp_err: 0
> tx12_xdp_cqes: 0
> tx13_xdp_xmit: 0
> tx13_xdp_full: 0
> tx13_xdp_err: 0
> tx13_xdp_cqes: 0
> tx14_xdp_xmit: 0
> tx14_xdp_full: 0
> tx14_xdp_err: 0
> tx14_xdp_cqes: 0
> tx15_xdp_xmit: 0
> tx15_xdp_full: 0
> tx15_xdp_err: 0
> tx15_xdp_cqes: 0
> tx16_xdp_xmit: 0
> tx16_xdp_full: 0
> tx16_xdp_err: 0
> tx16_xdp_cqes: 0
> tx17_xdp_xmit: 0
> tx17_xdp_full: 0
> tx17_xdp_err: 0
> tx17_xdp_cqes: 0
> tx18_xdp_xmit: 0
> tx18_xdp_full: 0
> tx18_xdp_err: 0
> tx18_xdp_cqes: 0
> tx19_xdp_xmit: 0
> tx19_xdp_full: 0
> tx19_xdp_err: 0
> tx19_xdp_cqes: 0
> tx20_xdp_xmit: 0
> tx20_xdp_full: 0
> tx20_xdp_err: 0
> tx20_xdp_cqes: 0
> tx21_xdp_xmit: 0
> tx21_xdp_full: 0
> tx21_xdp_err: 0
> tx21_xdp_cqes: 0
> tx22_xdp_xmit: 0
> tx22_xdp_full: 0
> tx22_xdp_err: 0
> tx22_xdp_cqes: 0
> tx23_xdp_xmit: 0
> tx23_xdp_full: 0
> tx23_xdp_err: 0
> tx23_xdp_cqes: 0
> tx24_xdp_xmit: 0
> tx24_xdp_full: 0
> tx24_xdp_err: 0
> tx24_xdp_cqes: 0
> tx25_xdp_xmit: 0
> tx25_xdp_full: 0
> tx25_xdp_err: 0
> tx25_xdp_cqes: 0
> tx26_xdp_xmit: 0
> tx26_xdp_full: 0
> tx26_xdp_err: 0
> tx26_xdp_cqes: 0
> tx27_xdp_xmit: 0
> tx27_xdp_full: 0
> tx27_xdp_err: 0
> tx27_xdp_cqes: 0
> tx28_xdp_xmit: 0
> tx28_xdp_full: 0
> tx28_xdp_err: 0
> tx28_xdp_cqes: 0
> tx29_xdp_xmit: 0
> tx29_xdp_full: 0
> tx29_xdp_err: 0
> tx29_xdp_cqes: 0
> tx30_xdp_xmit: 0
> tx30_xdp_full: 0
> tx30_xdp_err: 0
> tx30_xdp_cqes: 0
> tx31_xdp_xmit: 0
> tx31_xdp_full: 0
> tx31_xdp_err: 0
> tx31_xdp_cqes: 0
> tx32_xdp_xmit: 0
> tx32_xdp_full: 0
> tx32_xdp_err: 0
> tx32_xdp_cqes: 0
> tx33_xdp_xmit: 0
> tx33_xdp_full: 0
> tx33_xdp_err: 0
> tx33_xdp_cqes: 0
> tx34_xdp_xmit: 0
> tx34_xdp_full: 0
> tx34_xdp_err: 0
> tx34_xdp_cqes: 0
> tx35_xdp_xmit: 0
> tx35_xdp_full: 0
> tx35_xdp_err: 0
> tx35_xdp_cqes: 0
> tx36_xdp_xmit: 0
> tx36_xdp_full: 0
> tx36_xdp_err: 0
> tx36_xdp_cqes: 0
> tx37_xdp_xmit: 0
> tx37_xdp_full: 0
> tx37_xdp_err: 0
> tx37_xdp_cqes: 0
> tx38_xdp_xmit: 0
> tx38_xdp_full: 0
> tx38_xdp_err: 0
> tx38_xdp_cqes: 0
> tx39_xdp_xmit: 0
> tx39_xdp_full: 0
> tx39_xdp_err: 0
> tx39_xdp_cqes: 0
> tx40_xdp_xmit: 0
> tx40_xdp_full: 0
> tx40_xdp_err: 0
> tx40_xdp_cqes: 0
> tx41_xdp_xmit: 0
> tx41_xdp_full: 0
> tx41_xdp_err: 0
> tx41_xdp_cqes: 0
> tx42_xdp_xmit: 0
> tx42_xdp_full: 0
> tx42_xdp_err: 0
> tx42_xdp_cqes: 0
> tx43_xdp_xmit: 0
> tx43_xdp_full: 0
> tx43_xdp_err: 0
> tx43_xdp_cqes: 0
> tx44_xdp_xmit: 0
> tx44_xdp_full: 0
> tx44_xdp_err: 0
> tx44_xdp_cqes: 0
> tx45_xdp_xmit: 0
> tx45_xdp_full: 0
> tx45_xdp_err: 0
> tx45_xdp_cqes: 0
> tx46_xdp_xmit: 0
> tx46_xdp_full: 0
> tx46_xdp_err: 0
> tx46_xdp_cqes: 0
> tx47_xdp_xmit: 0
> tx47_xdp_full: 0
> tx47_xdp_err: 0
> tx47_xdp_cqes: 0
> tx48_xdp_xmit: 0
> tx48_xdp_full: 0
> tx48_xdp_err: 0
> tx48_xdp_cqes: 0
> tx49_xdp_xmit: 0
> tx49_xdp_full: 0
> tx49_xdp_err: 0
> tx49_xdp_cqes: 0
> tx50_xdp_xmit: 0
> tx50_xdp_full: 0
> tx50_xdp_err: 0
> tx50_xdp_cqes: 0
> tx51_xdp_xmit: 0
> tx51_xdp_full: 0
> tx51_xdp_err: 0
> tx51_xdp_cqes: 0
> tx52_xdp_xmit: 0
> tx52_xdp_full: 0
> tx52_xdp_err: 0
> tx52_xdp_cqes: 0
> tx53_xdp_xmit: 0
> tx53_xdp_full: 0
> tx53_xdp_err: 0
> tx53_xdp_cqes: 0
> tx54_xdp_xmit: 0
> tx54_xdp_full: 0
> tx54_xdp_err: 0
> tx54_xdp_cqes: 0
> tx55_xdp_xmit: 0
> tx55_xdp_full: 0
> tx55_xdp_err: 0
> tx55_xdp_cqes: 0
>
>
>> [...]
>>
>>>>> ethtool -S enp175s0f0
>>>>> NIC statistics:
>>>>> rx_packets: 141574897253
>>>>> rx_bytes: 184445040406258
>>>>> tx_packets: 172569543894
>>>>> tx_bytes: 99486882076365
>>>>> tx_tso_packets: 9367664195
>>>>> tx_tso_bytes: 56435233992948
>>>>> tx_tso_inner_packets: 0
>>>>> tx_tso_inner_bytes: 0
>>>>> tx_added_vlan_packets: 141297671626
>>>>> tx_nop: 2102916272
>>>>> rx_lro_packets: 0
>>>>> rx_lro_bytes: 0
>>>>> rx_ecn_mark: 0
>>>>> rx_removed_vlan_packets: 141574897252
>>>>> rx_csum_unnecessary: 0
>>>>> rx_csum_none: 23135854
>>>>> rx_csum_complete: 141551761398
>>>>> rx_csum_unnecessary_inner: 0
>>>>> rx_xdp_drop: 0
>>>>> rx_xdp_redirect: 0
>>>>> rx_xdp_tx_xmit: 0
>>>>> rx_xdp_tx_full: 0
>>>>> rx_xdp_tx_err: 0
>>>>> rx_xdp_tx_cqe: 0
>>>>> tx_csum_none: 127934791664
>>>> It is a good idea to look into this, tx is not requesting hw tx
>>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>>> on
>>>> calculating csum, or maybe this is just the rx csum complete..
>>>>
>>>>> tx_csum_partial: 13362879974
>>>>> tx_csum_partial_inner: 0
>>>>> tx_queue_stopped: 232561
>>>> TX queues are stalling, could be an indentation for the pcie
>>>> bottelneck.
>>>>
>>>>> tx_queue_dropped: 0
>>>>> tx_xmit_more: 1266021946
>>>>> tx_recover: 0
>>>>> tx_cqes: 140031716469
>>>>> tx_queue_wake: 232561
>>>>> tx_udp_seg_rem: 0
>>>>> tx_cqe_err: 0
>>>>> tx_xdp_xmit: 0
>>>>> tx_xdp_full: 0
>>>>> tx_xdp_err: 0
>>>>> tx_xdp_cqes: 0
>>>>> rx_wqe_err: 0
>>>>> rx_mpwqe_filler_cqes: 0
>>>>> rx_mpwqe_filler_strides: 0
>>>>> rx_buff_alloc_err: 0
>>>>> rx_cqe_compress_blks: 0
>>>>> rx_cqe_compress_pkts: 0
>>>>> rx_page_reuse: 0
>>>>> rx_cache_reuse: 16625975793
>>>>> rx_cache_full: 54161465914
>>>>> rx_cache_empty: 258048
>>>>> rx_cache_busy: 54161472735
>>>>> rx_cache_waive: 0
>>>>> rx_congst_umr: 0
>>>>> rx_arfs_err: 0
>>>>> ch_events: 40572621887
>>>>> ch_poll: 40885650979
>>>>> ch_arm: 40429276692
>>>>> ch_aff_change: 0
>>>>> ch_eq_rearm: 0
>>>>> rx_out_of_buffer: 2791690
>>>>> rx_if_down_packets: 74
>>>>> rx_vport_unicast_packets: 141843476308
>>>>> rx_vport_unicast_bytes: 185421265403318
>>>>> tx_vport_unicast_packets: 172569484005
>>>>> tx_vport_unicast_bytes: 100019940094298
>>>>> rx_vport_multicast_packets: 85122935
>>>>> rx_vport_multicast_bytes: 5761316431
>>>>> tx_vport_multicast_packets: 6452
>>>>> tx_vport_multicast_bytes: 643540
>>>>> rx_vport_broadcast_packets: 22423624
>>>>> rx_vport_broadcast_bytes: 1390127090
>>>>> tx_vport_broadcast_packets: 22024
>>>>> tx_vport_broadcast_bytes: 1321440
>>>>> rx_vport_rdma_unicast_packets: 0
>>>>> rx_vport_rdma_unicast_bytes: 0
>>>>> tx_vport_rdma_unicast_packets: 0
>>>>> tx_vport_rdma_unicast_bytes: 0
>>>>> rx_vport_rdma_multicast_packets: 0
>>>>> rx_vport_rdma_multicast_bytes: 0
>>>>> tx_vport_rdma_multicast_packets: 0
>>>>> tx_vport_rdma_multicast_bytes: 0
>>>>> tx_packets_phy: 172569501577
>>>>> rx_packets_phy: 142871314588
>>>>> rx_crc_errors_phy: 0
>>>>> tx_bytes_phy: 100710212814151
>>>>> rx_bytes_phy: 187209224289564
>>>>> tx_multicast_phy: 6452
>>>>> tx_broadcast_phy: 22024
>>>>> rx_multicast_phy: 85122933
>>>>> rx_broadcast_phy: 22423623
>>>>> rx_in_range_len_errors_phy: 2
>>>>> rx_out_of_range_len_phy: 0
>>>>> rx_oversize_pkts_phy: 0
>>>>> rx_symbol_err_phy: 0
>>>>> tx_mac_control_phy: 0
>>>>> rx_mac_control_phy: 0
>>>>> rx_unsupported_op_phy: 0
>>>>> rx_pause_ctrl_phy: 0
>>>>> tx_pause_ctrl_phy: 0
>>>>> rx_discards_phy: 920161423
>>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>>> to
>>>> the pcie bottleneck.
>>> Yes this side is receiving more traffic - second port is +10G more tx
>>>
>> [...]
>>
>>
>>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>>> 0.00 0.00 0.00 31.30
>>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>>> 0.00 0.00 0.00 24.90
>>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>>> 0.00 0.00 0.00 19.68
>>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>>> 0.00 0.00 0.00 18.00
>>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>>> 0.00 0.00 0.00 17.40
>>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>>> 0.00 0.00 0.00 26.03
>>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>>> 0.00 0.00 0.00 17.52
>>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>>> 0.00 0.00 0.00 18.20
>>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>>> 0.00 0.00 0.00 17.68
>>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>>> 0.00 0.00 0.00 18.20
>>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>>> 0.00 0.00 0.00 20.42
>>>> The numa cores are not at 100% util, you have around 20% of idle on
>>>> each one.
>>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>>> push
>>> aditional 1-2Gbit/s
>>>
>> yes but, it doens't look like the bottleneck is the cpu, although it is
>> close to be :)..
>>
>>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 96.10
>>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.70
>>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.50
>>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 97.40
>>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.10
>>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.40
>>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>>> 0.00 0.00 0.00 19.42
>>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>>> 0.00 0.00 0.00 26.60
>>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>>> 0.00 0.00 0.00 21.60
>>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>>> 0.00 0.00 0.00 20.50
>>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>>> 0.00 0.00 0.00 21.60
>>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>>> 0.00 0.00 0.00 24.58
>>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>>> 0.00 0.00 0.00 24.68
>>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>>> 0.00 0.00 0.00 28.34
>>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>>> 0.00 0.00 0.00 25.83
>>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>>> 0.00 0.00 0.00 27.20
>>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>>> 0.00 0.00 0.00 28.03
>>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>>> 0.00 0.00 0.00 24.70
>>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>>> 0.00 0.00 0.00 25.30
>>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>>> 0.00 0.00 0.00 26.77
>>>>>
>>>>>
>>>>> ethtool -k enp175s0f0
>>>>> Features for enp175s0f0:
>>>>> rx-checksumming: on
>>>>> tx-checksumming: on
>>>>> tx-checksum-ipv4: on
>>>>> tx-checksum-ip-generic: off [fixed]
>>>>> tx-checksum-ipv6: on
>>>>> tx-checksum-fcoe-crc: off [fixed]
>>>>> tx-checksum-sctp: off [fixed]
>>>>> scatter-gather: on
>>>>> tx-scatter-gather: on
>>>>> tx-scatter-gather-fraglist: off [fixed]
>>>>> tcp-segmentation-offload: on
>>>>> tx-tcp-segmentation: on
>>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>>> tx-tcp-mangleid-segmentation: off
>>>>> tx-tcp6-segmentation: on
>>>>> udp-fragmentation-offload: off
>>>>> generic-segmentation-offload: on
>>>>> generic-receive-offload: on
>>>>> large-receive-offload: off [fixed]
>>>>> rx-vlan-offload: on
>>>>> tx-vlan-offload: on
>>>>> ntuple-filters: off
>>>>> receive-hashing: on
>>>>> highdma: on [fixed]
>>>>> rx-vlan-filter: on
>>>>> vlan-challenged: off [fixed]
>>>>> tx-lockless: off [fixed]
>>>>> netns-local: off [fixed]
>>>>> tx-gso-robust: off [fixed]
>>>>> tx-fcoe-segmentation: off [fixed]
>>>>> tx-gre-segmentation: on
>>>>> tx-gre-csum-segmentation: on
>>>>> tx-ipxip4-segmentation: off [fixed]
>>>>> tx-ipxip6-segmentation: off [fixed]
>>>>> tx-udp_tnl-segmentation: on
>>>>> tx-udp_tnl-csum-segmentation: on
>>>>> tx-gso-partial: on
>>>>> tx-sctp-segmentation: off [fixed]
>>>>> tx-esp-segmentation: off [fixed]
>>>>> tx-udp-segmentation: on
>>>>> fcoe-mtu: off [fixed]
>>>>> tx-nocache-copy: off
>>>>> loopback: off [fixed]
>>>>> rx-fcs: off
>>>>> rx-all: off
>>>>> tx-vlan-stag-hw-insert: on
>>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>>> rx-vlan-stag-filter: on [fixed]
>>>>> l2-fwd-offload: off [fixed]
>>>>> hw-tc-offload: off
>>>>> esp-hw-offload: off [fixed]
>>>>> esp-tx-csum-hw-offload: off [fixed]
>>>>> rx-udp_tunnel-port-offload: on
>>>>> tls-hw-tx-offload: off [fixed]
>>>>> tls-hw-rx-offload: off [fixed]
>>>>> rx-gro-hw: off [fixed]
>>>>> tls-hw-record: off [fixed]
>>>>>
>>>>> ethtool -c enp175s0f0
>>>>> Coalesce parameters for enp175s0f0:
>>>>> Adaptive RX: off TX: on
>>>>> stats-block-usecs: 0
>>>>> sample-interval: 0
>>>>> pkt-rate-low: 0
>>>>> pkt-rate-high: 0
>>>>> dmac: 32703
>>>>>
>>>>> rx-usecs: 256
>>>>> rx-frames: 128
>>>>> rx-usecs-irq: 0
>>>>> rx-frames-irq: 0
>>>>>
>>>>> tx-usecs: 8
>>>>> tx-frames: 128
>>>>> tx-usecs-irq: 0
>>>>> tx-frames-irq: 0
>>>>>
>>>>> rx-usecs-low: 0
>>>>> rx-frame-low: 0
>>>>> tx-usecs-low: 0
>>>>> tx-frame-low: 0
>>>>>
>>>>> rx-usecs-high: 0
>>>>> rx-frame-high: 0
>>>>> tx-usecs-high: 0
>>>>> tx-frame-high: 0
>>>>>
>>>>> ethtool -g enp175s0f0
>>>>> Ring parameters for enp175s0f0:
>>>>> Pre-set maximums:
>>>>> RX: 8192
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 8192
>>>>> Current hardware settings:
>>>>> RX: 4096
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 4096
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>> Also changed a little coalesce params - and best for this config are:
>>> ethtool -c enp175s0f0
>>> Coalesce parameters for enp175s0f0:
>>> Adaptive RX: off TX: off
>>> stats-block-usecs: 0
>>> sample-interval: 0
>>> pkt-rate-low: 0
>>> pkt-rate-high: 0
>>> dmac: 32573
>>>
>>> rx-usecs: 40
>>> rx-frames: 128
>>> rx-usecs-irq: 0
>>> rx-frames-irq: 0
>>>
>>> tx-usecs: 8
>>> tx-frames: 8
>>> tx-usecs-irq: 0
>>> tx-frames-irq: 0
>>>
>>> rx-usecs-low: 0
>>> rx-frame-low: 0
>>> tx-usecs-low: 0
>>> tx-frame-low: 0
>>>
>>> rx-usecs-high: 0
>>> rx-frame-high: 0
>>> tx-usecs-high: 0
>>> tx-frame-high: 0
>>>
>>>
>>> Less drops on RX side - and more pps in overall forwarded.
>>>
>> how much improvement ? maybe we can improve our adaptive rx coal to be
>> efficient for this work load.
>>
>>
> I can prepare more stats with ethtool maybee to compare - but normally
> tested with simple icmp forwarded from interface to interface
> - before change coalescence params:
> adaptive-rx off rx-usecs 384 rx-frames 128
> 3% loss for icmp
> - after change to:
> adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8
> tx-frames 8
> 2% loss for icmp
>
> But yes - to know better will need to compare rx/tx counters from
> ethtool + /proc/net/dev
>
>
> Was trying to turn on adaptative-tx+rx - but 100% saturation at
> 43Gbit/s RX / 43Gbit/s TX
>
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 21:24 ` Paweł Staszewski
@ 2018-11-01 21:34 ` Paweł Staszewski
0 siblings, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-01 21:34 UTC (permalink / raw)
To: Saeed Mahameed, netdev
W dniu 01.11.2018 o 22:24, Paweł Staszewski pisze:
>
>
> W dniu 01.11.2018 o 22:18, Paweł Staszewski pisze:
>>
>>
>> W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
>>> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>>>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>>>> Hi
>>>>>>
>>>>>> So maybee someone will be interested how linux kernel handles
>>>>>> normal
>>>>>> traffic (not pktgen :) )
>>>>>>
>>>>>>
>>>>>> Server HW configuration:
>>>>>>
>>>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>>>
>>>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>>>
>>>>>>
>>>>>> Server software:
>>>>>>
>>>>>> FRR - as routing daemon
>>>>>>
>>>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>>>> local
>>>>>> numa
>>>>>> node)
>>>>>>
>>>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>>>> numa
>>>>>> node)
>>>>>>
>>>>>>
>>>>>> Maximum traffic that server can handle:
>>>>>>
>>>>>> Bandwidth
>>>>>>
>>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>> input: /proc/net/dev type: rate
>>>>>> \ iface Rx Tx Total
>>>>>> =================================================================
>>>>>> ====
>>>>>> =========
>>>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>>>> Gb/s
>>>>>> 65.74 Gb/s
>>>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>>>> Gb/s
>>>>>> 66.51 Gb/s
>>>>>> ---------------------------------------------------------------
>>>>>> ----
>>>>>> -----------
>>>>>> total: 66.58 Gb/s 65.67
>>>>>> Gb/s
>>>>>> 132.25 Gb/s
>>>>>>
>>>>>>
>>>>>> Packets per second:
>>>>>>
>>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>> input: /proc/net/dev type: rate
>>>>>> - iface Rx Tx Total
>>>>>> =================================================================
>>>>>> ====
>>>>>> =========
>>>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>>>> 8735207.00 P/s
>>>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>>>> 8790460.00 P/s
>>>>>> ---------------------------------------------------------------
>>>>>> ----
>>>>>> -----------
>>>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>>>> 17525668.00 P/s
>>>>>>
>>>>>>
>>>>>> After reaching that limits nics on the upstream side (more RX
>>>>>> traffic)
>>>>>> start to drop packets
>>>>>>
>>>>>>
>>>>>> I just dont understand that server can't handle more bandwidth
>>>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>>>> RX
>>>>>> side are increasing.
>>>>>>
>>>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>>>> NIC (
>>>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>>>> 132.25
>>>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>>>> hmm yes that was my concern also - cause cant find anywhere
>>>> informations
>>>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>>>> 8GT
>>>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>>>> bw
>>>> on both ports
>>> i think it is bidir
>>>
>>>> This can explain maybee also why cpuload is rising rapidly from
>>>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>>>> so
>>>> there can be some error in reading them when offloading (gro/gso/tso)
>>>> on
>>>> nic's is enabled that is why
>>>>
>>>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>>>> 8GT
>>>>>> is
>>>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>>>> and
>>>>>> pps
>>>>>> (like 4x more comparing to normal internet traffic)
>>>>>>
>>>>> Are you forwarding when using pktgen as well or you just testing
>>>>> the RX
>>>>> side pps ?
>>>> Yes pktgen was tested on single port RX
>>>> Can check also forwarding to eliminate pciex limits
>>>>
>>> So this explains why you have more RX pps, since tx is idle and pcie
>>> will be free to do only rx.
>>>
>>> [...]
>>>
>>>
>>>>>> ethtool -S enp175s0f1
>>>>>> NIC statistics:
>>>>>> rx_packets: 173730800927
>>>>>> rx_bytes: 99827422751332
>>>>>> tx_packets: 142532009512
>>>>>> tx_bytes: 184633045911222
>>>>>> tx_tso_packets: 25989113891
>>>>>> tx_tso_bytes: 132933363384458
>>>>>> tx_tso_inner_packets: 0
>>>>>> tx_tso_inner_bytes: 0
>>>>>> tx_added_vlan_packets: 74630239613
>>>>>> tx_nop: 2029817748
>>>>>> rx_lro_packets: 0
>>>>>> rx_lro_bytes: 0
>>>>>> rx_ecn_mark: 0
>>>>>> rx_removed_vlan_packets: 173730800927
>>>>>> rx_csum_unnecessary: 0
>>>>>> rx_csum_none: 434357
>>>>>> rx_csum_complete: 173730366570
>>>>>> rx_csum_unnecessary_inner: 0
>>>>>> rx_xdp_drop: 0
>>>>>> rx_xdp_redirect: 0
>>>>>> rx_xdp_tx_xmit: 0
>>>>>> rx_xdp_tx_full: 0
>>>>>> rx_xdp_tx_err: 0
>>>>>> rx_xdp_tx_cqe: 0
>>>>>> tx_csum_none: 38260960853
>>>>>> tx_csum_partial: 36369278774
>>>>>> tx_csum_partial_inner: 0
>>>>>> tx_queue_stopped: 1
>>>>>> tx_queue_dropped: 0
>>>>>> tx_xmit_more: 748638099
>>>>>> tx_recover: 0
>>>>>> tx_cqes: 73881645031
>>>>>> tx_queue_wake: 1
>>>>>> tx_udp_seg_rem: 0
>>>>>> tx_cqe_err: 0
>>>>>> tx_xdp_xmit: 0
>>>>>> tx_xdp_full: 0
>>>>>> tx_xdp_err: 0
>>>>>> tx_xdp_cqes: 0
>>>>>> rx_wqe_err: 0
>>>>>> rx_mpwqe_filler_cqes: 0
>>>>>> rx_mpwqe_filler_strides: 0
>>>>>> rx_buff_alloc_err: 0
>>>>>> rx_cqe_compress_blks: 0
>>>>>> rx_cqe_compress_pkts: 0
>>>>> If this is a pcie bottleneck it might be useful to enable CQE
>>>>> compression (to reduce PCIe completion descriptors transactions)
>>>>> you should see the above rx_cqe_compress_pkts increasing when
>>>>> enabled.
>>>>>
>>>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>>>> $ ethtool --show-priv-flags enp175s0f1
>>>>> Private flags for p6p1:
>>>>> rx_cqe_moder : on
>>>>> cqe_moder : off
>>>>> rx_cqe_compress : on
>>>>> ...
>>>>>
>>>>> try this on both interfaces.
>>>> Done
>>>> ethtool --show-priv-flags enp175s0f1
>>>> Private flags for enp175s0f1:
>>>> rx_cqe_moder : on
>>>> tx_cqe_moder : off
>>>> rx_cqe_compress : on
>>>> rx_striding_rq : off
>>>> rx_no_csum_complete: off
>>>>
>>>> ethtool --show-priv-flags enp175s0f0
>>>> Private flags for enp175s0f0:
>>>> rx_cqe_moder : on
>>>> tx_cqe_moder : off
>>>> rx_cqe_compress : on
>>>> rx_striding_rq : off
>>>> rx_no_csum_complete: off
>>>>
>>> did it help reduce the load on the pcie ? do you see more pps ?
>>> what is the ratio between rx_cqe_compress_pkts and over all rx packets
>>> ?
>> So - a little more pps
>> Before change top - graph / after bottom -> image with graph stats
>> from proc/net/dev
> Attached link to graph
> https://uploadfiles.io/5vgbh
>
>
Enabling tx_cqe_moder for both ports
helps alot with cpu load at same traffic lvl - the change is about -20%
on all 28 cores
>> cqe_compress enabled at 11:55
>>
>> Sorry - but for real life traffic it is hard to do any counter
>> differences - cause traffic just rising alone from minute to minute :)
>> But for that time the change is visible on graph - cause was almost
>> same for past 20minutes before change.
>>
>>
>> full ethtool below:
>> NIC statistics:
>> rx_packets: 516522465438
>> rx_bytes: 680052911258729
>> tx_packets: 677697545586
>> tx_bytes: 413647643141709
>> tx_tso_packets: 42530913279
>> tx_tso_bytes: 235655668554142
>> tx_tso_inner_packets: 0
>> tx_tso_inner_bytes: 0
>> tx_added_vlan_packets: 551156530885
>> tx_nop: 8536823558
>> rx_lro_packets: 0
>> rx_lro_bytes: 0
>> rx_ecn_mark: 0
>> rx_removed_vlan_packets: 516522465438
>> rx_csum_unnecessary: 0
>> rx_csum_none: 50382868
>> rx_csum_complete: 516472082570
>> rx_csum_unnecessary_inner: 0
>> rx_xdp_drop: 0
>> rx_xdp_redirect: 0
>> rx_xdp_tx_xmit: 0
>> rx_xdp_tx_full: 0
>> rx_xdp_tx_err: 0
>> rx_xdp_tx_cqe: 0
>> tx_csum_none: 494075047017
>> tx_csum_partial: 57081483898
>> tx_csum_partial_inner: 0
>> tx_queue_stopped: 518624
>> tx_queue_dropped: 0
>> tx_xmit_more: 1717880628
>> tx_recover: 0
>> tx_cqes: 549438869029
>> tx_queue_wake: 518627
>> tx_udp_seg_rem: 0
>> tx_cqe_err: 0
>> tx_xdp_xmit: 0
>> tx_xdp_full: 0
>> tx_xdp_err: 0
>> tx_xdp_cqes: 0
>> rx_wqe_err: 0
>> rx_mpwqe_filler_cqes: 0
>> rx_mpwqe_filler_strides: 0
>> rx_buff_alloc_err: 0
>> rx_cqe_compress_blks: 11483228712
>> rx_cqe_compress_pkts: 25794213324
>> rx_page_reuse: 0
>> rx_cache_reuse: 63610249810
>> rx_cache_full: 194650916511
>> rx_cache_empty: 1118208
>> rx_cache_busy: 194650982430
>> rx_cache_waive: 0
>> rx_congst_umr: 0
>> rx_arfs_err: 0
>> ch_events: 119556002196
>> ch_poll: 121107424977
>> ch_arm: 115856746008
>> ch_aff_change: 31
>> ch_eq_rearm: 0
>> rx_out_of_buffer: 6880325
>> rx_if_down_packets: 2062529
>> rx_vport_unicast_packets: 517433716795
>> rx_vport_unicast_bytes: 683464347301443
>> tx_vport_unicast_packets: 677697453738
>> tx_vport_unicast_bytes: 415788589663315
>> rx_vport_multicast_packets: 208258309
>> rx_vport_multicast_bytes: 14224046052
>> tx_vport_multicast_packets: 21689
>> tx_vport_multicast_bytes: 2158334
>> rx_vport_broadcast_packets: 75838646
>> rx_vport_broadcast_bytes: 4697944695
>> tx_vport_broadcast_packets: 68730
>> tx_vport_broadcast_bytes: 4123800
>> rx_vport_rdma_unicast_packets: 0
>> rx_vport_rdma_unicast_bytes: 0
>> tx_vport_rdma_unicast_packets: 0
>> tx_vport_rdma_unicast_bytes: 0
>> rx_vport_rdma_multicast_packets: 0
>> rx_vport_rdma_multicast_bytes: 0
>> tx_vport_rdma_multicast_packets: 0
>> tx_vport_rdma_multicast_bytes: 0
>> tx_packets_phy: 677697543252
>> rx_packets_phy: 521319491878
>> rx_crc_errors_phy: 0
>> tx_bytes_phy: 418499385791411
>> rx_bytes_phy: 690322537017274
>> tx_multicast_phy: 21689
>> tx_broadcast_phy: 68730
>> rx_multicast_phy: 208258305
>> rx_broadcast_phy: 75838646
>> rx_in_range_len_errors_phy: 4
>> rx_out_of_range_len_phy: 0
>> rx_oversize_pkts_phy: 0
>> rx_symbol_err_phy: 0
>> tx_mac_control_phy: 0
>> rx_mac_control_phy: 0
>> rx_unsupported_op_phy: 0
>> rx_pause_ctrl_phy: 0
>> tx_pause_ctrl_phy: 0
>> rx_discards_phy: 3601449265
>> tx_discards_phy: 0
>> tx_errors_phy: 0
>> rx_undersize_pkts_phy: 0
>> rx_fragments_phy: 0
>> rx_jabbers_phy: 0
>> rx_64_bytes_phy: 1416456771
>> rx_65_to_127_bytes_phy: 40750434737
>> rx_128_to_255_bytes_phy: 11518110310
>> rx_256_to_511_bytes_phy: 7055850637
>> rx_512_to_1023_bytes_phy: 7811550424
>> rx_1024_to_1518_bytes_phy: 265547564845
>> rx_1519_to_2047_bytes_phy: 187219522899
>> rx_2048_to_4095_bytes_phy: 0
>> rx_4096_to_8191_bytes_phy: 0
>> rx_8192_to_10239_bytes_phy: 0
>> link_down_events_phy: 0
>> rx_pcs_symbol_err_phy: 0
>> rx_corrected_bits_phy: 0
>> rx_pci_signal_integrity: 0
>> tx_pci_signal_integrity: 48
>> rx_prio0_bytes: 688807632117485
>> rx_prio0_packets: 516310309931
>> tx_prio0_bytes: 418499382756025
>> tx_prio0_packets: 677697534982
>> rx_prio1_bytes: 1497701612877
>> rx_prio1_packets: 1206768094
>> tx_prio1_bytes: 0
>> tx_prio1_packets: 0
>> rx_prio2_bytes: 112271227
>> rx_prio2_packets: 337295
>> tx_prio2_bytes: 0
>> tx_prio2_packets: 0
>> rx_prio3_bytes: 1165455555
>> rx_prio3_packets: 1544310
>> tx_prio3_bytes: 0
>> tx_prio3_packets: 0
>> rx_prio4_bytes: 161857240
>> rx_prio4_packets: 341392
>> tx_prio4_bytes: 0
>> tx_prio4_packets: 0
>> rx_prio5_bytes: 455031612
>> rx_prio5_packets: 2861469
>> tx_prio5_bytes: 0
>> tx_prio5_packets: 0
>> rx_prio6_bytes: 1873928697
>> rx_prio6_packets: 5146981
>> tx_prio6_bytes: 0
>> tx_prio6_packets: 0
>> rx_prio7_bytes: 13423452430
>> rx_prio7_packets: 190724796
>> tx_prio7_bytes: 0
>> tx_prio7_packets: 0
>> module_unplug: 0
>> module_bus_stuck: 0
>> module_high_temp: 0
>> module_bad_shorted: 0
>> ch0_events: 4252266777
>> ch0_poll: 4330804273
>> ch0_arm: 4120233182
>> ch0_aff_change: 2
>> ch0_eq_rearm: 0
>> ch1_events: 3938415938
>> ch1_poll: 4012621322
>> ch1_arm: 3810131188
>> ch1_aff_change: 2
>> ch1_eq_rearm: 0
>> ch2_events: 3897428860
>> ch2_poll: 3973886848
>> ch2_arm: 3773019397
>> ch2_aff_change: 1
>> ch2_eq_rearm: 0
>> ch3_events: 4108000541
>> ch3_poll: 4180139872
>> ch3_arm: 3982093366
>> ch3_aff_change: 1
>> ch3_eq_rearm: 0
>> ch4_events: 4652570079
>> ch4_poll: 4720541090
>> ch4_arm: 4524475054
>> ch4_aff_change: 2
>> ch4_eq_rearm: 0
>> ch5_events: 3899177385
>> ch5_poll: 3974274186
>> ch5_arm: 3772299186
>> ch5_aff_change: 2
>> ch5_eq_rearm: 0
>> ch6_events: 3915161350
>> ch6_poll: 3992338199
>> ch6_arm: 3794710989
>> ch6_aff_change: 0
>> ch6_eq_rearm: 0
>> ch7_events: 4008175631
>> ch7_poll: 4081321248
>> ch7_arm: 3882263723
>> ch7_aff_change: 0
>> ch7_eq_rearm: 0
>> ch8_events: 4207422352
>> ch8_poll: 4276465449
>> ch8_arm: 4077650366
>> ch8_aff_change: 0
>> ch8_eq_rearm: 0
>> ch9_events: 4036491879
>> ch9_poll: 4108975987
>> ch9_arm: 3914493694
>> ch9_aff_change: 0
>> ch9_eq_rearm: 0
>> ch10_events: 4066261595
>> ch10_poll: 4134419606
>> ch10_arm: 3936637711
>> ch10_aff_change: 1
>> ch10_eq_rearm: 0
>> ch11_events: 4440494043
>> ch11_poll: 4507578730
>> ch11_arm: 4318629438
>> ch11_aff_change: 0
>> ch11_eq_rearm: 0
>> ch12_events: 4066958252
>> ch12_poll: 4130191506
>> ch12_arm: 3934337782
>> ch12_aff_change: 0
>> ch12_eq_rearm: 0
>> ch13_events: 4051309159
>> ch13_poll: 4118864120
>> ch13_arm: 3921011919
>> ch13_aff_change: 0
>> ch13_eq_rearm: 0
>> ch14_events: 4321664800
>> ch14_poll: 4382433680
>> ch14_arm: 4186130552
>> ch14_aff_change: 0
>> ch14_eq_rearm: 0
>> ch15_events: 4701102075
>> ch15_poll: 4760373932
>> ch15_arm: 4570151468
>> ch15_aff_change: 0
>> ch15_eq_rearm: 0
>> ch16_events: 4311052687
>> ch16_poll: 4345937129
>> ch16_arm: 4170883819
>> ch16_aff_change: 0
>> ch16_eq_rearm: 0
>> ch17_events: 4647570931
>> ch17_poll: 4680218533
>> ch17_arm: 4509426288
>> ch17_aff_change: 0
>> ch17_eq_rearm: 0
>> ch18_events: 4598195702
>> ch18_poll: 4631314898
>> ch18_arm: 4457267084
>> ch18_aff_change: 0
>> ch18_eq_rearm: 0
>> ch19_events: 4808094560
>> ch19_poll: 4841368340
>> ch19_arm: 4670604358
>> ch19_aff_change: 0
>> ch19_eq_rearm: 0
>> ch20_events: 4240910605
>> ch20_poll: 4276531502
>> ch20_arm: 4101767278
>> ch20_aff_change: 1
>> ch20_eq_rearm: 0
>> ch21_events: 4389371472
>> ch21_poll: 4426870311
>> ch21_arm: 4249339045
>> ch21_aff_change: 2
>> ch21_eq_rearm: 0
>> ch22_events: 4282958754
>> ch22_poll: 4319228073
>> ch22_arm: 4145102991
>> ch22_aff_change: 2
>> ch22_eq_rearm: 0
>> ch23_events: 4440196528
>> ch23_poll: 4474090188
>> ch23_arm: 4300837147
>> ch23_aff_change: 2
>> ch23_eq_rearm: 0
>> ch24_events: 4326875785
>> ch24_poll: 4364971263
>> ch24_arm: 4186404526
>> ch24_aff_change: 2
>> ch24_eq_rearm: 0
>> ch25_events: 4286528453
>> ch25_poll: 4324089445
>> ch25_arm: 4147222616
>> ch25_aff_change: 3
>> ch25_eq_rearm: 0
>> ch26_events: 4098043104
>> ch26_poll: 4138133745
>> ch26_arm: 3967438971
>> ch26_aff_change: 4
>> ch26_eq_rearm: 0
>> ch27_events: 4563302840
>> ch27_poll: 4599441446
>> ch27_arm: 4432182806
>> ch27_aff_change: 4
>> ch27_eq_rearm: 0
>> ch28_events: 4
>> ch28_poll: 4
>> ch28_arm: 4
>> ch28_aff_change: 0
>> ch28_eq_rearm: 0
>> ch29_events: 6
>> ch29_poll: 6
>> ch29_arm: 6
>> ch29_aff_change: 0
>> ch29_eq_rearm: 0
>> ch30_events: 4
>> ch30_poll: 4
>> ch30_arm: 4
>> ch30_aff_change: 0
>> ch30_eq_rearm: 0
>> ch31_events: 4
>> ch31_poll: 4
>> ch31_arm: 4
>> ch31_aff_change: 0
>> ch31_eq_rearm: 0
>> ch32_events: 4
>> ch32_poll: 4
>> ch32_arm: 4
>> ch32_aff_change: 0
>> ch32_eq_rearm: 0
>> ch33_events: 4
>> ch33_poll: 4
>> ch33_arm: 4
>> ch33_aff_change: 0
>> ch33_eq_rearm: 0
>> ch34_events: 4
>> ch34_poll: 4
>> ch34_arm: 4
>> ch34_aff_change: 0
>> ch34_eq_rearm: 0
>> ch35_events: 4
>> ch35_poll: 4
>> ch35_arm: 4
>> ch35_aff_change: 0
>> ch35_eq_rearm: 0
>> ch36_events: 4
>> ch36_poll: 4
>> ch36_arm: 4
>> ch36_aff_change: 0
>> ch36_eq_rearm: 0
>> ch37_events: 4
>> ch37_poll: 4
>> ch37_arm: 4
>> ch37_aff_change: 0
>> ch37_eq_rearm: 0
>> ch38_events: 4
>> ch38_poll: 4
>> ch38_arm: 4
>> ch38_aff_change: 0
>> ch38_eq_rearm: 0
>> ch39_events: 4
>> ch39_poll: 4
>> ch39_arm: 4
>> ch39_aff_change: 0
>> ch39_eq_rearm: 0
>> ch40_events: 4
>> ch40_poll: 4
>> ch40_arm: 4
>> ch40_aff_change: 0
>> ch40_eq_rearm: 0
>> ch41_events: 4
>> ch41_poll: 4
>> ch41_arm: 4
>> ch41_aff_change: 0
>> ch41_eq_rearm: 0
>> ch42_events: 4
>> ch42_poll: 4
>> ch42_arm: 4
>> ch42_aff_change: 0
>> ch42_eq_rearm: 0
>> ch43_events: 4
>> ch43_poll: 4
>> ch43_arm: 4
>> ch43_aff_change: 0
>> ch43_eq_rearm: 0
>> ch44_events: 4
>> ch44_poll: 4
>> ch44_arm: 4
>> ch44_aff_change: 0
>> ch44_eq_rearm: 0
>> ch45_events: 4
>> ch45_poll: 4
>> ch45_arm: 4
>> ch45_aff_change: 0
>> ch45_eq_rearm: 0
>> ch46_events: 4
>> ch46_poll: 4
>> ch46_arm: 4
>> ch46_aff_change: 0
>> ch46_eq_rearm: 0
>> ch47_events: 4
>> ch47_poll: 4
>> ch47_arm: 4
>> ch47_aff_change: 0
>> ch47_eq_rearm: 0
>> ch48_events: 4
>> ch48_poll: 4
>> ch48_arm: 4
>> ch48_aff_change: 0
>> ch48_eq_rearm: 0
>> ch49_events: 4
>> ch49_poll: 4
>> ch49_arm: 4
>> ch49_aff_change: 0
>> ch49_eq_rearm: 0
>> ch50_events: 4
>> ch50_poll: 4
>> ch50_arm: 4
>> ch50_aff_change: 0
>> ch50_eq_rearm: 0
>> ch51_events: 4
>> ch51_poll: 4
>> ch51_arm: 4
>> ch51_aff_change: 0
>> ch51_eq_rearm: 0
>> ch52_events: 4
>> ch52_poll: 4
>> ch52_arm: 4
>> ch52_aff_change: 0
>> ch52_eq_rearm: 0
>> ch53_events: 4
>> ch53_poll: 4
>> ch53_arm: 4
>> ch53_aff_change: 0
>> ch53_eq_rearm: 0
>> ch54_events: 4
>> ch54_poll: 4
>> ch54_arm: 4
>> ch54_aff_change: 0
>> ch54_eq_rearm: 0
>> ch55_events: 4
>> ch55_poll: 4
>> ch55_arm: 4
>> ch55_aff_change: 0
>> ch55_eq_rearm: 0
>> rx0_packets: 21390033774
>> rx0_bytes: 27326856299122
>> rx0_csum_complete: 21339650906
>> rx0_csum_unnecessary: 0
>> rx0_csum_unnecessary_inner: 0
>> rx0_csum_none: 50382868
>> rx0_xdp_drop: 0
>> rx0_xdp_redirect: 0
>> rx0_lro_packets: 0
>> rx0_lro_bytes: 0
>> rx0_ecn_mark: 0
>> rx0_removed_vlan_packets: 21390033774
>> rx0_wqe_err: 0
>> rx0_mpwqe_filler_cqes: 0
>> rx0_mpwqe_filler_strides: 0
>> rx0_buff_alloc_err: 0
>> rx0_cqe_compress_blks: 481077641
>> rx0_cqe_compress_pkts: 1085647489
>> rx0_page_reuse: 0
>> rx0_cache_reuse: 19050049
>> rx0_cache_full: 10675964285
>> rx0_cache_empty: 37376
>> rx0_cache_busy: 10675966819
>> rx0_cache_waive: 0
>> rx0_congst_umr: 0
>> rx0_arfs_err: 0
>> rx0_xdp_tx_xmit: 0
>> rx0_xdp_tx_full: 0
>> rx0_xdp_tx_err: 0
>> rx0_xdp_tx_cqes: 0
>> rx1_packets: 19868919527
>> rx1_bytes: 26149716991561
>> rx1_csum_complete: 19868919527
>> rx1_csum_unnecessary: 0
>> rx1_csum_unnecessary_inner: 0
>> rx1_csum_none: 0
>> rx1_xdp_drop: 0
>> rx1_xdp_redirect: 0
>> rx1_lro_packets: 0
>> rx1_lro_bytes: 0
>> rx1_ecn_mark: 0
>> rx1_removed_vlan_packets: 19868919527
>> rx1_wqe_err: 0
>> rx1_mpwqe_filler_cqes: 0
>> rx1_mpwqe_filler_strides: 0
>> rx1_buff_alloc_err: 0
>> rx1_cqe_compress_blks: 420210560
>> rx1_cqe_compress_pkts: 941233388
>> rx1_page_reuse: 0
>> rx1_cache_reuse: 46200002
>> rx1_cache_full: 9888257242
>> rx1_cache_empty: 37376
>> rx1_cache_busy: 9888259746
>> rx1_cache_waive: 0
>> rx1_congst_umr: 0
>> rx1_arfs_err: 0
>> rx1_xdp_tx_xmit: 0
>> rx1_xdp_tx_full: 0
>> rx1_xdp_tx_err: 0
>> rx1_xdp_tx_cqes: 0
>> rx2_packets: 19575013662
>> rx2_bytes: 25759818417945
>> rx2_csum_complete: 19575013662
>> rx2_csum_unnecessary: 0
>> rx2_csum_unnecessary_inner: 0
>> rx2_csum_none: 0
>> rx2_xdp_drop: 0
>> rx2_xdp_redirect: 0
>> rx2_lro_packets: 0
>> rx2_lro_bytes: 0
>> rx2_ecn_mark: 0
>> rx2_removed_vlan_packets: 19575013662
>> rx2_wqe_err: 0
>> rx2_mpwqe_filler_cqes: 0
>> rx2_mpwqe_filler_strides: 0
>> rx2_buff_alloc_err: 0
>> rx2_cqe_compress_blks: 412345511
>> rx2_cqe_compress_pkts: 923376167
>> rx2_page_reuse: 0
>> rx2_cache_reuse: 38837731
>> rx2_cache_full: 9748666548
>> rx2_cache_empty: 37376
>> rx2_cache_busy: 9748669093
>> rx2_cache_waive: 0
>> rx2_congst_umr: 0
>> rx2_arfs_err: 0
>> rx2_xdp_tx_xmit: 0
>> rx2_xdp_tx_full: 0
>> rx2_xdp_tx_err: 0
>> rx2_xdp_tx_cqes: 0
>> rx3_packets: 19795911749
>> rx3_bytes: 25969475566905
>> rx3_csum_complete: 19795911749
>> rx3_csum_unnecessary: 0
>> rx3_csum_unnecessary_inner: 0
>> rx3_csum_none: 0
>> rx3_xdp_drop: 0
>> rx3_xdp_redirect: 0
>> rx3_lro_packets: 0
>> rx3_lro_bytes: 0
>> rx3_ecn_mark: 0
>> rx3_removed_vlan_packets: 19795911749
>> rx3_wqe_err: 0
>> rx3_mpwqe_filler_cqes: 0
>> rx3_mpwqe_filler_strides: 0
>> rx3_buff_alloc_err: 0
>> rx3_cqe_compress_blks: 416658765
>> rx3_cqe_compress_pkts: 934986266
>> rx3_page_reuse: 0
>> rx3_cache_reuse: 34542124
>> rx3_cache_full: 9863411232
>> rx3_cache_empty: 37376
>> rx3_cache_busy: 9863413732
>> rx3_cache_waive: 0
>> rx3_congst_umr: 0
>> rx3_arfs_err: 0
>> rx3_xdp_tx_xmit: 0
>> rx3_xdp_tx_full: 0
>> rx3_xdp_tx_err: 0
>> rx3_xdp_tx_cqes: 0
>> rx4_packets: 20445652378
>> rx4_bytes: 26949065110265
>> rx4_csum_complete: 20445652378
>> rx4_csum_unnecessary: 0
>> rx4_csum_unnecessary_inner: 0
>> rx4_csum_none: 0
>> rx4_xdp_drop: 0
>> rx4_xdp_redirect: 0
>> rx4_lro_packets: 0
>> rx4_lro_bytes: 0
>> rx4_ecn_mark: 0
>> rx4_removed_vlan_packets: 20445652378
>> rx4_wqe_err: 0
>> rx4_mpwqe_filler_cqes: 0
>> rx4_mpwqe_filler_strides: 0
>> rx4_buff_alloc_err: 0
>> rx4_cqe_compress_blks: 506085858
>> rx4_cqe_compress_pkts: 1147860328
>> rx4_page_reuse: 0
>> rx4_cache_reuse: 10122542864
>> rx4_cache_full: 100281206
>> rx4_cache_empty: 37376
>> rx4_cache_busy: 100283304
>> rx4_cache_waive: 0
>> rx4_congst_umr: 0
>> rx4_arfs_err: 0
>> rx4_xdp_tx_xmit: 0
>> rx4_xdp_tx_full: 0
>> rx4_xdp_tx_err: 0
>> rx4_xdp_tx_cqes: 0
>> rx5_packets: 19622362246
>> rx5_bytes: 25843450982982
>> rx5_csum_complete: 19622362246
>> rx5_csum_unnecessary: 0
>> rx5_csum_unnecessary_inner: 0
>> rx5_csum_none: 0
>> rx5_xdp_drop: 0
>> rx5_xdp_redirect: 0
>> rx5_lro_packets: 0
>> rx5_lro_bytes: 0
>> rx5_ecn_mark: 0
>> rx5_removed_vlan_packets: 19622362246
>> rx5_wqe_err: 0
>> rx5_mpwqe_filler_cqes: 0
>> rx5_mpwqe_filler_strides: 0
>> rx5_buff_alloc_err: 0
>> rx5_cqe_compress_blks: 422840924
>> rx5_cqe_compress_pkts: 948005878
>> rx5_page_reuse: 0
>> rx5_cache_reuse: 31285453
>> rx5_cache_full: 9779893117
>> rx5_cache_empty: 37376
>> rx5_cache_busy: 9779895647
>> rx5_cache_waive: 0
>> rx5_congst_umr: 0
>> rx5_arfs_err: 0
>> rx5_xdp_tx_xmit: 0
>> rx5_xdp_tx_full: 0
>> rx5_xdp_tx_err: 0
>> rx5_xdp_tx_cqes: 0
>> rx6_packets: 19788231278
>> rx6_bytes: 25985783006486
>> rx6_csum_complete: 19788231278
>> rx6_csum_unnecessary: 0
>> rx6_csum_unnecessary_inner: 0
>> rx6_csum_none: 0
>> rx6_xdp_drop: 0
>> rx6_xdp_redirect: 0
>> rx6_lro_packets: 0
>> rx6_lro_bytes: 0
>> rx6_ecn_mark: 0
>> rx6_removed_vlan_packets: 19788231278
>> rx6_wqe_err: 0
>> rx6_mpwqe_filler_cqes: 0
>> rx6_mpwqe_filler_strides: 0
>> rx6_buff_alloc_err: 0
>> rx6_cqe_compress_blks: 418799056
>> rx6_cqe_compress_pkts: 938282685
>> rx6_page_reuse: 0
>> rx6_cache_reuse: 18114793
>> rx6_cache_full: 9875998295
>> rx6_cache_empty: 37376
>> rx6_cache_busy: 9876000831
>> rx6_cache_waive: 0
>> rx6_congst_umr: 0
>> rx6_arfs_err: 0
>> rx6_xdp_tx_xmit: 0
>> rx6_xdp_tx_full: 0
>> rx6_xdp_tx_err: 0
>> rx6_xdp_tx_cqes: 0
>> rx7_packets: 19795759168
>> rx7_bytes: 26085056586860
>> rx7_csum_complete: 19795759168
>> rx7_csum_unnecessary: 0
>> rx7_csum_unnecessary_inner: 0
>> rx7_csum_none: 0
>> rx7_xdp_drop: 0
>> rx7_xdp_redirect: 0
>> rx7_lro_packets: 0
>> rx7_lro_bytes: 0
>> rx7_ecn_mark: 0
>> rx7_removed_vlan_packets: 19795759168
>> rx7_wqe_err: 0
>> rx7_mpwqe_filler_cqes: 0
>> rx7_mpwqe_filler_strides: 0
>> rx7_buff_alloc_err: 0
>> rx7_cqe_compress_blks: 413959224
>> rx7_cqe_compress_pkts: 927675936
>> rx7_page_reuse: 0
>> rx7_cache_reuse: 23902990
>> rx7_cache_full: 9873974042
>> rx7_cache_empty: 37376
>> rx7_cache_busy: 9873976574
>> rx7_cache_waive: 0
>> rx7_congst_umr: 0
>> rx7_arfs_err: 0
>> rx7_xdp_tx_xmit: 0
>> rx7_xdp_tx_full: 0
>> rx7_xdp_tx_err: 0
>> rx7_xdp_tx_cqes: 0
>> rx8_packets: 19963477439
>> rx8_bytes: 26384640501789
>> rx8_csum_complete: 19963477439
>> rx8_csum_unnecessary: 0
>> rx8_csum_unnecessary_inner: 0
>> rx8_csum_none: 0
>> rx8_xdp_drop: 0
>> rx8_xdp_redirect: 0
>> rx8_lro_packets: 0
>> rx8_lro_bytes: 0
>> rx8_ecn_mark: 0
>> rx8_removed_vlan_packets: 19963477439
>> rx8_wqe_err: 0
>> rx8_mpwqe_filler_cqes: 0
>> rx8_mpwqe_filler_strides: 0
>> rx8_buff_alloc_err: 0
>> rx8_cqe_compress_blks: 420422857
>> rx8_cqe_compress_pkts: 942720292
>> rx8_page_reuse: 0
>> rx8_cache_reuse: 88181713
>> rx8_cache_full: 9893554525
>> rx8_cache_empty: 37376
>> rx8_cache_busy: 9893556983
>> rx8_cache_waive: 0
>> rx8_congst_umr: 0
>> rx8_arfs_err: 0
>> rx8_xdp_tx_xmit: 0
>> rx8_xdp_tx_full: 0
>> rx8_xdp_tx_err: 0
>> rx8_xdp_tx_cqes: 0
>> rx9_packets: 19726642138
>> rx9_bytes: 26063924286499
>> rx9_csum_complete: 19726642138
>> rx9_csum_unnecessary: 0
>> rx9_csum_unnecessary_inner: 0
>> rx9_csum_none: 0
>> rx9_xdp_drop: 0
>> rx9_xdp_redirect: 0
>> rx9_lro_packets: 0
>> rx9_lro_bytes: 0
>> rx9_ecn_mark: 0
>> rx9_removed_vlan_packets: 19726642138
>> rx9_wqe_err: 0
>> rx9_mpwqe_filler_cqes: 0
>> rx9_mpwqe_filler_strides: 0
>> rx9_buff_alloc_err: 0
>> rx9_cqe_compress_blks: 424227411
>> rx9_cqe_compress_pkts: 951534873
>> rx9_page_reuse: 0
>> rx9_cache_reuse: 482901440
>> rx9_cache_full: 9380417487
>> rx9_cache_empty: 37376
>> rx9_cache_busy: 9380419608
>> rx9_cache_waive: 0
>> rx9_congst_umr: 0
>> rx9_arfs_err: 0
>> rx9_xdp_tx_xmit: 0
>> rx9_xdp_tx_full: 0
>> rx9_xdp_tx_err: 0
>> rx9_xdp_tx_cqes: 0
>> rx10_packets: 19901229170
>> rx10_bytes: 26300854495044
>> rx10_csum_complete: 19901229170
>> rx10_csum_unnecessary: 0
>> rx10_csum_unnecessary_inner: 0
>> rx10_csum_none: 0
>> rx10_xdp_drop: 0
>> rx10_xdp_redirect: 0
>> rx10_lro_packets: 0
>> rx10_lro_bytes: 0
>> rx10_ecn_mark: 0
>> rx10_removed_vlan_packets: 19901229170
>> rx10_wqe_err: 0
>> rx10_mpwqe_filler_cqes: 0
>> rx10_mpwqe_filler_strides: 0
>> rx10_buff_alloc_err: 0
>> rx10_cqe_compress_blks: 419082938
>> rx10_cqe_compress_pkts: 940791347
>> rx10_page_reuse: 0
>> rx10_cache_reuse: 14896055
>> rx10_cache_full: 9935715977
>> rx10_cache_empty: 37376
>> rx10_cache_busy: 9935718513
>> rx10_cache_waive: 0
>> rx10_congst_umr: 0
>> rx10_arfs_err: 0
>> rx10_xdp_tx_xmit: 0
>> rx10_xdp_tx_full: 0
>> rx10_xdp_tx_err: 0
>> rx10_xdp_tx_cqes: 0
>> rx11_packets: 20352190494
>> rx11_bytes: 26851034425372
>> rx11_csum_complete: 20352190494
>> rx11_csum_unnecessary: 0
>> rx11_csum_unnecessary_inner: 0
>> rx11_csum_none: 0
>> rx11_xdp_drop: 0
>> rx11_xdp_redirect: 0
>> rx11_lro_packets: 0
>> rx11_lro_bytes: 0
>> rx11_ecn_mark: 0
>> rx11_removed_vlan_packets: 20352190494
>> rx11_wqe_err: 0
>> rx11_mpwqe_filler_cqes: 0
>> rx11_mpwqe_filler_strides: 0
>> rx11_buff_alloc_err: 0
>> rx11_cqe_compress_blks: 501992147
>> rx11_cqe_compress_pkts: 1140398610
>> rx11_page_reuse: 0
>> rx11_cache_reuse: 10071721531
>> rx11_cache_full: 104371621
>> rx11_cache_empty: 37376
>> rx11_cache_busy: 104373697
>> rx11_cache_waive: 0
>> rx11_congst_umr: 0
>> rx11_arfs_err: 0
>> rx11_xdp_tx_xmit: 0
>> rx11_xdp_tx_full: 0
>> rx11_xdp_tx_err: 0
>> rx11_xdp_tx_cqes: 0
>> rx12_packets: 19934747149
>> rx12_bytes: 26296478787829
>> rx12_csum_complete: 19934747149
>> rx12_csum_unnecessary: 0
>> rx12_csum_unnecessary_inner: 0
>> rx12_csum_none: 0
>> rx12_xdp_drop: 0
>> rx12_xdp_redirect: 0
>> rx12_lro_packets: 0
>> rx12_lro_bytes: 0
>> rx12_ecn_mark: 0
>> rx12_removed_vlan_packets: 19934747149
>> rx12_wqe_err: 0
>> rx12_mpwqe_filler_cqes: 0
>> rx12_mpwqe_filler_strides: 0
>> rx12_buff_alloc_err: 0
>> rx12_cqe_compress_blks: 443350570
>> rx12_cqe_compress_pkts: 995997220
>> rx12_page_reuse: 0
>> rx12_cache_reuse: 9864934174
>> rx12_cache_full: 102437428
>> rx12_cache_empty: 37376
>> rx12_cache_busy: 102439382
>> rx12_cache_waive: 0
>> rx12_congst_umr: 0
>> rx12_arfs_err: 0
>> rx12_xdp_tx_xmit: 0
>> rx12_xdp_tx_full: 0
>> rx12_xdp_tx_err: 0
>> rx12_xdp_tx_cqes: 0
>> rx13_packets: 19866908096
>> rx13_bytes: 26160931936186
>> rx13_csum_complete: 19866908096
>> rx13_csum_unnecessary: 0
>> rx13_csum_unnecessary_inner: 0
>> rx13_csum_none: 0
>> rx13_xdp_drop: 0
>> rx13_xdp_redirect: 0
>> rx13_lro_packets: 0
>> rx13_lro_bytes: 0
>> rx13_ecn_mark: 0
>> rx13_removed_vlan_packets: 19866908096
>> rx13_wqe_err: 0
>> rx13_mpwqe_filler_cqes: 0
>> rx13_mpwqe_filler_strides: 0
>> rx13_buff_alloc_err: 0
>> rx13_cqe_compress_blks: 413640141
>> rx13_cqe_compress_pkts: 926175066
>> rx13_page_reuse: 0
>> rx13_cache_reuse: 36358610
>> rx13_cache_full: 9897092921
>> rx13_cache_empty: 37376
>> rx13_cache_busy: 9897095422
>> rx13_cache_waive: 0
>> rx13_congst_umr: 0
>> rx13_arfs_err: 0
>> rx13_xdp_tx_xmit: 0
>> rx13_xdp_tx_full: 0
>> rx13_xdp_tx_err: 0
>> rx13_xdp_tx_cqes: 0
>> rx14_packets: 20229035746
>> rx14_bytes: 26655092809172
>> rx14_csum_complete: 20229035746
>> rx14_csum_unnecessary: 0
>> rx14_csum_unnecessary_inner: 0
>> rx14_csum_none: 0
>> rx14_xdp_drop: 0
>> rx14_xdp_redirect: 0
>> rx14_lro_packets: 0
>> rx14_lro_bytes: 0
>> rx14_ecn_mark: 0
>> rx14_removed_vlan_packets: 20229035746
>> rx14_wqe_err: 0
>> rx14_mpwqe_filler_cqes: 0
>> rx14_mpwqe_filler_strides: 0
>> rx14_buff_alloc_err: 0
>> rx14_cqe_compress_blks: 460990337
>> rx14_cqe_compress_pkts: 1041287948
>> rx14_page_reuse: 0
>> rx14_cache_reuse: 25649275
>> rx14_cache_full: 10088866045
>> rx14_cache_empty: 37376
>> rx14_cache_busy: 10088868574
>> rx14_cache_waive: 0
>> rx14_congst_umr: 0
>> rx14_arfs_err: 0
>> rx14_xdp_tx_xmit: 0
>> rx14_xdp_tx_full: 0
>> rx14_xdp_tx_err: 0
>> rx14_xdp_tx_cqes: 0
>> rx15_packets: 20528177154
>> rx15_bytes: 27029263893264
>> rx15_csum_complete: 20528177154
>> rx15_csum_unnecessary: 0
>> rx15_csum_unnecessary_inner: 0
>> rx15_csum_none: 0
>> rx15_xdp_drop: 0
>> rx15_xdp_redirect: 0
>> rx15_lro_packets: 0
>> rx15_lro_bytes: 0
>> rx15_ecn_mark: 0
>> rx15_removed_vlan_packets: 20528177154
>> rx15_wqe_err: 0
>> rx15_mpwqe_filler_cqes: 0
>> rx15_mpwqe_filler_strides: 0
>> rx15_buff_alloc_err: 0
>> rx15_cqe_compress_blks: 476776176
>> rx15_cqe_compress_pkts: 1076153263
>> rx15_page_reuse: 0
>> rx15_cache_reuse: 48426735
>> rx15_cache_full: 10215659289
>> rx15_cache_empty: 37376
>> rx15_cache_busy: 10215661817
>> rx15_cache_waive: 0
>> rx15_congst_umr: 0
>> rx15_arfs_err: 0
>> rx15_xdp_tx_xmit: 0
>> rx15_xdp_tx_full: 0
>> rx15_xdp_tx_err: 0
>> rx15_xdp_tx_cqes: 0
>> rx16_packets: 16104078098
>> rx16_bytes: 21256361789679
>> rx16_csum_complete: 16104078098
>> rx16_csum_unnecessary: 0
>> rx16_csum_unnecessary_inner: 0
>> rx16_csum_none: 0
>> rx16_xdp_drop: 0
>> rx16_xdp_redirect: 0
>> rx16_lro_packets: 0
>> rx16_lro_bytes: 0
>> rx16_ecn_mark: 0
>> rx16_removed_vlan_packets: 16104078098
>> rx16_wqe_err: 0
>> rx16_mpwqe_filler_cqes: 0
>> rx16_mpwqe_filler_strides: 0
>> rx16_buff_alloc_err: 0
>> rx16_cqe_compress_blks: 352082054
>> rx16_cqe_compress_pkts: 787161670
>> rx16_page_reuse: 0
>> rx16_cache_reuse: 25912567
>> rx16_cache_full: 8026124051
>> rx16_cache_empty: 37376
>> rx16_cache_busy: 8026126465
>> rx16_cache_waive: 0
>> rx16_congst_umr: 0
>> rx16_arfs_err: 0
>> rx16_xdp_tx_xmit: 0
>> rx16_xdp_tx_full: 0
>> rx16_xdp_tx_err: 0
>> rx16_xdp_tx_cqes: 0
>> rx17_packets: 16314055017
>> rx17_bytes: 21589139030173
>> rx17_csum_complete: 16314055017
>> rx17_csum_unnecessary: 0
>> rx17_csum_unnecessary_inner: 0
>> rx17_csum_none: 0
>> rx17_xdp_drop: 0
>> rx17_xdp_redirect: 0
>> rx17_lro_packets: 0
>> rx17_lro_bytes: 0
>> rx17_ecn_mark: 0
>> rx17_removed_vlan_packets: 16314055017
>> rx17_wqe_err: 0
>> rx17_mpwqe_filler_cqes: 0
>> rx17_mpwqe_filler_strides: 0
>> rx17_buff_alloc_err: 0
>> rx17_cqe_compress_blks: 387834541
>> rx17_cqe_compress_pkts: 871851081
>> rx17_page_reuse: 0
>> rx17_cache_reuse: 24021313
>> rx17_cache_full: 8133003829
>> rx17_cache_empty: 37376
>> rx17_cache_busy: 8133006175
>> rx17_cache_waive: 0
>> rx17_congst_umr: 0
>> rx17_arfs_err: 0
>> rx17_xdp_tx_xmit: 0
>> rx17_xdp_tx_full: 0
>> rx17_xdp_tx_err: 0
>> rx17_xdp_tx_cqes: 0
>> rx18_packets: 16439016814
>> rx18_bytes: 21648651917475
>> rx18_csum_complete: 16439016814
>> rx18_csum_unnecessary: 0
>> rx18_csum_unnecessary_inner: 0
>> rx18_csum_none: 0
>> rx18_xdp_drop: 0
>> rx18_xdp_redirect: 0
>> rx18_lro_packets: 0
>> rx18_lro_bytes: 0
>> rx18_ecn_mark: 0
>> rx18_removed_vlan_packets: 16439016814
>> rx18_wqe_err: 0
>> rx18_mpwqe_filler_cqes: 0
>> rx18_mpwqe_filler_strides: 0
>> rx18_buff_alloc_err: 0
>> rx18_cqe_compress_blks: 375066666
>> rx18_cqe_compress_pkts: 843563974
>> rx18_page_reuse: 0
>> rx18_cache_reuse: 8151064266
>> rx18_cache_full: 68442025
>> rx18_cache_empty: 37376
>> rx18_cache_busy: 68444122
>> rx18_cache_waive: 0
>> rx18_congst_umr: 0
>> rx18_arfs_err: 0
>> rx18_xdp_tx_xmit: 0
>> rx18_xdp_tx_full: 0
>> rx18_xdp_tx_err: 0
>> rx18_xdp_tx_cqes: 0
>> rx19_packets: 16641223506
>> rx19_bytes: 21964749940935
>> rx19_csum_complete: 16641223506
>> rx19_csum_unnecessary: 0
>> rx19_csum_unnecessary_inner: 0
>> rx19_csum_none: 0
>> rx19_xdp_drop: 0
>> rx19_xdp_redirect: 0
>> rx19_lro_packets: 0
>> rx19_lro_bytes: 0
>> rx19_ecn_mark: 0
>> rx19_removed_vlan_packets: 16641223506
>> rx19_wqe_err: 0
>> rx19_mpwqe_filler_cqes: 0
>> rx19_mpwqe_filler_strides: 0
>> rx19_buff_alloc_err: 0
>> rx19_cqe_compress_blks: 387825932
>> rx19_cqe_compress_pkts: 872266355
>> rx19_page_reuse: 0
>> rx19_cache_reuse: 116433620
>> rx19_cache_full: 8204175954
>> rx19_cache_empty: 37376
>> rx19_cache_busy: 8204178120
>> rx19_cache_waive: 0
>> rx19_congst_umr: 0
>> rx19_arfs_err: 0
>> rx19_xdp_tx_xmit: 0
>> rx19_xdp_tx_full: 0
>> rx19_xdp_tx_err: 0
>> rx19_xdp_tx_cqes: 0
>> rx20_packets: 16206927741
>> rx20_bytes: 21387447038430
>> rx20_csum_complete: 16206927741
>> rx20_csum_unnecessary: 0
>> rx20_csum_unnecessary_inner: 0
>> rx20_csum_none: 0
>> rx20_xdp_drop: 0
>> rx20_xdp_redirect: 0
>> rx20_lro_packets: 0
>> rx20_lro_bytes: 0
>> rx20_ecn_mark: 0
>> rx20_removed_vlan_packets: 16206927741
>> rx20_wqe_err: 0
>> rx20_mpwqe_filler_cqes: 0
>> rx20_mpwqe_filler_strides: 0
>> rx20_buff_alloc_err: 0
>> rx20_cqe_compress_blks: 370144620
>> rx20_cqe_compress_pkts: 829122671
>> rx20_page_reuse: 0
>> rx20_cache_reuse: 8053733744
>> rx20_cache_full: 49728026
>> rx20_cache_empty: 37376
>> rx20_cache_busy: 49730116
>> rx20_cache_waive: 0
>> rx20_congst_umr: 0
>> rx20_arfs_err: 0
>> rx20_xdp_tx_xmit: 0
>> rx20_xdp_tx_full: 0
>> rx20_xdp_tx_err: 0
>> rx20_xdp_tx_cqes: 0
>> rx21_packets: 16562361314
>> rx21_bytes: 21856653284356
>> rx21_csum_complete: 16562361314
>> rx21_csum_unnecessary: 0
>> rx21_csum_unnecessary_inner: 0
>> rx21_csum_none: 0
>> rx21_xdp_drop: 0
>> rx21_xdp_redirect: 0
>> rx21_lro_packets: 0
>> rx21_lro_bytes: 0
>> rx21_ecn_mark: 0
>> rx21_removed_vlan_packets: 16562361314
>> rx21_wqe_err: 0
>> rx21_mpwqe_filler_cqes: 0
>> rx21_mpwqe_filler_strides: 0
>> rx21_buff_alloc_err: 0
>> rx21_cqe_compress_blks: 350790425
>> rx21_cqe_compress_pkts: 783850729
>> rx21_page_reuse: 0
>> rx21_cache_reuse: 28077493
>> rx21_cache_full: 8253100706
>> rx21_cache_empty: 37376
>> rx21_cache_busy: 8253103147
>> rx21_cache_waive: 0
>> rx21_congst_umr: 0
>> rx21_arfs_err: 0
>> rx21_xdp_tx_xmit: 0
>> rx21_xdp_tx_full: 0
>> rx21_xdp_tx_err: 0
>> rx21_xdp_tx_cqes: 0
>> rx22_packets: 16350307571
>> rx22_bytes: 21408575325592
>> rx22_csum_complete: 16350307571
>> rx22_csum_unnecessary: 0
>> rx22_csum_unnecessary_inner: 0
>> rx22_csum_none: 0
>> rx22_xdp_drop: 0
>> rx22_xdp_redirect: 0
>> rx22_lro_packets: 0
>> rx22_lro_bytes: 0
>> rx22_ecn_mark: 0
>> rx22_removed_vlan_packets: 16350307571
>> rx22_wqe_err: 0
>> rx22_mpwqe_filler_cqes: 0
>> rx22_mpwqe_filler_strides: 0
>> rx22_buff_alloc_err: 0
>> rx22_cqe_compress_blks: 353531065
>> rx22_cqe_compress_pkts: 790814415
>> rx22_page_reuse: 0
>> rx22_cache_reuse: 16934343
>> rx22_cache_full: 8158216889
>> rx22_cache_empty: 37376
>> rx22_cache_busy: 8158219417
>> rx22_cache_waive: 0
>> rx22_congst_umr: 0
>> rx22_arfs_err: 0
>> rx22_xdp_tx_xmit: 0
>> rx22_xdp_tx_full: 0
>> rx22_xdp_tx_err: 0
>> rx22_xdp_tx_cqes: 0
>> rx23_packets: 16019811764
>> rx23_bytes: 21137182570985
>> rx23_csum_complete: 16019811764
>> rx23_csum_unnecessary: 0
>> rx23_csum_unnecessary_inner: 0
>> rx23_csum_none: 0
>> rx23_xdp_drop: 0
>> rx23_xdp_redirect: 0
>> rx23_lro_packets: 0
>> rx23_lro_bytes: 0
>> rx23_ecn_mark: 0
>> rx23_removed_vlan_packets: 16019811764
>> rx23_wqe_err: 0
>> rx23_mpwqe_filler_cqes: 0
>> rx23_mpwqe_filler_strides: 0
>> rx23_buff_alloc_err: 0
>> rx23_cqe_compress_blks: 349733033
>> rx23_cqe_compress_pkts: 781248862
>> rx23_page_reuse: 0
>> rx23_cache_reuse: 33422343
>> rx23_cache_full: 7976481152
>> rx23_cache_empty: 37376
>> rx23_cache_busy: 7976483525
>> rx23_cache_waive: 0
>> rx23_congst_umr: 0
>> rx23_arfs_err: 0
>> rx23_xdp_tx_xmit: 0
>> rx23_xdp_tx_full: 0
>> rx23_xdp_tx_err: 0
>> rx23_xdp_tx_cqes: 0
>> rx24_packets: 16212040646
>> rx24_bytes: 21393399325700
>> rx24_csum_complete: 16212040646
>> rx24_csum_unnecessary: 0
>> rx24_csum_unnecessary_inner: 0
>> rx24_csum_none: 0
>> rx24_xdp_drop: 0
>> rx24_xdp_redirect: 0
>> rx24_lro_packets: 0
>> rx24_lro_bytes: 0
>> rx24_ecn_mark: 0
>> rx24_removed_vlan_packets: 16212040646
>> rx24_wqe_err: 0
>> rx24_mpwqe_filler_cqes: 0
>> rx24_mpwqe_filler_strides: 0
>> rx24_buff_alloc_err: 0
>> rx24_cqe_compress_blks: 379833752
>> rx24_cqe_compress_pkts: 852020179
>> rx24_page_reuse: 0
>> rx24_cache_reuse: 8033552512
>> rx24_cache_full: 72465843
>> rx24_cache_empty: 37376
>> rx24_cache_busy: 72467789
>> rx24_cache_waive: 0
>> rx24_congst_umr: 0
>> rx24_arfs_err: 0
>> rx24_xdp_tx_xmit: 0
>> rx24_xdp_tx_full: 0
>> rx24_xdp_tx_err: 0
>> rx24_xdp_tx_cqes: 0
>> rx25_packets: 16412186257
>> rx25_bytes: 21651198388407
>> rx25_csum_complete: 16412186257
>> rx25_csum_unnecessary: 0
>> rx25_csum_unnecessary_inner: 0
>> rx25_csum_none: 0
>> rx25_xdp_drop: 0
>> rx25_xdp_redirect: 0
>> rx25_lro_packets: 0
>> rx25_lro_bytes: 0
>> rx25_ecn_mark: 0
>> rx25_removed_vlan_packets: 16412186257
>> rx25_wqe_err: 0
>> rx25_mpwqe_filler_cqes: 0
>> rx25_mpwqe_filler_strides: 0
>> rx25_buff_alloc_err: 0
>> rx25_cqe_compress_blks: 383979685
>> rx25_cqe_compress_pkts: 861985772
>> rx25_page_reuse: 0
>> rx25_cache_reuse: 8129807841
>> rx25_cache_full: 76283342
>> rx25_cache_empty: 37376
>> rx25_cache_busy: 76285271
>> rx25_cache_waive: 0
>> rx25_congst_umr: 0
>> rx25_arfs_err: 0
>> rx25_xdp_tx_xmit: 0
>> rx25_xdp_tx_full: 0
>> rx25_xdp_tx_err: 0
>> rx25_xdp_tx_cqes: 0
>> rx26_packets: 16304310003
>> rx26_bytes: 21571217538721
>> rx26_csum_complete: 16304310003
>> rx26_csum_unnecessary: 0
>> rx26_csum_unnecessary_inner: 0
>> rx26_csum_none: 0
>> rx26_xdp_drop: 0
>> rx26_xdp_redirect: 0
>> rx26_lro_packets: 0
>> rx26_lro_bytes: 0
>> rx26_ecn_mark: 0
>> rx26_removed_vlan_packets: 16304310003
>> rx26_wqe_err: 0
>> rx26_mpwqe_filler_cqes: 0
>> rx26_mpwqe_filler_strides: 0
>> rx26_buff_alloc_err: 0
>> rx26_cqe_compress_blks: 353314041
>> rx26_cqe_compress_pkts: 788838424
>> rx26_page_reuse: 0
>> rx26_cache_reuse: 19673790
>> rx26_cache_full: 8132478659
>> rx26_cache_empty: 37376
>> rx26_cache_busy: 8132481198
>> rx26_cache_waive: 0
>> rx26_congst_umr: 0
>> rx26_arfs_err: 0
>> rx26_xdp_tx_xmit: 0
>> rx26_xdp_tx_full: 0
>> rx26_xdp_tx_err: 0
>> rx26_xdp_tx_cqes: 0
>> rx27_packets: 16171856079
>> rx27_bytes: 21376891736540
>> rx27_csum_complete: 16171856079
>> rx27_csum_unnecessary: 0
>> rx27_csum_unnecessary_inner: 0
>> rx27_csum_none: 0
>> rx27_xdp_drop: 0
>> rx27_xdp_redirect: 0
>> rx27_lro_packets: 0
>> rx27_lro_bytes: 0
>> rx27_ecn_mark: 0
>> rx27_removed_vlan_packets: 16171856079
>> rx27_wqe_err: 0
>> rx27_mpwqe_filler_cqes: 0
>> rx27_mpwqe_filler_strides: 0
>> rx27_buff_alloc_err: 0
>> rx27_cqe_compress_blks: 386632845
>> rx27_cqe_compress_pkts: 869362576
>> rx27_page_reuse: 0
>> rx27_cache_reuse: 10070560
>> rx27_cache_full: 8075854928
>> rx27_cache_empty: 37376
>> rx27_cache_busy: 8075857468
>> rx27_cache_waive: 0
>> rx27_congst_umr: 0
>> rx27_arfs_err: 0
>> rx27_xdp_tx_xmit: 0
>> rx27_xdp_tx_full: 0
>> rx27_xdp_tx_err: 0
>> rx27_xdp_tx_cqes: 0
>> rx28_packets: 0
>> rx28_bytes: 0
>> rx28_csum_complete: 0
>> rx28_csum_unnecessary: 0
>> rx28_csum_unnecessary_inner: 0
>> rx28_csum_none: 0
>> rx28_xdp_drop: 0
>> rx28_xdp_redirect: 0
>> rx28_lro_packets: 0
>> rx28_lro_bytes: 0
>> rx28_ecn_mark: 0
>> rx28_removed_vlan_packets: 0
>> rx28_wqe_err: 0
>> rx28_mpwqe_filler_cqes: 0
>> rx28_mpwqe_filler_strides: 0
>> rx28_buff_alloc_err: 0
>> rx28_cqe_compress_blks: 0
>> rx28_cqe_compress_pkts: 0
>> rx28_page_reuse: 0
>> rx28_cache_reuse: 0
>> rx28_cache_full: 0
>> rx28_cache_empty: 2560
>> rx28_cache_busy: 0
>> rx28_cache_waive: 0
>> rx28_congst_umr: 0
>> rx28_arfs_err: 0
>> rx28_xdp_tx_xmit: 0
>> rx28_xdp_tx_full: 0
>> rx28_xdp_tx_err: 0
>> rx28_xdp_tx_cqes: 0
>> rx29_packets: 0
>> rx29_bytes: 0
>> rx29_csum_complete: 0
>> rx29_csum_unnecessary: 0
>> rx29_csum_unnecessary_inner: 0
>> rx29_csum_none: 0
>> rx29_xdp_drop: 0
>> rx29_xdp_redirect: 0
>> rx29_lro_packets: 0
>> rx29_lro_bytes: 0
>> rx29_ecn_mark: 0
>> rx29_removed_vlan_packets: 0
>> rx29_wqe_err: 0
>> rx29_mpwqe_filler_cqes: 0
>> rx29_mpwqe_filler_strides: 0
>> rx29_buff_alloc_err: 0
>> rx29_cqe_compress_blks: 0
>> rx29_cqe_compress_pkts: 0
>> rx29_page_reuse: 0
>> rx29_cache_reuse: 0
>> rx29_cache_full: 0
>> rx29_cache_empty: 2560
>> rx29_cache_busy: 0
>> rx29_cache_waive: 0
>> rx29_congst_umr: 0
>> rx29_arfs_err: 0
>> rx29_xdp_tx_xmit: 0
>> rx29_xdp_tx_full: 0
>> rx29_xdp_tx_err: 0
>> rx29_xdp_tx_cqes: 0
>> rx30_packets: 0
>> rx30_bytes: 0
>> rx30_csum_complete: 0
>> rx30_csum_unnecessary: 0
>> rx30_csum_unnecessary_inner: 0
>> rx30_csum_none: 0
>> rx30_xdp_drop: 0
>> rx30_xdp_redirect: 0
>> rx30_lro_packets: 0
>> rx30_lro_bytes: 0
>> rx30_ecn_mark: 0
>> rx30_removed_vlan_packets: 0
>> rx30_wqe_err: 0
>> rx30_mpwqe_filler_cqes: 0
>> rx30_mpwqe_filler_strides: 0
>> rx30_buff_alloc_err: 0
>> rx30_cqe_compress_blks: 0
>> rx30_cqe_compress_pkts: 0
>> rx30_page_reuse: 0
>> rx30_cache_reuse: 0
>> rx30_cache_full: 0
>> rx30_cache_empty: 2560
>> rx30_cache_busy: 0
>> rx30_cache_waive: 0
>> rx30_congst_umr: 0
>> rx30_arfs_err: 0
>> rx30_xdp_tx_xmit: 0
>> rx30_xdp_tx_full: 0
>> rx30_xdp_tx_err: 0
>> rx30_xdp_tx_cqes: 0
>> rx31_packets: 0
>> rx31_bytes: 0
>> rx31_csum_complete: 0
>> rx31_csum_unnecessary: 0
>> rx31_csum_unnecessary_inner: 0
>> rx31_csum_none: 0
>> rx31_xdp_drop: 0
>> rx31_xdp_redirect: 0
>> rx31_lro_packets: 0
>> rx31_lro_bytes: 0
>> rx31_ecn_mark: 0
>> rx31_removed_vlan_packets: 0
>> rx31_wqe_err: 0
>> rx31_mpwqe_filler_cqes: 0
>> rx31_mpwqe_filler_strides: 0
>> rx31_buff_alloc_err: 0
>> rx31_cqe_compress_blks: 0
>> rx31_cqe_compress_pkts: 0
>> rx31_page_reuse: 0
>> rx31_cache_reuse: 0
>> rx31_cache_full: 0
>> rx31_cache_empty: 2560
>> rx31_cache_busy: 0
>> rx31_cache_waive: 0
>> rx31_congst_umr: 0
>> rx31_arfs_err: 0
>> rx31_xdp_tx_xmit: 0
>> rx31_xdp_tx_full: 0
>> rx31_xdp_tx_err: 0
>> rx31_xdp_tx_cqes: 0
>> rx32_packets: 0
>> rx32_bytes: 0
>> rx32_csum_complete: 0
>> rx32_csum_unnecessary: 0
>> rx32_csum_unnecessary_inner: 0
>> rx32_csum_none: 0
>> rx32_xdp_drop: 0
>> rx32_xdp_redirect: 0
>> rx32_lro_packets: 0
>> rx32_lro_bytes: 0
>> rx32_ecn_mark: 0
>> rx32_removed_vlan_packets: 0
>> rx32_wqe_err: 0
>> rx32_mpwqe_filler_cqes: 0
>> rx32_mpwqe_filler_strides: 0
>> rx32_buff_alloc_err: 0
>> rx32_cqe_compress_blks: 0
>> rx32_cqe_compress_pkts: 0
>> rx32_page_reuse: 0
>> rx32_cache_reuse: 0
>> rx32_cache_full: 0
>> rx32_cache_empty: 2560
>> rx32_cache_busy: 0
>> rx32_cache_waive: 0
>> rx32_congst_umr: 0
>> rx32_arfs_err: 0
>> rx32_xdp_tx_xmit: 0
>> rx32_xdp_tx_full: 0
>> rx32_xdp_tx_err: 0
>> rx32_xdp_tx_cqes: 0
>> rx33_packets: 0
>> rx33_bytes: 0
>> rx33_csum_complete: 0
>> rx33_csum_unnecessary: 0
>> rx33_csum_unnecessary_inner: 0
>> rx33_csum_none: 0
>> rx33_xdp_drop: 0
>> rx33_xdp_redirect: 0
>> rx33_lro_packets: 0
>> rx33_lro_bytes: 0
>> rx33_ecn_mark: 0
>> rx33_removed_vlan_packets: 0
>> rx33_wqe_err: 0
>> rx33_mpwqe_filler_cqes: 0
>> rx33_mpwqe_filler_strides: 0
>> rx33_buff_alloc_err: 0
>> rx33_cqe_compress_blks: 0
>> rx33_cqe_compress_pkts: 0
>> rx33_page_reuse: 0
>> rx33_cache_reuse: 0
>> rx33_cache_full: 0
>> rx33_cache_empty: 2560
>> rx33_cache_busy: 0
>> rx33_cache_waive: 0
>> rx33_congst_umr: 0
>> rx33_arfs_err: 0
>> rx33_xdp_tx_xmit: 0
>> rx33_xdp_tx_full: 0
>> rx33_xdp_tx_err: 0
>> rx33_xdp_tx_cqes: 0
>> rx34_packets: 0
>> rx34_bytes: 0
>> rx34_csum_complete: 0
>> rx34_csum_unnecessary: 0
>> rx34_csum_unnecessary_inner: 0
>> rx34_csum_none: 0
>> rx34_xdp_drop: 0
>> rx34_xdp_redirect: 0
>> rx34_lro_packets: 0
>> rx34_lro_bytes: 0
>> rx34_ecn_mark: 0
>> rx34_removed_vlan_packets: 0
>> rx34_wqe_err: 0
>> rx34_mpwqe_filler_cqes: 0
>> rx34_mpwqe_filler_strides: 0
>> rx34_buff_alloc_err: 0
>> rx34_cqe_compress_blks: 0
>> rx34_cqe_compress_pkts: 0
>> rx34_page_reuse: 0
>> rx34_cache_reuse: 0
>> rx34_cache_full: 0
>> rx34_cache_empty: 2560
>> rx34_cache_busy: 0
>> rx34_cache_waive: 0
>> rx34_congst_umr: 0
>> rx34_arfs_err: 0
>> rx34_xdp_tx_xmit: 0
>> rx34_xdp_tx_full: 0
>> rx34_xdp_tx_err: 0
>> rx34_xdp_tx_cqes: 0
>> rx35_packets: 0
>> rx35_bytes: 0
>> rx35_csum_complete: 0
>> rx35_csum_unnecessary: 0
>> rx35_csum_unnecessary_inner: 0
>> rx35_csum_none: 0
>> rx35_xdp_drop: 0
>> rx35_xdp_redirect: 0
>> rx35_lro_packets: 0
>> rx35_lro_bytes: 0
>> rx35_ecn_mark: 0
>> rx35_removed_vlan_packets: 0
>> rx35_wqe_err: 0
>> rx35_mpwqe_filler_cqes: 0
>> rx35_mpwqe_filler_strides: 0
>> rx35_buff_alloc_err: 0
>> rx35_cqe_compress_blks: 0
>> rx35_cqe_compress_pkts: 0
>> rx35_page_reuse: 0
>> rx35_cache_reuse: 0
>> rx35_cache_full: 0
>> rx35_cache_empty: 2560
>> rx35_cache_busy: 0
>> rx35_cache_waive: 0
>> rx35_congst_umr: 0
>> rx35_arfs_err: 0
>> rx35_xdp_tx_xmit: 0
>> rx35_xdp_tx_full: 0
>> rx35_xdp_tx_err: 0
>> rx35_xdp_tx_cqes: 0
>> rx36_packets: 0
>> rx36_bytes: 0
>> rx36_csum_complete: 0
>> rx36_csum_unnecessary: 0
>> rx36_csum_unnecessary_inner: 0
>> rx36_csum_none: 0
>> rx36_xdp_drop: 0
>> rx36_xdp_redirect: 0
>> rx36_lro_packets: 0
>> rx36_lro_bytes: 0
>> rx36_ecn_mark: 0
>> rx36_removed_vlan_packets: 0
>> rx36_wqe_err: 0
>> rx36_mpwqe_filler_cqes: 0
>> rx36_mpwqe_filler_strides: 0
>> rx36_buff_alloc_err: 0
>> rx36_cqe_compress_blks: 0
>> rx36_cqe_compress_pkts: 0
>> rx36_page_reuse: 0
>> rx36_cache_reuse: 0
>> rx36_cache_full: 0
>> rx36_cache_empty: 2560
>> rx36_cache_busy: 0
>> rx36_cache_waive: 0
>> rx36_congst_umr: 0
>> rx36_arfs_err: 0
>> rx36_xdp_tx_xmit: 0
>> rx36_xdp_tx_full: 0
>> rx36_xdp_tx_err: 0
>> rx36_xdp_tx_cqes: 0
>> rx37_packets: 0
>> rx37_bytes: 0
>> rx37_csum_complete: 0
>> rx37_csum_unnecessary: 0
>> rx37_csum_unnecessary_inner: 0
>> rx37_csum_none: 0
>> rx37_xdp_drop: 0
>> rx37_xdp_redirect: 0
>> rx37_lro_packets: 0
>> rx37_lro_bytes: 0
>> rx37_ecn_mark: 0
>> rx37_removed_vlan_packets: 0
>> rx37_wqe_err: 0
>> rx37_mpwqe_filler_cqes: 0
>> rx37_mpwqe_filler_strides: 0
>> rx37_buff_alloc_err: 0
>> rx37_cqe_compress_blks: 0
>> rx37_cqe_compress_pkts: 0
>> rx37_page_reuse: 0
>> rx37_cache_reuse: 0
>> rx37_cache_full: 0
>> rx37_cache_empty: 2560
>> rx37_cache_busy: 0
>> rx37_cache_waive: 0
>> rx37_congst_umr: 0
>> rx37_arfs_err: 0
>> rx37_xdp_tx_xmit: 0
>> rx37_xdp_tx_full: 0
>> rx37_xdp_tx_err: 0
>> rx37_xdp_tx_cqes: 0
>> rx38_packets: 0
>> rx38_bytes: 0
>> rx38_csum_complete: 0
>> rx38_csum_unnecessary: 0
>> rx38_csum_unnecessary_inner: 0
>> rx38_csum_none: 0
>> rx38_xdp_drop: 0
>> rx38_xdp_redirect: 0
>> rx38_lro_packets: 0
>> rx38_lro_bytes: 0
>> rx38_ecn_mark: 0
>> rx38_removed_vlan_packets: 0
>> rx38_wqe_err: 0
>> rx38_mpwqe_filler_cqes: 0
>> rx38_mpwqe_filler_strides: 0
>> rx38_buff_alloc_err: 0
>> rx38_cqe_compress_blks: 0
>> rx38_cqe_compress_pkts: 0
>> rx38_page_reuse: 0
>> rx38_cache_reuse: 0
>> rx38_cache_full: 0
>> rx38_cache_empty: 2560
>> rx38_cache_busy: 0
>> rx38_cache_waive: 0
>> rx38_congst_umr: 0
>> rx38_arfs_err: 0
>> rx38_xdp_tx_xmit: 0
>> rx38_xdp_tx_full: 0
>> rx38_xdp_tx_err: 0
>> rx38_xdp_tx_cqes: 0
>> rx39_packets: 0
>> rx39_bytes: 0
>> rx39_csum_complete: 0
>> rx39_csum_unnecessary: 0
>> rx39_csum_unnecessary_inner: 0
>> rx39_csum_none: 0
>> rx39_xdp_drop: 0
>> rx39_xdp_redirect: 0
>> rx39_lro_packets: 0
>> rx39_lro_bytes: 0
>> rx39_ecn_mark: 0
>> rx39_removed_vlan_packets: 0
>> rx39_wqe_err: 0
>> rx39_mpwqe_filler_cqes: 0
>> rx39_mpwqe_filler_strides: 0
>> rx39_buff_alloc_err: 0
>> rx39_cqe_compress_blks: 0
>> rx39_cqe_compress_pkts: 0
>> rx39_page_reuse: 0
>> rx39_cache_reuse: 0
>> rx39_cache_full: 0
>> rx39_cache_empty: 2560
>> rx39_cache_busy: 0
>> rx39_cache_waive: 0
>> rx39_congst_umr: 0
>> rx39_arfs_err: 0
>> rx39_xdp_tx_xmit: 0
>> rx39_xdp_tx_full: 0
>> rx39_xdp_tx_err: 0
>> rx39_xdp_tx_cqes: 0
>> rx40_packets: 0
>> rx40_bytes: 0
>> rx40_csum_complete: 0
>> rx40_csum_unnecessary: 0
>> rx40_csum_unnecessary_inner: 0
>> rx40_csum_none: 0
>> rx40_xdp_drop: 0
>> rx40_xdp_redirect: 0
>> rx40_lro_packets: 0
>> rx40_lro_bytes: 0
>> rx40_ecn_mark: 0
>> rx40_removed_vlan_packets: 0
>> rx40_wqe_err: 0
>> rx40_mpwqe_filler_cqes: 0
>> rx40_mpwqe_filler_strides: 0
>> rx40_buff_alloc_err: 0
>> rx40_cqe_compress_blks: 0
>> rx40_cqe_compress_pkts: 0
>> rx40_page_reuse: 0
>> rx40_cache_reuse: 0
>> rx40_cache_full: 0
>> rx40_cache_empty: 2560
>> rx40_cache_busy: 0
>> rx40_cache_waive: 0
>> rx40_congst_umr: 0
>> rx40_arfs_err: 0
>> rx40_xdp_tx_xmit: 0
>> rx40_xdp_tx_full: 0
>> rx40_xdp_tx_err: 0
>> rx40_xdp_tx_cqes: 0
>> rx41_packets: 0
>> rx41_bytes: 0
>> rx41_csum_complete: 0
>> rx41_csum_unnecessary: 0
>> rx41_csum_unnecessary_inner: 0
>> rx41_csum_none: 0
>> rx41_xdp_drop: 0
>> rx41_xdp_redirect: 0
>> rx41_lro_packets: 0
>> rx41_lro_bytes: 0
>> rx41_ecn_mark: 0
>> rx41_removed_vlan_packets: 0
>> rx41_wqe_err: 0
>> rx41_mpwqe_filler_cqes: 0
>> rx41_mpwqe_filler_strides: 0
>> rx41_buff_alloc_err: 0
>> rx41_cqe_compress_blks: 0
>> rx41_cqe_compress_pkts: 0
>> rx41_page_reuse: 0
>> rx41_cache_reuse: 0
>> rx41_cache_full: 0
>> rx41_cache_empty: 2560
>> rx41_cache_busy: 0
>> rx41_cache_waive: 0
>> rx41_congst_umr: 0
>> rx41_arfs_err: 0
>> rx41_xdp_tx_xmit: 0
>> rx41_xdp_tx_full: 0
>> rx41_xdp_tx_err: 0
>> rx41_xdp_tx_cqes: 0
>> rx42_packets: 0
>> rx42_bytes: 0
>> rx42_csum_complete: 0
>> rx42_csum_unnecessary: 0
>> rx42_csum_unnecessary_inner: 0
>> rx42_csum_none: 0
>> rx42_xdp_drop: 0
>> rx42_xdp_redirect: 0
>> rx42_lro_packets: 0
>> rx42_lro_bytes: 0
>> rx42_ecn_mark: 0
>> rx42_removed_vlan_packets: 0
>> rx42_wqe_err: 0
>> rx42_mpwqe_filler_cqes: 0
>> rx42_mpwqe_filler_strides: 0
>> rx42_buff_alloc_err: 0
>> rx42_cqe_compress_blks: 0
>> rx42_cqe_compress_pkts: 0
>> rx42_page_reuse: 0
>> rx42_cache_reuse: 0
>> rx42_cache_full: 0
>> rx42_cache_empty: 2560
>> rx42_cache_busy: 0
>> rx42_cache_waive: 0
>> rx42_congst_umr: 0
>> rx42_arfs_err: 0
>> rx42_xdp_tx_xmit: 0
>> rx42_xdp_tx_full: 0
>> rx42_xdp_tx_err: 0
>> rx42_xdp_tx_cqes: 0
>> rx43_packets: 0
>> rx43_bytes: 0
>> rx43_csum_complete: 0
>> rx43_csum_unnecessary: 0
>> rx43_csum_unnecessary_inner: 0
>> rx43_csum_none: 0
>> rx43_xdp_drop: 0
>> rx43_xdp_redirect: 0
>> rx43_lro_packets: 0
>> rx43_lro_bytes: 0
>> rx43_ecn_mark: 0
>> rx43_removed_vlan_packets: 0
>> rx43_wqe_err: 0
>> rx43_mpwqe_filler_cqes: 0
>> rx43_mpwqe_filler_strides: 0
>> rx43_buff_alloc_err: 0
>> rx43_cqe_compress_blks: 0
>> rx43_cqe_compress_pkts: 0
>> rx43_page_reuse: 0
>> rx43_cache_reuse: 0
>> rx43_cache_full: 0
>> rx43_cache_empty: 2560
>> rx43_cache_busy: 0
>> rx43_cache_waive: 0
>> rx43_congst_umr: 0
>> rx43_arfs_err: 0
>> rx43_xdp_tx_xmit: 0
>> rx43_xdp_tx_full: 0
>> rx43_xdp_tx_err: 0
>> rx43_xdp_tx_cqes: 0
>> rx44_packets: 0
>> rx44_bytes: 0
>> rx44_csum_complete: 0
>> rx44_csum_unnecessary: 0
>> rx44_csum_unnecessary_inner: 0
>> rx44_csum_none: 0
>> rx44_xdp_drop: 0
>> rx44_xdp_redirect: 0
>> rx44_lro_packets: 0
>> rx44_lro_bytes: 0
>> rx44_ecn_mark: 0
>> rx44_removed_vlan_packets: 0
>> rx44_wqe_err: 0
>> rx44_mpwqe_filler_cqes: 0
>> rx44_mpwqe_filler_strides: 0
>> rx44_buff_alloc_err: 0
>> rx44_cqe_compress_blks: 0
>> rx44_cqe_compress_pkts: 0
>> rx44_page_reuse: 0
>> rx44_cache_reuse: 0
>> rx44_cache_full: 0
>> rx44_cache_empty: 2560
>> rx44_cache_busy: 0
>> rx44_cache_waive: 0
>> rx44_congst_umr: 0
>> rx44_arfs_err: 0
>> rx44_xdp_tx_xmit: 0
>> rx44_xdp_tx_full: 0
>> rx44_xdp_tx_err: 0
>> rx44_xdp_tx_cqes: 0
>> rx45_packets: 0
>> rx45_bytes: 0
>> rx45_csum_complete: 0
>> rx45_csum_unnecessary: 0
>> rx45_csum_unnecessary_inner: 0
>> rx45_csum_none: 0
>> rx45_xdp_drop: 0
>> rx45_xdp_redirect: 0
>> rx45_lro_packets: 0
>> rx45_lro_bytes: 0
>> rx45_ecn_mark: 0
>> rx45_removed_vlan_packets: 0
>> rx45_wqe_err: 0
>> rx45_mpwqe_filler_cqes: 0
>> rx45_mpwqe_filler_strides: 0
>> rx45_buff_alloc_err: 0
>> rx45_cqe_compress_blks: 0
>> rx45_cqe_compress_pkts: 0
>> rx45_page_reuse: 0
>> rx45_cache_reuse: 0
>> rx45_cache_full: 0
>> rx45_cache_empty: 2560
>> rx45_cache_busy: 0
>> rx45_cache_waive: 0
>> rx45_congst_umr: 0
>> rx45_arfs_err: 0
>> rx45_xdp_tx_xmit: 0
>> rx45_xdp_tx_full: 0
>> rx45_xdp_tx_err: 0
>> rx45_xdp_tx_cqes: 0
>> rx46_packets: 0
>> rx46_bytes: 0
>> rx46_csum_complete: 0
>> rx46_csum_unnecessary: 0
>> rx46_csum_unnecessary_inner: 0
>> rx46_csum_none: 0
>> rx46_xdp_drop: 0
>> rx46_xdp_redirect: 0
>> rx46_lro_packets: 0
>> rx46_lro_bytes: 0
>> rx46_ecn_mark: 0
>> rx46_removed_vlan_packets: 0
>> rx46_wqe_err: 0
>> rx46_mpwqe_filler_cqes: 0
>> rx46_mpwqe_filler_strides: 0
>> rx46_buff_alloc_err: 0
>> rx46_cqe_compress_blks: 0
>> rx46_cqe_compress_pkts: 0
>> rx46_page_reuse: 0
>> rx46_cache_reuse: 0
>> rx46_cache_full: 0
>> rx46_cache_empty: 2560
>> rx46_cache_busy: 0
>> rx46_cache_waive: 0
>> rx46_congst_umr: 0
>> rx46_arfs_err: 0
>> rx46_xdp_tx_xmit: 0
>> rx46_xdp_tx_full: 0
>> rx46_xdp_tx_err: 0
>> rx46_xdp_tx_cqes: 0
>> rx47_packets: 0
>> rx47_bytes: 0
>> rx47_csum_complete: 0
>> rx47_csum_unnecessary: 0
>> rx47_csum_unnecessary_inner: 0
>> rx47_csum_none: 0
>> rx47_xdp_drop: 0
>> rx47_xdp_redirect: 0
>> rx47_lro_packets: 0
>> rx47_lro_bytes: 0
>> rx47_ecn_mark: 0
>> rx47_removed_vlan_packets: 0
>> rx47_wqe_err: 0
>> rx47_mpwqe_filler_cqes: 0
>> rx47_mpwqe_filler_strides: 0
>> rx47_buff_alloc_err: 0
>> rx47_cqe_compress_blks: 0
>> rx47_cqe_compress_pkts: 0
>> rx47_page_reuse: 0
>> rx47_cache_reuse: 0
>> rx47_cache_full: 0
>> rx47_cache_empty: 2560
>> rx47_cache_busy: 0
>> rx47_cache_waive: 0
>> rx47_congst_umr: 0
>> rx47_arfs_err: 0
>> rx47_xdp_tx_xmit: 0
>> rx47_xdp_tx_full: 0
>> rx47_xdp_tx_err: 0
>> rx47_xdp_tx_cqes: 0
>> rx48_packets: 0
>> rx48_bytes: 0
>> rx48_csum_complete: 0
>> rx48_csum_unnecessary: 0
>> rx48_csum_unnecessary_inner: 0
>> rx48_csum_none: 0
>> rx48_xdp_drop: 0
>> rx48_xdp_redirect: 0
>> rx48_lro_packets: 0
>> rx48_lro_bytes: 0
>> rx48_ecn_mark: 0
>> rx48_removed_vlan_packets: 0
>> rx48_wqe_err: 0
>> rx48_mpwqe_filler_cqes: 0
>> rx48_mpwqe_filler_strides: 0
>> rx48_buff_alloc_err: 0
>> rx48_cqe_compress_blks: 0
>> rx48_cqe_compress_pkts: 0
>> rx48_page_reuse: 0
>> rx48_cache_reuse: 0
>> rx48_cache_full: 0
>> rx48_cache_empty: 2560
>> rx48_cache_busy: 0
>> rx48_cache_waive: 0
>> rx48_congst_umr: 0
>> rx48_arfs_err: 0
>> rx48_xdp_tx_xmit: 0
>> rx48_xdp_tx_full: 0
>> rx48_xdp_tx_err: 0
>> rx48_xdp_tx_cqes: 0
>> rx49_packets: 0
>> rx49_bytes: 0
>> rx49_csum_complete: 0
>> rx49_csum_unnecessary: 0
>> rx49_csum_unnecessary_inner: 0
>> rx49_csum_none: 0
>> rx49_xdp_drop: 0
>> rx49_xdp_redirect: 0
>> rx49_lro_packets: 0
>> rx49_lro_bytes: 0
>> rx49_ecn_mark: 0
>> rx49_removed_vlan_packets: 0
>> rx49_wqe_err: 0
>> rx49_mpwqe_filler_cqes: 0
>> rx49_mpwqe_filler_strides: 0
>> rx49_buff_alloc_err: 0
>> rx49_cqe_compress_blks: 0
>> rx49_cqe_compress_pkts: 0
>> rx49_page_reuse: 0
>> rx49_cache_reuse: 0
>> rx49_cache_full: 0
>> rx49_cache_empty: 2560
>> rx49_cache_busy: 0
>> rx49_cache_waive: 0
>> rx49_congst_umr: 0
>> rx49_arfs_err: 0
>> rx49_xdp_tx_xmit: 0
>> rx49_xdp_tx_full: 0
>> rx49_xdp_tx_err: 0
>> rx49_xdp_tx_cqes: 0
>> rx50_packets: 0
>> rx50_bytes: 0
>> rx50_csum_complete: 0
>> rx50_csum_unnecessary: 0
>> rx50_csum_unnecessary_inner: 0
>> rx50_csum_none: 0
>> rx50_xdp_drop: 0
>> rx50_xdp_redirect: 0
>> rx50_lro_packets: 0
>> rx50_lro_bytes: 0
>> rx50_ecn_mark: 0
>> rx50_removed_vlan_packets: 0
>> rx50_wqe_err: 0
>> rx50_mpwqe_filler_cqes: 0
>> rx50_mpwqe_filler_strides: 0
>> rx50_buff_alloc_err: 0
>> rx50_cqe_compress_blks: 0
>> rx50_cqe_compress_pkts: 0
>> rx50_page_reuse: 0
>> rx50_cache_reuse: 0
>> rx50_cache_full: 0
>> rx50_cache_empty: 2560
>> rx50_cache_busy: 0
>> rx50_cache_waive: 0
>> rx50_congst_umr: 0
>> rx50_arfs_err: 0
>> rx50_xdp_tx_xmit: 0
>> rx50_xdp_tx_full: 0
>> rx50_xdp_tx_err: 0
>> rx50_xdp_tx_cqes: 0
>> rx51_packets: 0
>> rx51_bytes: 0
>> rx51_csum_complete: 0
>> rx51_csum_unnecessary: 0
>> rx51_csum_unnecessary_inner: 0
>> rx51_csum_none: 0
>> rx51_xdp_drop: 0
>> rx51_xdp_redirect: 0
>> rx51_lro_packets: 0
>> rx51_lro_bytes: 0
>> rx51_ecn_mark: 0
>> rx51_removed_vlan_packets: 0
>> rx51_wqe_err: 0
>> rx51_mpwqe_filler_cqes: 0
>> rx51_mpwqe_filler_strides: 0
>> rx51_buff_alloc_err: 0
>> rx51_cqe_compress_blks: 0
>> rx51_cqe_compress_pkts: 0
>> rx51_page_reuse: 0
>> rx51_cache_reuse: 0
>> rx51_cache_full: 0
>> rx51_cache_empty: 2560
>> rx51_cache_busy: 0
>> rx51_cache_waive: 0
>> rx51_congst_umr: 0
>> rx51_arfs_err: 0
>> rx51_xdp_tx_xmit: 0
>> rx51_xdp_tx_full: 0
>> rx51_xdp_tx_err: 0
>> rx51_xdp_tx_cqes: 0
>> rx52_packets: 0
>> rx52_bytes: 0
>> rx52_csum_complete: 0
>> rx52_csum_unnecessary: 0
>> rx52_csum_unnecessary_inner: 0
>> rx52_csum_none: 0
>> rx52_xdp_drop: 0
>> rx52_xdp_redirect: 0
>> rx52_lro_packets: 0
>> rx52_lro_bytes: 0
>> rx52_ecn_mark: 0
>> rx52_removed_vlan_packets: 0
>> rx52_wqe_err: 0
>> rx52_mpwqe_filler_cqes: 0
>> rx52_mpwqe_filler_strides: 0
>> rx52_buff_alloc_err: 0
>> rx52_cqe_compress_blks: 0
>> rx52_cqe_compress_pkts: 0
>> rx52_page_reuse: 0
>> rx52_cache_reuse: 0
>> rx52_cache_full: 0
>> rx52_cache_empty: 2560
>> rx52_cache_busy: 0
>> rx52_cache_waive: 0
>> rx52_congst_umr: 0
>> rx52_arfs_err: 0
>> rx52_xdp_tx_xmit: 0
>> rx52_xdp_tx_full: 0
>> rx52_xdp_tx_err: 0
>> rx52_xdp_tx_cqes: 0
>> rx53_packets: 0
>> rx53_bytes: 0
>> rx53_csum_complete: 0
>> rx53_csum_unnecessary: 0
>> rx53_csum_unnecessary_inner: 0
>> rx53_csum_none: 0
>> rx53_xdp_drop: 0
>> rx53_xdp_redirect: 0
>> rx53_lro_packets: 0
>> rx53_lro_bytes: 0
>> rx53_ecn_mark: 0
>> rx53_removed_vlan_packets: 0
>> rx53_wqe_err: 0
>> rx53_mpwqe_filler_cqes: 0
>> rx53_mpwqe_filler_strides: 0
>> rx53_buff_alloc_err: 0
>> rx53_cqe_compress_blks: 0
>> rx53_cqe_compress_pkts: 0
>> rx53_page_reuse: 0
>> rx53_cache_reuse: 0
>> rx53_cache_full: 0
>> rx53_cache_empty: 2560
>> rx53_cache_busy: 0
>> rx53_cache_waive: 0
>> rx53_congst_umr: 0
>> rx53_arfs_err: 0
>> rx53_xdp_tx_xmit: 0
>> rx53_xdp_tx_full: 0
>> rx53_xdp_tx_err: 0
>> rx53_xdp_tx_cqes: 0
>> rx54_packets: 0
>> rx54_bytes: 0
>> rx54_csum_complete: 0
>> rx54_csum_unnecessary: 0
>> rx54_csum_unnecessary_inner: 0
>> rx54_csum_none: 0
>> rx54_xdp_drop: 0
>> rx54_xdp_redirect: 0
>> rx54_lro_packets: 0
>> rx54_lro_bytes: 0
>> rx54_ecn_mark: 0
>> rx54_removed_vlan_packets: 0
>> rx54_wqe_err: 0
>> rx54_mpwqe_filler_cqes: 0
>> rx54_mpwqe_filler_strides: 0
>> rx54_buff_alloc_err: 0
>> rx54_cqe_compress_blks: 0
>> rx54_cqe_compress_pkts: 0
>> rx54_page_reuse: 0
>> rx54_cache_reuse: 0
>> rx54_cache_full: 0
>> rx54_cache_empty: 2560
>> rx54_cache_busy: 0
>> rx54_cache_waive: 0
>> rx54_congst_umr: 0
>> rx54_arfs_err: 0
>> rx54_xdp_tx_xmit: 0
>> rx54_xdp_tx_full: 0
>> rx54_xdp_tx_err: 0
>> rx54_xdp_tx_cqes: 0
>> rx55_packets: 0
>> rx55_bytes: 0
>> rx55_csum_complete: 0
>> rx55_csum_unnecessary: 0
>> rx55_csum_unnecessary_inner: 0
>> rx55_csum_none: 0
>> rx55_xdp_drop: 0
>> rx55_xdp_redirect: 0
>> rx55_lro_packets: 0
>> rx55_lro_bytes: 0
>> rx55_ecn_mark: 0
>> rx55_removed_vlan_packets: 0
>> rx55_wqe_err: 0
>> rx55_mpwqe_filler_cqes: 0
>> rx55_mpwqe_filler_strides: 0
>> rx55_buff_alloc_err: 0
>> rx55_cqe_compress_blks: 0
>> rx55_cqe_compress_pkts: 0
>> rx55_page_reuse: 0
>> rx55_cache_reuse: 0
>> rx55_cache_full: 0
>> rx55_cache_empty: 2560
>> rx55_cache_busy: 0
>> rx55_cache_waive: 0
>> rx55_congst_umr: 0
>> rx55_arfs_err: 0
>> rx55_xdp_tx_xmit: 0
>> rx55_xdp_tx_full: 0
>> rx55_xdp_tx_err: 0
>> rx55_xdp_tx_cqes: 0
>> tx0_packets: 24512439668
>> tx0_bytes: 15287569052791
>> tx0_tso_packets: 1536157106
>> tx0_tso_bytes: 8571753637944
>> tx0_tso_inner_packets: 0
>> tx0_tso_inner_bytes: 0
>> tx0_csum_partial: 2132156117
>> tx0_csum_partial_inner: 0
>> tx0_added_vlan_packets: 19906601448
>> tx0_nop: 308098536
>> tx0_csum_none: 17774445331
>> tx0_stopped: 19625
>> tx0_dropped: 0
>> tx0_xmit_more: 67864870
>> tx0_recover: 0
>> tx0_cqes: 19838744246
>> tx0_wake: 19624
>> tx0_cqe_err: 0
>> tx1_packets: 22598557053
>> tx1_bytes: 13568850145010
>> tx1_tso_packets: 1369529475
>> tx1_tso_bytes: 7661777265382
>> tx1_tso_inner_packets: 0
>> tx1_tso_inner_bytes: 0
>> tx1_csum_partial: 1884639496
>> tx1_csum_partial_inner: 0
>> tx1_added_vlan_packets: 18468333696
>> tx1_nop: 281301783
>> tx1_csum_none: 16583694200
>> tx1_stopped: 19457
>> tx1_dropped: 0
>> tx1_xmit_more: 55170875
>> tx1_recover: 0
>> tx1_cqes: 18413169824
>> tx1_wake: 19455
>> tx1_cqe_err: 0
>> tx2_packets: 22821611433
>> tx2_bytes: 13752535163683
>> tx2_tso_packets: 1396978825
>> tx2_tso_bytes: 7774704508463
>> tx2_tso_inner_packets: 0
>> tx2_tso_inner_bytes: 0
>> tx2_csum_partial: 1897834558
>> tx2_csum_partial_inner: 0
>> tx2_added_vlan_packets: 18641958085
>> tx2_nop: 286934891
>> tx2_csum_none: 16744123527
>> tx2_stopped: 13214
>> tx2_dropped: 0
>> tx2_xmit_more: 61749446
>> tx2_recover: 0
>> tx2_cqes: 18580215654
>> tx2_wake: 13214
>> tx2_cqe_err: 0
>> tx3_packets: 22580809948
>> tx3_bytes: 13730542936609
>> tx3_tso_packets: 1370434579
>> tx3_tso_bytes: 7605636711455
>> tx3_tso_inner_packets: 0
>> tx3_tso_inner_bytes: 0
>> tx3_csum_partial: 1865573748
>> tx3_csum_partial_inner: 0
>> tx3_added_vlan_packets: 18491873644
>> tx3_nop: 281195875
>> tx3_csum_none: 16626299896
>> tx3_stopped: 12542
>> tx3_dropped: 0
>> tx3_xmit_more: 57681647
>> tx3_recover: 0
>> tx3_cqes: 18434198757
>> tx3_wake: 12540
>> tx3_cqe_err: 0
>> tx4_packets: 27801801208
>> tx4_bytes: 17058453171137
>> tx4_tso_packets: 1740500105
>> tx4_tso_bytes: 9474905622036
>> tx4_tso_inner_packets: 0
>> tx4_tso_inner_bytes: 0
>> tx4_csum_partial: 2279225376
>> tx4_csum_partial_inner: 0
>> tx4_added_vlan_packets: 22744081633
>> tx4_nop: 349753979
>> tx4_csum_none: 20464856257
>> tx4_stopped: 14816
>> tx4_dropped: 0
>> tx4_xmit_more: 65469322
>> tx4_recover: 0
>> tx4_cqes: 22678618972
>> tx4_wake: 14816
>> tx4_cqe_err: 0
>> tx5_packets: 25099783024
>> tx5_bytes: 14917740698381
>> tx5_tso_packets: 1512988013
>> tx5_tso_bytes: 8571208921023
>> tx5_tso_inner_packets: 0
>> tx5_tso_inner_bytes: 0
>> tx5_csum_partial: 2078498561
>> tx5_csum_partial_inner: 0
>> tx5_added_vlan_packets: 20465533760
>> tx5_nop: 312614719
>> tx5_csum_none: 18387035199
>> tx5_stopped: 4605
>> tx5_dropped: 0
>> tx5_xmit_more: 64188936
>> tx5_recover: 0
>> tx5_cqes: 20401350718
>> tx5_wake: 4604
>> tx5_cqe_err: 0
>> tx6_packets: 25025504896
>> tx6_bytes: 14908021946070
>> tx6_tso_packets: 1515718977
>> tx6_tso_bytes: 8511442522461
>> tx6_tso_inner_packets: 0
>> tx6_tso_inner_bytes: 0
>> tx6_csum_partial: 2056378610
>> tx6_csum_partial_inner: 0
>> tx6_added_vlan_packets: 20434066400
>> tx6_nop: 310594020
>> tx6_csum_none: 18377687790
>> tx6_stopped: 15234
>> tx6_dropped: 0
>> tx6_xmit_more: 61130422
>> tx6_recover: 0
>> tx6_cqes: 20372943611
>> tx6_wake: 15234
>> tx6_cqe_err: 0
>> tx7_packets: 25457096169
>> tx7_bytes: 15456289446172
>> tx7_tso_packets: 1553342799
>> tx7_tso_bytes: 8764550988105
>> tx7_tso_inner_packets: 0
>> tx7_tso_inner_bytes: 0
>> tx7_csum_partial: 2105765233
>> tx7_csum_partial_inner: 0
>> tx7_added_vlan_packets: 20720382377
>> tx7_nop: 319044853
>> tx7_csum_none: 18614617145
>> tx7_stopped: 18745
>> tx7_dropped: 0
>> tx7_xmit_more: 57050107
>> tx7_recover: 0
>> tx7_cqes: 20663340775
>> tx7_wake: 18746
>> tx7_cqe_err: 0
>> tx8_packets: 25389771649
>> tx8_bytes: 15225503883962
>> tx8_tso_packets: 1563367648
>> tx8_tso_bytes: 8710384514258
>> tx8_tso_inner_packets: 0
>> tx8_tso_inner_bytes: 0
>> tx8_csum_partial: 2106586634
>> tx8_csum_partial_inner: 0
>> tx8_added_vlan_packets: 20704676274
>> tx8_nop: 318149261
>> tx8_csum_none: 18598089640
>> tx8_stopped: 4733
>> tx8_dropped: 0
>> tx8_xmit_more: 61014317
>> tx8_recover: 0
>> tx8_cqes: 20643667301
>> tx8_wake: 4735
>> tx8_cqe_err: 0
>> tx9_packets: 25521500166
>> tx9_bytes: 15302293145755
>> tx9_tso_packets: 1546316697
>> tx9_tso_bytes: 8770688145926
>> tx9_tso_inner_packets: 0
>> tx9_tso_inner_bytes: 0
>> tx9_csum_partial: 2097652880
>> tx9_csum_partial_inner: 0
>> tx9_added_vlan_packets: 20778408432
>> tx9_nop: 318538543
>> tx9_csum_none: 18680755556
>> tx9_stopped: 16118
>> tx9_dropped: 0
>> tx9_xmit_more: 68509728
>> tx9_recover: 0
>> tx9_cqes: 20709906498
>> tx9_wake: 16118
>> tx9_cqe_err: 0
>> tx10_packets: 25451605829
>> tx10_bytes: 15386896170792
>> tx10_tso_packets: 1576473520
>> tx10_tso_bytes: 8880888676383
>> tx10_tso_inner_packets: 0
>> tx10_tso_inner_bytes: 0
>> tx10_csum_partial: 2129796141
>> tx10_csum_partial_inner: 0
>> tx10_added_vlan_packets: 20659622590
>> tx10_nop: 319117433
>> tx10_csum_none: 18529826450
>> tx10_stopped: 20187
>> tx10_dropped: 0
>> tx10_xmit_more: 58892184
>> tx10_recover: 0
>> tx10_cqes: 20600737739
>> tx10_wake: 20188
>> tx10_cqe_err: 0
>> tx11_packets: 27008919793
>> tx11_bytes: 16587719213058
>> tx11_tso_packets: 1734884654
>> tx11_tso_bytes: 9475681471870
>> tx11_tso_inner_packets: 0
>> tx11_tso_inner_bytes: 0
>> tx11_csum_partial: 2296162292
>> tx11_csum_partial_inner: 0
>> tx11_added_vlan_packets: 21943096263
>> tx11_nop: 344188182
>> tx11_csum_none: 19646933971
>> tx11_stopped: 9703
>> tx11_dropped: 0
>> tx11_xmit_more: 66530718
>> tx11_recover: 0
>> tx11_cqes: 21876571667
>> tx11_wake: 9704
>> tx11_cqe_err: 0
>> tx12_packets: 25969493269
>> tx12_bytes: 15980767963416
>> tx12_tso_packets: 1671396456
>> tx12_tso_bytes: 9268973672821
>> tx12_tso_inner_packets: 0
>> tx12_tso_inner_bytes: 0
>> tx12_csum_partial: 2243809182
>> tx12_csum_partial_inner: 0
>> tx12_added_vlan_packets: 20980642456
>> tx12_nop: 330241007
>> tx12_csum_none: 18736833276
>> tx12_stopped: 10341
>> tx12_dropped: 0
>> tx12_xmit_more: 57834100
>> tx12_recover: 0
>> tx12_cqes: 20922815079
>> tx12_wake: 10342
>> tx12_cqe_err: 0
>> tx13_packets: 25332762261
>> tx13_bytes: 15353213283280
>> tx13_tso_packets: 1577433599
>> tx13_tso_bytes: 8785240284281
>> tx13_tso_inner_packets: 0
>> tx13_tso_inner_bytes: 0
>> tx13_csum_partial: 2110640515
>> tx13_csum_partial_inner: 0
>> tx13_added_vlan_packets: 20605670910
>> tx13_nop: 319805741
>> tx13_csum_none: 18495030395
>> tx13_stopped: 7006
>> tx13_dropped: 0
>> tx13_xmit_more: 58314402
>> tx13_recover: 0
>> tx13_cqes: 20547362770
>> tx13_wake: 7008
>> tx13_cqe_err: 0
>> tx14_packets: 26333743548
>> tx14_bytes: 16070719060573
>> tx14_tso_packets: 1677922970
>> tx14_tso_bytes: 9240299765487
>> tx14_tso_inner_packets: 0
>> tx14_tso_inner_bytes: 0
>> tx14_csum_partial: 2215668906
>> tx14_csum_partial_inner: 0
>> tx14_added_vlan_packets: 21384410786
>> tx14_nop: 332734939
>> tx14_csum_none: 19168741880
>> tx14_stopped: 13160
>> tx14_dropped: 0
>> tx14_xmit_more: 57650391
>> tx14_recover: 0
>> tx14_cqes: 21326767783
>> tx14_wake: 13161
>> tx14_cqe_err: 0
>> tx15_packets: 26824968971
>> tx15_bytes: 16687994233452
>> tx15_tso_packets: 1755745052
>> tx15_tso_bytes: 9533814012441
>> tx15_tso_inner_packets: 0
>> tx15_tso_inner_bytes: 0
>> tx15_csum_partial: 2304778064
>> tx15_csum_partial_inner: 0
>> tx15_added_vlan_packets: 21740906107
>> tx15_nop: 344143287
>> tx15_csum_none: 19436128058
>> tx15_stopped: 75
>> tx15_dropped: 0
>> tx15_xmit_more: 63325832
>> tx15_recover: 0
>> tx15_cqes: 21677585345
>> tx15_wake: 74
>> tx15_cqe_err: 0
>> tx16_packets: 24488158946
>> tx16_bytes: 15027415004570
>> tx16_tso_packets: 1559127391
>> tx16_tso_bytes: 8658691917845
>> tx16_tso_inner_packets: 0
>> tx16_tso_inner_bytes: 0
>> tx16_csum_partial: 2075856395
>> tx16_csum_partial_inner: 0
>> tx16_added_vlan_packets: 19835695731
>> tx16_nop: 308464189
>> tx16_csum_none: 17759839340
>> tx16_stopped: 4567
>> tx16_dropped: 0
>> tx16_xmit_more: 62631422
>> tx16_recover: 0
>> tx16_cqes: 19773070012
>> tx16_wake: 4568
>> tx16_cqe_err: 0
>> tx17_packets: 24700413784
>> tx17_bytes: 15216529713715
>> tx17_tso_packets: 1597555108
>> tx17_tso_bytes: 8773728661243
>> tx17_tso_inner_packets: 0
>> tx17_tso_inner_bytes: 0
>> tx17_csum_partial: 2127177297
>> tx17_csum_partial_inner: 0
>> tx17_added_vlan_packets: 20003144561
>> tx17_nop: 313356918
>> tx17_csum_none: 17875967264
>> tx17_stopped: 12572
>> tx17_dropped: 0
>> tx17_xmit_more: 62742980
>> tx17_recover: 0
>> tx17_cqes: 19940407615
>> tx17_wake: 12573
>> tx17_cqe_err: 0
>> tx18_packets: 24887710046
>> tx18_bytes: 15245034034664
>> tx18_tso_packets: 1582550520
>> tx18_tso_bytes: 8782692335483
>> tx18_tso_inner_packets: 0
>> tx18_tso_inner_bytes: 0
>> tx18_csum_partial: 2084514331
>> tx18_csum_partial_inner: 0
>> tx18_added_vlan_packets: 20173879181
>> tx18_nop: 314818702
>> tx18_csum_none: 18089364850
>> tx18_stopped: 21366
>> tx18_dropped: 0
>> tx18_xmit_more: 62485819
>> tx18_recover: 0
>> tx18_cqes: 20111400935
>> tx18_wake: 21366
>> tx18_cqe_err: 0
>> tx19_packets: 24831057648
>> tx19_bytes: 15164663890576
>> tx19_tso_packets: 1599135489
>> tx19_tso_bytes: 8756045449746
>> tx19_tso_inner_packets: 0
>> tx19_tso_inner_bytes: 0
>> tx19_csum_partial: 2119746608
>> tx19_csum_partial_inner: 0
>> tx19_added_vlan_packets: 20143573903
>> tx19_nop: 316966450
>> tx19_csum_none: 18023827295
>> tx19_stopped: 11431
>> tx19_dropped: 0
>> tx19_xmit_more: 57535904
>> tx19_recover: 0
>> tx19_cqes: 20086045325
>> tx19_wake: 11431
>> tx19_cqe_err: 0
>> tx20_packets: 21943735263
>> tx20_bytes: 13528749492187
>> tx20_tso_packets: 1390048103
>> tx20_tso_bytes: 7629058809637
>> tx20_tso_inner_packets: 0
>> tx20_tso_inner_bytes: 0
>> tx20_csum_partial: 1848533941
>> tx20_csum_partial_inner: 0
>> tx20_added_vlan_packets: 17861417651
>> tx20_nop: 276840365
>> tx20_csum_none: 16012883710
>> tx20_stopped: 38457
>> tx20_dropped: 0
>> tx20_xmit_more: 57042753
>> tx20_recover: 0
>> tx20_cqes: 17804384839
>> tx20_wake: 38457
>> tx20_cqe_err: 0
>> tx21_packets: 21476926958
>> tx21_bytes: 13096410597896
>> tx21_tso_packets: 1367724090
>> tx21_tso_bytes: 7568364585127
>> tx21_tso_inner_packets: 0
>> tx21_tso_inner_bytes: 0
>> tx21_csum_partial: 1830570727
>> tx21_csum_partial_inner: 0
>> tx21_added_vlan_packets: 17421087814
>> tx21_nop: 270611519
>> tx21_csum_none: 15590517087
>> tx21_stopped: 31213
>> tx21_dropped: 0
>> tx21_xmit_more: 60305389
>> tx21_recover: 0
>> tx21_cqes: 17360791205
>> tx21_wake: 31213
>> tx21_cqe_err: 0
>> tx22_packets: 21819106444
>> tx22_bytes: 13492871887100
>> tx22_tso_packets: 1387002018
>> tx22_tso_bytes: 7617705727669
>> tx22_tso_inner_packets: 0
>> tx22_tso_inner_bytes: 0
>> tx22_csum_partial: 1853632107
>> tx22_csum_partial_inner: 0
>> tx22_added_vlan_packets: 17743255447
>> tx22_nop: 274820992
>> tx22_csum_none: 15889623340
>> tx22_stopped: 24814
>> tx22_dropped: 0
>> tx22_xmit_more: 60811304
>> tx22_recover: 0
>> tx22_cqes: 17682451111
>> tx22_wake: 24815
>> tx22_cqe_err: 0
>> tx23_packets: 21830455800
>> tx23_bytes: 13427551902532
>> tx23_tso_packets: 1388556038
>> tx23_tso_bytes: 7604040587125
>> tx23_tso_inner_packets: 0
>> tx23_tso_inner_bytes: 0
>> tx23_csum_partial: 1850819694
>> tx23_csum_partial_inner: 0
>> tx23_added_vlan_packets: 17761271122
>> tx23_nop: 275142775
>> tx23_csum_none: 15910451428
>> tx23_stopped: 29899
>> tx23_dropped: 0
>> tx23_xmit_more: 58924909
>> tx23_recover: 0
>> tx23_cqes: 17702355187
>> tx23_wake: 29898
>> tx23_cqe_err: 0
>> tx24_packets: 21961484213
>> tx24_bytes: 13531373062497
>> tx24_tso_packets: 1394697504
>> tx24_tso_bytes: 7663866609308
>> tx24_tso_inner_packets: 0
>> tx24_tso_inner_bytes: 0
>> tx24_csum_partial: 1857072074
>> tx24_csum_partial_inner: 0
>> tx24_added_vlan_packets: 17856887568
>> tx24_nop: 276352855
>> tx24_csum_none: 15999815494
>> tx24_stopped: 33924
>> tx24_dropped: 0
>> tx24_xmit_more: 63992426
>> tx24_recover: 0
>> tx24_cqes: 17792905243
>> tx24_wake: 33923
>> tx24_cqe_err: 0
>> tx25_packets: 21853593838
>> tx25_bytes: 13357487830519
>> tx25_tso_packets: 1398822411
>> tx25_tso_bytes: 7691191518838
>> tx25_tso_inner_packets: 0
>> tx25_tso_inner_bytes: 0
>> tx25_csum_partial: 1869483109
>> tx25_csum_partial_inner: 0
>> tx25_added_vlan_packets: 17734634614
>> tx25_nop: 276327643
>> tx25_csum_none: 15865151505
>> tx25_stopped: 38651
>> tx25_dropped: 0
>> tx25_xmit_more: 56410535
>> tx25_recover: 0
>> tx25_cqes: 17678234537
>> tx25_wake: 38650
>> tx25_cqe_err: 0
>> tx26_packets: 21480261205
>> tx26_bytes: 13148973015935
>> tx26_tso_packets: 1348132284
>> tx26_tso_bytes: 7523489481775
>> tx26_tso_inner_packets: 0
>> tx26_tso_inner_bytes: 0
>> tx26_csum_partial: 1839740745
>> tx26_csum_partial_inner: 0
>> tx26_added_vlan_packets: 17430592911
>> tx26_nop: 270367836
>> tx26_csum_none: 15590852166
>> tx26_stopped: 34044
>> tx26_dropped: 0
>> tx26_xmit_more: 59870114
>> tx26_recover: 0
>> tx26_cqes: 17370736612
>> tx26_wake: 34043
>> tx26_cqe_err: 0
>> tx27_packets: 22694273108
>> tx27_bytes: 14135473431004
>> tx27_tso_packets: 1418371875
>> tx27_tso_bytes: 7784842263038
>> tx27_tso_inner_packets: 0
>> tx27_tso_inner_bytes: 0
>> tx27_csum_partial: 1919170584
>> tx27_csum_partial_inner: 0
>> tx27_added_vlan_packets: 18520826023
>> tx27_nop: 286296272
>> tx27_csum_none: 16601655439
>> tx27_stopped: 38125
>> tx27_dropped: 0
>> tx27_xmit_more: 72749775
>> tx27_recover: 0
>> tx27_cqes: 18448090270
>> tx27_wake: 38127
>> tx27_cqe_err: 0
>> tx28_packets: 0
>> tx28_bytes: 0
>> tx28_tso_packets: 0
>> tx28_tso_bytes: 0
>> tx28_tso_inner_packets: 0
>> tx28_tso_inner_bytes: 0
>> tx28_csum_partial: 0
>> tx28_csum_partial_inner: 0
>> tx28_added_vlan_packets: 0
>> tx28_nop: 0
>> tx28_csum_none: 0
>> tx28_stopped: 0
>> tx28_dropped: 0
>> tx28_xmit_more: 0
>> tx28_recover: 0
>> tx28_cqes: 0
>> tx28_wake: 0
>> tx28_cqe_err: 0
>> tx29_packets: 3
>> tx29_bytes: 266
>> tx29_tso_packets: 0
>> tx29_tso_bytes: 0
>> tx29_tso_inner_packets: 0
>> tx29_tso_inner_bytes: 0
>> tx29_csum_partial: 0
>> tx29_csum_partial_inner: 0
>> tx29_added_vlan_packets: 0
>> tx29_nop: 0
>> tx29_csum_none: 3
>> tx29_stopped: 0
>> tx29_dropped: 0
>> tx29_xmit_more: 1
>> tx29_recover: 0
>> tx29_cqes: 2
>> tx29_wake: 0
>> tx29_cqe_err: 0
>> tx30_packets: 0
>> tx30_bytes: 0
>> tx30_tso_packets: 0
>> tx30_tso_bytes: 0
>> tx30_tso_inner_packets: 0
>> tx30_tso_inner_bytes: 0
>> tx30_csum_partial: 0
>> tx30_csum_partial_inner: 0
>> tx30_added_vlan_packets: 0
>> tx30_nop: 0
>> tx30_csum_none: 0
>> tx30_stopped: 0
>> tx30_dropped: 0
>> tx30_xmit_more: 0
>> tx30_recover: 0
>> tx30_cqes: 0
>> tx30_wake: 0
>> tx30_cqe_err: 0
>> tx31_packets: 0
>> tx31_bytes: 0
>> tx31_tso_packets: 0
>> tx31_tso_bytes: 0
>> tx31_tso_inner_packets: 0
>> tx31_tso_inner_bytes: 0
>> tx31_csum_partial: 0
>> tx31_csum_partial_inner: 0
>> tx31_added_vlan_packets: 0
>> tx31_nop: 0
>> tx31_csum_none: 0
>> tx31_stopped: 0
>> tx31_dropped: 0
>> tx31_xmit_more: 0
>> tx31_recover: 0
>> tx31_cqes: 0
>> tx31_wake: 0
>> tx31_cqe_err: 0
>> tx32_packets: 0
>> tx32_bytes: 0
>> tx32_tso_packets: 0
>> tx32_tso_bytes: 0
>> tx32_tso_inner_packets: 0
>> tx32_tso_inner_bytes: 0
>> tx32_csum_partial: 0
>> tx32_csum_partial_inner: 0
>> tx32_added_vlan_packets: 0
>> tx32_nop: 0
>> tx32_csum_none: 0
>> tx32_stopped: 0
>> tx32_dropped: 0
>> tx32_xmit_more: 0
>> tx32_recover: 0
>> tx32_cqes: 0
>> tx32_wake: 0
>> tx32_cqe_err: 0
>> tx33_packets: 0
>> tx33_bytes: 0
>> tx33_tso_packets: 0
>> tx33_tso_bytes: 0
>> tx33_tso_inner_packets: 0
>> tx33_tso_inner_bytes: 0
>> tx33_csum_partial: 0
>> tx33_csum_partial_inner: 0
>> tx33_added_vlan_packets: 0
>> tx33_nop: 0
>> tx33_csum_none: 0
>> tx33_stopped: 0
>> tx33_dropped: 0
>> tx33_xmit_more: 0
>> tx33_recover: 0
>> tx33_cqes: 0
>> tx33_wake: 0
>> tx33_cqe_err: 0
>> tx34_packets: 0
>> tx34_bytes: 0
>> tx34_tso_packets: 0
>> tx34_tso_bytes: 0
>> tx34_tso_inner_packets: 0
>> tx34_tso_inner_bytes: 0
>> tx34_csum_partial: 0
>> tx34_csum_partial_inner: 0
>> tx34_added_vlan_packets: 0
>> tx34_nop: 0
>> tx34_csum_none: 0
>> tx34_stopped: 0
>> tx34_dropped: 0
>> tx34_xmit_more: 0
>> tx34_recover: 0
>> tx34_cqes: 0
>> tx34_wake: 0
>> tx34_cqe_err: 0
>> tx35_packets: 0
>> tx35_bytes: 0
>> tx35_tso_packets: 0
>> tx35_tso_bytes: 0
>> tx35_tso_inner_packets: 0
>> tx35_tso_inner_bytes: 0
>> tx35_csum_partial: 0
>> tx35_csum_partial_inner: 0
>> tx35_added_vlan_packets: 0
>> tx35_nop: 0
>> tx35_csum_none: 0
>> tx35_stopped: 0
>> tx35_dropped: 0
>> tx35_xmit_more: 0
>> tx35_recover: 0
>> tx35_cqes: 0
>> tx35_wake: 0
>> tx35_cqe_err: 0
>> tx36_packets: 0
>> tx36_bytes: 0
>> tx36_tso_packets: 0
>> tx36_tso_bytes: 0
>> tx36_tso_inner_packets: 0
>> tx36_tso_inner_bytes: 0
>> tx36_csum_partial: 0
>> tx36_csum_partial_inner: 0
>> tx36_added_vlan_packets: 0
>> tx36_nop: 0
>> tx36_csum_none: 0
>> tx36_stopped: 0
>> tx36_dropped: 0
>> tx36_xmit_more: 0
>> tx36_recover: 0
>> tx36_cqes: 0
>> tx36_wake: 0
>> tx36_cqe_err: 0
>> tx37_packets: 0
>> tx37_bytes: 0
>> tx37_tso_packets: 0
>> tx37_tso_bytes: 0
>> tx37_tso_inner_packets: 0
>> tx37_tso_inner_bytes: 0
>> tx37_csum_partial: 0
>> tx37_csum_partial_inner: 0
>> tx37_added_vlan_packets: 0
>> tx37_nop: 0
>> tx37_csum_none: 0
>> tx37_stopped: 0
>> tx37_dropped: 0
>> tx37_xmit_more: 0
>> tx37_recover: 0
>> tx37_cqes: 0
>> tx37_wake: 0
>> tx37_cqe_err: 0
>> tx38_packets: 0
>> tx38_bytes: 0
>> tx38_tso_packets: 0
>> tx38_tso_bytes: 0
>> tx38_tso_inner_packets: 0
>> tx38_tso_inner_bytes: 0
>> tx38_csum_partial: 0
>> tx38_csum_partial_inner: 0
>> tx38_added_vlan_packets: 0
>> tx38_nop: 0
>> tx38_csum_none: 0
>> tx38_stopped: 0
>> tx38_dropped: 0
>> tx38_xmit_more: 0
>> tx38_recover: 0
>> tx38_cqes: 0
>> tx38_wake: 0
>> tx38_cqe_err: 0
>> tx39_packets: 0
>> tx39_bytes: 0
>> tx39_tso_packets: 0
>> tx39_tso_bytes: 0
>> tx39_tso_inner_packets: 0
>> tx39_tso_inner_bytes: 0
>> tx39_csum_partial: 0
>> tx39_csum_partial_inner: 0
>> tx39_added_vlan_packets: 0
>> tx39_nop: 0
>> tx39_csum_none: 0
>> tx39_stopped: 0
>> tx39_dropped: 0
>> tx39_xmit_more: 0
>> tx39_recover: 0
>> tx39_cqes: 0
>> tx39_wake: 0
>> tx39_cqe_err: 0
>> tx40_packets: 0
>> tx40_bytes: 0
>> tx40_tso_packets: 0
>> tx40_tso_bytes: 0
>> tx40_tso_inner_packets: 0
>> tx40_tso_inner_bytes: 0
>> tx40_csum_partial: 0
>> tx40_csum_partial_inner: 0
>> tx40_added_vlan_packets: 0
>> tx40_nop: 0
>> tx40_csum_none: 0
>> tx40_stopped: 0
>> tx40_dropped: 0
>> tx40_xmit_more: 0
>> tx40_recover: 0
>> tx40_cqes: 0
>> tx40_wake: 0
>> tx40_cqe_err: 0
>> tx41_packets: 0
>> tx41_bytes: 0
>> tx41_tso_packets: 0
>> tx41_tso_bytes: 0
>> tx41_tso_inner_packets: 0
>> tx41_tso_inner_bytes: 0
>> tx41_csum_partial: 0
>> tx41_csum_partial_inner: 0
>> tx41_added_vlan_packets: 0
>> tx41_nop: 0
>> tx41_csum_none: 0
>> tx41_stopped: 0
>> tx41_dropped: 0
>> tx41_xmit_more: 0
>> tx41_recover: 0
>> tx41_cqes: 0
>> tx41_wake: 0
>> tx41_cqe_err: 0
>> tx42_packets: 0
>> tx42_bytes: 0
>> tx42_tso_packets: 0
>> tx42_tso_bytes: 0
>> tx42_tso_inner_packets: 0
>> tx42_tso_inner_bytes: 0
>> tx42_csum_partial: 0
>> tx42_csum_partial_inner: 0
>> tx42_added_vlan_packets: 0
>> tx42_nop: 0
>> tx42_csum_none: 0
>> tx42_stopped: 0
>> tx42_dropped: 0
>> tx42_xmit_more: 0
>> tx42_recover: 0
>> tx42_cqes: 0
>> tx42_wake: 0
>> tx42_cqe_err: 0
>> tx43_packets: 0
>> tx43_bytes: 0
>> tx43_tso_packets: 0
>> tx43_tso_bytes: 0
>> tx43_tso_inner_packets: 0
>> tx43_tso_inner_bytes: 0
>> tx43_csum_partial: 0
>> tx43_csum_partial_inner: 0
>> tx43_added_vlan_packets: 0
>> tx43_nop: 0
>> tx43_csum_none: 0
>> tx43_stopped: 0
>> tx43_dropped: 0
>> tx43_xmit_more: 0
>> tx43_recover: 0
>> tx43_cqes: 0
>> tx43_wake: 0
>> tx43_cqe_err: 0
>> tx44_packets: 0
>> tx44_bytes: 0
>> tx44_tso_packets: 0
>> tx44_tso_bytes: 0
>> tx44_tso_inner_packets: 0
>> tx44_tso_inner_bytes: 0
>> tx44_csum_partial: 0
>> tx44_csum_partial_inner: 0
>> tx44_added_vlan_packets: 0
>> tx44_nop: 0
>> tx44_csum_none: 0
>> tx44_stopped: 0
>> tx44_dropped: 0
>> tx44_xmit_more: 0
>> tx44_recover: 0
>> tx44_cqes: 0
>> tx44_wake: 0
>> tx44_cqe_err: 0
>> tx45_packets: 0
>> tx45_bytes: 0
>> tx45_tso_packets: 0
>> tx45_tso_bytes: 0
>> tx45_tso_inner_packets: 0
>> tx45_tso_inner_bytes: 0
>> tx45_csum_partial: 0
>> tx45_csum_partial_inner: 0
>> tx45_added_vlan_packets: 0
>> tx45_nop: 0
>> tx45_csum_none: 0
>> tx45_stopped: 0
>> tx45_dropped: 0
>> tx45_xmit_more: 0
>> tx45_recover: 0
>> tx45_cqes: 0
>> tx45_wake: 0
>> tx45_cqe_err: 0
>> tx46_packets: 0
>> tx46_bytes: 0
>> tx46_tso_packets: 0
>> tx46_tso_bytes: 0
>> tx46_tso_inner_packets: 0
>> tx46_tso_inner_bytes: 0
>> tx46_csum_partial: 0
>> tx46_csum_partial_inner: 0
>> tx46_added_vlan_packets: 0
>> tx46_nop: 0
>> tx46_csum_none: 0
>> tx46_stopped: 0
>> tx46_dropped: 0
>> tx46_xmit_more: 0
>> tx46_recover: 0
>> tx46_cqes: 0
>> tx46_wake: 0
>> tx46_cqe_err: 0
>> tx47_packets: 0
>> tx47_bytes: 0
>> tx47_tso_packets: 0
>> tx47_tso_bytes: 0
>> tx47_tso_inner_packets: 0
>> tx47_tso_inner_bytes: 0
>> tx47_csum_partial: 0
>> tx47_csum_partial_inner: 0
>> tx47_added_vlan_packets: 0
>> tx47_nop: 0
>> tx47_csum_none: 0
>> tx47_stopped: 0
>> tx47_dropped: 0
>> tx47_xmit_more: 0
>> tx47_recover: 0
>> tx47_cqes: 0
>> tx47_wake: 0
>> tx47_cqe_err: 0
>> tx48_packets: 0
>> tx48_bytes: 0
>> tx48_tso_packets: 0
>> tx48_tso_bytes: 0
>> tx48_tso_inner_packets: 0
>> tx48_tso_inner_bytes: 0
>> tx48_csum_partial: 0
>> tx48_csum_partial_inner: 0
>> tx48_added_vlan_packets: 0
>> tx48_nop: 0
>> tx48_csum_none: 0
>> tx48_stopped: 0
>> tx48_dropped: 0
>> tx48_xmit_more: 0
>> tx48_recover: 0
>> tx48_cqes: 0
>> tx48_wake: 0
>> tx48_cqe_err: 0
>> tx49_packets: 0
>> tx49_bytes: 0
>> tx49_tso_packets: 0
>> tx49_tso_bytes: 0
>> tx49_tso_inner_packets: 0
>> tx49_tso_inner_bytes: 0
>> tx49_csum_partial: 0
>> tx49_csum_partial_inner: 0
>> tx49_added_vlan_packets: 0
>> tx49_nop: 0
>> tx49_csum_none: 0
>> tx49_stopped: 0
>> tx49_dropped: 0
>> tx49_xmit_more: 0
>> tx49_recover: 0
>> tx49_cqes: 0
>> tx49_wake: 0
>> tx49_cqe_err: 0
>> tx50_packets: 0
>> tx50_bytes: 0
>> tx50_tso_packets: 0
>> tx50_tso_bytes: 0
>> tx50_tso_inner_packets: 0
>> tx50_tso_inner_bytes: 0
>> tx50_csum_partial: 0
>> tx50_csum_partial_inner: 0
>> tx50_added_vlan_packets: 0
>> tx50_nop: 0
>> tx50_csum_none: 0
>> tx50_stopped: 0
>> tx50_dropped: 0
>> tx50_xmit_more: 0
>> tx50_recover: 0
>> tx50_cqes: 0
>> tx50_wake: 0
>> tx50_cqe_err: 0
>> tx51_packets: 0
>> tx51_bytes: 0
>> tx51_tso_packets: 0
>> tx51_tso_bytes: 0
>> tx51_tso_inner_packets: 0
>> tx51_tso_inner_bytes: 0
>> tx51_csum_partial: 0
>> tx51_csum_partial_inner: 0
>> tx51_added_vlan_packets: 0
>> tx51_nop: 0
>> tx51_csum_none: 0
>> tx51_stopped: 0
>> tx51_dropped: 0
>> tx51_xmit_more: 0
>> tx51_recover: 0
>> tx51_cqes: 0
>> tx51_wake: 0
>> tx51_cqe_err: 0
>> tx52_packets: 0
>> tx52_bytes: 0
>> tx52_tso_packets: 0
>> tx52_tso_bytes: 0
>> tx52_tso_inner_packets: 0
>> tx52_tso_inner_bytes: 0
>> tx52_csum_partial: 0
>> tx52_csum_partial_inner: 0
>> tx52_added_vlan_packets: 0
>> tx52_nop: 0
>> tx52_csum_none: 0
>> tx52_stopped: 0
>> tx52_dropped: 0
>> tx52_xmit_more: 0
>> tx52_recover: 0
>> tx52_cqes: 0
>> tx52_wake: 0
>> tx52_cqe_err: 0
>> tx53_packets: 0
>> tx53_bytes: 0
>> tx53_tso_packets: 0
>> tx53_tso_bytes: 0
>> tx53_tso_inner_packets: 0
>> tx53_tso_inner_bytes: 0
>> tx53_csum_partial: 0
>> tx53_csum_partial_inner: 0
>> tx53_added_vlan_packets: 0
>> tx53_nop: 0
>> tx53_csum_none: 0
>> tx53_stopped: 0
>> tx53_dropped: 0
>> tx53_xmit_more: 0
>> tx53_recover: 0
>> tx53_cqes: 0
>> tx53_wake: 0
>> tx53_cqe_err: 0
>> tx54_packets: 0
>> tx54_bytes: 0
>> tx54_tso_packets: 0
>> tx54_tso_bytes: 0
>> tx54_tso_inner_packets: 0
>> tx54_tso_inner_bytes: 0
>> tx54_csum_partial: 0
>> tx54_csum_partial_inner: 0
>> tx54_added_vlan_packets: 0
>> tx54_nop: 0
>> tx54_csum_none: 0
>> tx54_stopped: 0
>> tx54_dropped: 0
>> tx54_xmit_more: 0
>> tx54_recover: 0
>> tx54_cqes: 0
>> tx54_wake: 0
>> tx54_cqe_err: 0
>> tx55_packets: 0
>> tx55_bytes: 0
>> tx55_tso_packets: 0
>> tx55_tso_bytes: 0
>> tx55_tso_inner_packets: 0
>> tx55_tso_inner_bytes: 0
>> tx55_csum_partial: 0
>> tx55_csum_partial_inner: 0
>> tx55_added_vlan_packets: 0
>> tx55_nop: 0
>> tx55_csum_none: 0
>> tx55_stopped: 0
>> tx55_dropped: 0
>> tx55_xmit_more: 0
>> tx55_recover: 0
>> tx55_cqes: 0
>> tx55_wake: 0
>> tx55_cqe_err: 0
>> tx0_xdp_xmit: 0
>> tx0_xdp_full: 0
>> tx0_xdp_err: 0
>> tx0_xdp_cqes: 0
>> tx1_xdp_xmit: 0
>> tx1_xdp_full: 0
>> tx1_xdp_err: 0
>> tx1_xdp_cqes: 0
>> tx2_xdp_xmit: 0
>> tx2_xdp_full: 0
>> tx2_xdp_err: 0
>> tx2_xdp_cqes: 0
>> tx3_xdp_xmit: 0
>> tx3_xdp_full: 0
>> tx3_xdp_err: 0
>> tx3_xdp_cqes: 0
>> tx4_xdp_xmit: 0
>> tx4_xdp_full: 0
>> tx4_xdp_err: 0
>> tx4_xdp_cqes: 0
>> tx5_xdp_xmit: 0
>> tx5_xdp_full: 0
>> tx5_xdp_err: 0
>> tx5_xdp_cqes: 0
>> tx6_xdp_xmit: 0
>> tx6_xdp_full: 0
>> tx6_xdp_err: 0
>> tx6_xdp_cqes: 0
>> tx7_xdp_xmit: 0
>> tx7_xdp_full: 0
>> tx7_xdp_err: 0
>> tx7_xdp_cqes: 0
>> tx8_xdp_xmit: 0
>> tx8_xdp_full: 0
>> tx8_xdp_err: 0
>> tx8_xdp_cqes: 0
>> tx9_xdp_xmit: 0
>> tx9_xdp_full: 0
>> tx9_xdp_err: 0
>> tx9_xdp_cqes: 0
>> tx10_xdp_xmit: 0
>> tx10_xdp_full: 0
>> tx10_xdp_err: 0
>> tx10_xdp_cqes: 0
>> tx11_xdp_xmit: 0
>> tx11_xdp_full: 0
>> tx11_xdp_err: 0
>> tx11_xdp_cqes: 0
>> tx12_xdp_xmit: 0
>> tx12_xdp_full: 0
>> tx12_xdp_err: 0
>> tx12_xdp_cqes: 0
>> tx13_xdp_xmit: 0
>> tx13_xdp_full: 0
>> tx13_xdp_err: 0
>> tx13_xdp_cqes: 0
>> tx14_xdp_xmit: 0
>> tx14_xdp_full: 0
>> tx14_xdp_err: 0
>> tx14_xdp_cqes: 0
>> tx15_xdp_xmit: 0
>> tx15_xdp_full: 0
>> tx15_xdp_err: 0
>> tx15_xdp_cqes: 0
>> tx16_xdp_xmit: 0
>> tx16_xdp_full: 0
>> tx16_xdp_err: 0
>> tx16_xdp_cqes: 0
>> tx17_xdp_xmit: 0
>> tx17_xdp_full: 0
>> tx17_xdp_err: 0
>> tx17_xdp_cqes: 0
>> tx18_xdp_xmit: 0
>> tx18_xdp_full: 0
>> tx18_xdp_err: 0
>> tx18_xdp_cqes: 0
>> tx19_xdp_xmit: 0
>> tx19_xdp_full: 0
>> tx19_xdp_err: 0
>> tx19_xdp_cqes: 0
>> tx20_xdp_xmit: 0
>> tx20_xdp_full: 0
>> tx20_xdp_err: 0
>> tx20_xdp_cqes: 0
>> tx21_xdp_xmit: 0
>> tx21_xdp_full: 0
>> tx21_xdp_err: 0
>> tx21_xdp_cqes: 0
>> tx22_xdp_xmit: 0
>> tx22_xdp_full: 0
>> tx22_xdp_err: 0
>> tx22_xdp_cqes: 0
>> tx23_xdp_xmit: 0
>> tx23_xdp_full: 0
>> tx23_xdp_err: 0
>> tx23_xdp_cqes: 0
>> tx24_xdp_xmit: 0
>> tx24_xdp_full: 0
>> tx24_xdp_err: 0
>> tx24_xdp_cqes: 0
>> tx25_xdp_xmit: 0
>> tx25_xdp_full: 0
>> tx25_xdp_err: 0
>> tx25_xdp_cqes: 0
>> tx26_xdp_xmit: 0
>> tx26_xdp_full: 0
>> tx26_xdp_err: 0
>> tx26_xdp_cqes: 0
>> tx27_xdp_xmit: 0
>> tx27_xdp_full: 0
>> tx27_xdp_err: 0
>> tx27_xdp_cqes: 0
>> tx28_xdp_xmit: 0
>> tx28_xdp_full: 0
>> tx28_xdp_err: 0
>> tx28_xdp_cqes: 0
>> tx29_xdp_xmit: 0
>> tx29_xdp_full: 0
>> tx29_xdp_err: 0
>> tx29_xdp_cqes: 0
>> tx30_xdp_xmit: 0
>> tx30_xdp_full: 0
>> tx30_xdp_err: 0
>> tx30_xdp_cqes: 0
>> tx31_xdp_xmit: 0
>> tx31_xdp_full: 0
>> tx31_xdp_err: 0
>> tx31_xdp_cqes: 0
>> tx32_xdp_xmit: 0
>> tx32_xdp_full: 0
>> tx32_xdp_err: 0
>> tx32_xdp_cqes: 0
>> tx33_xdp_xmit: 0
>> tx33_xdp_full: 0
>> tx33_xdp_err: 0
>> tx33_xdp_cqes: 0
>> tx34_xdp_xmit: 0
>> tx34_xdp_full: 0
>> tx34_xdp_err: 0
>> tx34_xdp_cqes: 0
>> tx35_xdp_xmit: 0
>> tx35_xdp_full: 0
>> tx35_xdp_err: 0
>> tx35_xdp_cqes: 0
>> tx36_xdp_xmit: 0
>> tx36_xdp_full: 0
>> tx36_xdp_err: 0
>> tx36_xdp_cqes: 0
>> tx37_xdp_xmit: 0
>> tx37_xdp_full: 0
>> tx37_xdp_err: 0
>> tx37_xdp_cqes: 0
>> tx38_xdp_xmit: 0
>> tx38_xdp_full: 0
>> tx38_xdp_err: 0
>> tx38_xdp_cqes: 0
>> tx39_xdp_xmit: 0
>> tx39_xdp_full: 0
>> tx39_xdp_err: 0
>> tx39_xdp_cqes: 0
>> tx40_xdp_xmit: 0
>> tx40_xdp_full: 0
>> tx40_xdp_err: 0
>> tx40_xdp_cqes: 0
>> tx41_xdp_xmit: 0
>> tx41_xdp_full: 0
>> tx41_xdp_err: 0
>> tx41_xdp_cqes: 0
>> tx42_xdp_xmit: 0
>> tx42_xdp_full: 0
>> tx42_xdp_err: 0
>> tx42_xdp_cqes: 0
>> tx43_xdp_xmit: 0
>> tx43_xdp_full: 0
>> tx43_xdp_err: 0
>> tx43_xdp_cqes: 0
>> tx44_xdp_xmit: 0
>> tx44_xdp_full: 0
>> tx44_xdp_err: 0
>> tx44_xdp_cqes: 0
>> tx45_xdp_xmit: 0
>> tx45_xdp_full: 0
>> tx45_xdp_err: 0
>> tx45_xdp_cqes: 0
>> tx46_xdp_xmit: 0
>> tx46_xdp_full: 0
>> tx46_xdp_err: 0
>> tx46_xdp_cqes: 0
>> tx47_xdp_xmit: 0
>> tx47_xdp_full: 0
>> tx47_xdp_err: 0
>> tx47_xdp_cqes: 0
>> tx48_xdp_xmit: 0
>> tx48_xdp_full: 0
>> tx48_xdp_err: 0
>> tx48_xdp_cqes: 0
>> tx49_xdp_xmit: 0
>> tx49_xdp_full: 0
>> tx49_xdp_err: 0
>> tx49_xdp_cqes: 0
>> tx50_xdp_xmit: 0
>> tx50_xdp_full: 0
>> tx50_xdp_err: 0
>> tx50_xdp_cqes: 0
>> tx51_xdp_xmit: 0
>> tx51_xdp_full: 0
>> tx51_xdp_err: 0
>> tx51_xdp_cqes: 0
>> tx52_xdp_xmit: 0
>> tx52_xdp_full: 0
>> tx52_xdp_err: 0
>> tx52_xdp_cqes: 0
>> tx53_xdp_xmit: 0
>> tx53_xdp_full: 0
>> tx53_xdp_err: 0
>> tx53_xdp_cqes: 0
>> tx54_xdp_xmit: 0
>> tx54_xdp_full: 0
>> tx54_xdp_err: 0
>> tx54_xdp_cqes: 0
>> tx55_xdp_xmit: 0
>> tx55_xdp_full: 0
>> tx55_xdp_err: 0
>> tx55_xdp_cqes: 0
>>
>>
>>> [...]
>>>
>>>>>> ethtool -S enp175s0f0
>>>>>> NIC statistics:
>>>>>> rx_packets: 141574897253
>>>>>> rx_bytes: 184445040406258
>>>>>> tx_packets: 172569543894
>>>>>> tx_bytes: 99486882076365
>>>>>> tx_tso_packets: 9367664195
>>>>>> tx_tso_bytes: 56435233992948
>>>>>> tx_tso_inner_packets: 0
>>>>>> tx_tso_inner_bytes: 0
>>>>>> tx_added_vlan_packets: 141297671626
>>>>>> tx_nop: 2102916272
>>>>>> rx_lro_packets: 0
>>>>>> rx_lro_bytes: 0
>>>>>> rx_ecn_mark: 0
>>>>>> rx_removed_vlan_packets: 141574897252
>>>>>> rx_csum_unnecessary: 0
>>>>>> rx_csum_none: 23135854
>>>>>> rx_csum_complete: 141551761398
>>>>>> rx_csum_unnecessary_inner: 0
>>>>>> rx_xdp_drop: 0
>>>>>> rx_xdp_redirect: 0
>>>>>> rx_xdp_tx_xmit: 0
>>>>>> rx_xdp_tx_full: 0
>>>>>> rx_xdp_tx_err: 0
>>>>>> rx_xdp_tx_cqe: 0
>>>>>> tx_csum_none: 127934791664
>>>>> It is a good idea to look into this, tx is not requesting hw tx
>>>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>>>> on
>>>>> calculating csum, or maybe this is just the rx csum complete..
>>>>>
>>>>>> tx_csum_partial: 13362879974
>>>>>> tx_csum_partial_inner: 0
>>>>>> tx_queue_stopped: 232561
>>>>> TX queues are stalling, could be an indentation for the pcie
>>>>> bottelneck.
>>>>>
>>>>>> tx_queue_dropped: 0
>>>>>> tx_xmit_more: 1266021946
>>>>>> tx_recover: 0
>>>>>> tx_cqes: 140031716469
>>>>>> tx_queue_wake: 232561
>>>>>> tx_udp_seg_rem: 0
>>>>>> tx_cqe_err: 0
>>>>>> tx_xdp_xmit: 0
>>>>>> tx_xdp_full: 0
>>>>>> tx_xdp_err: 0
>>>>>> tx_xdp_cqes: 0
>>>>>> rx_wqe_err: 0
>>>>>> rx_mpwqe_filler_cqes: 0
>>>>>> rx_mpwqe_filler_strides: 0
>>>>>> rx_buff_alloc_err: 0
>>>>>> rx_cqe_compress_blks: 0
>>>>>> rx_cqe_compress_pkts: 0
>>>>>> rx_page_reuse: 0
>>>>>> rx_cache_reuse: 16625975793
>>>>>> rx_cache_full: 54161465914
>>>>>> rx_cache_empty: 258048
>>>>>> rx_cache_busy: 54161472735
>>>>>> rx_cache_waive: 0
>>>>>> rx_congst_umr: 0
>>>>>> rx_arfs_err: 0
>>>>>> ch_events: 40572621887
>>>>>> ch_poll: 40885650979
>>>>>> ch_arm: 40429276692
>>>>>> ch_aff_change: 0
>>>>>> ch_eq_rearm: 0
>>>>>> rx_out_of_buffer: 2791690
>>>>>> rx_if_down_packets: 74
>>>>>> rx_vport_unicast_packets: 141843476308
>>>>>> rx_vport_unicast_bytes: 185421265403318
>>>>>> tx_vport_unicast_packets: 172569484005
>>>>>> tx_vport_unicast_bytes: 100019940094298
>>>>>> rx_vport_multicast_packets: 85122935
>>>>>> rx_vport_multicast_bytes: 5761316431
>>>>>> tx_vport_multicast_packets: 6452
>>>>>> tx_vport_multicast_bytes: 643540
>>>>>> rx_vport_broadcast_packets: 22423624
>>>>>> rx_vport_broadcast_bytes: 1390127090
>>>>>> tx_vport_broadcast_packets: 22024
>>>>>> tx_vport_broadcast_bytes: 1321440
>>>>>> rx_vport_rdma_unicast_packets: 0
>>>>>> rx_vport_rdma_unicast_bytes: 0
>>>>>> tx_vport_rdma_unicast_packets: 0
>>>>>> tx_vport_rdma_unicast_bytes: 0
>>>>>> rx_vport_rdma_multicast_packets: 0
>>>>>> rx_vport_rdma_multicast_bytes: 0
>>>>>> tx_vport_rdma_multicast_packets: 0
>>>>>> tx_vport_rdma_multicast_bytes: 0
>>>>>> tx_packets_phy: 172569501577
>>>>>> rx_packets_phy: 142871314588
>>>>>> rx_crc_errors_phy: 0
>>>>>> tx_bytes_phy: 100710212814151
>>>>>> rx_bytes_phy: 187209224289564
>>>>>> tx_multicast_phy: 6452
>>>>>> tx_broadcast_phy: 22024
>>>>>> rx_multicast_phy: 85122933
>>>>>> rx_broadcast_phy: 22423623
>>>>>> rx_in_range_len_errors_phy: 2
>>>>>> rx_out_of_range_len_phy: 0
>>>>>> rx_oversize_pkts_phy: 0
>>>>>> rx_symbol_err_phy: 0
>>>>>> tx_mac_control_phy: 0
>>>>>> rx_mac_control_phy: 0
>>>>>> rx_unsupported_op_phy: 0
>>>>>> rx_pause_ctrl_phy: 0
>>>>>> tx_pause_ctrl_phy: 0
>>>>>> rx_discards_phy: 920161423
>>>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>>>> to
>>>>> the pcie bottleneck.
>>>> Yes this side is receiving more traffic - second port is +10G more tx
>>>>
>>> [...]
>>>
>>>
>>>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>>>> 0.00 0.00 0.00 31.30
>>>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>>>> 0.00 0.00 0.00 24.90
>>>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>>>> 0.00 0.00 0.00 19.68
>>>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>>>> 0.00 0.00 0.00 18.00
>>>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>>>> 0.00 0.00 0.00 17.40
>>>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>>>> 0.00 0.00 0.00 26.03
>>>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>>>> 0.00 0.00 0.00 17.52
>>>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>>>> 0.00 0.00 0.00 18.20
>>>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>>>> 0.00 0.00 0.00 17.68
>>>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>>>> 0.00 0.00 0.00 18.20
>>>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>>>> 0.00 0.00 0.00 20.42
>>>>> The numa cores are not at 100% util, you have around 20% of idle on
>>>>> each one.
>>>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>>>> push
>>>> aditional 1-2Gbit/s
>>>>
>>> yes but, it doens't look like the bottleneck is the cpu, although it is
>>> close to be :)..
>>>
>>>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 96.10
>>>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.70
>>>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.50
>>>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 97.40
>>>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.10
>>>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.40
>>>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>>>> 0.00 0.00 0.00 19.42
>>>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>>>> 0.00 0.00 0.00 26.60
>>>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>>>> 0.00 0.00 0.00 21.60
>>>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>>>> 0.00 0.00 0.00 20.50
>>>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>>>> 0.00 0.00 0.00 21.60
>>>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>>>> 0.00 0.00 0.00 24.58
>>>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>>>> 0.00 0.00 0.00 24.68
>>>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>>>> 0.00 0.00 0.00 28.34
>>>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>>>> 0.00 0.00 0.00 25.83
>>>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>>>> 0.00 0.00 0.00 27.20
>>>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>>>> 0.00 0.00 0.00 28.03
>>>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>>>> 0.00 0.00 0.00 24.70
>>>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>>>> 0.00 0.00 0.00 25.30
>>>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>>>> 0.00 0.00 0.00 26.77
>>>>>>
>>>>>>
>>>>>> ethtool -k enp175s0f0
>>>>>> Features for enp175s0f0:
>>>>>> rx-checksumming: on
>>>>>> tx-checksumming: on
>>>>>> tx-checksum-ipv4: on
>>>>>> tx-checksum-ip-generic: off [fixed]
>>>>>> tx-checksum-ipv6: on
>>>>>> tx-checksum-fcoe-crc: off [fixed]
>>>>>> tx-checksum-sctp: off [fixed]
>>>>>> scatter-gather: on
>>>>>> tx-scatter-gather: on
>>>>>> tx-scatter-gather-fraglist: off [fixed]
>>>>>> tcp-segmentation-offload: on
>>>>>> tx-tcp-segmentation: on
>>>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>>>> tx-tcp-mangleid-segmentation: off
>>>>>> tx-tcp6-segmentation: on
>>>>>> udp-fragmentation-offload: off
>>>>>> generic-segmentation-offload: on
>>>>>> generic-receive-offload: on
>>>>>> large-receive-offload: off [fixed]
>>>>>> rx-vlan-offload: on
>>>>>> tx-vlan-offload: on
>>>>>> ntuple-filters: off
>>>>>> receive-hashing: on
>>>>>> highdma: on [fixed]
>>>>>> rx-vlan-filter: on
>>>>>> vlan-challenged: off [fixed]
>>>>>> tx-lockless: off [fixed]
>>>>>> netns-local: off [fixed]
>>>>>> tx-gso-robust: off [fixed]
>>>>>> tx-fcoe-segmentation: off [fixed]
>>>>>> tx-gre-segmentation: on
>>>>>> tx-gre-csum-segmentation: on
>>>>>> tx-ipxip4-segmentation: off [fixed]
>>>>>> tx-ipxip6-segmentation: off [fixed]
>>>>>> tx-udp_tnl-segmentation: on
>>>>>> tx-udp_tnl-csum-segmentation: on
>>>>>> tx-gso-partial: on
>>>>>> tx-sctp-segmentation: off [fixed]
>>>>>> tx-esp-segmentation: off [fixed]
>>>>>> tx-udp-segmentation: on
>>>>>> fcoe-mtu: off [fixed]
>>>>>> tx-nocache-copy: off
>>>>>> loopback: off [fixed]
>>>>>> rx-fcs: off
>>>>>> rx-all: off
>>>>>> tx-vlan-stag-hw-insert: on
>>>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>>>> rx-vlan-stag-filter: on [fixed]
>>>>>> l2-fwd-offload: off [fixed]
>>>>>> hw-tc-offload: off
>>>>>> esp-hw-offload: off [fixed]
>>>>>> esp-tx-csum-hw-offload: off [fixed]
>>>>>> rx-udp_tunnel-port-offload: on
>>>>>> tls-hw-tx-offload: off [fixed]
>>>>>> tls-hw-rx-offload: off [fixed]
>>>>>> rx-gro-hw: off [fixed]
>>>>>> tls-hw-record: off [fixed]
>>>>>>
>>>>>> ethtool -c enp175s0f0
>>>>>> Coalesce parameters for enp175s0f0:
>>>>>> Adaptive RX: off TX: on
>>>>>> stats-block-usecs: 0
>>>>>> sample-interval: 0
>>>>>> pkt-rate-low: 0
>>>>>> pkt-rate-high: 0
>>>>>> dmac: 32703
>>>>>>
>>>>>> rx-usecs: 256
>>>>>> rx-frames: 128
>>>>>> rx-usecs-irq: 0
>>>>>> rx-frames-irq: 0
>>>>>>
>>>>>> tx-usecs: 8
>>>>>> tx-frames: 128
>>>>>> tx-usecs-irq: 0
>>>>>> tx-frames-irq: 0
>>>>>>
>>>>>> rx-usecs-low: 0
>>>>>> rx-frame-low: 0
>>>>>> tx-usecs-low: 0
>>>>>> tx-frame-low: 0
>>>>>>
>>>>>> rx-usecs-high: 0
>>>>>> rx-frame-high: 0
>>>>>> tx-usecs-high: 0
>>>>>> tx-frame-high: 0
>>>>>>
>>>>>> ethtool -g enp175s0f0
>>>>>> Ring parameters for enp175s0f0:
>>>>>> Pre-set maximums:
>>>>>> RX: 8192
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: 8192
>>>>>> Current hardware settings:
>>>>>> RX: 4096
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: 4096
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> Also changed a little coalesce params - and best for this config are:
>>>> ethtool -c enp175s0f0
>>>> Coalesce parameters for enp175s0f0:
>>>> Adaptive RX: off TX: off
>>>> stats-block-usecs: 0
>>>> sample-interval: 0
>>>> pkt-rate-low: 0
>>>> pkt-rate-high: 0
>>>> dmac: 32573
>>>>
>>>> rx-usecs: 40
>>>> rx-frames: 128
>>>> rx-usecs-irq: 0
>>>> rx-frames-irq: 0
>>>>
>>>> tx-usecs: 8
>>>> tx-frames: 8
>>>> tx-usecs-irq: 0
>>>> tx-frames-irq: 0
>>>>
>>>> rx-usecs-low: 0
>>>> rx-frame-low: 0
>>>> tx-usecs-low: 0
>>>> tx-frame-low: 0
>>>>
>>>> rx-usecs-high: 0
>>>> rx-frame-high: 0
>>>> tx-usecs-high: 0
>>>> tx-frame-high: 0
>>>>
>>>>
>>>> Less drops on RX side - and more pps in overall forwarded.
>>>>
>>> how much improvement ? maybe we can improve our adaptive rx coal to be
>>> efficient for this work load.
>>>
>>>
>> I can prepare more stats with ethtool maybee to compare - but
>> normally tested with simple icmp forwarded from interface to interface
>> - before change coalescence params:
>> adaptive-rx off rx-usecs 384 rx-frames 128
>> 3% loss for icmp
>> - after change to:
>> adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8
>> tx-frames 8
>> 2% loss for icmp
>>
>> But yes - to know better will need to compare rx/tx counters from
>> ethtool + /proc/net/dev
>>
>>
>> Was trying to turn on adaptative-tx+rx - but 100% saturation at
>> 43Gbit/s RX / 43Gbit/s TX
>>
>>
>>
>>
>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 20:23 ` Saeed Mahameed
@ 2018-11-02 5:23 ` Aaron Lu
2018-11-02 11:40 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 77+ messages in thread
From: Aaron Lu @ 2018-11-02 5:23 UTC (permalink / raw)
To: Saeed Mahameed
Cc: brouer, pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> > On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> > wrote:
> > ... ...
> > > Section copied out:
> > >
> > > mlx5e_poll_tx_cq
> > > |
> > > --16.34%--napi_consume_skb
> > > |
> > > |--12.65%--__free_pages_ok
> > > | |
> > > | --11.86%--free_one_page
> > > | |
> > > | |--10.10%
> > > --queued_spin_lock_slowpath
> > > | |
> > > | --0.65%--_raw_spin_lock
> >
> > This callchain looks like it is freeing higher order pages than order
> > 0:
> > __free_pages_ok is only called for pages whose order are bigger than
> > 0.
>
> mlx5 rx uses only order 0 pages, so i don't know where these high order
> tx SKBs are coming from..
Perhaps here:
__netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
__napi_alloc_frag() will all call page_frag_alloc(), which will use
__page_frag_cache_refill() to get an order 3 page if possible, or fall
back to an order 0 page if order 3 page is not available.
I'm not sure if your workload will use the above code path though.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-02 5:23 ` Aaron Lu
@ 2018-11-02 11:40 ` Jesper Dangaard Brouer
2018-11-02 14:20 ` Aaron Lu
0 siblings, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-02 11:40 UTC (permalink / raw)
To: Aaron Lu
Cc: Saeed Mahameed, pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman, brouer
On Fri, 2 Nov 2018 13:23:56 +0800
Aaron Lu <aaron.lu@intel.com> wrote:
> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> > On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> > > On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> > > wrote:
> > > ... ...
> > > > Section copied out:
> > > >
> > > > mlx5e_poll_tx_cq
> > > > |
> > > > --16.34%--napi_consume_skb
> > > > |
> > > > |--12.65%--__free_pages_ok
> > > > | |
> > > > | --11.86%--free_one_page
> > > > | |
> > > > | |--10.10%
> > > > --queued_spin_lock_slowpath
> > > > | |
> > > > | --0.65%--_raw_spin_lock
> > >
> > > This callchain looks like it is freeing higher order pages than order
> > > 0:
> > > __free_pages_ok is only called for pages whose order are bigger than
> > > 0.
> >
> > mlx5 rx uses only order 0 pages, so i don't know where these high order
> > tx SKBs are coming from..
>
> Perhaps here:
> __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
> __napi_alloc_frag() will all call page_frag_alloc(), which will use
> __page_frag_cache_refill() to get an order 3 page if possible, or fall
> back to an order 0 page if order 3 page is not available.
>
> I'm not sure if your workload will use the above code path though.
TL;DR: this is order-0 pages (code-walk trough proof below)
To Aaron, the network stack *can* call __free_pages_ok() with order-0
pages, via:
static void skb_free_head(struct sk_buff *skb)
{
unsigned char *head = skb->head;
if (skb->head_frag)
skb_free_frag(head);
else
kfree(head);
}
static inline void skb_free_frag(void *addr)
{
page_frag_free(addr);
}
/*
* Frees a page fragment allocated out of either a compound or order 0 page.
*/
void page_frag_free(void *addr)
{
struct page *page = virt_to_head_page(addr);
if (unlikely(put_page_testzero(page)))
__free_pages_ok(page, compound_order(page));
}
EXPORT_SYMBOL(page_frag_free);
Notice for the mlx5 driver it support several RX-memory models, so it
can be hard to follow, but from the perf report output we can see that
is uses mlx5e_skb_from_cqe_linear, which use build_skb.
--13.63%--mlx5e_skb_from_cqe_linear
|
--5.02%--build_skb
|
--1.85%--__build_skb
|
--1.00%--kmem_cache_alloc
/* build_skb() is wrapper over __build_skb(), that specifically
* takes care of skb->head and skb->pfmemalloc
* This means that if @frag_size is not zero, then @data must be backed
* by a page fragment, not kmalloc() or vmalloc()
*/
struct sk_buff *build_skb(void *data, unsigned int frag_size)
{
struct sk_buff *skb = __build_skb(data, frag_size);
if (skb && frag_size) {
skb->head_frag = 1;
if (page_is_pfmemalloc(virt_to_head_page(data)))
skb->pfmemalloc = 1;
}
return skb;
}
EXPORT_SYMBOL(build_skb);
It still doesn't prove, that the @data is backed by by a order-0 page.
For the mlx5 driver is uses mlx5e_page_alloc_mapped ->
page_pool_dev_alloc_pages(), and I can see perf report using
__page_pool_alloc_pages_slow().
The setup for page_pool in mlx5 uses order=0.
/* Create a page_pool and register it with rxq */
pp_params.order = 0;
pp_params.flags = 0; /* No-internal DMA mapping in page_pool */
pp_params.pool_size = pool_size;
pp_params.nid = cpu_to_node(c->cpu);
pp_params.dev = c->pdev;
pp_params.dma_dir = rq->buff.map_dir;
/* page_pool can be used even when there is no rq->xdp_prog,
* given page_pool does not handle DMA mapping there is no
* required state to clear. And page_pool gracefully handle
* elevated refcnt.
*/
rq->page_pool = page_pool_create(&pp_params);
if (IS_ERR(rq->page_pool)) {
err = PTR_ERR(rq->page_pool);
rq->page_pool = NULL;
goto err_free;
}
err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
MEM_TYPE_PAGE_POOL, rq->page_pool);
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-02 11:40 ` Jesper Dangaard Brouer
@ 2018-11-02 14:20 ` Aaron Lu
2018-11-02 19:02 ` Paweł Staszewski
2018-11-03 12:53 ` Jesper Dangaard Brouer
0 siblings, 2 replies; 77+ messages in thread
From: Aaron Lu @ 2018-11-02 14:20 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Saeed Mahameed, pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
> On Fri, 2 Nov 2018 13:23:56 +0800
> Aaron Lu <aaron.lu@intel.com> wrote:
>
> > On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> > > On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> > > > On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> > > > wrote:
> > > > ... ...
> > > > > Section copied out:
> > > > >
> > > > > mlx5e_poll_tx_cq
> > > > > |
> > > > > --16.34%--napi_consume_skb
> > > > > |
> > > > > |--12.65%--__free_pages_ok
> > > > > | |
> > > > > | --11.86%--free_one_page
> > > > > | |
> > > > > | |--10.10%
> > > > > --queued_spin_lock_slowpath
> > > > > | |
> > > > > | --0.65%--_raw_spin_lock
> > > >
> > > > This callchain looks like it is freeing higher order pages than order
> > > > 0:
> > > > __free_pages_ok is only called for pages whose order are bigger than
> > > > 0.
> > >
> > > mlx5 rx uses only order 0 pages, so i don't know where these high order
> > > tx SKBs are coming from..
> >
> > Perhaps here:
> > __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
> > __napi_alloc_frag() will all call page_frag_alloc(), which will use
> > __page_frag_cache_refill() to get an order 3 page if possible, or fall
> > back to an order 0 page if order 3 page is not available.
> >
> > I'm not sure if your workload will use the above code path though.
>
> TL;DR: this is order-0 pages (code-walk trough proof below)
>
> To Aaron, the network stack *can* call __free_pages_ok() with order-0
> pages, via:
>
> static void skb_free_head(struct sk_buff *skb)
> {
> unsigned char *head = skb->head;
>
> if (skb->head_frag)
> skb_free_frag(head);
> else
> kfree(head);
> }
>
> static inline void skb_free_frag(void *addr)
> {
> page_frag_free(addr);
> }
>
> /*
> * Frees a page fragment allocated out of either a compound or order 0 page.
> */
> void page_frag_free(void *addr)
> {
> struct page *page = virt_to_head_page(addr);
>
> if (unlikely(put_page_testzero(page)))
> __free_pages_ok(page, compound_order(page));
> }
> EXPORT_SYMBOL(page_frag_free);
I think here is a problem - order 0 pages are freed directly to buddy,
bypassing per-cpu-pages. This might be the reason lock contention
appeared on free path. Can someone apply below diff and see if lock
contention is gone?
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index e2ef1c17942f..65c0ae13215a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
{
struct page *page = virt_to_head_page(addr);
- if (unlikely(put_page_testzero(page)))
- __free_pages_ok(page, compound_order(page));
+ if (unlikely(put_page_testzero(page))) {
+ unsigned int order = compound_order(page);
+
+ if (order == 0)
+ free_unref_page(page);
+ else
+ __free_pages_ok(page, order);
+ }
}
EXPORT_SYMBOL(page_frag_free);
> Notice for the mlx5 driver it support several RX-memory models, so it
> can be hard to follow, but from the perf report output we can see that
> is uses mlx5e_skb_from_cqe_linear, which use build_skb.
>
> --13.63%--mlx5e_skb_from_cqe_linear
> |
> --5.02%--build_skb
> |
> --1.85%--__build_skb
> |
> --1.00%--kmem_cache_alloc
>
> /* build_skb() is wrapper over __build_skb(), that specifically
> * takes care of skb->head and skb->pfmemalloc
> * This means that if @frag_size is not zero, then @data must be backed
> * by a page fragment, not kmalloc() or vmalloc()
> */
> struct sk_buff *build_skb(void *data, unsigned int frag_size)
> {
> struct sk_buff *skb = __build_skb(data, frag_size);
>
> if (skb && frag_size) {
> skb->head_frag = 1;
> if (page_is_pfmemalloc(virt_to_head_page(data)))
> skb->pfmemalloc = 1;
> }
> return skb;
> }
> EXPORT_SYMBOL(build_skb);
>
> It still doesn't prove, that the @data is backed by by a order-0 page.
> For the mlx5 driver is uses mlx5e_page_alloc_mapped ->
> page_pool_dev_alloc_pages(), and I can see perf report using
> __page_pool_alloc_pages_slow().
>
> The setup for page_pool in mlx5 uses order=0.
>
> /* Create a page_pool and register it with rxq */
> pp_params.order = 0;
> pp_params.flags = 0; /* No-internal DMA mapping in page_pool */
> pp_params.pool_size = pool_size;
> pp_params.nid = cpu_to_node(c->cpu);
> pp_params.dev = c->pdev;
> pp_params.dma_dir = rq->buff.map_dir;
>
> /* page_pool can be used even when there is no rq->xdp_prog,
> * given page_pool does not handle DMA mapping there is no
> * required state to clear. And page_pool gracefully handle
> * elevated refcnt.
> */
> rq->page_pool = page_pool_create(&pp_params);
> if (IS_ERR(rq->page_pool)) {
> err = PTR_ERR(rq->page_pool);
> rq->page_pool = NULL;
> goto err_free;
> }
> err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
> MEM_TYPE_PAGE_POOL, rq->page_pool);
Thanks for the detailed analysis, I'll need more time to understand the
whole picture :-)
^ permalink raw reply related [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-02 14:20 ` Aaron Lu
@ 2018-11-02 19:02 ` Paweł Staszewski
2018-11-03 0:16 ` Paweł Staszewski
2018-11-03 12:53 ` Jesper Dangaard Brouer
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-02 19:02 UTC (permalink / raw)
To: Aaron Lu, Jesper Dangaard Brouer
Cc: Saeed Mahameed, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
W dniu 02.11.2018 o 15:20, Aaron Lu pisze:
> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
>> On Fri, 2 Nov 2018 13:23:56 +0800
>> Aaron Lu <aaron.lu@intel.com> wrote:
>>
>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
>>>>> wrote:
>>>>> ... ...
>>>>>> Section copied out:
>>>>>>
>>>>>> mlx5e_poll_tx_cq
>>>>>> |
>>>>>> --16.34%--napi_consume_skb
>>>>>> |
>>>>>> |--12.65%--__free_pages_ok
>>>>>> | |
>>>>>> | --11.86%--free_one_page
>>>>>> | |
>>>>>> | |--10.10%
>>>>>> --queued_spin_lock_slowpath
>>>>>> | |
>>>>>> | --0.65%--_raw_spin_lock
>>>>> This callchain looks like it is freeing higher order pages than order
>>>>> 0:
>>>>> __free_pages_ok is only called for pages whose order are bigger than
>>>>> 0.
>>>> mlx5 rx uses only order 0 pages, so i don't know where these high order
>>>> tx SKBs are coming from..
>>> Perhaps here:
>>> __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
>>> __napi_alloc_frag() will all call page_frag_alloc(), which will use
>>> __page_frag_cache_refill() to get an order 3 page if possible, or fall
>>> back to an order 0 page if order 3 page is not available.
>>>
>>> I'm not sure if your workload will use the above code path though.
>> TL;DR: this is order-0 pages (code-walk trough proof below)
>>
>> To Aaron, the network stack *can* call __free_pages_ok() with order-0
>> pages, via:
>>
>> static void skb_free_head(struct sk_buff *skb)
>> {
>> unsigned char *head = skb->head;
>>
>> if (skb->head_frag)
>> skb_free_frag(head);
>> else
>> kfree(head);
>> }
>>
>> static inline void skb_free_frag(void *addr)
>> {
>> page_frag_free(addr);
>> }
>>
>> /*
>> * Frees a page fragment allocated out of either a compound or order 0 page.
>> */
>> void page_frag_free(void *addr)
>> {
>> struct page *page = virt_to_head_page(addr);
>>
>> if (unlikely(put_page_testzero(page)))
>> __free_pages_ok(page, compound_order(page));
>> }
>> EXPORT_SYMBOL(page_frag_free);
> I think here is a problem - order 0 pages are freed directly to buddy,
> bypassing per-cpu-pages. This might be the reason lock contention
> appeared on free path. Can someone apply below diff and see if lock
> contention is gone?
Will test it tonight
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index e2ef1c17942f..65c0ae13215a 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
> {
> struct page *page = virt_to_head_page(addr);
>
> - if (unlikely(put_page_testzero(page)))
> - __free_pages_ok(page, compound_order(page));
> + if (unlikely(put_page_testzero(page))) {
> + unsigned int order = compound_order(page);
> +
> + if (order == 0)
> + free_unref_page(page);
> + else
> + __free_pages_ok(page, order);
> + }
> }
> EXPORT_SYMBOL(page_frag_free);
>
>> Notice for the mlx5 driver it support several RX-memory models, so it
>> can be hard to follow, but from the perf report output we can see that
>> is uses mlx5e_skb_from_cqe_linear, which use build_skb.
>>
>> --13.63%--mlx5e_skb_from_cqe_linear
>> |
>> --5.02%--build_skb
>> |
>> --1.85%--__build_skb
>> |
>> --1.00%--kmem_cache_alloc
>>
>> /* build_skb() is wrapper over __build_skb(), that specifically
>> * takes care of skb->head and skb->pfmemalloc
>> * This means that if @frag_size is not zero, then @data must be backed
>> * by a page fragment, not kmalloc() or vmalloc()
>> */
>> struct sk_buff *build_skb(void *data, unsigned int frag_size)
>> {
>> struct sk_buff *skb = __build_skb(data, frag_size);
>>
>> if (skb && frag_size) {
>> skb->head_frag = 1;
>> if (page_is_pfmemalloc(virt_to_head_page(data)))
>> skb->pfmemalloc = 1;
>> }
>> return skb;
>> }
>> EXPORT_SYMBOL(build_skb);
>>
>> It still doesn't prove, that the @data is backed by by a order-0 page.
>> For the mlx5 driver is uses mlx5e_page_alloc_mapped ->
>> page_pool_dev_alloc_pages(), and I can see perf report using
>> __page_pool_alloc_pages_slow().
>>
>> The setup for page_pool in mlx5 uses order=0.
>>
>> /* Create a page_pool and register it with rxq */
>> pp_params.order = 0;
>> pp_params.flags = 0; /* No-internal DMA mapping in page_pool */
>> pp_params.pool_size = pool_size;
>> pp_params.nid = cpu_to_node(c->cpu);
>> pp_params.dev = c->pdev;
>> pp_params.dma_dir = rq->buff.map_dir;
>>
>> /* page_pool can be used even when there is no rq->xdp_prog,
>> * given page_pool does not handle DMA mapping there is no
>> * required state to clear. And page_pool gracefully handle
>> * elevated refcnt.
>> */
>> rq->page_pool = page_pool_create(&pp_params);
>> if (IS_ERR(rq->page_pool)) {
>> err = PTR_ERR(rq->page_pool);
>> rq->page_pool = NULL;
>> goto err_free;
>> }
>> err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
>> MEM_TYPE_PAGE_POOL, rq->page_pool);
> Thanks for the detailed analysis, I'll need more time to understand the
> whole picture :-)
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-02 19:02 ` Paweł Staszewski
@ 2018-11-03 0:16 ` Paweł Staszewski
2018-11-03 12:01 ` Paweł Staszewski
2018-11-03 12:58 ` Jesper Dangaard Brouer
0 siblings, 2 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-03 0:16 UTC (permalink / raw)
To: Aaron Lu, Jesper Dangaard Brouer
Cc: Saeed Mahameed, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
W dniu 02.11.2018 o 20:02, Paweł Staszewski pisze:
>
>
> W dniu 02.11.2018 o 15:20, Aaron Lu pisze:
>> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
>>> On Fri, 2 Nov 2018 13:23:56 +0800
>>> Aaron Lu <aaron.lu@intel.com> wrote:
>>>
>>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
>>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
>>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
>>>>>> wrote:
>>>>>> ... ...
>>>>>>> Section copied out:
>>>>>>>
>>>>>>> mlx5e_poll_tx_cq
>>>>>>> |
>>>>>>> --16.34%--napi_consume_skb
>>>>>>> |
>>>>>>> |--12.65%--__free_pages_ok
>>>>>>> | |
>>>>>>> | --11.86%--free_one_page
>>>>>>> | |
>>>>>>> | |--10.10%
>>>>>>> --queued_spin_lock_slowpath
>>>>>>> | |
>>>>>>> | --0.65%--_raw_spin_lock
>>>>>> This callchain looks like it is freeing higher order pages than
>>>>>> order
>>>>>> 0:
>>>>>> __free_pages_ok is only called for pages whose order are bigger than
>>>>>> 0.
>>>>> mlx5 rx uses only order 0 pages, so i don't know where these high
>>>>> order
>>>>> tx SKBs are coming from..
>>>> Perhaps here:
>>>> __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
>>>> __napi_alloc_frag() will all call page_frag_alloc(), which will use
>>>> __page_frag_cache_refill() to get an order 3 page if possible, or fall
>>>> back to an order 0 page if order 3 page is not available.
>>>>
>>>> I'm not sure if your workload will use the above code path though.
>>> TL;DR: this is order-0 pages (code-walk trough proof below)
>>>
>>> To Aaron, the network stack *can* call __free_pages_ok() with order-0
>>> pages, via:
>>>
>>> static void skb_free_head(struct sk_buff *skb)
>>> {
>>> unsigned char *head = skb->head;
>>>
>>> if (skb->head_frag)
>>> skb_free_frag(head);
>>> else
>>> kfree(head);
>>> }
>>>
>>> static inline void skb_free_frag(void *addr)
>>> {
>>> page_frag_free(addr);
>>> }
>>>
>>> /*
>>> * Frees a page fragment allocated out of either a compound or
>>> order 0 page.
>>> */
>>> void page_frag_free(void *addr)
>>> {
>>> struct page *page = virt_to_head_page(addr);
>>>
>>> if (unlikely(put_page_testzero(page)))
>>> __free_pages_ok(page, compound_order(page));
>>> }
>>> EXPORT_SYMBOL(page_frag_free);
>> I think here is a problem - order 0 pages are freed directly to buddy,
>> bypassing per-cpu-pages. This might be the reason lock contention
>> appeared on free path. Can someone apply below diff and see if lock
>> contention is gone?
> Will test it tonight
>
Patch applied
perf report:
https://ufile.io/sytfh
But i need to wait also with more traffic currently cpu's are sleeping
>
>
>>
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index e2ef1c17942f..65c0ae13215a 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
>> {
>> struct page *page = virt_to_head_page(addr);
>> - if (unlikely(put_page_testzero(page)))
>> - __free_pages_ok(page, compound_order(page));
>> + if (unlikely(put_page_testzero(page))) {
>> + unsigned int order = compound_order(page);
>> +
>> + if (order == 0)
>> + free_unref_page(page);
>> + else
>> + __free_pages_ok(page, order);
>> + }
>> }
>> EXPORT_SYMBOL(page_frag_free);
>>> Notice for the mlx5 driver it support several RX-memory models, so it
>>> can be hard to follow, but from the perf report output we can see that
>>> is uses mlx5e_skb_from_cqe_linear, which use build_skb.
>>>
>>> --13.63%--mlx5e_skb_from_cqe_linear
>>> |
>>> --5.02%--build_skb
>>> |
>>> --1.85%--__build_skb
>>> |
>>> --1.00%--kmem_cache_alloc
>>>
>>> /* build_skb() is wrapper over __build_skb(), that specifically
>>> * takes care of skb->head and skb->pfmemalloc
>>> * This means that if @frag_size is not zero, then @data must be
>>> backed
>>> * by a page fragment, not kmalloc() or vmalloc()
>>> */
>>> struct sk_buff *build_skb(void *data, unsigned int frag_size)
>>> {
>>> struct sk_buff *skb = __build_skb(data, frag_size);
>>>
>>> if (skb && frag_size) {
>>> skb->head_frag = 1;
>>> if (page_is_pfmemalloc(virt_to_head_page(data)))
>>> skb->pfmemalloc = 1;
>>> }
>>> return skb;
>>> }
>>> EXPORT_SYMBOL(build_skb);
>>>
>>> It still doesn't prove, that the @data is backed by by a order-0 page.
>>> For the mlx5 driver is uses mlx5e_page_alloc_mapped ->
>>> page_pool_dev_alloc_pages(), and I can see perf report using
>>> __page_pool_alloc_pages_slow().
>>>
>>> The setup for page_pool in mlx5 uses order=0.
>>>
>>> /* Create a page_pool and register it with rxq */
>>> pp_params.order = 0;
>>> pp_params.flags = 0; /* No-internal DMA mapping in page_pool */
>>> pp_params.pool_size = pool_size;
>>> pp_params.nid = cpu_to_node(c->cpu);
>>> pp_params.dev = c->pdev;
>>> pp_params.dma_dir = rq->buff.map_dir;
>>>
>>> /* page_pool can be used even when there is no rq->xdp_prog,
>>> * given page_pool does not handle DMA mapping there is no
>>> * required state to clear. And page_pool gracefully handle
>>> * elevated refcnt.
>>> */
>>> rq->page_pool = page_pool_create(&pp_params);
>>> if (IS_ERR(rq->page_pool)) {
>>> err = PTR_ERR(rq->page_pool);
>>> rq->page_pool = NULL;
>>> goto err_free;
>>> }
>>> err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
>>> MEM_TYPE_PAGE_POOL, rq->page_pool);
>> Thanks for the detailed analysis, I'll need more time to understand the
>> whole picture :-)
>>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 20:37 ` Saeed Mahameed
2018-11-01 21:18 ` Paweł Staszewski
@ 2018-11-03 0:18 ` Paweł Staszewski
2018-11-08 19:12 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-03 0:18 UTC (permalink / raw)
To: Saeed Mahameed, netdev
W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>> Hi
>>>>
>>>> So maybee someone will be interested how linux kernel handles
>>>> normal
>>>> traffic (not pktgen :) )
>>>>
>>>>
>>>> Server HW configuration:
>>>>
>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>
>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>
>>>>
>>>> Server software:
>>>>
>>>> FRR - as routing daemon
>>>>
>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>> local
>>>> numa
>>>> node)
>>>>
>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>> numa
>>>> node)
>>>>
>>>>
>>>> Maximum traffic that server can handle:
>>>>
>>>> Bandwidth
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> \ iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>> Gb/s
>>>> 65.74 Gb/s
>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>> Gb/s
>>>> 66.51 Gb/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 66.58 Gb/s 65.67
>>>> Gb/s
>>>> 132.25 Gb/s
>>>>
>>>>
>>>> Packets per second:
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>> 8735207.00 P/s
>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>> 8790460.00 P/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>> 17525668.00 P/s
>>>>
>>>>
>>>> After reaching that limits nics on the upstream side (more RX
>>>> traffic)
>>>> start to drop packets
>>>>
>>>>
>>>> I just dont understand that server can't handle more bandwidth
>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>> RX
>>>> side are increasing.
>>>>
>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>> NIC (
>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>> 132.25
>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>> hmm yes that was my concern also - cause cant find anywhere
>> informations
>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>> 8GT
>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>> bw
>> on both ports
> i think it is bidir
So yes - we are hitting there other problem i think pcie is most
probabbly bidirectional max bw 126Gbit so RX 126Gbit and at same time TX
should be 126Gbit
>> This can explain maybee also why cpuload is rising rapidly from
>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>> so
>> there can be some error in reading them when offloading (gro/gso/tso)
>> on
>> nic's is enabled that is why
>>
>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>> 8GT
>>>> is
>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>> and
>>>> pps
>>>> (like 4x more comparing to normal internet traffic)
>>>>
>>> Are you forwarding when using pktgen as well or you just testing
>>> the RX
>>> side pps ?
>> Yes pktgen was tested on single port RX
>> Can check also forwarding to eliminate pciex limits
>>
> So this explains why you have more RX pps, since tx is idle and pcie
> will be free to do only rx.
>
> [...]
>
>
>>>> ethtool -S enp175s0f1
>>>> NIC statistics:
>>>> rx_packets: 173730800927
>>>> rx_bytes: 99827422751332
>>>> tx_packets: 142532009512
>>>> tx_bytes: 184633045911222
>>>> tx_tso_packets: 25989113891
>>>> tx_tso_bytes: 132933363384458
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 74630239613
>>>> tx_nop: 2029817748
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 173730800927
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 434357
>>>> rx_csum_complete: 173730366570
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 38260960853
>>>> tx_csum_partial: 36369278774
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 1
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 748638099
>>>> tx_recover: 0
>>>> tx_cqes: 73881645031
>>>> tx_queue_wake: 1
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>> If this is a pcie bottleneck it might be useful to enable CQE
>>> compression (to reduce PCIe completion descriptors transactions)
>>> you should see the above rx_cqe_compress_pkts increasing when
>>> enabled.
>>>
>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>> $ ethtool --show-priv-flags enp175s0f1
>>> Private flags for p6p1:
>>> rx_cqe_moder : on
>>> cqe_moder : off
>>> rx_cqe_compress : on
>>> ...
>>>
>>> try this on both interfaces.
>> Done
>> ethtool --show-priv-flags enp175s0f1
>> Private flags for enp175s0f1:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
>> ethtool --show-priv-flags enp175s0f0
>> Private flags for enp175s0f0:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
> did it help reduce the load on the pcie ? do you see more pps ?
> what is the ratio between rx_cqe_compress_pkts and over all rx packets
> ?
>
> [...]
>
>>>> ethtool -S enp175s0f0
>>>> NIC statistics:
>>>> rx_packets: 141574897253
>>>> rx_bytes: 184445040406258
>>>> tx_packets: 172569543894
>>>> tx_bytes: 99486882076365
>>>> tx_tso_packets: 9367664195
>>>> tx_tso_bytes: 56435233992948
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 141297671626
>>>> tx_nop: 2102916272
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 141574897252
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 23135854
>>>> rx_csum_complete: 141551761398
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 127934791664
>>> It is a good idea to look into this, tx is not requesting hw tx
>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>> on
>>> calculating csum, or maybe this is just the rx csum complete..
>>>
>>>> tx_csum_partial: 13362879974
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 232561
>>> TX queues are stalling, could be an indentation for the pcie
>>> bottelneck.
>>>
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 1266021946
>>>> tx_recover: 0
>>>> tx_cqes: 140031716469
>>>> tx_queue_wake: 232561
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>>> rx_page_reuse: 0
>>>> rx_cache_reuse: 16625975793
>>>> rx_cache_full: 54161465914
>>>> rx_cache_empty: 258048
>>>> rx_cache_busy: 54161472735
>>>> rx_cache_waive: 0
>>>> rx_congst_umr: 0
>>>> rx_arfs_err: 0
>>>> ch_events: 40572621887
>>>> ch_poll: 40885650979
>>>> ch_arm: 40429276692
>>>> ch_aff_change: 0
>>>> ch_eq_rearm: 0
>>>> rx_out_of_buffer: 2791690
>>>> rx_if_down_packets: 74
>>>> rx_vport_unicast_packets: 141843476308
>>>> rx_vport_unicast_bytes: 185421265403318
>>>> tx_vport_unicast_packets: 172569484005
>>>> tx_vport_unicast_bytes: 100019940094298
>>>> rx_vport_multicast_packets: 85122935
>>>> rx_vport_multicast_bytes: 5761316431
>>>> tx_vport_multicast_packets: 6452
>>>> tx_vport_multicast_bytes: 643540
>>>> rx_vport_broadcast_packets: 22423624
>>>> rx_vport_broadcast_bytes: 1390127090
>>>> tx_vport_broadcast_packets: 22024
>>>> tx_vport_broadcast_bytes: 1321440
>>>> rx_vport_rdma_unicast_packets: 0
>>>> rx_vport_rdma_unicast_bytes: 0
>>>> tx_vport_rdma_unicast_packets: 0
>>>> tx_vport_rdma_unicast_bytes: 0
>>>> rx_vport_rdma_multicast_packets: 0
>>>> rx_vport_rdma_multicast_bytes: 0
>>>> tx_vport_rdma_multicast_packets: 0
>>>> tx_vport_rdma_multicast_bytes: 0
>>>> tx_packets_phy: 172569501577
>>>> rx_packets_phy: 142871314588
>>>> rx_crc_errors_phy: 0
>>>> tx_bytes_phy: 100710212814151
>>>> rx_bytes_phy: 187209224289564
>>>> tx_multicast_phy: 6452
>>>> tx_broadcast_phy: 22024
>>>> rx_multicast_phy: 85122933
>>>> rx_broadcast_phy: 22423623
>>>> rx_in_range_len_errors_phy: 2
>>>> rx_out_of_range_len_phy: 0
>>>> rx_oversize_pkts_phy: 0
>>>> rx_symbol_err_phy: 0
>>>> tx_mac_control_phy: 0
>>>> rx_mac_control_phy: 0
>>>> rx_unsupported_op_phy: 0
>>>> rx_pause_ctrl_phy: 0
>>>> tx_pause_ctrl_phy: 0
>>>> rx_discards_phy: 920161423
>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>> to
>>> the pcie bottleneck.
>> Yes this side is receiving more traffic - second port is +10G more tx
>>
> [...]
>
>
>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>> 0.00 0.00 0.00 31.30
>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>> 0.00 0.00 0.00 24.90
>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>> 0.00 0.00 0.00 19.68
>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>> 0.00 0.00 0.00 18.00
>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>> 0.00 0.00 0.00 17.40
>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>> 0.00 0.00 0.00 26.03
>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>> 0.00 0.00 0.00 17.52
>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>> 0.00 0.00 0.00 17.68
>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>> 0.00 0.00 0.00 20.42
>>> The numa cores are not at 100% util, you have around 20% of idle on
>>> each one.
>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>> push
>> aditional 1-2Gbit/s
>>
> yes but, it doens't look like the bottleneck is the cpu, although it is
> close to be :)..
>
>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 96.10
>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.70
>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.50
>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 97.40
>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.10
>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.40
>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>> 0.00 0.00 0.00 19.42
>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>> 0.00 0.00 0.00 26.60
>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>> 0.00 0.00 0.00 20.50
>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>> 0.00 0.00 0.00 24.58
>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>> 0.00 0.00 0.00 24.68
>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>> 0.00 0.00 0.00 28.34
>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>> 0.00 0.00 0.00 25.83
>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>> 0.00 0.00 0.00 27.20
>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>> 0.00 0.00 0.00 28.03
>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>> 0.00 0.00 0.00 24.70
>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>> 0.00 0.00 0.00 25.30
>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>> 0.00 0.00 0.00 26.77
>>>>
>>>>
>>>> ethtool -k enp175s0f0
>>>> Features for enp175s0f0:
>>>> rx-checksumming: on
>>>> tx-checksumming: on
>>>> tx-checksum-ipv4: on
>>>> tx-checksum-ip-generic: off [fixed]
>>>> tx-checksum-ipv6: on
>>>> tx-checksum-fcoe-crc: off [fixed]
>>>> tx-checksum-sctp: off [fixed]
>>>> scatter-gather: on
>>>> tx-scatter-gather: on
>>>> tx-scatter-gather-fraglist: off [fixed]
>>>> tcp-segmentation-offload: on
>>>> tx-tcp-segmentation: on
>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>> tx-tcp-mangleid-segmentation: off
>>>> tx-tcp6-segmentation: on
>>>> udp-fragmentation-offload: off
>>>> generic-segmentation-offload: on
>>>> generic-receive-offload: on
>>>> large-receive-offload: off [fixed]
>>>> rx-vlan-offload: on
>>>> tx-vlan-offload: on
>>>> ntuple-filters: off
>>>> receive-hashing: on
>>>> highdma: on [fixed]
>>>> rx-vlan-filter: on
>>>> vlan-challenged: off [fixed]
>>>> tx-lockless: off [fixed]
>>>> netns-local: off [fixed]
>>>> tx-gso-robust: off [fixed]
>>>> tx-fcoe-segmentation: off [fixed]
>>>> tx-gre-segmentation: on
>>>> tx-gre-csum-segmentation: on
>>>> tx-ipxip4-segmentation: off [fixed]
>>>> tx-ipxip6-segmentation: off [fixed]
>>>> tx-udp_tnl-segmentation: on
>>>> tx-udp_tnl-csum-segmentation: on
>>>> tx-gso-partial: on
>>>> tx-sctp-segmentation: off [fixed]
>>>> tx-esp-segmentation: off [fixed]
>>>> tx-udp-segmentation: on
>>>> fcoe-mtu: off [fixed]
>>>> tx-nocache-copy: off
>>>> loopback: off [fixed]
>>>> rx-fcs: off
>>>> rx-all: off
>>>> tx-vlan-stag-hw-insert: on
>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>> rx-vlan-stag-filter: on [fixed]
>>>> l2-fwd-offload: off [fixed]
>>>> hw-tc-offload: off
>>>> esp-hw-offload: off [fixed]
>>>> esp-tx-csum-hw-offload: off [fixed]
>>>> rx-udp_tunnel-port-offload: on
>>>> tls-hw-tx-offload: off [fixed]
>>>> tls-hw-rx-offload: off [fixed]
>>>> rx-gro-hw: off [fixed]
>>>> tls-hw-record: off [fixed]
>>>>
>>>> ethtool -c enp175s0f0
>>>> Coalesce parameters for enp175s0f0:
>>>> Adaptive RX: off TX: on
>>>> stats-block-usecs: 0
>>>> sample-interval: 0
>>>> pkt-rate-low: 0
>>>> pkt-rate-high: 0
>>>> dmac: 32703
>>>>
>>>> rx-usecs: 256
>>>> rx-frames: 128
>>>> rx-usecs-irq: 0
>>>> rx-frames-irq: 0
>>>>
>>>> tx-usecs: 8
>>>> tx-frames: 128
>>>> tx-usecs-irq: 0
>>>> tx-frames-irq: 0
>>>>
>>>> rx-usecs-low: 0
>>>> rx-frame-low: 0
>>>> tx-usecs-low: 0
>>>> tx-frame-low: 0
>>>>
>>>> rx-usecs-high: 0
>>>> rx-frame-high: 0
>>>> tx-usecs-high: 0
>>>> tx-frame-high: 0
>>>>
>>>> ethtool -g enp175s0f0
>>>> Ring parameters for enp175s0f0:
>>>> Pre-set maximums:
>>>> RX: 8192
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 8192
>>>> Current hardware settings:
>>>> RX: 4096
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 4096
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Also changed a little coalesce params - and best for this config are:
>> ethtool -c enp175s0f0
>> Coalesce parameters for enp175s0f0:
>> Adaptive RX: off TX: off
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>> dmac: 32573
>>
>> rx-usecs: 40
>> rx-frames: 128
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 8
>> tx-frames: 8
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
>> rx-usecs-low: 0
>> rx-frame-low: 0
>> tx-usecs-low: 0
>> tx-frame-low: 0
>>
>> rx-usecs-high: 0
>> rx-frame-high: 0
>> tx-usecs-high: 0
>> tx-frame-high: 0
>>
>>
>> Less drops on RX side - and more pps in overall forwarded.
>>
> how much improvement ? maybe we can improve our adaptive rx coal to be
> efficient for this work load.
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 0:16 ` Paweł Staszewski
@ 2018-11-03 12:01 ` Paweł Staszewski
2018-11-03 12:58 ` Jesper Dangaard Brouer
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-03 12:01 UTC (permalink / raw)
To: Aaron Lu, Jesper Dangaard Brouer
Cc: Saeed Mahameed, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
W dniu 03.11.2018 o 01:16, Paweł Staszewski pisze:
>
>
> W dniu 02.11.2018 o 20:02, Paweł Staszewski pisze:
>>
>>
>> W dniu 02.11.2018 o 15:20, Aaron Lu pisze:
>>> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
>>>> On Fri, 2 Nov 2018 13:23:56 +0800
>>>> Aaron Lu <aaron.lu@intel.com> wrote:
>>>>
>>>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
>>>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
>>>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
>>>>>>> wrote:
>>>>>>> ... ...
>>>>>>>> Section copied out:
>>>>>>>>
>>>>>>>> mlx5e_poll_tx_cq
>>>>>>>> |
>>>>>>>> --16.34%--napi_consume_skb
>>>>>>>> |
>>>>>>>> |--12.65%--__free_pages_ok
>>>>>>>> | |
>>>>>>>> | --11.86%--free_one_page
>>>>>>>> | |
>>>>>>>> | |--10.10%
>>>>>>>> --queued_spin_lock_slowpath
>>>>>>>> | |
>>>>>>>> | --0.65%--_raw_spin_lock
>>>>>>> This callchain looks like it is freeing higher order pages than
>>>>>>> order
>>>>>>> 0:
>>>>>>> __free_pages_ok is only called for pages whose order are bigger
>>>>>>> than
>>>>>>> 0.
>>>>>> mlx5 rx uses only order 0 pages, so i don't know where these high
>>>>>> order
>>>>>> tx SKBs are coming from..
>>>>> Perhaps here:
>>>>> __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
>>>>> __napi_alloc_frag() will all call page_frag_alloc(), which will use
>>>>> __page_frag_cache_refill() to get an order 3 page if possible, or
>>>>> fall
>>>>> back to an order 0 page if order 3 page is not available.
>>>>>
>>>>> I'm not sure if your workload will use the above code path though.
>>>> TL;DR: this is order-0 pages (code-walk trough proof below)
>>>>
>>>> To Aaron, the network stack *can* call __free_pages_ok() with order-0
>>>> pages, via:
>>>>
>>>> static void skb_free_head(struct sk_buff *skb)
>>>> {
>>>> unsigned char *head = skb->head;
>>>>
>>>> if (skb->head_frag)
>>>> skb_free_frag(head);
>>>> else
>>>> kfree(head);
>>>> }
>>>>
>>>> static inline void skb_free_frag(void *addr)
>>>> {
>>>> page_frag_free(addr);
>>>> }
>>>>
>>>> /*
>>>> * Frees a page fragment allocated out of either a compound or
>>>> order 0 page.
>>>> */
>>>> void page_frag_free(void *addr)
>>>> {
>>>> struct page *page = virt_to_head_page(addr);
>>>>
>>>> if (unlikely(put_page_testzero(page)))
>>>> __free_pages_ok(page, compound_order(page));
>>>> }
>>>> EXPORT_SYMBOL(page_frag_free);
>>> I think here is a problem - order 0 pages are freed directly to buddy,
>>> bypassing per-cpu-pages. This might be the reason lock contention
>>> appeared on free path. Can someone apply below diff and see if lock
>>> contention is gone?
>> Will test it tonight
>>
> Patch applied
> perf report:
> https://ufile.io/sytfh
>
>
>
> But i need to wait also with more traffic currently cpu's are sleeping
before patch:
| | | |
|--13.55%--mlx5e_poll_tx_cq
| | |
| | |
| | |
| | --10.32%--napi_consume_skb
| | |
| | |
| | |
| | |--8.52%--__free_pages_ok
| | |
| | | |
| | |
| | | --7.67%--free_one_page
| | |
| | | |
| | |
| | | |--6.05%--queued_spin_lock_slowpath
| | |
| | | |
| | |
| | | --0.64%--_raw_spin_lock
| | |
| | |
| | |
| | |--0.77%--skb_release_data
| | |
| | |
| | |
| | --0.72%--page_frag_free
after patch:
| | | | |
|--3.75%--mlx5e_poll_tx_cq
| | | |
| | |
| | | |
| | --1.53%--napi_consume_skb
| | | |
| | |
| | | |
| | --0.54%--skb_release_data
| | | |
| |
| | | | |
--3.09%--mlx5e_post_rx_wqes
| | | |
| |
| | | | |
--1.21%--__page_pool_alloc_pages_slow
| | | |
| |
| | | | |
--1.16%--__alloc_pages_nodemask
| | | |
| |
| | | | |
--1.05%--get_page_from_freelist
currently waiting for more traffic also
>
>
>
>
>>
>>
>>>
>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>> index e2ef1c17942f..65c0ae13215a 100644
>>> --- a/mm/page_alloc.c
>>> +++ b/mm/page_alloc.c
>>> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
>>> {
>>> struct page *page = virt_to_head_page(addr);
>>> - if (unlikely(put_page_testzero(page)))
>>> - __free_pages_ok(page, compound_order(page));
>>> + if (unlikely(put_page_testzero(page))) {
>>> + unsigned int order = compound_order(page);
>>> +
>>> + if (order == 0)
>>> + free_unref_page(page);
>>> + else
>>> + __free_pages_ok(page, order);
>>> + }
>>> }
>>> EXPORT_SYMBOL(page_frag_free);
>>>> Notice for the mlx5 driver it support several RX-memory models, so it
>>>> can be hard to follow, but from the perf report output we can see that
>>>> is uses mlx5e_skb_from_cqe_linear, which use build_skb.
>>>>
>>>> --13.63%--mlx5e_skb_from_cqe_linear
>>>> |
>>>> --5.02%--build_skb
>>>> |
>>>> --1.85%--__build_skb
>>>> |
>>>> --1.00%--kmem_cache_alloc
>>>>
>>>> /* build_skb() is wrapper over __build_skb(), that specifically
>>>> * takes care of skb->head and skb->pfmemalloc
>>>> * This means that if @frag_size is not zero, then @data must be
>>>> backed
>>>> * by a page fragment, not kmalloc() or vmalloc()
>>>> */
>>>> struct sk_buff *build_skb(void *data, unsigned int frag_size)
>>>> {
>>>> struct sk_buff *skb = __build_skb(data, frag_size);
>>>>
>>>> if (skb && frag_size) {
>>>> skb->head_frag = 1;
>>>> if (page_is_pfmemalloc(virt_to_head_page(data)))
>>>> skb->pfmemalloc = 1;
>>>> }
>>>> return skb;
>>>> }
>>>> EXPORT_SYMBOL(build_skb);
>>>>
>>>> It still doesn't prove, that the @data is backed by by a order-0 page.
>>>> For the mlx5 driver is uses mlx5e_page_alloc_mapped ->
>>>> page_pool_dev_alloc_pages(), and I can see perf report using
>>>> __page_pool_alloc_pages_slow().
>>>>
>>>> The setup for page_pool in mlx5 uses order=0.
>>>>
>>>> /* Create a page_pool and register it with rxq */
>>>> pp_params.order = 0;
>>>> pp_params.flags = 0; /* No-internal DMA mapping in
>>>> page_pool */
>>>> pp_params.pool_size = pool_size;
>>>> pp_params.nid = cpu_to_node(c->cpu);
>>>> pp_params.dev = c->pdev;
>>>> pp_params.dma_dir = rq->buff.map_dir;
>>>>
>>>> /* page_pool can be used even when there is no rq->xdp_prog,
>>>> * given page_pool does not handle DMA mapping there is no
>>>> * required state to clear. And page_pool gracefully handle
>>>> * elevated refcnt.
>>>> */
>>>> rq->page_pool = page_pool_create(&pp_params);
>>>> if (IS_ERR(rq->page_pool)) {
>>>> err = PTR_ERR(rq->page_pool);
>>>> rq->page_pool = NULL;
>>>> goto err_free;
>>>> }
>>>> err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
>>>> MEM_TYPE_PAGE_POOL, rq->page_pool);
>>> Thanks for the detailed analysis, I'll need more time to understand the
>>> whole picture :-)
>>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-02 14:20 ` Aaron Lu
2018-11-02 19:02 ` Paweł Staszewski
@ 2018-11-03 12:53 ` Jesper Dangaard Brouer
2018-11-05 6:28 ` Aaron Lu
2018-11-05 8:42 ` Tariq Toukan
1 sibling, 2 replies; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-03 12:53 UTC (permalink / raw)
To: Aaron Lu
Cc: Saeed Mahameed, pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman, brouer
On Fri, 2 Nov 2018 22:20:24 +0800 Aaron Lu <aaron.lu@intel.com> wrote:
> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
> > On Fri, 2 Nov 2018 13:23:56 +0800
> > Aaron Lu <aaron.lu@intel.com> wrote:
> >
> > > On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> > > > On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> > > > > On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> > > > > wrote:
> > > > > ... ...
> > > > > > Section copied out:
> > > > > >
> > > > > > mlx5e_poll_tx_cq
> > > > > > |
> > > > > > --16.34%--napi_consume_skb
> > > > > > |
> > > > > > |--12.65%--__free_pages_ok
> > > > > > | |
> > > > > > | --11.86%--free_one_page
> > > > > > | |
> > > > > > | |--10.10%
> > > > > > --queued_spin_lock_slowpath
> > > > > > | |
> > > > > > | --0.65%--_raw_spin_lock
> > > > >
> > > > > This callchain looks like it is freeing higher order pages than order
> > > > > 0:
> > > > > __free_pages_ok is only called for pages whose order are bigger than
> > > > > 0.
> > > >
> > > > mlx5 rx uses only order 0 pages, so i don't know where these high order
> > > > tx SKBs are coming from..
> > >
> > > Perhaps here:
> > > __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
> > > __napi_alloc_frag() will all call page_frag_alloc(), which will use
> > > __page_frag_cache_refill() to get an order 3 page if possible, or fall
> > > back to an order 0 page if order 3 page is not available.
> > >
> > > I'm not sure if your workload will use the above code path though.
> >
> > TL;DR: this is order-0 pages (code-walk trough proof below)
> >
> > To Aaron, the network stack *can* call __free_pages_ok() with order-0
> > pages, via:
> >
> > static void skb_free_head(struct sk_buff *skb)
> > {
> > unsigned char *head = skb->head;
> >
> > if (skb->head_frag)
> > skb_free_frag(head);
> > else
> > kfree(head);
> > }
> >
> > static inline void skb_free_frag(void *addr)
> > {
> > page_frag_free(addr);
> > }
> >
> > /*
> > * Frees a page fragment allocated out of either a compound or order 0 page.
> > */
> > void page_frag_free(void *addr)
> > {
> > struct page *page = virt_to_head_page(addr);
> >
> > if (unlikely(put_page_testzero(page)))
> > __free_pages_ok(page, compound_order(page));
> > }
> > EXPORT_SYMBOL(page_frag_free);
>
> I think here is a problem - order 0 pages are freed directly to buddy,
> bypassing per-cpu-pages. This might be the reason lock contention
> appeared on free path.
OMG - you just found a significant issue with the network stacks
interaction with the page allocator! This explains why I could not get
the PCP (Per-Cpu-Pages) system to have good performance, in my
performance networking benchmarks. As we are basically only using the
alloc side of PCP, and not the free side.
We have spend years adding different driver level recycle tricks to
avoid this code path getting activated, exactly because it is rather
slow and problematic that we hit this zone->lock.
> Can someone apply below diff and see if lock contention is gone?
I have also applied and tested this patch, and yes the lock contention
is gone. As mentioned is it rather difficult to hit this code path, as
the driver page recycle mechanism tries to hide/avoid it, but mlx5 +
page_pool + CPU-map recycling have a known weakness that bypass the
driver page recycle scheme (that I've not fixed yet). I observed a 7%
speedup for this micro benchmark.
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index e2ef1c17942f..65c0ae13215a 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
> {
> struct page *page = virt_to_head_page(addr);
>
> - if (unlikely(put_page_testzero(page)))
> - __free_pages_ok(page, compound_order(page));
> + if (unlikely(put_page_testzero(page))) {
> + unsigned int order = compound_order(page);
> +
> + if (order == 0)
> + free_unref_page(page);
> + else
> + __free_pages_ok(page, order);
> + }
> }
> EXPORT_SYMBOL(page_frag_free);
Thank you Aaron for spotting this!!!
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 0:16 ` Paweł Staszewski
2018-11-03 12:01 ` Paweł Staszewski
@ 2018-11-03 12:58 ` Jesper Dangaard Brouer
2018-11-03 15:23 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-03 12:58 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Aaron Lu, Saeed Mahameed, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman, brouer
On Sat, 3 Nov 2018 01:16:08 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 02.11.2018 o 20:02, Paweł Staszewski pisze:
> >
> >
> > W dniu 02.11.2018 o 15:20, Aaron Lu pisze:
> >> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
> >>> On Fri, 2 Nov 2018 13:23:56 +0800
> >>> Aaron Lu <aaron.lu@intel.com> wrote:
> >>>
> >>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> >>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> >>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> >>>>>> wrote:
> >>>>>> ... ...
[...]
> >>> TL;DR: this is order-0 pages (code-walk trough proof below)
> >>>
> >>> To Aaron, the network stack *can* call __free_pages_ok() with order-0
> >>> pages, via:
[...]
> >>
> >> I think here is a problem - order 0 pages are freed directly to buddy,
> >> bypassing per-cpu-pages. This might be the reason lock contention
> >> appeared on free path. Can someone apply below diff and see if lock
> >> contention is gone?
> >>
> > Will test it tonight
> >
> Patch applied
> perf report:
> https://ufile.io/sytfh
>
>
> But i need to wait also with more traffic currently cpu's are sleeping
>
Well, that would be the expected result, that the CPUs get more time to
sleep, if the lock contention is gone...
What is the measured bandwidth now?
Notice, you might still be limited by the PCIe bandwidth, but then your
CPUs might actually decide to sleep, as they are getting data fast
enough.
[...]
> >>
> >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> >> index e2ef1c17942f..65c0ae13215a 100644
> >> --- a/mm/page_alloc.c
> >> +++ b/mm/page_alloc.c
> >> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
> >> {
> >> struct page *page = virt_to_head_page(addr);
> >> - if (unlikely(put_page_testzero(page)))
> >> - __free_pages_ok(page, compound_order(page));
> >> + if (unlikely(put_page_testzero(page))) {
> >> + unsigned int order = compound_order(page);
> >> +
> >> + if (order == 0)
> >> + free_unref_page(page);
> >> + else
> >> + __free_pages_ok(page, order);
> >> + }
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 12:58 ` Jesper Dangaard Brouer
@ 2018-11-03 15:23 ` Paweł Staszewski
2018-11-03 15:43 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-03 15:23 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Aaron Lu, Saeed Mahameed, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
W dniu 03.11.2018 o 13:58, Jesper Dangaard Brouer pisze:
> On Sat, 3 Nov 2018 01:16:08 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 02.11.2018 o 20:02, Paweł Staszewski pisze:
>>>
>>> W dniu 02.11.2018 o 15:20, Aaron Lu pisze:
>>>> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
>>>>> On Fri, 2 Nov 2018 13:23:56 +0800
>>>>> Aaron Lu <aaron.lu@intel.com> wrote:
>>>>>
>>>>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
>>>>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
>>>>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
>>>>>>>> wrote:
>>>>>>>> ... ...
> [...]
>>>>> TL;DR: this is order-0 pages (code-walk trough proof below)
>>>>>
>>>>> To Aaron, the network stack *can* call __free_pages_ok() with order-0
>>>>> pages, via:
> [...]
>>>>
>>>> I think here is a problem - order 0 pages are freed directly to buddy,
>>>> bypassing per-cpu-pages. This might be the reason lock contention
>>>> appeared on free path. Can someone apply below diff and see if lock
>>>> contention is gone?
>>>>
>>> Will test it tonight
>>>
>> Patch applied
>> perf report:
>> https://ufile.io/sytfh
>>
>>
>> But i need to wait also with more traffic currently cpu's are sleeping
>>
> Well, that would be the expected result, that the CPUs get more time to
> sleep, if the lock contention is gone...
>
> What is the measured bandwidth now?
30 RX /30 TX Gbit/s
>
> Notice, you might still be limited by the PCIe bandwidth, but then your
> CPUs might actually decide to sleep, as they are getting data fast
> enough.
Yes - i will replace network controller to two separate nic's in two
separate x16 pcie
But after monday.
But i dont think i hit pcie limit there - it looks like pcie x16 gen3
have 16GB/s RX and 16GB/s TX so bidirectional
>
> [...]
>>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>>> index e2ef1c17942f..65c0ae13215a 100644
>>>> --- a/mm/page_alloc.c
>>>> +++ b/mm/page_alloc.c
>>>> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
>>>> {
>>>> struct page *page = virt_to_head_page(addr);
>>>> - if (unlikely(put_page_testzero(page)))
>>>> - __free_pages_ok(page, compound_order(page));
>>>> + if (unlikely(put_page_testzero(page))) {
>>>> + unsigned int order = compound_order(page);
>>>> +
>>>> + if (order == 0)
>>>> + free_unref_page(page);
>>>> + else
>>>> + __free_pages_ok(page, order);
>>>> + }
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 15:23 ` Paweł Staszewski
@ 2018-11-03 15:43 ` Paweł Staszewski
0 siblings, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-03 15:43 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Aaron Lu, Saeed Mahameed, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
W dniu 03.11.2018 o 16:23, Paweł Staszewski pisze:
>
>
> W dniu 03.11.2018 o 13:58, Jesper Dangaard Brouer pisze:
>> On Sat, 3 Nov 2018 01:16:08 +0100
>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 02.11.2018 o 20:02, Paweł Staszewski pisze:
>>>>
>>>> W dniu 02.11.2018 o 15:20, Aaron Lu pisze:
>>>>> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer
>>>>> wrote:
>>>>>> On Fri, 2 Nov 2018 13:23:56 +0800
>>>>>> Aaron Lu <aaron.lu@intel.com> wrote:
>>>>>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
>>>>>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
>>>>>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
>>>>>>>>> wrote:
>>>>>>>>> ... ...
>> [...]
>>>>>> TL;DR: this is order-0 pages (code-walk trough proof below)
>>>>>>
>>>>>> To Aaron, the network stack *can* call __free_pages_ok() with
>>>>>> order-0
>>>>>> pages, via:
>> [...]
>>>>> I think here is a problem - order 0 pages are freed directly to
>>>>> buddy,
>>>>> bypassing per-cpu-pages. This might be the reason lock contention
>>>>> appeared on free path. Can someone apply below diff and see if lock
>>>>> contention is gone?
>>>>>
>>>> Will test it tonight
>>> Patch applied
>>> perf report:
>>> https://ufile.io/sytfh
>>>
>>>
>>> But i need to wait also with more traffic currently cpu's are sleeping
>>>
>> Well, that would be the expected result, that the CPUs get more time to
>> sleep, if the lock contention is gone...
>>
>> What is the measured bandwidth now?
> 30 RX /30 TX Gbit/s
>
>>
>> Notice, you might still be limited by the PCIe bandwidth, but then your
>> CPUs might actually decide to sleep, as they are getting data fast
>> enough.
> Yes - i will replace network controller to two separate nic's in two
> separate x16 pcie
> But after monday.
>
> But i dont think i hit pcie limit there - it looks like pcie x16 gen3
> have 16GB/s RX and 16GB/s TX so bidirectional
>
Was thinking that maybee memory limit - but also there is 4 channel DDR4
2666MHz - so total bandwidth for memory is bigger (48GB/s) than needed
for 100Gbit ethernet
>
>>
>> [...]
>>>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>>>> index e2ef1c17942f..65c0ae13215a 100644
>>>>> --- a/mm/page_alloc.c
>>>>> +++ b/mm/page_alloc.c
>>>>> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
>>>>> {
>>>>> struct page *page = virt_to_head_page(addr);
>>>>> - if (unlikely(put_page_testzero(page)))
>>>>> - __free_pages_ok(page, compound_order(page));
>>>>> + if (unlikely(put_page_testzero(page))) {
>>>>> + unsigned int order = compound_order(page);
>>>>> +
>>>>> + if (order == 0)
>>>>> + free_unref_page(page);
>>>>> + else
>>>>> + __free_pages_ok(page, order);
>>>>> + }
>>
>>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-01 17:30 ` Paweł Staszewski
@ 2018-11-03 17:32 ` David Ahern
2018-11-04 0:24 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: David Ahern @ 2018-11-03 17:32 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/1/18 11:30 AM, Paweł Staszewski wrote:
>>>>
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/xdp_fwd_kern.c
>>>>
>>>>
>>> I can try some tests on same hw but testlab configuration - will give it
>>> a try :)
>>>
>> That version does not work with VLANs. I have patches for it but it
>> needs a bit more work before sending out. Perhaps I can get back to it
>> next week.
>>
> Will be nice - next week i will be able to replace network controller
> and install separate two 100Gbit nics into two pciex x16 slots - so can
> test without hitting pcie bandwidth limits.
>
>
Does your setup have any other device types besides physical ports with
VLANs (e.g., any macvlans or bonds)?
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 17:32 ` David Ahern
@ 2018-11-04 0:24 ` Paweł Staszewski
2018-11-05 20:17 ` Jesper Dangaard Brouer
2018-11-07 21:06 ` David Ahern
0 siblings, 2 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-04 0:24 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
n
W dniu 03.11.2018 o 18:32, David Ahern pisze:
> On 11/1/18 11:30 AM, Paweł Staszewski wrote:
>>>>>
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/xdp_fwd_kern.c
>>>>>
>>>>>
>>>> I can try some tests on same hw but testlab configuration - will give it
>>>> a try :)
>>>>
>>> That version does not work with VLANs. I have patches for it but it
>>> needs a bit more work before sending out. Perhaps I can get back to it
>>> next week.
>>>
>> Will be nice - next week i will be able to replace network controller
>> and install separate two 100Gbit nics into two pciex x16 slots - so can
>> test without hitting pcie bandwidth limits.
>>
>>
> Does your setup have any other device types besides physical ports with
> VLANs (e.g., any macvlans or bonds)?
>
>
no.
just
phy(mlnx)->vlans only config
And today again after allpy patch for page allocator - reached again
64/64 Gbit/s
with only 50-60% cpu load
today no slowpath hit for netwoking :)
But again dropped pckt at 64GbitRX and 64TX ....
And as it should not be pcie express limit -i think something more is
going on there - and hard to catch - cause perf top doestn chenged
besides there is no queued slowpath hit now
I ordered now also intel cards to compare - but 3 weeks eta
Faster - cause 3 days - i will have mellanox connectx 5 - so can
separate traffic to two different x16 pcie busses
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 12:53 ` Jesper Dangaard Brouer
@ 2018-11-05 6:28 ` Aaron Lu
2018-11-05 9:10 ` Jesper Dangaard Brouer
2018-11-05 8:42 ` Tariq Toukan
1 sibling, 1 reply; 77+ messages in thread
From: Aaron Lu @ 2018-11-05 6:28 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Saeed Mahameed, pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
On Sat, Nov 03, 2018 at 01:53:25PM +0100, Jesper Dangaard Brouer wrote:
>
> On Fri, 2 Nov 2018 22:20:24 +0800 Aaron Lu <aaron.lu@intel.com> wrote:
>
> > On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
> > > On Fri, 2 Nov 2018 13:23:56 +0800
> > > Aaron Lu <aaron.lu@intel.com> wrote:
> > >
> > > > On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> > > > > On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> > > > > > On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> > > > > > wrote:
> > > > > > ... ...
> > > > > > > Section copied out:
> > > > > > >
> > > > > > > mlx5e_poll_tx_cq
> > > > > > > |
> > > > > > > --16.34%--napi_consume_skb
> > > > > > > |
> > > > > > > |--12.65%--__free_pages_ok
> > > > > > > | |
> > > > > > > | --11.86%--free_one_page
> > > > > > > | |
> > > > > > > | |--10.10%
> > > > > > > --queued_spin_lock_slowpath
> > > > > > > | |
> > > > > > > | --0.65%--_raw_spin_lock
> > > > > >
> > > > > > This callchain looks like it is freeing higher order pages than order
> > > > > > 0:
> > > > > > __free_pages_ok is only called for pages whose order are bigger than
> > > > > > 0.
> > > > >
> > > > > mlx5 rx uses only order 0 pages, so i don't know where these high order
> > > > > tx SKBs are coming from..
> > > >
> > > > Perhaps here:
> > > > __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
> > > > __napi_alloc_frag() will all call page_frag_alloc(), which will use
> > > > __page_frag_cache_refill() to get an order 3 page if possible, or fall
> > > > back to an order 0 page if order 3 page is not available.
> > > >
> > > > I'm not sure if your workload will use the above code path though.
> > >
> > > TL;DR: this is order-0 pages (code-walk trough proof below)
> > >
> > > To Aaron, the network stack *can* call __free_pages_ok() with order-0
> > > pages, via:
> > >
> > > static void skb_free_head(struct sk_buff *skb)
> > > {
> > > unsigned char *head = skb->head;
> > >
> > > if (skb->head_frag)
> > > skb_free_frag(head);
> > > else
> > > kfree(head);
> > > }
> > >
> > > static inline void skb_free_frag(void *addr)
> > > {
> > > page_frag_free(addr);
> > > }
> > >
> > > /*
> > > * Frees a page fragment allocated out of either a compound or order 0 page.
> > > */
> > > void page_frag_free(void *addr)
> > > {
> > > struct page *page = virt_to_head_page(addr);
> > >
> > > if (unlikely(put_page_testzero(page)))
> > > __free_pages_ok(page, compound_order(page));
> > > }
> > > EXPORT_SYMBOL(page_frag_free);
> >
> > I think here is a problem - order 0 pages are freed directly to buddy,
> > bypassing per-cpu-pages. This might be the reason lock contention
> > appeared on free path.
>
> OMG - you just found a significant issue with the network stacks
> interaction with the page allocator! This explains why I could not get
> the PCP (Per-Cpu-Pages) system to have good performance, in my
> performance networking benchmarks. As we are basically only using the
> alloc side of PCP, and not the free side.
Exactly.
> We have spend years adding different driver level recycle tricks to
> avoid this code path getting activated, exactly because it is rather
> slow and problematic that we hit this zone->lock.
I can see when this code path is hit, it causes unnecessary taking of
zone lock for order-0 pages and cause lock contention.
>
> > Can someone apply below diff and see if lock contention is gone?
>
> I have also applied and tested this patch, and yes the lock contention
> is gone. As mentioned is it rather difficult to hit this code path, as
> the driver page recycle mechanism tries to hide/avoid it, but mlx5 +
> page_pool + CPU-map recycling have a known weakness that bypass the
> driver page recycle scheme (that I've not fixed yet). I observed a 7%
> speedup for this micro benchmark.
Good to know this, I will prepare a formal patch.
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index e2ef1c17942f..65c0ae13215a 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
> > {
> > struct page *page = virt_to_head_page(addr);
> >
> > - if (unlikely(put_page_testzero(page)))
> > - __free_pages_ok(page, compound_order(page));
> > + if (unlikely(put_page_testzero(page))) {
> > + unsigned int order = compound_order(page);
> > +
> > + if (order == 0)
> > + free_unref_page(page);
> > + else
> > + __free_pages_ok(page, order);
> > + }
> > }
> > EXPORT_SYMBOL(page_frag_free);
>
> Thank you Aaron for spotting this!!!
Which is impossible without your analysis :-)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 12:53 ` Jesper Dangaard Brouer
2018-11-05 6:28 ` Aaron Lu
@ 2018-11-05 8:42 ` Tariq Toukan
2018-11-05 8:48 ` Aaron Lu
1 sibling, 1 reply; 77+ messages in thread
From: Tariq Toukan @ 2018-11-05 8:42 UTC (permalink / raw)
To: Jesper Dangaard Brouer, Aaron Lu
Cc: Saeed Mahameed, pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman
On 03/11/2018 2:53 PM, Jesper Dangaard Brouer wrote:
>
> On Fri, 2 Nov 2018 22:20:24 +0800 Aaron Lu <aaron.lu@intel.com> wrote:
>
>> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
>>> On Fri, 2 Nov 2018 13:23:56 +0800
>>> Aaron Lu <aaron.lu@intel.com> wrote:
>>>
>>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
>>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
>>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
>>>>>> wrote:
>>>>>> ... ...
>>>>>>> Section copied out:
>>>>>>>
>>>>>>> mlx5e_poll_tx_cq
>>>>>>> |
>>>>>>> --16.34%--napi_consume_skb
>>>>>>> |
>>>>>>> |--12.65%--__free_pages_ok
>>>>>>> | |
>>>>>>> | --11.86%--free_one_page
>>>>>>> | |
>>>>>>> | |--10.10%
>>>>>>> --queued_spin_lock_slowpath
>>>>>>> | |
>>>>>>> | --0.65%--_raw_spin_lock
>>>>>>
>>>>>> This callchain looks like it is freeing higher order pages than order
>>>>>> 0:
>>>>>> __free_pages_ok is only called for pages whose order are bigger than
>>>>>> 0.
>>>>>
>>>>> mlx5 rx uses only order 0 pages, so i don't know where these high order
>>>>> tx SKBs are coming from..
>>>>
>>>> Perhaps here:
>>>> __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
>>>> __napi_alloc_frag() will all call page_frag_alloc(), which will use
>>>> __page_frag_cache_refill() to get an order 3 page if possible, or fall
>>>> back to an order 0 page if order 3 page is not available.
>>>>
>>>> I'm not sure if your workload will use the above code path though.
>>>
>>> TL;DR: this is order-0 pages (code-walk trough proof below)
>>>
>>> To Aaron, the network stack *can* call __free_pages_ok() with order-0
>>> pages, via:
>>>
>>> static void skb_free_head(struct sk_buff *skb)
>>> {
>>> unsigned char *head = skb->head;
>>>
>>> if (skb->head_frag)
>>> skb_free_frag(head);
>>> else
>>> kfree(head);
>>> }
>>>
>>> static inline void skb_free_frag(void *addr)
>>> {
>>> page_frag_free(addr);
>>> }
>>>
>>> /*
>>> * Frees a page fragment allocated out of either a compound or order 0 page.
>>> */
>>> void page_frag_free(void *addr)
>>> {
>>> struct page *page = virt_to_head_page(addr);
>>>
>>> if (unlikely(put_page_testzero(page)))
>>> __free_pages_ok(page, compound_order(page));
>>> }
>>> EXPORT_SYMBOL(page_frag_free);
>>
>> I think here is a problem - order 0 pages are freed directly to buddy,
>> bypassing per-cpu-pages. This might be the reason lock contention
>> appeared on free path.
>
> OMG - you just found a significant issue with the network stacks
> interaction with the page allocator! This explains why I could not get
> the PCP (Per-Cpu-Pages) system to have good performance, in my
> performance networking benchmarks. As we are basically only using the
> alloc side of PCP, and not the free side.
> We have spend years adding different driver level recycle tricks to
> avoid this code path getting activated, exactly because it is rather
> slow and problematic that we hit this zone->lock.
>
Oh! It has been behaving this way for too long.
Good catch!
>> Can someone apply below diff and see if lock contention is gone?
>
> I have also applied and tested this patch, and yes the lock contention
> is gone. As mentioned is it rather difficult to hit this code path, as
> the driver page recycle mechanism tries to hide/avoid it, but mlx5 +
> page_pool + CPU-map recycling have a known weakness that bypass the
> driver page recycle scheme (that I've not fixed yet). I observed a 7%
> speedup for this micro benchmark.
>
Great news. I also have a benchmark that uses orde-r0 pages and stresses
the zone-lock. I'll test your patch during this week.
>
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index e2ef1c17942f..65c0ae13215a 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
>> {
>> struct page *page = virt_to_head_page(addr);
>>
>> - if (unlikely(put_page_testzero(page)))
>> - __free_pages_ok(page, compound_order(page));
>> + if (unlikely(put_page_testzero(page))) {
>> + unsigned int order = compound_order(page);
>> +
>> + if (order == 0)
>> + free_unref_page(page);
>> + else
>> + __free_pages_ok(page, order);
>> + }
>> }
>> EXPORT_SYMBOL(page_frag_free);
>
> Thank you Aaron for spotting this!!!
>
Thanks Aaron :) !!
Does it conflict with your recent work that optimizes order-0 allocation?
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-05 8:42 ` Tariq Toukan
@ 2018-11-05 8:48 ` Aaron Lu
0 siblings, 0 replies; 77+ messages in thread
From: Aaron Lu @ 2018-11-05 8:48 UTC (permalink / raw)
To: Tariq Toukan
Cc: Jesper Dangaard Brouer, Saeed Mahameed, pstaszewski,
eric.dumazet, netdev, ilias.apalodimas, yoel, mgorman
On Mon, Nov 05, 2018 at 08:42:33AM +0000, Tariq Toukan wrote:
>
> On 03/11/2018 2:53 PM, Jesper Dangaard Brouer wrote:
> >
> > On Fri, 2 Nov 2018 22:20:24 +0800 Aaron Lu <aaron.lu@intel.com> wrote:
> >>
> >> I think here is a problem - order 0 pages are freed directly to buddy,
> >> bypassing per-cpu-pages. This might be the reason lock contention
> >> appeared on free path.
> >
> > OMG - you just found a significant issue with the network stacks
> > interaction with the page allocator! This explains why I could not get
> > the PCP (Per-Cpu-Pages) system to have good performance, in my
> > performance networking benchmarks. As we are basically only using the
> > alloc side of PCP, and not the free side.
> > We have spend years adding different driver level recycle tricks to
> > avoid this code path getting activated, exactly because it is rather
> > slow and problematic that we hit this zone->lock.
> >
>
> Oh! It has been behaving this way for too long.
> Good catch!
Thanks.
> >> Can someone apply below diff and see if lock contention is gone?
> >
> > I have also applied and tested this patch, and yes the lock contention
> > is gone. As mentioned is it rather difficult to hit this code path, as
> > the driver page recycle mechanism tries to hide/avoid it, but mlx5 +
> > page_pool + CPU-map recycling have a known weakness that bypass the
> > driver page recycle scheme (that I've not fixed yet). I observed a 7%
> > speedup for this micro benchmark.
> >
>
> Great news. I also have a benchmark that uses orde-r0 pages and stresses
> the zone-lock. I'll test your patch during this week.
Note this patch only helps when order-0 pages are freed through
page_frag_free().
I'll send a formal patch later.
> >
> >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> >> index e2ef1c17942f..65c0ae13215a 100644
> >> --- a/mm/page_alloc.c
> >> +++ b/mm/page_alloc.c
> >> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
> >> {
> >> struct page *page = virt_to_head_page(addr);
> >>
> >> - if (unlikely(put_page_testzero(page)))
> >> - __free_pages_ok(page, compound_order(page));
> >> + if (unlikely(put_page_testzero(page))) {
> >> + unsigned int order = compound_order(page);
> >> +
> >> + if (order == 0)
> >> + free_unref_page(page);
> >> + else
> >> + __free_pages_ok(page, order);
> >> + }
> >> }
> >> EXPORT_SYMBOL(page_frag_free);
> >
> > Thank you Aaron for spotting this!!!
> >
> Thanks Aaron :) !!
>
> Does it conflict with your recent work that optimizes order-0 allocation?
No it doesn't. This patch optimize code outside of zone lock(by reducing
the need to take zone lock) while my recent work optimize code inside
the zone lock :-)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-05 6:28 ` Aaron Lu
@ 2018-11-05 9:10 ` Jesper Dangaard Brouer
0 siblings, 0 replies; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-05 9:10 UTC (permalink / raw)
To: Aaron Lu
Cc: Saeed Mahameed, pstaszewski, eric.dumazet, netdev, Tariq Toukan,
ilias.apalodimas, yoel, mgorman, brouer, Jérôme Glisse
On Mon, 5 Nov 2018 14:28:36 +0800
Aaron Lu <aaron.lu@intel.com> wrote:
> On Sat, Nov 03, 2018 at 01:53:25PM +0100, Jesper Dangaard Brouer wrote:
> >
> > On Fri, 2 Nov 2018 22:20:24 +0800 Aaron Lu <aaron.lu@intel.com> wrote:
> >
> > > On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:
> > > > On Fri, 2 Nov 2018 13:23:56 +0800
> > > > Aaron Lu <aaron.lu@intel.com> wrote:
> > > >
> > > > > On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> > > > > > On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> > > > > > > On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> > > > > > > wrote:
> > > > > > > ... ...
> > > > > > > > Section copied out:
> > > > > > > >
> > > > > > > > mlx5e_poll_tx_cq
> > > > > > > > |
> > > > > > > > --16.34%--napi_consume_skb
> > > > > > > > |
> > > > > > > > |--12.65%--__free_pages_ok
> > > > > > > > | |
> > > > > > > > | --11.86%--free_one_page
> > > > > > > > | |
> > > > > > > > | |--10.10%
> > > > > > > > --queued_spin_lock_slowpath
> > > > > > > > | |
> > > > > > > > | --0.65%--_raw_spin_lock
> > > > > > >
> > > > > > > This callchain looks like it is freeing higher order pages than order
> > > > > > > 0:
> > > > > > > __free_pages_ok is only called for pages whose order are bigger than
> > > > > > > 0.
> > > > > >
> > > > > > mlx5 rx uses only order 0 pages, so i don't know where these high order
> > > > > > tx SKBs are coming from..
> > > > >
> > > > > Perhaps here:
> > > > > __netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
> > > > > __napi_alloc_frag() will all call page_frag_alloc(), which will use
> > > > > __page_frag_cache_refill() to get an order 3 page if possible, or fall
> > > > > back to an order 0 page if order 3 page is not available.
> > > > >
> > > > > I'm not sure if your workload will use the above code path though.
> > > >
> > > > TL;DR: this is order-0 pages (code-walk trough proof below)
> > > >
> > > > To Aaron, the network stack *can* call __free_pages_ok() with order-0
> > > > pages, via:
> > > >
> > > > static void skb_free_head(struct sk_buff *skb)
> > > > {
> > > > unsigned char *head = skb->head;
> > > >
> > > > if (skb->head_frag)
> > > > skb_free_frag(head);
> > > > else
> > > > kfree(head);
> > > > }
> > > >
> > > > static inline void skb_free_frag(void *addr)
> > > > {
> > > > page_frag_free(addr);
> > > > }
> > > >
> > > > /*
> > > > * Frees a page fragment allocated out of either a compound or order 0 page.
> > > > */
> > > > void page_frag_free(void *addr)
> > > > {
> > > > struct page *page = virt_to_head_page(addr);
> > > >
> > > > if (unlikely(put_page_testzero(page)))
> > > > __free_pages_ok(page, compound_order(page));
> > > > }
> > > > EXPORT_SYMBOL(page_frag_free);
> > >
> > > I think here is a problem - order 0 pages are freed directly to buddy,
> > > bypassing per-cpu-pages. This might be the reason lock contention
> > > appeared on free path.
> >
> > OMG - you just found a significant issue with the network stacks
> > interaction with the page allocator! This explains why I could not get
> > the PCP (Per-Cpu-Pages) system to have good performance, in my
> > performance networking benchmarks. As we are basically only using the
> > alloc side of PCP, and not the free side.
>
> Exactly.
>
> > We have spend years adding different driver level recycle tricks to
> > avoid this code path getting activated, exactly because it is rather
> > slow and problematic that we hit this zone->lock.
>
> I can see when this code path is hit, it causes unnecessary taking of
> zone lock for order-0 pages and cause lock contention.
>
> >
> > > Can someone apply below diff and see if lock contention is gone?
> >
> > I have also applied and tested this patch, and yes the lock contention
> > is gone. As mentioned is it rather difficult to hit this code path, as
> > the driver page recycle mechanism tries to hide/avoid it, but mlx5 +
> > page_pool + CPU-map recycling have a known weakness that bypass the
> > driver page recycle scheme (that I've not fixed yet). I observed a 7%
> > speedup for this micro benchmark.
>
> Good to know this, I will prepare a formal patch.
I wonder if this code is still missing something. I was looking at
using put_devmap_managed_page() infrastructure, but I realized that
page_frag_free() is also skipping this code path. I guess, I can add
it later when I show/proof (performance wise) that this is a good idea
(as we currently don't have any users).
> > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > > index e2ef1c17942f..65c0ae13215a 100644
> > > --- a/mm/page_alloc.c
> > > +++ b/mm/page_alloc.c
> > > @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
> > > {
> > > struct page *page = virt_to_head_page(addr);
> > >
> > > - if (unlikely(put_page_testzero(page)))
> > > - __free_pages_ok(page, compound_order(page));
> > > + if (unlikely(put_page_testzero(page))) {
> > > + unsigned int order = compound_order(page);
> > > +
> > > + if (order == 0)
> > > + free_unref_page(page);
> > > + else
> > > + __free_pages_ok(page, order);
> > > + }
> > > }
> > > EXPORT_SYMBOL(page_frag_free);
> >
> > Thank you Aaron for spotting this!!!
>
> Which is impossible without your analysis :-)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-04 0:24 ` Paweł Staszewski
@ 2018-11-05 20:17 ` Jesper Dangaard Brouer
2018-11-08 0:59 ` Paweł Staszewski
2018-11-07 21:06 ` David Ahern
1 sibling, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-05 20:17 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: David Ahern, netdev, Yoel Caspersen, brouer
On Sun, 4 Nov 2018 01:24:03 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> And today again after allpy patch for page allocator - reached again
> 64/64 Gbit/s
>
> with only 50-60% cpu load
Great.
> today no slowpath hit for netwoking :)
>
> But again dropped pckt at 64GbitRX and 64TX ....
> And as it should not be pcie express limit -i think something more is
Well, this does sounds like a PCIe bandwidth limit to me.
See the PCIe BW here: https://en.wikipedia.org/wiki/PCI_Express
You likely have PCIe v3, where 1-lane have 984.6 MBytes/s or 7.87 Gbit/s
Thus, x16-lanes have 15.75 GBytes or 126 Gbit/s. It does say "in each
direction", but you are also forwarding this RX->TX on both (dual) ports
NIC that is sharing the same PCIe slot.
> going on there - and hard to catch - cause perf top doestn chenged
> besides there is no queued slowpath hit now
>
> I ordered now also intel cards to compare - but 3 weeks eta
> Faster - cause 3 days - i will have mellanox connectx 5 - so can
> separate traffic to two different x16 pcie busses
I do think you need to separate traffic to two different x16 PCIe
slots. I have found that the ConnectX-5 is significantly faster
packet-per-sec performance than ConnectX-4, but that is not your
use-case (max BW). I've not tested these NICs for maximum
_bidirectional_ bandwidth limits, I've only made sure I can do 100G
unidirectional, which can hit some funny motherboard memory limits
(remember to equip motherboard with 4 RAM blocks for full memory BW).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-04 0:24 ` Paweł Staszewski
2018-11-05 20:17 ` Jesper Dangaard Brouer
@ 2018-11-07 21:06 ` David Ahern
2018-11-08 13:33 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: David Ahern @ 2018-11-07 21:06 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/3/18 6:24 PM, Paweł Staszewski wrote:
>> Does your setup have any other device types besides physical ports with
>> VLANs (e.g., any macvlans or bonds)?
>>
>>
> no.
> just
> phy(mlnx)->vlans only config
VLAN and non-VLAN (and a mix) seem to work ok. Patches are here:
https://github.com/dsahern/linux.git bpf/kernel-tables-wip
I got lazy with the vlan exports; right now it requires 8021q to be
builtin (CONFIG_VLAN_8021Q=y)
You can use the xdp_fwd sample:
make O=kbuild -C samples/bpf -j 8
Copy samples/bpf/xdp_fwd_kern.o and samples/bpf/xdp_fwd to the server
and run:
./xdp_fwd <list of NIC ports>
e.g., in my testing I run:
xdp_fwd eth1 eth2 eth3 eth4
All of the relevant forwarding ports need to be on the same command
line. This version populates a second map to verify the egress port has
XDP enabled.
>
> And today again after allpy patch for page allocator - reached again
> 64/64 Gbit/s
>
> with only 50-60% cpu load
you should see the cpu load drop considerably.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-05 20:17 ` Jesper Dangaard Brouer
@ 2018-11-08 0:59 ` Paweł Staszewski
2018-11-08 1:13 ` Paweł Staszewski
2018-11-08 14:43 ` Paweł Staszewski
0 siblings, 2 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 0:59 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: David Ahern, netdev, Yoel Caspersen
W dniu 05.11.2018 o 21:17, Jesper Dangaard Brouer pisze:
> On Sun, 4 Nov 2018 01:24:03 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> And today again after allpy patch for page allocator - reached again
>> 64/64 Gbit/s
>>
>> with only 50-60% cpu load
> Great.
>
>> today no slowpath hit for netwoking :)
>>
>> But again dropped pckt at 64GbitRX and 64TX ....
>> And as it should not be pcie express limit -i think something more is
> Well, this does sounds like a PCIe bandwidth limit to me.
>
> See the PCIe BW here: https://en.wikipedia.org/wiki/PCI_Express
>
> You likely have PCIe v3, where 1-lane have 984.6 MBytes/s or 7.87 Gbit/s
> Thus, x16-lanes have 15.75 GBytes or 126 Gbit/s. It does say "in each
> direction", but you are also forwarding this RX->TX on both (dual) ports
> NIC that is sharing the same PCIe slot.
Network controller changed from 2-port 100G connectx4 to 2 separate
cards 100G connectx5
PerfTop: 92239 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6.65% [kernel] [k] irq_entries_start
5.57% [kernel] [k] tasklet_action_common.isra.21
4.60% [kernel] [k] mlx5_eq_int
4.04% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
3.66% [kernel] [k] _raw_spin_lock_irqsave
3.58% [kernel] [k] mlx5e_sq_xmit
2.66% [kernel] [k] fib_table_lookup
2.52% [kernel] [k] _raw_spin_lock
2.51% [kernel] [k] build_skb
2.50% [kernel] [k] _raw_spin_lock_irq
2.04% [kernel] [k] try_to_wake_up
1.83% [kernel] [k] queued_spin_lock_slowpath
1.81% [kernel] [k] mlx5e_poll_tx_cq
1.65% [kernel] [k] do_idle
1.50% [kernel] [k] mlx5e_poll_rx_cq
1.34% [kernel] [k] __sched_text_start
1.32% [kernel] [k] cmd_exec
1.30% [kernel] [k] cmd_work_handler
1.16% [kernel] [k] vlan_do_receive
1.15% [kernel] [k] memcpy_erms
1.15% [kernel] [k] __dev_queue_xmit
1.07% [kernel] [k] mlx5_cmd_comp_handler
1.06% [kernel] [k] sched_ttwu_pending
1.00% [kernel] [k] ipt_do_table
0.98% [kernel] [k] ip_finish_output2
0.92% [kernel] [k] pfifo_fast_dequeue
0.88% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
0.78% [kernel] [k] dev_gro_receive
0.78% [kernel] [k] mlx5e_napi_poll
0.76% [kernel] [k] mlx5e_post_rx_mpwqes
0.70% [kernel] [k] process_one_work
0.67% [kernel] [k] __netif_receive_skb_core
0.65% [kernel] [k] __build_skb
0.63% [kernel] [k] llist_add_batch
0.62% [kernel] [k] tcp_gro_receive
0.60% [kernel] [k] inet_gro_receive
0.59% [kernel] [k] ip_route_input_rcu
0.59% [kernel] [k] rcu_irq_exit
0.56% [kernel] [k] napi_complete_done
0.52% [kernel] [k] kmem_cache_alloc
0.48% [kernel] [k] __softirqentry_text_start
0.48% [kernel] [k] mlx5e_xmit
0.47% [kernel] [k] __queue_work
0.46% [kernel] [k] memset_erms
0.46% [kernel] [k] dev_hard_start_xmit
0.45% [kernel] [k] insert_work
0.45% [kernel] [k] enqueue_task_fair
0.44% [kernel] [k] __wake_up_common
0.43% [kernel] [k] finish_task_switch
0.43% [kernel] [k] kmem_cache_free_bulk
0.42% [kernel] [k] ip_forward
0.42% [kernel] [k] worker_thread
0.41% [kernel] [k] schedule
0.41% [kernel] [k] _raw_spin_unlock_irqrestore
0.40% [kernel] [k] netif_skb_features
0.40% [kernel] [k] queue_work_on
0.40% [kernel] [k] pfifo_fast_enqueue
0.39% [kernel] [k] vlan_dev_hard_start_xmit
0.39% [kernel] [k] page_frag_free
0.36% [kernel] [k] swiotlb_map_page
0.36% [kernel] [k] update_cfs_rq_h_load
0.35% [kernel] [k] validate_xmit_skb.isra.142
0.35% [kernel] [k] dev_ifconf
0.35% [kernel] [k] check_preempt_curr
0.34% [kernel] [k] _raw_spin_trylock
0.34% [kernel] [k] rcu_idle_exit
0.33% [kernel] [k] ip_rcv_core.isra.20.constprop.25
0.33% [kernel] [k] __qdisc_run
0.33% [kernel] [k] skb_release_data
0.32% [kernel] [k] native_sched_clock
0.30% [kernel] [k] add_interrupt_randomness
0.29% [kernel] [k] interrupt_entry
0.28% [kernel] [k] skb_gro_receive
0.26% [kernel] [k] read_tsc
0.26% [kernel] [k] __get_xps_queue_idx
0.26% [kernel] [k] inet_gifconf
0.26% [kernel] [k] skb_segment
0.25% [kernel] [k] __tasklet_schedule_common
0.25% [kernel] [k] smpboot_thread_fn
0.23% [kernel] [k] __update_load_avg_se
0.22% [kernel] [k] tcp4_gro_receive
Not much traffic now:
bwm-ng v0.6.1 (probing every 0.500s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
enp175s0: 6.95 Gb/s 4.20 Gb/s
11.15 Gb/s
enp216s0: 4.23 Gb/s 6.98 Gb/s
11.21 Gb/s
------------------------------------------------------------------------------
total: 11.18 Gb/s 11.18 Gb/s
22.37 Gb/s
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
enp175s0: 700264.50 P/s 923890.25 P/s 1624154.75 P/s
enp216s0: 932598.81 P/s 708771.50 P/s 1641370.25 P/s
------------------------------------------------------------------------------
total: 1632863.38 P/s 1632661.75 P/s 3265525.00 P/s
>
>
>> going on there - and hard to catch - cause perf top doestn chenged
>> besides there is no queued slowpath hit now
>>
>> I ordered now also intel cards to compare - but 3 weeks eta
>> Faster - cause 3 days - i will have mellanox connectx 5 - so can
>> separate traffic to two different x16 pcie busses
> I do think you need to separate traffic to two different x16 PCIe
> slots. I have found that the ConnectX-5 is significantly faster
> packet-per-sec performance than ConnectX-4, but that is not your
> use-case (max BW). I've not tested these NICs for maximum
> _bidirectional_ bandwidth limits, I've only made sure I can do 100G
> unidirectional, which can hit some funny motherboard memory limits
> (remember to equip motherboard with 4 RAM blocks for full memory BW).
>
Yes memory channels are separated and there are 4 modules per cpu :)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 0:59 ` Paweł Staszewski
@ 2018-11-08 1:13 ` Paweł Staszewski
2018-11-08 14:43 ` Paweł Staszewski
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 1:13 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: David Ahern, netdev, Yoel Caspersen
W dniu 08.11.2018 o 01:59, Paweł Staszewski pisze:
>
>
> W dniu 05.11.2018 o 21:17, Jesper Dangaard Brouer pisze:
>> On Sun, 4 Nov 2018 01:24:03 +0100 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> And today again after allpy patch for page allocator - reached again
>>> 64/64 Gbit/s
>>>
>>> with only 50-60% cpu load
>> Great.
>>
>>> today no slowpath hit for netwoking :)
>>>
>>> But again dropped pckt at 64GbitRX and 64TX ....
>>> And as it should not be pcie express limit -i think something more is
>> Well, this does sounds like a PCIe bandwidth limit to me.
>>
>> See the PCIe BW here: https://en.wikipedia.org/wiki/PCI_Express
>>
>> You likely have PCIe v3, where 1-lane have 984.6 MBytes/s or 7.87 Gbit/s
>> Thus, x16-lanes have 15.75 GBytes or 126 Gbit/s. It does say "in each
>> direction", but you are also forwarding this RX->TX on both (dual) ports
>> NIC that is sharing the same PCIe slot.
> Network controller changed from 2-port 100G connectx4 to 2 separate
> cards 100G connectx5
>
>
> PerfTop: 92239 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> 6.65% [kernel] [k] irq_entries_start
> 5.57% [kernel] [k] tasklet_action_common.isra.21
> 4.60% [kernel] [k] mlx5_eq_int
> 4.04% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
> 3.66% [kernel] [k] _raw_spin_lock_irqsave
> 3.58% [kernel] [k] mlx5e_sq_xmit
> 2.66% [kernel] [k] fib_table_lookup
> 2.52% [kernel] [k] _raw_spin_lock
> 2.51% [kernel] [k] build_skb
> 2.50% [kernel] [k] _raw_spin_lock_irq
> 2.04% [kernel] [k] try_to_wake_up
> 1.83% [kernel] [k] queued_spin_lock_slowpath
> 1.81% [kernel] [k] mlx5e_poll_tx_cq
> 1.65% [kernel] [k] do_idle
> 1.50% [kernel] [k] mlx5e_poll_rx_cq
> 1.34% [kernel] [k] __sched_text_start
> 1.32% [kernel] [k] cmd_exec
> 1.30% [kernel] [k] cmd_work_handler
> 1.16% [kernel] [k] vlan_do_receive
> 1.15% [kernel] [k] memcpy_erms
> 1.15% [kernel] [k] __dev_queue_xmit
> 1.07% [kernel] [k] mlx5_cmd_comp_handler
> 1.06% [kernel] [k] sched_ttwu_pending
> 1.00% [kernel] [k] ipt_do_table
> 0.98% [kernel] [k] ip_finish_output2
> 0.92% [kernel] [k] pfifo_fast_dequeue
> 0.88% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
> 0.78% [kernel] [k] dev_gro_receive
> 0.78% [kernel] [k] mlx5e_napi_poll
> 0.76% [kernel] [k] mlx5e_post_rx_mpwqes
> 0.70% [kernel] [k] process_one_work
> 0.67% [kernel] [k] __netif_receive_skb_core
> 0.65% [kernel] [k] __build_skb
> 0.63% [kernel] [k] llist_add_batch
> 0.62% [kernel] [k] tcp_gro_receive
> 0.60% [kernel] [k] inet_gro_receive
> 0.59% [kernel] [k] ip_route_input_rcu
> 0.59% [kernel] [k] rcu_irq_exit
> 0.56% [kernel] [k] napi_complete_done
> 0.52% [kernel] [k] kmem_cache_alloc
> 0.48% [kernel] [k] __softirqentry_text_start
> 0.48% [kernel] [k] mlx5e_xmit
> 0.47% [kernel] [k] __queue_work
> 0.46% [kernel] [k] memset_erms
> 0.46% [kernel] [k] dev_hard_start_xmit
> 0.45% [kernel] [k] insert_work
> 0.45% [kernel] [k] enqueue_task_fair
> 0.44% [kernel] [k] __wake_up_common
> 0.43% [kernel] [k] finish_task_switch
> 0.43% [kernel] [k] kmem_cache_free_bulk
> 0.42% [kernel] [k] ip_forward
> 0.42% [kernel] [k] worker_thread
> 0.41% [kernel] [k] schedule
> 0.41% [kernel] [k] _raw_spin_unlock_irqrestore
> 0.40% [kernel] [k] netif_skb_features
> 0.40% [kernel] [k] queue_work_on
> 0.40% [kernel] [k] pfifo_fast_enqueue
> 0.39% [kernel] [k] vlan_dev_hard_start_xmit
> 0.39% [kernel] [k] page_frag_free
> 0.36% [kernel] [k] swiotlb_map_page
> 0.36% [kernel] [k] update_cfs_rq_h_load
> 0.35% [kernel] [k] validate_xmit_skb.isra.142
> 0.35% [kernel] [k] dev_ifconf
> 0.35% [kernel] [k] check_preempt_curr
> 0.34% [kernel] [k] _raw_spin_trylock
> 0.34% [kernel] [k] rcu_idle_exit
> 0.33% [kernel] [k] ip_rcv_core.isra.20.constprop.25
> 0.33% [kernel] [k] __qdisc_run
> 0.33% [kernel] [k] skb_release_data
> 0.32% [kernel] [k] native_sched_clock
> 0.30% [kernel] [k] add_interrupt_randomness
> 0.29% [kernel] [k] interrupt_entry
> 0.28% [kernel] [k] skb_gro_receive
> 0.26% [kernel] [k] read_tsc
> 0.26% [kernel] [k] __get_xps_queue_idx
> 0.26% [kernel] [k] inet_gifconf
> 0.26% [kernel] [k] skb_segment
> 0.25% [kernel] [k] __tasklet_schedule_common
> 0.25% [kernel] [k] smpboot_thread_fn
> 0.23% [kernel] [k] __update_load_avg_se
> 0.22% [kernel] [k] tcp4_gro_receive
>
>
> Not much traffic now:
> bwm-ng v0.6.1 (probing every 0.500s), press 'h' for help
> input: /proc/net/dev type: rate
> | iface Rx Tx Total
> ==============================================================================
>
> enp175s0: 6.95 Gb/s 4.20 Gb/s
> 11.15 Gb/s
> enp216s0: 4.23 Gb/s 6.98 Gb/s
> 11.21 Gb/s
> ------------------------------------------------------------------------------
>
> total: 11.18 Gb/s 11.18 Gb/s
> 22.37 Gb/s
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> | iface Rx Tx Total
> ==============================================================================
>
> enp175s0: 700264.50 P/s 923890.25 P/s 1624154.75
> P/s
> enp216s0: 932598.81 P/s 708771.50 P/s 1641370.25
> P/s
> ------------------------------------------------------------------------------
>
> total: 1632863.38 P/s 1632661.75 P/s 3265525.00
> P/s
>
>
>
>
Also is that normal that some kworker procs takes 10%+ of cpu ?
below top
2913 root 20 0 0 0 0 I 10.3 0.0 6:58.29
kworker/u112:1-
7 root 20 0 0 0 0 I 8.6 0.0 6:17.18
kworker/u112:0-
10289 root 20 0 0 0 0 I 6.6 0.0 6:33.90
kworker/u112:4-
2939 root 20 0 0 0 0 R 3.6 0.0 7:37.68
kworker/u112:2-
4557 root 20 0 0 0 0 I 1.3 0.0 0:08.82
kworker/45:4-ev
6775 root 20 0 0 0 0 I 1.3 0.0 0:26.30
kworker/50:4-ev
6833 root 20 0 0 0 0 D 1.3 0.0 0:04.96
kworker/15:0+ev
6840 root 20 0 0 0 0 I 1.3 0.0 0:09.32
kworker/55:2-ev
6874 root 20 0 0 0 0 D 1.3 0.0 0:08.51
kworker/53:0+ev
7710 root 20 0 0 0 0 I 1.3 0.0 0:07.78
kworker/14:1-ev
12075 root 20 0 0 0 0 I 1.3 0.0 1:19.22
kworker/23:3-ev
31209 root 20 0 0 0 0 I 1.3 0.0 0:07.02
kworker/20:1-ev
32351 root 20 0 0 0 0 R 1.3 0.0 0:06.99
kworker/51:2+ev
39869 root 20 0 0 0 0 D 1.3 0.0 0:06.15
kworker/42:0+ev
39959 root 20 0 0 0 0 I 1.3 0.0 0:16.23
kworker/51:1-ev
42858 root 20 0 0 0 0 I 1.3 0.0 0:47.72
kworker/27:2-ev
43281 root 20 0 0 0 0 I 1.3 0.0 0:14.99
kworker/14:4-ev
43282 root 20 0 0 0 0 I 1.3 0.0 0:13.38
kworker/16:1-ev
43389 root 20 0 0 0 0 D 1.3 0.0 0:08.92
kworker/54:2+ev
45214 root 20 0 0 0 0 I 1.3 0.0 0:05.82
kworker/55:0-ev
46894 root 20 0 0 0 0 I 1.3 0.0 0:04.11
kworker/46:1-ev
47027 root 20 0 0 0 0 D 1.3 0.0 0:03.79
kworker/47:1+ev
47129 root 20 0 0 0 0 D 1.3 0.0 0:03.15
kworker/52:0+ev
47133 root 20 0 0 0 0 I 1.3 0.0 0:03.19
kworker/49:1-ev
47179 root 20 0 0 0 0 I 1.3 0.0 0:02.83
kworker/17:3-ev
48062 root 20 0 0 0 0 I 1.3 0.0 0:02.54
kworker/44:1-ev
48158 root 20 0 0 0 0 I 1.3 0.0 0:02.17
kworker/16:2-ev
48168 root 20 0 0 0 0 I 1.3 0.0 0:02.13
kworker/27:3-ev
48247 root 20 0 0 0 0 I 1.3 0.0 0:01.83
kworker/22:0-ev
48337 root 20 0 0 0 0 I 1.3 0.0 0:01.57
kworker/15:1-ev
48345 root 20 0 0 0 0 I 1.3 0.0 0:01.49
kworker/24:3-ev
49302 root 20 0 0 0 0 I 1.3 0.0 0:00.71
kworker/54:1-ev
49366 root 20 0 0 0 0 I 1.3 0.0 0:00.38
kworker/20:3-ev
49400 root 20 0 0 0 0 I 1.3 0.0 0:00.31
kworker/26:2-ev
49430 root 20 0 0 0 0 I 1.3 0.0 0:00.21
kworker/42:2-ev
49463 root 20 0 0 0 0 D 1.3 0.0 0:00.08
kworker/50:2+ev
51698 root 20 0 0 0 0 D 1.3 0.0 0:14.85
kworker/46:2+ev
54238 root 20 0 0 0 0 I 1.3 0.0 0:23.73
kworker/52:1-ev
2507 root 20 0 0 0 0 I 1.0 0.0 0:09.60
kworker/44:2-ev
4525 root 20 0 0 0 0 I 1.0 0.0 0:08.07
kworker/26:1-ev
4556 root 20 0 0 0 0 I 1.0 0.0 0:05.15
kworker/48:0-ev
4604 root 20 0 0 0 0 I 1.0 0.0 0:10.90
kworker/19:0-ev
5789 root 20 0 0 0 0 I 1.0 0.0 0:08.24
kworker/18:0-ev
6868 root 20 0 0 0 0 I 1.0 0.0 0:09.68
kworker/47:0-ev
6900 root 20 0 0 0 0 I 1.0 0.0 0:28.83
kworker/18:1-ev
7764 root 20 0 0 0 0 I 1.0 0.0 0:03.00
kworker/49:2-ev
12045 root 20 0 0 0 0 I 1.0 0.0 1:16.98
kworker/24:2-ev
32218 root 20 0 0 0 0 I 1.0 0.0 0:04.13
kworker/45:2-ev
34082 root 20 0 0 0 0 I 1.0 0.0 0:06.29
kworker/17:1-ev
39791 root 20 0 0 0 0 I 1.0 0.0 0:19.51
kworker/21:4-ev
39973 root 20 0 0 0 0 I 1.0 0.0 0:17.12
kworker/53:2-ev
43223 root 20 0 0 0 0 I 1.0 0.0 0:07.88
kworker/25:0-ev
43295 root 20 0 0 0 0 I 1.0 0.0 0:10.89
kworker/22:4-ev
46055 root 20 0 0 0 0 I 1.0 0.0 0:04.00
kworker/21:2-ev
46077 root 20 0 0 0 0 I 1.0 0.0 0:04.62
kworker/19:1-ev
47204 root 20 0 0 0 0 I 1.0 0.0 0:03.03
kworker/25:2-ev
47989 root 20 0 0 0 0 I 1.0 0.0 0:02.65
kworker/43:1-ev
49127 root 20 0 0 0 0 I 1.0 0.0 0:01.10
kworker/48:2-ev
49317 root 20 0 0 0 0 I 1.0 0.0 0:00.56
kworker/23:1-ev
54191 root 20 0 0 0 0 R 1.0 0.0 0:30.27
kworker/43:2+ev
81 root 20 0 0 0 0 S 0.7 0.0 0:50.27
ksoftirqd/14
87 root 20 0 0 0 0 S 0.7 0.0 1:02.92
ksoftirqd/15
102 root 20 0 0 0 0 S 0.7 0.0 0:29.78
ksoftirqd/18
117 root 20 0 0 0 0 S 0.7 0.0 0:30.73
ksoftirqd/21
127 root 20 0 0 0 0 S 0.7 0.0 0:24.45
ksoftirqd/23
137 root 20 0 0 0 0 S 0.7 0.0 0:24.94
ksoftirqd/25
142 root 20 0 0 0 0 S 0.7 0.0 0:21.74
ksoftirqd/26
222 root 20 0 0 0 0 S 0.7 0.0 0:27.83
ksoftirqd/42
227 root 20 0 0 0 0 S 0.7 0.0 0:25.35
ksoftirqd/43
242 root 20 0 0 0 0 S 0.7 0.0 0:21.40
ksoftirqd/46
267 root 20 0 0 0 0 S 0.7 0.0 0:08.62
ksoftirqd/51
5174 root 20 0 0 0 0 I 0.7 0.0 5:57.10
kworker/u112:3-
>
>
>>
>>
>>> going on there - and hard to catch - cause perf top doestn chenged
>>> besides there is no queued slowpath hit now
>>>
>>> I ordered now also intel cards to compare - but 3 weeks eta
>>> Faster - cause 3 days - i will have mellanox connectx 5 - so can
>>> separate traffic to two different x16 pcie busses
>> I do think you need to separate traffic to two different x16 PCIe
>> slots. I have found that the ConnectX-5 is significantly faster
>> packet-per-sec performance than ConnectX-4, but that is not your
>> use-case (max BW). I've not tested these NICs for maximum
>> _bidirectional_ bandwidth limits, I've only made sure I can do 100G
>> unidirectional, which can hit some funny motherboard memory limits
>> (remember to equip motherboard with 4 RAM blocks for full memory BW).
>>
> Yes memory channels are separated and there are 4 modules per cpu :)
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-07 21:06 ` David Ahern
@ 2018-11-08 13:33 ` Paweł Staszewski
2018-11-08 16:06 ` David Ahern
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 13:33 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 07.11.2018 o 22:06, David Ahern pisze:
> On 11/3/18 6:24 PM, Paweł Staszewski wrote:
>>> Does your setup have any other device types besides physical ports with
>>> VLANs (e.g., any macvlans or bonds)?
>>>
>>>
>> no.
>> just
>> phy(mlnx)->vlans only config
> VLAN and non-VLAN (and a mix) seem to work ok. Patches are here:
> https://github.com/dsahern/linux.git bpf/kernel-tables-wip
>
> I got lazy with the vlan exports; right now it requires 8021q to be
> builtin (CONFIG_VLAN_8021Q=y)
>
> You can use the xdp_fwd sample:
> make O=kbuild -C samples/bpf -j 8
>
> Copy samples/bpf/xdp_fwd_kern.o and samples/bpf/xdp_fwd to the server
> and run:
> ./xdp_fwd <list of NIC ports>
>
> e.g., in my testing I run:
> xdp_fwd eth1 eth2 eth3 eth4
>
> All of the relevant forwarding ports need to be on the same command
> line. This version populates a second map to verify the egress port has
> XDP enabled.
Installed today on some lab server with mellanox connectx4
And trying some simple static routing first - but after enabling xdp
program - receiver is not receiving frames
Route table is simple as possible for tests :)
icmp ping test send from 192.168.22.237 to 172.16.0.2 - incomming
packets on vlan 4081
ip r
default via 192.168.22.236 dev vlan4081
172.16.0.0/30 dev vlan1740 proto kernel scope link src 172.16.0.1
192.168.22.0/24 dev vlan4081 proto kernel scope link src 192.168.22.205
neigh table:
ip neigh ls
192.168.22.237 dev vlan4081 lladdr 00:25:90:fb:a6:8d REACHABLE
172.16.0.2 dev vlan1740 lladdr ac:1f:6b:2c:2e:5a REACHABLE
and interfaces:
4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5 qdisc
mq state UP group default qlen 1000
link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
valid_lft forever preferred_lft forever
6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP group default qlen 1000
link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
inet 192.168.22.205/24 scope global vlan4081
valid_lft forever preferred_lft forever
inet6 fe80::ae1f:6bff:fe07:c890/64 scope link
valid_lft forever preferred_lft forever
7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP group default qlen 1000
link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.1/30 scope global vlan1740
valid_lft forever preferred_lft forever
inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
valid_lft forever preferred_lft forever
xdp program detached:
Receiving side tcpdump:
14:28:09.141233 IP 192.168.22.237 > 172.16.0.2: ICMP echo request, id
30227, seq 487, length 64
I can see icmp requests
enabling xdp
./xdp_fwd enp175s0f1 enp175s0f0
4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
prog/xdp id 5 tag 3c231ff1e5e77f3f
5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
prog/xdp id 5 tag 3c231ff1e5e77f3f
6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
Receiving side no icmp echo requests incommint to interface.
And some ethtool stats for xdp interface trat receiving icmp requests
from sender to be forwarded:
ethtool -S enp175s0f0 | grep 'rx_xdp_redirect'
rx_xdp_redirect: 321
ethtool stats for interface that should forward icmp requests to
receiver on vlan id 1740
ethtool -S enp175s0f1 | grep 'tx_xdp'
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
No frames tx-ed.
>
>> And today again after allpy patch for page allocator - reached again
>> 64/64 Gbit/s
>>
>> with only 50-60% cpu load
> you should see the cpu load drop considerably.
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 0:59 ` Paweł Staszewski
2018-11-08 1:13 ` Paweł Staszewski
@ 2018-11-08 14:43 ` Paweł Staszewski
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 14:43 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: David Ahern, netdev, Yoel Caspersen
W dniu 08.11.2018 o 01:59, Paweł Staszewski pisze:
>
>
> W dniu 05.11.2018 o 21:17, Jesper Dangaard Brouer pisze:
>> On Sun, 4 Nov 2018 01:24:03 +0100 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> And today again after allpy patch for page allocator - reached again
>>> 64/64 Gbit/s
>>>
>>> with only 50-60% cpu load
>> Great.
>>
>>> today no slowpath hit for netwoking :)
>>>
>>> But again dropped pckt at 64GbitRX and 64TX ....
>>> And as it should not be pcie express limit -i think something more is
>> Well, this does sounds like a PCIe bandwidth limit to me.
>>
>> See the PCIe BW here: https://en.wikipedia.org/wiki/PCI_Express
>>
>> You likely have PCIe v3, where 1-lane have 984.6 MBytes/s or 7.87 Gbit/s
>> Thus, x16-lanes have 15.75 GBytes or 126 Gbit/s. It does say "in each
>> direction", but you are also forwarding this RX->TX on both (dual) ports
>> NIC that is sharing the same PCIe slot.
> Network controller changed from 2-port 100G connectx4 to 2 separate
> cards 100G connectx5
>
>
> PerfTop: 92239 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> 6.65% [kernel] [k] irq_entries_start
> 5.57% [kernel] [k] tasklet_action_common.isra.21
> 4.60% [kernel] [k] mlx5_eq_int
> 4.04% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
> 3.66% [kernel] [k] _raw_spin_lock_irqsave
> 3.58% [kernel] [k] mlx5e_sq_xmit
> 2.66% [kernel] [k] fib_table_lookup
> 2.52% [kernel] [k] _raw_spin_lock
> 2.51% [kernel] [k] build_skb
> 2.50% [kernel] [k] _raw_spin_lock_irq
> 2.04% [kernel] [k] try_to_wake_up
> 1.83% [kernel] [k] queued_spin_lock_slowpath
> 1.81% [kernel] [k] mlx5e_poll_tx_cq
> 1.65% [kernel] [k] do_idle
> 1.50% [kernel] [k] mlx5e_poll_rx_cq
> 1.34% [kernel] [k] __sched_text_start
> 1.32% [kernel] [k] cmd_exec
> 1.30% [kernel] [k] cmd_work_handler
> 1.16% [kernel] [k] vlan_do_receive
> 1.15% [kernel] [k] memcpy_erms
> 1.15% [kernel] [k] __dev_queue_xmit
> 1.07% [kernel] [k] mlx5_cmd_comp_handler
> 1.06% [kernel] [k] sched_ttwu_pending
> 1.00% [kernel] [k] ipt_do_table
> 0.98% [kernel] [k] ip_finish_output2
> 0.92% [kernel] [k] pfifo_fast_dequeue
> 0.88% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
> 0.78% [kernel] [k] dev_gro_receive
> 0.78% [kernel] [k] mlx5e_napi_poll
> 0.76% [kernel] [k] mlx5e_post_rx_mpwqes
> 0.70% [kernel] [k] process_one_work
> 0.67% [kernel] [k] __netif_receive_skb_core
> 0.65% [kernel] [k] __build_skb
> 0.63% [kernel] [k] llist_add_batch
> 0.62% [kernel] [k] tcp_gro_receive
> 0.60% [kernel] [k] inet_gro_receive
> 0.59% [kernel] [k] ip_route_input_rcu
> 0.59% [kernel] [k] rcu_irq_exit
> 0.56% [kernel] [k] napi_complete_done
> 0.52% [kernel] [k] kmem_cache_alloc
> 0.48% [kernel] [k] __softirqentry_text_start
> 0.48% [kernel] [k] mlx5e_xmit
> 0.47% [kernel] [k] __queue_work
> 0.46% [kernel] [k] memset_erms
> 0.46% [kernel] [k] dev_hard_start_xmit
> 0.45% [kernel] [k] insert_work
> 0.45% [kernel] [k] enqueue_task_fair
> 0.44% [kernel] [k] __wake_up_common
> 0.43% [kernel] [k] finish_task_switch
> 0.43% [kernel] [k] kmem_cache_free_bulk
> 0.42% [kernel] [k] ip_forward
> 0.42% [kernel] [k] worker_thread
> 0.41% [kernel] [k] schedule
> 0.41% [kernel] [k] _raw_spin_unlock_irqrestore
> 0.40% [kernel] [k] netif_skb_features
> 0.40% [kernel] [k] queue_work_on
> 0.40% [kernel] [k] pfifo_fast_enqueue
> 0.39% [kernel] [k] vlan_dev_hard_start_xmit
> 0.39% [kernel] [k] page_frag_free
> 0.36% [kernel] [k] swiotlb_map_page
> 0.36% [kernel] [k] update_cfs_rq_h_load
> 0.35% [kernel] [k] validate_xmit_skb.isra.142
> 0.35% [kernel] [k] dev_ifconf
> 0.35% [kernel] [k] check_preempt_curr
> 0.34% [kernel] [k] _raw_spin_trylock
> 0.34% [kernel] [k] rcu_idle_exit
> 0.33% [kernel] [k] ip_rcv_core.isra.20.constprop.25
> 0.33% [kernel] [k] __qdisc_run
> 0.33% [kernel] [k] skb_release_data
> 0.32% [kernel] [k] native_sched_clock
> 0.30% [kernel] [k] add_interrupt_randomness
> 0.29% [kernel] [k] interrupt_entry
> 0.28% [kernel] [k] skb_gro_receive
> 0.26% [kernel] [k] read_tsc
> 0.26% [kernel] [k] __get_xps_queue_idx
> 0.26% [kernel] [k] inet_gifconf
> 0.26% [kernel] [k] skb_segment
> 0.25% [kernel] [k] __tasklet_schedule_common
> 0.25% [kernel] [k] smpboot_thread_fn
> 0.23% [kernel] [k] __update_load_avg_se
> 0.22% [kernel] [k] tcp4_gro_receive
>
>
> Not much traffic now:
> bwm-ng v0.6.1 (probing every 0.500s), press 'h' for help
> input: /proc/net/dev type: rate
> | iface Rx Tx Total
> ==============================================================================
>
> enp175s0: 6.95 Gb/s 4.20 Gb/s
> 11.15 Gb/s
> enp216s0: 4.23 Gb/s 6.98 Gb/s
> 11.21 Gb/s
> ------------------------------------------------------------------------------
>
> total: 11.18 Gb/s 11.18 Gb/s
> 22.37 Gb/s
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> | iface Rx Tx Total
> ==============================================================================
>
> enp175s0: 700264.50 P/s 923890.25 P/s 1624154.75
> P/s
> enp216s0: 932598.81 P/s 708771.50 P/s 1641370.25
> P/s
> ------------------------------------------------------------------------------
>
> total: 1632863.38 P/s 1632661.75 P/s 3265525.00
> P/s
>
>
>
updated perf top - more traffic 37Gbit/37Gbit total traffic
bwm-ng v0.6.1 (probing every 0.500s), press 'h' for help
input: /proc/net/dev type: rate
/ iface Rx Tx Total
==============================================================================
enp175s0: 28.91 Gb/s 8.89 Gb/s
37.80 Gb/s
enp216s0: 8.91 Gb/s 28.95 Gb/s
37.86 Gb/s
------------------------------------------------------------------------------
total: 37.82 Gb/s 37.84 Gb/s
75.67 Gb/s
bwm-ng v0.6.1 (probing every 0.500s), press 'h' for help
input: /proc/net/dev type: rate
- iface Rx Tx Total
==============================================================================
enp175s0: 2721518.75 P/s 2460930.50 P/s 5182449.50 P/s
enp216s0: 2471451.25 P/s 2731946.25 P/s 5203397.50 P/s
------------------------------------------------------------------------------
total: 5192970.00 P/s 5192876.50 P/s 10385847.00 P/s
PerfTop: 56488 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.41% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
7.73% [kernel] [k] mlx5e_sq_xmit
6.05% [kernel] [k] build_skb
5.63% [kernel] [k] fib_table_lookup
2.75% [kernel] [k] mlx5e_poll_rx_cq
2.74% [kernel] [k] memcpy_erms
2.33% [kernel] [k] vlan_do_receive
2.00% [kernel] [k] __dev_queue_xmit
1.81% [kernel] [k] ip_finish_output2
1.79% [kernel] [k] dev_gro_receive
1.78% [kernel] [k] ipt_do_table
1.78% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
1.76% [kernel] [k] pfifo_fast_dequeue
1.70% [kernel] [k] mlx5e_post_rx_mpwqes
1.52% [kernel] [k] mlx5e_poll_tx_cq
1.49% [kernel] [k] irq_entries_start
1.47% [kernel] [k] _raw_spin_lock
1.45% [kernel] [k] inet_gro_receive
1.42% [kernel] [k] __netif_receive_skb_core
1.39% [kernel] [k] mlx5_eq_int
1.39% [kernel] [k] tcp_gro_receive
1.23% [kernel] [k] __build_skb
1.14% [kernel] [k] ip_route_input_rcu
1.00% [kernel] [k] vlan_dev_hard_start_xmit
0.92% [kernel] [k] _raw_spin_lock_irqsave
0.89% [kernel] [k] kmem_cache_alloc
0.88% [kernel] [k] dev_hard_start_xmit
0.88% [kernel] [k] swiotlb_map_page
0.86% [kernel] [k] mlx5e_xmit
0.81% [kernel] [k] ip_forward
0.80% [kernel] [k] tasklet_action_common.isra.21
0.79% [kernel] [k] netif_skb_features
0.77% [kernel] [k] pfifo_fast_enqueue
0.66% [kernel] [k] validate_xmit_skb.isra.142
0.64% [kernel] [k] ip_rcv_core.isra.20.constprop.25
0.63% [kernel] [k] find_busiest_group
0.60% [kernel] [k] __qdisc_run
0.59% [kernel] [k] skb_release_data
0.59% [kernel] [k] skb_gro_receive
0.58% [kernel] [k] page_frag_free
0.53% [kernel] [k] skb_segment
0.52% [kernel] [k] try_to_wake_up
0.52% [kernel] [k] _raw_spin_lock_irq
0.50% [kernel] [k] tcp4_gro_receive
0.47% [kernel] [k] kmem_cache_free_bulk
0.45% [kernel] [k] mlx5e_page_release
0.43% [kernel] [k] _raw_spin_trylock
0.39% [kernel] [k] kmem_cache_free
0.38% [kernel] [k] __sched_text_start
0.38% [kernel] [k] sch_direct_xmit
0.38% [kernel] [k] do_idle
0.34% [kernel] [k] vlan_passthru_hard_header
0.34% [kernel] [k] cmd_exec
0.34% [kernel] [k] __local_bh_enable_ip
0.33% [kernel] [k] inet_lookup_ifaddr_rcu
0.33% [kernel] [k] skb_network_protocol
0.33% [kernel] [k] netdev_pick_tx
0.33% [kernel] [k] eth_type_trans
0.32% [kernel] [k] __get_xps_queue_idx
0.31% [kernel] [k] __slab_free.isra.79
0.29% [kernel] [k] mlx5e_xdp_handle
0.27% [kernel] [k] sched_ttwu_pending
0.26% [kernel] [k] cmd_work_handler
0.24% [kernel] [k] ip_finish_output
0.23% [kernel] [k] neigh_connected_output
0.23% [kernel] [k] napi_gro_receive
0.23% [kernel] [k] mlx5e_napi_poll
0.23% [kernel] [k] mlx5e_features_check
0.22% [kernel] [k] ip_output
0.21% [kernel] [k] ip_rcv_finish_core.isra.17
0.21% [kernel] [k] fib_validate_source
0.20% [kernel] [k] dev_ifconf
0.20% [kernel] [k] eth_header
0.20% [kernel] [k] __netdev_pick_tx
0.20% [kernel] [k] mlx5_cmd_comp_handler
0.19% [kernel] [k] memset_erms
0.18% [kernel] [k] __netif_receive_skb_one_core
0.18% [kernel] [k] __memcpy
0.18% [kernel] [k] queued_spin_lock_slowpath
0.18% [kernel] [k] nf_hook_slow
0.17% [kernel] [k] enqueue_task_fair
Also modified a little coal settings for connectx5 compared to connectx4
ethtool -c enp175s0
Coalesce parameters for enp175s0:
Adaptive RX: off TX: on
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32588
rx-usecs: 128
rx-frames: 128
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 8
tx-frames: 128
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
So far cpu load (looks better than in previous configuration with 2 port
100G connectx4):
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.05 0.00 0.64 0.01 0.00 8.79 0.00
0.00 0.00 90.51
Average: 0 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 7 0.10 0.00 1.30 0.00 0.00 0.00 0.00
0.00 0.00 98.60
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 13 0.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 14 0.10 0.00 0.80 0.00 0.00 22.80 0.00
0.00 0.00 76.30
Average: 15 0.10 0.00 0.70 0.00 0.00 21.20 0.00
0.00 0.00 78.00
Average: 16 0.00 0.00 0.80 0.00 0.00 17.70 0.00
0.00 0.00 81.50
Average: 17 0.00 0.00 0.50 0.00 0.00 15.10 0.00
0.00 0.00 84.40
Average: 18 0.00 0.00 0.70 0.00 0.00 20.90 0.00
0.00 0.00 78.40
Average: 19 0.10 0.00 0.70 0.00 0.00 20.50 0.00
0.00 0.00 78.70
Average: 20 0.50 0.00 1.70 0.00 0.00 18.80 0.00
0.00 0.00 79.00
Average: 21 0.10 0.00 1.30 0.00 0.00 20.90 0.00
0.00 0.00 77.70
Average: 22 0.00 0.00 0.70 0.00 0.00 19.40 0.00
0.00 0.00 79.90
Average: 23 0.00 0.00 0.90 0.00 0.00 18.50 0.00
0.00 0.00 80.60
Average: 24 0.10 0.00 1.00 0.00 0.00 15.80 0.00
0.00 0.00 83.10
Average: 25 0.00 0.00 0.70 0.00 0.00 19.50 0.00
0.00 0.00 79.80
Average: 26 0.00 0.00 0.50 0.00 0.00 18.30 0.00
0.00 0.00 81.20
Average: 27 0.00 0.00 0.70 0.00 0.00 17.60 0.00
0.00 0.00 81.70
Average: 28 0.00 0.00 0.70 0.00 0.00 0.00 0.00
0.00 0.00 99.30
Average: 29 0.00 0.00 2.00 0.00 0.00 0.00 0.00
0.00 0.00 98.00
Average: 30 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 31 0.00 0.00 2.50 0.00 0.00 0.00 0.00
0.00 0.00 97.50
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.70 0.00 0.00 0.00 0.00
0.00 0.00 99.30
Average: 36 0.00 0.00 2.00 0.00 0.00 0.00 0.00
0.00 0.00 98.00
Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 39 0.00 0.00 1.40 0.00 0.00 0.00 0.00
0.00 0.00 98.60
Average: 40 0.60 0.00 0.40 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 42 0.00 0.00 1.20 0.00 0.00 17.70 0.00
0.00 0.00 81.10
Average: 43 0.00 0.00 0.70 0.00 0.00 20.00 0.00
0.00 0.00 79.30
Average: 44 0.00 0.00 0.50 0.00 0.00 16.10 0.00
0.00 0.00 83.40
Average: 45 0.30 0.00 1.10 0.00 0.00 16.10 0.00
0.00 0.00 82.50
Average: 46 0.00 0.00 0.80 0.00 0.00 14.80 0.00
0.00 0.00 84.40
Average: 47 0.10 0.00 1.60 0.00 0.00 17.20 0.00
0.00 0.00 81.10
Average: 48 0.00 0.00 0.60 0.00 0.00 15.00 0.00
0.00 0.00 84.40
Average: 49 0.10 0.00 0.80 0.00 0.00 14.90 0.00
0.00 0.00 84.20
Average: 50 0.20 0.00 0.50 0.70 0.00 13.60 0.00
0.00 0.00 85.00
Average: 51 0.00 0.00 0.70 0.00 0.00 14.10 0.00
0.00 0.00 85.20
Average: 52 0.20 0.00 1.60 0.00 0.00 16.80 0.00
0.00 0.00 81.40
Average: 53 0.00 0.00 0.80 0.00 0.00 13.20 0.00
0.00 0.00 86.00
Average: 54 0.20 0.00 0.50 0.00 0.00 17.20 0.00
0.00 0.00 82.10
Average: 55 0.00 0.00 0.40 0.00 0.00 18.30 0.00
0.00 0.00 81.30
>
>
>
>>
>>
>>> going on there - and hard to catch - cause perf top doestn chenged
>>> besides there is no queued slowpath hit now
>>>
>>> I ordered now also intel cards to compare - but 3 weeks eta
>>> Faster - cause 3 days - i will have mellanox connectx 5 - so can
>>> separate traffic to two different x16 pcie busses
>> I do think you need to separate traffic to two different x16 PCIe
>> slots. I have found that the ConnectX-5 is significantly faster
>> packet-per-sec performance than ConnectX-4, but that is not your
>> use-case (max BW). I've not tested these NICs for maximum
>> _bidirectional_ bandwidth limits, I've only made sure I can do 100G
>> unidirectional, which can hit some funny motherboard memory limits
>> (remember to equip motherboard with 4 RAM blocks for full memory BW).
>>
> Yes memory channels are separated and there are 4 modules per cpu :)
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 13:33 ` Paweł Staszewski
@ 2018-11-08 16:06 ` David Ahern
2018-11-08 16:25 ` Paweł Staszewski
2018-11-09 10:20 ` Paweł Staszewski
0 siblings, 2 replies; 77+ messages in thread
From: David Ahern @ 2018-11-08 16:06 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/8/18 6:33 AM, Paweł Staszewski wrote:
>
>
> W dniu 07.11.2018 o 22:06, David Ahern pisze:
>> On 11/3/18 6:24 PM, Paweł Staszewski wrote:
>>>> Does your setup have any other device types besides physical ports with
>>>> VLANs (e.g., any macvlans or bonds)?
>>>>
>>>>
>>> no.
>>> just
>>> phy(mlnx)->vlans only config
>> VLAN and non-VLAN (and a mix) seem to work ok. Patches are here:
>> https://github.com/dsahern/linux.git bpf/kernel-tables-wip
>>
>> I got lazy with the vlan exports; right now it requires 8021q to be
>> builtin (CONFIG_VLAN_8021Q=y)
>>
>> You can use the xdp_fwd sample:
>> make O=kbuild -C samples/bpf -j 8
>>
>> Copy samples/bpf/xdp_fwd_kern.o and samples/bpf/xdp_fwd to the server
>> and run:
>> ./xdp_fwd <list of NIC ports>
>>
>> e.g., in my testing I run:
>> xdp_fwd eth1 eth2 eth3 eth4
>>
>> All of the relevant forwarding ports need to be on the same command
>> line. This version populates a second map to verify the egress port has
>> XDP enabled.
> Installed today on some lab server with mellanox connectx4
>
> And trying some simple static routing first - but after enabling xdp
> program - receiver is not receiving frames
>
> Route table is simple as possible for tests :)
>
> icmp ping test send from 192.168.22.237 to 172.16.0.2 - incomming
> packets on vlan 4081
>
> ip r
> default via 192.168.22.236 dev vlan4081
> 172.16.0.0/30 dev vlan1740 proto kernel scope link src 172.16.0.1
> 192.168.22.0/24 dev vlan4081 proto kernel scope link src 192.168.22.205
>
> neigh table:
> ip neigh ls
>
> 192.168.22.237 dev vlan4081 lladdr 00:25:90:fb:a6:8d REACHABLE
> 172.16.0.2 dev vlan1740 lladdr ac:1f:6b:2c:2e:5a REACHABLE
>
> and interfaces:
> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
> UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
> UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>
> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5 qdisc
> mq state UP group default qlen 1000
> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
> valid_lft forever preferred_lft forever
> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP group default qlen 1000
> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
> inet 192.168.22.205/24 scope global vlan4081
> valid_lft forever preferred_lft forever
> inet6 fe80::ae1f:6bff:fe07:c890/64 scope link
> valid_lft forever preferred_lft forever
> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP group default qlen 1000
> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
> inet 172.16.0.1/30 scope global vlan1740
> valid_lft forever preferred_lft forever
> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
> valid_lft forever preferred_lft forever
>
>
> xdp program detached:
> Receiving side tcpdump:
> 14:28:09.141233 IP 192.168.22.237 > 172.16.0.2: ICMP echo request, id
> 30227, seq 487, length 64
>
> I can see icmp requests
>
>
> enabling xdp
> ./xdp_fwd enp175s0f1 enp175s0f0
>
> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
> state UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
> prog/xdp id 5 tag 3c231ff1e5e77f3f
> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
> state UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
> prog/xdp id 5 tag 3c231ff1e5e77f3f
> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>
What hardware is this?
Start with:
echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
cat /sys/kernel/debug/tracing/trace_pipe
>From there, you can check the FIB lookups:
sysctl -w kernel.perf_event_max_stack=16
perf record -e fib:* -a -g -- sleep 5
perf script
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 16:06 ` David Ahern
@ 2018-11-08 16:25 ` Paweł Staszewski
2018-11-08 16:27 ` Paweł Staszewski
2018-11-09 10:20 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 16:25 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 08.11.2018 o 17:06, David Ahern pisze:
> On 11/8/18 6:33 AM, Paweł Staszewski wrote:
>>
>> W dniu 07.11.2018 o 22:06, David Ahern pisze:
>>> On 11/3/18 6:24 PM, Paweł Staszewski wrote:
>>>>> Does your setup have any other device types besides physical ports with
>>>>> VLANs (e.g., any macvlans or bonds)?
>>>>>
>>>>>
>>>> no.
>>>> just
>>>> phy(mlnx)->vlans only config
>>> VLAN and non-VLAN (and a mix) seem to work ok. Patches are here:
>>> https://github.com/dsahern/linux.git bpf/kernel-tables-wip
>>>
>>> I got lazy with the vlan exports; right now it requires 8021q to be
>>> builtin (CONFIG_VLAN_8021Q=y)
>>>
>>> You can use the xdp_fwd sample:
>>> make O=kbuild -C samples/bpf -j 8
>>>
>>> Copy samples/bpf/xdp_fwd_kern.o and samples/bpf/xdp_fwd to the server
>>> and run:
>>> ./xdp_fwd <list of NIC ports>
>>>
>>> e.g., in my testing I run:
>>> xdp_fwd eth1 eth2 eth3 eth4
>>>
>>> All of the relevant forwarding ports need to be on the same command
>>> line. This version populates a second map to verify the egress port has
>>> XDP enabled.
>> Installed today on some lab server with mellanox connectx4
>>
>> And trying some simple static routing first - but after enabling xdp
>> program - receiver is not receiving frames
>>
>> Route table is simple as possible for tests :)
>>
>> icmp ping test send from 192.168.22.237 to 172.16.0.2 - incomming
>> packets on vlan 4081
>>
>> ip r
>> default via 192.168.22.236 dev vlan4081
>> 172.16.0.0/30 dev vlan1740 proto kernel scope link src 172.16.0.1
>> 192.168.22.0/24 dev vlan4081 proto kernel scope link src 192.168.22.205
>>
>> neigh table:
>> ip neigh ls
>>
>> 192.168.22.237 dev vlan4081 lladdr 00:25:90:fb:a6:8d REACHABLE
>> 172.16.0.2 dev vlan1740 lladdr ac:1f:6b:2c:2e:5a REACHABLE
>>
>> and interfaces:
>> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
>> UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
>> UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>
>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5 qdisc
>> mq state UP group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
>> valid_lft forever preferred_lft forever
>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> inet 192.168.22.205/24 scope global vlan4081
>> valid_lft forever preferred_lft forever
>> inet6 fe80::ae1f:6bff:fe07:c890/64 scope link
>> valid_lft forever preferred_lft forever
>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> inet 172.16.0.1/30 scope global vlan1740
>> valid_lft forever preferred_lft forever
>> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
>> valid_lft forever preferred_lft forever
>>
>>
>> xdp program detached:
>> Receiving side tcpdump:
>> 14:28:09.141233 IP 192.168.22.237 > 172.16.0.2: ICMP echo request, id
>> 30227, seq 487, length 64
>>
>> I can see icmp requests
>>
>>
>> enabling xdp
>> ./xdp_fwd enp175s0f1 enp175s0f0
>>
>> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
>> state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> prog/xdp id 5 tag 3c231ff1e5e77f3f
>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
>> state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> prog/xdp id 5 tag 3c231ff1e5e77f3f
>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>
> What hardware is this?
>
> Start with:
>
> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
> cat /sys/kernel/debug/tracing/trace_pipe
cat /sys/kernel/debug/tracing/trace_pipe
<idle>-0 [045] ..s. 68469.467752: xdp_devmap_xmit:
ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
from_ifindex=4 to_ifindex=5 err=-6
<idle>-0 [045] ..s. 68470.483836: xdp_redirect_map:
prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
map_index=5
<idle>-0 [045] ..s. 68470.483837: xdp_devmap_xmit:
ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
from_ifindex=4 to_ifindex=5 err=-6
<idle>-0 [045] ..s. 68471.503853: xdp_redirect_map:
prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
map_index=5
<idle>-0 [045] ..s. 68471.503853: xdp_devmap_xmit:
ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
from_ifindex=4 to_ifindex=5 err=-6
<idle>-0 [045] ..s. 68472.527871: xdp_redirect_map:
prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
map_index=5
<idle>-0 [045] ..s. 68472.527877: xdp_devmap_xmit:
ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
from_ifindex=4 to_ifindex=5 err=-6
<idle>-0 [045] ..s. 68473.551876: xdp_redirect_map:
prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
map_index=5
<idle>-0 [045] ..s. 68473.551880: xdp_devmap_xmit:
ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
from_ifindex=4 to_ifindex=5 err=-6
<idle>-0 [045] ..s. 68474.575893: xdp_redirect_map:
prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
map_index=5
<idle>-0 [045] ..s. 68474.575897: xdp_devmap_xmit:
ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
from_ifindex=4 to_ifindex=5 err=-6
<idle>-0 [045] ..s. 68475.599909: xdp_redirect_map:
prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
map_index=5
<idle>-0 [045] ..s. 68475.599912: xdp_devmap_xmit:
ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
from_ifindex=4 to_ifindex=5 err=-6
>
> >From there, you can check the FIB lookups:
> sysctl -w kernel.perf_event_max_stack=16
> perf record -e fib:* -a -g -- sleep 5
> perf script
>
swapper 0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif 0
iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 ==>
dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif 0
iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 ==>
dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif 0
iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 ==>
dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif 0
iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 ==>
dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
swapper 0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif 0
iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 ==>
dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 16:25 ` Paweł Staszewski
@ 2018-11-08 16:27 ` Paweł Staszewski
2018-11-08 16:32 ` David Ahern
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 16:27 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 08.11.2018 o 17:25, Paweł Staszewski pisze:
>
>
> W dniu 08.11.2018 o 17:06, David Ahern pisze:
>> On 11/8/18 6:33 AM, Paweł Staszewski wrote:
>>>
>>> W dniu 07.11.2018 o 22:06, David Ahern pisze:
>>>> On 11/3/18 6:24 PM, Paweł Staszewski wrote:
>>>>>> Does your setup have any other device types besides physical
>>>>>> ports with
>>>>>> VLANs (e.g., any macvlans or bonds)?
>>>>>>
>>>>>>
>>>>> no.
>>>>> just
>>>>> phy(mlnx)->vlans only config
>>>> VLAN and non-VLAN (and a mix) seem to work ok. Patches are here:
>>>> https://github.com/dsahern/linux.git bpf/kernel-tables-wip
>>>>
>>>> I got lazy with the vlan exports; right now it requires 8021q to be
>>>> builtin (CONFIG_VLAN_8021Q=y)
>>>>
>>>> You can use the xdp_fwd sample:
>>>> make O=kbuild -C samples/bpf -j 8
>>>>
>>>> Copy samples/bpf/xdp_fwd_kern.o and samples/bpf/xdp_fwd to the server
>>>> and run:
>>>> ./xdp_fwd <list of NIC ports>
>>>>
>>>> e.g., in my testing I run:
>>>> xdp_fwd eth1 eth2 eth3 eth4
>>>>
>>>> All of the relevant forwarding ports need to be on the same command
>>>> line. This version populates a second map to verify the egress port
>>>> has
>>>> XDP enabled.
>>> Installed today on some lab server with mellanox connectx4
>>>
>>> And trying some simple static routing first - but after enabling xdp
>>> program - receiver is not receiving frames
>>>
>>> Route table is simple as possible for tests :)
>>>
>>> icmp ping test send from 192.168.22.237 to 172.16.0.2 - incomming
>>> packets on vlan 4081
>>>
>>> ip r
>>> default via 192.168.22.236 dev vlan4081
>>> 172.16.0.0/30 dev vlan1740 proto kernel scope link src 172.16.0.1
>>> 192.168.22.0/24 dev vlan4081 proto kernel scope link src 192.168.22.205
>>>
>>> neigh table:
>>> ip neigh ls
>>>
>>> 192.168.22.237 dev vlan4081 lladdr 00:25:90:fb:a6:8d REACHABLE
>>> 172.16.0.2 dev vlan1740 lladdr ac:1f:6b:2c:2e:5a REACHABLE
>>>
>>> and interfaces:
>>> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>>> state
>>> UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>>> state
>>> UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>>> qdisc
>>> noqueue state UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>>> qdisc
>>> noqueue state UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>>
>>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5
>>> qdisc
>>> mq state UP group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
>>> valid_lft forever preferred_lft forever
>>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>>> qdisc
>>> noqueue state UP group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>>> inet 192.168.22.205/24 scope global vlan4081
>>> valid_lft forever preferred_lft forever
>>> inet6 fe80::ae1f:6bff:fe07:c890/64 scope link
>>> valid_lft forever preferred_lft forever
>>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>>> qdisc
>>> noqueue state UP group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>> inet 172.16.0.1/30 scope global vlan1740
>>> valid_lft forever preferred_lft forever
>>> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
>>> valid_lft forever preferred_lft forever
>>>
>>>
>>> xdp program detached:
>>> Receiving side tcpdump:
>>> 14:28:09.141233 IP 192.168.22.237 > 172.16.0.2: ICMP echo request, id
>>> 30227, seq 487, length 64
>>>
>>> I can see icmp requests
>>>
>>>
>>> enabling xdp
>>> ./xdp_fwd enp175s0f1 enp175s0f0
>>>
>>> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
>>> state UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>>> prog/xdp id 5 tag 3c231ff1e5e77f3f
>>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
>>> state UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>> prog/xdp id 5 tag 3c231ff1e5e77f3f
>>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>>> qdisc
>>> noqueue state UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>>> qdisc
>>> noqueue state UP mode DEFAULT group default qlen 1000
>>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>>
>> What hardware is this?
>>
mellanox connectx 4
ethtool -i enp175s0f0
driver: mlx5_core
version: 5.0-0
firmware-version: 12.21.1000 (SM_2001000001033)
expansion-rom-version:
bus-info: 0000:af:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
ethtool -i enp175s0f1
driver: mlx5_core
version: 5.0-0
firmware-version: 12.21.1000 (SM_2001000001033)
expansion-rom-version:
bus-info: 0000:af:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
>> Start with:
>>
>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
>> cat /sys/kernel/debug/tracing/trace_pipe
> cat /sys/kernel/debug/tracing/trace_pipe
> <idle>-0 [045] ..s. 68469.467752: xdp_devmap_xmit:
> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
> from_ifindex=4 to_ifindex=5 err=-6
> <idle>-0 [045] ..s. 68470.483836: xdp_redirect_map:
> prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
> map_index=5
> <idle>-0 [045] ..s. 68470.483837: xdp_devmap_xmit:
> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
> from_ifindex=4 to_ifindex=5 err=-6
> <idle>-0 [045] ..s. 68471.503853: xdp_redirect_map:
> prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
> map_index=5
> <idle>-0 [045] ..s. 68471.503853: xdp_devmap_xmit:
> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
> from_ifindex=4 to_ifindex=5 err=-6
> <idle>-0 [045] ..s. 68472.527871: xdp_redirect_map:
> prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
> map_index=5
> <idle>-0 [045] ..s. 68472.527877: xdp_devmap_xmit:
> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
> from_ifindex=4 to_ifindex=5 err=-6
> <idle>-0 [045] ..s. 68473.551876: xdp_redirect_map:
> prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
> map_index=5
> <idle>-0 [045] ..s. 68473.551880: xdp_devmap_xmit:
> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
> from_ifindex=4 to_ifindex=5 err=-6
> <idle>-0 [045] ..s. 68474.575893: xdp_redirect_map:
> prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
> map_index=5
> <idle>-0 [045] ..s. 68474.575897: xdp_devmap_xmit:
> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
> from_ifindex=4 to_ifindex=5 err=-6
> <idle>-0 [045] ..s. 68475.599909: xdp_redirect_map:
> prog_id=30 action=REDIRECT ifindex=4 to_ifindex=5 err=0 map_id=32
> map_index=5
> <idle>-0 [045] ..s. 68475.599912: xdp_devmap_xmit:
> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
> from_ifindex=4 to_ifindex=5 err=-6
>
>
>
>>
>> >From there, you can check the FIB lookups:
>> sysctl -w kernel.perf_event_max_stack=16
>> perf record -e fib:* -a -g -- sleep 5
>> perf script
>>
> swapper 0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif
> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>
> swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif
> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>
> swapper 0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif
> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>
> swapper 0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif
> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>
> swapper 0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif
> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 16:27 ` Paweł Staszewski
@ 2018-11-08 16:32 ` David Ahern
2018-11-08 17:30 ` Paweł Staszewski
2018-11-09 0:40 ` Paweł Staszewski
0 siblings, 2 replies; 77+ messages in thread
From: David Ahern @ 2018-11-08 16:32 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/8/18 9:27 AM, Paweł Staszewski wrote:
>>> What hardware is this?
>>>
> mellanox connectx 4
> ethtool -i enp175s0f0
> driver: mlx5_core
> version: 5.0-0
> firmware-version: 12.21.1000 (SM_2001000001033)
> expansion-rom-version:
> bus-info: 0000:af:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: yes
>
> ethtool -i enp175s0f1
> driver: mlx5_core
> version: 5.0-0
> firmware-version: 12.21.1000 (SM_2001000001033)
> expansion-rom-version:
> bus-info: 0000:af:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: yes
>
>>> Start with:
>>>
>>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
>>> cat /sys/kernel/debug/tracing/trace_pipe
>> cat /sys/kernel/debug/tracing/trace_pipe
>> <idle>-0 [045] ..s. 68469.467752: xdp_devmap_xmit:
>> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
>> from_ifindex=4 to_ifindex=5 err=-6
FIB lookup is good, the redirect is happening, but the mlx5 driver does
not like it.
I think the -6 is coming from the mlx5 driver and the packet is getting
dropped. Perhaps this check in mlx5e_xdp_xmit:
if (unlikely(sq_num >= priv->channels.num))
return -ENXIO;
>> swapper 0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper 0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper 0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>
>> swapper 0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif
>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 16:32 ` David Ahern
@ 2018-11-08 17:30 ` Paweł Staszewski
2018-11-08 18:05 ` David Ahern
2018-11-09 0:40 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 17:30 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 08.11.2018 o 17:32, David Ahern pisze:
> On 11/8/18 9:27 AM, Paweł Staszewski wrote:
>>>> What hardware is this?
>>>>
>> mellanox connectx 4
>> ethtool -i enp175s0f0
>> driver: mlx5_core
>> version: 5.0-0
>> firmware-version: 12.21.1000 (SM_2001000001033)
>> expansion-rom-version:
>> bus-info: 0000:af:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: no
>> supports-register-dump: no
>> supports-priv-flags: yes
>>
>> ethtool -i enp175s0f1
>> driver: mlx5_core
>> version: 5.0-0
>> firmware-version: 12.21.1000 (SM_2001000001033)
>> expansion-rom-version:
>> bus-info: 0000:af:00.1
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: no
>> supports-register-dump: no
>> supports-priv-flags: yes
>>
>>>> Start with:
>>>>
>>>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
>>>> cat /sys/kernel/debug/tracing/trace_pipe
>>> cat /sys/kernel/debug/tracing/trace_pipe
>>> <idle>-0 [045] ..s. 68469.467752: xdp_devmap_xmit:
>>> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
>>> from_ifindex=4 to_ifindex=5 err=-6
> FIB lookup is good, the redirect is happening, but the mlx5 driver does
> not like it.
>
> I think the -6 is coming from the mlx5 driver and the packet is getting
> dropped. Perhaps this check in mlx5e_xdp_xmit:
>
> if (unlikely(sq_num >= priv->channels.num))
> return -ENXIO;
>
Wondering about this:
swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif 0
iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 ==>
dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
oif 0 ?
Is that correct here ?
>
>>> swapper 0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 17:30 ` Paweł Staszewski
@ 2018-11-08 18:05 ` David Ahern
0 siblings, 0 replies; 77+ messages in thread
From: David Ahern @ 2018-11-08 18:05 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/8/18 10:30 AM, Paweł Staszewski wrote:
> Wondering about this:
> swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif 0
> iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0 ==>
> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>
> oif 0 ?
>
> Is that correct here ?
ingress path so iif is set to the vlan device and oif is 0.
egress lookups (e.g., locally generated traffic) have oif non-0.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-03 0:18 ` Paweł Staszewski
@ 2018-11-08 19:12 ` Paweł Staszewski
2018-11-09 22:20 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-08 19:12 UTC (permalink / raw)
To: Saeed Mahameed, netdev
W dniu 03.11.2018 o 01:18, Paweł Staszewski pisze:
>
>
> W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
>> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>>> Hi
>>>>>
>>>>> So maybee someone will be interested how linux kernel handles
>>>>> normal
>>>>> traffic (not pktgen :) )
>>>>>
>>>>>
>>>>> Server HW configuration:
>>>>>
>>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>>
>>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>>
>>>>>
>>>>> Server software:
>>>>>
>>>>> FRR - as routing daemon
>>>>>
>>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>>> local
>>>>> numa
>>>>> node)
>>>>>
>>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>>> numa
>>>>> node)
>>>>>
>>>>>
>>>>> Maximum traffic that server can handle:
>>>>>
>>>>> Bandwidth
>>>>>
>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>> input: /proc/net/dev type: rate
>>>>> \ iface Rx Tx Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>>> Gb/s
>>>>> 65.74 Gb/s
>>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>>> Gb/s
>>>>> 66.51 Gb/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>> total: 66.58 Gb/s 65.67
>>>>> Gb/s
>>>>> 132.25 Gb/s
>>>>>
>>>>>
>>>>> Packets per second:
>>>>>
>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>> input: /proc/net/dev type: rate
>>>>> - iface Rx Tx Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>>> 8735207.00 P/s
>>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>>> 8790460.00 P/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>>> 17525668.00 P/s
>>>>>
>>>>>
>>>>> After reaching that limits nics on the upstream side (more RX
>>>>> traffic)
>>>>> start to drop packets
>>>>>
>>>>>
>>>>> I just dont understand that server can't handle more bandwidth
>>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>>> RX
>>>>> side are increasing.
>>>>>
>>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>>> NIC (
>>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>>> 132.25
>>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>>> hmm yes that was my concern also - cause cant find anywhere
>>> informations
>>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>>> 8GT
>>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>>> bw
>>> on both ports
>> i think it is bidir
> So yes - we are hitting there other problem i think pcie is most
> probabbly bidirectional max bw 126Gbit so RX 126Gbit and at same time
> TX should be 126Gbit
>
>
So one 2-port 100G card connectx4 replaced with two separate connectx5
placed in two different pcie x16 gen 3.0
lspci -vvv -s af:00.0
af:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
[ConnectX-5]
Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 90
NUMA node: 1
Region 0: Memory at 39bffe000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at ee600000 [disabled] [size=1M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not
supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [48] Vital Product Data
Product Name: CX515A - ConnectX-5 QSFP28
Read-only fields:
[PN] Part number: MCX515A-CCAT
[EC] Engineering changes: A6
[V2] Vendor specific: MCX515A-CCAT
[SN] Serial number: MT1831J00221
[V3] Vendor specific:
14a5c73bee92e811800098039b1ee5f0
[VA] Vendor specific:
MLX:MODL=CX515A:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0
[V0] Vendor specific: PCIeGen3 x16
[RV] Reserved: checksum good, 2 byte(s) reserved
End
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
PME(D0-,D1-,D2-,D3hot-,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 04, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 0, Total VFs: 0, Number of VFs: 0,
Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 1018
Supported Page Size: 000007ff, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1c0 v1] #19
Kernel driver in use: mlx5_core
d8:00.0 Ethernet controller: Mellanox Technologies MT27800 Family
[ConnectX-5]
Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 159
NUMA node: 1
Region 0: Memory at 39fffe000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fbe00000 [disabled] [size=1M]
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not
supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Capabilities: [48] Vital Product Data
Product Name: CX515A - ConnectX-5 QSFP28
Read-only fields:
[PN] Part number: MCX515A-CCAT
[EC] Engineering changes: A6
[V2] Vendor specific: MCX515A-CCAT
[SN] Serial number: MT1831J00169
[V3] Vendor specific:
c06757e6e092e811800098039b1ee520
[VA] Vendor specific:
MLX:MODL=CX515A:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0
[V0] Vendor specific: PCIeGen3 x16
[RV] Reserved: checksum good, 2 byte(s) reserved
End
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
PME(D0-,D1-,D2-,D3hot-,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 04, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 0, Total VFs: 0, Number of VFs: 0,
Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 1018
Supported Page Size: 000007ff, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1c0 v1] #19
Kernel driver in use: mlx5_core
CPU load is lower than for connectx4 - but it looks like bandwidth limit
is the same :)
But also after reaching 60Gbit/60Gbit
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
- iface Rx Tx Total
==============================================================================
enp175s0: 45.09 Gb/s 15.09 Gb/s
60.18 Gb/s
enp216s0: 15.14 Gb/s 45.19 Gb/s
60.33 Gb/s
------------------------------------------------------------------------------
total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
Nics start to drop packets (discards from nic's where is more rx traffic):
ethtool -S enp175s0 |grep 'disc'
rx_discards_phy: 47265611
after 20 secs
ethtool -S enp175s0 |grep 'disc'
rx_discards_phy: 49434472
current coalescence params:
ethtool -c enp175s0
Coalesce parameters for enp175s0:
Adaptive RX: off TX: on
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32651
rx-usecs: 128
rx-frames: 128
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 8
tx-frames: 128
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
and perf top:
PerfTop: 86898 irqs/sec kernel:99.5% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
12.76% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
8.68% [kernel] [k] mlx5e_sq_xmit
6.47% [kernel] [k] build_skb
4.78% [kernel] [k] fib_table_lookup
4.58% [kernel] [k] memcpy_erms
3.47% [kernel] [k] mlx5e_poll_rx_cq
2.59% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
2.37% [kernel] [k] mlx5e_post_rx_mpwqes
2.33% [kernel] [k] vlan_do_receive
1.94% [kernel] [k] __dev_queue_xmit
1.89% [kernel] [k] mlx5e_poll_tx_cq
1.74% [kernel] [k] ip_finish_output2
1.67% [kernel] [k] dev_gro_receive
1.64% [kernel] [k] ipt_do_table
1.58% [kernel] [k] tcp_gro_receive
1.49% [kernel] [k] pfifo_fast_dequeue
1.28% [kernel] [k] mlx5_eq_int
1.26% [kernel] [k] inet_gro_receive
1.26% [kernel] [k] _raw_spin_lock
1.20% [kernel] [k] __netif_receive_skb_core
1.19% [kernel] [k] irq_entries_start
1.17% [kernel] [k] swiotlb_map_page
1.13% [kernel] [k] vlan_dev_hard_start_xmit
1.12% [kernel] [k] ip_route_input_rcu
0.97% [kernel] [k] __build_skb
0.84% [kernel] [k] _raw_spin_lock_irqsave
0.78% [kernel] [k] kmem_cache_alloc
0.77% [kernel] [k] mlx5e_xmit
0.77% [kernel] [k] dev_hard_start_xmit
0.76% [kernel] [k] ip_forward
0.73% [kernel] [k] netif_skb_features
0.70% [kernel] [k] tasklet_action_common.isra.21
0.58% [kernel] [k] validate_xmit_skb.isra.142
0.55% [kernel] [k] ip_rcv_core.isra.20.constprop.25
0.55% [kernel] [k] mlx5e_page_release
0.55% [kernel] [k] __qdisc_run
0.51% [kernel] [k] __memcpy
0.48% [kernel] [k] kmem_cache_free_bulk
0.48% [kernel] [k] page_frag_free
0.47% [kernel] [k] inet_lookup_ifaddr_rcu
0.47% [kernel] [k] queued_spin_lock_slowpath
0.46% [kernel] [k] pfifo_fast_enqueue
0.43% [kernel] [k] tcp4_gro_receive
0.40% [kernel] [k] skb_gro_receive
0.39% [kernel] [k] skb_release_data
0.38% [kernel] [k] find_busiest_group
0.36% [kernel] [k] _raw_spin_trylock
0.36% [kernel] [k] skb_segment
0.33% [kernel] [k] eth_type_trans
0.32% [kernel] [k] __sched_text_start
0.32% [kernel] [k] __netif_schedule
0.32% [kernel] [k] try_to_wake_up
0.31% [kernel] [k] _raw_spin_lock_irq
0.31% [kernel] [k] __local_bh_enable_ip
Also mpstat:
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.06 0.00 1.00 0.02 0.00 21.61 0.00
0.00 0.00 77.32
Average: 0 0.00 0.00 0.60 0.00 0.00 0.00 0.00
0.00 0.00 99.40
Average: 1 0.10 0.00 1.30 0.00 0.00 0.00 0.00
0.00 0.00 98.60
Average: 2 0.00 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.80
Average: 3 0.00 0.00 1.60 0.00 0.00 0.00 0.00
0.00 0.00 98.40
Average: 4 0.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 5 0.20 0.00 4.60 0.00 0.00 0.00 0.00
0.00 0.00 95.20
Average: 6 0.00 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.80
Average: 7 0.60 0.00 3.00 0.00 0.00 0.00 0.00
0.00 0.00 96.40
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.70 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.00 0.00 2.00 0.00 0.00 0.00 0.00
0.00 0.00 98.00
Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 14 0.00 0.00 1.00 0.00 0.00 50.40 0.00
0.00 0.00 48.60
Average: 15 0.00 0.00 1.30 0.00 0.00 47.90 0.00
0.00 0.00 50.80
Average: 16 0.00 0.00 2.00 0.00 0.00 47.80 0.00
0.00 0.00 50.20
Average: 17 0.00 0.00 1.30 0.00 0.00 50.20 0.00
0.00 0.00 48.50
Average: 18 0.10 0.00 1.10 0.00 0.00 42.40 0.00
0.00 0.00 56.40
Average: 19 0.00 0.00 1.50 0.00 0.00 44.40 0.00
0.00 0.00 54.10
Average: 20 0.00 0.00 1.40 0.00 0.00 45.90 0.00
0.00 0.00 52.70
Average: 21 0.00 0.00 0.70 0.00 0.00 44.50 0.00
0.00 0.00 54.80
Average: 22 0.10 0.00 1.40 0.00 0.00 47.00 0.00
0.00 0.00 51.50
Average: 23 0.00 0.00 0.30 0.00 0.00 45.50 0.00
0.00 0.00 54.20
Average: 24 0.00 0.00 1.60 0.00 0.00 50.00 0.00
0.00 0.00 48.40
Average: 25 0.10 0.00 0.70 0.00 0.00 47.00 0.00
0.00 0.00 52.20
Average: 26 0.00 0.00 1.80 0.00 0.00 48.70 0.00
0.00 0.00 49.50
Average: 27 0.00 0.00 1.10 0.00 0.00 44.80 0.00
0.00 0.00 54.10
Average: 28 0.30 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.70
Average: 29 0.10 0.00 0.60 0.00 0.00 0.00 0.00
0.00 0.00 99.30
Average: 30 0.00 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.80
Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 32 0.00 0.00 1.20 0.00 0.00 0.00 0.00
0.00 0.00 98.80
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 38 0.20 0.00 0.80 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 39 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 40 0.00 0.00 3.30 0.00 0.00 0.00 0.00
0.00 0.00 96.70
Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 42 0.00 0.00 0.80 0.00 0.00 45.00 0.00
0.00 0.00 54.20
Average: 43 0.00 0.00 1.60 0.00 0.00 48.30 0.00
0.00 0.00 50.10
Average: 44 0.00 0.00 1.60 0.00 0.00 37.90 0.00
0.00 0.00 60.50
Average: 45 0.30 0.00 1.40 0.00 0.00 32.90 0.00
0.00 0.00 65.40
Average: 46 0.00 0.00 1.50 0.90 0.00 37.60 0.00
0.00 0.00 60.00
Average: 47 0.10 0.00 0.40 0.00 0.00 41.40 0.00
0.00 0.00 58.10
Average: 48 0.20 0.00 1.70 0.00 0.00 38.20 0.00
0.00 0.00 59.90
Average: 49 0.00 0.00 1.40 0.00 0.00 37.20 0.00
0.00 0.00 61.40
Average: 50 0.00 0.00 1.30 0.00 0.00 38.10 0.00
0.00 0.00 60.60
Average: 51 0.00 0.00 0.80 0.00 0.00 39.40 0.00
0.00 0.00 59.80
Average: 52 0.00 0.00 1.70 0.00 0.00 39.50 0.00
0.00 0.00 58.80
Average: 53 0.10 0.00 0.90 0.00 0.00 38.20 0.00
0.00 0.00 60.80
Average: 54 0.00 0.00 1.30 0.00 0.00 42.10 0.00
0.00 0.00 56.60
Average: 55 0.00 0.00 1.60 0.00 0.00 37.70 0.00
0.00 0.00 60.70
So it looks like previously there was also no problem with pciexpress x16
>
>
>
>>> This can explain maybee also why cpuload is rising rapidly from
>>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>>> so
>>> there can be some error in reading them when offloading (gro/gso/tso)
>>> on
>>> nic's is enabled that is why
>>>
>>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>>> 8GT
>>>>> is
>>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>>> and
>>>>> pps
>>>>> (like 4x more comparing to normal internet traffic)
>>>>>
>>>> Are you forwarding when using pktgen as well or you just testing
>>>> the RX
>>>> side pps ?
>>> Yes pktgen was tested on single port RX
>>> Can check also forwarding to eliminate pciex limits
>>>
>> So this explains why you have more RX pps, since tx is idle and pcie
>> will be free to do only rx.
>>
>> [...]
>>
>>
>>>>> ethtool -S enp175s0f1
>>>>> NIC statistics:
>>>>> rx_packets: 173730800927
>>>>> rx_bytes: 99827422751332
>>>>> tx_packets: 142532009512
>>>>> tx_bytes: 184633045911222
>>>>> tx_tso_packets: 25989113891
>>>>> tx_tso_bytes: 132933363384458
>>>>> tx_tso_inner_packets: 0
>>>>> tx_tso_inner_bytes: 0
>>>>> tx_added_vlan_packets: 74630239613
>>>>> tx_nop: 2029817748
>>>>> rx_lro_packets: 0
>>>>> rx_lro_bytes: 0
>>>>> rx_ecn_mark: 0
>>>>> rx_removed_vlan_packets: 173730800927
>>>>> rx_csum_unnecessary: 0
>>>>> rx_csum_none: 434357
>>>>> rx_csum_complete: 173730366570
>>>>> rx_csum_unnecessary_inner: 0
>>>>> rx_xdp_drop: 0
>>>>> rx_xdp_redirect: 0
>>>>> rx_xdp_tx_xmit: 0
>>>>> rx_xdp_tx_full: 0
>>>>> rx_xdp_tx_err: 0
>>>>> rx_xdp_tx_cqe: 0
>>>>> tx_csum_none: 38260960853
>>>>> tx_csum_partial: 36369278774
>>>>> tx_csum_partial_inner: 0
>>>>> tx_queue_stopped: 1
>>>>> tx_queue_dropped: 0
>>>>> tx_xmit_more: 748638099
>>>>> tx_recover: 0
>>>>> tx_cqes: 73881645031
>>>>> tx_queue_wake: 1
>>>>> tx_udp_seg_rem: 0
>>>>> tx_cqe_err: 0
>>>>> tx_xdp_xmit: 0
>>>>> tx_xdp_full: 0
>>>>> tx_xdp_err: 0
>>>>> tx_xdp_cqes: 0
>>>>> rx_wqe_err: 0
>>>>> rx_mpwqe_filler_cqes: 0
>>>>> rx_mpwqe_filler_strides: 0
>>>>> rx_buff_alloc_err: 0
>>>>> rx_cqe_compress_blks: 0
>>>>> rx_cqe_compress_pkts: 0
>>>> If this is a pcie bottleneck it might be useful to enable CQE
>>>> compression (to reduce PCIe completion descriptors transactions)
>>>> you should see the above rx_cqe_compress_pkts increasing when
>>>> enabled.
>>>>
>>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>>> $ ethtool --show-priv-flags enp175s0f1
>>>> Private flags for p6p1:
>>>> rx_cqe_moder : on
>>>> cqe_moder : off
>>>> rx_cqe_compress : on
>>>> ...
>>>>
>>>> try this on both interfaces.
>>> Done
>>> ethtool --show-priv-flags enp175s0f1
>>> Private flags for enp175s0f1:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : on
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>>> ethtool --show-priv-flags enp175s0f0
>>> Private flags for enp175s0f0:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : on
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>> did it help reduce the load on the pcie ? do you see more pps ?
>> what is the ratio between rx_cqe_compress_pkts and over all rx packets
>> ?
>>
>> [...]
>>
>>>>> ethtool -S enp175s0f0
>>>>> NIC statistics:
>>>>> rx_packets: 141574897253
>>>>> rx_bytes: 184445040406258
>>>>> tx_packets: 172569543894
>>>>> tx_bytes: 99486882076365
>>>>> tx_tso_packets: 9367664195
>>>>> tx_tso_bytes: 56435233992948
>>>>> tx_tso_inner_packets: 0
>>>>> tx_tso_inner_bytes: 0
>>>>> tx_added_vlan_packets: 141297671626
>>>>> tx_nop: 2102916272
>>>>> rx_lro_packets: 0
>>>>> rx_lro_bytes: 0
>>>>> rx_ecn_mark: 0
>>>>> rx_removed_vlan_packets: 141574897252
>>>>> rx_csum_unnecessary: 0
>>>>> rx_csum_none: 23135854
>>>>> rx_csum_complete: 141551761398
>>>>> rx_csum_unnecessary_inner: 0
>>>>> rx_xdp_drop: 0
>>>>> rx_xdp_redirect: 0
>>>>> rx_xdp_tx_xmit: 0
>>>>> rx_xdp_tx_full: 0
>>>>> rx_xdp_tx_err: 0
>>>>> rx_xdp_tx_cqe: 0
>>>>> tx_csum_none: 127934791664
>>>> It is a good idea to look into this, tx is not requesting hw tx
>>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>>> on
>>>> calculating csum, or maybe this is just the rx csum complete..
>>>>
>>>>> tx_csum_partial: 13362879974
>>>>> tx_csum_partial_inner: 0
>>>>> tx_queue_stopped: 232561
>>>> TX queues are stalling, could be an indentation for the pcie
>>>> bottelneck.
>>>>
>>>>> tx_queue_dropped: 0
>>>>> tx_xmit_more: 1266021946
>>>>> tx_recover: 0
>>>>> tx_cqes: 140031716469
>>>>> tx_queue_wake: 232561
>>>>> tx_udp_seg_rem: 0
>>>>> tx_cqe_err: 0
>>>>> tx_xdp_xmit: 0
>>>>> tx_xdp_full: 0
>>>>> tx_xdp_err: 0
>>>>> tx_xdp_cqes: 0
>>>>> rx_wqe_err: 0
>>>>> rx_mpwqe_filler_cqes: 0
>>>>> rx_mpwqe_filler_strides: 0
>>>>> rx_buff_alloc_err: 0
>>>>> rx_cqe_compress_blks: 0
>>>>> rx_cqe_compress_pkts: 0
>>>>> rx_page_reuse: 0
>>>>> rx_cache_reuse: 16625975793
>>>>> rx_cache_full: 54161465914
>>>>> rx_cache_empty: 258048
>>>>> rx_cache_busy: 54161472735
>>>>> rx_cache_waive: 0
>>>>> rx_congst_umr: 0
>>>>> rx_arfs_err: 0
>>>>> ch_events: 40572621887
>>>>> ch_poll: 40885650979
>>>>> ch_arm: 40429276692
>>>>> ch_aff_change: 0
>>>>> ch_eq_rearm: 0
>>>>> rx_out_of_buffer: 2791690
>>>>> rx_if_down_packets: 74
>>>>> rx_vport_unicast_packets: 141843476308
>>>>> rx_vport_unicast_bytes: 185421265403318
>>>>> tx_vport_unicast_packets: 172569484005
>>>>> tx_vport_unicast_bytes: 100019940094298
>>>>> rx_vport_multicast_packets: 85122935
>>>>> rx_vport_multicast_bytes: 5761316431
>>>>> tx_vport_multicast_packets: 6452
>>>>> tx_vport_multicast_bytes: 643540
>>>>> rx_vport_broadcast_packets: 22423624
>>>>> rx_vport_broadcast_bytes: 1390127090
>>>>> tx_vport_broadcast_packets: 22024
>>>>> tx_vport_broadcast_bytes: 1321440
>>>>> rx_vport_rdma_unicast_packets: 0
>>>>> rx_vport_rdma_unicast_bytes: 0
>>>>> tx_vport_rdma_unicast_packets: 0
>>>>> tx_vport_rdma_unicast_bytes: 0
>>>>> rx_vport_rdma_multicast_packets: 0
>>>>> rx_vport_rdma_multicast_bytes: 0
>>>>> tx_vport_rdma_multicast_packets: 0
>>>>> tx_vport_rdma_multicast_bytes: 0
>>>>> tx_packets_phy: 172569501577
>>>>> rx_packets_phy: 142871314588
>>>>> rx_crc_errors_phy: 0
>>>>> tx_bytes_phy: 100710212814151
>>>>> rx_bytes_phy: 187209224289564
>>>>> tx_multicast_phy: 6452
>>>>> tx_broadcast_phy: 22024
>>>>> rx_multicast_phy: 85122933
>>>>> rx_broadcast_phy: 22423623
>>>>> rx_in_range_len_errors_phy: 2
>>>>> rx_out_of_range_len_phy: 0
>>>>> rx_oversize_pkts_phy: 0
>>>>> rx_symbol_err_phy: 0
>>>>> tx_mac_control_phy: 0
>>>>> rx_mac_control_phy: 0
>>>>> rx_unsupported_op_phy: 0
>>>>> rx_pause_ctrl_phy: 0
>>>>> tx_pause_ctrl_phy: 0
>>>>> rx_discards_phy: 920161423
>>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>>> to
>>>> the pcie bottleneck.
>>> Yes this side is receiving more traffic - second port is +10G more tx
>>>
>> [...]
>>
>>
>>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>>> 0.00 0.00 0.00 31.30
>>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>>> 0.00 0.00 0.00 24.90
>>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>>> 0.00 0.00 0.00 19.68
>>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>>> 0.00 0.00 0.00 18.00
>>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>>> 0.00 0.00 0.00 17.40
>>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>>> 0.00 0.00 0.00 26.03
>>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>>> 0.00 0.00 0.00 17.52
>>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>>> 0.00 0.00 0.00 18.20
>>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>>> 0.00 0.00 0.00 17.68
>>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>>> 0.00 0.00 0.00 18.20
>>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>>> 0.00 0.00 0.00 20.42
>>>> The numa cores are not at 100% util, you have around 20% of idle on
>>>> each one.
>>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>>> push
>>> aditional 1-2Gbit/s
>>>
>> yes but, it doens't look like the bottleneck is the cpu, although it is
>> close to be :)..
>>
>>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 96.10
>>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.70
>>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.50
>>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 97.40
>>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.10
>>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.40
>>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>>> 0.00 0.00 0.00 19.42
>>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>>> 0.00 0.00 0.00 26.60
>>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>>> 0.00 0.00 0.00 21.60
>>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>>> 0.00 0.00 0.00 20.50
>>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>>> 0.00 0.00 0.00 21.60
>>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>>> 0.00 0.00 0.00 24.58
>>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>>> 0.00 0.00 0.00 24.68
>>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>>> 0.00 0.00 0.00 28.34
>>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>>> 0.00 0.00 0.00 25.83
>>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>>> 0.00 0.00 0.00 27.20
>>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>>> 0.00 0.00 0.00 28.03
>>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>>> 0.00 0.00 0.00 24.70
>>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>>> 0.00 0.00 0.00 25.30
>>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>>> 0.00 0.00 0.00 26.77
>>>>>
>>>>>
>>>>> ethtool -k enp175s0f0
>>>>> Features for enp175s0f0:
>>>>> rx-checksumming: on
>>>>> tx-checksumming: on
>>>>> tx-checksum-ipv4: on
>>>>> tx-checksum-ip-generic: off [fixed]
>>>>> tx-checksum-ipv6: on
>>>>> tx-checksum-fcoe-crc: off [fixed]
>>>>> tx-checksum-sctp: off [fixed]
>>>>> scatter-gather: on
>>>>> tx-scatter-gather: on
>>>>> tx-scatter-gather-fraglist: off [fixed]
>>>>> tcp-segmentation-offload: on
>>>>> tx-tcp-segmentation: on
>>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>>> tx-tcp-mangleid-segmentation: off
>>>>> tx-tcp6-segmentation: on
>>>>> udp-fragmentation-offload: off
>>>>> generic-segmentation-offload: on
>>>>> generic-receive-offload: on
>>>>> large-receive-offload: off [fixed]
>>>>> rx-vlan-offload: on
>>>>> tx-vlan-offload: on
>>>>> ntuple-filters: off
>>>>> receive-hashing: on
>>>>> highdma: on [fixed]
>>>>> rx-vlan-filter: on
>>>>> vlan-challenged: off [fixed]
>>>>> tx-lockless: off [fixed]
>>>>> netns-local: off [fixed]
>>>>> tx-gso-robust: off [fixed]
>>>>> tx-fcoe-segmentation: off [fixed]
>>>>> tx-gre-segmentation: on
>>>>> tx-gre-csum-segmentation: on
>>>>> tx-ipxip4-segmentation: off [fixed]
>>>>> tx-ipxip6-segmentation: off [fixed]
>>>>> tx-udp_tnl-segmentation: on
>>>>> tx-udp_tnl-csum-segmentation: on
>>>>> tx-gso-partial: on
>>>>> tx-sctp-segmentation: off [fixed]
>>>>> tx-esp-segmentation: off [fixed]
>>>>> tx-udp-segmentation: on
>>>>> fcoe-mtu: off [fixed]
>>>>> tx-nocache-copy: off
>>>>> loopback: off [fixed]
>>>>> rx-fcs: off
>>>>> rx-all: off
>>>>> tx-vlan-stag-hw-insert: on
>>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>>> rx-vlan-stag-filter: on [fixed]
>>>>> l2-fwd-offload: off [fixed]
>>>>> hw-tc-offload: off
>>>>> esp-hw-offload: off [fixed]
>>>>> esp-tx-csum-hw-offload: off [fixed]
>>>>> rx-udp_tunnel-port-offload: on
>>>>> tls-hw-tx-offload: off [fixed]
>>>>> tls-hw-rx-offload: off [fixed]
>>>>> rx-gro-hw: off [fixed]
>>>>> tls-hw-record: off [fixed]
>>>>>
>>>>> ethtool -c enp175s0f0
>>>>> Coalesce parameters for enp175s0f0:
>>>>> Adaptive RX: off TX: on
>>>>> stats-block-usecs: 0
>>>>> sample-interval: 0
>>>>> pkt-rate-low: 0
>>>>> pkt-rate-high: 0
>>>>> dmac: 32703
>>>>>
>>>>> rx-usecs: 256
>>>>> rx-frames: 128
>>>>> rx-usecs-irq: 0
>>>>> rx-frames-irq: 0
>>>>>
>>>>> tx-usecs: 8
>>>>> tx-frames: 128
>>>>> tx-usecs-irq: 0
>>>>> tx-frames-irq: 0
>>>>>
>>>>> rx-usecs-low: 0
>>>>> rx-frame-low: 0
>>>>> tx-usecs-low: 0
>>>>> tx-frame-low: 0
>>>>>
>>>>> rx-usecs-high: 0
>>>>> rx-frame-high: 0
>>>>> tx-usecs-high: 0
>>>>> tx-frame-high: 0
>>>>>
>>>>> ethtool -g enp175s0f0
>>>>> Ring parameters for enp175s0f0:
>>>>> Pre-set maximums:
>>>>> RX: 8192
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 8192
>>>>> Current hardware settings:
>>>>> RX: 4096
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 4096
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>> Also changed a little coalesce params - and best for this config are:
>>> ethtool -c enp175s0f0
>>> Coalesce parameters for enp175s0f0:
>>> Adaptive RX: off TX: off
>>> stats-block-usecs: 0
>>> sample-interval: 0
>>> pkt-rate-low: 0
>>> pkt-rate-high: 0
>>> dmac: 32573
>>>
>>> rx-usecs: 40
>>> rx-frames: 128
>>> rx-usecs-irq: 0
>>> rx-frames-irq: 0
>>>
>>> tx-usecs: 8
>>> tx-frames: 8
>>> tx-usecs-irq: 0
>>> tx-frames-irq: 0
>>>
>>> rx-usecs-low: 0
>>> rx-frame-low: 0
>>> tx-usecs-low: 0
>>> tx-frame-low: 0
>>>
>>> rx-usecs-high: 0
>>> rx-frame-high: 0
>>> tx-usecs-high: 0
>>> tx-frame-high: 0
>>>
>>>
>>> Less drops on RX side - and more pps in overall forwarded.
>>>
>> how much improvement ? maybe we can improve our adaptive rx coal to be
>> efficient for this work load.
>>
>>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 16:32 ` David Ahern
2018-11-08 17:30 ` Paweł Staszewski
@ 2018-11-09 0:40 ` Paweł Staszewski
2018-11-09 0:42 ` David Ahern
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-09 0:40 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 08.11.2018 o 17:32, David Ahern pisze:
> On 11/8/18 9:27 AM, Paweł Staszewski wrote:
>>>> What hardware is this?
>>>>
>> mellanox connectx 4
>> ethtool -i enp175s0f0
>> driver: mlx5_core
>> version: 5.0-0
>> firmware-version: 12.21.1000 (SM_2001000001033)
>> expansion-rom-version:
>> bus-info: 0000:af:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: no
>> supports-register-dump: no
>> supports-priv-flags: yes
>>
>> ethtool -i enp175s0f1
>> driver: mlx5_core
>> version: 5.0-0
>> firmware-version: 12.21.1000 (SM_2001000001033)
>> expansion-rom-version:
>> bus-info: 0000:af:00.1
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: no
>> supports-register-dump: no
>> supports-priv-flags: yes
>>
>>>> Start with:
>>>>
>>>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
>>>> cat /sys/kernel/debug/tracing/trace_pipe
>>> cat /sys/kernel/debug/tracing/trace_pipe
>>> <idle>-0 [045] ..s. 68469.467752: xdp_devmap_xmit:
>>> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
>>> from_ifindex=4 to_ifindex=5 err=-6
> FIB lookup is good, the redirect is happening, but the mlx5 driver does
> not like it.
>
> I think the -6 is coming from the mlx5 driver and the packet is getting
> dropped. Perhaps this check in mlx5e_xdp_xmit:
>
> if (unlikely(sq_num >= priv->channels.num))
> return -ENXIO;
I removed that part and recompiled - but after running now xdp_fwd i
have kernel pamic :)
>
>
>>> swapper 0 [045] 68493.746274: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68494.770287: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68495.794304: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68496.818308: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
>>>
>>> swapper 0 [045] 68497.842313: fib:fib_table_lookup: table 254 oif
>>> 0 iif 6 proto 1 192.168.22.237/0 -> 172.16.0.2/0 tos 0 scope 0 flags 0
>>> ==> dev vlan1740 gw 0.0.0.0 src 172.16.0.1 err 0
>>> 7fff818c13b5 fib_table_lookup ([kernel.kallsyms])
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 0:40 ` Paweł Staszewski
@ 2018-11-09 0:42 ` David Ahern
2018-11-09 4:52 ` Saeed Mahameed
0 siblings, 1 reply; 77+ messages in thread
From: David Ahern @ 2018-11-09 0:42 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/8/18 5:40 PM, Paweł Staszewski wrote:
>
>
> W dniu 08.11.2018 o 17:32, David Ahern pisze:
>> On 11/8/18 9:27 AM, Paweł Staszewski wrote:
>>>>> What hardware is this?
>>>>>
>>> mellanox connectx 4
>>> ethtool -i enp175s0f0
>>> driver: mlx5_core
>>> version: 5.0-0
>>> firmware-version: 12.21.1000 (SM_2001000001033)
>>> expansion-rom-version:
>>> bus-info: 0000:af:00.0
>>> supports-statistics: yes
>>> supports-test: yes
>>> supports-eeprom-access: no
>>> supports-register-dump: no
>>> supports-priv-flags: yes
>>>
>>> ethtool -i enp175s0f1
>>> driver: mlx5_core
>>> version: 5.0-0
>>> firmware-version: 12.21.1000 (SM_2001000001033)
>>> expansion-rom-version:
>>> bus-info: 0000:af:00.1
>>> supports-statistics: yes
>>> supports-test: yes
>>> supports-eeprom-access: no
>>> supports-register-dump: no
>>> supports-priv-flags: yes
>>>
>>>>> Start with:
>>>>>
>>>>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
>>>>> cat /sys/kernel/debug/tracing/trace_pipe
>>>> cat /sys/kernel/debug/tracing/trace_pipe
>>>> <idle>-0 [045] ..s. 68469.467752: xdp_devmap_xmit:
>>>> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0 drops=1
>>>> from_ifindex=4 to_ifindex=5 err=-6
>> FIB lookup is good, the redirect is happening, but the mlx5 driver does
>> not like it.
>>
>> I think the -6 is coming from the mlx5 driver and the packet is getting
>> dropped. Perhaps this check in mlx5e_xdp_xmit:
>>
>> if (unlikely(sq_num >= priv->channels.num))
>> return -ENXIO;
> I removed that part and recompiled - but after running now xdp_fwd i
> have kernel pamic :)
Jesper or one of the Mellanox folks needs to respond about the config
needed to run XDP with this NIC. I don't have a 40G or 100G card to play
with.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 0:42 ` David Ahern
@ 2018-11-09 4:52 ` Saeed Mahameed
2018-11-09 7:52 ` Jesper Dangaard Brouer
2018-11-09 9:56 ` Paweł Staszewski
0 siblings, 2 replies; 77+ messages in thread
From: Saeed Mahameed @ 2018-11-09 4:52 UTC (permalink / raw)
To: dsahern, pstaszewski, brouer; +Cc: netdev, yoel
On Thu, 2018-11-08 at 17:42 -0700, David Ahern wrote:
> On 11/8/18 5:40 PM, Paweł Staszewski wrote:
> >
> > W dniu 08.11.2018 o 17:32, David Ahern pisze:
> > > On 11/8/18 9:27 AM, Paweł Staszewski wrote:
> > > > > > What hardware is this?
> > > > > >
> > > > mellanox connectx 4
> > > > ethtool -i enp175s0f0
> > > > driver: mlx5_core
> > > > version: 5.0-0
> > > > firmware-version: 12.21.1000 (SM_2001000001033)
> > > > expansion-rom-version:
> > > > bus-info: 0000:af:00.0
> > > > supports-statistics: yes
> > > > supports-test: yes
> > > > supports-eeprom-access: no
> > > > supports-register-dump: no
> > > > supports-priv-flags: yes
> > > >
> > > > ethtool -i enp175s0f1
> > > > driver: mlx5_core
> > > > version: 5.0-0
> > > > firmware-version: 12.21.1000 (SM_2001000001033)
> > > > expansion-rom-version:
> > > > bus-info: 0000:af:00.1
> > > > supports-statistics: yes
> > > > supports-test: yes
> > > > supports-eeprom-access: no
> > > > supports-register-dump: no
> > > > supports-priv-flags: yes
> > > >
> > > > > > Start with:
> > > > > >
> > > > > > echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
> > > > > > cat /sys/kernel/debug/tracing/trace_pipe
> > > > > cat /sys/kernel/debug/tracing/trace_pipe
> > > > > <idle>-0 [045] ..s. 68469.467752:
> > > > > xdp_devmap_xmit:
> > > > > ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0
> > > > > drops=1
> > > > > from_ifindex=4 to_ifindex=5 err=-6
> > > FIB lookup is good, the redirect is happening, but the mlx5
> > > driver does
> > > not like it.
> > >
> > > I think the -6 is coming from the mlx5 driver and the packet is
> > > getting
> > > dropped. Perhaps this check in mlx5e_xdp_xmit:
> > >
> > > if (unlikely(sq_num >= priv->channels.num))
> > > return -ENXIO;
> > I removed that part and recompiled - but after running now xdp_fwd
> > i
> > have kernel pamic :)
>
hh, no please don't do such thing :)
It must be because the tx netdev has less tx queues than the rx netdev.
or the rx netdev rings are bound to a high cpu indexes.
anyway, best practice is to open #cores RX/TX netdev on both sides
ethtool -L enp175s0f0 combined $(nproc)
ethtool -L enp175s0f1 combined $(nproc)
> Jesper or one of the Mellanox folks needs to respond about the config
> needed to run XDP with this NIC. I don't have a 40G or 100G card to
> play
> with.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 4:52 ` Saeed Mahameed
@ 2018-11-09 7:52 ` Jesper Dangaard Brouer
2018-11-09 9:56 ` Paweł Staszewski
1 sibling, 0 replies; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-09 7:52 UTC (permalink / raw)
To: Saeed Mahameed
Cc: dsahern, pstaszewski, netdev, yoel, brouer, John Fastabend,
Tariq Toukan, Toke Høiland-Jørgensen
On Fri, 9 Nov 2018 04:52:01 +0000
Saeed Mahameed <saeedm@mellanox.com> wrote:
> On Thu, 2018-11-08 at 17:42 -0700, David Ahern wrote:
> > On 11/8/18 5:40 PM, Paweł Staszewski wrote:
> > >
> > > W dniu 08.11.2018 o 17:32, David Ahern pisze:
> > > > On 11/8/18 9:27 AM, Paweł Staszewski wrote:
> > > > > > > What hardware is this?
> > > > > > >
> > > > > mellanox connectx 4
> > > > > ethtool -i enp175s0f0
> > > > > driver: mlx5_core
> > > > > version: 5.0-0
> > > > > firmware-version: 12.21.1000 (SM_2001000001033)
> > > > > expansion-rom-version:
> > > > > bus-info: 0000:af:00.0
> > > > > supports-statistics: yes
> > > > > supports-test: yes
> > > > > supports-eeprom-access: no
> > > > > supports-register-dump: no
> > > > > supports-priv-flags: yes
> > > > >
> > > > > ethtool -i enp175s0f1
> > > > > driver: mlx5_core
> > > > > version: 5.0-0
> > > > > firmware-version: 12.21.1000 (SM_2001000001033)
> > > > > expansion-rom-version:
> > > > > bus-info: 0000:af:00.1
> > > > > supports-statistics: yes
> > > > > supports-test: yes
> > > > > supports-eeprom-access: no
> > > > > supports-register-dump: no
> > > > > supports-priv-flags: yes
> > > > >
> > > > > > > Start with:
> > > > > > >
> > > > > > > echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
> > > > > > > cat /sys/kernel/debug/tracing/trace_pipe
> > > > > > cat /sys/kernel/debug/tracing/trace_pipe
> > > > > > <idle>-0 [045] ..s. 68469.467752:
> > > > > > xdp_devmap_xmit:
> > > > > > ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0
> > > > > > drops=1
> > > > > > from_ifindex=4 to_ifindex=5 err=-6
> > > > FIB lookup is good, the redirect is happening, but the mlx5
> > > > driver does
> > > > not like it.
> > > >
> > > > I think the -6 is coming from the mlx5 driver and the packet is
> > > > getting
> > > > dropped. Perhaps this check in mlx5e_xdp_xmit:
> > > >
> > > > if (unlikely(sq_num >= priv->channels.num))
> > > > return -ENXIO;
> > > I removed that part and recompiled - but after running now xdp_fwd
> > > i
> > > have kernel pamic :)
> >
>
> hh, no please don't do such thing :)
>
> It must be because the tx netdev has less tx queues than the rx netdev.
> or the rx netdev rings are bound to a high cpu indexes.
>
> anyway, best practice is to open #cores RX/TX netdev on both sides
>
> ethtool -L enp175s0f0 combined $(nproc)
> ethtool -L enp175s0f1 combined $(nproc)
>
> > Jesper or one of the Mellanox folks needs to respond about the config
> > needed to run XDP with this NIC. I don't have a 40G or 100G card to
> > play with.
Saeed already answered with a solution... you need to increase the
number of RX/TX queues to be equal to the number of CPUs.
IHMO this again shows that the resource allocations around ndo_xdp_xmit
needs a better API. The implicit requirement is that once ndo_xdp_xmit
is enabled the driver MUST allocate for each CPU a dedicated TX for
XDP. It seems for mlx5 that this is a manual process. And as Pawel
discovered it is hard to troubleshoot and only via tracepoints.
I think we need to do better in this area, both regarding usability and
more graceful handling when the HW doesn't have the resources. The
original requirement for a XDP-TX queue per CPU was necessary because
the ndo_xdp_xmit was only sending 1-packet at the time. After my
recent changes, the ndo_xdp_xmit can now send in bulks. Thus,
performance wise it is feasible to use an (array of) locks, if e.g. the
HW cannot allocated more TX-HW queues, or e.g. allow sysadm to set the
mode of operation (if the system as a hole have issues allocating TX
completion IRQs for all these queues).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 4:52 ` Saeed Mahameed
2018-11-09 7:52 ` Jesper Dangaard Brouer
@ 2018-11-09 9:56 ` Paweł Staszewski
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-09 9:56 UTC (permalink / raw)
To: Saeed Mahameed, dsahern, brouer; +Cc: netdev, yoel
W dniu 09.11.2018 o 05:52, Saeed Mahameed pisze:
> On Thu, 2018-11-08 at 17:42 -0700, David Ahern wrote:
>> On 11/8/18 5:40 PM, Paweł Staszewski wrote:
>>> W dniu 08.11.2018 o 17:32, David Ahern pisze:
>>>> On 11/8/18 9:27 AM, Paweł Staszewski wrote:
>>>>>>> What hardware is this?
>>>>>>>
>>>>> mellanox connectx 4
>>>>> ethtool -i enp175s0f0
>>>>> driver: mlx5_core
>>>>> version: 5.0-0
>>>>> firmware-version: 12.21.1000 (SM_2001000001033)
>>>>> expansion-rom-version:
>>>>> bus-info: 0000:af:00.0
>>>>> supports-statistics: yes
>>>>> supports-test: yes
>>>>> supports-eeprom-access: no
>>>>> supports-register-dump: no
>>>>> supports-priv-flags: yes
>>>>>
>>>>> ethtool -i enp175s0f1
>>>>> driver: mlx5_core
>>>>> version: 5.0-0
>>>>> firmware-version: 12.21.1000 (SM_2001000001033)
>>>>> expansion-rom-version:
>>>>> bus-info: 0000:af:00.1
>>>>> supports-statistics: yes
>>>>> supports-test: yes
>>>>> supports-eeprom-access: no
>>>>> supports-register-dump: no
>>>>> supports-priv-flags: yes
>>>>>
>>>>>>> Start with:
>>>>>>>
>>>>>>> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
>>>>>>> cat /sys/kernel/debug/tracing/trace_pipe
>>>>>> cat /sys/kernel/debug/tracing/trace_pipe
>>>>>> <idle>-0 [045] ..s. 68469.467752:
>>>>>> xdp_devmap_xmit:
>>>>>> ndo_xdp_xmit map_id=32 map_index=5 action=REDIRECT sent=0
>>>>>> drops=1
>>>>>> from_ifindex=4 to_ifindex=5 err=-6
>>>> FIB lookup is good, the redirect is happening, but the mlx5
>>>> driver does
>>>> not like it.
>>>>
>>>> I think the -6 is coming from the mlx5 driver and the packet is
>>>> getting
>>>> dropped. Perhaps this check in mlx5e_xdp_xmit:
>>>>
>>>> if (unlikely(sq_num >= priv->channels.num))
>>>> return -ENXIO;
>>> I removed that part and recompiled - but after running now xdp_fwd
>>> i
>>> have kernel pamic :)
> hh, no please don't do such thing :)
yes - dirty "try" :)
Code back in place :)
>
> It must be because the tx netdev has less tx queues than the rx netdev.
> or the rx netdev rings are bound to a high cpu indexes.
>
> anyway, best practice is to open #cores RX/TX netdev on both sides
>
> ethtool -L enp175s0f0 combined $(nproc)
> ethtool -L enp175s0f1 combined $(nproc)
Ok now it is working.
Time for some tests :)
Thanks
>> Jesper or one of the Mellanox folks needs to respond about the config
>> needed to run XDP with this NIC. I don't have a 40G or 100G card to
>> play
>> with.
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 16:06 ` David Ahern
2018-11-08 16:25 ` Paweł Staszewski
@ 2018-11-09 10:20 ` Paweł Staszewski
2018-11-09 16:21 ` David Ahern
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-09 10:20 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 08.11.2018 o 17:06, David Ahern pisze:
> On 11/8/18 6:33 AM, Paweł Staszewski wrote:
>>
>> W dniu 07.11.2018 o 22:06, David Ahern pisze:
>>> On 11/3/18 6:24 PM, Paweł Staszewski wrote:
>>>>> Does your setup have any other device types besides physical ports with
>>>>> VLANs (e.g., any macvlans or bonds)?
>>>>>
>>>>>
>>>> no.
>>>> just
>>>> phy(mlnx)->vlans only config
>>> VLAN and non-VLAN (and a mix) seem to work ok. Patches are here:
>>> https://github.com/dsahern/linux.git bpf/kernel-tables-wip
>>>
>>> I got lazy with the vlan exports; right now it requires 8021q to be
>>> builtin (CONFIG_VLAN_8021Q=y)
>>>
>>> You can use the xdp_fwd sample:
>>> make O=kbuild -C samples/bpf -j 8
>>>
>>> Copy samples/bpf/xdp_fwd_kern.o and samples/bpf/xdp_fwd to the server
>>> and run:
>>> ./xdp_fwd <list of NIC ports>
>>>
>>> e.g., in my testing I run:
>>> xdp_fwd eth1 eth2 eth3 eth4
>>>
>>> All of the relevant forwarding ports need to be on the same command
>>> line. This version populates a second map to verify the egress port has
>>> XDP enabled.
>> Installed today on some lab server with mellanox connectx4
>>
>> And trying some simple static routing first - but after enabling xdp
>> program - receiver is not receiving frames
>>
>> Route table is simple as possible for tests :)
>>
>> icmp ping test send from 192.168.22.237 to 172.16.0.2 - incomming
>> packets on vlan 4081
>>
>> ip r
>> default via 192.168.22.236 dev vlan4081
>> 172.16.0.0/30 dev vlan1740 proto kernel scope link src 172.16.0.1
>> 192.168.22.0/24 dev vlan4081 proto kernel scope link src 192.168.22.205
>>
>> neigh table:
>> ip neigh ls
>>
>> 192.168.22.237 dev vlan4081 lladdr 00:25:90:fb:a6:8d REACHABLE
>> 172.16.0.2 dev vlan1740 lladdr ac:1f:6b:2c:2e:5a REACHABLE
>>
>> and interfaces:
>> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
>> UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
>> UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>
>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:5 qdisc
>> mq state UP group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
>> valid_lft forever preferred_lft forever
>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> inet 192.168.22.205/24 scope global vlan4081
>> valid_lft forever preferred_lft forever
>> inet6 fe80::ae1f:6bff:fe07:c890/64 scope link
>> valid_lft forever preferred_lft forever
>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> inet 172.16.0.1/30 scope global vlan1740
>> valid_lft forever preferred_lft forever
>> inet6 fe80::ae1f:6bff:fe07:c891/64 scope link
>> valid_lft forever preferred_lft forever
>>
>>
>> xdp program detached:
>> Receiving side tcpdump:
>> 14:28:09.141233 IP 192.168.22.237 > 172.16.0.2: ICMP echo request, id
>> 30227, seq 487, length 64
>>
>> I can see icmp requests
>>
>>
>> enabling xdp
>> ./xdp_fwd enp175s0f1 enp175s0f0
>>
>> 4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
>> state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> prog/xdp id 5 tag 3c231ff1e5e77f3f
>> 5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq
>> state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>> prog/xdp id 5 tag 3c231ff1e5e77f3f
>> 6: vlan4081@enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:90 brd ff:ff:ff:ff:ff:ff
>> 7: vlan1740@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP mode DEFAULT group default qlen 1000
>> link/ether ac:1f:6b:07:c8:91 brd ff:ff:ff:ff:ff:ff
>>
> What hardware is this?
>
> Start with:
>
> echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
> cat /sys/kernel/debug/tracing/trace_pipe
>
> >From there, you can check the FIB lookups:
> sysctl -w kernel.perf_event_max_stack=16
> perf record -e fib:* -a -g -- sleep 5
> perf script
>
I just catch some weird behavior :)
All was working fine for about 20k packets
Then after xdp start to forward every 10 packets
ping 172.16.0.2 -i 0.1
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.20 ms
64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=4.85 ms
64 bytes from 172.16.0.2: icmp_seq=29 ttl=64 time=4.91 ms
64 bytes from 172.16.0.2: icmp_seq=38 ttl=64 time=4.85 ms
64 bytes from 172.16.0.2: icmp_seq=48 ttl=64 time=5.00 ms
^C
--- 172.16.0.2 ping statistics ---
55 packets transmitted, 6 received, 89% packet loss, time 5655ms
rtt min/avg/max/mdev = 4.850/4.992/5.203/0.145 ms
And again after some time back to normal
ping 172.16.0.2 -i 0.1
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.02 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=5.06 ms
64 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=5.19 ms
64 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=5.07 ms
64 bytes from 172.16.0.2: icmp_seq=5 ttl=64 time=5.08 ms
64 bytes from 172.16.0.2: icmp_seq=6 ttl=64 time=5.14 ms
64 bytes from 172.16.0.2: icmp_seq=7 ttl=64 time=5.08 ms
64 bytes from 172.16.0.2: icmp_seq=8 ttl=64 time=5.17 ms
64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.04 ms
64 bytes from 172.16.0.2: icmp_seq=10 ttl=64 time=5.10 ms
64 bytes from 172.16.0.2: icmp_seq=11 ttl=64 time=5.11 ms
64 bytes from 172.16.0.2: icmp_seq=12 ttl=64 time=5.13 ms
64 bytes from 172.16.0.2: icmp_seq=13 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=14 ttl=64 time=5.15 ms
64 bytes from 172.16.0.2: icmp_seq=15 ttl=64 time=5.13 ms
64 bytes from 172.16.0.2: icmp_seq=16 ttl=64 time=5.04 ms
64 bytes from 172.16.0.2: icmp_seq=17 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=18 ttl=64 time=5.07 ms
64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=5.06 ms
64 bytes from 172.16.0.2: icmp_seq=20 ttl=64 time=5.12 ms
64 bytes from 172.16.0.2: icmp_seq=21 ttl=64 time=5.21 ms
64 bytes from 172.16.0.2: icmp_seq=22 ttl=64 time=4.98 ms
^C
--- 172.16.0.2 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 2105ms
rtt min/avg/max/mdev = 4.988/5.104/5.210/0.089 ms
I will try to catch this with debug enabled
Wondering also - cause xdp will bypass now vlan counters and other stuff
like tcpdump
Is there possible to add only counters from xdp for vlans ?
This will help me in testing.
And also - for non lab scenario there should be possible to sniff
sometimes on interface :)
Soo wondering if need to attack another xdp program to interface or all
this can be done by one
I think this is time where i will need to learn more about xdp :)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 10:20 ` Paweł Staszewski
@ 2018-11-09 16:21 ` David Ahern
2018-11-09 19:59 ` Paweł Staszewski
2018-11-10 0:06 ` David Ahern
0 siblings, 2 replies; 77+ messages in thread
From: David Ahern @ 2018-11-09 16:21 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/9/18 3:20 AM, Paweł Staszewski wrote:
>
> I just catch some weird behavior :)
> All was working fine for about 20k packets
>
> Then after xdp start to forward every 10 packets
Interesting. Any counter showing drops?
> ping 172.16.0.2 -i 0.1
> PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
> 64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.12 ms
> 64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.20 ms
> 64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=4.85 ms
> 64 bytes from 172.16.0.2: icmp_seq=29 ttl=64 time=4.91 ms
> 64 bytes from 172.16.0.2: icmp_seq=38 ttl=64 time=4.85 ms
> 64 bytes from 172.16.0.2: icmp_seq=48 ttl=64 time=5.00 ms
> ^C
> --- 172.16.0.2 ping statistics ---
> 55 packets transmitted, 6 received, 89% packet loss, time 5655ms
> rtt min/avg/max/mdev = 4.850/4.992/5.203/0.145 ms
>
>
> And again after some time back to normal
>
> ping 172.16.0.2 -i 0.1
> PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
> 64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.02 ms
> 64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=5.06 ms
> 64 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=5.19 ms
> 64 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=5.07 ms
> 64 bytes from 172.16.0.2: icmp_seq=5 ttl=64 time=5.08 ms
> 64 bytes from 172.16.0.2: icmp_seq=6 ttl=64 time=5.14 ms
> 64 bytes from 172.16.0.2: icmp_seq=7 ttl=64 time=5.08 ms
> 64 bytes from 172.16.0.2: icmp_seq=8 ttl=64 time=5.17 ms
> 64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.04 ms
> 64 bytes from 172.16.0.2: icmp_seq=10 ttl=64 time=5.10 ms
> 64 bytes from 172.16.0.2: icmp_seq=11 ttl=64 time=5.11 ms
> 64 bytes from 172.16.0.2: icmp_seq=12 ttl=64 time=5.13 ms
> 64 bytes from 172.16.0.2: icmp_seq=13 ttl=64 time=5.12 ms
> 64 bytes from 172.16.0.2: icmp_seq=14 ttl=64 time=5.15 ms
> 64 bytes from 172.16.0.2: icmp_seq=15 ttl=64 time=5.13 ms
> 64 bytes from 172.16.0.2: icmp_seq=16 ttl=64 time=5.04 ms
> 64 bytes from 172.16.0.2: icmp_seq=17 ttl=64 time=5.12 ms
> 64 bytes from 172.16.0.2: icmp_seq=18 ttl=64 time=5.07 ms
> 64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=5.06 ms
> 64 bytes from 172.16.0.2: icmp_seq=20 ttl=64 time=5.12 ms
> 64 bytes from 172.16.0.2: icmp_seq=21 ttl=64 time=5.21 ms
> 64 bytes from 172.16.0.2: icmp_seq=22 ttl=64 time=4.98 ms
> ^C
> --- 172.16.0.2 ping statistics ---
> 22 packets transmitted, 22 received, 0% packet loss, time 2105ms
> rtt min/avg/max/mdev = 4.988/5.104/5.210/0.089 ms
>
>
> I will try to catch this with debug enabled
>
>
>
>
>
> Wondering also - cause xdp will bypass now vlan counters and other stuff
> like tcpdump
yes, xdp is before tcpdump based sockets.
And the counters (vlan just being the current example) is another
problem to be solved. The vlan net_device never sees the packet and you
can not arbitrarily bump the counters just because the device lookups
reference them.
>
> Is there possible to add only counters from xdp for vlans ?
> This will help me in testing.
I will take a look today at adding counters that you can dump using
bpftool. It will be a temporary solution for this xdp program only.
>
>
> And also - for non lab scenario there should be possible to sniff
> sometimes on interface :)
Yes, sampling is another problem.
> Soo wondering if need to attack another xdp program to interface or all
> this can be done by one
>
> I think this is time where i will need to learn more about xdp :)
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 16:21 ` David Ahern
@ 2018-11-09 19:59 ` Paweł Staszewski
2018-11-10 0:06 ` David Ahern
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-09 19:59 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 09.11.2018 o 17:21, David Ahern pisze:
> On 11/9/18 3:20 AM, Paweł Staszewski wrote:
>> I just catch some weird behavior :)
>> All was working fine for about 20k packets
>>
>> Then after xdp start to forward every 10 packets
> Interesting. Any counter showing drops?
nothing that will fit
NIC statistics:
rx_packets: 187041
rx_bytes: 10600954
tx_packets: 40316
tx_bytes: 16526844
tx_tso_packets: 797
tx_tso_bytes: 3876084
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 38391
tx_nop: 2
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 187041
rx_csum_unnecessary: 0
rx_csum_none: 150011
rx_csum_complete: 37030
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 64893
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 2468
tx_csum_partial: 35955
tx_csum_partial_inner: 0
tx_queue_stopped: 0
tx_queue_dropped: 0
tx_xmit_more: 0
tx_recover: 0
tx_cqes: 38423
tx_queue_wake: 0
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 186302
rx_cache_full: 0
rx_cache_empty: 666768
rx_cache_busy: 174
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 249320
ch_poll: 249321
ch_arm: 249001
ch_aff_change: 0
ch_eq_rearm: 0
rx_out_of_buffer: 0
rx_if_down_packets: 57
rx_vport_unicast_packets: 142659
rx_vport_unicast_bytes: 42706914
tx_vport_unicast_packets: 40167
tx_vport_unicast_bytes: 16668096
rx_vport_multicast_packets: 39188170
rx_vport_multicast_bytes: 3466527450
tx_vport_multicast_packets: 58
tx_vport_multicast_bytes: 4556
rx_vport_broadcast_packets: 16343520
rx_vport_broadcast_bytes: 1031334602
tx_vport_broadcast_packets: 91
tx_vport_broadcast_bytes: 5460
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 40316
rx_packets_phy: 55674361
rx_crc_errors_phy: 0
tx_bytes_phy: 16839376
rx_bytes_phy: 4763267396
tx_multicast_phy: 58
tx_broadcast_phy: 91
rx_multicast_phy: 39188180
rx_broadcast_phy: 16343521
rx_in_range_len_errors_phy: 0
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 1
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 3792455
rx_65_to_127_bytes_phy: 51821620
rx_128_to_255_bytes_phy: 37669
rx_256_to_511_bytes_phy: 1481
rx_512_to_1023_bytes_phy: 434
rx_1024_to_1518_bytes_phy: 694
rx_1519_to_2047_bytes_phy: 20008
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 6
rx_err_lane_0_phy: 0
rx_err_lane_1_phy: 0
rx_err_lane_2_phy: 0
rx_err_lane_3_phy: 6
rx_buffer_passed_thres_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 82
outbound_pci_stalled_rd: 0
outbound_pci_stalled_wr: 0
outbound_pci_stalled_rd_events: 0
outbound_pci_stalled_wr_events: 0
rx_prio0_bytes: 4144920388
rx_prio0_packets: 48310037
tx_prio0_bytes: 16839376
tx_prio0_packets: 40316
rx_prio1_bytes: 481032
rx_prio1_packets: 7074
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 9074194
rx_prio2_packets: 106207
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 0
rx_prio3_packets: 0
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 0
rx_prio4_packets: 0
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 0
rx_prio5_packets: 0
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 371961810
rx_prio6_packets: 4006281
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 236830040
rx_prio7_packets: 3244761
tx_prio7_bytes: 0
tx_prio7_packets: 0
tx_pause_storm_warning_events : 0
tx_pause_storm_error_events: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
NIC statistics:
rx_packets: 843
rx_bytes: 58889
tx_packets: 324
tx_bytes: 23324
tx_tso_packets: 0
tx_tso_bytes: 0
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 293
tx_nop: 0
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 843
rx_csum_unnecessary: 0
rx_csum_none: 190
rx_csum_complete: 653
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 324
tx_csum_partial: 0
tx_csum_partial_inner: 0
tx_queue_stopped: 0
tx_queue_dropped: 0
tx_xmit_more: 1
tx_recover: 0
tx_cqes: 323
tx_queue_wake: 0
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 64926
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 47958
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
rx_page_reuse: 0
rx_cache_reuse: 648
rx_cache_full: 0
rx_cache_empty: 602112
rx_cache_busy: 0
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 49628
ch_poll: 49628
ch_arm: 49626
ch_aff_change: 0
ch_eq_rearm: 0
rx_out_of_buffer: 0
rx_if_down_packets: 46
rx_vport_unicast_packets: 5953
rx_vport_unicast_bytes: 4927049
tx_vport_unicast_packets: 65194
tx_vport_unicast_bytes: 31820150
rx_vport_multicast_packets: 37085249
rx_vport_multicast_bytes: 2449620421
tx_vport_multicast_packets: 55
tx_vport_multicast_bytes: 4278
rx_vport_broadcast_packets: 434654
rx_vport_broadcast_bytes: 31881063
tx_vport_broadcast_packets: 1
tx_vport_broadcast_bytes: 60
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 65250
rx_packets_phy: 37525857
rx_crc_errors_phy: 0
tx_bytes_phy: 32085488
rx_bytes_phy: 2636532027
tx_multicast_phy: 55
tx_broadcast_phy: 1
rx_multicast_phy: 37085250
rx_broadcast_phy: 434654
rx_in_range_len_errors_phy: 0
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 0
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 63346
rx_65_to_127_bytes_phy: 37434768
rx_128_to_255_bytes_phy: 14088
rx_256_to_511_bytes_phy: 10461
rx_512_to_1023_bytes_phy: 96
rx_1024_to_1518_bytes_phy: 1933
rx_1519_to_2047_bytes_phy: 1165
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 5
rx_err_lane_0_phy: 1
rx_err_lane_1_phy: 0
rx_err_lane_2_phy: 0
rx_err_lane_3_phy: 4
rx_buffer_passed_thres_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 82
outbound_pci_stalled_rd: 0
outbound_pci_stalled_wr: 0
outbound_pci_stalled_rd_events: 0
outbound_pci_stalled_wr_events: 0
rx_prio0_bytes: 23157221
rx_prio0_packets: 195789
tx_prio0_bytes: 32085488
tx_prio0_packets: 65250
rx_prio1_bytes: 0
rx_prio1_packets: 0
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 0
rx_prio2_packets: 0
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 23397578
rx_prio3_packets: 343182
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 0
rx_prio4_packets: 0
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 0
rx_prio5_packets: 0
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 14643472
rx_prio6_packets: 203589
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 2575333474
rx_prio7_packets: 36783293
tx_prio7_bytes: 0
tx_prio7_packets: 0
tx_pause_storm_warning_events : 0
tx_pause_storm_error_events: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
But wondering if any offloading now can do some things that we dont want
for xdp
currently all offloads are enabled.
ethtool -k enp175s0f0
Features for enp175s0f0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
Also at the time when xdp is forwarding 1/10 frame - same problem is
with local input/output traffic - testing server is also responding to
1/10 icmp request
>
>
>> ping 172.16.0.2 -i 0.1
>> PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
>> 64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.20 ms
>> 64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=4.85 ms
>> 64 bytes from 172.16.0.2: icmp_seq=29 ttl=64 time=4.91 ms
>> 64 bytes from 172.16.0.2: icmp_seq=38 ttl=64 time=4.85 ms
>> 64 bytes from 172.16.0.2: icmp_seq=48 ttl=64 time=5.00 ms
>> ^C
>> --- 172.16.0.2 ping statistics ---
>> 55 packets transmitted, 6 received, 89% packet loss, time 5655ms
>> rtt min/avg/max/mdev = 4.850/4.992/5.203/0.145 ms
>>
>>
>> And again after some time back to normal
>>
>> ping 172.16.0.2 -i 0.1
>> PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
>> 64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=5.02 ms
>> 64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=5.06 ms
>> 64 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=5.19 ms
>> 64 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=5.07 ms
>> 64 bytes from 172.16.0.2: icmp_seq=5 ttl=64 time=5.08 ms
>> 64 bytes from 172.16.0.2: icmp_seq=6 ttl=64 time=5.14 ms
>> 64 bytes from 172.16.0.2: icmp_seq=7 ttl=64 time=5.08 ms
>> 64 bytes from 172.16.0.2: icmp_seq=8 ttl=64 time=5.17 ms
>> 64 bytes from 172.16.0.2: icmp_seq=9 ttl=64 time=5.04 ms
>> 64 bytes from 172.16.0.2: icmp_seq=10 ttl=64 time=5.10 ms
>> 64 bytes from 172.16.0.2: icmp_seq=11 ttl=64 time=5.11 ms
>> 64 bytes from 172.16.0.2: icmp_seq=12 ttl=64 time=5.13 ms
>> 64 bytes from 172.16.0.2: icmp_seq=13 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=14 ttl=64 time=5.15 ms
>> 64 bytes from 172.16.0.2: icmp_seq=15 ttl=64 time=5.13 ms
>> 64 bytes from 172.16.0.2: icmp_seq=16 ttl=64 time=5.04 ms
>> 64 bytes from 172.16.0.2: icmp_seq=17 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=18 ttl=64 time=5.07 ms
>> 64 bytes from 172.16.0.2: icmp_seq=19 ttl=64 time=5.06 ms
>> 64 bytes from 172.16.0.2: icmp_seq=20 ttl=64 time=5.12 ms
>> 64 bytes from 172.16.0.2: icmp_seq=21 ttl=64 time=5.21 ms
>> 64 bytes from 172.16.0.2: icmp_seq=22 ttl=64 time=4.98 ms
>> ^C
>> --- 172.16.0.2 ping statistics ---
>> 22 packets transmitted, 22 received, 0% packet loss, time 2105ms
>> rtt min/avg/max/mdev = 4.988/5.104/5.210/0.089 ms
>>
>>
>> I will try to catch this with debug enabled
>>
>>
>>
>>
>>
>> Wondering also - cause xdp will bypass now vlan counters and other stuff
>> like tcpdump
> yes, xdp is before tcpdump based sockets.
>
> And the counters (vlan just being the current example) is another
> problem to be solved. The vlan net_device never sees the packet and you
> can not arbitrarily bump the counters just because the device lookups
> reference them.
Ok.
>> Is there possible to add only counters from xdp for vlans ?
>> This will help me in testing.
> I will take a look today at adding counters that you can dump using
> bpftool. It will be a temporary solution for this xdp program only.
Yes anything that can give me counters to check traffic lvls
>>
>> And also - for non lab scenario there should be possible to sniff
>> sometimes on interface :)
> Yes, sampling is another problem.
>
>
>> Soo wondering if need to attack another xdp program to interface or all
>> this can be done by one
>>
>> I think this is time where i will need to learn more about xdp :)
>>
>>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-08 19:12 ` Paweł Staszewski
@ 2018-11-09 22:20 ` Paweł Staszewski
2018-11-10 19:34 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-09 22:20 UTC (permalink / raw)
To: Saeed Mahameed, netdev, Jesper Dangaard Brouer
W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> CPU load is lower than for connectx4 - but it looks like bandwidth
> limit is the same :)
> But also after reaching 60Gbit/60Gbit
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> - iface Rx Tx Total
> ==============================================================================
>
> enp175s0: 45.09 Gb/s 15.09 Gb/s
> 60.18 Gb/s
> enp216s0: 15.14 Gb/s 45.19 Gb/s
> 60.33 Gb/s
> ------------------------------------------------------------------------------
>
> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
Today reached 65/65Gbit/s
But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
(with 50%CPU on all 28cores) - so still there is cpu power to use :).
So checked other stats.
softnet_stats shows average 1k squeezed per sec:
cpu total dropped squeezed collision rps flow_limit
0 18554 0 1 0 0 0
1 16728 0 1 0 0 0
2 18033 0 1 0 0 0
3 17757 0 1 0 0 0
4 18861 0 0 0 0 0
5 0 0 1 0 0 0
6 2 0 1 0 0 0
7 0 0 1 0 0 0
8 0 0 0 0 0 0
9 0 0 1 0 0 0
10 0 0 0 0 0 0
11 0 0 1 0 0 0
12 50 0 1 0 0 0
13 257 0 0 0 0 0
14 3629115363 0 3353259 0 0 0
15 255167835 0 3138271 0 0 0
16 4240101961 0 3036130 0 0 0
17 599810018 0 3072169 0 0 0
18 432796524 0 3034191 0 0 0
19 41803906 0 3037405 0 0 0
20 900382666 0 3112294 0 0 0
21 620926085 0 3086009 0 0 0
22 41861198 0 3023142 0 0 0
23 4090425574 0 2990412 0 0 0
24 4264870218 0 3010272 0 0 0
25 141401811 0 3027153 0 0 0
26 104155188 0 3051251 0 0 0
27 4261258691 0 3039765 0 0 0
28 4 0 1 0 0 0
29 4 0 0 0 0 0
30 0 0 1 0 0 0
31 0 0 0 0 0 0
32 3 0 1 0 0 0
33 1 0 1 0 0 0
34 0 0 1 0 0 0
35 0 0 0 0 0 0
36 0 0 1 0 0 0
37 0 0 1 0 0 0
38 0 0 1 0 0 0
39 0 0 1 0 0 0
40 0 0 0 0 0 0
41 0 0 1 0 0 0
42 299758202 0 3139693 0 0 0
43 4254727979 0 3103577 0 0 0
44 1959555543 0 2554885 0 0 0
45 1675702723 0 2513481 0 0 0
46 1908435503 0 2519698 0 0 0
47 1877799710 0 2537768 0 0 0
48 2384274076 0 2584673 0 0 0
49 2598104878 0 2593616 0 0 0
50 1897566829 0 2530857 0 0 0
51 1712741629 0 2489089 0 0 0
52 1704033648 0 2495892 0 0 0
53 1636781820 0 2499783 0 0 0
54 1861997734 0 2541060 0 0 0
55 2113521616 0 2555673 0 0 0
So i rised netdev backlog and budged to rly high values
524288 for netdev_budget and same for backlog
This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
But after this changes i have less packets drops.
Below perf top from max traffic reached:
PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
8.44% [kernel] [k] mlx5e_sq_xmit
6.69% [kernel] [k] build_skb
5.21% [kernel] [k] fib_table_lookup
3.54% [kernel] [k] memcpy_erms
3.20% [kernel] [k] mlx5e_poll_rx_cq
2.25% [kernel] [k] vlan_do_receive
2.20% [kernel] [k] mlx5e_post_rx_mpwqes
2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
1.95% [kernel] [k] __dev_queue_xmit
1.83% [kernel] [k] dev_gro_receive
1.79% [kernel] [k] tcp_gro_receive
1.73% [kernel] [k] ip_finish_output2
1.63% [kernel] [k] mlx5e_poll_tx_cq
1.49% [kernel] [k] ipt_do_table
1.38% [kernel] [k] inet_gro_receive
1.31% [kernel] [k] __netif_receive_skb_core
1.30% [kernel] [k] _raw_spin_lock
1.28% [kernel] [k] mlx5_eq_int
1.24% [kernel] [k] irq_entries_start
1.19% [kernel] [k] __build_skb
1.15% [kernel] [k] swiotlb_map_page
1.02% [kernel] [k] vlan_dev_hard_start_xmit
0.94% [kernel] [k] pfifo_fast_dequeue
0.92% [kernel] [k] ip_route_input_rcu
0.86% [kernel] [k] kmem_cache_alloc
0.80% [kernel] [k] mlx5e_xmit
0.79% [kernel] [k] dev_hard_start_xmit
0.78% [kernel] [k] _raw_spin_lock_irqsave
0.74% [kernel] [k] ip_forward
0.72% [kernel] [k] tasklet_action_common.isra.21
0.68% [kernel] [k] pfifo_fast_enqueue
0.67% [kernel] [k] netif_skb_features
0.66% [kernel] [k] skb_segment
0.60% [kernel] [k] skb_gro_receive
0.56% [kernel] [k] validate_xmit_skb.isra.142
0.53% [kernel] [k] skb_release_data
0.51% [kernel] [k] mlx5e_page_release
0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
0.51% [kernel] [k] __qdisc_run
0.50% [kernel] [k] tcp4_gro_receive
0.49% [kernel] [k] page_frag_free
0.46% [kernel] [k] kmem_cache_free_bulk
0.43% [kernel] [k] kmem_cache_free
0.42% [kernel] [k] try_to_wake_up
0.39% [kernel] [k] _raw_spin_lock_irq
0.39% [kernel] [k] find_busiest_group
0.37% [kernel] [k] __memcpy
Remember those tests are now on two separate connectx5 connected to two
separate pcie x16 gen 3.0
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 16:21 ` David Ahern
2018-11-09 19:59 ` Paweł Staszewski
@ 2018-11-10 0:06 ` David Ahern
2018-11-10 13:18 ` Paweł Staszewski
2018-11-19 21:59 ` David Ahern
1 sibling, 2 replies; 77+ messages in thread
From: David Ahern @ 2018-11-10 0:06 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/9/18 9:21 AM, David Ahern wrote:
>> Is there possible to add only counters from xdp for vlans ?
>> This will help me in testing.
> I will take a look today at adding counters that you can dump using
> bpftool. It will be a temporary solution for this xdp program only.
>
Same tree, kernel-tables-wip-02 branch. Compile kernel and install.
Compile samples as before.
If you give the userspace program a -t arg, it loop showing stats.
Ctrl-C to break. The xdp programs are not detached on exit.
Example:
./xdp_fwd -t 5 eth1 eth2 eth3 eth4
15:59:32: rx tx dropped skipped l3_dev fib_dev
index 3: 901158 901158 0 18 0 0
index 4: 901159 901158 0 20 0 901139
index 10: 0 0 0 0 19 19
index 11: 0 0 0 0 901139 901139
index 15: 0 0 0 0 19 19
index 16: 0 0 0 0 901139 0
Rx and Tx counters are for the physical port.
VLANs show up as l3_dev (ingress) and fib_dev (egress).
dropped is anytime the xdp program returns XDP_DROP (e.g., invalid packet)
skipped is anytime the program returns XDP_PASS (e.g., not ipv4 or ipv6,
local traffic, or needs full stack assist).
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 0:06 ` David Ahern
@ 2018-11-10 13:18 ` Paweł Staszewski
2018-11-10 14:56 ` David Ahern
2018-11-19 21:59 ` David Ahern
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-10 13:18 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 10.11.2018 o 01:06, David Ahern pisze:
> On 11/9/18 9:21 AM, David Ahern wrote:
>>> Is there possible to add only counters from xdp for vlans ?
>>> This will help me in testing.
>> I will take a look today at adding counters that you can dump using
>> bpftool. It will be a temporary solution for this xdp program only.
>>
> Same tree, kernel-tables-wip-02 branch. Compile kernel and install.
> Compile samples as before.
>
> If you give the userspace program a -t arg, it loop showing stats.
> Ctrl-C to break. The xdp programs are not detached on exit.
>
> Example:
>
> ./xdp_fwd -t 5 eth1 eth2 eth3 eth4
>
> 15:59:32: rx tx dropped skipped l3_dev fib_dev
> index 3: 901158 901158 0 18 0 0
> index 4: 901159 901158 0 20 0 901139
> index 10: 0 0 0 0 19 19
> index 11: 0 0 0 0 901139 901139
> index 15: 0 0 0 0 19 19
> index 16: 0 0 0 0 901139 0
>
> Rx and Tx counters are for the physical port.
>
> VLANs show up as l3_dev (ingress) and fib_dev (egress).
>
> dropped is anytime the xdp program returns XDP_DROP (e.g., invalid packet)
>
> skipped is anytime the program returns XDP_PASS (e.g., not ipv4 or ipv6,
> local traffic, or needs full stack assist).
>
recompiled new version but:
./xdp_fwd enp175s0f0 enp175s0f1
libbpf: failed to create map (name: 'stats_map'): Operation not permitted
libbpf: failed to load object './xdp_fwd_kern.o'
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 13:18 ` Paweł Staszewski
@ 2018-11-10 14:56 ` David Ahern
0 siblings, 0 replies; 77+ messages in thread
From: David Ahern @ 2018-11-10 14:56 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/10/18 6:18 AM, Paweł Staszewski wrote:
>
> ./xdp_fwd enp175s0f0 enp175s0f1
> libbpf: failed to create map (name: 'stats_map'): Operation not permitted
> libbpf: failed to load object './xdp_fwd_kern.o'
Forgot I had increased locked memory:
ulimit -l unlimited
./xdp_fwd enp175s0f0 enp175s0f1
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-09 22:20 ` Paweł Staszewski
@ 2018-11-10 19:34 ` Jesper Dangaard Brouer
2018-11-10 19:49 ` Paweł Staszewski
2018-11-10 20:02 ` Paweł Staszewski
0 siblings, 2 replies; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-10 19:34 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev, brouer
On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> > CPU load is lower than for connectx4 - but it looks like bandwidth
> > limit is the same :)
> > But also after reaching 60Gbit/60Gbit
> >
> > bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> > input: /proc/net/dev type: rate
> > - iface Rx Tx Total
> > ==========================================================================
> >
> > enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
> > enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
> > --------------------------------------------------------------------------
> >
> > total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>
> Today reached 65/65Gbit/s
>
> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
This is weird!
How do you see / measure these drops?
> So checked other stats.
> softnet_stats shows average 1k squeezed per sec:
Is below output the raw counters? not per sec?
It would be valuable to see the per sec stats instead...
I use this tool:
https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
> cpu total dropped squeezed collision rps flow_limit
> 0 18554 0 1 0 0 0
> 1 16728 0 1 0 0 0
> 2 18033 0 1 0 0 0
> 3 17757 0 1 0 0 0
> 4 18861 0 0 0 0 0
> 5 0 0 1 0 0 0
> 6 2 0 1 0 0 0
> 7 0 0 1 0 0 0
> 8 0 0 0 0 0 0
> 9 0 0 1 0 0 0
> 10 0 0 0 0 0 0
> 11 0 0 1 0 0 0
> 12 50 0 1 0 0 0
> 13 257 0 0 0 0 0
> 14 3629115363 0 3353259 0 0 0
> 15 255167835 0 3138271 0 0 0
> 16 4240101961 0 3036130 0 0 0
> 17 599810018 0 3072169 0 0 0
> 18 432796524 0 3034191 0 0 0
> 19 41803906 0 3037405 0 0 0
> 20 900382666 0 3112294 0 0 0
> 21 620926085 0 3086009 0 0 0
> 22 41861198 0 3023142 0 0 0
> 23 4090425574 0 2990412 0 0 0
> 24 4264870218 0 3010272 0 0 0
> 25 141401811 0 3027153 0 0 0
> 26 104155188 0 3051251 0 0 0
> 27 4261258691 0 3039765 0 0 0
> 28 4 0 1 0 0 0
> 29 4 0 0 0 0 0
> 30 0 0 1 0 0 0
> 31 0 0 0 0 0 0
> 32 3 0 1 0 0 0
> 33 1 0 1 0 0 0
> 34 0 0 1 0 0 0
> 35 0 0 0 0 0 0
> 36 0 0 1 0 0 0
> 37 0 0 1 0 0 0
> 38 0 0 1 0 0 0
> 39 0 0 1 0 0 0
> 40 0 0 0 0 0 0
> 41 0 0 1 0 0 0
> 42 299758202 0 3139693 0 0 0
> 43 4254727979 0 3103577 0 0 0
> 44 1959555543 0 2554885 0 0 0
> 45 1675702723 0 2513481 0 0 0
> 46 1908435503 0 2519698 0 0 0
> 47 1877799710 0 2537768 0 0 0
> 48 2384274076 0 2584673 0 0 0
> 49 2598104878 0 2593616 0 0 0
> 50 1897566829 0 2530857 0 0 0
> 51 1712741629 0 2489089 0 0 0
> 52 1704033648 0 2495892 0 0 0
> 53 1636781820 0 2499783 0 0 0
> 54 1861997734 0 2541060 0 0 0
> 55 2113521616 0 2555673 0 0 0
>
>
> So i rised netdev backlog and budged to rly high values
> 524288 for netdev_budget and same for backlog
Does it affect the squeezed counters?
Notice, this (crazy) huge netdev_budget limit will also be limited
by /proc/sys/net/core/netdev_budget_usecs.
> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
Hmmm, this could indicated not enough NAPI bulking is occurring.
I have a BPF tool, that can give you some insight into NAPI bulking and
softirq idle/kthread starting. Called 'napi_monitor', could you try to
run this, so can try to understand this? You find the tool here:
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
> But after this changes i have less packets drops.
>
>
> Below perf top from max traffic reached:
> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ------------------------------------------------------------------------------------------
>
> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
> 8.44% [kernel] [k] mlx5e_sq_xmit
> 6.69% [kernel] [k] build_skb
> 5.21% [kernel] [k] fib_table_lookup
> 3.54% [kernel] [k] memcpy_erms
> 3.20% [kernel] [k] mlx5e_poll_rx_cq
> 2.25% [kernel] [k] vlan_do_receive
> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
> 1.95% [kernel] [k] __dev_queue_xmit
> 1.83% [kernel] [k] dev_gro_receive
> 1.79% [kernel] [k] tcp_gro_receive
> 1.73% [kernel] [k] ip_finish_output2
> 1.63% [kernel] [k] mlx5e_poll_tx_cq
> 1.49% [kernel] [k] ipt_do_table
> 1.38% [kernel] [k] inet_gro_receive
> 1.31% [kernel] [k] __netif_receive_skb_core
> 1.30% [kernel] [k] _raw_spin_lock
> 1.28% [kernel] [k] mlx5_eq_int
> 1.24% [kernel] [k] irq_entries_start
> 1.19% [kernel] [k] __build_skb
> 1.15% [kernel] [k] swiotlb_map_page
> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
> 0.94% [kernel] [k] pfifo_fast_dequeue
> 0.92% [kernel] [k] ip_route_input_rcu
> 0.86% [kernel] [k] kmem_cache_alloc
> 0.80% [kernel] [k] mlx5e_xmit
> 0.79% [kernel] [k] dev_hard_start_xmit
> 0.78% [kernel] [k] _raw_spin_lock_irqsave
> 0.74% [kernel] [k] ip_forward
> 0.72% [kernel] [k] tasklet_action_common.isra.21
> 0.68% [kernel] [k] pfifo_fast_enqueue
> 0.67% [kernel] [k] netif_skb_features
> 0.66% [kernel] [k] skb_segment
> 0.60% [kernel] [k] skb_gro_receive
> 0.56% [kernel] [k] validate_xmit_skb.isra.142
> 0.53% [kernel] [k] skb_release_data
> 0.51% [kernel] [k] mlx5e_page_release
> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
> 0.51% [kernel] [k] __qdisc_run
> 0.50% [kernel] [k] tcp4_gro_receive
> 0.49% [kernel] [k] page_frag_free
> 0.46% [kernel] [k] kmem_cache_free_bulk
> 0.43% [kernel] [k] kmem_cache_free
> 0.42% [kernel] [k] try_to_wake_up
> 0.39% [kernel] [k] _raw_spin_lock_irq
> 0.39% [kernel] [k] find_busiest_group
> 0.37% [kernel] [k] __memcpy
>
>
>
> Remember those tests are now on two separate connectx5 connected to
> two separate pcie x16 gen 3.0
That is strange... I still suspect some HW NIC issue, can you provide
ethtool stats info via tool:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
$ ethtool_stats.pl --dev enp175s0 --dev enp216s0
The tool remove zero-stats counters and report per sec stats. It makes
it easier to spot that is relevant for the given workload.
Can you give output put from:
$ ethtool --show-priv-flag DEVICE
I want you to experiment with:
ethtool --set-priv-flags DEVICE rx_striding_rq off
I think you already have played with 'rx_cqe_compress', right.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 19:34 ` Jesper Dangaard Brouer
@ 2018-11-10 19:49 ` Paweł Staszewski
2018-11-10 19:56 ` Paweł Staszewski
2018-11-10 20:02 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-10 19:49 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>> limit is the same :)
>>> But also after reaching 60Gbit/60Gbit
>>>
>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>> input: /proc/net/dev type: rate
>>> - iface Rx Tx Total
>>> ==========================================================================
>>>
>>> enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
>>> enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
>>> --------------------------------------------------------------------------
>>>
>>> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>> Today reached 65/65Gbit/s
>>
>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
> This is weird!
>
> How do you see / measure these drops?
Simple icmp test like ping -i 0.1
And im testing by icmp management ip address on vlan that is attacked to
one NIC (the side that is more stressed with RX)
And another icmp test is forward thru this router - host behind it
Both measurements shows same loss ratio from 0.1 to 0.5% after reaching
~45Gbit/s RX side - depends how much RX side is pushed drops vary
between 0.1 to 0.5 - even 0.6%:)
>
>
>> So checked other stats.
>> softnet_stats shows average 1k squeezed per sec:
> Is below output the raw counters? not per sec?
>
> It would be valuable to see the per sec stats instead...
> I use this tool:
> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
>
>> cpu total dropped squeezed collision rps flow_limit
>> 0 18554 0 1 0 0 0
>> 1 16728 0 1 0 0 0
>> 2 18033 0 1 0 0 0
>> 3 17757 0 1 0 0 0
>> 4 18861 0 0 0 0 0
>> 5 0 0 1 0 0 0
>> 6 2 0 1 0 0 0
>> 7 0 0 1 0 0 0
>> 8 0 0 0 0 0 0
>> 9 0 0 1 0 0 0
>> 10 0 0 0 0 0 0
>> 11 0 0 1 0 0 0
>> 12 50 0 1 0 0 0
>> 13 257 0 0 0 0 0
>> 14 3629115363 0 3353259 0 0 0
>> 15 255167835 0 3138271 0 0 0
>> 16 4240101961 0 3036130 0 0 0
>> 17 599810018 0 3072169 0 0 0
>> 18 432796524 0 3034191 0 0 0
>> 19 41803906 0 3037405 0 0 0
>> 20 900382666 0 3112294 0 0 0
>> 21 620926085 0 3086009 0 0 0
>> 22 41861198 0 3023142 0 0 0
>> 23 4090425574 0 2990412 0 0 0
>> 24 4264870218 0 3010272 0 0 0
>> 25 141401811 0 3027153 0 0 0
>> 26 104155188 0 3051251 0 0 0
>> 27 4261258691 0 3039765 0 0 0
>> 28 4 0 1 0 0 0
>> 29 4 0 0 0 0 0
>> 30 0 0 1 0 0 0
>> 31 0 0 0 0 0 0
>> 32 3 0 1 0 0 0
>> 33 1 0 1 0 0 0
>> 34 0 0 1 0 0 0
>> 35 0 0 0 0 0 0
>> 36 0 0 1 0 0 0
>> 37 0 0 1 0 0 0
>> 38 0 0 1 0 0 0
>> 39 0 0 1 0 0 0
>> 40 0 0 0 0 0 0
>> 41 0 0 1 0 0 0
>> 42 299758202 0 3139693 0 0 0
>> 43 4254727979 0 3103577 0 0 0
>> 44 1959555543 0 2554885 0 0 0
>> 45 1675702723 0 2513481 0 0 0
>> 46 1908435503 0 2519698 0 0 0
>> 47 1877799710 0 2537768 0 0 0
>> 48 2384274076 0 2584673 0 0 0
>> 49 2598104878 0 2593616 0 0 0
>> 50 1897566829 0 2530857 0 0 0
>> 51 1712741629 0 2489089 0 0 0
>> 52 1704033648 0 2495892 0 0 0
>> 53 1636781820 0 2499783 0 0 0
>> 54 1861997734 0 2541060 0 0 0
>> 55 2113521616 0 2555673 0 0 0
>>
>>
>> So i rised netdev backlog and budged to rly high values
>> 524288 for netdev_budget and same for backlog
> Does it affect the squeezed counters?
a little - but not much
After change budget from 65536 to to 524k - number of squeezed counters
for all cpus changed from 1.5k per second to 0.9-1k per second - but
increasing it more like above 524k change nothing - same 0.9 to 1k/s
squeezed
>
> Notice, this (crazy) huge netdev_budget limit will also be limited
> by /proc/sys/net/core/netdev_budget_usecs.
Yes changed that also to 1000 / 2000 / 3000 / 4000 not much difference
on squeezed - even cant see the difference
>
>> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
> Hmmm, this could indicated not enough NAPI bulking is occurring.
>
> I have a BPF tool, that can give you some insight into NAPI bulking and
> softirq idle/kthread starting. Called 'napi_monitor', could you try to
> run this, so can try to understand this? You find the tool here:
>
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
yes will try it
>
>> But after this changes i have less packets drops.
>>
>>
>> Below perf top from max traffic reached:
>> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> ------------------------------------------------------------------------------------------
>>
>> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
>> 8.44% [kernel] [k] mlx5e_sq_xmit
>> 6.69% [kernel] [k] build_skb
>> 5.21% [kernel] [k] fib_table_lookup
>> 3.54% [kernel] [k] memcpy_erms
>> 3.20% [kernel] [k] mlx5e_poll_rx_cq
>> 2.25% [kernel] [k] vlan_do_receive
>> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
>> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
>> 1.95% [kernel] [k] __dev_queue_xmit
>> 1.83% [kernel] [k] dev_gro_receive
>> 1.79% [kernel] [k] tcp_gro_receive
>> 1.73% [kernel] [k] ip_finish_output2
>> 1.63% [kernel] [k] mlx5e_poll_tx_cq
>> 1.49% [kernel] [k] ipt_do_table
>> 1.38% [kernel] [k] inet_gro_receive
>> 1.31% [kernel] [k] __netif_receive_skb_core
>> 1.30% [kernel] [k] _raw_spin_lock
>> 1.28% [kernel] [k] mlx5_eq_int
>> 1.24% [kernel] [k] irq_entries_start
>> 1.19% [kernel] [k] __build_skb
>> 1.15% [kernel] [k] swiotlb_map_page
>> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.94% [kernel] [k] pfifo_fast_dequeue
>> 0.92% [kernel] [k] ip_route_input_rcu
>> 0.86% [kernel] [k] kmem_cache_alloc
>> 0.80% [kernel] [k] mlx5e_xmit
>> 0.79% [kernel] [k] dev_hard_start_xmit
>> 0.78% [kernel] [k] _raw_spin_lock_irqsave
>> 0.74% [kernel] [k] ip_forward
>> 0.72% [kernel] [k] tasklet_action_common.isra.21
>> 0.68% [kernel] [k] pfifo_fast_enqueue
>> 0.67% [kernel] [k] netif_skb_features
>> 0.66% [kernel] [k] skb_segment
>> 0.60% [kernel] [k] skb_gro_receive
>> 0.56% [kernel] [k] validate_xmit_skb.isra.142
>> 0.53% [kernel] [k] skb_release_data
>> 0.51% [kernel] [k] mlx5e_page_release
>> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>> 0.51% [kernel] [k] __qdisc_run
>> 0.50% [kernel] [k] tcp4_gro_receive
>> 0.49% [kernel] [k] page_frag_free
>> 0.46% [kernel] [k] kmem_cache_free_bulk
>> 0.43% [kernel] [k] kmem_cache_free
>> 0.42% [kernel] [k] try_to_wake_up
>> 0.39% [kernel] [k] _raw_spin_lock_irq
>> 0.39% [kernel] [k] find_busiest_group
>> 0.37% [kernel] [k] __memcpy
>>
>>
>>
>> Remember those tests are now on two separate connectx5 connected to
>> two separate pcie x16 gen 3.0
>
> That is strange... I still suspect some HW NIC issue, can you provide
> ethtool stats info via tool:
>
> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>
> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>
> The tool remove zero-stats counters and report per sec stats. It makes
> it easier to spot that is relevant for the given workload.
yes mlnx have just too many counters that are always 0 for my case :)
Will try this also
>
> Can you give output put from:
> $ ethtool --show-priv-flag DEVICE
>
> I want you to experiment with:
ethtool --show-priv-flags enp175s0
Private flags for enp175s0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : off
rx_striding_rq : on
rx_no_csum_complete: off
>
> ethtool --set-priv-flags DEVICE rx_striding_rq off
ok i will first check on test server if this will reset my interface and
will not produce kernel panic :)
>
> I think you already have played with 'rx_cqe_compress', right.
yes - and compress increasing number of irq's but doing not much for
bandwidth same limit 60-64Gbit/s total RX+TX on one 100G port
And what is weird - that limit is in overall symetric - cause if for
example 100G port is receiving 42G traffic and transmitting 20G traffic
- when i flood rx side with pktgen or other for example icmp traffic
1/2/3/4/5G - then receiving side increase with 1/2/3/4/5Gbit of traffic
but transmitting is going down for same lvl's
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 19:49 ` Paweł Staszewski
@ 2018-11-10 19:56 ` Paweł Staszewski
2018-11-10 22:06 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-10 19:56 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
>
>
> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>>> limit is the same :)
>>>> But also after reaching 60Gbit/60Gbit
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> ==========================================================================
>>>>
>>>>
>>>> enp175s0: 45.09 Gb/s 15.09 Gb/s
>>>> 60.18 Gb/s
>>>> enp216s0: 15.14 Gb/s 45.19 Gb/s
>>>> 60.33 Gb/s
>>>> --------------------------------------------------------------------------
>>>>
>>>>
>>>> total: 60.45 Gb/s 60.48 Gb/s 120.93
>>>> Gb/s
>>> Today reached 65/65Gbit/s
>>>
>>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
>> This is weird!
>>
>> How do you see / measure these drops?
> Simple icmp test like ping -i 0.1
> And im testing by icmp management ip address on vlan that is attacked
> to one NIC (the side that is more stressed with RX)
> And another icmp test is forward thru this router - host behind it
>
> Both measurements shows same loss ratio from 0.1 to 0.5% after
> reaching ~45Gbit/s RX side - depends how much RX side is pushed drops
> vary between 0.1 to 0.5 - even 0.6%:)
>
>
>>
>>
>>> So checked other stats.
>>> softnet_stats shows average 1k squeezed per sec:
>> Is below output the raw counters? not per sec?
>>
>> It would be valuable to see the per sec stats instead...
>> I use this tool:
>> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
CPU total/sec dropped/sec squeezed/sec
collision/sec rx_rps/sec flow_limit/sec
CPU:00 0 0 0 0
0 0
CPU:01 0 0 0 0
0 0
CPU:02 0 0 0 0
0 0
CPU:03 0 0 0 0
0 0
CPU:04 0 0 0 0
0 0
CPU:05 0 0 0 0
0 0
CPU:06 0 0 0 0
0 0
CPU:07 0 0 0 0
0 0
CPU:08 0 0 0 0
0 0
CPU:09 0 0 0 0
0 0
CPU:10 0 0 0 0
0 0
CPU:11 0 0 0 0
0 0
CPU:12 0 0 0 0
0 0
CPU:13 0 0 0 0
0 0
CPU:14 485538 0 43 0
0 0
CPU:15 474794 0 51 0
0 0
CPU:16 449322 0 41 0
0 0
CPU:17 476420 0 46 0
0 0
CPU:18 440436 0 38 0
0 0
CPU:19 501499 0 49 0
0 0
CPU:20 459468 0 49 0
0 0
CPU:21 438928 0 47 0
0 0
CPU:22 468983 0 40 0
0 0
CPU:23 446253 0 47 0
0 0
CPU:24 451909 0 46 0
0 0
CPU:25 479373 0 55 0
0 0
CPU:26 467848 0 49 0
0 0
CPU:27 453153 0 51 0
0 0
CPU:28 0 0 0 0
0 0
CPU:29 0 0 0 0
0 0
CPU:30 0 0 0 0
0 0
CPU:31 0 0 0 0
0 0
CPU:32 0 0 0 0
0 0
CPU:33 0 0 0 0
0 0
CPU:34 0 0 0 0
0 0
CPU:35 0 0 0 0
0 0
CPU:36 0 0 0 0
0 0
CPU:37 0 0 0 0
0 0
CPU:38 0 0 0 0
0 0
CPU:39 0 0 0 0
0 0
CPU:40 0 0 0 0
0 0
CPU:41 0 0 0 0
0 0
CPU:42 466853 0 43 0
0 0
CPU:43 453059 0 54 0
0 0
CPU:44 363219 0 34 0
0 0
CPU:45 353632 0 38 0
0 0
CPU:46 371618 0 40 0
0 0
CPU:47 350518 0 46 0
0 0
CPU:48 397544 0 40 0
0 0
CPU:49 364873 0 38 0
0 0
CPU:50 383630 0 38 0
0 0
CPU:51 358771 0 39 0
0 0
CPU:52 372547 0 38 0
0 0
CPU:53 372882 0 36 0
0 0
CPU:54 366244 0 43 0
0 0
CPU:55 365886 0 39 0
0 0
Summed: 11835201 0 1217 0
0 0
>>
>>> cpu total dropped squeezed collision rps flow_limit
>>> 0 18554 0 1 0 0 0
>>> 1 16728 0 1 0 0 0
>>> 2 18033 0 1 0 0 0
>>> 3 17757 0 1 0 0 0
>>> 4 18861 0 0 0 0 0
>>> 5 0 0 1 0 0 0
>>> 6 2 0 1 0 0 0
>>> 7 0 0 1 0 0 0
>>> 8 0 0 0 0 0 0
>>> 9 0 0 1 0 0 0
>>> 10 0 0 0 0 0 0
>>> 11 0 0 1 0 0 0
>>> 12 50 0 1 0 0 0
>>> 13 257 0 0 0 0 0
>>> 14 3629115363 0 3353259 0 0 0
>>> 15 255167835 0 3138271 0 0 0
>>> 16 4240101961 0 3036130 0 0 0
>>> 17 599810018 0 3072169 0 0 0
>>> 18 432796524 0 3034191 0 0 0
>>> 19 41803906 0 3037405 0 0 0
>>> 20 900382666 0 3112294 0 0 0
>>> 21 620926085 0 3086009 0 0 0
>>> 22 41861198 0 3023142 0 0 0
>>> 23 4090425574 0 2990412 0 0 0
>>> 24 4264870218 0 3010272 0 0 0
>>> 25 141401811 0 3027153 0 0 0
>>> 26 104155188 0 3051251 0 0 0
>>> 27 4261258691 0 3039765 0 0 0
>>> 28 4 0 1 0 0 0
>>> 29 4 0 0 0 0 0
>>> 30 0 0 1 0 0 0
>>> 31 0 0 0 0 0 0
>>> 32 3 0 1 0 0 0
>>> 33 1 0 1 0 0 0
>>> 34 0 0 1 0 0 0
>>> 35 0 0 0 0 0 0
>>> 36 0 0 1 0 0 0
>>> 37 0 0 1 0 0 0
>>> 38 0 0 1 0 0 0
>>> 39 0 0 1 0 0 0
>>> 40 0 0 0 0 0 0
>>> 41 0 0 1 0 0 0
>>> 42 299758202 0 3139693 0 0 0
>>> 43 4254727979 0 3103577 0 0 0
>>> 44 1959555543 0 2554885 0 0 0
>>> 45 1675702723 0 2513481 0 0 0
>>> 46 1908435503 0 2519698 0 0 0
>>> 47 1877799710 0 2537768 0 0 0
>>> 48 2384274076 0 2584673 0 0 0
>>> 49 2598104878 0 2593616 0 0 0
>>> 50 1897566829 0 2530857 0 0 0
>>> 51 1712741629 0 2489089 0 0 0
>>> 52 1704033648 0 2495892 0 0 0
>>> 53 1636781820 0 2499783 0 0 0
>>> 54 1861997734 0 2541060 0 0 0
>>> 55 2113521616 0 2555673 0 0 0
>>>
>>>
>>> So i rised netdev backlog and budged to rly high values
>>> 524288 for netdev_budget and same for backlog
>> Does it affect the squeezed counters?
> a little - but not much
> After change budget from 65536 to to 524k - number of squeezed
> counters for all cpus changed from 1.5k per second to 0.9-1k per
> second - but increasing it more like above 524k change nothing - same
> 0.9 to 1k/s squeezed
>>
>> Notice, this (crazy) huge netdev_budget limit will also be limited
>> by /proc/sys/net/core/netdev_budget_usecs.
> Yes changed that also to 1000 / 2000 / 3000 / 4000 not much
> difference on squeezed - even cant see the difference
>
>>
>>> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
>> Hmmm, this could indicated not enough NAPI bulking is occurring.
>>
>> I have a BPF tool, that can give you some insight into NAPI bulking and
>> softirq idle/kthread starting. Called 'napi_monitor', could you try to
>> run this, so can try to understand this? You find the tool here:
>>
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
> yes will try it
>
>>> But after this changes i have less packets drops.
>>>
>>>
>>> Below perf top from max traffic reached:
>>> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>> ------------------------------------------------------------------------------------------
>>>
>>>
>>> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
>>> 8.44% [kernel] [k] mlx5e_sq_xmit
>>> 6.69% [kernel] [k] build_skb
>>> 5.21% [kernel] [k] fib_table_lookup
>>> 3.54% [kernel] [k] memcpy_erms
>>> 3.20% [kernel] [k] mlx5e_poll_rx_cq
>>> 2.25% [kernel] [k] vlan_do_receive
>>> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
>>> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
>>> 1.95% [kernel] [k] __dev_queue_xmit
>>> 1.83% [kernel] [k] dev_gro_receive
>>> 1.79% [kernel] [k] tcp_gro_receive
>>> 1.73% [kernel] [k] ip_finish_output2
>>> 1.63% [kernel] [k] mlx5e_poll_tx_cq
>>> 1.49% [kernel] [k] ipt_do_table
>>> 1.38% [kernel] [k] inet_gro_receive
>>> 1.31% [kernel] [k] __netif_receive_skb_core
>>> 1.30% [kernel] [k] _raw_spin_lock
>>> 1.28% [kernel] [k] mlx5_eq_int
>>> 1.24% [kernel] [k] irq_entries_start
>>> 1.19% [kernel] [k] __build_skb
>>> 1.15% [kernel] [k] swiotlb_map_page
>>> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
>>> 0.94% [kernel] [k] pfifo_fast_dequeue
>>> 0.92% [kernel] [k] ip_route_input_rcu
>>> 0.86% [kernel] [k] kmem_cache_alloc
>>> 0.80% [kernel] [k] mlx5e_xmit
>>> 0.79% [kernel] [k] dev_hard_start_xmit
>>> 0.78% [kernel] [k] _raw_spin_lock_irqsave
>>> 0.74% [kernel] [k] ip_forward
>>> 0.72% [kernel] [k] tasklet_action_common.isra.21
>>> 0.68% [kernel] [k] pfifo_fast_enqueue
>>> 0.67% [kernel] [k] netif_skb_features
>>> 0.66% [kernel] [k] skb_segment
>>> 0.60% [kernel] [k] skb_gro_receive
>>> 0.56% [kernel] [k] validate_xmit_skb.isra.142
>>> 0.53% [kernel] [k] skb_release_data
>>> 0.51% [kernel] [k] mlx5e_page_release
>>> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>>> 0.51% [kernel] [k] __qdisc_run
>>> 0.50% [kernel] [k] tcp4_gro_receive
>>> 0.49% [kernel] [k] page_frag_free
>>> 0.46% [kernel] [k] kmem_cache_free_bulk
>>> 0.43% [kernel] [k] kmem_cache_free
>>> 0.42% [kernel] [k] try_to_wake_up
>>> 0.39% [kernel] [k] _raw_spin_lock_irq
>>> 0.39% [kernel] [k] find_busiest_group
>>> 0.37% [kernel] [k] __memcpy
>>>
>>>
>>>
>>> Remember those tests are now on two separate connectx5 connected to
>>> two separate pcie x16 gen 3.0
>> That is strange... I still suspect some HW NIC issue, can you provide
>> ethtool stats info via tool:
>>
>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>
>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>>
>> The tool remove zero-stats counters and report per sec stats. It makes
>> it easier to spot that is relevant for the given workload.
> yes mlnx have just too many counters that are always 0 for my case :)
> Will try this also
>
But still alot of non 0 counters
Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
Ethtool(enp175s0) stat: 10265 ( 10,265) <= ch0_events /sec
Ethtool(enp175s0) stat: 11072 ( 11,072) <= ch0_poll /sec
Ethtool(enp175s0) stat: 9003 ( 9,003) <= ch10_arm /sec
Ethtool(enp175s0) stat: 10476 ( 10,476) <= ch10_events /sec
Ethtool(enp175s0) stat: 11284 ( 11,284) <= ch10_poll /sec
Ethtool(enp175s0) stat: 11211 ( 11,211) <= ch11_arm /sec
Ethtool(enp175s0) stat: 12645 ( 12,645) <= ch11_events /sec
Ethtool(enp175s0) stat: 13450 ( 13,450) <= ch11_poll /sec
Ethtool(enp175s0) stat: 9012 ( 9,012) <= ch12_arm /sec
Ethtool(enp175s0) stat: 10366 ( 10,366) <= ch12_events /sec
Ethtool(enp175s0) stat: 11074 ( 11,074) <= ch12_poll /sec
Ethtool(enp175s0) stat: 8810 ( 8,810) <= ch13_arm /sec
Ethtool(enp175s0) stat: 10177 ( 10,177) <= ch13_events /sec
Ethtool(enp175s0) stat: 10886 ( 10,886) <= ch13_poll /sec
Ethtool(enp175s0) stat: 9794 ( 9,794) <= ch14_arm /sec
Ethtool(enp175s0) stat: 11159 ( 11,159) <= ch14_events /sec
Ethtool(enp175s0) stat: 11932 ( 11,932) <= ch14_poll /sec
Ethtool(enp175s0) stat: 8703 ( 8,703) <= ch15_arm /sec
Ethtool(enp175s0) stat: 10052 ( 10,052) <= ch15_events /sec
Ethtool(enp175s0) stat: 10774 ( 10,774) <= ch15_poll /sec
Ethtool(enp175s0) stat: 6429 ( 6,429) <= ch16_arm /sec
Ethtool(enp175s0) stat: 7591 ( 7,591) <= ch16_events /sec
Ethtool(enp175s0) stat: 8223 ( 8,223) <= ch16_poll /sec
Ethtool(enp175s0) stat: 8981 ( 8,981) <= ch17_arm /sec
Ethtool(enp175s0) stat: 10229 ( 10,229) <= ch17_events /sec
Ethtool(enp175s0) stat: 10887 ( 10,887) <= ch17_poll /sec
Ethtool(enp175s0) stat: 6786 ( 6,786) <= ch18_arm /sec
Ethtool(enp175s0) stat: 7887 ( 7,887) <= ch18_events /sec
Ethtool(enp175s0) stat: 8484 ( 8,484) <= ch18_poll /sec
Ethtool(enp175s0) stat: 6080 ( 6,080) <= ch19_arm /sec
Ethtool(enp175s0) stat: 7377 ( 7,377) <= ch19_events /sec
Ethtool(enp175s0) stat: 8124 ( 8,124) <= ch19_poll /sec
Ethtool(enp175s0) stat: 7715 ( 7,715) <= ch1_arm /sec
Ethtool(enp175s0) stat: 9109 ( 9,109) <= ch1_events /sec
Ethtool(enp175s0) stat: 9923 ( 9,923) <= ch1_poll /sec
Ethtool(enp175s0) stat: 7303 ( 7,303) <= ch20_arm /sec
Ethtool(enp175s0) stat: 8514 ( 8,514) <= ch20_events /sec
Ethtool(enp175s0) stat: 9169 ( 9,169) <= ch20_poll /sec
Ethtool(enp175s0) stat: 8972 ( 8,972) <= ch21_arm /sec
Ethtool(enp175s0) stat: 10060 ( 10,060) <= ch21_events /sec
Ethtool(enp175s0) stat: 10647 ( 10,647) <= ch21_poll /sec
Ethtool(enp175s0) stat: 7729 ( 7,729) <= ch22_arm /sec
Ethtool(enp175s0) stat: 8932 ( 8,932) <= ch22_events /sec
Ethtool(enp175s0) stat: 9585 ( 9,585) <= ch22_poll /sec
Ethtool(enp175s0) stat: 8125 ( 8,125) <= ch23_arm /sec
Ethtool(enp175s0) stat: 9218 ( 9,218) <= ch23_events /sec
Ethtool(enp175s0) stat: 9805 ( 9,805) <= ch23_poll /sec
Ethtool(enp175s0) stat: 7212 ( 7,212) <= ch24_arm /sec
Ethtool(enp175s0) stat: 8369 ( 8,369) <= ch24_events /sec
Ethtool(enp175s0) stat: 8993 ( 8,993) <= ch24_poll /sec
Ethtool(enp175s0) stat: 6328 ( 6,328) <= ch25_arm /sec
Ethtool(enp175s0) stat: 7567 ( 7,567) <= ch25_events /sec
Ethtool(enp175s0) stat: 8274 ( 8,274) <= ch25_poll /sec
Ethtool(enp175s0) stat: 6210 ( 6,210) <= ch26_arm /sec
Ethtool(enp175s0) stat: 7409 ( 7,409) <= ch26_events /sec
Ethtool(enp175s0) stat: 8062 ( 8,062) <= ch26_poll /sec
Ethtool(enp175s0) stat: 7035 ( 7,035) <= ch27_arm /sec
Ethtool(enp175s0) stat: 8203 ( 8,203) <= ch27_events /sec
Ethtool(enp175s0) stat: 8840 ( 8,840) <= ch27_poll /sec
Ethtool(enp175s0) stat: 11278 ( 11,278) <= ch2_arm /sec
Ethtool(enp175s0) stat: 12632 ( 12,632) <= ch2_events /sec
Ethtool(enp175s0) stat: 13348 ( 13,348) <= ch2_poll /sec
Ethtool(enp175s0) stat: 10612 ( 10,612) <= ch3_arm /sec
Ethtool(enp175s0) stat: 11900 ( 11,900) <= ch3_events /sec
Ethtool(enp175s0) stat: 12567 ( 12,567) <= ch3_poll /sec
Ethtool(enp175s0) stat: 8936 ( 8,936) <= ch4_arm /sec
Ethtool(enp175s0) stat: 10248 ( 10,248) <= ch4_events /sec
Ethtool(enp175s0) stat: 10962 ( 10,962) <= ch4_poll /sec
Ethtool(enp175s0) stat: 11631 ( 11,631) <= ch5_arm /sec
Ethtool(enp175s0) stat: 12953 ( 12,953) <= ch5_events /sec
Ethtool(enp175s0) stat: 13629 ( 13,629) <= ch5_poll /sec
Ethtool(enp175s0) stat: 9877 ( 9,877) <= ch6_arm /sec
Ethtool(enp175s0) stat: 11114 ( 11,114) <= ch6_events /sec
Ethtool(enp175s0) stat: 11800 ( 11,800) <= ch6_poll /sec
Ethtool(enp175s0) stat: 8228 ( 8,228) <= ch7_arm /sec
Ethtool(enp175s0) stat: 9577 ( 9,577) <= ch7_events /sec
Ethtool(enp175s0) stat: 10320 ( 10,320) <= ch7_poll /sec
Ethtool(enp175s0) stat: 11808 ( 11,808) <= ch8_arm /sec
Ethtool(enp175s0) stat: 13135 ( 13,135) <= ch8_events /sec
Ethtool(enp175s0) stat: 13828 ( 13,828) <= ch8_poll /sec
Ethtool(enp175s0) stat: 10566 ( 10,566) <= ch9_arm /sec
Ethtool(enp175s0) stat: 11904 ( 11,904) <= ch9_events /sec
Ethtool(enp175s0) stat: 12634 ( 12,634) <= ch9_poll /sec
Ethtool(enp175s0) stat: 243256 ( 243,256) <= ch_arm /sec
Ethtool(enp175s0) stat: 279057 ( 279,057) <= ch_events /sec
Ethtool(enp175s0) stat: 298563 ( 298,563) <= ch_poll /sec
Ethtool(enp175s0) stat: 186677525 ( 186,677,525) <= rx0_bytes /sec
Ethtool(enp175s0) stat: 72870 ( 72,870) <=
rx0_cache_reuse /sec
Ethtool(enp175s0) stat: 145627 ( 145,627) <=
rx0_csum_complete /sec
Ethtool(enp175s0) stat: 88 ( 88) <= rx0_csum_none /sec
Ethtool(enp175s0) stat: 145715 ( 145,715) <= rx0_packets /sec
Ethtool(enp175s0) stat: 145715 ( 145,715) <=
rx0_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 198552827 ( 198,552,827) <= rx10_bytes /sec
Ethtool(enp175s0) stat: 75553 ( 75,553) <=
rx10_cache_reuse /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <=
rx10_csum_complete /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <= rx10_packets /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <=
rx10_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 200924148 ( 200,924,148) <= rx11_bytes /sec
Ethtool(enp175s0) stat: 76589 ( 76,589) <=
rx11_cache_reuse /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <=
rx11_csum_complete /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <= rx11_packets /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <=
rx11_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 186259790 ( 186,259,790) <= rx12_bytes /sec
Ethtool(enp175s0) stat: 70675 ( 70,675) <=
rx12_cache_reuse /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <=
rx12_csum_complete /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <= rx12_packets /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <=
rx12_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 189627451 ( 189,627,451) <= rx13_bytes /sec
Ethtool(enp175s0) stat: 72626 ( 72,626) <=
rx13_cache_reuse /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <=
rx13_csum_complete /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <= rx13_packets /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <=
rx13_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 199246096 ( 199,246,096) <= rx14_bytes /sec
Ethtool(enp175s0) stat: 77992 ( 77,992) <=
rx14_cache_reuse /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <=
rx14_csum_complete /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <= rx14_packets /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <=
rx14_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 189698176 ( 189,698,176) <= rx15_bytes /sec
Ethtool(enp175s0) stat: 72382 ( 72,382) <=
rx15_cache_reuse /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <=
rx15_csum_complete /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <= rx15_packets /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <=
rx15_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 143896232 ( 143,896,232) <= rx16_bytes /sec
Ethtool(enp175s0) stat: 55369 ( 55,369) <=
rx16_cache_reuse /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <=
rx16_csum_complete /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <= rx16_packets /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <=
rx16_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 171449483 ( 171,449,483) <= rx17_bytes /sec
Ethtool(enp175s0) stat: 65308 ( 65,308) <=
rx17_cache_reuse /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <=
rx17_csum_complete /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <= rx17_packets /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <=
rx17_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 141033264 ( 141,033,264) <= rx18_bytes /sec
Ethtool(enp175s0) stat: 54515 ( 54,515) <=
rx18_cache_reuse /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <=
rx18_csum_complete /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <= rx18_packets /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <=
rx18_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 163097410 ( 163,097,410) <= rx19_bytes /sec
Ethtool(enp175s0) stat: 61894 ( 61,894) <=
rx19_cache_reuse /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <=
rx19_csum_complete /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <= rx19_packets /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <=
rx19_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 181707418 ( 181,707,418) <= rx1_bytes /sec
Ethtool(enp175s0) stat: 71223 ( 71,223) <=
rx1_cache_reuse /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <=
rx1_csum_complete /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <= rx1_packets /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <=
rx1_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 161626368 ( 161,626,368) <= rx20_bytes /sec
Ethtool(enp175s0) stat: 61345 ( 61,345) <=
rx20_cache_reuse /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <=
rx20_csum_complete /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <= rx20_packets /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <=
rx20_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 138593554 ( 138,593,554) <= rx21_bytes /sec
Ethtool(enp175s0) stat: 53478 ( 53,478) <=
rx21_cache_reuse /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <=
rx21_csum_complete /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <= rx21_packets /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <=
rx21_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 149217722 ( 149,217,722) <= rx22_bytes /sec
Ethtool(enp175s0) stat: 58174 ( 58,174) <=
rx22_cache_reuse /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <=
rx22_csum_complete /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <= rx22_packets /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <=
rx22_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 147968086 ( 147,968,086) <= rx23_bytes /sec
Ethtool(enp175s0) stat: 55979 ( 55,979) <=
rx23_cache_reuse /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <=
rx23_csum_complete /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <= rx23_packets /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <=
rx23_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 145955524 ( 145,955,524) <= rx24_bytes /sec
Ethtool(enp175s0) stat: 55491 ( 55,491) <=
rx24_cache_reuse /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <=
rx24_csum_complete /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <= rx24_packets /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <=
rx24_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 155552699 ( 155,552,699) <= rx25_bytes /sec
Ethtool(enp175s0) stat: 59028 ( 59,028) <=
rx25_cache_reuse /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <=
rx25_csum_complete /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <= rx25_packets /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <=
rx25_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 144880442 ( 144,880,442) <= rx26_bytes /sec
Ethtool(enp175s0) stat: 56223 ( 56,223) <=
rx26_cache_reuse /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <=
rx26_csum_complete /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <= rx26_packets /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <=
rx26_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 154545288 ( 154,545,288) <= rx27_bytes /sec
Ethtool(enp175s0) stat: 58784 ( 58,784) <=
rx27_cache_reuse /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <=
rx27_csum_complete /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <= rx27_packets /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <=
rx27_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 182425129 ( 182,425,129) <= rx2_bytes /sec
Ethtool(enp175s0) stat: 71406 ( 71,406) <=
rx2_cache_reuse /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <=
rx2_csum_complete /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <= rx2_packets /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <=
rx2_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 188368405 ( 188,368,405) <= rx3_bytes /sec
Ethtool(enp175s0) stat: 72138 ( 72,138) <=
rx3_cache_reuse /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <=
rx3_csum_complete /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <= rx3_packets /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <=
rx3_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 186009984 ( 186,009,984) <= rx4_bytes /sec
Ethtool(enp175s0) stat: 70004 ( 70,004) <=
rx4_cache_reuse /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <=
rx4_csum_complete /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <= rx4_packets /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <=
rx4_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 198040550 ( 198,040,550) <= rx5_bytes /sec
Ethtool(enp175s0) stat: 75492 ( 75,492) <=
rx5_cache_reuse /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <=
rx5_csum_complete /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <= rx5_packets /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <=
rx5_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 182607101 ( 182,607,101) <= rx6_bytes /sec
Ethtool(enp175s0) stat: 69699 ( 69,699) <=
rx6_cache_reuse /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <=
rx6_csum_complete /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <= rx6_packets /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <=
rx6_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 174999243 ( 174,999,243) <= rx7_bytes /sec
Ethtool(enp175s0) stat: 66650 ( 66,650) <=
rx7_cache_reuse /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <=
rx7_csum_complete /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <= rx7_packets /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <=
rx7_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 204109286 ( 204,109,286) <= rx8_bytes /sec
Ethtool(enp175s0) stat: 76711 ( 76,711) <=
rx8_cache_reuse /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <=
rx8_csum_complete /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <= rx8_packets /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <=
rx8_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 183703752 ( 183,703,752) <= rx9_bytes /sec
Ethtool(enp175s0) stat: 71101 ( 71,101) <=
rx9_cache_reuse /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <=
rx9_csum_complete /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <= rx9_packets /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <=
rx9_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 2024072 ( 2,024,072) <=
rx_1024_to_1518_bytes_phy /sec
Ethtool(enp175s0) stat: 106582 ( 106,582) <=
rx_128_to_255_bytes_phy /sec
Ethtool(enp175s0) stat: 1296735 ( 1,296,735) <=
rx_1519_to_2047_bytes_phy /sec
Ethtool(enp175s0) stat: 59460 ( 59,460) <=
rx_256_to_511_bytes_phy /sec
Ethtool(enp175s0) stat: 57326 ( 57,326) <=
rx_512_to_1023_bytes_phy /sec
Ethtool(enp175s0) stat: 7159 ( 7,159) <=
rx_64_bytes_phy /sec
Ethtool(enp175s0) stat: 310730 ( 310,730) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp175s0) stat: 232 ( 232) <=
rx_broadcast_phy /sec
Ethtool(enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
Ethtool(enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
Ethtool(enp175s0) stat: 1858636 ( 1,858,636) <= rx_cache_reuse
/sec
Ethtool(enp175s0) stat: 3717060 ( 3,717,060) <=
rx_csum_complete /sec
Ethtool(enp175s0) stat: 88 ( 88) <= rx_csum_none /sec
Ethtool(enp175s0) stat: 139602 ( 139,602) <=
rx_discards_phy /sec
Ethtool(enp175s0) stat: 354 ( 354) <=
rx_multicast_phy /sec
Ethtool(enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
Ethtool(enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy
/sec
Ethtool(enp175s0) stat: 5063355121 ( 5,063,355,121) <= rx_prio0_bytes
/sec
Ethtool(enp175s0) stat: 3718759 ( 3,718,759) <=
rx_prio0_packets /sec
Ethtool(enp175s0) stat: 7193190 ( 7,193,190) <= rx_prio1_bytes
/sec
Ethtool(enp175s0) stat: 5031 ( 5,031) <=
rx_prio1_packets /sec
Ethtool(enp175s0) stat: 557 ( 557) <= rx_prio2_bytes
/sec
Ethtool(enp175s0) stat: 5 ( 5) <=
rx_prio2_packets /sec
Ethtool(enp175s0) stat: 61 ( 61) <= rx_prio3_bytes
/sec
Ethtool(enp175s0) stat: 1 ( 1) <=
rx_prio3_packets /sec
Ethtool(enp175s0) stat: 21010 ( 21,010) <= rx_prio4_bytes
/sec
Ethtool(enp175s0) stat: 39 ( 39) <=
rx_prio4_packets /sec
Ethtool(enp175s0) stat: 187 ( 187) <= rx_prio5_bytes
/sec
Ethtool(enp175s0) stat: 2 ( 2) <=
rx_prio5_packets /sec
Ethtool(enp175s0) stat: 1711 ( 1,711) <= rx_prio6_bytes
/sec
Ethtool(enp175s0) stat: 15 ( 15) <=
rx_prio6_packets /sec
Ethtool(enp175s0) stat: 19498 ( 19,498) <= rx_prio7_bytes
/sec
Ethtool(enp175s0) stat: 273 ( 273) <=
rx_prio7_packets /sec
Ethtool(enp175s0) stat: 3717148 ( 3,717,148) <=
rx_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 5737 ( 5,737) <=
rx_steer_missed_packets /sec
Ethtool(enp175s0) stat: 14573 ( 14,573) <=
rx_vport_broadcast_bytes /sec
Ethtool(enp175s0) stat: 232 ( 232) <=
rx_vport_broadcast_packets /sec
Ethtool(enp175s0) stat: 25491 ( 25,491) <=
rx_vport_multicast_bytes /sec
Ethtool(enp175s0) stat: 354 ( 354) <=
rx_vport_multicast_packets /sec
Ethtool(enp175s0) stat: 4872354516 ( 4,872,354,516) <=
rx_vport_unicast_bytes /sec
Ethtool(enp175s0) stat: 3721920 ( 3,721,920) <=
rx_vport_unicast_packets /sec
Ethtool(enp175s0) stat: 158883 ( 158,883) <=
tx0_added_vlan_packets /sec
Ethtool(enp175s0) stat: 93790423 ( 93,790,423) <= tx0_bytes /sec
Ethtool(enp175s0) stat: 158854 ( 158,854) <= tx0_cqes /sec
Ethtool(enp175s0) stat: 146499 ( 146,499) <= tx0_csum_none /sec
Ethtool(enp175s0) stat: 12384 ( 12,384) <=
tx0_csum_partial /sec
Ethtool(enp175s0) stat: 2144 ( 2,144) <= tx0_nop /sec
Ethtool(enp175s0) stat: 188173 ( 188,173) <= tx0_packets /sec
Ethtool(enp175s0) stat: 53068613 ( 53,068,613) <= tx0_tso_bytes /sec
Ethtool(enp175s0) stat: 8839 ( 8,839) <=
tx0_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx0_xmit_more /sec
Ethtool(enp175s0) stat: 165538 ( 165,538) <=
tx10_added_vlan_packets /sec
Ethtool(enp175s0) stat: 102395057 ( 102,395,057) <= tx10_bytes /sec
Ethtool(enp175s0) stat: 165303 ( 165,303) <= tx10_cqes /sec
Ethtool(enp175s0) stat: 151089 ( 151,089) <= tx10_csum_none
/sec
Ethtool(enp175s0) stat: 14448 ( 14,448) <=
tx10_csum_partial /sec
Ethtool(enp175s0) stat: 2391 ( 2,391) <= tx10_nop /sec
Ethtool(enp175s0) stat: 198798 ( 198,798) <= tx10_packets /sec
Ethtool(enp175s0) stat: 58951449 ( 58,951,449) <= tx10_tso_bytes
/sec
Ethtool(enp175s0) stat: 8987 ( 8,987) <=
tx10_tso_packets /sec
Ethtool(enp175s0) stat: 234 ( 234) <= tx10_xmit_more
/sec
Ethtool(enp175s0) stat: 166402 ( 166,402) <=
tx11_added_vlan_packets /sec
Ethtool(enp175s0) stat: 98591304 ( 98,591,304) <= tx11_bytes /sec
Ethtool(enp175s0) stat: 166384 ( 166,384) <= tx11_cqes /sec
Ethtool(enp175s0) stat: 152456 ( 152,456) <= tx11_csum_none
/sec
Ethtool(enp175s0) stat: 13946 ( 13,946) <=
tx11_csum_partial /sec
Ethtool(enp175s0) stat: 2386 ( 2,386) <= tx11_nop /sec
Ethtool(enp175s0) stat: 201615 ( 201,615) <= tx11_packets /sec
Ethtool(enp175s0) stat: 63844660 ( 63,844,660) <= tx11_tso_bytes
/sec
Ethtool(enp175s0) stat: 10515 ( 10,515) <=
tx11_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx11_xmit_more
/sec
Ethtool(enp175s0) stat: 156312 ( 156,312) <=
tx12_added_vlan_packets /sec
Ethtool(enp175s0) stat: 97068537 ( 97,068,537) <= tx12_bytes /sec
Ethtool(enp175s0) stat: 156302 ( 156,302) <= tx12_cqes /sec
Ethtool(enp175s0) stat: 142468 ( 142,468) <= tx12_csum_none
/sec
Ethtool(enp175s0) stat: 13844 ( 13,844) <=
tx12_csum_partial /sec
Ethtool(enp175s0) stat: 2278 ( 2,278) <= tx12_nop /sec
Ethtool(enp175s0) stat: 187535 ( 187,535) <= tx12_packets /sec
Ethtool(enp175s0) stat: 58368798 ( 58,368,798) <= tx12_tso_bytes
/sec
Ethtool(enp175s0) stat: 10398 ( 10,398) <=
tx12_tso_packets /sec
Ethtool(enp175s0) stat: 10 ( 10) <= tx12_xmit_more
/sec
Ethtool(enp175s0) stat: 161768 ( 161,768) <=
tx13_added_vlan_packets /sec
Ethtool(enp175s0) stat: 120232518 ( 120,232,518) <= tx13_bytes /sec
Ethtool(enp175s0) stat: 161584 ( 161,584) <= tx13_cqes /sec
Ethtool(enp175s0) stat: 144388 ( 144,388) <= tx13_csum_none
/sec
Ethtool(enp175s0) stat: 17380 ( 17,380) <=
tx13_csum_partial /sec
Ethtool(enp175s0) stat: 2425 ( 2,425) <= tx13_nop /sec
Ethtool(enp175s0) stat: 202823 ( 202,823) <= tx13_packets /sec
Ethtool(enp175s0) stat: 72804507 ( 72,804,507) <= tx13_tso_bytes
/sec
Ethtool(enp175s0) stat: 10865 ( 10,865) <=
tx13_tso_packets /sec
Ethtool(enp175s0) stat: 185 ( 185) <= tx13_xmit_more
/sec
Ethtool(enp175s0) stat: 165762 ( 165,762) <=
tx14_added_vlan_packets /sec
Ethtool(enp175s0) stat: 99271622 ( 99,271,622) <= tx14_bytes /sec
Ethtool(enp175s0) stat: 165688 ( 165,688) <= tx14_cqes /sec
Ethtool(enp175s0) stat: 153195 ( 153,195) <= tx14_csum_none
/sec
Ethtool(enp175s0) stat: 12566 ( 12,566) <=
tx14_csum_partial /sec
Ethtool(enp175s0) stat: 2195 ( 2,195) <= tx14_nop /sec
Ethtool(enp175s0) stat: 195504 ( 195,504) <= tx14_packets /sec
Ethtool(enp175s0) stat: 53717277 ( 53,717,277) <= tx14_tso_bytes
/sec
Ethtool(enp175s0) stat: 8743 ( 8,743) <=
tx14_tso_packets /sec
Ethtool(enp175s0) stat: 32 ( 32) <= tx14_xmit_more
/sec
Ethtool(enp175s0) stat: 162803 ( 162,803) <=
tx15_added_vlan_packets /sec
Ethtool(enp175s0) stat: 105591893 ( 105,591,893) <= tx15_bytes /sec
Ethtool(enp175s0) stat: 162673 ( 162,673) <= tx15_cqes /sec
Ethtool(enp175s0) stat: 147080 ( 147,080) <= tx15_csum_none
/sec
Ethtool(enp175s0) stat: 15723 ( 15,723) <=
tx15_csum_partial /sec
Ethtool(enp175s0) stat: 2355 ( 2,355) <= tx15_nop /sec
Ethtool(enp175s0) stat: 198282 ( 198,282) <= tx15_packets /sec
Ethtool(enp175s0) stat: 64278573 ( 64,278,573) <= tx15_tso_bytes
/sec
Ethtool(enp175s0) stat: 10704 ( 10,704) <=
tx15_tso_packets /sec
Ethtool(enp175s0) stat: 183 ( 183) <= tx15_xmit_more
/sec
Ethtool(enp175s0) stat: 125282 ( 125,282) <=
tx16_added_vlan_packets /sec
Ethtool(enp175s0) stat: 81835815 ( 81,835,815) <= tx16_bytes /sec
Ethtool(enp175s0) stat: 125264 ( 125,264) <= tx16_cqes /sec
Ethtool(enp175s0) stat: 113284 ( 113,284) <= tx16_csum_none
/sec
Ethtool(enp175s0) stat: 11998 ( 11,998) <=
tx16_csum_partial /sec
Ethtool(enp175s0) stat: 1773 ( 1,773) <= tx16_nop /sec
Ethtool(enp175s0) stat: 150812 ( 150,812) <= tx16_packets /sec
Ethtool(enp175s0) stat: 46027767 ( 46,027,767) <= tx16_tso_bytes
/sec
Ethtool(enp175s0) stat: 7361 ( 7,361) <=
tx16_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx16_xmit_more
/sec
Ethtool(enp175s0) stat: 131865 ( 131,865) <=
tx17_added_vlan_packets /sec
Ethtool(enp175s0) stat: 89213555 ( 89,213,555) <= tx17_bytes /sec
Ethtool(enp175s0) stat: 131409 ( 131,409) <= tx17_cqes /sec
Ethtool(enp175s0) stat: 117199 ( 117,199) <= tx17_csum_none
/sec
Ethtool(enp175s0) stat: 14665 ( 14,665) <=
tx17_csum_partial /sec
Ethtool(enp175s0) stat: 1933 ( 1,933) <= tx17_nop /sec
Ethtool(enp175s0) stat: 161576 ( 161,576) <= tx17_packets /sec
Ethtool(enp175s0) stat: 53682297 ( 53,682,297) <= tx17_tso_bytes
/sec
Ethtool(enp175s0) stat: 8825 ( 8,825) <=
tx17_tso_packets /sec
Ethtool(enp175s0) stat: 459 ( 459) <= tx17_xmit_more
/sec
Ethtool(enp175s0) stat: 122492 ( 122,492) <=
tx18_added_vlan_packets /sec
Ethtool(enp175s0) stat: 87373476 ( 87,373,476) <= tx18_bytes /sec
Ethtool(enp175s0) stat: 122138 ( 122,138) <= tx18_cqes /sec
Ethtool(enp175s0) stat: 109037 ( 109,037) <= tx18_csum_none
/sec
Ethtool(enp175s0) stat: 13455 ( 13,455) <=
tx18_csum_partial /sec
Ethtool(enp175s0) stat: 1933 ( 1,933) <= tx18_nop /sec
Ethtool(enp175s0) stat: 152821 ( 152,821) <= tx18_packets /sec
Ethtool(enp175s0) stat: 54163102 ( 54,163,102) <= tx18_tso_bytes
/sec
Ethtool(enp175s0) stat: 8674 ( 8,674) <=
tx18_tso_packets /sec
Ethtool(enp175s0) stat: 354 ( 354) <= tx18_xmit_more
/sec
Ethtool(enp175s0) stat: 119294 ( 119,294) <=
tx19_added_vlan_packets /sec
Ethtool(enp175s0) stat: 67719609 ( 67,719,609) <= tx19_bytes /sec
Ethtool(enp175s0) stat: 119262 ( 119,262) <= tx19_cqes /sec
Ethtool(enp175s0) stat: 108591 ( 108,591) <= tx19_csum_none
/sec
Ethtool(enp175s0) stat: 10703 ( 10,703) <=
tx19_csum_partial /sec
Ethtool(enp175s0) stat: 1551 ( 1,551) <= tx19_nop /sec
Ethtool(enp175s0) stat: 141344 ( 141,344) <= tx19_packets /sec
Ethtool(enp175s0) stat: 39539778 ( 39,539,778) <= tx19_tso_bytes
/sec
Ethtool(enp175s0) stat: 6310 ( 6,310) <=
tx19_tso_packets /sec
Ethtool(enp175s0) stat: 31 ( 31) <= tx19_xmit_more
/sec
Ethtool(enp175s0) stat: 157094 ( 157,094) <=
tx1_added_vlan_packets /sec
Ethtool(enp175s0) stat: 93505806 ( 93,505,806) <= tx1_bytes /sec
Ethtool(enp175s0) stat: 156935 ( 156,935) <= tx1_cqes /sec
Ethtool(enp175s0) stat: 144805 ( 144,805) <= tx1_csum_none /sec
Ethtool(enp175s0) stat: 12289 ( 12,289) <=
tx1_csum_partial /sec
Ethtool(enp175s0) stat: 2201 ( 2,201) <= tx1_nop /sec
Ethtool(enp175s0) stat: 184561 ( 184,561) <= tx1_packets /sec
Ethtool(enp175s0) stat: 50513729 ( 50,513,729) <= tx1_tso_bytes /sec
Ethtool(enp175s0) stat: 8699 ( 8,699) <=
tx1_tso_packets /sec
Ethtool(enp175s0) stat: 159 ( 159) <= tx1_xmit_more /sec
Ethtool(enp175s0) stat: 134411 ( 134,411) <=
tx20_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88898658 ( 88,898,658) <= tx20_bytes /sec
Ethtool(enp175s0) stat: 134221 ( 134,221) <= tx20_cqes /sec
Ethtool(enp175s0) stat: 120787 ( 120,787) <= tx20_csum_none
/sec
Ethtool(enp175s0) stat: 13624 ( 13,624) <=
tx20_csum_partial /sec
Ethtool(enp175s0) stat: 2064 ( 2,064) <= tx20_nop /sec
Ethtool(enp175s0) stat: 165064 ( 165,064) <= tx20_packets /sec
Ethtool(enp175s0) stat: 54707259 ( 54,707,259) <= tx20_tso_bytes
/sec
Ethtool(enp175s0) stat: 8769 ( 8,769) <=
tx20_tso_packets /sec
Ethtool(enp175s0) stat: 188 ( 188) <= tx20_xmit_more
/sec
Ethtool(enp175s0) stat: 127029 ( 127,029) <=
tx21_added_vlan_packets /sec
Ethtool(enp175s0) stat: 98012630 ( 98,012,630) <= tx21_bytes /sec
Ethtool(enp175s0) stat: 126990 ( 126,990) <= tx21_cqes /sec
Ethtool(enp175s0) stat: 111553 ( 111,553) <= tx21_csum_none
/sec
Ethtool(enp175s0) stat: 15476 ( 15,476) <=
tx21_csum_partial /sec
Ethtool(enp175s0) stat: 2002 ( 2,002) <= tx21_nop /sec
Ethtool(enp175s0) stat: 159688 ( 159,688) <= tx21_packets /sec
Ethtool(enp175s0) stat: 59637304 ( 59,637,304) <= tx21_tso_bytes
/sec
Ethtool(enp175s0) stat: 9988 ( 9,988) <=
tx21_tso_packets /sec
Ethtool(enp175s0) stat: 39 ( 39) <= tx21_xmit_more
/sec
Ethtool(enp175s0) stat: 122610 ( 122,610) <=
tx22_added_vlan_packets /sec
Ethtool(enp175s0) stat: 70972052 ( 70,972,052) <= tx22_bytes /sec
Ethtool(enp175s0) stat: 122600 ( 122,600) <= tx22_cqes /sec
Ethtool(enp175s0) stat: 111526 ( 111,526) <= tx22_csum_none
/sec
Ethtool(enp175s0) stat: 11085 ( 11,085) <=
tx22_csum_partial /sec
Ethtool(enp175s0) stat: 1544 ( 1,544) <= tx22_nop /sec
Ethtool(enp175s0) stat: 144874 ( 144,874) <= tx22_packets /sec
Ethtool(enp175s0) stat: 40175057 ( 40,175,057) <= tx22_tso_bytes
/sec
Ethtool(enp175s0) stat: 6517 ( 6,517) <=
tx22_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx22_xmit_more
/sec
Ethtool(enp175s0) stat: 126809 ( 126,809) <=
tx23_added_vlan_packets /sec
Ethtool(enp175s0) stat: 76906314 ( 76,906,314) <= tx23_bytes /sec
Ethtool(enp175s0) stat: 126791 ( 126,791) <= tx23_cqes /sec
Ethtool(enp175s0) stat: 116656 ( 116,656) <= tx23_csum_none
/sec
Ethtool(enp175s0) stat: 10153 ( 10,153) <=
tx23_csum_partial /sec
Ethtool(enp175s0) stat: 1686 ( 1,686) <= tx23_nop /sec
Ethtool(enp175s0) stat: 148551 ( 148,551) <= tx23_packets /sec
Ethtool(enp175s0) stat: 39913764 ( 39,913,764) <= tx23_tso_bytes
/sec
Ethtool(enp175s0) stat: 6978 ( 6,978) <=
tx23_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx23_xmit_more
/sec
Ethtool(enp175s0) stat: 133910 ( 133,910) <=
tx24_added_vlan_packets /sec
Ethtool(enp175s0) stat: 69609913 ( 69,609,913) <= tx24_bytes /sec
Ethtool(enp175s0) stat: 133890 ( 133,890) <= tx24_cqes /sec
Ethtool(enp175s0) stat: 124462 ( 124,462) <= tx24_csum_none
/sec
Ethtool(enp175s0) stat: 9448 ( 9,448) <=
tx24_csum_partial /sec
Ethtool(enp175s0) stat: 1639 ( 1,639) <= tx24_nop /sec
Ethtool(enp175s0) stat: 154475 ( 154,475) <= tx24_packets /sec
Ethtool(enp175s0) stat: 37456148 ( 37,456,148) <= tx24_tso_bytes
/sec
Ethtool(enp175s0) stat: 6247 ( 6,247) <=
tx24_tso_packets /sec
Ethtool(enp175s0) stat: 20 ( 20) <= tx24_xmit_more
/sec
Ethtool(enp175s0) stat: 118528 ( 118,528) <=
tx25_added_vlan_packets /sec
Ethtool(enp175s0) stat: 62435525 ( 62,435,525) <= tx25_bytes /sec
Ethtool(enp175s0) stat: 118471 ( 118,471) <= tx25_cqes /sec
Ethtool(enp175s0) stat: 108887 ( 108,887) <= tx25_csum_none
/sec
Ethtool(enp175s0) stat: 9640 ( 9,640) <=
tx25_csum_partial /sec
Ethtool(enp175s0) stat: 1528 ( 1,528) <= tx25_nop /sec
Ethtool(enp175s0) stat: 138748 ( 138,748) <= tx25_packets /sec
Ethtool(enp175s0) stat: 36592045 ( 36,592,045) <= tx25_tso_bytes
/sec
Ethtool(enp175s0) stat: 5993 ( 5,993) <=
tx25_tso_packets /sec
Ethtool(enp175s0) stat: 44 ( 44) <= tx25_xmit_more
/sec
Ethtool(enp175s0) stat: 119890 ( 119,890) <=
tx26_added_vlan_packets /sec
Ethtool(enp175s0) stat: 76743929 ( 76,743,929) <= tx26_bytes /sec
Ethtool(enp175s0) stat: 119873 ( 119,873) <= tx26_cqes /sec
Ethtool(enp175s0) stat: 108181 ( 108,181) <= tx26_csum_none
/sec
Ethtool(enp175s0) stat: 11709 ( 11,709) <=
tx26_csum_partial /sec
Ethtool(enp175s0) stat: 1690 ( 1,690) <= tx26_nop /sec
Ethtool(enp175s0) stat: 144495 ( 144,495) <= tx26_packets /sec
Ethtool(enp175s0) stat: 43922304 ( 43,922,304) <= tx26_tso_bytes
/sec
Ethtool(enp175s0) stat: 7043 ( 7,043) <=
tx26_tso_packets /sec
Ethtool(enp175s0) stat: 17 ( 17) <= tx26_xmit_more
/sec
Ethtool(enp175s0) stat: 130825 ( 130,825) <=
tx27_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88723162 ( 88,723,162) <= tx27_bytes /sec
Ethtool(enp175s0) stat: 130769 ( 130,769) <= tx27_cqes /sec
Ethtool(enp175s0) stat: 116555 ( 116,555) <= tx27_csum_none
/sec
Ethtool(enp175s0) stat: 14270 ( 14,270) <=
tx27_csum_partial /sec
Ethtool(enp175s0) stat: 2042 ( 2,042) <= tx27_nop /sec
Ethtool(enp175s0) stat: 161228 ( 161,228) <= tx27_packets /sec
Ethtool(enp175s0) stat: 55637023 ( 55,637,023) <= tx27_tso_bytes
/sec
Ethtool(enp175s0) stat: 9707 ( 9,707) <=
tx27_tso_packets /sec
Ethtool(enp175s0) stat: 71 ( 71) <= tx27_xmit_more
/sec
Ethtool(enp175s0) stat: 166973 ( 166,973) <=
tx2_added_vlan_packets /sec
Ethtool(enp175s0) stat: 103659503 ( 103,659,503) <= tx2_bytes /sec
Ethtool(enp175s0) stat: 166941 ( 166,941) <= tx2_cqes /sec
Ethtool(enp175s0) stat: 151389 ( 151,389) <= tx2_csum_none /sec
Ethtool(enp175s0) stat: 15585 ( 15,585) <=
tx2_csum_partial /sec
Ethtool(enp175s0) stat: 2455 ( 2,455) <= tx2_nop /sec
Ethtool(enp175s0) stat: 201854 ( 201,854) <= tx2_packets /sec
Ethtool(enp175s0) stat: 65384298 ( 65,384,298) <= tx2_tso_bytes /sec
Ethtool(enp175s0) stat: 11809 ( 11,809) <=
tx2_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx2_xmit_more /sec
Ethtool(enp175s0) stat: 172353 ( 172,353) <=
tx3_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88248541 ( 88,248,541) <= tx3_bytes /sec
Ethtool(enp175s0) stat: 172277 ( 172,277) <= tx3_cqes /sec
Ethtool(enp175s0) stat: 160714 ( 160,714) <= tx3_csum_none /sec
Ethtool(enp175s0) stat: 11639 ( 11,639) <=
tx3_csum_partial /sec
Ethtool(enp175s0) stat: 2258 ( 2,258) <= tx3_nop /sec
Ethtool(enp175s0) stat: 199853 ( 199,853) <= tx3_packets /sec
Ethtool(enp175s0) stat: 49776463 ( 49,776,463) <= tx3_tso_bytes /sec
Ethtool(enp175s0) stat: 8353 ( 8,353) <=
tx3_tso_packets /sec
Ethtool(enp175s0) stat: 76 ( 76) <= tx3_xmit_more /sec
Ethtool(enp175s0) stat: 157527 ( 157,527) <=
tx4_added_vlan_packets /sec
Ethtool(enp175s0) stat: 110770979 ( 110,770,979) <= tx4_bytes /sec
Ethtool(enp175s0) stat: 157492 ( 157,492) <= tx4_cqes /sec
Ethtool(enp175s0) stat: 141858 ( 141,858) <= tx4_csum_none /sec
Ethtool(enp175s0) stat: 15670 ( 15,670) <=
tx4_csum_partial /sec
Ethtool(enp175s0) stat: 2320 ( 2,320) <= tx4_nop /sec
Ethtool(enp175s0) stat: 192429 ( 192,429) <= tx4_packets /sec
Ethtool(enp175s0) stat: 64689503 ( 64,689,503) <= tx4_tso_bytes /sec
Ethtool(enp175s0) stat: 11367 ( 11,367) <=
tx4_tso_packets /sec
Ethtool(enp175s0) stat: 35 ( 35) <= tx4_xmit_more /sec
Ethtool(enp175s0) stat: 169077 ( 169,077) <=
tx5_added_vlan_packets /sec
Ethtool(enp175s0) stat: 121536690 ( 121,536,690) <= tx5_bytes /sec
Ethtool(enp175s0) stat: 168906 ( 168,906) <= tx5_cqes /sec
Ethtool(enp175s0) stat: 150099 ( 150,099) <= tx5_csum_none /sec
Ethtool(enp175s0) stat: 18978 ( 18,978) <=
tx5_csum_partial /sec
Ethtool(enp175s0) stat: 2678 ( 2,678) <= tx5_nop /sec
Ethtool(enp175s0) stat: 210733 ( 210,733) <= tx5_packets /sec
Ethtool(enp175s0) stat: 76448346 ( 76,448,346) <= tx5_tso_bytes /sec
Ethtool(enp175s0) stat: 13238 ( 13,238) <=
tx5_tso_packets /sec
Ethtool(enp175s0) stat: 171 ( 171) <= tx5_xmit_more /sec
Ethtool(enp175s0) stat: 156881 ( 156,881) <=
tx6_added_vlan_packets /sec
Ethtool(enp175s0) stat: 85752393 ( 85,752,393) <= tx6_bytes /sec
Ethtool(enp175s0) stat: 156859 ( 156,859) <= tx6_cqes /sec
Ethtool(enp175s0) stat: 144843 ( 144,843) <= tx6_csum_none /sec
Ethtool(enp175s0) stat: 12038 ( 12,038) <=
tx6_csum_partial /sec
Ethtool(enp175s0) stat: 2034 ( 2,034) <= tx6_nop /sec
Ethtool(enp175s0) stat: 181315 ( 181,315) <= tx6_packets /sec
Ethtool(enp175s0) stat: 44625307 ( 44,625,307) <= tx6_tso_bytes /sec
Ethtool(enp175s0) stat: 7752 ( 7,752) <=
tx6_tso_packets /sec
Ethtool(enp175s0) stat: 24 ( 24) <= tx6_xmit_more /sec
Ethtool(enp175s0) stat: 157744 ( 157,744) <=
tx7_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88795890 ( 88,795,890) <= tx7_bytes /sec
Ethtool(enp175s0) stat: 157666 ( 157,666) <= tx7_cqes /sec
Ethtool(enp175s0) stat: 146023 ( 146,023) <= tx7_csum_none /sec
Ethtool(enp175s0) stat: 11720 ( 11,720) <=
tx7_csum_partial /sec
Ethtool(enp175s0) stat: 2190 ( 2,190) <= tx7_nop /sec
Ethtool(enp175s0) stat: 184845 ( 184,845) <= tx7_packets /sec
Ethtool(enp175s0) stat: 48986735 ( 48,986,735) <= tx7_tso_bytes /sec
Ethtool(enp175s0) stat: 8401 ( 8,401) <=
tx7_tso_packets /sec
Ethtool(enp175s0) stat: 78 ( 78) <= tx7_xmit_more /sec
Ethtool(enp175s0) stat: 165190 ( 165,190) <=
tx8_added_vlan_packets /sec
Ethtool(enp175s0) stat: 108303283 ( 108,303,283) <= tx8_bytes /sec
Ethtool(enp175s0) stat: 165145 ( 165,145) <= tx8_cqes /sec
Ethtool(enp175s0) stat: 150047 ( 150,047) <= tx8_csum_none /sec
Ethtool(enp175s0) stat: 15143 ( 15,143) <=
tx8_csum_partial /sec
Ethtool(enp175s0) stat: 2497 ( 2,497) <= tx8_nop /sec
Ethtool(enp175s0) stat: 201615 ( 201,615) <= tx8_packets /sec
Ethtool(enp175s0) stat: 67575199 ( 67,575,199) <= tx8_tso_bytes /sec
Ethtool(enp175s0) stat: 11828 ( 11,828) <=
tx8_tso_packets /sec
Ethtool(enp175s0) stat: 45 ( 45) <= tx8_xmit_more /sec
Ethtool(enp175s0) stat: 168256 ( 168,256) <=
tx9_added_vlan_packets /sec
Ethtool(enp175s0) stat: 115981227 ( 115,981,227) <= tx9_bytes /sec
Ethtool(enp175s0) stat: 168225 ( 168,225) <= tx9_cqes /sec
Ethtool(enp175s0) stat: 147980 ( 147,980) <= tx9_csum_none /sec
Ethtool(enp175s0) stat: 20275 ( 20,275) <=
tx9_csum_partial /sec
Ethtool(enp175s0) stat: 2522 ( 2,522) <= tx9_nop /sec
Ethtool(enp175s0) stat: 203398 ( 203,398) <= tx9_packets /sec
Ethtool(enp175s0) stat: 63891360 ( 63,891,360) <= tx9_tso_bytes /sec
Ethtool(enp175s0) stat: 10610 ( 10,610) <=
tx9_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx9_xmit_more /sec
Ethtool(enp175s0) stat: 4121491 ( 4,121,491) <=
tx_added_vlan_packets /sec
Ethtool(enp175s0) stat: 1 ( 1) <=
tx_broadcast_phy /sec
Ethtool(enp175s0) stat: 2591851137 ( 2,591,851,137) <= tx_bytes /sec
Ethtool(enp175s0) stat: 2626884154 ( 2,626,884,154) <= tx_bytes_phy /sec
Ethtool(enp175s0) stat: 4118829 ( 4,118,829) <= tx_cqes /sec
Ethtool(enp175s0) stat: 3741682 ( 3,741,682) <= tx_csum_none /sec
Ethtool(enp175s0) stat: 379810 ( 379,810) <=
tx_csum_partial /sec
Ethtool(enp175s0) stat: 58718 ( 58,718) <= tx_nop /sec
Ethtool(enp175s0) stat: 4956995 ( 4,956,995) <= tx_packets /sec
Ethtool(enp175s0) stat: 4956579 ( 4,956,579) <= tx_packets_phy
/sec
Ethtool(enp175s0) stat: 2627152852 ( 2,627,152,852) <= tx_prio0_bytes
/sec
Ethtool(enp175s0) stat: 4957649 ( 4,957,649) <=
tx_prio0_packets /sec
Ethtool(enp175s0) stat: 1518383905 ( 1,518,383,905) <= tx_tso_bytes /sec
Ethtool(enp175s0) stat: 253526 ( 253,526) <= tx_tso_packets
/sec
Ethtool(enp175s0) stat: 57 ( 57) <=
tx_vport_broadcast_bytes /sec
Ethtool(enp175s0) stat: 1 ( 1) <=
tx_vport_broadcast_packets /sec
Ethtool(enp175s0) stat: 2607061269 ( 2,607,061,269) <=
tx_vport_unicast_bytes /sec
Ethtool(enp175s0) stat: 4956492 ( 4,956,492) <=
tx_vport_unicast_packets /sec
Ethtool(enp175s0) stat: 2630 ( 2,630) <= tx_xmit_more /sec
Ethtool(enp216s0) stat: 6923 ( 6,923) <= ch0_arm /sec
Ethtool(enp216s0) stat: 8367 ( 8,367) <= ch0_events /sec
Ethtool(enp216s0) stat: 9797 ( 9,797) <= ch0_poll /sec
Ethtool(enp216s0) stat: 7703 ( 7,703) <= ch10_arm /sec
Ethtool(enp216s0) stat: 9253 ( 9,253) <= ch10_events /sec
Ethtool(enp216s0) stat: 10807 ( 10,807) <= ch10_poll /sec
Ethtool(enp216s0) stat: 7886 ( 7,886) <= ch11_arm /sec
Ethtool(enp216s0) stat: 9391 ( 9,391) <= ch11_events /sec
Ethtool(enp216s0) stat: 10896 ( 10,896) <= ch11_poll /sec
Ethtool(enp216s0) stat: 9793 ( 9,793) <= ch12_arm /sec
Ethtool(enp216s0) stat: 11299 ( 11,299) <= ch12_events /sec
Ethtool(enp216s0) stat: 12637 ( 12,637) <= ch12_poll /sec
Ethtool(enp216s0) stat: 9119 ( 9,119) <= ch13_arm /sec
Ethtool(enp216s0) stat: 10671 ( 10,671) <= ch13_events /sec
Ethtool(enp216s0) stat: 12205 ( 12,205) <= ch13_poll /sec
Ethtool(enp216s0) stat: 8784 ( 8,784) <= ch14_arm /sec
Ethtool(enp216s0) stat: 10189 ( 10,189) <= ch14_events /sec
Ethtool(enp216s0) stat: 11565 ( 11,565) <= ch14_poll /sec
Ethtool(enp216s0) stat: 9248 ( 9,248) <= ch15_arm /sec
Ethtool(enp216s0) stat: 10739 ( 10,739) <= ch15_events /sec
Ethtool(enp216s0) stat: 12238 ( 12,238) <= ch15_poll /sec
Ethtool(enp216s0) stat: 5201 ( 5,201) <= ch16_arm /sec
Ethtool(enp216s0) stat: 6554 ( 6,554) <= ch16_events /sec
Ethtool(enp216s0) stat: 7667 ( 7,667) <= ch16_poll /sec
Ethtool(enp216s0) stat: 7799 ( 7,799) <= ch17_arm /sec
Ethtool(enp216s0) stat: 9120 ( 9,120) <= ch17_events /sec
Ethtool(enp216s0) stat: 10194 ( 10,194) <= ch17_poll /sec
Ethtool(enp216s0) stat: 6889 ( 6,889) <= ch18_arm /sec
Ethtool(enp216s0) stat: 8266 ( 8,266) <= ch18_events /sec
Ethtool(enp216s0) stat: 9332 ( 9,332) <= ch18_poll /sec
Ethtool(enp216s0) stat: 5494 ( 5,494) <= ch19_arm /sec
Ethtool(enp216s0) stat: 6813 ( 6,813) <= ch19_events /sec
Ethtool(enp216s0) stat: 7837 ( 7,837) <= ch19_poll /sec
Ethtool(enp216s0) stat: 6672 ( 6,672) <= ch1_arm /sec
Ethtool(enp216s0) stat: 8124 ( 8,124) <= ch1_events /sec
Ethtool(enp216s0) stat: 9604 ( 9,604) <= ch1_poll /sec
Ethtool(enp216s0) stat: 7705 ( 7,705) <= ch20_arm /sec
Ethtool(enp216s0) stat: 9102 ( 9,102) <= ch20_events /sec
Ethtool(enp216s0) stat: 10287 ( 10,287) <= ch20_poll /sec
Ethtool(enp216s0) stat: 5929 ( 5,929) <= ch21_arm /sec
Ethtool(enp216s0) stat: 7333 ( 7,333) <= ch21_events /sec
Ethtool(enp216s0) stat: 8463 ( 8,463) <= ch21_poll /sec
Ethtool(enp216s0) stat: 5495 ( 5,495) <= ch22_arm /sec
Ethtool(enp216s0) stat: 6813 ( 6,813) <= ch22_events /sec
Ethtool(enp216s0) stat: 7843 ( 7,843) <= ch22_poll /sec
Ethtool(enp216s0) stat: 7091 ( 7,091) <= ch23_arm /sec
Ethtool(enp216s0) stat: 8367 ( 8,367) <= ch23_events /sec
Ethtool(enp216s0) stat: 9344 ( 9,344) <= ch23_poll /sec
Ethtool(enp216s0) stat: 5481 ( 5,481) <= ch24_arm /sec
Ethtool(enp216s0) stat: 6879 ( 6,879) <= ch24_events /sec
Ethtool(enp216s0) stat: 7995 ( 7,995) <= ch24_poll /sec
Ethtool(enp216s0) stat: 5642 ( 5,642) <= ch25_arm /sec
Ethtool(enp216s0) stat: 6959 ( 6,959) <= ch25_events /sec
Ethtool(enp216s0) stat: 7927 ( 7,927) <= ch25_poll /sec
Ethtool(enp216s0) stat: 5289 ( 5,289) <= ch26_arm /sec
Ethtool(enp216s0) stat: 6643 ( 6,643) <= ch26_events /sec
Ethtool(enp216s0) stat: 7691 ( 7,691) <= ch26_poll /sec
Ethtool(enp216s0) stat: 7313 ( 7,313) <= ch27_arm /sec
Ethtool(enp216s0) stat: 8719 ( 8,719) <= ch27_events /sec
Ethtool(enp216s0) stat: 9876 ( 9,876) <= ch27_poll /sec
Ethtool(enp216s0) stat: 7791 ( 7,791) <= ch2_arm /sec
Ethtool(enp216s0) stat: 9328 ( 9,328) <= ch2_events /sec
Ethtool(enp216s0) stat: 10838 ( 10,838) <= ch2_poll /sec
Ethtool(enp216s0) stat: 9595 ( 9,595) <= ch3_arm /sec
Ethtool(enp216s0) stat: 11015 ( 11,015) <= ch3_events /sec
Ethtool(enp216s0) stat: 12402 ( 12,402) <= ch3_poll /sec
Ethtool(enp216s0) stat: 9653 ( 9,653) <= ch4_arm /sec
Ethtool(enp216s0) stat: 11163 ( 11,163) <= ch4_events /sec
Ethtool(enp216s0) stat: 12553 ( 12,553) <= ch4_poll /sec
Ethtool(enp216s0) stat: 10269 ( 10,269) <= ch5_arm /sec
Ethtool(enp216s0) stat: 11727 ( 11,727) <= ch5_events /sec
Ethtool(enp216s0) stat: 13160 ( 13,160) <= ch5_poll /sec
Ethtool(enp216s0) stat: 8806 ( 8,806) <= ch6_arm /sec
Ethtool(enp216s0) stat: 10191 ( 10,191) <= ch6_events /sec
Ethtool(enp216s0) stat: 11431 ( 11,431) <= ch6_poll /sec
Ethtool(enp216s0) stat: 6866 ( 6,866) <= ch7_arm /sec
Ethtool(enp216s0) stat: 8412 ( 8,412) <= ch7_events /sec
Ethtool(enp216s0) stat: 9815 ( 9,815) <= ch7_poll /sec
Ethtool(enp216s0) stat: 10533 ( 10,533) <= ch8_arm /sec
Ethtool(enp216s0) stat: 11945 ( 11,945) <= ch8_events /sec
Ethtool(enp216s0) stat: 13286 ( 13,286) <= ch8_poll /sec
Ethtool(enp216s0) stat: 7126 ( 7,126) <= ch9_arm /sec
Ethtool(enp216s0) stat: 8643 ( 8,643) <= ch9_events /sec
Ethtool(enp216s0) stat: 10210 ( 10,210) <= ch9_poll /sec
Ethtool(enp216s0) stat: 212206 ( 212,206) <= ch_arm /sec
Ethtool(enp216s0) stat: 252118 ( 252,118) <= ch_events /sec
Ethtool(enp216s0) stat: 288009 ( 288,009) <= ch_poll /sec
Ethtool(enp216s0) stat: 93281134 ( 93,281,134) <= rx0_bytes /sec
Ethtool(enp216s0) stat: 94200 ( 94,200) <=
rx0_cache_reuse /sec
Ethtool(enp216s0) stat: 188327 ( 188,327) <=
rx0_csum_complete /sec
Ethtool(enp216s0) stat: 7 ( 7) <= rx0_csum_none /sec
Ethtool(enp216s0) stat: 188334 ( 188,334) <= rx0_packets /sec
Ethtool(enp216s0) stat: 188334 ( 188,334) <=
rx0_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 102443052 ( 102,443,052) <= rx10_bytes /sec
Ethtool(enp216s0) stat: 99816 ( 99,816) <=
rx10_cache_reuse /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <=
rx10_csum_complete /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <= rx10_packets /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <=
rx10_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 97655592 ( 97,655,592) <= rx11_bytes /sec
Ethtool(enp216s0) stat: 100854 ( 100,854) <=
rx11_cache_reuse /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <=
rx11_csum_complete /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <= rx11_packets /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <=
rx11_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 97076715 ( 97,076,715) <= rx12_bytes /sec
Ethtool(enp216s0) stat: 94078 ( 94,078) <=
rx12_cache_reuse /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <=
rx12_csum_complete /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <= rx12_packets /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <=
rx12_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 120547921 ( 120,547,921) <= rx13_bytes /sec
Ethtool(enp216s0) stat: 101709 ( 101,709) <=
rx13_cache_reuse /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <=
rx13_csum_complete /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <= rx13_packets /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <=
rx13_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 98367936 ( 98,367,936) <= rx14_bytes /sec
Ethtool(enp216s0) stat: 97741 ( 97,741) <=
rx14_cache_reuse /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <=
rx14_csum_complete /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <= rx14_packets /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <=
rx14_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 106726542 ( 106,726,542) <= rx15_bytes /sec
Ethtool(enp216s0) stat: 99694 ( 99,694) <=
rx15_cache_reuse /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <=
rx15_csum_complete /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <= rx15_packets /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <=
rx15_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 81928969 ( 81,928,969) <= rx16_bytes /sec
Ethtool(enp216s0) stat: 75580 ( 75,580) <=
rx16_cache_reuse /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <=
rx16_csum_complete /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <= rx16_packets /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <=
rx16_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 90509227 ( 90,509,227) <= rx17_bytes /sec
Ethtool(enp216s0) stat: 81196 ( 81,196) <=
rx17_cache_reuse /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <=
rx17_csum_complete /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <= rx17_packets /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <=
rx17_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 87854385 ( 87,854,385) <= rx18_bytes /sec
Ethtool(enp216s0) stat: 76923 ( 76,923) <=
rx18_cache_reuse /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <=
rx18_csum_complete /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <= rx18_packets /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <=
rx18_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 67849725 ( 67,849,725) <= rx19_bytes /sec
Ethtool(enp216s0) stat: 71001 ( 71,001) <=
rx19_cache_reuse /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <=
rx19_csum_complete /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <= rx19_packets /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <=
rx19_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 92611021 ( 92,611,021) <= rx1_bytes /sec
Ethtool(enp216s0) stat: 92307 ( 92,307) <=
rx1_cache_reuse /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <=
rx1_csum_complete /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <= rx1_packets /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <=
rx1_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88902617 ( 88,902,617) <= rx20_bytes /sec
Ethtool(enp216s0) stat: 82844 ( 82,844) <=
rx20_cache_reuse /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <=
rx20_csum_complete /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <= rx20_packets /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <=
rx20_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 98096942 ( 98,096,942) <= rx21_bytes /sec
Ethtool(enp216s0) stat: 79975 ( 79,975) <=
rx21_cache_reuse /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <=
rx21_csum_complete /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <= rx21_packets /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <=
rx21_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 71885445 ( 71,885,445) <= rx22_bytes /sec
Ethtool(enp216s0) stat: 73565 ( 73,565) <=
rx22_cache_reuse /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <=
rx22_csum_complete /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <= rx22_packets /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <=
rx22_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 77030721 ( 77,030,721) <= rx23_bytes /sec
Ethtool(enp216s0) stat: 74481 ( 74,481) <=
rx23_cache_reuse /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <=
rx23_csum_complete /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <= rx23_packets /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <=
rx23_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 69603951 ( 69,603,951) <= rx24_bytes /sec
Ethtool(enp216s0) stat: 77472 ( 77,472) <=
rx24_cache_reuse /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <=
rx24_csum_complete /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <= rx24_packets /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <=
rx24_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 62277522 ( 62,277,522) <= rx25_bytes /sec
Ethtool(enp216s0) stat: 69414 ( 69,414) <=
rx25_cache_reuse /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <=
rx25_csum_complete /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <= rx25_packets /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <=
rx25_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 77498258 ( 77,498,258) <= rx26_bytes /sec
Ethtool(enp216s0) stat: 72466 ( 72,466) <=
rx26_cache_reuse /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <=
rx26_csum_complete /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <= rx26_packets /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <=
rx26_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88962856 ( 88,962,856) <= rx27_bytes /sec
Ethtool(enp216s0) stat: 80952 ( 80,952) <=
rx27_cache_reuse /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <=
rx27_csum_complete /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <= rx27_packets /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <=
rx27_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 103624977 ( 103,624,977) <= rx2_bytes /sec
Ethtool(enp216s0) stat: 101098 ( 101,098) <=
rx2_cache_reuse /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <=
rx2_csum_complete /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <= rx2_packets /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <=
rx2_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 87213368 ( 87,213,368) <= rx3_bytes /sec
Ethtool(enp216s0) stat: 99877 ( 99,877) <=
rx3_cache_reuse /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <=
rx3_csum_complete /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <= rx3_packets /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <=
rx3_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 110458845 ( 110,458,845) <= rx4_bytes /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx4_cache_busy
/sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx4_cache_full
/sec
Ethtool(enp216s0) stat: 95523 ( 95,523) <=
rx4_cache_reuse /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <=
rx4_csum_complete /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <= rx4_packets /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <=
rx4_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 121222062 ( 121,222,062) <= rx5_bytes /sec
Ethtool(enp216s0) stat: 105616 ( 105,616) <=
rx5_cache_reuse /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <=
rx5_csum_complete /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <= rx5_packets /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <=
rx5_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 85541470 ( 85,541,470) <= rx6_bytes /sec
Ethtool(enp216s0) stat: 91147 ( 91,147) <=
rx6_cache_reuse /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <=
rx6_csum_complete /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <= rx6_packets /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <=
rx6_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88443649 ( 88,443,649) <= rx7_bytes /sec
Ethtool(enp216s0) stat: 92368 ( 92,368) <=
rx7_cache_reuse /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <=
rx7_csum_complete /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <= rx7_packets /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <=
rx7_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 108419072 ( 108,419,072) <= rx8_bytes /sec
Ethtool(enp216s0) stat: 101098 ( 101,098) <=
rx8_cache_reuse /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <=
rx8_csum_complete /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <= rx8_packets /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <=
rx8_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 116210958 ( 116,210,958) <= rx9_bytes /sec
Ethtool(enp216s0) stat: 102014 ( 102,014) <=
rx9_cache_reuse /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <=
rx9_csum_complete /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <= rx9_packets /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <=
rx9_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 1065697 ( 1,065,697) <=
rx_1024_to_1518_bytes_phy /sec
Ethtool(enp216s0) stat: 141705 ( 141,705) <=
rx_128_to_255_bytes_phy /sec
Ethtool(enp216s0) stat: 512465 ( 512,465) <=
rx_1519_to_2047_bytes_phy /sec
Ethtool(enp216s0) stat: 49523 ( 49,523) <=
rx_256_to_511_bytes_phy /sec
Ethtool(enp216s0) stat: 56321 ( 56,321) <=
rx_512_to_1023_bytes_phy /sec
Ethtool(enp216s0) stat: 957286 ( 957,286) <=
rx_64_bytes_phy /sec
Ethtool(enp216s0) stat: 2193644 ( 2,193,644) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_broadcast_phy /sec
Ethtool(enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
Ethtool(enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx_cache_busy /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx_cache_full /sec
Ethtool(enp216s0) stat: 2484928 ( 2,484,928) <= rx_cache_reuse
/sec
Ethtool(enp216s0) stat: 4971670 ( 4,971,670) <=
rx_csum_complete /sec
Ethtool(enp216s0) stat: 7 ( 7) <= rx_csum_none /sec
Ethtool(enp216s0) stat: 464 ( 464) <=
rx_discards_phy /sec
Ethtool(enp216s0) stat: 1376 ( 1,376) <=
rx_multicast_phy /sec
Ethtool(enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
Ethtool(enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy
/sec
Ethtool(enp216s0) stat: 2634727119 ( 2,634,727,119) <= rx_prio0_bytes
/sec
Ethtool(enp216s0) stat: 4975012 ( 4,975,012) <=
rx_prio0_packets /sec
Ethtool(enp216s0) stat: 61 ( 61) <= rx_prio1_bytes
/sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_prio1_packets /sec
Ethtool(enp216s0) stat: 2003 ( 2,003) <= rx_prio3_bytes
/sec
Ethtool(enp216s0) stat: 7 ( 7) <=
rx_prio3_packets /sec
Ethtool(enp216s0) stat: 2497 ( 2,497) <= rx_prio4_bytes
/sec
Ethtool(enp216s0) stat: 3 ( 3) <=
rx_prio4_packets /sec
Ethtool(enp216s0) stat: 10697 ( 10,697) <= rx_prio5_bytes
/sec
Ethtool(enp216s0) stat: 163 ( 163) <=
rx_prio5_packets /sec
Ethtool(enp216s0) stat: 3519 ( 3,519) <= rx_prio6_bytes
/sec
Ethtool(enp216s0) stat: 26 ( 26) <=
rx_prio6_packets /sec
Ethtool(enp216s0) stat: 95583 ( 95,583) <= rx_prio7_bytes
/sec
Ethtool(enp216s0) stat: 1365 ( 1,365) <=
rx_prio7_packets /sec
Ethtool(enp216s0) stat: 4971677 ( 4,971,677) <=
rx_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 1377 ( 1,377) <=
rx_steer_missed_packets /sec
Ethtool(enp216s0) stat: 150 ( 150) <=
rx_vport_broadcast_bytes /sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_vport_broadcast_packets /sec
Ethtool(enp216s0) stat: 90890 ( 90,890) <=
rx_vport_multicast_bytes /sec
Ethtool(enp216s0) stat: 1376 ( 1,376) <=
rx_vport_multicast_packets /sec
Ethtool(enp216s0) stat: 2612989221 ( 2,612,989,221) <=
rx_vport_unicast_bytes /sec
Ethtool(enp216s0) stat: 4973191 ( 4,973,191) <=
rx_vport_unicast_packets /sec
Ethtool(enp216s0) stat: 82580 ( 82,580) <=
tx0_added_vlan_packets /sec
Ethtool(enp216s0) stat: 186927644 ( 186,927,644) <= tx0_bytes /sec
Ethtool(enp216s0) stat: 81347 ( 81,347) <= tx0_cqes /sec
Ethtool(enp216s0) stat: 49836 ( 49,836) <= tx0_csum_none /sec
Ethtool(enp216s0) stat: 32745 ( 32,745) <=
tx0_csum_partial /sec
Ethtool(enp216s0) stat: 2093 ( 2,093) <= tx0_nop /sec
Ethtool(enp216s0) stat: 146450 ( 146,450) <= tx0_packets /sec
Ethtool(enp216s0) stat: 131547357 ( 131,547,357) <= tx0_tso_bytes /sec
Ethtool(enp216s0) stat: 28323 ( 28,323) <=
tx0_tso_packets /sec
Ethtool(enp216s0) stat: 1226 ( 1,226) <= tx0_xmit_more /sec
Ethtool(enp216s0) stat: 85851 ( 85,851) <=
tx10_added_vlan_packets /sec
Ethtool(enp216s0) stat: 198440153 ( 198,440,153) <= tx10_bytes /sec
Ethtool(enp216s0) stat: 84107 ( 84,107) <= tx10_cqes /sec
Ethtool(enp216s0) stat: 51644 ( 51,644) <= tx10_csum_none
/sec
Ethtool(enp216s0) stat: 34207 ( 34,207) <=
tx10_csum_partial /sec
Ethtool(enp216s0) stat: 2256 ( 2,256) <= tx10_nop /sec
Ethtool(enp216s0) stat: 151642 ( 151,642) <= tx10_packets /sec
Ethtool(enp216s0) stat: 135145744 ( 135,145,744) <= tx10_tso_bytes
/sec
Ethtool(enp216s0) stat: 29140 ( 29,140) <=
tx10_tso_packets /sec
Ethtool(enp216s0) stat: 1739 ( 1,739) <= tx10_xmit_more
/sec
Ethtool(enp216s0) stat: 85526 ( 85,526) <=
tx11_added_vlan_packets /sec
Ethtool(enp216s0) stat: 199463985 ( 199,463,985) <= tx11_bytes /sec
Ethtool(enp216s0) stat: 83793 ( 83,793) <= tx11_cqes /sec
Ethtool(enp216s0) stat: 50188 ( 50,188) <= tx11_csum_none
/sec
Ethtool(enp216s0) stat: 35338 ( 35,338) <=
tx11_csum_partial /sec
Ethtool(enp216s0) stat: 2227 ( 2,227) <= tx11_nop /sec
Ethtool(enp216s0) stat: 152815 ( 152,815) <= tx11_packets /sec
Ethtool(enp216s0) stat: 138134263 ( 138,134,263) <= tx11_tso_bytes
/sec
Ethtool(enp216s0) stat: 29948 ( 29,948) <=
tx11_tso_packets /sec
Ethtool(enp216s0) stat: 1733 ( 1,733) <= tx11_xmit_more
/sec
Ethtool(enp216s0) stat: 77464 ( 77,464) <=
tx12_added_vlan_packets /sec
Ethtool(enp216s0) stat: 185723223 ( 185,723,223) <= tx12_bytes /sec
Ethtool(enp216s0) stat: 75907 ( 75,907) <= tx12_cqes /sec
Ethtool(enp216s0) stat: 43727 ( 43,727) <= tx12_csum_none
/sec
Ethtool(enp216s0) stat: 33738 ( 33,738) <=
tx12_csum_partial /sec
Ethtool(enp216s0) stat: 2060 ( 2,060) <= tx12_nop /sec
Ethtool(enp216s0) stat: 141903 ( 141,903) <= tx12_packets /sec
Ethtool(enp216s0) stat: 133227574 ( 133,227,574) <= tx12_tso_bytes
/sec
Ethtool(enp216s0) stat: 29139 ( 29,139) <=
tx12_tso_packets /sec
Ethtool(enp216s0) stat: 1559 ( 1,559) <= tx12_xmit_more
/sec
Ethtool(enp216s0) stat: 79682 ( 79,682) <=
tx13_added_vlan_packets /sec
Ethtool(enp216s0) stat: 189943899 ( 189,943,899) <= tx13_bytes /sec
Ethtool(enp216s0) stat: 78110 ( 78,110) <= tx13_cqes /sec
Ethtool(enp216s0) stat: 46294 ( 46,294) <= tx13_csum_none
/sec
Ethtool(enp216s0) stat: 33388 ( 33,388) <=
tx13_csum_partial /sec
Ethtool(enp216s0) stat: 2109 ( 2,109) <= tx13_nop /sec
Ethtool(enp216s0) stat: 145877 ( 145,877) <= tx13_packets /sec
Ethtool(enp216s0) stat: 136076991 ( 136,076,991) <= tx13_tso_bytes
/sec
Ethtool(enp216s0) stat: 29367 ( 29,367) <=
tx13_tso_packets /sec
Ethtool(enp216s0) stat: 1572 ( 1,572) <= tx13_xmit_more
/sec
Ethtool(enp216s0) stat: 86096 ( 86,096) <=
tx14_added_vlan_packets /sec
Ethtool(enp216s0) stat: 199318635 ( 199,318,635) <= tx14_bytes /sec
Ethtool(enp216s0) stat: 84335 ( 84,335) <= tx14_cqes /sec
Ethtool(enp216s0) stat: 50934 ( 50,934) <= tx14_csum_none
/sec
Ethtool(enp216s0) stat: 35163 ( 35,163) <=
tx14_csum_partial /sec
Ethtool(enp216s0) stat: 2263 ( 2,263) <= tx14_nop /sec
Ethtool(enp216s0) stat: 156445 ( 156,445) <= tx14_packets /sec
Ethtool(enp216s0) stat: 142293555 ( 142,293,555) <= tx14_tso_bytes
/sec
Ethtool(enp216s0) stat: 29703 ( 29,703) <=
tx14_tso_packets /sec
Ethtool(enp216s0) stat: 1761 ( 1,761) <= tx14_xmit_more
/sec
Ethtool(enp216s0) stat: 79698 ( 79,698) <=
tx15_added_vlan_packets /sec
Ethtool(enp216s0) stat: 189620424 ( 189,620,424) <= tx15_bytes /sec
Ethtool(enp216s0) stat: 78356 ( 78,356) <= tx15_cqes /sec
Ethtool(enp216s0) stat: 45528 ( 45,528) <= tx15_csum_none
/sec
Ethtool(enp216s0) stat: 34170 ( 34,170) <=
tx15_csum_partial /sec
Ethtool(enp216s0) stat: 2156 ( 2,156) <= tx15_nop /sec
Ethtool(enp216s0) stat: 144935 ( 144,935) <= tx15_packets /sec
Ethtool(enp216s0) stat: 135023821 ( 135,023,821) <= tx15_tso_bytes
/sec
Ethtool(enp216s0) stat: 29400 ( 29,400) <=
tx15_tso_packets /sec
Ethtool(enp216s0) stat: 1344 ( 1,344) <= tx15_xmit_more
/sec
Ethtool(enp216s0) stat: 59598 ( 59,598) <=
tx16_added_vlan_packets /sec
Ethtool(enp216s0) stat: 143187495 ( 143,187,495) <= tx16_bytes /sec
Ethtool(enp216s0) stat: 58408 ( 58,408) <= tx16_cqes /sec
Ethtool(enp216s0) stat: 35002 ( 35,002) <= tx16_csum_none
/sec
Ethtool(enp216s0) stat: 24595 ( 24,595) <=
tx16_csum_partial /sec
Ethtool(enp216s0) stat: 1585 ( 1,585) <= tx16_nop /sec
Ethtool(enp216s0) stat: 110250 ( 110,250) <= tx16_packets /sec
Ethtool(enp216s0) stat: 101339613 ( 101,339,613) <= tx16_tso_bytes
/sec
Ethtool(enp216s0) stat: 20698 ( 20,698) <=
tx16_tso_packets /sec
Ethtool(enp216s0) stat: 1179 ( 1,179) <= tx16_xmit_more
/sec
Ethtool(enp216s0) stat: 69504 ( 69,504) <=
tx17_added_vlan_packets /sec
Ethtool(enp216s0) stat: 171534675 ( 171,534,675) <= tx17_bytes /sec
Ethtool(enp216s0) stat: 68155 ( 68,155) <= tx17_cqes /sec
Ethtool(enp216s0) stat: 39445 ( 39,445) <= tx17_csum_none
/sec
Ethtool(enp216s0) stat: 30059 ( 30,059) <=
tx17_csum_partial /sec
Ethtool(enp216s0) stat: 1886 ( 1,886) <= tx17_nop /sec
Ethtool(enp216s0) stat: 130910 ( 130,910) <= tx17_packets /sec
Ethtool(enp216s0) stat: 123978012 ( 123,978,012) <= tx17_tso_bytes
/sec
Ethtool(enp216s0) stat: 26215 ( 26,215) <=
tx17_tso_packets /sec
Ethtool(enp216s0) stat: 1349 ( 1,349) <= tx17_xmit_more
/sec
Ethtool(enp216s0) stat: 58880 ( 58,880) <=
tx18_added_vlan_packets /sec
Ethtool(enp216s0) stat: 141863299 ( 141,863,299) <= tx18_bytes /sec
Ethtool(enp216s0) stat: 57755 ( 57,755) <= tx18_cqes /sec
Ethtool(enp216s0) stat: 33875 ( 33,875) <= tx18_csum_none
/sec
Ethtool(enp216s0) stat: 25005 ( 25,005) <=
tx18_csum_partial /sec
Ethtool(enp216s0) stat: 1592 ( 1,592) <= tx18_nop /sec
Ethtool(enp216s0) stat: 110248 ( 110,248) <= tx18_packets /sec
Ethtool(enp216s0) stat: 103105544 ( 103,105,544) <= tx18_tso_bytes
/sec
Ethtool(enp216s0) stat: 21243 ( 21,243) <=
tx18_tso_packets /sec
Ethtool(enp216s0) stat: 1108 ( 1,108) <= tx18_xmit_more
/sec
Ethtool(enp216s0) stat: 69804 ( 69,804) <=
tx19_added_vlan_packets /sec
Ethtool(enp216s0) stat: 164225730 ( 164,225,730) <= tx19_bytes /sec
Ethtool(enp216s0) stat: 68756 ( 68,756) <= tx19_cqes /sec
Ethtool(enp216s0) stat: 42003 ( 42,003) <= tx19_csum_none
/sec
Ethtool(enp216s0) stat: 27801 ( 27,801) <=
tx19_csum_partial /sec
Ethtool(enp216s0) stat: 1790 ( 1,790) <= tx19_nop /sec
Ethtool(enp216s0) stat: 124794 ( 124,794) <= tx19_packets /sec
Ethtool(enp216s0) stat: 110814620 ( 110,814,620) <= tx19_tso_bytes
/sec
Ethtool(enp216s0) stat: 23278 ( 23,278) <=
tx19_tso_packets /sec
Ethtool(enp216s0) stat: 1045 ( 1,045) <= tx19_xmit_more
/sec
Ethtool(enp216s0) stat: 79346 ( 79,346) <=
tx1_added_vlan_packets /sec
Ethtool(enp216s0) stat: 181179251 ( 181,179,251) <= tx1_bytes /sec
Ethtool(enp216s0) stat: 78062 ( 78,062) <= tx1_cqes /sec
Ethtool(enp216s0) stat: 46525 ( 46,525) <= tx1_csum_none /sec
Ethtool(enp216s0) stat: 32821 ( 32,821) <=
tx1_csum_partial /sec
Ethtool(enp216s0) stat: 1996 ( 1,996) <= tx1_nop /sec
Ethtool(enp216s0) stat: 142716 ( 142,716) <= tx1_packets /sec
Ethtool(enp216s0) stat: 129507562 ( 129,507,562) <= tx1_tso_bytes /sec
Ethtool(enp216s0) stat: 27579 ( 27,579) <=
tx1_tso_packets /sec
Ethtool(enp216s0) stat: 1281 ( 1,281) <= tx1_xmit_more /sec
Ethtool(enp216s0) stat: 66641 ( 66,641) <=
tx20_added_vlan_packets /sec
Ethtool(enp216s0) stat: 161452661 ( 161,452,661) <= tx20_bytes /sec
Ethtool(enp216s0) stat: 65374 ( 65,374) <= tx20_cqes /sec
Ethtool(enp216s0) stat: 37657 ( 37,657) <= tx20_csum_none
/sec
Ethtool(enp216s0) stat: 28983 ( 28,983) <=
tx20_csum_partial /sec
Ethtool(enp216s0) stat: 1739 ( 1,739) <= tx20_nop /sec
Ethtool(enp216s0) stat: 122824 ( 122,824) <= tx20_packets /sec
Ethtool(enp216s0) stat: 115707977 ( 115,707,977) <= tx20_tso_bytes
/sec
Ethtool(enp216s0) stat: 24823 ( 24,823) <=
tx20_tso_packets /sec
Ethtool(enp216s0) stat: 1263 ( 1,263) <= tx20_xmit_more
/sec
Ethtool(enp216s0) stat: 60564 ( 60,564) <=
tx21_added_vlan_packets /sec
Ethtool(enp216s0) stat: 138260273 ( 138,260,273) <= tx21_bytes /sec
Ethtool(enp216s0) stat: 59611 ( 59,611) <= tx21_cqes /sec
Ethtool(enp216s0) stat: 36105 ( 36,105) <= tx21_csum_none
/sec
Ethtool(enp216s0) stat: 24459 ( 24,459) <=
tx21_csum_partial /sec
Ethtool(enp216s0) stat: 1490 ( 1,490) <= tx21_nop /sec
Ethtool(enp216s0) stat: 106913 ( 106,913) <= tx21_packets /sec
Ethtool(enp216s0) stat: 97216016 ( 97,216,016) <= tx21_tso_bytes
/sec
Ethtool(enp216s0) stat: 21944 ( 21,944) <=
tx21_tso_packets /sec
Ethtool(enp216s0) stat: 930 ( 930) <= tx21_xmit_more
/sec
Ethtool(enp216s0) stat: 66527 ( 66,527) <=
tx22_added_vlan_packets /sec
Ethtool(enp216s0) stat: 149686537 ( 149,686,537) <= tx22_bytes /sec
Ethtool(enp216s0) stat: 65262 ( 65,262) <= tx22_cqes /sec
Ethtool(enp216s0) stat: 40609 ( 40,609) <= tx22_csum_none
/sec
Ethtool(enp216s0) stat: 25918 ( 25,918) <=
tx22_csum_partial /sec
Ethtool(enp216s0) stat: 1676 ( 1,676) <= tx22_nop /sec
Ethtool(enp216s0) stat: 118071 ( 118,071) <= tx22_packets /sec
Ethtool(enp216s0) stat: 104822520 ( 104,822,520) <= tx22_tso_bytes
/sec
Ethtool(enp216s0) stat: 22046 ( 22,046) <=
tx22_tso_packets /sec
Ethtool(enp216s0) stat: 1265 ( 1,265) <= tx22_xmit_more
/sec
Ethtool(enp216s0) stat: 59973 ( 59,973) <=
tx23_added_vlan_packets /sec
Ethtool(enp216s0) stat: 147788578 ( 147,788,578) <= tx23_bytes /sec
Ethtool(enp216s0) stat: 58827 ( 58,827) <= tx23_cqes /sec
Ethtool(enp216s0) stat: 33197 ( 33,197) <= tx23_csum_none
/sec
Ethtool(enp216s0) stat: 26776 ( 26,776) <=
tx23_csum_partial /sec
Ethtool(enp216s0) stat: 1669 ( 1,669) <= tx23_nop /sec
Ethtool(enp216s0) stat: 112053 ( 112,053) <= tx23_packets /sec
Ethtool(enp216s0) stat: 106098597 ( 106,098,597) <= tx23_tso_bytes
/sec
Ethtool(enp216s0) stat: 22433 ( 22,433) <=
tx23_tso_packets /sec
Ethtool(enp216s0) stat: 1126 ( 1,126) <= tx23_xmit_more
/sec
Ethtool(enp216s0) stat: 58819 ( 58,819) <=
tx24_added_vlan_packets /sec
Ethtool(enp216s0) stat: 146231570 ( 146,231,570) <= tx24_bytes /sec
Ethtool(enp216s0) stat: 57661 ( 57,661) <= tx24_cqes /sec
Ethtool(enp216s0) stat: 32150 ( 32,150) <= tx24_csum_none
/sec
Ethtool(enp216s0) stat: 26669 ( 26,669) <=
tx24_csum_partial /sec
Ethtool(enp216s0) stat: 1578 ( 1,578) <= tx24_nop /sec
Ethtool(enp216s0) stat: 111230 ( 111,230) <= tx24_packets /sec
Ethtool(enp216s0) stat: 106359530 ( 106,359,530) <= tx24_tso_bytes
/sec
Ethtool(enp216s0) stat: 22402 ( 22,402) <=
tx24_tso_packets /sec
Ethtool(enp216s0) stat: 1158 ( 1,158) <= tx24_xmit_more
/sec
Ethtool(enp216s0) stat: 64116 ( 64,116) <=
tx25_added_vlan_packets /sec
Ethtool(enp216s0) stat: 156132090 ( 156,132,090) <= tx25_bytes /sec
Ethtool(enp216s0) stat: 62901 ( 62,901) <= tx25_cqes /sec
Ethtool(enp216s0) stat: 36357 ( 36,357) <= tx25_csum_none
/sec
Ethtool(enp216s0) stat: 27759 ( 27,759) <=
tx25_csum_partial /sec
Ethtool(enp216s0) stat: 1717 ( 1,717) <= tx25_nop /sec
Ethtool(enp216s0) stat: 118618 ( 118,618) <= tx25_packets /sec
Ethtool(enp216s0) stat: 110752893 ( 110,752,893) <= tx25_tso_bytes
/sec
Ethtool(enp216s0) stat: 23282 ( 23,282) <=
tx25_tso_packets /sec
Ethtool(enp216s0) stat: 1212 ( 1,212) <= tx25_xmit_more
/sec
Ethtool(enp216s0) stat: 62028 ( 62,028) <=
tx26_added_vlan_packets /sec
Ethtool(enp216s0) stat: 144966495 ( 144,966,495) <= tx26_bytes /sec
Ethtool(enp216s0) stat: 60906 ( 60,906) <= tx26_cqes /sec
Ethtool(enp216s0) stat: 36345 ( 36,345) <= tx26_csum_none
/sec
Ethtool(enp216s0) stat: 25684 ( 25,684) <=
tx26_csum_partial /sec
Ethtool(enp216s0) stat: 1615 ( 1,615) <= tx26_nop /sec
Ethtool(enp216s0) stat: 112487 ( 112,487) <= tx26_packets /sec
Ethtool(enp216s0) stat: 102083200 ( 102,083,200) <= tx26_tso_bytes
/sec
Ethtool(enp216s0) stat: 21598 ( 21,598) <=
tx26_tso_packets /sec
Ethtool(enp216s0) stat: 1123 ( 1,123) <= tx26_xmit_more
/sec
Ethtool(enp216s0) stat: 64029 ( 64,029) <=
tx27_added_vlan_packets /sec
Ethtool(enp216s0) stat: 154747343 ( 154,747,343) <= tx27_bytes /sec
Ethtool(enp216s0) stat: 62881 ( 62,881) <= tx27_cqes /sec
Ethtool(enp216s0) stat: 35343 ( 35,343) <= tx27_csum_none
/sec
Ethtool(enp216s0) stat: 28686 ( 28,686) <=
tx27_csum_partial /sec
Ethtool(enp216s0) stat: 1707 ( 1,707) <= tx27_nop /sec
Ethtool(enp216s0) stat: 118074 ( 118,074) <= tx27_packets /sec
Ethtool(enp216s0) stat: 110960676 ( 110,960,676) <= tx27_tso_bytes
/sec
Ethtool(enp216s0) stat: 24139 ( 24,139) <=
tx27_tso_packets /sec
Ethtool(enp216s0) stat: 1148 ( 1,148) <= tx27_xmit_more
/sec
Ethtool(enp216s0) stat: 82542 ( 82,542) <=
tx2_added_vlan_packets /sec
Ethtool(enp216s0) stat: 183007051 ( 183,007,051) <= tx2_bytes /sec
Ethtool(enp216s0) stat: 81320 ( 81,320) <= tx2_cqes /sec
Ethtool(enp216s0) stat: 49989 ( 49,989) <= tx2_csum_none /sec
Ethtool(enp216s0) stat: 32553 ( 32,553) <=
tx2_csum_partial /sec
Ethtool(enp216s0) stat: 2142 ( 2,142) <= tx2_nop /sec
Ethtool(enp216s0) stat: 143613 ( 143,613) <= tx2_packets /sec
Ethtool(enp216s0) stat: 126895404 ( 126,895,404) <= tx2_tso_bytes /sec
Ethtool(enp216s0) stat: 28173 ( 28,173) <=
tx2_tso_packets /sec
Ethtool(enp216s0) stat: 1220 ( 1,220) <= tx2_xmit_more /sec
Ethtool(enp216s0) stat: 78737 ( 78,737) <=
tx3_added_vlan_packets /sec
Ethtool(enp216s0) stat: 188546556 ( 188,546,556) <= tx3_bytes /sec
Ethtool(enp216s0) stat: 77389 ( 77,389) <= tx3_cqes /sec
Ethtool(enp216s0) stat: 46147 ( 46,147) <= tx3_csum_none /sec
Ethtool(enp216s0) stat: 32590 ( 32,590) <=
tx3_csum_partial /sec
Ethtool(enp216s0) stat: 2171 ( 2,171) <= tx3_nop /sec
Ethtool(enp216s0) stat: 144570 ( 144,570) <= tx3_packets /sec
Ethtool(enp216s0) stat: 134231500 ( 134,231,500) <= tx3_tso_bytes /sec
Ethtool(enp216s0) stat: 28857 ( 28,857) <=
tx3_tso_packets /sec
Ethtool(enp216s0) stat: 1348 ( 1,348) <= tx3_xmit_more /sec
Ethtool(enp216s0) stat: 76284 ( 76,284) <=
tx4_added_vlan_packets /sec
Ethtool(enp216s0) stat: 185633403 ( 185,633,403) <= tx4_bytes /sec
Ethtool(enp216s0) stat: 74710 ( 74,710) <= tx4_cqes /sec
Ethtool(enp216s0) stat: 42803 ( 42,803) <= tx4_csum_none /sec
Ethtool(enp216s0) stat: 33482 ( 33,482) <=
tx4_csum_partial /sec
Ethtool(enp216s0) stat: 2005 ( 2,005) <= tx4_nop /sec
Ethtool(enp216s0) stat: 139978 ( 139,978) <= tx4_packets /sec
Ethtool(enp216s0) stat: 131762891 ( 131,762,891) <= tx4_tso_bytes /sec
Ethtool(enp216s0) stat: 28843 ( 28,843) <=
tx4_tso_packets /sec
Ethtool(enp216s0) stat: 1563 ( 1,563) <= tx4_xmit_more /sec
Ethtool(enp216s0) stat: 83878 ( 83,878) <=
tx5_added_vlan_packets /sec
Ethtool(enp216s0) stat: 197920426 ( 197,920,426) <= tx5_bytes /sec
Ethtool(enp216s0) stat: 82146 ( 82,146) <= tx5_cqes /sec
Ethtool(enp216s0) stat: 48122 ( 48,122) <= tx5_csum_none /sec
Ethtool(enp216s0) stat: 35755 ( 35,755) <=
tx5_csum_partial /sec
Ethtool(enp216s0) stat: 2227 ( 2,227) <= tx5_nop /sec
Ethtool(enp216s0) stat: 151214 ( 151,214) <= tx5_packets /sec
Ethtool(enp216s0) stat: 139113072 ( 139,113,072) <= tx5_tso_bytes /sec
Ethtool(enp216s0) stat: 30390 ( 30,390) <=
tx5_tso_packets /sec
Ethtool(enp216s0) stat: 1730 ( 1,730) <= tx5_xmit_more /sec
Ethtool(enp216s0) stat: 76307 ( 76,307) <=
tx6_added_vlan_packets /sec
Ethtool(enp216s0) stat: 182153538 ( 182,153,538) <= tx6_bytes /sec
Ethtool(enp216s0) stat: 74744 ( 74,744) <= tx6_cqes /sec
Ethtool(enp216s0) stat: 44381 ( 44,381) <= tx6_csum_none /sec
Ethtool(enp216s0) stat: 31926 ( 31,926) <=
tx6_csum_partial /sec
Ethtool(enp216s0) stat: 2068 ( 2,068) <= tx6_nop /sec
Ethtool(enp216s0) stat: 139745 ( 139,745) <= tx6_packets /sec
Ethtool(enp216s0) stat: 129175625 ( 129,175,625) <= tx6_tso_bytes /sec
Ethtool(enp216s0) stat: 27349 ( 27,349) <=
tx6_tso_packets /sec
Ethtool(enp216s0) stat: 1562 ( 1,562) <= tx6_xmit_more /sec
Ethtool(enp216s0) stat: 72151 ( 72,151) <=
tx7_added_vlan_packets /sec
Ethtool(enp216s0) stat: 175340247 ( 175,340,247) <= tx7_bytes /sec
Ethtool(enp216s0) stat: 70565 ( 70,565) <= tx7_cqes /sec
Ethtool(enp216s0) stat: 40824 ( 40,824) <= tx7_csum_none /sec
Ethtool(enp216s0) stat: 31327 ( 31,327) <=
tx7_csum_partial /sec
Ethtool(enp216s0) stat: 1962 ( 1,962) <= tx7_nop /sec
Ethtool(enp216s0) stat: 133790 ( 133,790) <= tx7_packets /sec
Ethtool(enp216s0) stat: 124931972 ( 124,931,972) <= tx7_tso_bytes /sec
Ethtool(enp216s0) stat: 26314 ( 26,314) <=
tx7_tso_packets /sec
Ethtool(enp216s0) stat: 1586 ( 1,586) <= tx7_xmit_more /sec
Ethtool(enp216s0) stat: 83983 ( 83,983) <=
tx8_added_vlan_packets /sec
Ethtool(enp216s0) stat: 203970262 ( 203,970,262) <= tx8_bytes /sec
Ethtool(enp216s0) stat: 82475 ( 82,475) <= tx8_cqes /sec
Ethtool(enp216s0) stat: 47937 ( 47,937) <= tx8_csum_none /sec
Ethtool(enp216s0) stat: 36046 ( 36,046) <=
tx8_csum_partial /sec
Ethtool(enp216s0) stat: 2203 ( 2,203) <= tx8_nop /sec
Ethtool(enp216s0) stat: 153743 ( 153,743) <= tx8_packets /sec
Ethtool(enp216s0) stat: 143296525 ( 143,296,525) <= tx8_tso_bytes /sec
Ethtool(enp216s0) stat: 30936 ( 30,936) <=
tx8_tso_packets /sec
Ethtool(enp216s0) stat: 1500 ( 1,500) <= tx8_xmit_more /sec
Ethtool(enp216s0) stat: 79406 ( 79,406) <=
tx9_added_vlan_packets /sec
Ethtool(enp216s0) stat: 183286836 ( 183,286,836) <= tx9_bytes /sec
Ethtool(enp216s0) stat: 77769 ( 77,769) <= tx9_cqes /sec
Ethtool(enp216s0) stat: 48014 ( 48,014) <= tx9_csum_none /sec
Ethtool(enp216s0) stat: 31392 ( 31,392) <=
tx9_csum_partial /sec
Ethtool(enp216s0) stat: 2146 ( 2,146) <= tx9_nop /sec
Ethtool(enp216s0) stat: 142229 ( 142,229) <= tx9_packets /sec
Ethtool(enp216s0) stat: 127419807 ( 127,419,807) <= tx9_tso_bytes /sec
Ethtool(enp216s0) stat: 26644 ( 26,644) <=
tx9_tso_packets /sec
Ethtool(enp216s0) stat: 1633 ( 1,633) <= tx9_xmit_more /sec
Ethtool(enp216s0) stat: 2050335 ( 2,050,335) <=
tx_added_vlan_packets /sec
Ethtool(enp216s0) stat: 4851410368 ( 4,851,410,368) <= tx_bytes /sec
Ethtool(enp216s0) stat: 4881854912 ( 4,881,854,912) <= tx_bytes_phy /sec
Ethtool(enp216s0) stat: 2011989 ( 2,011,989) <= tx_cqes /sec
Ethtool(enp216s0) stat: 1191137 ( 1,191,137) <= tx_csum_none /sec
Ethtool(enp216s0) stat: 859197 ( 859,197) <=
tx_csum_partial /sec
Ethtool(enp216s0) stat: 54140 ( 54,140) <= tx_nop /sec
Ethtool(enp216s0) stat: 3728759 ( 3,728,759) <= tx_packets /sec
Ethtool(enp216s0) stat: 3729416 ( 3,729,416) <= tx_packets_phy
/sec
Ethtool(enp216s0) stat: 4882774572 ( 4,882,774,572) <= tx_prio0_bytes
/sec
Ethtool(enp216s0) stat: 3730195 ( 3,730,195) <=
tx_prio0_packets /sec
Ethtool(enp216s0) stat: 3431642829 ( 3,431,642,829) <= tx_tso_bytes /sec
Ethtool(enp216s0) stat: 734343 ( 734,343) <= tx_tso_packets
/sec
Ethtool(enp216s0) stat: 4866327192 ( 4,866,327,192) <=
tx_vport_unicast_bytes /sec
Ethtool(enp216s0) stat: 3728966 ( 3,728,966) <=
tx_vport_unicast_packets /sec
Ethtool(enp216s0) stat: 38268 ( 38,268) <= tx_xmit_more /sec
>>
>> Can you give output put from:
>> $ ethtool --show-priv-flag DEVICE
>>
>> I want you to experiment with:
> ethtool --show-priv-flags enp175s0
> Private flags for enp175s0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : off
> rx_striding_rq : on
> rx_no_csum_complete: off
>
>>
>> ethtool --set-priv-flags DEVICE rx_striding_rq off
> ok i will first check on test server if this will reset my interface
> and will not produce kernel panic :)
>>
>> I think you already have played with 'rx_cqe_compress', right.
> yes - and compress increasing number of irq's but doing not much for
> bandwidth same limit 60-64Gbit/s total RX+TX on one 100G port
>
> And what is weird - that limit is in overall symetric - cause if for
> example 100G port is receiving 42G traffic and transmitting 20G
> traffic - when i flood rx side with pktgen or other for example icmp
> traffic 1/2/3/4/5G - then receiving side increase with 1/2/3/4/5Gbit
> of traffic but transmitting is going down for same lvl's
>
>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 19:34 ` Jesper Dangaard Brouer
2018-11-10 19:49 ` Paweł Staszewski
@ 2018-11-10 20:02 ` Paweł Staszewski
2018-11-10 21:01 ` Jesper Dangaard Brouer
1 sibling, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-10 20:02 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> I want you to experiment with:
>
> ethtool --set-priv-flags DEVICE rx_striding_rq off
just checked that previously connectx4 was have thos disabled:
ethtool --show-priv-flags enp175s0f0
Private flags for enp175s0f0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : off
rx_striding_rq : off
rx_no_csum_complete: off
So now we are on connectx5 and we have enabled - for sure connectx5
changed cpu load - where i have now max 50/60% cpu where with connectx4
there was sometimes near 100% with same configuration.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 20:02 ` Paweł Staszewski
@ 2018-11-10 21:01 ` Jesper Dangaard Brouer
2018-11-10 21:53 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-10 21:01 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev, brouer
On Sat, 10 Nov 2018 21:02:10 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> > I want you to experiment with:
> >
> > ethtool --set-priv-flags DEVICE rx_striding_rq off
>
> just checked that previously connectx4 was have thos disabled:
> ethtool --show-priv-flags enp175s0f0
>
> Private flags for enp175s0f0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : off
> rx_striding_rq : off
> rx_no_csum_complete: off
>
The CX4 hardware does not have this feature (p.s. the CX4-Lx does).
> So now we are on connectx5 and we have enabled - for sure connectx5
> changed cpu load - where i have now max 50/60% cpu where with connectx4
> there was sometimes near 100% with same configuration.
I (strongly) believe the CPU load was related to the page-alloactor
lock congestion, that Aaron fixed.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 21:01 ` Jesper Dangaard Brouer
@ 2018-11-10 21:53 ` Paweł Staszewski
2018-11-10 22:04 ` Paweł Staszewski
2018-11-11 8:56 ` Jesper Dangaard Brouer
0 siblings, 2 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-10 21:53 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 10.11.2018 o 22:01, Jesper Dangaard Brouer pisze:
> On Sat, 10 Nov 2018 21:02:10 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>> I want you to experiment with:
>>>
>>> ethtool --set-priv-flags DEVICE rx_striding_rq off
>> just checked that previously connectx4 was have thos disabled:
>> ethtool --show-priv-flags enp175s0f0
>>
>> Private flags for enp175s0f0:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : off
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
> The CX4 hardware does not have this feature (p.s. the CX4-Lx does).
>
>
>> So now we are on connectx5 and we have enabled - for sure connectx5
>> changed cpu load - where i have now max 50/60% cpu where with connectx4
>> there was sometimes near 100% with same configuration.
> I (strongly) believe the CPU load was related to the page-alloactor
> lock congestion, that Aaron fixed.
>
Yes i think both - most problems with cpu was due to page-allocator
problems.
But also after change connctx4 to connectx5 there is cpu load difference
- about 10% in total - but yes most of this like 40% is cause of Aaron
patch :) - rly good job :)
Now im messing with ring configuration for connectx5 nics.
And after reading that paper:
https://netdevconf.org/2.1/slides/apr6/network-performance/
04-amir-RX_and_TX_bulking_v2.pdf
changed from RX:8192 / TX: 4096 to RX:8192 / TX: 256
after this i gain about 5Gbit/s RX and TX traffic and less cpu load....
before change there was 59/59 Gbit/s
After change there is 64/64 Gbit/s
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
enp175s0: 44.45 Gb/s 19.69 Gb/s
64.14 Gb/s
enp216s0: 19.69 Gb/s 44.49 Gb/s
64.19 Gb/s
------------------------------------------------------------------------------
total: 64.14 Gb/s 64.18 Gb/s 128.33 Gb/s
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 21:53 ` Paweł Staszewski
@ 2018-11-10 22:04 ` Paweł Staszewski
2018-11-11 8:56 ` Jesper Dangaard Brouer
1 sibling, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-10 22:04 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 10.11.2018 o 22:53, Paweł Staszewski pisze:
>
>
> W dniu 10.11.2018 o 22:01, Jesper Dangaard Brouer pisze:
>> On Sat, 10 Nov 2018 21:02:10 +0100
>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>>> I want you to experiment with:
>>>>
>>>> ethtool --set-priv-flags DEVICE rx_striding_rq off
>>> just checked that previously connectx4 was have thos disabled:
>>> ethtool --show-priv-flags enp175s0f0
>>>
>>> Private flags for enp175s0f0:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : off
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>> The CX4 hardware does not have this feature (p.s. the CX4-Lx does).
>>
>>> So now we are on connectx5 and we have enabled - for sure connectx5
>>> changed cpu load - where i have now max 50/60% cpu where with connectx4
>>> there was sometimes near 100% with same configuration.
>> I (strongly) believe the CPU load was related to the page-alloactor
>> lock congestion, that Aaron fixed.
>>
> Yes i think both - most problems with cpu was due to page-allocator
> problems.
> But also after change connctx4 to connectx5 there is cpu load
> difference - about 10% in total - but yes most of this like 40% is
> cause of Aaron patch :) - rly good job :)
>
>
> Now im messing with ring configuration for connectx5 nics.
> And after reading that paper:
> https://netdevconf.org/2.1/slides/apr6/network-performance/
> 04-amir-RX_and_TX_bulking_v2.pdf
>
> changed from RX:8192 / TX: 4096 to RX:8192 / TX: 256
>
> after this i gain about 5Gbit/s RX and TX traffic and less cpu load....
> before change there was 59/59 Gbit/s
>
> After change there is 64/64 Gbit/s
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> | iface Rx Tx Total
> ==============================================================================
>
> enp175s0: 44.45 Gb/s 19.69 Gb/s
> 64.14 Gb/s
> enp216s0: 19.69 Gb/s 44.49 Gb/s
> 64.19 Gb/s
> ------------------------------------------------------------------------------
>
> total: 64.14 Gb/s 64.18 Gb/s 128.33 Gb/s
>
>
Also after this change kernel freed some memory... like 500MB
Still squeezed but less with more traffic...
CPU total/sec dropped/sec squeezed/sec
collision/sec rx_rps/sec flow_limit/sec
CPU:00 0 0 0 0
0 0
CPU:01 0 0 0 0
0 0
CPU:02 0 0 0 0
0 0
CPU:03 0 0 0 0
0 0
CPU:04 0 0 0 0
0 0
CPU:05 0 0 0 0
0 0
CPU:06 0 0 0 0
0 0
CPU:07 0 0 0 0
0 0
CPU:08 0 0 0 0
0 0
CPU:09 0 0 0 0
0 0
CPU:10 0 0 0 0
0 0
CPU:11 0 0 0 0
0 0
CPU:12 0 0 0 0
0 0
CPU:13 0 0 0 0
0 0
CPU:14 389270 0 41 0
0 0
CPU:15 375543 0 32 0
0 0
CPU:16 385847 0 22 0
0 0
CPU:17 412293 0 34 0
0 0
CPU:18 401287 0 30 0
0 0
CPU:19 368345 0 30 0
0 0
CPU:20 395452 0 28 0
0 0
CPU:21 374032 0 38 0
0 0
CPU:22 342036 0 32 0
0 0
CPU:23 374773 0 34 0
0 0
CPU:24 356139 0 31 0
0 0
CPU:25 392725 0 32 0
0 0
CPU:26 385937 0 37 0
0 0
CPU:27 385282 0 37 0
0 0
CPU:28 0 0 0 0
0 0
CPU:29 0 0 0 0
0 0
CPU:30 0 0 0 0
0 0
CPU:31 0 0 0 0
0 0
CPU:32 0 0 0 0
0 0
CPU:33 0 0 0 0
0 0
CPU:34 0 0 0 0
0 0
CPU:35 0 0 0 0
0 0
CPU:36 0 0 0 0
0 0
CPU:37 0 0 0 0
0 0
CPU:38 0 0 0 0
0 0
CPU:39 0 0 0 0
0 0
CPU:40 0 0 0 0
0 0
CPU:41 0 0 0 0
0 0
CPU:42 340817 0 33 0
0 0
CPU:43 364805 0 42 0
0 0
CPU:44 298484 0 29 0
0 0
CPU:45 292798 0 30 0
0 0
CPU:46 301739 0 24 0
0 0
CPU:47 275116 0 20 0
0 0
CPU:48 319237 0 34 0
0 0
CPU:49 290350 0 29 0
0 0
CPU:50 307084 0 30 0
0 0
CPU:51 332908 0 24 0
0 0
CPU:52 300151 0 24 0
0 0
CPU:53 310140 0 28 0
0 0
CPU:54 341788 0 28 0
0 0
CPU:55 320344 0 28 0
0 0
Summed: 9734722 0 860 0
0 0
>
>
>
>
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 19:56 ` Paweł Staszewski
@ 2018-11-10 22:06 ` Jesper Dangaard Brouer
2018-11-10 22:19 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-10 22:06 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev, brouer
On Sat, 10 Nov 2018 20:56:02 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
> >
> >
> > W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> >> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
> >> <pstaszewski@itcare.pl> wrote:
> >>
> >>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> >>>> CPU load is lower than for connectx4 - but it looks like bandwidth
> >>>> limit is the same :)
> >>>> But also after reaching 60Gbit/60Gbit
> >>>>
> >>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> >>>> input: /proc/net/dev type: rate
> >>>> - iface Rx Tx Total
> >>>> ===================================================================
> >>>>
> >>>>
> >>>> enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
> >>>> enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
> >>>> -------------------------------------------------------------------
> >>>>
> >>>>
> >>>> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
> >>> Today reached 65/65Gbit/s
> >>>
> >>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
> >>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
> >> This is weird!
> >>
> >> How do you see / measure these drops?
> >
> > Simple icmp test like ping -i 0.1
> > And im testing by icmp management ip address on vlan that is attacked
> > to one NIC (the side that is more stressed with RX)
> > And another icmp test is forward thru this router - host behind it
> >
> > Both measurements shows same loss ratio from 0.1 to 0.5% after
> > reaching ~45Gbit/s RX side - depends how much RX side is pushed drops
> > vary between 0.1 to 0.5 - even 0.6%:)
> >
Okay good to know, you use an external measurement for this. I do
think packets are getting dropped by the NIC.
> >>> So checked other stats.
> >>> softnet_stats shows average 1k squeezed per sec:
> >> Is below output the raw counters? not per sec?
> >>
> >> It would be valuable to see the per sec stats instead...
> >> I use this tool:
> >> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
> CPU total/sec dropped/sec squeezed/sec collision/sec rx_rps/sec flow_limit/sec
> CPU:00 0 0 0 0 0 0
[...]
> CPU:13 0 0 0 0 0 0
> CPU:14 485538 0 43 0 0 0
> CPU:15 474794 0 51 0 0 0
> CPU:16 449322 0 41 0 0 0
> CPU:17 476420 0 46 0 0 0
> CPU:18 440436 0 38 0 0 0
> CPU:19 501499 0 49 0 0 0
> CPU:20 459468 0 49 0 0 0
> CPU:21 438928 0 47 0 0 0
> CPU:22 468983 0 40 0 0 0
> CPU:23 446253 0 47 0 0 0
> CPU:24 451909 0 46 0 0 0
> CPU:25 479373 0 55 0 0 0
> CPU:26 467848 0 49 0 0 0
> CPU:27 453153 0 51 0 0 0
> CPU:28 0 0 0 0 0 0
[...]
> CPU:40 0 0 0 0 0 0
> CPU:41 0 0 0 0 0 0
> CPU:42 466853 0 43 0 0 0
> CPU:43 453059 0 54 0 0 0
> CPU:44 363219 0 34 0 0 0
> CPU:45 353632 0 38 0 0 0
> CPU:46 371618 0 40 0 0 0
> CPU:47 350518 0 46 0 0 0
> CPU:48 397544 0 40 0 0 0
> CPU:49 364873 0 38 0 0 0
> CPU:50 383630 0 38 0 0 0
> CPU:51 358771 0 39 0 0 0
> CPU:52 372547 0 38 0 0 0
> CPU:53 372882 0 36 0 0 0
> CPU:54 366244 0 43 0 0 0
> CPU:55 365886 0 39 0 0 0
>
> Summed: 11835201 0 1217 0 0 0
Do notice, the per CPU squeeze is not too large.
The summed 11.8 Mpps is a little high compared to:
Ethtool(enp216s0) stat: 4971677 (4,971,677) <= rx_packets /sec
Ethtool(enp175s0) stat: 3717148 (3,717,148) <= rx_packets /sec
Sum: 3717148+4971677 = 8688825 (8,688,825)
[...]
> >>>
> >>> Remember those tests are now on two separate connectx5 connected to
> >>> two separate pcie x16 gen 3.0
> >> That is strange... I still suspect some HW NIC issue, can you provide
> >> ethtool stats info via tool:
> >>
> >> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> >>
> >> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
> >>
> >> The tool remove zero-stats counters and report per sec stats. It makes
> >> it easier to spot that is relevant for the given workload.
> > yes mlnx have just too many counters that are always 0 for my case :)
> > Will try this also
> >
> But still alot of non 0 counters
> Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
> Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
[...]
I have copied the stats over in another document so I can better looks
at it... and I've found some interesting stats.
E.g. we can see that the NIC hardware is dropping packets.
RX-drops on enp175s0:
(enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
(enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
-218308971 ( -218,308,971) Dropped bytes /sec
(enp175s0) stat: 139602 ( 139,602) <= rx_discards_phy /sec
(enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
(enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy /sec
-145272 ( -145,272) Dropped packets /sec
RX-drops on enp216s0 is less:
(enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
(enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
-41288962 ( -41,288,962) Dropped bytes /sec
(enp216s0) stat: 464 (464) <= rx_discards_phy /sec
(enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
(enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy /sec
-3886 ( -3,886) Dropped packets /sec
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 22:06 ` Jesper Dangaard Brouer
@ 2018-11-10 22:19 ` Paweł Staszewski
2018-11-11 8:03 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-10 22:19 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 10.11.2018 o 23:06, Jesper Dangaard Brouer pisze:
> On Sat, 10 Nov 2018 20:56:02 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
>>>
>>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
>>>> <pstaszewski@itcare.pl> wrote:
>>>>
>>>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>>>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>>>>> limit is the same :)
>>>>>> But also after reaching 60Gbit/60Gbit
>>>>>>
>>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>> input: /proc/net/dev type: rate
>>>>>> - iface Rx Tx Total
>>>>>> ===================================================================
>>>>>>
>>>>>>
>>>>>> enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
>>>>>> enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
>>>>>> -------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>>>>> Today reached 65/65Gbit/s
>>>>>
>>>>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>>>>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
>>>> This is weird!
>>>>
>>>> How do you see / measure these drops?
>>> Simple icmp test like ping -i 0.1
>>> And im testing by icmp management ip address on vlan that is attacked
>>> to one NIC (the side that is more stressed with RX)
>>> And another icmp test is forward thru this router - host behind it
>>>
>>> Both measurements shows same loss ratio from 0.1 to 0.5% after
>>> reaching ~45Gbit/s RX side - depends how much RX side is pushed drops
>>> vary between 0.1 to 0.5 - even 0.6%:)
>>>
> Okay good to know, you use an external measurement for this. I do
> think packets are getting dropped by the NIC.
>
>>>>> So checked other stats.
>>>>> softnet_stats shows average 1k squeezed per sec:
>>>> Is below output the raw counters? not per sec?
>>>>
>>>> It would be valuable to see the per sec stats instead...
>>>> I use this tool:
>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
>> CPU total/sec dropped/sec squeezed/sec collision/sec rx_rps/sec flow_limit/sec
>> CPU:00 0 0 0 0 0 0
> [...]
>> CPU:13 0 0 0 0 0 0
>> CPU:14 485538 0 43 0 0 0
>> CPU:15 474794 0 51 0 0 0
>> CPU:16 449322 0 41 0 0 0
>> CPU:17 476420 0 46 0 0 0
>> CPU:18 440436 0 38 0 0 0
>> CPU:19 501499 0 49 0 0 0
>> CPU:20 459468 0 49 0 0 0
>> CPU:21 438928 0 47 0 0 0
>> CPU:22 468983 0 40 0 0 0
>> CPU:23 446253 0 47 0 0 0
>> CPU:24 451909 0 46 0 0 0
>> CPU:25 479373 0 55 0 0 0
>> CPU:26 467848 0 49 0 0 0
>> CPU:27 453153 0 51 0 0 0
>> CPU:28 0 0 0 0 0 0
> [...]
>> CPU:40 0 0 0 0 0 0
>> CPU:41 0 0 0 0 0 0
>> CPU:42 466853 0 43 0 0 0
>> CPU:43 453059 0 54 0 0 0
>> CPU:44 363219 0 34 0 0 0
>> CPU:45 353632 0 38 0 0 0
>> CPU:46 371618 0 40 0 0 0
>> CPU:47 350518 0 46 0 0 0
>> CPU:48 397544 0 40 0 0 0
>> CPU:49 364873 0 38 0 0 0
>> CPU:50 383630 0 38 0 0 0
>> CPU:51 358771 0 39 0 0 0
>> CPU:52 372547 0 38 0 0 0
>> CPU:53 372882 0 36 0 0 0
>> CPU:54 366244 0 43 0 0 0
>> CPU:55 365886 0 39 0 0 0
>>
>> Summed: 11835201 0 1217 0 0 0
>
> Do notice, the per CPU squeeze is not too large.
Yes - but im searching invisible thing now :) something invisible is
slowing down packet processing :)
So trying to find any counter that have something to do with packet
processing.
> The summed 11.8 Mpps is a little high compared to:
>
> Ethtool(enp216s0) stat: 4971677 (4,971,677) <= rx_packets /sec
> Ethtool(enp175s0) stat: 3717148 (3,717,148) <= rx_packets /sec
> Sum: 3717148+4971677 = 8688825 (8,688,825)
Yes i was mentioning this that stats from /net/dev for nics are weird if
u compare them to ethtool - there are big differences for mellanox nic's
Especially with packets/s
For example when i change
- cqe to compress i have more interrupts - same as more packets/s - but
same bw
- change ring settings - like half hour before - changed TX fing from
4096 to 256 and i have less interrupts and less packets but more
bandwidth... weird...
Cause in normal traffic more packets/s need to be more bandwidth - if
average frame is 500-600 if I gain like 1M+pps - then it should mean in
average +5/6Gbit/s more
But it looks like it is more comparable to number of interrupts not
number of packets.
>
> [...]
>>>>> Remember those tests are now on two separate connectx5 connected to
>>>>> two separate pcie x16 gen 3.0
>>>> That is strange... I still suspect some HW NIC issue, can you provide
>>>> ethtool stats info via tool:
>>>>
>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>>>
>>>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>>>>
>>>> The tool remove zero-stats counters and report per sec stats. It makes
>>>> it easier to spot that is relevant for the given workload.
>>> yes mlnx have just too many counters that are always 0 for my case :)
>>> Will try this also
>>>
>> But still alot of non 0 counters
>> Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
>> Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
> [...]
>
> I have copied the stats over in another document so I can better looks
> at it... and I've found some interesting stats.
>
> E.g. we can see that the NIC hardware is dropping packets.
>
> RX-drops on enp175s0:
>
> (enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
> (enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
> -218308971 ( -218,308,971) Dropped bytes /sec
>
> (enp175s0) stat: 139602 ( 139,602) <= rx_discards_phy /sec
>
> (enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
> (enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy /sec
> -145272 ( -145,272) Dropped packets /sec
>
>
> RX-drops on enp216s0 is less:
>
> (enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
> (enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
> -41288962 ( -41,288,962) Dropped bytes /sec
>
> (enp216s0) stat: 464 (464) <= rx_discards_phy /sec
>
> (enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
> (enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy /sec
> -3886 ( -3,886) Dropped packets /sec
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 22:19 ` Paweł Staszewski
@ 2018-11-11 8:03 ` Jesper Dangaard Brouer
2018-11-11 10:26 ` Paweł Staszewski
0 siblings, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-11 8:03 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev, brouer
On Sat, 10 Nov 2018 23:19:50 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 10.11.2018 o 23:06, Jesper Dangaard Brouer pisze:
> > On Sat, 10 Nov 2018 20:56:02 +0100
> > Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >
> >> W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
> >>>
> >>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> >>>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
> >>>> <pstaszewski@itcare.pl> wrote:
> >>>>
> >>>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
[...]
> > Do notice, the per CPU squeeze is not too large.
>
> Yes - but im searching invisible thing now :) something invisible is
> slowing down packet processing :)
> So trying to find any counter that have something to do with packet
> processing.
NOTICE, I have given you the counters you need (below)
> >
> > [...]
> >>>>> Remember those tests are now on two separate connectx5 connected to
> >>>>> two separate pcie x16 gen 3.0
> >>>> That is strange... I still suspect some HW NIC issue, can you provide
> >>>> ethtool stats info via tool:
> >>>>
> >>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> >>>>
> >>>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
> >>>>
> >>>> The tool remove zero-stats counters and report per sec stats. It makes
> >>>> it easier to spot that is relevant for the given workload.
> >>> yes mlnx have just too many counters that are always 0 for my case :)
> >>> Will try this also
> >>>
> >> But still alot of non 0 counters
> >> Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
> >> Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
> > [...]
> >
> > I have copied the stats over in another document so I can better looks
> > at it... and I've found some interesting stats.
> >
> > E.g. we can see that the NIC hardware is dropping packets.
> >
> > RX-drops on enp175s0:
> >
> > (enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
> > (enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
> > -218308971 ( -218,308,971) Dropped bytes /sec
> >
> > (enp175s0) stat: 139602 ( 139,602) <= rx_discards_phy /sec
> >
> > (enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
> > (enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy /sec
> > -145272 ( -145,272) Dropped packets /sec
> >
> >
> > RX-drops on enp216s0 is less:
> >
> > (enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
> > (enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
> > -41288962 ( -41,288,962) Dropped bytes /sec
> >
> > (enp216s0) stat: 464 (464) <= rx_discards_phy /sec
> >
> > (enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
> > (enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy /sec
> > -3886 ( -3,886) Dropped packets /sec
> >
I would recommend, that you use ethtool stats and monitor rx_discards_phy.
The PHY are the counters from the hardware, and it shows that packets
are getting dropped at HW level. This can be because software is not
fast enough to empty RX-queue, but in this case where CPUs are mostly
idle I don't think that is the case.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 21:53 ` Paweł Staszewski
2018-11-10 22:04 ` Paweł Staszewski
@ 2018-11-11 8:56 ` Jesper Dangaard Brouer
2018-11-12 19:19 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: Jesper Dangaard Brouer @ 2018-11-11 8:56 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev, brouer
On Sat, 10 Nov 2018 22:53:53 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> Now im messing with ring configuration for connectx5 nics.
> And after reading that paper:
> https://netdevconf.org/2.1/slides/apr6/network-performance/04-amir-RX_and_TX_bulking_v2.pdf
>
Do notice that some of the ideas in that slide deck, was never
implemented. But they are still on my todo list ;-).
Notice how that it show that TX bulking is very important, but based on
your ethtool_stats.pl, I can see that not much TX bulking is happening
in your case. This is indicated via the xmit_more counters.
Ethtool(enp175s0) stat: 2630 ( 2,630) <= tx_xmit_more /sec
Ethtool(enp175s0) stat: 4956995 ( 4,956,995) <= tx_packets /sec
And the per queue levels are also avail:
Ethtool(enp175s0) stat: 184845 ( 184,845) <= tx7_packets /sec
Ethtool(enp175s0) stat: 78 ( 78) <= tx7_xmit_more /sec
This means that you are doing too many doorbell's to the NIC hardware
at TX time, which I worry could be what cause the NIC and PCIe hardware
not to operate at optimal speeds.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-11 8:03 ` Jesper Dangaard Brouer
@ 2018-11-11 10:26 ` Paweł Staszewski
0 siblings, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-11 10:26 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 11.11.2018 o 09:03, Jesper Dangaard Brouer pisze:
> On Sat, 10 Nov 2018 23:19:50 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 10.11.2018 o 23:06, Jesper Dangaard Brouer pisze:
>>> On Sat, 10 Nov 2018 20:56:02 +0100
>>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>
>>>> W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
>>>>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>>>>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>
>>>>>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> [...]
>>> Do notice, the per CPU squeeze is not too large.
>> Yes - but im searching invisible thing now :) something invisible is
>> slowing down packet processing :)
>> So trying to find any counter that have something to do with packet
>> processing.
> NOTICE, I have given you the counters you need (below)
Yes noticed this :)
>>> [...]
>>>>>>> Remember those tests are now on two separate connectx5 connected to
>>>>>>> two separate pcie x16 gen 3.0
>>>>>> That is strange... I still suspect some HW NIC issue, can you provide
>>>>>> ethtool stats info via tool:
>>>>>>
>>>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>>>>>
>>>>>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>>>>>>
>>>>>> The tool remove zero-stats counters and report per sec stats. It makes
>>>>>> it easier to spot that is relevant for the given workload.
>>>>> yes mlnx have just too many counters that are always 0 for my case :)
>>>>> Will try this also
>>>>>
>>>> But still alot of non 0 counters
>>>> Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
>>>> Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
>>> [...]
>>>
>>> I have copied the stats over in another document so I can better looks
>>> at it... and I've found some interesting stats.
>>>
>>> E.g. we can see that the NIC hardware is dropping packets.
>>>
>>> RX-drops on enp175s0:
>>>
>>> (enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
>>> (enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
>>> -218308971 ( -218,308,971) Dropped bytes /sec
>>>
>>> (enp175s0) stat: 139602 ( 139,602) <= rx_discards_phy /sec
>>>
>>> (enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
>>> (enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy /sec
>>> -145272 ( -145,272) Dropped packets /sec
>>>
>>>
>>> RX-drops on enp216s0 is less:
>>>
>>> (enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
>>> (enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
>>> -41288962 ( -41,288,962) Dropped bytes /sec
>>>
>>> (enp216s0) stat: 464 (464) <= rx_discards_phy /sec
>>>
>>> (enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
>>> (enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy /sec
>>> -3886 ( -3,886) Dropped packets /sec
>>>
>
> I would recommend, that you use ethtool stats and monitor rx_discards_phy.
> The PHY are the counters from the hardware, and it shows that packets
> are getting dropped at HW level. This can be because software is not
> fast enough to empty RX-queue, but in this case where CPUs are mostly
> idle I don't think that is the case.
>
That is why i was searching some counter for software - where is
something wrong.
Cause in earlier reports from ethtool there was also phy drops reported
- just when cpu's was saturated that was normal for me that phy can drop
packets if no more cpu cycles available to pickup them from hw
But in case where i have 50% idle cpu's - there should be no problem -
that is why i start to modify ethtool params for tx/rx ring and coalescence
Currently waiting for more traffic with new ethtool settings:
ethtool -g enp175s0
Ring parameters for enp175s0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 128
ethtool -c enp175s0
Coalesce parameters for enp175s0:
Adaptive RX: off TX: on
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32517
rx-usecs: 64
rx-frames: 128
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 8
tx-frames: 128
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
Both ports same settings.
Current traffic:
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
enp175s0: 37.85 Gb/s 7.77 Gb/s
45.62 Gb/s
enp216s0: 7.80 Gb/s 37.90 Gb/s
45.70 Gb/s
------------------------------------------------------------------------------
total: 45.61 Gb/s 45.63 Gb/s
91.24 Gb/s
and mpstat for cpu's
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.33 0.00 1.48 0.01 0.00 12.11 0.00
0.00 0.00 86.06
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 1 0.00 0.00 0.90 0.00 0.00 0.00 0.00
0.00 0.00 99.10
Average: 2 0.10 0.00 0.20 0.80 0.00 0.00 0.00
0.00 0.00 98.90
Average: 3 0.10 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.60
Average: 4 14.10 0.00 1.00 0.00 0.00 0.00 0.00
0.00 0.00 84.90
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 6 0.00 0.00 1.50 0.00 0.00 0.00 0.00
0.00 0.00 98.50
Average: 7 0.20 0.00 2.00 0.00 0.00 0.00 0.00
0.00 0.00 97.80
Average: 8 0.10 0.00 0.40 0.00 0.00 0.00 0.00
0.00 0.00 99.50
Average: 9 0.00 0.00 0.60 0.00 0.00 0.00 0.00
0.00 0.00 99.40
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.00 0.00 5.60 0.00 0.00 0.00 0.00
0.00 0.00 94.40
Average: 12 0.00 0.00 4.10 0.00 0.00 0.00 0.00
0.00 0.00 95.90
Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 14 0.00 0.00 1.90 0.00 0.00 27.30 0.00
0.00 0.00 70.80
Average: 15 0.00 0.00 2.10 0.00 0.00 26.00 0.00
0.00 0.00 71.90
Average: 16 0.00 0.00 2.10 0.00 0.00 25.40 0.00
0.00 0.00 72.50
Average: 17 0.20 0.00 1.80 0.00 0.00 23.10 0.00
0.00 0.00 74.90
Average: 18 0.00 0.00 2.00 0.00 0.00 25.50 0.00
0.00 0.00 72.50
Average: 19 0.00 0.00 1.90 0.00 0.00 20.20 0.00
0.00 0.00 77.90
Average: 20 0.10 0.00 1.00 0.00 0.00 26.90 0.00
0.00 0.00 72.00
Average: 21 0.10 0.00 2.80 0.00 0.00 24.70 0.00
0.00 0.00 72.40
Average: 22 0.80 0.00 3.30 0.00 0.00 24.30 0.00
0.00 0.00 71.60
Average: 23 0.10 0.00 1.80 0.00 0.00 26.60 0.00
0.00 0.00 71.50
Average: 24 0.10 0.00 1.20 0.00 0.00 23.60 0.00
0.00 0.00 75.10
Average: 25 0.00 0.00 1.80 0.00 0.00 26.60 0.00
0.00 0.00 71.60
Average: 26 0.00 0.00 1.50 0.00 0.00 26.70 0.00
0.00 0.00 71.80
Average: 27 0.10 0.00 0.70 0.00 0.00 26.70 0.00
0.00 0.00 72.50
Average: 28 0.70 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 29 0.20 0.00 1.50 0.00 0.00 0.00 0.00
0.00 0.00 98.30
Average: 30 0.10 0.00 0.60 0.00 0.00 0.00 0.00
0.00 0.00 99.30
Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 38 0.00 0.00 2.80 0.00 0.00 0.00 0.00
0.00 0.00 97.20
Average: 39 0.00 0.00 7.40 0.00 0.00 0.00 0.00
0.00 0.00 92.60
Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 42 0.00 0.00 2.10 0.00 0.00 28.40 0.00
0.00 0.00 69.50
Average: 43 0.00 0.00 1.60 0.00 0.00 25.00 0.00
0.00 0.00 73.40
Average: 44 0.10 0.00 1.60 0.00 0.00 23.90 0.00
0.00 0.00 74.40
Average: 45 0.00 0.00 1.60 0.00 0.00 21.00 0.00
0.00 0.00 77.40
Average: 46 0.00 0.00 2.20 0.00 0.00 28.00 0.00
0.00 0.00 69.80
Average: 47 0.00 0.00 2.80 0.00 0.00 20.30 0.00
0.00 0.00 76.90
Average: 48 0.00 0.00 2.50 0.00 0.00 21.60 0.00
0.00 0.00 75.90
Average: 49 0.00 0.00 0.80 0.00 0.00 22.50 0.00
0.00 0.00 76.70
Average: 50 0.40 0.00 3.00 0.00 0.00 23.50 0.00
0.00 0.00 73.10
Average: 51 0.60 0.00 2.50 0.00 0.00 25.00 0.00
0.00 0.00 71.90
Average: 52 0.10 0.00 1.30 0.00 0.00 20.70 0.00
0.00 0.00 77.90
Average: 53 0.00 0.00 2.20 0.00 0.00 22.80 0.00
0.00 0.00 75.00
Average: 54 0.00 0.00 1.40 0.00 0.00 20.80 0.00
0.00 0.00 77.80
Average: 55 0.00 0.00 2.10 0.00 0.00 21.30 0.00
0.00 0.00 76.60
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-11 8:56 ` Jesper Dangaard Brouer
@ 2018-11-12 19:19 ` Paweł Staszewski
0 siblings, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-12 19:19 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev
W dniu 11.11.2018 o 09:56, Jesper Dangaard Brouer pisze:
> On Sat, 10 Nov 2018 22:53:53 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> Now im messing with ring configuration for connectx5 nics.
>> And after reading that paper:
>> https://netdevconf.org/2.1/slides/apr6/network-performance/04-amir-RX_and_TX_bulking_v2.pdf
>>
> Do notice that some of the ideas in that slide deck, was never
> implemented. But they are still on my todo list ;-).
>
> Notice how that it show that TX bulking is very important, but based on
> your ethtool_stats.pl, I can see that not much TX bulking is happening
> in your case. This is indicated via the xmit_more counters.
>
> Ethtool(enp175s0) stat: 2630 ( 2,630) <= tx_xmit_more /sec
> Ethtool(enp175s0) stat: 4956995 ( 4,956,995) <= tx_packets /sec
>
> And the per queue levels are also avail:
>
> Ethtool(enp175s0) stat: 184845 ( 184,845) <= tx7_packets /sec
> Ethtool(enp175s0) stat: 78 ( 78) <= tx7_xmit_more /sec
>
> This means that you are doing too many doorbell's to the NIC hardware
> at TX time, which I worry could be what cause the NIC and PCIe hardware
> not to operate at optimal speeds.
After tunning coal/ring a little with ethtool
Reached today:
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
enp175s0: 50.68 Gb/s 21.53 Gb/s
72.20 Gb/s
enp216s0: 21.62 Gb/s 50.81 Gb/s
72.42 Gb/s
------------------------------------------------------------------------------
total: 72.30 Gb/s 72.33 Gb/s
144.63 Gb/s
And still no packet loss (icmp side to side test every 100ms)
Below perf top
PerfTop: 104692 irqs/sec kernel:99.5% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
9.06% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
6.43% [kernel] [k] tasklet_action_common.isra.21
5.68% [kernel] [k] fib_table_lookup
4.89% [kernel] [k] irq_entries_start
4.53% [kernel] [k] mlx5_eq_int
4.10% [kernel] [k] build_skb
3.39% [kernel] [k] mlx5e_poll_tx_cq
3.38% [kernel] [k] mlx5e_sq_xmit
2.73% [kernel] [k] mlx5e_poll_rx_cq
2.18% [kernel] [k] __dev_queue_xmit
2.13% [kernel] [k] vlan_do_receive
2.12% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
2.00% [kernel] [k] ip_finish_output2
1.87% [kernel] [k] mlx5e_post_rx_mpwqes
1.86% [kernel] [k] memcpy_erms
1.85% [kernel] [k] ipt_do_table
1.70% [kernel] [k] dev_gro_receive
1.39% [kernel] [k] __netif_receive_skb_core
1.31% [kernel] [k] inet_gro_receive
1.21% [kernel] [k] ip_route_input_rcu
1.21% [kernel] [k] tcp_gro_receive
1.13% [kernel] [k] _raw_spin_lock
1.08% [kernel] [k] __build_skb
1.06% [kernel] [k] kmem_cache_free_bulk
1.05% [kernel] [k] __softirqentry_text_start
1.03% [kernel] [k] vlan_dev_hard_start_xmit
0.98% [kernel] [k] pfifo_fast_dequeue
0.95% [kernel] [k] mlx5e_xmit
0.95% [kernel] [k] page_frag_free
0.88% [kernel] [k] ip_forward
0.81% [kernel] [k] dev_hard_start_xmit
0.78% [kernel] [k] rcu_irq_exit
0.77% [kernel] [k] netif_skb_features
0.72% [kernel] [k] napi_complete_done
0.72% [kernel] [k] kmem_cache_alloc
0.68% [kernel] [k] validate_xmit_skb.isra.142
0.66% [kernel] [k] ip_rcv_core.isra.20.constprop.25
0.58% [kernel] [k] swiotlb_map_page
0.57% [kernel] [k] __qdisc_run
0.56% [kernel] [k] tasklet_action
0.54% [kernel] [k] __get_xps_queue_idx
0.54% [kernel] [k] inet_lookup_ifaddr_rcu
0.50% [kernel] [k] tcp4_gro_receive
0.49% [kernel] [k] skb_release_data
0.47% [kernel] [k] eth_type_trans
0.40% [kernel] [k] sch_direct_xmit
0.40% [kernel] [k] net_rx_action
0.39% [kernel] [k] __local_bh_enable_ip
And perf record/report
https://ufile.io/zguq0
So now i know what was causing cpu load for some processes like:
2913 root 20 0 0 0 0 I 10.3 0.0 6:58.29
kworker/u112:1-
7 root 20 0 0 0 0 I 8.6 0.0 6:17.18
kworker/u112:0-
10289 root 20 0 0 0 0 I 6.6 0.0 6:33.90
kworker/u112:4-
2939 root 20 0 0 0 0 R 3.6 0.0 7:37.68
kworker/u112:2-
After disabling adaptative tx for coalescense - all this processes gone.
lavg drops from 40 to 1
Current settings for coalescence:
ethtool -c enp175s0
Coalesce parameters for enp175s0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32548
rx-usecs: 24
rx-frames: 256
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 0
tx-frames: 64
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
And currently with that traffiv lvls - have no packet loss (cpu is avg.
60% for all 28 cores)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-10 0:06 ` David Ahern
2018-11-10 13:18 ` Paweł Staszewski
@ 2018-11-19 21:59 ` David Ahern
2018-11-20 23:00 ` Paweł Staszewski
1 sibling, 1 reply; 77+ messages in thread
From: David Ahern @ 2018-11-19 21:59 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
On 11/9/18 5:06 PM, David Ahern wrote:
> On 11/9/18 9:21 AM, David Ahern wrote:
>>> Is there possible to add only counters from xdp for vlans ?
>>> This will help me in testing.
>> I will take a look today at adding counters that you can dump using
>> bpftool. It will be a temporary solution for this xdp program only.
>>
>
> Same tree, kernel-tables-wip-02 branch. Compile kernel and install.
> Compile samples as before.
new version:
https://github.com/dsahern/linux.git bpf/kernel-tables-wip-03
This one prototypes incrementing counters for VLAN devices (rx/tx,
packets and bytes). Counters for netdevices representing physical ports
should be managed by the NIC driver.
I will look at what can be done for packet captures (e.g., xdpdump and
https://github.com/facebookincubator/katran/tree/master/tools). Most
likely a project for next week.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
2018-11-19 21:59 ` David Ahern
@ 2018-11-20 23:00 ` Paweł Staszewski
0 siblings, 0 replies; 77+ messages in thread
From: Paweł Staszewski @ 2018-11-20 23:00 UTC (permalink / raw)
To: David Ahern, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
W dniu 19.11.2018 o 22:59, David Ahern pisze:
> On 11/9/18 5:06 PM, David Ahern wrote:
>> On 11/9/18 9:21 AM, David Ahern wrote:
>>>> Is there possible to add only counters from xdp for vlans ?
>>>> This will help me in testing.
>>> I will take a look today at adding counters that you can dump using
>>> bpftool. It will be a temporary solution for this xdp program only.
>>>
>> Same tree, kernel-tables-wip-02 branch. Compile kernel and install.
>> Compile samples as before.
> new version:
> https://github.com/dsahern/linux.git bpf/kernel-tables-wip-03
>
> This one prototypes incrementing counters for VLAN devices (rx/tx,
> packets and bytes). Counters for netdevices representing physical ports
> should be managed by the NIC driver.
Will test it today
Thanks
Paweł
>
> I will look at what can be done for packet captures (e.g., xdpdump and
> https://github.com/facebookincubator/katran/tree/master/tools). Most
> likely a project for next week.
>
^ permalink raw reply [flat|nested] 77+ messages in thread
end of thread, other threads:[~2018-11-21 9:32 UTC | newest]
Thread overview: 77+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-31 21:57 Kernel 4.19 network performance - forwarding/routing normal users traffic Paweł Staszewski
2018-10-31 22:09 ` Eric Dumazet
2018-10-31 22:20 ` Paweł Staszewski
2018-10-31 22:45 ` Paweł Staszewski
2018-11-01 9:22 ` Jesper Dangaard Brouer
2018-11-01 10:34 ` Paweł Staszewski
2018-11-01 15:27 ` Aaron Lu
2018-11-01 20:23 ` Saeed Mahameed
2018-11-02 5:23 ` Aaron Lu
2018-11-02 11:40 ` Jesper Dangaard Brouer
2018-11-02 14:20 ` Aaron Lu
2018-11-02 19:02 ` Paweł Staszewski
2018-11-03 0:16 ` Paweł Staszewski
2018-11-03 12:01 ` Paweł Staszewski
2018-11-03 12:58 ` Jesper Dangaard Brouer
2018-11-03 15:23 ` Paweł Staszewski
2018-11-03 15:43 ` Paweł Staszewski
2018-11-03 12:53 ` Jesper Dangaard Brouer
2018-11-05 6:28 ` Aaron Lu
2018-11-05 9:10 ` Jesper Dangaard Brouer
2018-11-05 8:42 ` Tariq Toukan
2018-11-05 8:48 ` Aaron Lu
2018-11-01 3:37 ` David Ahern
2018-11-01 10:55 ` Jesper Dangaard Brouer
2018-11-01 13:52 ` Paweł Staszewski
2018-11-01 17:23 ` David Ahern
2018-11-01 17:30 ` Paweł Staszewski
2018-11-03 17:32 ` David Ahern
2018-11-04 0:24 ` Paweł Staszewski
2018-11-05 20:17 ` Jesper Dangaard Brouer
2018-11-08 0:59 ` Paweł Staszewski
2018-11-08 1:13 ` Paweł Staszewski
2018-11-08 14:43 ` Paweł Staszewski
2018-11-07 21:06 ` David Ahern
2018-11-08 13:33 ` Paweł Staszewski
2018-11-08 16:06 ` David Ahern
2018-11-08 16:25 ` Paweł Staszewski
2018-11-08 16:27 ` Paweł Staszewski
2018-11-08 16:32 ` David Ahern
2018-11-08 17:30 ` Paweł Staszewski
2018-11-08 18:05 ` David Ahern
2018-11-09 0:40 ` Paweł Staszewski
2018-11-09 0:42 ` David Ahern
2018-11-09 4:52 ` Saeed Mahameed
2018-11-09 7:52 ` Jesper Dangaard Brouer
2018-11-09 9:56 ` Paweł Staszewski
2018-11-09 10:20 ` Paweł Staszewski
2018-11-09 16:21 ` David Ahern
2018-11-09 19:59 ` Paweł Staszewski
2018-11-10 0:06 ` David Ahern
2018-11-10 13:18 ` Paweł Staszewski
2018-11-10 14:56 ` David Ahern
2018-11-19 21:59 ` David Ahern
2018-11-20 23:00 ` Paweł Staszewski
2018-11-01 9:50 ` Saeed Mahameed
2018-11-01 11:09 ` Paweł Staszewski
2018-11-01 16:49 ` Paweł Staszewski
2018-11-01 20:37 ` Saeed Mahameed
2018-11-01 21:18 ` Paweł Staszewski
2018-11-01 21:24 ` Paweł Staszewski
2018-11-01 21:34 ` Paweł Staszewski
2018-11-03 0:18 ` Paweł Staszewski
2018-11-08 19:12 ` Paweł Staszewski
2018-11-09 22:20 ` Paweł Staszewski
2018-11-10 19:34 ` Jesper Dangaard Brouer
2018-11-10 19:49 ` Paweł Staszewski
2018-11-10 19:56 ` Paweł Staszewski
2018-11-10 22:06 ` Jesper Dangaard Brouer
2018-11-10 22:19 ` Paweł Staszewski
2018-11-11 8:03 ` Jesper Dangaard Brouer
2018-11-11 10:26 ` Paweł Staszewski
2018-11-10 20:02 ` Paweł Staszewski
2018-11-10 21:01 ` Jesper Dangaard Brouer
2018-11-10 21:53 ` Paweł Staszewski
2018-11-10 22:04 ` Paweł Staszewski
2018-11-11 8:56 ` Jesper Dangaard Brouer
2018-11-12 19:19 ` Paweł Staszewski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.