All of lore.kernel.org
 help / color / mirror / Atom feed
* Why our soft-RoCE throughput is quite low compared with TCP
@ 2019-11-15 13:26 QWang
  2019-11-15 16:07 ` Leon Romanovsky
  0 siblings, 1 reply; 10+ messages in thread
From: QWang @ 2019-11-15 13:26 UTC (permalink / raw)
  To: linux-rdma

Dear experts on RDMA,
       We are sorry to disturb you. Because of a project, we need to 
integrate soft-RoCE in our system. However ,we are very confused by our 
soft-RoCE throughput results, which are quite low compared with TCP 
throughput. The throughput of soft-RoCE in our tests measured by 
ib_send_bw and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 
Gbps and the two Xeon E5 servers with Mellanox ConnectX-4 cards are 
connected via back-to-back, the OS is ubuntu16.04 with kernel 
4.15.0-041500-generic). The throughput of hard-RoCE and TCP are normal, 
which are 100 Gbps and 20 Gbps, respectively. But in the figure 6 in the 
attached paper "A Performance Comparison of Container Networking 
Alternatives", the throughput of soft-RoCE can be up to 23 Gbps.  In our 
tests, we get the open-source soft-RoCE from github in 
https://github.com/linux-rdma. Do you know how can we get such high 
bandwidth? Do we need to configure some OS system settings?
       We find that in 2017, someone finds the same problem and he posts 
all his detailed results on 
https://bugzilla.kernel.org/show_bug.cgi?id=190951   . But it remains 
unsolved. His results are nearly the same with our's. For simplicity,  
we do not post our results in this email. You can get very detailed 
information in the web page listed above.
       We are very confused by our results. We will very appreciate it 
if we can receive your early reply. Best wishes,
Wang Qi


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-15 13:26 Why our soft-RoCE throughput is quite low compared with TCP QWang
@ 2019-11-15 16:07 ` Leon Romanovsky
  2019-11-18  6:38   ` [question]Why " wangqi
  0 siblings, 1 reply; 10+ messages in thread
From: Leon Romanovsky @ 2019-11-15 16:07 UTC (permalink / raw)
  To: QWang; +Cc: linux-rdma

On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
> Dear experts on RDMA,
>       We are sorry to disturb you. Because of a project, we need to
> integrate soft-RoCE in our system. However ,we are very confused by our
> soft-RoCE throughput results, which are quite low compared with TCP
> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
> respectively. But in the figure 6 in the attached paper "A Performance
> Comparison of Container Networking Alternatives", the throughput of
> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
> we get such high bandwidth? Do we need to configure some OS system settings?
>       We find that in 2017, someone finds the same problem and he posts all
> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
> . But it remains unsolved. His results are nearly the same with our's. For
> simplicity,  we do not post our results in this email. You can get very
> detailed information in the web page listed above.
>       We are very confused by our results. We will very appreciate it if we
> can receive your early reply. Best wishes,
> Wang Qi

Can you please fix your email client?
The email text looks like one big sentence.

From the perf report attached to this bugzilla, looks like RXE does a
lot of CRC32 calculations and it is consistent with what Matan said
a long time ago, RXE "stuck" in ICRC calculations required by spec.

I'm curios what are your CONFIG_CRYPTO_* configs?

Thanks

>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-15 16:07 ` Leon Romanovsky
@ 2019-11-18  6:38   ` wangqi
  2019-11-18  9:49     ` Leon Romanovsky
  0 siblings, 1 reply; 10+ messages in thread
From: wangqi @ 2019-11-18  6:38 UTC (permalink / raw)
  To: Leon Romanovsky; +Cc: linux-rdma

[-- Attachment #1: Type: text/plain, Size: 3452 bytes --]

On 2019/11/16 上午12:07, Leon Romanovsky wrote:

> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
>> Dear experts on RDMA,
>>       We are sorry to disturb you. Because of a project, we need to
>> integrate soft-RoCE in our system. However ,we are very confused by our
>> soft-RoCE throughput results, which are quite low compared with TCP
>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
>> respectively. But in the figure 6 in the attached paper "A Performance
>> Comparison of Container Networking Alternatives", the throughput of
>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
>> we get such high bandwidth? Do we need to configure some OS system settings?
>>       We find that in 2017, someone finds the same problem and he posts all
>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
>> . But it remains unsolved. His results are nearly the same with our's. For
>> simplicity,  we do not post our results in this email. You can get very
>> detailed information in the web page listed above.
>>       We are very confused by our results. We will very appreciate it if we
>> can receive your early reply. Best wishes,
>> Wang Qi
> Can you please fix your email client?
> The email text looks like one big sentence.
>
> From the perf report attached to this bugzilla, looks like RXE does a
> lot of CRC32 calculations and it is consistent with what Matan said
> a long time ago, RXE "stuck" in ICRC calculations required by spec.
>
> I'm curios what are your CONFIG_CRYPTO_* configs?
>
> ThanksCONFIG_CRYPTO_* configs
>
>


I'm sorry for the editor problem in my last email. Now I use another editor.

We get our rdma-core and perftest from

https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.

We attach five files to clarify our problem.

* The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP

and softroce throughput in our two servers (connected via back to back).

* The second file "server_CONFIG_CRYPTO_result.txt" is the

CONFIG_CRYPTO_* config results in the two servers..

* The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20

& perf record -ags sleep 10 & wait" results in our two servers (we use

"perf report --header >perf" to make the file).

* The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP

and softroce throughput in two virtual machines with the latest linux kernel

5.4.0-rc7

(we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).

* The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual

machines.

* The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228

& perf record -ags sleep 10 & wait " result in the two virtual machines.

On the other side, we tried to use the rxe command "rxe_cfg crc disable"

in the two servers, but it didn't improve the throughput performance

at all. 

Qi



[-- Attachment #2: server_tcp_vs_softroce_performance.txt --]
[-- Type: text/plain, Size: 2975 bytes --]

12:52 root@train-gpu9:~ # iperf -c 192.168.0.20
------------------------------------------------------------
Client connecting to 192.168.0.20, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.19 port 52514 connected with 192.168.0.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  22.3 GBytes  19.2 Gbits/sec
12:52 root@train-gpu9:~ # ib_send_bw -a 192.168.0.20
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF          Device         : rxe0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0012 PSN 0x986722
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:00:19
 remote address: LID 0000 QPN 0x0011 PSN 0xe3c4d3
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:00:20
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000             0.34               0.31               0.163311
 4          1000             0.63               0.63               0.166140
 8          1000             1.24               1.24               0.162059
 16         1000             2.54               2.54               0.166631
 32         1000             4.95               4.95               0.162102
 64         1000             9.85               9.85               0.161369
 128        1000             19.49              19.48              0.159607
 256        1000             38.48              38.47              0.157580
 512        1000             76.76              76.66              0.156990
 1024       1000             151.62             149.43             0.153018
 2048       1000             179.50             179.50             0.091904
 4096       1000             256.69             249.99             0.063996
 8192       1000             254.56             252.30             0.032295
 16384      1000             288.98             284.32             0.018196
 32768      1000             306.12             306.10             0.009795
 65536      1000             310.75             310.50             0.004968
 131072     1000             311.21             311.21             0.002490
 262144     1000             310.83             310.79             0.001243
 524288     1000             311.09             310.87             0.000622
 1048576    1000             310.78             310.55             0.000311
^C
12:53 root@train-gpu9:~ #

[-- Attachment #3: server_CONFIG_CRYPTO_result.txt --]
[-- Type: text/plain, Size: 4250 bytes --]

 root@train-gpu10:/usr/src# cat /boot/config-4.15.0-041500-generic  | grep CONFIG_CRYPTO_
CONFIG_CRYPTO_SKEIN=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=y
CONFIG_CRYPTO_AKCIPHER2=y
CONFIG_CRYPTO_AKCIPHER=y
CONFIG_CRYPTO_KPP2=y
CONFIG_CRYPTO_KPP=y
CONFIG_CRYPTO_ACOMP2=y
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_DH=y
CONFIG_CRYPTO_ECDH=m
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=y
CONFIG_CRYPTO_NULL2=y
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_MCRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_ABLK_HELPER=m
CONFIG_CRYPTO_SIMD=m
CONFIG_CRYPTO_GLUE_HELPER_X86=m
CONFIG_CRYPTO_ENGINE=m
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=y
CONFIG_CRYPTO_CHACHA20POLY1305=m
CONFIG_CRYPTO_SEQIV=y
CONFIG_CRYPTO_ECHAINIV=m
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=y
CONFIG_CRYPTO_KEYWRAP=m
CONFIG_CRYPTO_CMAC=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=y
CONFIG_CRYPTO_CRC32=m
CONFIG_CRYPTO_CRC32_PCLMUL=m
CONFIG_CRYPTO_CRCT10DIF=y
CONFIG_CRYPTO_CRCT10DIF_PCLMUL=m
CONFIG_CRYPTO_GHASH=y
CONFIG_CRYPTO_POLY1305=m
CONFIG_CRYPTO_POLY1305_X86_64=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m
CONFIG_CRYPTO_SHA1_MB=m
CONFIG_CRYPTO_SHA256_MB=m
CONFIG_CRYPTO_SHA512_MB=m
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_SHA3=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_TI=m
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAMELLIA_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST5_AVX_X86_64=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_CAST6_AVX_X86_64=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_DES3_EDE_X86_64=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_X86_64=m
CONFIG_CRYPTO_CHACHA20=m
CONFIG_CRYPTO_CHACHA20_X86_64=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX2_X86_64=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_842=m
CONFIG_CRYPTO_LZ4=m
CONFIG_CRYPTO_LZ4HC=m
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_DRBG_MENU=y
CONFIG_CRYPTO_DRBG_HMAC=y
CONFIG_CRYPTO_DRBG_HASH=y
CONFIG_CRYPTO_DRBG_CTR=y
CONFIG_CRYPTO_DRBG=y
CONFIG_CRYPTO_JITTERENTROPY=y
CONFIG_CRYPTO_USER_API=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_HASH_INFO=y
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=y
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
# CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_DESC is not set
CONFIG_CRYPTO_DEV_CCP=y
CONFIG_CRYPTO_DEV_CCP_DD=m
CONFIG_CRYPTO_DEV_SP_CCP=y
CONFIG_CRYPTO_DEV_CCP_CRYPTO=m
CONFIG_CRYPTO_DEV_QAT=m
CONFIG_CRYPTO_DEV_QAT_DH895xCC=m
# CONFIG_CRYPTO_DEV_QAT_C3XXX is not set
# CONFIG_CRYPTO_DEV_QAT_C62X is not set
CONFIG_CRYPTO_DEV_QAT_DH895xCCVF=m
# CONFIG_CRYPTO_DEV_QAT_C3XXXVF is not set
# CONFIG_CRYPTO_DEV_QAT_C62XVF is not set
CONFIG_CRYPTO_DEV_NITROX=m
CONFIG_CRYPTO_DEV_NITROX_CNN55XX=m
CONFIG_CRYPTO_DEV_CHELSIO=m
CONFIG_CRYPTO_DEV_VIRTIO=m




[-- Attachment #4: server_perf.txt --]
[-- Type: text/plain, Size: 481529 bytes --]

# ========
# captured on: Mon Nov 18 09:20:12 2019
# hostname : train-gpu9
# os release : 4.15.0-041500-generic
# perf version : 4.15.0
# arch : x86_64
# nrcpus online : 32
# nrcpus avail : 32
# cpudesc : Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
# cpuid : GenuineIntel,6,79,1
# total memory : 263890636 kB
# cmdline : /root/softroce/linux-4.15/tools/perf/perf record -ags sleep 10 
# event : name = cycles:ppp, , id = { 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249 }, size = 112, { sample_period, sample_freq } = 4000, sample_type = IP|TID|TIME|CALLCHAIN|CPU|PERIOD, read_format = TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID, disabled = 1, inherit = 1, mmap = 1, comm = 1, freq = 1, inherit_stat = 1, task = 1, precise_ip = 3, sample_id_all = 1, exclude_guest = 1, mmap2 = 1, comm_exec = 1
# CPU_TOPOLOGY info available, use -I to display
# NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: intel_pt = 6, uncore_imc_4 = 18, uncore_cbox_5 = 25, uncore_ha_0 = 13, uncore_imc_2 = 16, uncore_cbox_3 = 23, uncore_qpi_0 = 8, cstate_pkg = 31, breakpoint = 5, uncore_imc_0 = 14, uncore_ubox = 19, uncore_pcu = 28, uncore_cbox_1 = 21, uncore_r3qpi_0 = 10, power = 29, cpu = 4, software = 1, uncore_cbox_6 = 26, uncore_r2pcie = 12, uncore_imc_3 = 17, uncore_cbox_4 = 24, uncore_qpi_1 = 9, uncore_imc_1 = 15, uncore_cbox_2 = 22, uncore_r3qpi_1 = 11, uncore_cbox_0 = 20, cstate_core = 30, tracepoint = 2, msr = 7, uncore_cbox_7 = 27
# CACHE info available, use -I to display
# missing features: TRACING_DATA BRANCH_STACK GROUP_DESC AUXTRACE STAT 
# ========
#
#
# Total Lost Samples: 0
#
# Samples: 14K of event 'cycles:ppp'
# Event count (approx.): 6330923850
#
# Children      Self  Command          Shared Object            Symbol                                                                                                                           
# ........  ........  ...............  .......................  .................................................................................................................................
#
    52.49%     0.00%  ib_send_bw       [unknown]                [k] 0000000000000000
            |
            ---0
               |          
               |--39.07%--run_iter_bw
               |          
               |--11.24%--rxe_query_device
               |          0
               |          |          
               |          |--8.12%--rxe_poll_cq
               |          |          
               |          |--1.83%--pthread_spin_lock@plt
               |          |          
               |           --1.02%--pthread_spin_lock
               |          
                --2.12%--rxe_poll_cq

    39.07%    39.07%  ib_send_bw       ib_send_bw               [.] run_iter_bw
            |
            ---0
               run_iter_bw

    20.08%    20.08%  ib_send_bw       ib_send_bw               [.] print_report_bw
            |
            ---0x2200be0
               print_report_bw

    20.08%     0.00%  ib_send_bw       [unknown]                [.] 0x0000000002200be0
            |
            ---0x2200be0
               print_report_bw

    14.79%    14.79%  ib_send_bw       librxe-rdmav25.so        [.] rxe_poll_cq
            |          
            |--10.23%--0
            |          |          
            |          |--8.12%--rxe_query_device
            |          |          0
            |          |          rxe_poll_cq
            |          |          
            |           --2.12%--rxe_poll_cq
            |          
             --4.55%--rxe_poll_cq

    11.24%     0.00%  ib_send_bw       librxe-rdmav25.so        [.] rxe_query_device
            |
            ---rxe_query_device
               0
               |          
               |--8.12%--rxe_poll_cq
               |          
               |--1.83%--pthread_spin_lock@plt
               |          
                --1.02%--pthread_spin_lock

     6.35%     1.41%  ib_send_bw       ib_send_bw               [.] sample_get_cpu_mhz
            |
            ---sample_get_cpu_mhz
               |          
                --4.82%--__vdso_gettimeofday
                          |          
                           --1.12%--0x974

     6.31%     0.00%  swapper          [kernel.kallsyms]        [k] secondary_startup_64
            |
            ---secondary_startup_64
               |          
                --6.17%--start_secondary
                          |          
                           --6.16%--cpu_startup_entry
                                     |          
                                      --6.15%--do_idle
                                                |          
                                                 --5.90%--default_idle_call
                                                           |          
                                                            --5.90%--arch_cpu_idle
                                                                      |          
                                                                       --5.89%--default_idle
                                                                                 |          
                                                                                  --5.86%--native_safe_halt
                                                                                            |          
                                                                                             --1.71%--ret_from_intr
                                                                                                       do_IRQ
                                                                                                       |          
                                                                                                        --1.60%--irq_exit
                                                                                                                  |          
                                                                                                                   --1.50%--__softirqentry_text_start
                                                                                                                             |          
                                                                                                                              --1.00%--tasklet_action
                                                                                                                                        |          
                                                                                                                                         --0.99%--rxe_do_task
                                                                                                                                                   rxe_requester
                                                                                                                                                   |          
                                                                                                                                                    --0.63%--rxe_send
                                                                                                                                                              |          
                                                                                                                                                               --0.58%--ip_local_out

     6.30%     0.00%  swapper          [kernel.kallsyms]        [k] cpu_startup_entry
            |          
             --6.30%--cpu_startup_entry
                       |          
                        --6.29%--do_idle
                                  |          
                                   --6.03%--default_idle_call
                                             |          
                                              --6.03%--arch_cpu_idle
                                                        |          
                                                         --6.03%--default_idle
                                                                   |          
                                                                    --5.99%--native_safe_halt
                                                                              |          
                                                                               --1.72%--ret_from_intr
                                                                                         do_IRQ
                                                                                         |          
                                                                                          --1.61%--irq_exit
                                                                                                    |          
                                                                                                     --1.52%--__softirqentry_text_start
                                                                                                               |          
                                                                                                                --1.00%--tasklet_action
                                                                                                                          |          
                                                                                                                           --0.99%--rxe_do_task
                                                                                                                                     rxe_requester
                                                                                                                                     |          
                                                                                                                                      --0.63%--rxe_send
                                                                                                                                                |          
                                                                                                                                                 --0.58%--ip_local_out

     6.30%     0.03%  swapper          [kernel.kallsyms]        [k] do_idle
            |          
             --6.27%--do_idle
                       |          
                        --6.03%--default_idle_call
                                  |          
                                   --6.03%--arch_cpu_idle
                                             |          
                                              --6.03%--default_idle
                                                        |          
                                                         --5.99%--native_safe_halt
                                                                   |          
                                                                    --1.72%--ret_from_intr
                                                                              do_IRQ
                                                                              |          
                                                                               --1.61%--irq_exit
                                                                                         |          
                                                                                          --1.52%--__softirqentry_text_start
                                                                                                    |          
                                                                                                     --1.00%--tasklet_action
                                                                                                               |          
                                                                                                                --0.99%--rxe_do_task
                                                                                                                          rxe_requester
                                                                                                                          |          
                                                                                                                           --0.63%--rxe_send
                                                                                                                                     |          
                                                                                                                                      --0.58%--ip_local_out

     6.17%     0.00%  swapper          [kernel.kallsyms]        [k] start_secondary
            |
            ---start_secondary
               |          
                --6.16%--cpu_startup_entry
                          |          
                           --6.15%--do_idle
                                     |          
                                      --5.90%--default_idle_call
                                                |          
                                                 --5.90%--arch_cpu_idle
                                                           |          
                                                            --5.89%--default_idle
                                                                      |          
                                                                       --5.86%--native_safe_halt
                                                                                 |          
                                                                                  --1.71%--ret_from_intr
                                                                                            do_IRQ
                                                                                            |          
                                                                                             --1.60%--irq_exit
                                                                                                       |          
                                                                                                        --1.50%--__softirqentry_text_start
                                                                                                                  |          
                                                                                                                   --1.00%--tasklet_action
                                                                                                                             |          
                                                                                                                              --0.99%--rxe_do_task
                                                                                                                                        rxe_requester
                                                                                                                                        |          
                                                                                                                                         --0.63%--rxe_send
                                                                                                                                                   |          
                                                                                                                                                    --0.58%--ip_local_out

     6.04%     0.01%  swapper          [kernel.kallsyms]        [k] default_idle_call
            |          
             --6.03%--default_idle_call
                       |          
                        --6.03%--arch_cpu_idle
                                  |          
                                   --6.03%--default_idle
                                             |          
                                              --5.99%--native_safe_halt
                                                        |          
                                                         --1.72%--ret_from_intr
                                                                   do_IRQ
                                                                   |          
                                                                    --1.61%--irq_exit
                                                                              |          
                                                                               --1.52%--__softirqentry_text_start
                                                                                         |          
                                                                                          --1.00%--tasklet_action
                                                                                                    |          
                                                                                                     --0.99%--rxe_do_task
                                                                                                               rxe_requester
                                                                                                               |          
                                                                                                                --0.63%--rxe_send
                                                                                                                          |          
                                                                                                                           --0.58%--ip_local_out

     6.04%     0.01%  swapper          [kernel.kallsyms]        [k] arch_cpu_idle
            |          
             --6.03%--arch_cpu_idle
                       |          
                        --6.03%--default_idle
                                  |          
                                   --5.99%--native_safe_halt
                                             |          
                                              --1.72%--ret_from_intr
                                                        do_IRQ
                                                        |          
                                                         --1.61%--irq_exit
                                                                   |          
                                                                    --1.52%--__softirqentry_text_start
                                                                              |          
                                                                               --1.00%--tasklet_action
                                                                                         |          
                                                                                          --0.99%--rxe_do_task
                                                                                                    rxe_requester
                                                                                                    |          
                                                                                                     --0.63%--rxe_send
                                                                                                               |          
                                                                                                                --0.58%--ip_local_out

     6.03%     0.00%  swapper          [kernel.kallsyms]        [k] default_idle
            |          
             --6.02%--default_idle
                       |          
                        --5.99%--native_safe_halt
                                  |          
                                   --1.72%--ret_from_intr
                                             do_IRQ
                                             |          
                                              --1.61%--irq_exit
                                                        |          
                                                         --1.52%--__softirqentry_text_start
                                                                   |          
                                                                    --1.00%--tasklet_action
                                                                              |          
                                                                               --0.99%--rxe_do_task
                                                                                         rxe_requester
                                                                                         |          
                                                                                          --0.63%--rxe_send
                                                                                                    |          
                                                                                                     --0.58%--ip_local_out

     6.00%     3.95%  swapper          [kernel.kallsyms]        [k] native_safe_halt
            |          
            |--3.95%--secondary_startup_64
            |          |          
            |           --3.85%--start_secondary
            |                     cpu_startup_entry
            |                     do_idle
            |                     default_idle_call
            |                     arch_cpu_idle
            |                     |          
            |                      --3.84%--default_idle
            |                                native_safe_halt
            |          
             --2.05%--native_safe_halt
                       |          
                        --1.72%--ret_from_intr
                                  do_IRQ
                                  |          
                                   --1.61%--irq_exit
                                             |          
                                              --1.52%--__softirqentry_text_start
                                                        |          
                                                         --1.00%--tasklet_action
                                                                   |          
                                                                    --0.99%--rxe_do_task
                                                                              rxe_requester
                                                                              |          
                                                                               --0.63%--rxe_send
                                                                                         |          
                                                                                          --0.58%--ip_local_out

     5.27%     3.97%  ib_send_bw       [vdso]                   [.] __vdso_gettimeofday
            |          
            |--3.52%--sample_get_cpu_mhz
            |          __vdso_gettimeofday
            |          
             --1.75%--__vdso_gettimeofday
                       |          
                        --1.12%--0x974

     1.83%     1.83%  ib_send_bw       librxe-rdmav25.so        [.] pthread_spin_lock@plt
            |
            ---0
               rxe_query_device
               0
               pthread_spin_lock@plt

     1.79%     0.00%  swapper          [kernel.kallsyms]        [k] irq_exit
            |          
             --1.78%--irq_exit
                       |          
                        --1.68%--__softirqentry_text_start
                                  |          
                                   --1.00%--tasklet_action
                                             |          
                                              --0.99%--rxe_do_task
                                                        rxe_requester
                                                        |          
                                                         --0.63%--rxe_send
                                                                   |          
                                                                    --0.58%--ip_local_out

     1.73%     0.01%  swapper          [kernel.kallsyms]        [k] ret_from_intr
            |          
             --1.73%--ret_from_intr
                       |          
                        --1.72%--do_IRQ
                                  |          
                                   --1.61%--irq_exit
                                             |          
                                              --1.52%--__softirqentry_text_start
                                                        |          
                                                         --1.00%--tasklet_action
                                                                   |          
                                                                    --0.99%--rxe_do_task
                                                                              rxe_requester
                                                                              |          
                                                                               --0.63%--rxe_send
                                                                                         |          
                                                                                          --0.58%--ip_local_out

     1.72%     0.00%  swapper          [kernel.kallsyms]        [k] do_IRQ
            |
            ---do_IRQ
               |          
                --1.61%--irq_exit
                          |          
                           --1.52%--__softirqentry_text_start
                                     |          
                                      --1.00%--tasklet_action
                                                |          
                                                 --0.99%--rxe_do_task
                                                           rxe_requester
                                                           |          
                                                            --0.63%--rxe_send
                                                                      |          
                                                                       --0.58%--ip_local_out

     1.68%     0.01%  swapper          [kernel.kallsyms]        [k] __softirqentry_text_start
            |          
             --1.66%--__softirqentry_text_start
                       |          
                        --1.00%--tasklet_action
                                  |          
                                   --0.99%--rxe_do_task
                                             rxe_requester
                                             |          
                                              --0.63%--rxe_send
                                                        |          
                                                         --0.58%--ip_local_out

     1.65%     1.65%  ib_send_bw       libpthread-2.23.so       [.] pthread_spin_unlock
            |          
             --1.64%--pthread_spin_unlock

     1.65%     1.65%  ib_send_bw       librxe-rdmav25.so        [.] pthread_spin_unlock@plt
            |
            ---pthread_spin_unlock@plt

     1.52%     0.00%  kubelet          [unknown]                [k] 0xfffd7673e8240489
            |
            ---0xfffd7673e8240489
               x_cgo_notify_runtime_init_done
               0x30e6d
               |          
                --1.51%--0x2d4a109
                          |          
                           --1.03%--0xdfa457
                                     |          
                                      --0.61%--0x3c7e29
                                                |          
                                                 --0.56%--0x3c632b
                                                           0x3c6532
                                                           |          
                                                            --0.56%--0x3c6638
                                                                      |          
                                                                       --0.53%--0x2ea9b2
                                                                                 0x2eafd4
                                                                                 0x2eb8fd

     1.52%     0.00%  kubelet          kubelet                  [.] x_cgo_notify_runtime_init_done
            |
            ---x_cgo_notify_runtime_init_done
               0x30e6d
               |          
                --1.51%--0x2d4a109
                          |          
                           --1.03%--0xdfa457
                                     |          
                                      --0.61%--0x3c7e29
                                                |          
                                                 --0.56%--0x3c632b
                                                           0x3c6532
                                                           |          
                                                            --0.56%--0x3c6638
                                                                      |          
                                                                       --0.53%--0x2ea9b2
                                                                                 0x2eafd4
                                                                                 0x2eb8fd

     1.52%     0.00%  kubelet          kubelet                  [.] 0x0000000000030e6d
            |
            ---0x30e6d
               |          
                --1.51%--0x2d4a109
                          |          
                           --1.03%--0xdfa457
                                     |          
                                      --0.61%--0x3c7e29
                                                |          
                                                 --0.56%--0x3c632b
                                                           0x3c6532
                                                           |          
                                                            --0.56%--0x3c6638
                                                                      |          
                                                                       --0.53%--0x2ea9b2
                                                                                 0x2eafd4
                                                                                 0x2eb8fd

     1.51%     0.00%  kubelet          kubelet                  [.] 0x0000000002d4a109
            |
            ---0x2d4a109
               |          
                --1.03%--0xdfa457
                          |          
                           --0.61%--0x3c7e29
                                     |          
                                      --0.56%--0x3c632b
                                                0x3c6532
                                                |          
                                                 --0.56%--0x3c6638
                                                           |          
                                                            --0.53%--0x2ea9b2
                                                                      0x2eafd4
                                                                      0x2eb8fd

     1.12%     1.12%  ib_send_bw       [vdso]                   [.] 0x0000000000000974
            |
            ---sample_get_cpu_mhz
               __vdso_gettimeofday
               0x974

     1.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa457
            |
            ---0xdfa457
               |          
                --0.61%--0x3c7e29
                          |          
                           --0.56%--0x3c632b
                                     0x3c6532
                                     |          
                                      --0.56%--0x3c6638
                                                |          
                                                 --0.53%--0x2ea9b2
                                                           0x2eafd4
                                                           0x2eb8fd

     1.02%     1.01%  ib_send_bw       libpthread-2.23.so       [.] pthread_spin_lock
            |          
             --1.01%--0
                       rxe_query_device
                       0
                       pthread_spin_lock

     1.00%     0.00%  swapper          [kernel.kallsyms]        [k] rxe_do_task
            |
            ---rxe_do_task
               |          
                --0.99%--rxe_requester
                          |          
                           --0.63%--rxe_send
                                     |          
                                      --0.58%--ip_local_out

     1.00%     0.01%  swapper          [kernel.kallsyms]        [k] tasklet_action
            |          
             --0.99%--tasklet_action
                       rxe_do_task
                       rxe_requester
                       |          
                        --0.63%--rxe_send
                                  |          
                                   --0.58%--ip_local_out

     0.99%     0.04%  swapper          [kernel.kallsyms]        [k] rxe_requester
            |          
             --0.95%--rxe_requester
                       |          
                        --0.63%--rxe_send
                                  |          
                                   --0.58%--ip_local_out

     0.93%     0.00%  kubelet          kubelet                  [.] 0x00000000002ea9b2
            |
            ---0x2ea9b2
               0x2eafd4
               0x2eb8fd
               |          
                --0.88%--0x2ef1c9
                          |          
                           --0.81%--0x3c58b8
                                     |          
                                      --0.77%--0x2ea9b2
                                                0x2eafd4
                                                0x2eb8fd
                                                |          
                                                 --0.76%--0x2ee67e
                                                           0x2eb8fd
                                                           |          
                                                            --0.73%--0x2ee67e
                                                                      |          
                                                                       --0.72%--0x2eb8fd
                                                                                 |          
                                                                                  --0.68%--0x2ef1c9
                                                                                            |          
                                                                                             --0.50%--0x3c58b8

     0.93%     0.00%  kubelet          kubelet                  [.] 0x00000000002eafd4
            |
            ---0x2eafd4
               0x2eb8fd
               |          
                --0.88%--0x2ef1c9
                          |          
                           --0.81%--0x3c58b8
                                     |          
                                      --0.77%--0x2ea9b2
                                                0x2eafd4
                                                0x2eb8fd
                                                |          
                                                 --0.76%--0x2ee67e
                                                           0x2eb8fd
                                                           |          
                                                            --0.73%--0x2ee67e
                                                                      |          
                                                                       --0.72%--0x2eb8fd
                                                                                 |          
                                                                                  --0.68%--0x2ef1c9
                                                                                            |          
                                                                                             --0.50%--0x3c58b8

     0.93%     0.00%  kubelet          kubelet                  [.] 0x00000000002eb8fd
            |
            ---0x2eb8fd
               |          
                --0.88%--0x2ef1c9
                          |          
                           --0.81%--0x3c58b8
                                     |          
                                      --0.77%--0x2ea9b2
                                                0x2eafd4
                                                0x2eb8fd
                                                |          
                                                 --0.76%--0x2ee67e
                                                           0x2eb8fd
                                                           |          
                                                            --0.73%--0x2ee67e
                                                                      |          
                                                                       --0.72%--0x2eb8fd
                                                                                 |          
                                                                                  --0.68%--0x2ef1c9
                                                                                            |          
                                                                                             --0.50%--0x3c58b8

     0.92%     0.00%  kubelet          kubelet                  [.] 0x00000000003c6532
            |
            ---0x3c6532
               |          
                --0.90%--0x3c6638
                          |          
                           --0.86%--0x2ea9b2
                                     0x2eafd4
                                     0x2eb8fd
                                     |          
                                      --0.82%--0x2ef1c9
                                                |          
                                                 --0.75%--0x3c58b8
                                                           |          
                                                            --0.71%--0x2ea9b2
                                                                      0x2eafd4
                                                                      0x2eb8fd
                                                                      |          
                                                                       --0.70%--0x2ee67e
                                                                                 0x2eb8fd
                                                                                 |          
                                                                                  --0.68%--0x2ee67e
                                                                                            0x2eb8fd
                                                                                            |          
                                                                                             --0.64%--0x2ef1c9

     0.90%     0.00%  kubelet          kubelet                  [.] 0x00000000003c6638
            |
            ---0x3c6638
               |          
                --0.86%--0x2ea9b2
                          0x2eafd4
                          0x2eb8fd
                          |          
                           --0.82%--0x2ef1c9
                                     |          
                                      --0.75%--0x3c58b8
                                                |          
                                                 --0.71%--0x2ea9b2
                                                           0x2eafd4
                                                           0x2eb8fd
                                                           |          
                                                            --0.70%--0x2ee67e
                                                                      0x2eb8fd
                                                                      |          
                                                                       --0.68%--0x2ee67e
                                                                                 0x2eb8fd
                                                                                 |          
                                                                                  --0.64%--0x2ef1c9

     0.88%     0.00%  kubelet          kubelet                  [.] 0x00000000002ef1c9
            |
            ---0x2ef1c9
               |          
                --0.81%--0x3c58b8
                          |          
                           --0.77%--0x2ea9b2
                                     0x2eafd4
                                     0x2eb8fd
                                     |          
                                      --0.76%--0x2ee67e
                                                0x2eb8fd
                                                |          
                                                 --0.73%--0x2ee67e
                                                           |          
                                                            --0.72%--0x2eb8fd
                                                                      |          
                                                                       --0.68%--0x2ef1c9
                                                                                 |          
                                                                                  --0.50%--0x3c58b8

     0.82%     0.00%  kubelet          kubelet                  [.] 0x00000000002ee67e
            |
            ---0x2ee67e
               0x2eb8fd
               |          
                --0.73%--0x2ee67e
                          |          
                           --0.72%--0x2eb8fd
                                     |          
                                      --0.68%--0x2ef1c9
                                                |          
                                                 --0.50%--0x3c58b8

     0.81%     0.00%  kubelet          kubelet                  [.] 0x00000000003c58b8
            |
            ---0x3c58b8
               |          
                --0.77%--0x2ea9b2
                          0x2eafd4
                          0x2eb8fd
                          |          
                           --0.76%--0x2ee67e
                                     0x2eb8fd
                                     |          
                                      --0.73%--0x2ee67e
                                                |          
                                                 --0.72%--0x2eb8fd
                                                           |          
                                                            --0.68%--0x2ef1c9
                                                                      |          
                                                                       --0.50%--0x3c58b8

     0.63%     0.02%  swapper          [kernel.kallsyms]        [k] rxe_send
            |          
             --0.61%--rxe_send
                       |          
                        --0.58%--ip_local_out

     0.61%     0.00%  kubelet          kubelet                  [.] 0x00000000003c7e29
            |
            ---0x3c7e29
               |          
                --0.56%--0x3c632b
                          0x3c6532
                          |          
                           --0.56%--0x3c6638
                                     |          
                                      --0.53%--0x2ea9b2
                                                0x2eafd4
                                                0x2eb8fd

     0.58%     0.00%  swapper          [kernel.kallsyms]        [k] ip_local_out
            |
            ---ip_local_out

     0.56%     0.00%  kubelet          kubelet                  [.] 0x00000000003c632b
            |
            ---0x3c632b
               0x3c6532
               |          
                --0.56%--0x3c6638
                          |          
                           --0.53%--0x2ea9b2
                                     0x2eafd4
                                     0x2eb8fd

     0.54%     0.00%  kubelet          kubelet                  [.] 0x000000000005d306
            |
            ---0x5d306

     0.49%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.45%     0.00%  kubelet          [kernel.kallsyms]        [k] page_fault
     0.45%     0.00%  kubelet          [kernel.kallsyms]        [k] do_page_fault
     0.45%     0.00%  kubelet          [kernel.kallsyms]        [k] __do_page_fault
     0.44%     0.01%  kubelet          [kernel.kallsyms]        [k] handle_mm_fault
     0.43%     0.00%  kubelet          [kernel.kallsyms]        [k] __handle_mm_fault
     0.42%     0.01%  swapper          [kernel.kallsyms]        [k] net_rx_action
     0.39%     0.00%  kubelet          kubelet                  [.] 0x00000000003c7e1f
     0.39%     0.00%  kubelet          kubelet                  [.] 0x00000000003c1792
     0.39%     0.39%  ib_send_bw       ib_send_bw               [.] gettimeofday@plt
     0.36%     0.00%  kubelet          kubelet                  [.] 0x00000000003c1806
     0.34%     0.04%  swapper          [kernel.kallsyms]        [k] ip_output
     0.32%     0.00%  kubelet          [kernel.kallsyms]        [k] __do_fault
     0.32%     0.01%  kubelet          [kernel.kallsyms]        [k] ext4_filemap_fault
     0.32%     0.00%  kubelet          [kernel.kallsyms]        [k] filemap_fault
     0.31%     0.00%  kubelet          [unknown]                [k] 0x0000016c850f0038
     0.30%     0.00%  ib_send_bw       libpthread-2.23.so       [.] __write_nocancel
     0.30%     0.01%  swapper          [kernel.kallsyms]        [k] mlx5e_napi_poll
     0.29%     0.00%  kubelet          kubelet                  [.] 0x00000000003c59ae
     0.28%     0.00%  swapper          [kernel.kallsyms]        [k] apic_timer_interrupt
     0.28%     0.00%  swapper          [kernel.kallsyms]        [k] smp_apic_timer_interrupt
     0.28%     0.01%  kubelet          [kernel.kallsyms]        [k] __do_page_cache_readahead
     0.27%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.26%     0.05%  swapper          [kernel.kallsyms]        [k] mlx5e_poll_tx_cq
     0.25%     0.01%  swapper          [kernel.kallsyms]        [k] ip_finish_output
     0.25%     0.00%  swapper          [kernel.kallsyms]        [k] nf_hook_slow
     0.25%     0.01%  swapper          [kernel.kallsyms]        [k] ip_finish_output2
     0.24%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] sys_write
     0.24%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] vfs_write
     0.24%     0.01%  swapper          [kernel.kallsyms]        [k] __ip_local_out
     0.22%     0.00%  kubelet          kubelet                  [.] 0x00000000002ea8f6
     0.22%     0.01%  swapper          [kernel.kallsyms]        [k] neigh_resolve_output
     0.21%     0.01%  swapper          [kernel.kallsyms]        [k] skb_release_all
     0.21%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] __vfs_write
     0.21%     0.00%  kubelet          kubelet                  [.] 0x000000000005f434
     0.21%     0.01%  swapper          [kernel.kallsyms]        [k] napi_consume_skb
     0.20%     0.00%  kubelet          kubelet                  [.] 0x00000000000215ce
     0.20%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] ib_uverbs_write
     0.19%     0.01%  swapper          [kernel.kallsyms]        [k] dev_queue_xmit
     0.19%     0.00%  systemd          [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.19%     0.02%  ib_send_bw       [kernel.kallsyms]        [k] ib_uverbs_post_send
     0.19%     0.04%  swapper          [kernel.kallsyms]        [k] __dev_queue_xmit
     0.19%     0.00%  kubelet          [unknown]                [k] 0xe5ce258d4c544155
     0.18%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_exit_group
     0.18%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_group_exit
     0.18%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_exit
     0.17%     0.17%  ib_send_bw       [vdso]                   [.] 0x0000000000000977
     0.17%     0.00%  kubelet          kubelet                  [.] 0x000000000005d18a
     0.16%     0.02%  swapper          [kernel.kallsyms]        [k] __tick_nohz_idle_enter
     0.15%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] mmput
     0.15%     0.00%  kubelet          kubelet                  [.] 0x000000000005b61a
     0.15%     0.01%  swapper          [kernel.kallsyms]        [k] copy_data
     0.15%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] exit_mmap
     0.14%     0.00%  kubelet          [kernel.kallsyms]        [k] ext4_readpages
     0.14%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] user_path_at_empty
     0.14%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] filename_lookup
     0.14%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] path_lookupat
     0.14%     0.00%  swapper          [kernel.kallsyms]        [k] x86_64_start_kernel
     0.14%     0.00%  swapper          [kernel.kallsyms]        [k] x86_64_start_reservations
     0.14%     0.00%  swapper          [kernel.kallsyms]        [k] start_kernel
     0.14%     0.00%  swapper          [kernel.kallsyms]        [k] rest_init
     0.14%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000000efba
     0.14%     0.00%  irqbalance       [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.14%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] page_fault
     0.14%     0.01%  kubelet          [kernel.kallsyms]        [k] ext4_mpage_readpages
     0.14%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] __do_page_fault
     0.14%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_page_fault
     0.13%     0.00%  kubelet          kubelet                  [.] 0x00000000003c5a8a
     0.13%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] rxe_post_send
     0.13%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] rxe_run_task
     0.13%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] rxe_do_task
     0.13%     0.00%  systemd-udevd    libc-2.23.so             [.] __libc_fork
     0.13%     0.01%  swapper          [kernel.kallsyms]        [k] crc32_pclmul_update
     0.13%     0.00%  kubelet          kubelet                  [.] 0x00000000002eeef1
     0.13%     0.01%  swapper          [kernel.kallsyms]        [k] crypto_shash_update
     0.12%     0.02%  swapper          [kernel.kallsyms]        [k] skb_release_data
     0.12%     0.12%  ib_send_bw       [vdso]                   [.] 0x000000000000098d
     0.12%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa493
     0.12%     0.00%  kubelet          kubelet                  [.] 0x0000000000b97754
     0.12%     0.01%  kubelet          kubelet                  [.] 0x00000000002fbd66
     0.12%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] ret_from_fork
     0.12%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] kthread
     0.12%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] worker_thread
     0.12%     0.00%  kubelet          kubelet                  [.] 0x000000000005abd0
     0.12%     0.04%  swapper          [kernel.kallsyms]        [k] nf_conntrack_in
     0.12%     0.00%  perf             libc-2.23.so             [.] __libc_start_main
     0.12%     0.00%  perf             perf                     [.] main
     0.12%     0.00%  perf             perf                     [.] run_builtin
     0.12%     0.00%  perf             perf                     [.] cmd_record
     0.12%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] ret_from_fork
     0.12%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] kthread
     0.12%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] worker_thread
     0.12%     0.00%  swapper          [kernel.kallsyms]        [k] ipv4_conntrack_local
     0.12%     0.00%  swapper          [kernel.kallsyms]        [k] sch_direct_xmit
     0.12%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] copy_process.part.35
     0.12%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.12%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_syscall_64
     0.12%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_clone
     0.12%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] _do_fork
     0.11%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.11%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] do_syscall_64
     0.11%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] process_one_work
     0.11%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_os_execute_deferred
     0.11%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] link_path_walk
     0.11%     0.00%  kubelet          kubelet                  [.] 0x0000000000032072
     0.11%     0.00%  kubelet          kubelet                  [.] 0x000000000000c4c8
     0.11%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] process_one_work
     0.11%     0.01%  swapper          [kernel.kallsyms]        [k] rxe_mem_copy
     0.11%     0.00%  dockerd          dockerd-ce               [.] runtime.goexit
     0.11%     0.00%  kubelet          kubelet                  [.] 0x00000000002f22ba
     0.11%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ev_asynch_execute_gpe_method
     0.11%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ns_evaluate
     0.11%     0.00%  swapper          [kernel.kallsyms]        [k] handle_irq
     0.11%     0.06%  swapper          [kernel.kallsyms]        [k] kfree
     0.11%     0.02%  ib_send_bw       [kernel.kallsyms]        [k] rxe_requester
     0.11%     0.00%  kubelet          kubelet                  [.] 0x00000000002ef154
     0.11%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_execute_method
     0.11%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_parse_aml
     0.11%     0.00%  swapper          [kernel.kallsyms]        [k] handle_irq_event
     0.10%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] handle_mm_fault
     0.10%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __handle_mm_fault
     0.10%     0.01%  swapper          [kernel.kallsyms]        [k] handle_irq_event_percpu
     0.10%     0.00%  systemd-udevd    libc-2.23.so             [.] __lxstat64
     0.10%     0.00%  kubelet          kubelet                  [.] 0x00000000002f1e4c
     0.10%     0.01%  swapper          [kernel.kallsyms]        [k] hrtimer_interrupt
     0.10%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.10%     0.00%  swapper          [kernel.kallsyms]        [k] handle_edge_irq
     0.10%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa4d4
     0.10%     0.00%  kubelet          kubelet                  [.] 0x000000000005ac8f
     0.10%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vfs_statx
     0.10%     0.01%  swapper          [kernel.kallsyms]        [k] rxe_prepare
     0.09%     0.00%  swapper          [kernel.kallsyms]        [k] skb_free_head
     0.09%     0.00%  kubelet          kubelet                  [.] 0x0000000000b9677b
     0.09%     0.00%  swapper          [kernel.kallsyms]        [k] tick_nohz_irq_exit
     0.09%     0.00%  zabbix_agentd    libc-2.23.so             [.] __GI___libc_read
     0.09%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] sys_read
     0.09%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] vfs_read
     0.09%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __vfs_read
     0.09%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] proc_reg_read
     0.09%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] seq_read
     0.09%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_newlstat
     0.09%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] SYSC_newlstat
     0.09%     0.00%  kubelet          kubelet                  [.] 0x000000000005acda
     0.09%     0.00%  kubelet          [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.09%     0.00%  irqbalance       libc-2.23.so             [.] __GI___libc_read
     0.09%     0.00%  irqbalance       [kernel.kallsyms]        [k] sys_read
     0.09%     0.00%  irqbalance       [kernel.kallsyms]        [k] vfs_read
     0.09%     0.00%  irqbalance       [kernel.kallsyms]        [k] __vfs_read
     0.09%     0.00%  irqbalance       [kernel.kallsyms]        [k] proc_reg_read
     0.09%     0.00%  irqbalance       [kernel.kallsyms]        [k] seq_read
     0.09%     0.00%  perf             [unknown]                [k] 0xf85e258d4c544155
     0.09%     0.00%  swapper          [kernel.kallsyms]        [k] schedule_idle
     0.09%     0.01%  swapper          [kernel.kallsyms]        [k] get_next_timer_interrupt
     0.09%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] show_stat
     0.09%     0.00%  systemd-udevd    [unknown]                [.] 0000000000000000
     0.09%     0.00%  kubelet          kubelet                  [.] 0x000000000000f538
     0.09%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_parse_loop
     0.09%     0.03%  swapper          [kernel.kallsyms]        [k] __sched_text_start
     0.09%     0.01%  zabbix_agentd    [kernel.kallsyms]        [k] kstat_irqs_usr
     0.09%     0.01%  kubelet          [kernel.kallsyms]        [k] __alloc_pages_nodemask
     0.09%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb719058
     0.09%     0.00%  kubelet          kubelet                  [.] 0x0000000000d58380
     0.09%     0.00%  kubelet          kubelet                  [.] 0x0000000000d40b4c
     0.08%     0.00%  swapper          [kernel.kallsyms]        [k] run_rebalance_domains
     0.08%     0.01%  swapper          [kernel.kallsyms]        [k] rebalance_domains
     0.08%     0.08%  kworker/u64:0    [kernel.kallsyms]        [k] ixgbe_read_reg
     0.08%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] ixgbe_service_task
     0.08%     0.00%  kubelet          kubelet                  [.] 0x00000000002eb688
     0.08%     0.00%  kubelet          kubelet                  [.] 0x0000000000075813
     0.08%     0.01%  irqbalance       [kernel.kallsyms]        [k] show_interrupts
     0.08%     0.00%  kubelet          kubelet                  [.] 0x000000000038603a
     0.08%     0.00%  perf             [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.08%     0.00%  kubelet          [kernel.kallsyms]        [k] add_to_page_cache_lru
     0.08%     0.00%  containerd       containerd               [.] runtime.goexit
     0.08%     0.00%  kubelet          kubelet                  [.] 0x0000000000d40a2f
     0.08%     0.00%  kubelet          kubelet                  [.] 0x0000000000238d92
     0.08%     0.00%  kubelet          kubelet                  [.] 0x00000000002389b0
     0.08%     0.00%  kubelet          kubelet                  [.] 0x00000000002399d7
     0.08%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] ixgbe_update_stats
     0.08%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] mutex_lock
     0.08%     0.00%  dockerd          [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.08%     0.00%  swapper          [kernel.kallsyms]        [k] blk_done_softirq
     0.08%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_softirq_done
     0.08%     0.00%  kubelet          kubelet                  [.] 0x00000000002f1e9f
     0.07%     0.00%  swapper          [kernel.kallsyms]        [k] tick_nohz_idle_enter
     0.07%     0.00%  swapper          [kernel.kallsyms]        [k] __hrtimer_run_queues
     0.07%     0.00%  kubelet          kubelet                  [.] 0x00000000003c6392
     0.07%     0.04%  systemd-udevd    [kernel.kallsyms]        [k] unmap_page_range
     0.07%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] unmap_vmas
     0.07%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] unmap_single_vma
     0.07%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_finish_command
     0.07%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_io_completion
     0.07%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_end_request
     0.07%     0.03%  swapper          [kernel.kallsyms]        [k] skb_release_head_state
     0.07%     0.04%  swapper          [kernel.kallsyms]        [k] __slab_free
     0.07%     0.05%  swapper          [kernel.kallsyms]        [k] kmem_cache_free_bulk
     0.07%     0.00%  swapper          [kernel.kallsyms]        [k] __kfree_skb_flush
     0.07%     0.06%  swapper          [kernel.kallsyms]        [k] _raw_spin_lock
     0.07%     0.01%  swapper          [kernel.kallsyms]        [k] __handle_irq_event_percpu
     0.07%     0.00%  kubelet          kubelet                  [.] 0x000000000005d27b
     0.07%     0.00%  kubelet          kubelet                  [.] 0x00000000003c6498
     0.06%     0.00%  swapper          [kernel.kallsyms]        [k] dev_hard_start_xmit
     0.06%     0.03%  swapper          [kernel.kallsyms]        [k] __next_timer_interrupt
     0.06%     0.03%  swapper          [kernel.kallsyms]        [k] rxe_icrc_hdr
     0.06%     0.00%  kubelet          [kernel.kallsyms]        [k] alloc_pages_current
     0.06%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_open
     0.06%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_sys_open
     0.06%     0.00%  kubelet          kubelet                  [.] 0x00000000002eb823
     0.06%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa47f
     0.06%     0.01%  kubelet          kubelet                  [.] 0x000000000000bfba
     0.06%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] rxe_send
     0.06%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] ip_local_out
     0.06%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] walk_component
     0.06%     0.00%  kubelet          kubelet                  [.] 0x0000000000239881
     0.06%     0.01%  swapper          [kernel.kallsyms]        [k] iptable_filter_hook
     0.06%     0.06%  kubelet          [kernel.kallsyms]        [k] get_page_from_freelist
     0.06%     0.01%  kubelet          [kernel.kallsyms]        [k] __add_to_page_cache_locked
     0.06%     0.00%  kubelet          kubelet                  [.] 0x00000000002eb989
     0.06%     0.00%  kubelet          kubelet                  [.] 0x0000000000684bfb
     0.05%     0.00%  systemd          [kernel.kallsyms]        [k] walk_component
     0.05%     0.00%  kubelet          kubelet                  [.] 0x00000000002f2501
     0.05%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] copy_page_range
     0.05%     0.00%  swapper          [kernel.kallsyms]        [k] crc32_pclmul_le
     0.05%     0.01%  swapper          [kernel.kallsyms]        [k] rxe_init_packet
     0.05%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa46b
     0.05%     0.00%  kubelet          kubelet                  [.] 0x00000000003c59df
     0.05%     0.00%  kubelet          kubelet                  [.] 0x00000000003c2f91
     0.05%     0.00%  kubelet          kubelet                  [.] 0x00000000003c008f
     0.05%     0.05%  swapper          [kernel.kallsyms]        [k] crc32_le
     0.05%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] page_fault
     0.05%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_page_fault
     0.05%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __do_page_fault
     0.05%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_wp_page
     0.05%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] path_openat
     0.05%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_filp_open
     0.05%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_exec_end_op
     0.05%     0.00%  kubelet          [kernel.kallsyms]        [k] __page_cache_alloc
     0.05%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] __mutex_lock.isra.2
     0.05%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __mutex_lock_slowpath
     0.05%     0.00%  kubelet          [unknown]                [k] 0x000000c0004707b0
     0.05%     0.00%  irqbalance       libc-2.23.so             [.] __GI___libc_open
     0.05%     0.00%  irqbalance       [kernel.kallsyms]        [k] sys_open
     0.05%     0.00%  irqbalance       [kernel.kallsyms]        [k] do_sys_open
     0.05%     0.00%  irqbalance       [kernel.kallsyms]        [k] do_filp_open
     0.05%     0.01%  irqbalance       [kernel.kallsyms]        [k] path_openat
     0.05%     0.05%  swapper          [kernel.kallsyms]        [k] mlx5e_xmit
     0.05%     0.00%  systemd          [unknown]                [.] 0x706f746b73656465
     0.05%     0.00%  ifconfig         [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.05%     0.05%  swapper          [kernel.kallsyms]        [k] ipt_do_table
     0.05%     0.00%  dockerd          dockerd-ce               [.] runtime.futex
     0.05%     0.00%  kubelet          kubelet                  [.] 0x000000000059b134
     0.05%     0.00%  kubelet          kubelet                  [.] 0x000000000045fd6d
     0.05%     0.03%  systemd-udevd    libc-2.23.so             [.] _int_free
     0.05%     0.00%  systemd          [kernel.kallsyms]        [k] vfs_statx
     0.05%     0.00%  systemd          [kernel.kallsyms]        [k] user_path_at_empty
     0.05%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] copy_pte_range
     0.05%     0.00%  systemd-udevd    libc-2.23.so             [.] __GI___access
     0.05%     0.00%  systemd-journal  [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.05%     0.00%  kubelet          [kernel.kallsyms]        [k] schedule
     0.05%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_finish_plug
     0.05%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_flush_plug_list
     0.05%     0.00%  kubelet          kubelet                  [.] 0x00000000002fc749
     0.05%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] inode_permission
     0.05%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] handle_mm_fault
     0.05%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __handle_mm_fault
     0.05%     0.00%  kubelet          [kernel.kallsyms]        [k] submit_bio
     0.05%     0.02%  zabbix_agentd    [kernel.kallsyms]        [k] kstat_irqs
     0.05%     0.01%  swapper          [kernel.kallsyms]        [k] __alloc_skb
     0.05%     0.01%  kubelet          [kernel.kallsyms]        [k] __sched_text_start
     0.05%     0.00%  kubelet          kubelet                  [.] 0x00000000002eba9e
     0.05%     0.00%  (rm)             [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.05%     0.00%  containerd       [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.05%     0.00%  perf             perf                     [.] record__mmap_read_evlist.constprop.26
     0.05%     0.00%  swapper          [kernel.kallsyms]        [k] tick_sched_timer
     0.05%     0.00%  kubelet          kubelet                  [.] 0x0000000000b95621
     0.05%     0.00%  kubelet          kubelet                  [.] 0x0000000000b950a5
     0.04%     0.00%  kubelet          kubelet                  [.] 0x0000000000021df1
     0.04%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] wp_page_copy
     0.04%     0.00%  zabbix_agentd    [unknown]                [k] 0x57c6258d4c544155
     0.04%     0.00%  zabbix_agentd    libc-2.23.so             [.] __libc_start_main
     0.04%     0.00%  zabbix_agentd    zabbix_agentd            [.] main
     0.04%     0.00%  zabbix_agentd    zabbix_agentd            [.] daemon_start
     0.04%     0.00%  zabbix_agentd    zabbix_agentd            [.] MAIN_ZABBIX_ENTRY
     0.04%     0.00%  zabbix_agentd    zabbix_agentd            [.] zbx_thread_start
     0.04%     0.00%  perf             libc-2.23.so             [.] __GI___ioctl
     0.04%     0.00%  kubelet          kubelet                  [.] 0x000000000003709a
     0.04%     0.00%  systemd          [kernel.kallsyms]        [k] link_path_walk
     0.04%     0.04%  perf             [kernel.kallsyms]        [k] smp_call_function_single
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] sys_ioctl
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] do_vfs_ioctl
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] perf_ioctl
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] perf_event_for_each_child
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] _perf_event_enable
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] event_function_call
     0.04%     0.01%  systemd-udevd    libc-2.23.so             [.] _int_malloc
     0.04%     0.00%  kubelet          kubelet                  [.] 0x00000000000611b3
     0.04%     0.00%  perf             perf                     [.] perf_evlist__enable
     0.04%     0.00%  dockerd          [kernel.kallsyms]        [k] sys_futex
     0.04%     0.00%  dockerd          [kernel.kallsyms]        [k] do_futex
     0.04%     0.04%  swapper          [kernel.kallsyms]        [k] nf_nat_ipv4_fn
     0.04%     0.00%  kubelet          kubelet                  [.] 0x00000000002edc22
     0.04%     0.00%  kubelet          kubelet                  [.] 0x00000000000c2f1f
     0.04%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb721f80
     0.04%     0.00%  systemd-udevd    libc-2.23.so             [.] __GI___libc_open
     0.04%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] __blk_run_queue
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] scsi_request_fn
     0.04%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_access
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] generic_make_request
     0.04%     0.00%  perf             perf                     [.] perf_mmap__push
     0.04%     0.04%  swapper          [kernel.kallsyms]        [k] rxe_skb_tx_dtor
     0.04%     0.00%  kubelet          kubelet                  [.] 0x0000000000019585
     0.04%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __inode_permission
     0.04%     0.00%  kubelet          kubelet                  [.] 0x00000000000373be
     0.04%     0.00%  kubelet          kubelet                  [.] 0x00000000003c6312
     0.04%     0.01%  swapper          [kernel.kallsyms]        [k] load_balance
     0.04%     0.00%  kubelet          kubelet                  [.] 0x000000000000f153
     0.04%     0.00%  kubelet          kubelet                  [.] 0x000000000005a4d2
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] __elv_add_request
     0.04%     0.04%  swapper          [kernel.kallsyms]        [k] mlx5_eq_int
     0.04%     0.03%  systemd-udevd    [kernel.kallsyms]        [k] filemap_map_pages
     0.04%     0.00%  kubelet          kubelet                  [.] 0x000000000001e16c
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] napi_gro_receive
     0.04%     0.01%  kubelet          kubelet                  [.] 0x000000000002b74a
     0.04%     0.00%  kubelet          kubelet                  [.] 0x00000000000199c8
     0.04%     0.00%  systemd          libc-2.23.so             [.] __GI___mkdir
     0.04%     0.00%  systemd          [kernel.kallsyms]        [k] sys_mkdir
     0.04%     0.00%  kubelet          [unknown]                [.] 0x000000c0000b6fb0
     0.04%     0.00%  kubelet          kubelet                  [.] 0x00000000002eba13
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] ixgbe_poll
     0.04%     0.01%  systemd-cgroups  ld-2.23.so               [.] _dl_sysdep_start
     0.04%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).reader
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_process_callbacks
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] blk_update_request
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] bio_endio
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] clone_endio
     0.04%     0.01%  irqbalance       [kernel.kallsyms]        [k] seq_vprintf
     0.04%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_iop_permission
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] cfq_insert_request
     0.04%     0.00%  perf             libpthread-2.23.so       [.] __write_nocancel
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] call_function_single_interrupt
     0.04%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] free_pgtables
     0.04%     0.00%  systemd          [kernel.kallsyms]        [k] filename_lookup
     0.04%     0.00%  systemd          [kernel.kallsyms]        [k] path_lookupat
     0.04%     0.04%  kubelet          kubelet                  [.] 0x00000000002fcae9
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] alloc_pages_vma
     0.04%     0.00%  kubelet          kubelet                  [.] 0x0000000000019e74
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] page_cache_async_readahead
     0.04%     0.00%  kubelet          [kernel.kallsyms]        [k] ondemand_readahead
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] smp_call_function_single_interrupt
     0.04%     0.00%  kubelet          kubelet                  [.] 0x0000000000595e39
     0.04%     0.00%  kubelet          kubelet                  [.] 0x000000000058f7ef
     0.04%     0.00%  kubelet          [unknown]                [k] 0000000000000000
     0.04%     0.00%  kubelet          kubelet                  [.] 0x0000000000046157
     0.04%     0.00%  perf             perf                     [.] record__pushfn
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] sys_write
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] vfs_write
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] __vfs_write
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] ext4_file_write_iter
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] __generic_file_write_iter
     0.04%     0.00%  perf             [kernel.kallsyms]        [k] generic_perform_write
     0.04%     0.00%  kubelet          kubelet                  [.] 0x000000000005f371
     0.04%     0.01%  swapper          [kernel.kallsyms]        [k] hrtimer_start_range_ns
     0.04%     0.01%  swapper          [kernel.kallsyms]        [k] nf_nat_ipv4_local_fn
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] iptable_nat_ipv4_local_fn
     0.04%     0.00%  swapper          [kernel.kallsyms]        [k] tick_nohz_idle_exit
     0.04%     0.04%  systemd          libc-2.23.so             [.] strlen
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000000214a7
     0.03%     0.00%  systemd          libc-2.23.so             [.] __fxstatat64
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] sys_newfstatat
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] SYSC_newfstatat
     0.03%     0.00%  kubelet          [unknown]                [.] 0x000000c0000b67b0
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000000bfb7
     0.03%     0.02%  zabbix_agentd    [kernel.kallsyms]        [k] cpumask_next
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000058d629
     0.03%     0.02%  swapper          [kernel.kallsyms]        [k] switch_mm_irqs_off
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000004804e
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000002175f
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000024829
     0.03%     0.03%  systemd-udevd    [kernel.kallsyms]        [k] osq_lock
     0.03%     0.01%  systemd          [kernel.kallsyms]        [k] cgroup_addrm_files
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000000ef00
     0.03%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_futex
     0.03%     0.00%  kubelet          [kernel.kallsyms]        [k] do_futex
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] mlx5e_poll_rx_cq
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] mlx5e_handle_rx_cqe
     0.03%     0.03%  systemd-udevd    [kernel.kallsyms]        [k] release_pages
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] tlb_finish_mmu
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] arch_tlb_finish_mmu
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] tlb_flush_mmu_free
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] free_pages_and_swap_cache
     0.03%     0.02%  kubelet          [kernel.kallsyms]        [k] filemap_map_pages
     0.03%     0.03%  ib_send_bw       [kernel.kallsyms]        [k] syscall_return_via_sysret
     0.03%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] __ip_local_out
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] vfs_mkdir
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_iop_mkdir
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup_mkdir
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000003c1814
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000002ec493
     0.03%     0.01%  swapper          [kernel.kallsyms]        [k] __kmalloc_node_track_caller
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] __kmalloc_reserve.isra.43
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000011b28
     0.03%     0.00%  kubelet          [unknown]                [.] 0x00020b5a0000000a
     0.03%     0.03%  kubelet          kubelet                  [.] 0x00000000002fbd4e
     0.03%     0.00%  irqbalance       [kernel.kallsyms]        [k] seq_printf
     0.03%     0.00%  kubelet          [unknown]                [.] 0x000000c0000b77b0
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] netif_receive_skb_internal
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] __netif_receive_skb
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] __netif_receive_skb_core
     0.03%     0.00%  systemd          systemd                  [.] 0x00000000000cb600
     0.03%     0.00%  systemd          [unknown]                [.] 0x00005626e254f4f0
     0.03%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_queue_bio
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000003203d
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000004c384
     0.03%     0.01%  dockerd          dockerd-ce               [.] runtime.selectgo
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] lookup_slow
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000677cd8
     0.03%     0.00%  kubelet          [unknown]                [k] 0x000000c0000b07b0
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000000eec3
     0.03%     0.00%  zabbix_agentd    zabbix_agentd            [.] collector_thread
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] sys_open
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] do_sys_open
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] do_filp_open
     0.03%     0.00%  systemd          [kernel.kallsyms]        [k] path_openat
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] try_to_wake_up
     0.03%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] unlink_anon_vmas
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] ttwu_do_activate
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] dec_pending
     0.03%     0.03%  swapper          [kernel.kallsyms]        [k] ixgbe_read_reg
     0.03%     0.00%  perf             libc-2.23.so             [.] __GI___execve
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] do_syscall_64
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] sys_execve
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000031e1ad
     0.03%     0.00%  systemd-udevd    [unknown]                [.] 0xfd32efe808ec8348
     0.03%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000036bf0
     0.03%     0.00%  (rm)             [unknown]                [k] 0xf566cfe808ec8348
     0.03%     0.00%  (rm)             systemd                  [.] 0x00000000000cb600
     0.03%     0.00%  (rm)             [unknown]                [.] 0x00005626e254f4f0
     0.03%     0.00%  (rm)             [unknown]                [.] 0x00005626e24640b0
     0.03%     0.00%  (rm)             [unknown]                [.] 0x0000000100000000
     0.03%     0.00%  (rm)             systemd                  [.] 0x00000000000d7e1a
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000002fcae4
     0.03%     0.00%  kubelet          [kernel.kallsyms]        [k] __lock_page_or_retry
     0.03%     0.00%  kubelet          [kernel.kallsyms]        [k] io_schedule
     0.03%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] ret_from_fork
     0.03%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] kthread
     0.03%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] worker_thread
     0.03%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] process_one_work
     0.03%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_dop_revalidate
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __alloc_pages_nodemask
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000003c125b
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] tick_sched_handle
     0.03%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_map_object
     0.03%     0.00%  systemd          [unknown]                [.] 0xf566cfe808ec8348
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] activate_task
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] irq_enter
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] lookup_fast
     0.03%     0.01%  swapper          [kernel.kallsyms]        [k] ___slab_alloc
     0.03%     0.00%  systemd-cgroups  ld-2.23.so               [.] dl_main
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000003c0303
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000003bdb1c
     0.03%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] kmem_cache_free
     0.03%     0.00%  dockerd          dockerd-ce               [.] runtime.mcall
     0.03%     0.00%  dockerd          dockerd-ce               [.] runtime.park_m
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000031dab1
     0.03%     0.00%  kubelet          [unknown]                [.] 0x8b4800001258888b
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000003d46c70
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa42f
     0.03%     0.01%  swapper          [kernel.kallsyms]        [k] netif_skb_features
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] validate_xmit_skb_list
     0.03%     0.03%  swapper          [kernel.kallsyms]        [k] __indirect_thunk_start
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] alloc_pages_current
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000052d87
     0.03%     0.03%  swapper          [kernel.kallsyms]        [k] hash_conntrack_raw
     0.03%     0.01%  swapper          [kernel.kallsyms]        [k] netdev_pick_tx
     0.03%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] ret_from_fork
     0.03%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] kthread
     0.03%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] worker_thread
     0.03%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] process_one_work
     0.03%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] memcg_kmem_cache_create_func
     0.03%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] memcg_create_kmem_cache
     0.03%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] create_cache
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa41b
     0.03%     0.03%  swapper          [kernel.kallsyms]        [k] memcpy_erms
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] tick_irq_enter
     0.03%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
     0.03%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_resolve_to_value
     0.03%     0.00%  modprobe         [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.03%     0.01%  dockerd          dockerd-ce               [.] runtime.findrunnable
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] generic_smp_call_function_single_interrupt
     0.03%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] mpage_end_io
     0.03%     0.01%  swapper          [kernel.kallsyms]        [k] sched_clock
     0.03%     0.00%  systemd          [unknown]                [.] 0x00005626e24640b0
     0.03%     0.00%  systemd          [unknown]                [.] 0x0000000100000000
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] ip_rcv
     0.03%     0.03%  swapper          [kernel.kallsyms]        [k] _find_next_bit
     0.03%     0.00%  perf             perf                     [.] perf_evlist__prepare_workload
     0.03%     0.01%  kubelet          [kernel.kallsyms]        [k] radix_tree_lookup
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] update_process_times
     0.03%     0.00%  systemd-udevd    libc-2.23.so             [.] readlinkat
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_readlinkat
     0.03%     0.00%  zabbix_agentd    zabbix_agentd            [.] collect_cpustat
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000045bc30
     0.03%     0.00%  systemd-journal  [kernel.kallsyms]        [k] sys_read
     0.03%     0.00%  systemd-journal  [kernel.kallsyms]        [k] vfs_read
     0.03%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] kmem_cache_alloc
     0.03%     0.03%  kubelet          [kernel.kallsyms]        [k] clear_page_erms
     0.03%     0.00%  dockerd          [unknown]                [k] 0000000000000000
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] task_work_run
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ____fput
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __fput
     0.03%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3
     0.03%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2
     0.03%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams
     0.03%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams
     0.03%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __pte_alloc
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] flush_smp_call_function_queue
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] ip_rcv_finish
     0.03%     0.00%  swapper          [kernel.kallsyms]        [k] enqueue_task_fair
     0.03%     0.00%  kubelet          kubelet                  [.] 0x00000000003c09b6
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000076efa
     0.03%     0.00%  kubelet          kubelet                  [.] 0x0000000000077333
     0.03%     0.00%  kubelet          kubelet                  [.] 0x000000000005d3df
     0.03%     0.02%  swapper          [kernel.kallsyms]        [k] find_busiest_group
     0.03%     0.00%  systemd-udevd    [unknown]                [k] 0x6f646e65765f7379
     0.03%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000001dec6
     0.03%     0.03%  swapper          [kernel.kallsyms]        [k] native_irq_return_iret
     0.03%     0.03%  kubelet          kubelet                  [.] 0x0000000000021577
     0.03%     0.00%  zabbix_agentd    libc-2.23.so             [.] __GI___semop
     0.03%     0.00%  dockerd          dockerd-ce               [.] runtime.schedule
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] do_open_execat
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] do_filp_open
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] path_openat
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] ext4_lookup
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] ext4_iget_normal
     0.03%     0.00%  perf             [kernel.kallsyms]        [k] ext4_iget
     0.03%     0.01%  irqbalance       [kernel.kallsyms]        [k] vsnprintf
     0.03%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3
     0.03%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*loopyWriter).run
     0.03%     0.00%  dockerd          [kernel.kallsyms]        [k] schedule
     0.03%     0.00%  dockerd          [kernel.kallsyms]        [k] __sched_text_start
     0.03%     0.01%  systemd          [kernel.kallsyms]        [k] kmem_cache_alloc
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000003c7ada
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000003c079f
     0.02%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000004a20b
     0.02%     0.01%  swapper          [kernel.kallsyms]        [k] run_timer_softirq
     0.02%     0.01%  systemd          libc-2.23.so             [.] _int_malloc
     0.02%     0.02%  ib_send_bw       [kernel.kallsyms]        [k] __indirect_thunk_start
     0.02%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] ip_finish_output
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] ip_output
     0.02%     0.02%  systemd          [kernel.kallsyms]        [k] _raw_spin_lock
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] link_path_walk
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] walk_component
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] lookup_slow
     0.02%     0.02%  kubelet          kubelet                  [.] 0x00000000002eb82d
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] futex_wait
     0.02%     0.00%  kubelet          [unknown]                [.] 0xe80002dfd7e8ffff
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000011b0a
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002edb12
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] native_sched_clock
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002f1f76
     0.02%     0.00%  systemd-cgroups  ld-2.23.so               [.] mmap64
     0.02%     0.00%  irqbalance       [kernel.kallsyms]        [k] proc_lookup
     0.02%     0.00%  irqbalance       [kernel.kallsyms]        [k] proc_lookup_de
     0.02%     0.00%  systemd          libc-2.23.so             [.] __GI___rmdir
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] sys_rmdir
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] do_rmdir
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000459147
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000419689
     0.02%     0.00%  systemd          libpthread-2.23.so       [.] __sendmsg_nocancel
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] sys_sendmsg
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] __sys_sendmsg
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] ___sys_sendmsg
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] page_cache_tree_insert
     0.02%     0.00%  systemd-journal  libc-2.23.so             [.] __GI___libc_read
     0.02%     0.00%  systemd-journal  [kernel.kallsyms]        [k] __vfs_read
     0.02%     0.00%  systemd-journal  [kernel.kallsyms]        [k] seq_read
     0.02%     0.02%  kubelet          kubelet                  [.] 0x00000000000221f1
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] enqueue_entity
     0.02%     0.02%  kubelet          kubelet                  [.] 0x00000000002eb821
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000012c45
     0.02%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/github.com/containerd/containerd.(*Client).IsServing
     0.02%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/health/grpc_health_v1.(*healthClient).Check
     0.02%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __kmem_cache_create
     0.02%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] sysfs_slab_add
     0.02%     0.02%  kubelet          kubelet                  [.] 0x0000000000016e52
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] scheduler_tick
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] megasas_isr_fusion
     0.02%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000b96776
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000000ca59
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000040f7ff
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000040da80
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] kmem_cache_free
     0.02%     0.02%  systemd          [kernel.kallsyms]        [k] __d_lookup
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] lookup_fast
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000385fb1
     0.02%     0.00%  systemd          [unknown]                [.] 0x8b48fb89483574ff
     0.02%     0.00%  systemd          systemd                  [.] 0x0000000000079850
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] page_fault
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] do_page_fault
     0.02%     0.01%  systemd          [kernel.kallsyms]        [k] __do_page_fault
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] sched_clock_cpu
     0.02%     0.00%  zabbix_agentd    zabbix_agentd            [.] 0x000000000001320c
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002f1cb8
     0.02%     0.00%  kubelet          [unknown]                [.] 0x000000c000470fb0
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000003c7795
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] pte_alloc_one
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] eth_header
     0.02%     0.00%  ib_send_bw       libc-2.23.so             [.] __GI___libc_read
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] sys_read
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] vfs_read
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] __vfs_read
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] proc_reg_read
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] seq_read
     0.02%     0.00%  containerd       containerd               [.] syscall.Syscall
     0.02%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleStream
     0.02%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa466
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002f26a9
     0.02%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] ret_from_fork
     0.02%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] kthread
     0.02%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] worker_thread
     0.02%     0.00%  perf             [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.02%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] vfs_getattr_nosec
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vfs_getattr
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000048462
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] tick_program_event
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000056fe31
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000005367f6
     0.02%     0.00%  systemd-udevd    libpthread-2.23.so       [.] __GI___libc_open
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000003c65ba
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000003c0fd4
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000003c0f4e
     0.02%     0.00%  rcu_sched        [kernel.kallsyms]        [k] ret_from_fork
     0.02%     0.00%  rcu_sched        [kernel.kallsyms]        [k] kthread
     0.02%     0.00%  rcu_sched        [kernel.kallsyms]        [k] rcu_gp_kthread
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa4c5
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000cb8138
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] pagevec_lru_move_fn
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] __lru_cache_add
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] loop_64
     0.02%     0.02%  kubelet          [kernel.kallsyms]        [k] __radix_tree_lookup
     0.02%     0.00%  systemd          systemd                  [.] 0x00000000000d7907
     0.02%     0.00%  systemd          libc-2.23.so             [.] __libc_fork
     0.02%     0.01%  swapper          [kernel.kallsyms]        [k] ipv4_confirm
     0.02%     0.00%  dockerd          dockerd-ce               [.] runtime.epollwait
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000003658c
     0.02%     0.01%  swapper          [kernel.kallsyms]        [k] mlx5e_select_queue
     0.02%     0.02%  systemd-udevd    [kernel.kallsyms]        [.] native_irq_return_iret
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] validate_xmit_skb
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] kfree_skb
     0.02%     0.01%  kubelet          [kernel.kallsyms]        [k] futex_wake
     0.02%     0.01%  kubelet          [kernel.kallsyms]        [k] mem_cgroup_try_charge
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] cmpxchg_double_slab.isra.61
     0.02%     0.00%  systemd-cgroups  [unknown]                [k] 0000000000000000
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000000fc1fd
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000252fe7
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000024cc67
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000038b388
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_mmap
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_mmap_pgoff
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] vm_mmap_pgoff
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] mmap_region
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_mmap
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] find_next_bit
     0.02%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] ptep_clear_flush
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] update_blocked_averages
     0.02%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5e_update_stats_work
     0.02%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5e_update_ndo_stats
     0.02%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5e_update_stats
     0.02%     0.01%  swapper          [kernel.kallsyms]        [k] update_load_avg
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath
     0.02%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] memcg_kmem_charge
     0.02%     0.00%  systemd-udevd    [unknown]                [.] 0x65686361635f6564
     0.02%     0.00%  systemd          libc-2.23.so             [.] __libc_waitid
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] sys_waitid
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] SYSC_waitid
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] kernel_waitid
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] do_wait
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] wait_consider_task
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] release_task
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] proc_flush_task
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] d_invalidate
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] pick_next_task_fair
     0.02%     0.01%  swapper          [kernel.kallsyms]        [k] cfq_completed_request
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] blk_finish_request
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] __blk_put_request
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] elv_completed_request
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000002613a
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000005b0bc
     0.02%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.Invoke
     0.02%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*ClientConn).Invoke
     0.02%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.invoke
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] unlazy_walk
     0.02%     0.02%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_track_stack_ptr
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] try_to_wake_up
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] css_populate_dir
     0.02%     0.01%  dockerd          dockerd-ce               [.] runtime.mallocgc
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000037a82
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000021d73
     0.02%     0.00%  kubelet          libc-2.23.so             [.] __clone
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] dput
     0.02%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] dput.part.22
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup_apply_control_enable
     0.02%     0.00%  (rm)             libc-2.23.so             [.] __GI___chdir
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] sys_chdir
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] user_path_at_empty
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] filename_lookup
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] path_lookupat
     0.02%     0.00%  (rm)             [kernel.kallsyms]        [k] ext4_lookup
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] rcu_idle_exit
     0.02%     0.01%  swapper          [kernel.kallsyms]        [k] clockevents_program_event
     0.02%     0.01%  kubelet          [kernel.kallsyms]        [k] alloc_set_pte
     0.02%     0.00%  perf             [kernel.kallsyms]        [k] link_path_walk
     0.02%     0.00%  perf             [kernel.kallsyms]        [k] walk_component
     0.02%     0.00%  perf             [kernel.kallsyms]        [k] lookup_slow
     0.02%     0.00%  (ifconfig)       libc-2.23.so             [.] __GI___execve
     0.02%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.02%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] do_syscall_64
     0.02%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] sys_execve
     0.02%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.02%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_evaluate_name_path
     0.02%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_resolve_node_to_value
     0.02%     0.02%  systemd-cgroups  ld-2.23.so               [.] strcmp
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_peek_request
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000001114e
     0.02%     0.01%  perf             [kernel.kallsyms]        [k] ll_rw_block
     0.02%     0.00%  perf             [kernel.kallsyms]        [k] __breadahead
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000000200a0
     0.02%     0.01%  systemd          [kernel.kallsyms]        [k] copy_process.part.35
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] do_syscall_64
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] sys_clone
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] _do_fork
     0.02%     0.00%  kubelet          ld-2.23.so               [.] _dl_map_object
     0.02%     0.02%  zabbix_agentd    [kernel.kallsyms]        [k] syscall_return_via_sysret
     0.02%     0.02%  systemd-udevd    libc-2.23.so             [.] malloc_consolidate
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000230dbc
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] do_wp_page
     0.02%     0.02%  ifconfig         [kernel.kallsyms]        [k] ixgbe_read_reg
     0.02%     0.00%  ifconfig         libc-2.23.so             [.] __GI___ioctl
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] sys_ioctl
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_vfs_ioctl
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] sock_ioctl
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] sock_do_ioctl
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] inet_ioctl
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] devinet_ioctl
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] dev_change_flags
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] __dev_change_flags
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] dev_set_rx_mode
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] __dev_set_rx_mode
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] ixgbe_set_rx_mode
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] ixgbe_restore_vf_multicasts
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] ixgbe_full_sync_mac_table
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] ixgbe_clear_rar_generic
     0.02%     0.00%  ifconfig         [kernel.kallsyms]        [k] ixgbe_clear_vmdq_generic
     0.02%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] filemap_map_pages
     0.02%     0.00%  rm               [kernel.kallsyms]        [k] page_fault
     0.02%     0.00%  rm               [kernel.kallsyms]        [k] do_page_fault
     0.02%     0.00%  rm               [kernel.kallsyms]        [k] __do_page_fault
     0.02%     0.00%  rm               [kernel.kallsyms]        [k] handle_mm_fault
     0.02%     0.00%  rm               [kernel.kallsyms]        [k] __handle_mm_fault
     0.02%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_field_datum_io
     0.02%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_access_region
     0.02%     0.00%  kubelet          [unknown]                [.] 0x000000c00046cfb0
     0.02%     0.00%  containerd       containerd               [.] runtime.futex
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] native_write_msr
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] __slab_alloc
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] ttwu_do_activate
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002eda3b
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] deactivate_task
     0.02%     0.00%  systemd-journal  [kernel.kallsyms]        [k] proc_single_show
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] megasas_build_and_issue_cmd_fusion
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] scsi_dispatch_cmd
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] megasas_queue_command
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] wake_up_page_bit
     0.02%     0.00%  systemd-cgroups  ld-2.23.so               [.] open64
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_open
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_sys_open
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_filp_open
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] path_openat
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000004c45f
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] futex_wake
     0.02%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader
     0.02%     0.00%  dockerd          dockerd-ce               [.] io.ReadFull
     0.02%     0.00%  dockerd          dockerd-ce               [.] io.ReadAtLeast
     0.02%     0.00%  dockerd          dockerd-ce               [.] bufio.(*Reader).Read
     0.02%     0.00%  dockerd          dockerd-ce               [.] net.(*conn).Read
     0.02%     0.00%  dockerd          dockerd-ce               [.] net.(*netFD).Read
     0.02%     0.00%  dockerd          dockerd-ce               [.] internal/poll.(*FD).Read
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000011f49
     0.02%     0.02%  systemd          libc-2.23.so             [.] malloc_consolidate
     0.02%     0.00%  zabbix_agentd    zabbix_agentd            [.] 0x0000000000012ede
     0.02%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*controlBuffer).get
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] lru_cache_add
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000045a587
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] unlock_page
     0.02%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] _vm_normal_page
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] vfs_rmdir
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_iop_rmdir
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] d_alloc_parallel
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] d_alloc
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] quiet_vmstat
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000000b2b9
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000005cc93
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] lookup_slow
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] wake_up_process
     0.02%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] show_cpuinfo
     0.02%     0.00%  systemd-udevd    [vdso]                   [.] __vdso_clock_gettime
     0.02%     0.00%  containerd       [kernel.kallsyms]        [k] sys_futex
     0.02%     0.00%  containerd       [kernel.kallsyms]        [k] do_futex
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] dequeue_task_fair
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000000bf27
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000000d3ac
     0.02%     0.00%  modprobe         [kernel.kallsyms]        [k] page_fault
     0.02%     0.00%  modprobe         [kernel.kallsyms]        [k] do_page_fault
     0.02%     0.00%  modprobe         [kernel.kallsyms]        [k] __do_page_fault
     0.02%     0.02%  kubelet          kubelet                  [.] 0x000000000000d433
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000002505f
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000024c37
     0.02%     0.00%  systemd-udevd    libc-2.23.so             [.] __GI___munmap
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_munmap
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vm_munmap
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_munmap
     0.02%     0.00%  kubelet          [unknown]                [.] 0x000000c0000b7fb0
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000cb8084
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_iop_getattr
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000461073
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] ip_local_deliver
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] ip_local_deliver_finish
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000024ff6
     0.02%     0.00%  systemd-udevd    [unknown]                [.] 0xfffc1f54fffc1ec4
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] tick_do_update_jiffies64.part.10
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] enqueue_entity
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] activate_task
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] enqueue_task_fair
     0.02%     0.02%  ib_send_bw       librxe-rdmav25.so        [.] rxe_post_send
     0.02%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] rxe_prepare
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] ip_finish_output2
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] neigh_resolve_output
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] dev_queue_xmit
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] __dev_queue_xmit
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] do_syscall_64
     0.02%     0.02%  ib_send_bw       libc-2.23.so             [.] __memcpy_avx_unaligned
     0.02%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] iptable_filter_hook
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] nf_hook_slow
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] __wake_up_common
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] wp_page_copy
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000002b6e6
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000016206
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000012c1f
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000001010d
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002f2887
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002ed872
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000595e2f
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000004cccaa
     0.02%     0.02%  kubelet          kubelet                  [.] 0x00000000002fca89
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] dequeue_entity
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002f1db9
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002f21c6
     0.02%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2
     0.02%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).run
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] switch_mm
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] enter_lazy_tlb
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000031db3f
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] exit_to_usermode_loop
     0.02%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] task_work_run
     0.02%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] syscall_return_via_sysret
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] legitimize_path.isra.29
     0.02%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000038609d
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000003c7201
     0.02%     0.02%  systemd          [kernel.kallsyms]        [k] error_entry
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] udp_rcv
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] __udp4_lib_rcv
     0.02%     0.02%  swapper          [kernel.kallsyms]        [k] ktime_get
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002ed834
     0.02%     0.02%  irqbalance       [kernel.kallsyms]        [k] pde_subdir_find
     0.02%     0.02%  systemd-udevd    [kernel.kallsyms]        [k] get_page_from_freelist
     0.02%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] alloc_pages_vma
     0.02%     0.00%  irqbalance       [kernel.kallsyms]        [k] link_path_walk
     0.02%     0.00%  irqbalance       [kernel.kallsyms]        [k] walk_component
     0.02%     0.00%  irqbalance       [kernel.kallsyms]        [k] lookup_slow
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002f230f
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] sock_sendmsg
     0.02%     0.00%  systemd          [kernel.kallsyms]        [k] unix_stream_sendmsg
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] megasas_build_io_fusion
     0.02%     0.00%  containerd       [kernel.kallsyms]        [k] try_to_wake_up
     0.02%     0.00%  containerd       [kernel.kallsyms]        [k] ttwu_do_activate
     0.02%     0.00%  kubelet          [unknown]                [.] 0x000000c00046c7b0
     0.02%     0.02%  kubelet          kubelet                  [.] 0x00000000002fcb29
     0.02%     0.00%  (kubelet)        [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.02%     0.00%  (kubelet)        [kernel.kallsyms]        [k] do_syscall_64
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] sched_clock_idle_sleep_event
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000001fd92
     0.02%     0.00%  kubelet          kubelet                  [.] 0x0000000000024202
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000002fc81b
     0.02%     0.00%  systemd          [unknown]                [.] 0x00005626e246dff0
     0.02%     0.00%  containerd       containerd               [.] runtime.morestack
     0.02%     0.00%  containerd       containerd               [.] runtime.newstack
     0.02%     0.00%  containerd       containerd               [.] runtime.copystack
     0.02%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] vma_link
     0.02%     0.02%  kubelet          [kernel.kallsyms]        [k] blk_queue_split
     0.02%     0.00%  (ifconfig)       [unknown]                [k] 0xf566cfe808ec8348
     0.02%     0.00%  (ifconfig)       systemd                  [.] 0x00000000000cb600
     0.02%     0.00%  (ifconfig)       [unknown]                [.] 0x00005626e254f4f0
     0.02%     0.00%  (ifconfig)       [unknown]                [.] 0x00005626e24640b0
     0.02%     0.00%  (ifconfig)       [unknown]                [.] 0x0000000100000000
     0.02%     0.00%  (ifconfig)       systemd                  [.] 0x00000000000d7e1a
     0.02%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_read_data_from_field
     0.02%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_extract_from_field
     0.02%     0.01%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_trace_ptr
     0.02%     0.00%  dockerd          dockerd-ce               [.] syscall.Syscall
     0.02%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5_cmd_exec
     0.02%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] cmd_exec
     0.02%     0.02%  kubelet          kubelet                  [.] 0x000000000005f3ee
     0.02%     0.00%  dockerd          [kernel.kallsyms]        [k] wake_up_q
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000001ffa9
     0.02%     0.00%  kubelet          kubelet                  [.] 0x00000000000612d7
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] iptable_nat_ipv4_out
     0.02%     0.00%  swapper          [kernel.kallsyms]        [k] nf_nat_ipv4_out
     0.02%     0.01%  swapper          [kernel.kallsyms]        [k] complete_cmd_fusion
     0.02%     0.02%  irqbalance       [kernel.kallsyms]        [k] format_decode
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_read
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] vfs_read
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] __vfs_read
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] seq_read
     0.02%     0.00%  kubelet          kubelet                  [.] 0x000000000002b87e
     0.02%     0.00%  containerd       containerd               [.] runtime.mcall
     0.02%     0.00%  containerd       containerd               [.] runtime.park_m
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.02%     0.00%  kubelet          [kernel.kallsyms]        [k] do_syscall_64
     0.02%     0.00%  dockerd          dockerd-ce               [.] runtime.newobject
     0.02%     0.00%  zabbix_agentd    [unknown]                [k] 0x3733363931303733
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] handle_mm_fault
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __handle_mm_fault
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] radix_tree_lookup
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] proc_cgroup_show
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fbd6c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002fc795
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] futex_wait_queue_me
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] hrtimer_wakeup
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003bf7d8
     0.01%     0.01%  zabbix_agentd    [kernel.kallsyms]        [k] _find_next_bit
     0.01%     0.01%  perf             [kernel.kallsyms]        [k] copy_user_enhanced_fast_string
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] iov_iter_copy_from_user_atomic
     0.01%     0.00%  kubelet          [unknown]                [.] 0x000000c0000b47b0
     0.01%     0.01%  zabbix_agentd    libc-2.23.so             [.] _IO_vfscanf
     0.01%     0.00%  zabbix_agentd    [unknown]                [.] 0x20756c2520756c25
     0.01%     0.00%  zabbix_agentd    libc-2.23.so             [.] __isoc99_vsscanf
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000578486
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000535e39
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000533e6c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000005f376
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] enqueue_hrtimer
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] call_timer_fn
     0.01%     0.00%  containerd       [unknown]                [k] 0000000000000000
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __nf_conntrack_find_get
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] __kernfs_new_node
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __kernfs_create_file
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_new_node
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] _IO_file_close
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] default_wake_function
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] ip_route_input_rcu
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] ip_route_input_noref
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000104f4
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000f62a
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] native_flush_tlb_single
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] flush_tlb_mm_range
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] flush_tlb_func_common.constprop.10
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __udp4_lib_lookup
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] ipv4_mtu
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] find_vma
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] mlx5e_features_check
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] find_vma
     0.01%     0.00%  systemd          libc-2.23.so             [.] __GI___libc_open
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __netdev_pick_tx
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] less_64
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] terminate_walk
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] nf_ct_deliver_cached_events
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] rxe_find_route.isra.15
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] dst_release
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] kmem_cache_alloc
     0.01%     0.00%  containerd       containerd               [.] runtime.schedule
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fc703
     0.01%     0.00%  systemd-cgroups  ld-2.23.so               [.] 0x00000000002164c8
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] set_next_entity
     0.01%     0.00%  systemd          libc-2.23.so             [.] __xstat64
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] sys_newstat
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] SYSC_newstat
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] ext4_lookup
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] ext4_iget_normal
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] ext4_iget
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000036c00
     0.01%     0.00%  dbus-daemon      libpthread-2.23.so       [.] __sendmsg_nocancel
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] sys_sendmsg
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] __sys_sendmsg
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] ___sys_sendmsg
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] sock_sendmsg
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] unix_stream_sendmsg
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] sock_def_readable
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] __wake_up_sync_key
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] __wake_up_common_lock
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] __wake_up_common
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] ep_poll_callback
     0.01%     0.00%  dbus-daemon      [unknown]                [k] 0x000056033749e770
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fbd3a
     0.01%     0.01%  irqbalance       libglib-2.0.so.0.4800.2  [.] g_list_find_custom
     0.01%     0.00%  irqbalance       [unknown]                [.] 0x0000000000841f0f
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] sys_read
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3235333232303733
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_clone
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] _do_fork
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c7acb
     0.01%     0.00%  sleep            [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).WriteStatus
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/status.(*Status).Proto
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/golang/protobuf/proto.Clone
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/golang/protobuf/proto.Merge
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/golang/protobuf/proto.mergeStruct
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] switch_mm_irqs_off
     0.01%     0.01%  zabbix_agentd    [kernel.kallsyms]        [k] radix_tree_lookup
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] ext4_iget_normal
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] ext4_iget
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __breadahead
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000010f3f
     0.01%     0.01%  modprobe         ld-2.23.so               [.] _dl_sysdep_start
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] submit_bh_wbc
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] submit_bio
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] generic_make_request
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] blk_queue_bio
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002ed97a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002facfb
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002f9979
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000b4d0c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000b5ed5
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] sysfs_create_group
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] internal_create_group
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] sysfs_add_file_mode_ns
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __kernfs_create_file
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] kernfs_new_node
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __kernfs_new_node
     0.01%     0.00%  systemd-cgroups  [unknown]                [.] 0x0000000009691a75
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] sys_read
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] __wake_up_locked_key_bookmark
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] autoremove_wake_function
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] update_wall_time
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000005ad8d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000033428
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] __remove_hrtimer
     0.01%     0.00%  containerd       containerd               [.] runtime.gentraceback
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __getblk_gfp
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] pagecache_get_page
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __nf_ct_refresh_acct
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] udp_packet
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup_rmdir
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup_destroy_locked
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kill_css
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] css_clear_dir
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __fxstat64
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] iput
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] shrink_dentry_list
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __dentry_kill
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] dentry_unlink_inode
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] sched_ttwu_pending
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000249fd
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __do_fault
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ext4_filemap_fault
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] alloc_inode
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000238f1b
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] generic_permission
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] inode_permission
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __inode_permission
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] vfs_read
     0.01%     0.00%  zabbix_agentd    zabbix_agentd            [.] active_checks_thread
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] search_binary_handler
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] load_elf_binary
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] flush_old_exec
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] mmput
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] exit_mmap
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] unmap_vmas
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] unmap_single_vma
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] do_filp_open
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] path_openat
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] filename_parentat
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] path_parentat
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] handle_mm_fault
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] __handle_mm_fault
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000002eb0f
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] schedule
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] __sched_text_start
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] note_gp_changes
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000024cb4d
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] syscall_return_via_sysret
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] submit_bio
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] generic_make_request
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] blk_queue_bio
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000025e99
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] get_mem_cgroup_from_mm
     0.01%     0.00%  ifconfig         [unknown]                [k] 0x53434f4953006425
     0.01%     0.00%  ifconfig         [unknown]                [k] 0000000000000000
     0.01%     0.00%  ifconfig         [unknown]                [k] 0x000000000060f7e0
     0.01%     0.00%  ifconfig         libc-2.23.so             [.] _IO_file_open
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sys_open
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_sys_open
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_filp_open
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] path_openat
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ext4_lookup
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ext4_iget_normal
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ext4_iget
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __breadahead
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ll_rw_block
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] submit_bh_wbc
     0.01%     0.00%  (rm)             libc-2.23.so             [.] __GI___execve
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] do_syscall_64
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] sys_execve
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] do_wp_page
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] wp_page_copy
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000045a27d
     0.01%     0.00%  containerd       containerd               [.] runtime.findrunnable
     0.01%     0.00%  ifconfig         ld-2.23.so               [.] _dl_sysdep_start
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] show_stat
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000086160
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] proc_reg_read
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.systemstack
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] kmem_cache_alloc
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_exit_group
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_group_exit
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_exit
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] mmput
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] exit_mmap
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] vma_interval_tree_insert
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] fib_table_lookup
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000d548
     0.01%     0.00%  kubelet          [unknown]                [.] 0x000000c00046d7b0
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] read_tsc
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000073fe4
     0.01%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] schedule
     0.01%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] __sched_text_start
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] __radix_tree_create
     0.01%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_map_object_from_fd
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] futex_wait
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] sys_sched_yield
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __slab_alloc
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] ___slab_alloc
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] lockref_get_not_dead
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] __radix_tree_lookup
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] __vfs_read
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000053d36
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000047883
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000004929a
     0.01%     0.00%  (kubelet)        libc-2.23.so             [.] __GI___execve
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] sys_execve
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.01%     0.01%  systemd          libc-2.23.so             [.] __strcmp_sse2_unaligned
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000cb489f
     0.01%     0.00%  containerd       containerd               [.] runtime.systemstack
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] reschedule_interrupt
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] enqueue_task_fair
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] activate_task
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000da02
     0.01%     0.00%  containerd       [unknown]                [k] 0x000056273942be98
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] copy_process.part.35
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] kmem_cache_alloc
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] ext4_da_write_end
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] generic_write_end
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] __mark_inode_dirty
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] ext4_dirty_inode
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] ext4_mark_inode_dirty
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] ext4_reserve_inode_write
     0.01%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] ret_from_fork
     0.01%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] kthread
     0.01%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] worker_thread
     0.01%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] process_one_work
     0.01%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] fb_flashcursor
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000020fe5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000048fb5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000056fe20
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000239784
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000040f5e3
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] timerqueue_add
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] search_binary_handler
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] load_elf_binary
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000040f45d
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_trace
     0.01%     0.00%  systemd-cgroups  ld-2.23.so               [.] mprotect
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_mprotect
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_mprotect_pkey
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] mprotect_fixup
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] remote_function
     0.01%     0.01%  modprobe         libc-2.23.so             [.] _int_malloc
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001fa54f
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] _raw_spin_lock
     0.01%     0.00%  systemd          [unknown]                [.] 0x002f70756f726763
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] filemap_fault
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __do_page_cache_readahead
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] smp_reschedule_interrupt
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] scheduler_ipi
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] page_fault
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] do_page_fault
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] __do_page_fault
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] handle_mm_fault
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] __handle_mm_fault
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] __slab_free
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] anon_vma_fork
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __cmpxchg_double_slab.isra.51
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] skb_clone
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] wake_page_function
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000023979c
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000038ab9f
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000387aa9
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000008d3b31
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] schedule_hrtimeout_range
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] schedule_hrtimeout_range_clock
     0.01%     0.01%  kworker/1:2      [kernel.kallsyms]        [k] kmem_cache_alloc
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] sys_epoll_pwait
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] page_remove_rmap
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000000320000000b
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x0000000100000007
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] getname_flags
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __GI___libc_read
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] pick_next_task_fair
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] mem_cgroup_commit_charge
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005ea04
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] filename_lookup
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] path_lookupat
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] walk_component
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] lookup_fast
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] futex_wait_setup
     0.01%     0.00%  systemd          libpthread-2.23.so       [.] __GI___libc_open
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000b3a91
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] udp_pkt_to_tuple
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] nf_ct_get_tuple
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] complete_walk
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] rcu_eqs_enter_common.constprop.71
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_remove_reference
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ret_from_fork
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] call_usermodehelper_exec_async
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_execve
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] search_binary_handler
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] load_elf_binary
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_get_next_namepath
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ns_lookup
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] syscall_return_slowpath
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] exit_to_usermode_loop
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb72cae0
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb72cb20
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb72caa0
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb71a8e0
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb71a8a0
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb735b90
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb72cba0
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb72cb60
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000216d6
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] vsnprintf
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] seq_printf
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] seq_vprintf
     0.01%     0.01%  systemd-udevd    libc-2.23.so             [.] free
     0.01%     0.00%  irqbalance       [unknown]                [.] 0000000000000000
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x00007fb112170b78
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000046325
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ev_address_space_dispatch
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] schedule
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] __GI___getsockopt
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __vma_link_file
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000022031
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] __wake_up_locked
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] default_wake_function
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] error_entry
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000466345
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000004662e1
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000004659fa
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000465b1a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000465e19
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000046bf53
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000046d760
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000b712c
     0.01%     0.00%  containerd       containerd               [.] runtime.selectgo
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] anon_vma_interval_tree_remove
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005edaa
     0.01%     0.00%  dockerd          dockerd-ce               [.] internal/poll.(*pollDesc).waitRead
     0.01%     0.00%  kubelet          libc-2.23.so             [.] __mmap64
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_mmap_pgoff
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000005a536
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000602d3
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_call_control_method
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_init_aml_walk
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] vfs_read
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] __vfs_read
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000616be7
     0.01%     0.00%  kubelet          [unknown]                [k] 0xe900001388054824
     0.01%     0.00%  systemd-cgroups  ld-2.23.so               [.] 0x00000000002169b8
     0.01%     0.00%  (ifconfig)       [unknown]                [k] 0000000000000000
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000005265ce
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_delete_result_if_not_used
     0.01%     0.00%  systemd-journal  [unknown]                [.] 0x247c894800000008
     0.01%     0.00%  systemd-journal  systemd-journald         [.] 0x000000000002a1a0
     0.01%     0.00%  systemd-journal  [unknown]                [.] 0x0000000000000018
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] mempool_free
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] mempool_free_slab
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] futex_wake
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] wake_up_q
     0.01%     0.01%  kworker/1:1H     [kernel.kallsyms]        [k] process_one_work
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] bio_put
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] kobject_uevent
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] kobject_uevent_env
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] netlink_broadcast_filtered
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __netlink_sendskb
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] sock_def_readable
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __wake_up_sync_key
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __wake_up_common_lock
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __wake_up_common
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] ep_poll_callback
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __wake_up_locked
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] default_wake_function
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] try_to_wake_up
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] user_path_at_empty
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000031e99
     0.01%     0.01%  systemd-journal  systemd-journald         [.] 0x0000000000014dd2
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] prefetch_freepointer
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb7220b0
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).readMetaFrame
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2/hpack.(*Decoder).parseFieldIndexed
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2/hpack.(*Decoder).Write
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2/hpack.(*Decoder).parseHeaderFieldRepr
     0.01%     0.01%  systemd-udevd    libc-2.23.so             [.] __stpcpy_sse2_unaligned
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3639343932303733
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.usleep
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] sys_pselect6
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000035413
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d17
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] kstat_irqs_usr
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __dentry_kill
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] dentry_unlink_inode
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] iput
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] evict
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] destroy_inode
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __destroy_inode
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __fsnotify_inode_delete
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000012183
     0.01%     0.00%  kubelet          ld-2.23.so               [.] _dl_sysdep_start
     0.01%     0.00%  kubelet          ld-2.23.so               [.] dl_main
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __split_vma
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] split_vma
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d0c
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] handle_fasteoi_irq
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_irq
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_ev_sci_xrupt_handler
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_complete_op
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] load_new_mm_cr3
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d31
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.notesleep
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d3b
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] __pagevec_lru_add_fn
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000004cc9c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000053ee0
     0.01%     0.00%  systemd-udevd    [unknown]                [k] 0x0000000000000228
     0.01%     0.01%  zabbix_agentd    [kernel.kallsyms]        [k] mutex_lock
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] load_balance
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3038353932303733
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000578529
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] __do_fault
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] ext4_filemap_fault
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] filemap_fault
     0.01%     0.01%  kworker/u65:1    [kernel.kallsyms]        [k] console_unlock
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000012151
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sys_exit_group
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_group_exit
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_exit
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] lapic_next_deadline
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] need_update
     0.01%     0.01%  dockerd          dockerd-ce               [.] runtime.(*mcentral).cacheSpan
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.(*mcache).nextFree.func1
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.(*mcache).refill
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_wp_page
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] d_alloc
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] d_alloc_parallel
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader
     0.01%     0.00%  containerd       containerd               [.] io.ReadFull
     0.01%     0.00%  containerd       containerd               [.] io.ReadAtLeast
     0.01%     0.00%  containerd       containerd               [.] bufio.(*Reader).Read
     0.01%     0.00%  containerd       containerd               [.] net.(*conn).Read
     0.01%     0.00%  containerd       containerd               [.] net.(*netFD).Read
     0.01%     0.00%  containerd       containerd               [.] internal/poll.(*FD).Read
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] sock_read_iter
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] sock_recvmsg
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] unix_stream_recvmsg
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] unix_stream_read_generic
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] enqueue_entity
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_run_queue
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] blk_run_queue
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] intel_pstate_update_util
     0.01%     0.01%  systemd-cgroups  ld-2.23.so               [.] do_lookup_x
     0.01%     0.00%  systemd-cgroups  ld-2.23.so               [.] memset
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021cf6
     0.01%     0.00%  systemd          [unknown]                [k] 0000000000000000
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] release_pages
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] kstat_irqs_cpu
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] sys_exit_group
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] do_group_exit
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] do_exit
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] mmput
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] exit_mmap
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_syscall_64
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] filemap_map_pages
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] page_fault
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] do_page_fault
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] __do_page_fault
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] handle_mm_fault
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] __handle_mm_fault
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_is_result_used
     0.01%     0.01%  systemd-udevd    libc-2.23.so             [.] strlen
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] rcu_all_qs
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] unmap_page_range
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] unmap_vmas
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] unmap_single_vma
     0.01%     0.00%  rcu_sched        [kernel.kallsyms]        [k] schedule_timeout
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] resched_curr
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] wake_up_q
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] try_to_wake_up
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ttwu_do_activate
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ttwu_do_wakeup
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] check_preempt_curr
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __update_load_avg_se.isra.38
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] rxe_completer
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] udp_queue_rcv_skb
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] rxe_udp_encap_recv
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] rxe_rcv
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] rxe_comp_queue_pkt
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] rxe_run_task
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] new_slab
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] __slab_alloc
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ___slab_alloc
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d14
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000f082
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000000ea6d
     0.01%     0.00%  systemd          libc-2.23.so             [.] _int_free
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.wakep
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.startm
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).operateHeaders
     0.01%     0.01%  systemd-journal  [kernel.kallsyms]        [k] proc_pid_status
     0.01%     0.00%  containerd       containerd               [.] runtime.mallocgc
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_resolve_operands
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_update_object_reference
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x00000000ffffffff
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] kernel_fpu_begin
     0.01%     0.00%  modprobe         libc-2.23.so             [.] __GI___libc_read
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] blk_finish_plug
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] blk_flush_plug_list
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __elv_add_request
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] cfq_insert_request
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d1f
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] bio_free
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000030a62b
     0.01%     0.00%  containerd-shim  [unknown]                [.] 0xe80000016f850fc0
     0.01%     0.00%  containerd-shim  [unknown]                [.] 0x000000c4201d2fb0
     0.01%     0.00%  containerd-shim  containerd-shim          [.] 0x0000000000052ba9
     0.01%     0.00%  containerd-shim  containerd-shim          [.] 0x0000000000050577
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3736303732303733
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000003706c
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] new_slab
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] __slab_alloc
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] ___slab_alloc
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] acpi_os_read_port
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_hw_read
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_hw_read_port
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/runtime/restart/monitor.(*monitor).run
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/runtime/restart/monitor.(*monitor).reconcile
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/metadata.(*DB).View
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).View
     0.01%     0.00%  dockerd          dockerd-ce               [.] internal/poll.(*pollDesc).wait
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] vfs_statx
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] user_path_at_empty
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] filename_lookup
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] path_lookupat
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] walk_component
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] lookup_slow
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] ext4_lookup
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] 0x000000000001f8a0
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000056fbb1
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] swapgs_restore_regs_and_return_to_usermode
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] clear_page_erms
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000060b5d3
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000060c736
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] rcu_irq_exit
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3631363932303733
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_newfstat
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] SYSC_newfstat
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] perf_event_mmap
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] find_busiest_group
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __fdget_pos
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] task_numa_work
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] swapgs_restore_regs_and_return_to_usermode
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] prepare_exit_to_usermode
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] _raw_spin_unlock_irqrestore
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] security_file_permission
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] rw_verify_area
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] kfree
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] copy_user_generic_unrolled
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] entry_SYSCALL_64_stage2
     0.01%     0.01%  ib_send_bw       [unknown]                [k] 0xfffffe00003fa01b
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __srcu_read_lock
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] rdma_lookup_put_uobject
     0.01%     0.01%  ib_send_bw       [unknown]                [k] 0x00007fcf8110349e
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __fsnotify_parent
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __kmalloc
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] crc32_le
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] rxe_icrc_hdr
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] crypto_shash_update
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] crc32_pclmul_update
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] crc32_pclmul_le
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __alloc_skb
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] rxe_init_packet
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] sch_direct_xmit
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] dev_hard_start_xmit
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __netdev_pick_tx
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] netdev_pick_tx
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] mlx5e_select_queue
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] ip_send_check
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] ipt_do_table
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __nf_conntrack_find_get
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] ipv4_conntrack_local
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] nf_conntrack_in
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000008d3b36
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000008a01af
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b71e
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000024726
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000038b241
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000022a2b
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000024503
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b53d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000173ab
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000024d173
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c641a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c0d34
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c0cae
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000024caf7
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b529
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002eec9c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000c2647
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005b5f8
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000c5540
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000001851e
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000011054
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b534
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002ee762
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005f39b
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005d2c9
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000011cb9
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b879
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002ed075
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000f025
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000011182
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005d2f1
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000ef0b
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000017329
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] pmd_devmap_trans_unstable
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000015a00
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c5ae9
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000049f09
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fcaa9
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002f21fe
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000024e9f2
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000024d719
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000bfb75
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002ec34a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002ebdac
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fc64b
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000016a3e
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000049f84
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b4f9
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002f12bb
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000c31b5
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000022a26
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b599
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005d2d8
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b846
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fc5c4
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fbd61
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] alloc_set_pte
     0.01%     0.00%  systemd-journal  libc-2.23.so             [.] __fxstat64
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002f24d3
     0.01%     0.00%  systemd          [unknown]                [.] 0x00000000000000f1
     0.01%     0.00%  systemd          [unknown]                [.] 0x0000000000000361
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb804
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb838
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] hrtimer_force_reprogram
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] megasas_build_ldio_fusion
     0.01%     0.00%  kubelet          libc-2.23.so             [.] __GI___libc_connect
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_connect
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] SYSC_connect
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] unix_stream_connect
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] unix_find_other
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] kern_path
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] filename_lookup
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] path_lookupat
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] link_path_walk
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] walk_component
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] lookup_slow
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ext4_lookup
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ext4_iget_normal
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ext4_iget
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] __breadahead
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ll_rw_block
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] submit_bh_wbc
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] clear_asid_other
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003d7ed0
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005d2dc
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c69a5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002fbdfb
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006bb66
     0.01%     0.00%  systemd          [unknown]                [.] 0xf66487bb82814d87
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb7f9
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000156c9
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __GI___libc_fcntl
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] gup_pgd_range
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] get_futex_key
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] get_user_pages_fast
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000046503
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002fc3ed
     0.01%     0.01%  irqbalance       [vdso]                   [.] __vdso_gettimeofday
     0.01%     0.00%  irqbalance       irqbalance               [.] 0x00000000000060c3
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cb62c
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b745
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] unix_scm_to_skb
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002f230a
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006b8dd
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e24e05a0
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005d31a
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] dput
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fcaa1
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb800
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002ecc3d
     0.01%     0.01%  kworker/0:1      [kernel.kallsyms]        [k] kmem_cache_alloc
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000c9803
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e2581fe0
     0.01%     0.00%  systemd-udevd    [unknown]                [k] 0xbdddbbd14ef42b00
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000033c5b
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.stopm
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb6a6
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005d2fe
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000ca8f
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006ba6b
     0.01%     0.00%  systemd          [unknown]                [.] 0xaa47b93a15f1603e
     0.01%     0.01%  irqbalance       libc-2.23.so             [.] _int_malloc
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] __radix_tree_replace
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.sysmon
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000049f63
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002ed1a0
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000016e23
     0.01%     0.00%  kubelet          [unknown]                [.] 0x04c7167dc8394830
     0.01%     0.01%  systemd          libc-2.23.so             [.] _IO_default_uflow
     0.01%     0.00%  systemd          [unknown]                [.] 0xe25b27a000000000
     0.01%     0.00%  systemd          libc-2.23.so             [.] fscanf
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] kmem_cache_alloc
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] d_alloc_parallel
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] d_alloc
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] __d_alloc
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).handleData
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3634353932303733
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fc53c
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000001ce5da
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c031b
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001cf119
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] cfq_dispatch_requests
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002ed8e1
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] _raw_spin_lock
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] proc_get_inode
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] new_inode_pseudo
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000035395
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000001ed52
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000001ffde
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000002b230
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006bbc8
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002e871
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000038611a
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000004237
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000001cde39
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c02d9
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001cf0d2
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001cf81d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001d08d0
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] security_inode_free
     0.01%     0.00%  ib_send_bw       [unknown]                [k] 0x0000000002200c00
     0.01%     0.00%  ib_send_bw       libpthread-2.23.so       [.] __GI___libc_close
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] syscall_return_slowpath
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] ____fput
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] page_remove_rmap
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] __fput
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] dput
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] sys_exit_group
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] do_group_exit
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] dput.part.22
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] __dentry_kill
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] do_exit
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] dentry_unlink_inode
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] mmput
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] exit_mmap
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] iput
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] evict
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] unmap_vmas
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] destroy_inode
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] unmap_single_vma
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] __destroy_inode
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] unmap_page_range
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cbe76
     0.01%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] find_next_bit
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000015989
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fca97
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] vmacache_find
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006bb0f
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e25b5590
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000c5bd3
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002edbb2
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cc320
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e2501060
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] kstat_irqs
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] show_stat
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] kstat_irqs_usr
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000003c0368
     0.01%     0.00%  systemd-cgroups  [unknown]                [.] 0x42494c4700352e32
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] check_stack_object
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] copy_msghdr_from_user
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] import_iovec
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] rw_copy_check_uvector
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __check_object_size
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000049f75
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000001f4bc
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000026661
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000002679d
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] ep_poll
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb81a
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cc389
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e2573620
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ___pte_free_tlb
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] free_pgtables
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] arch_show_interrupts
     0.01%     0.00%  kubelet          [unknown]                [k] 0xc4834848246c8b48
     0.01%     0.00%  kubelet          [unknown]                [k] 0x000000c0000b8840
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000355c9
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] sync_regs
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000001560d
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000024721
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] try_to_wake_up
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] sock_def_readable
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __wake_up_sync_key
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __wake_up_common_lock
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __wake_up_common
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] pollwake
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] default_wake_function
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __vma_adjust
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] __rb_insert_augmented
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_mmap
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] vm_mmap_pgoff
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] lookup_mem
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] irq_to_desc
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000001626f
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002f17d0
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] idle_cpu
     0.01%     0.01%  perf             [kernel.kallsyms]        [k] do_page_mkwrite
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] do_wp_page
     0.01%     0.00%  systemd          [unknown]                [.] 0x9048638b47650068
     0.01%     0.00%  systemd          [unknown]                [k] 0x0000000100000031
     0.01%     0.00%  systemd          systemd                  [.] 0x00000000000ebca8
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] proc_tgid_base_lookup
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] proc_pident_lookup
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] proc_pident_instantiate
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] proc_pid_make_inode
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] new_inode
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] new_inode_pseudo
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] proc_alloc_inode
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_create_op
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006bb55
     0.01%     0.00%  systemd          [unknown]                [.] 0x3a6e4248c32088bf
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x00000000000308aa
     0.01%     0.01%  irqbalance       irqbalance               [.] 0x0000000000007b92
     0.01%     0.00%  irqbalance       [unknown]                [.] 0x000000000110508a
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] account_entity_dequeue
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000ca318
     0.01%     0.01%  kworker/24:0     [kernel.kallsyms]        [k] _find_next_bit
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] cpumask_next
     0.01%     0.01%  systemd          libc-2.23.so             [.] __stpcpy_sse2_unaligned
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_tgid_base_lookup
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000c97e9
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e25a6e50
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000004bf43d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000b7265
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000b9027
     0.01%     0.00%  rcu_sched        [kernel.kallsyms]        [k] force_qs_rnp
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] find_next_bit
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002600d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002ee952
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000c4edc
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000f598
     0.01%     0.00%  systemd          libc-2.23.so             [.] __closedir
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] _raw_spin_lock
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000004470e1
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000c9cc0
     0.01%     0.00%  containerd       containerd               [.] runtime.epollwait
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006b911
     0.01%     0.00%  kubelet          [unknown]                [k] 0xdd4bb00d8d482824
     0.01%     0.00%  kubelet          [unknown]                [k] 0x000000c00005d270
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000207a9
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000021907
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] generic_make_request_checks
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] blkg_lookup_slowpath
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b739
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002fc434
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] scsi_prep_fn
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006bb3c
     0.01%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] try_to_wake_up
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] schedule_hrtimeout_range_clock
     0.01%     0.00%  systemd          [unknown]                [.] 0x9d5546e5602c1c03
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000c93e7
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000c95ee
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] memchr_inv
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).operateHeaders
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] page_add_new_anon_rmap
     0.01%     0.00%  systemd          libc-2.23.so             [.] __vsnprintf_chk
     0.01%     0.00%  systemd          libc-2.23.so             [.] vfprintf
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000011985
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] ext4_map_blocks
     0.01%     0.01%  systemd          libc-2.23.so             [.] __libc_calloc
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).NewStream
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000024776
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] update_cfs_group
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006bb58
     0.01%     0.00%  systemd          [unknown]                [.] 0x366dbd1a9ebcc948
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] d_walk
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] detach_and_collect
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] select_collect
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] d_lru_del
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] list_lru_del
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] ___slab_alloc
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __slab_alloc
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002daf25
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000269c1a
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] blkcg_css_offline
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000229b8
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000d58385
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000d4fe58
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000bdd353
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000012373
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005f42f
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000016b2b
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000036153
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000021ab9
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] irq_fpu_usable
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] lock_page_memcg
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] page_add_file_rmap
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] copy_page
     0.01%     0.00%  systemd          [unknown]                [.] 0x0000000000000030
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cb586
     0.01%     0.00%  (ifconfig)       libc-2.23.so             [.] __getdents64
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] sys_getdents
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] iterate_dir
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] proc_readfd
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] proc_readfd_common
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] proc_fill_cache
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] proc_fd_instantiate
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] proc_pid_make_inode
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] new_inode
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] new_inode_pseudo
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] alloc_inode
     0.01%     0.01%  kubelet          ld-2.23.so               [.] memset
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fc06b
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] entry_SYSCALL_64_after_hwframe
     0.01%     0.00%  dockerd          [unknown]                [k] 0x637072672f6e6f69
     0.01%     0.00%  dockerd          [unknown]                [k] 0x0000000000000010
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] memcpy_erms
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] cmpxchg_double_slab.isra.61
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_remove_by_name_ns
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __kernfs_remove.part.12
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_put
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kfree_const
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kfree
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __slab_free
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] mlx5_cq_completion
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] napi_complete_done
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] kfree_skbmem
     0.01%     0.00%  kubelet          [unknown]                [.] 0xd2df058d48000001
     0.01%     0.00%  kubelet          [unknown]                [.] 0x000000c000470f18
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000005a8bd
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000001e77e
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000245e8
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] perf_event_task_tick
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] ip_send_check
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] compute_score
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] dev_gro_receive
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] inet_gro_receive
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] udp4_gro_receive
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] udp_gro_receive
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] udp4_lib_lookup_skb
     0.01%     0.00%  (kubelet)        libc-2.23.so             [.] __getdents64
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000006087a
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000d54b
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] fold_64
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] lookup_iova
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] rxe_get_av
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] swiotlb_map_page
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] udp_error
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] memcpy_erms
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] megasas_build_and_issue_cmd_fusion
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __breadahead
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] ll_rw_block
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] submit_bh_wbc
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] submit_bio
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] generic_make_request
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] blk_queue_bio
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __blk_run_queue
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] scsi_request_fn
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] scsi_dispatch_cmd
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] megasas_queue_command
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] get_partial_node.isra.72
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __local_bh_enable_ip
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] skb_network_protocol
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001c9551
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001b6123
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000001b3869
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000a3d86
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000a3e9f
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000a5f65
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000007ea10
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000081b4f
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000007e679
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000007e5b6
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] ipv4_pkt_to_tuple
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] skb_push
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000f19c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000239012
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000c9b8
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000602d7
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000239537
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000235d97
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000463f9
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cb62d
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000c9843
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e24b2380
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] loop_16
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] update_rq_clock
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5e_update_pport_counters
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5_core_access_reg
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3036353932303733
     0.01%     0.00%  zabbix_agentd    zabbix_agentd            [.] zbx_handle_log
     0.01%     0.00%  containerd       containerd               [.] runtime.step
     0.01%     0.00%  containerd       containerd               [.] runtime.funcspdelta
     0.01%     0.00%  containerd       containerd               [.] runtime.pcvalue
     0.01%     0.00%  systemd          [unknown]                [k] 0x00005626ffffffff
     0.01%     0.00%  systemd          systemd                  [.] 0x00000000000ee69f
     0.01%     0.00%  kubelet          [unknown]                [k] 0x00020b5a1508ee10
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] update_curr
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb719cd0
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000bfa5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000036fce
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000035dcc
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] aperfmperf_snapshot_khz
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/prometheus/client_golang/prometheus.(*metricVec).getMetricWithLabelValues
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/prometheus/client_golang/prometheus.(*CounterVec).WithLabelValues
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/prometheus/client_golang/prometheus.(*CounterVec).GetMetricWithLabelValues
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb719dc0
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000018534
     0.01%     0.00%  systemd-udevd    libpthread-2.23.so       [.] __GI___libc_recvmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_recvmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __sys_recvmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ___sys_recvmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sock_recvmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] unix_dgram_recvmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] skb_free_datagram
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] consume_skb
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kfree_skbmem
     0.01%     0.00%  systemd          [unknown]                [k] 0x00005626e24cb4d0
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/runtime/restart/monitor.(*monitor).monitor
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd.(*Client).Containers
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd.(*remoteContainers).List
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd.(*remoteContainers).stream
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/containers.(*local).ListStream
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/containers.(*local).withStoreView
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/containers.(*local).withStore.func1
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/containers.(*local).ListStream.func1
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/metadata.(*containerStore).List
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] get_page_from_freelist
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __getblk_gfp
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] pagecache_get_page
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __page_cache_alloc
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] alloc_pages_current
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __alloc_pages_nodemask
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] __close_fd
     0.01%     0.00%  kubelet          ld-2.23.so               [.] close
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_close
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cb563
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] radix_tree_node_ctor
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] add_to_page_cache_lru
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __add_to_page_cache_locked
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] page_cache_tree_insert
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __radix_tree_create
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] radix_tree_node_alloc.constprop.19
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] kmem_cache_alloc
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] new_slab
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000f1a85
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000023666a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002363ca
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002376e5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000f3e94
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000f3f10
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000237679
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000234c01
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000234b9e
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000f161d
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] _vm_normal_page
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000004f9bf5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000004f87e5
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b696
     0.01%     0.00%  kubelet          [unknown]                [k] 0x043d83000003d48f
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] memcmp
     0.01%     0.00%  systemd          libc-2.23.so             [.] __GI___libc_write
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] sys_write
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] vfs_write
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __vfs_write
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_fop_write
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup_file_write
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup1_procs_write
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] __cgroup1_procs_write.constprop.12
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup_attach_task
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] cgroup_migrate_prepare_dst
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ext4_lookup
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ext4_iget_normal
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ext4_iget
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006bb81
     0.01%     0.00%  systemd          [unknown]                [.] 0xdf2865bdb75b98f8
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000459a9b
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000181d7
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002ecc94
     0.01%     0.00%  zabbix_agentd    libc-2.23.so             [.] __nanosleep
     0.01%     0.00%  containerd       [unknown]                [k] 0x0000000000000001
     0.01%     0.00%  kubelet          [unknown]                [k] 0x0000000000000020
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000677fa5
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000ed97
     0.01%     0.00%  systemd          [unknown]                [.] 0x9f20db2b08951eaf
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] sync_regs
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] error_entry
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __memset_avx2
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_scope_stack_push
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] ixgbe_clean_rx_irq
     0.01%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] ret_from_fork
     0.01%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] kthread
     0.01%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] worker_thread
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] update_load_avg
     0.01%     0.00%  systemd          systemd                  [.] 0x00000000000574c2
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] online_css
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] cpu_cgroup_css_online
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] sched_online_group
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] online_fair_sched_group
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] attach_entity_cfs_rq
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] __queue_work
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] queue_work_on
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002b6bf
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] cpu_load_update_nohz_stop
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000603bd
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000239391
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005ea0b
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] tty_insert_flip_string_fixed_flag
     0.01%     0.00%  ib_send_bw       libc-2.23.so             [.] __GI___libc_write
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] tty_write
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] n_tty_write
     0.01%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] pty_write
     0.01%     0.00%  containerd       [unknown]                [k] 0x5024448b482575c0
     0.01%     0.00%  containerd       [unknown]                [k] 0x000000c420537680
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3832353932303733
     0.01%     0.00%  zabbix_agentd    [unknown]                [k] 0x3439353932303733
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] __msecs_to_jiffies
     0.01%     0.01%  ifconfig         libc-2.23.so             [.] _dl_addr
     0.01%     0.00%  ifconfig         [unknown]                [.] 0x0000077e0000077c
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_ev_gpe_detect
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] cgroup_exit
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] css_set_move_task
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] cgroup_update_populated
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] cgroup_file_notify
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] cpumask_next
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb7f0
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] deactivate_task
     0.01%     0.01%  systemd          libc-2.23.so             [.] free
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e249fdd0
     0.01%     0.00%  systemd          libc-2.23.so             [.] __opendir
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_iop_permission
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000002d4a0f5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa4ac
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000ba26cb
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000ba258b
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000b9e33f
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000b9e05e
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000b9d5d7
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000b3b1
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000004b4df
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000004b76d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000004d5ef
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000004ae9c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000046ce
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] scsi_dispatch_cmd
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] ll_rw_block
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] submit_bh_wbc
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] submit_bio
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] generic_make_request
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] blk_queue_bio
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __blk_run_queue
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] scsi_request_fn
     0.01%     0.00%  kubelet          [unknown]                [.] 0x000000c0000b0fb0
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] lock_page_memcg
     0.01%     0.01%  irqbalance       libc-2.23.so             [.] __GI_____strtoull_l_internal
     0.01%     0.00%  irqbalance       [unknown]                [.] 0x000000000110531b
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] cfq_dispatch_requests
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] do_open_execat
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] ext4_lookup
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] ext4_iget_normal
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] ext4_iget
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] __breadahead
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] ll_rw_block
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] submit_bh_wbc
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] submit_bio
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] generic_make_request
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] blk_queue_bio
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] __blk_run_queue
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] scsi_request_fn
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] blk_peek_request
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005edb9
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000239081
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000073f0e
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000010831
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa44d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000353289
     0.01%     0.01%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_system_memory_space_handler
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] timerqueue_add
     0.01%     0.00%  (rm)             systemd                  [.] 0x000000000006d1cb
     0.01%     0.00%  (rm)             libc-2.23.so             [.] __GI_setrlimit64
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] sys_setrlimit
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] security_task_setrlimit
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] apic_timer_interrupt
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] smp_apic_timer_interrupt
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] hrtimer_interrupt
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __hrtimer_run_queues
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] enqueue_hrtimer
     0.01%     0.01%  modprobe         [kernel.kallsyms]        [k] page_remove_rmap
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] unmap_vmas
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] unmap_single_vma
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] unmap_page_range
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003bc6ec
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003b574b
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] css_free_work_fn
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] pick_next_task_fair
     0.01%     0.01%  systemd          systemd                  [.] 0x000000000006b998
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000004cebd
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003c7171
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002eb862
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002fbe84
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000054238
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000004d73f
     0.01%     0.01%  kworker/1:2      [kernel.kallsyms]        [k] enqueue_entity
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] ttwu_do_activate
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] activate_task
     0.01%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] enqueue_task_fair
     0.01%     0.01%  systemd          [kernel.kallsyms]        [.] native_irq_return_iret
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] llist_reverse_order
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] update_cfs_group
     0.01%     0.00%  systemd-udevd    libpthread-2.23.so       [.] __sendmsg_nocancel
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_sendmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __sys_sendmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ___sys_sendmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sock_sendmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] netlink_sendmsg
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] netlink_unicast
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __netlink_sendskb
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sock_def_readable
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __wake_up_sync_key
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __wake_up_common_lock
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __wake_up_common
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ep_poll_callback
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __wake_up_locked
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] default_wake_function
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] try_to_wake_up
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ttwu_do_activate
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] activate_task
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] enqueue_task_fair
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] enqueue_entity
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] update_cfs_group
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] sys_write
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] vfs_write
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] __sched_text_start
     0.01%     0.01%  systemd          libc-2.23.so             [.] _IO_default_xsputn
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000460fb0
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] cpu_load_update
     0.01%     0.00%  dockerd          [unknown]                [k] 0x0000557d681f6490
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] perf_output_copy
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] perf_event_fork
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] perf_event_task
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] perf_iterate_sb
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] perf_event_task_output
     0.01%     0.01%  kworker/1:1H     [kernel.kallsyms]        [k] update_load_avg
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [k] MR_BuildRaidContext
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __blk_run_queue
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] scsi_request_fn
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] scsi_dispatch_cmd
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] megasas_queue_command
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] megasas_build_and_issue_cmd_fusion
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] megasas_build_io_fusion
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] megasas_build_ldio_fusion
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000010f28
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000239334
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [k] lapic_next_deadline
     0.01%     0.00%  ifconfig         [unknown]                [k] 0x6c61636f6c2f6572
     0.01%     0.00%  ifconfig         libc-2.23.so             [.] _nl_find_domain
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] apic_timer_interrupt
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] smp_apic_timer_interrupt
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] hrtimer_interrupt
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] tick_program_event
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_munmap
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] unmap_region
     0.01%     0.00%  ib_send_bw       [unknown]                [k] 0x6970762074706520
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [k] prefetch_freepointer
     0.01%     0.00%  ifconfig         ld-2.23.so               [.] access
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sys_access
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] prepare_creds
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [.] native_irq_return_iret
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] _vm_normal_page
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] search_binary_handler
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] load_elf_binary
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] flush_old_exec
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] mmput
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] exit_mmap
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] unmap_vmas
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] unmap_single_vma
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] css_killed_work_fn
     0.01%     0.01%  rm               [kernel.kallsyms]        [k] clear_page_erms
     0.01%     0.00%  rm               ld-2.23.so               [.] _dl_allocate_tls_storage
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] alloc_pages_vma
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] __alloc_pages_nodemask
     0.01%     0.01%  rcu_sched        [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.01%     0.00%  systemd          [unknown]                [.] 0x0000000000000040
     0.01%     0.00%  systemd-udevd    libkmod.so.2.3.0         [.] 0x00000000000047b7
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] filename_create
     0.01%     0.01%  dbus-daemon      [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.01%     0.00%  dbus-daemon      libdbus-1.so.3.14.6      [.] 0x000000000002925e
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000f0ee
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000b910b6
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] workingset_activation
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] ext4_find_entry
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] ext4_bread_batch
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] ext4_getblk
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __find_get_block
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] mark_page_accessed
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000605c1
     0.01%     0.01%  systemd-journal  systemd-journald         [.] 0x0000000000014e3a
     0.01%     0.00%  containerd       containerd               [.] runtime.gfget.func1
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002eb686
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] down_read
     0.01%     0.00%  (ifconfig)       libpthread-2.23.so       [.] __GI___libc_open
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] sys_open
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] do_sys_open
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] link_path_walk
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] walk_component
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] lookup_slow
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [k] native_sched_clock
     0.01%     0.00%  ifconfig         libc-2.23.so             [.] __GI___execve
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sys_execve
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] search_binary_handler
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] load_elf_binary
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] clear_user
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __clear_user
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] page_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] do_page_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __do_page_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] handle_mm_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __handle_mm_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __do_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ext4_filemap_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] filemap_fault
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __do_page_cache_readahead
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ext4_readpages
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ext4_mpage_readpages
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] get_request
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] blk_rq_init
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sched_clock
     0.01%     0.00%  kubelet          [unknown]                [.] 0x000000c0004717b0
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x0000000000032d10
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x62616c732f6c656e
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] cfq_add_rq_rb
     0.01%     0.01%  modprobe         libc-2.23.so             [.] __strchrnul
     0.01%     0.00%  modprobe         libc-2.23.so             [.] __vsnprintf_chk
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] free_unref_page
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] free_pgd_range
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ___pte_free_tlb
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] tlb_remove_table
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] free_page_and_swap_cache
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __put_page
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] new_slab
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __slab_alloc
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ___slab_alloc
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] mm_init
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] pgd_alloc
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __get_free_pages
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] unlock_page
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x0000000000032032
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000e9ce
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] _atomic_dec_and_lock
     0.01%     0.00%  systemd          [unknown]                [k] 0x0000000000000231
     0.01%     0.00%  systemd          [unknown]                [k] 0x00007f375ede1b78
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fc44b
     0.01%     0.00%  kubelet          [unknown]                [.] 0x000000c0000b17b0
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] pick_next_task_fair
     0.01%     0.00%  systemd-cgroups  systemd-cgroups-agent    [.] 0x000000000002e749
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] ppoll
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_ppoll
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_sys_poll
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] poll_schedule_timeout
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] schedule_hrtimeout_range
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] schedule_hrtimeout_range_clock
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] schedule
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __sched_text_start
     0.01%     0.01%  (kubelet)        [kernel.kallsyms]        [k] ext4_file_open
     0.01%     0.00%  (kubelet)        [unknown]                [k] 0xf566cfe808ec8348
     0.01%     0.00%  (kubelet)        systemd                  [.] 0x00000000000cb600
     0.01%     0.00%  (kubelet)        [unknown]                [.] 0x00005626e254f4f0
     0.01%     0.00%  (kubelet)        [unknown]                [.] 0x00005626e24640b0
     0.01%     0.00%  (kubelet)        [unknown]                [.] 0x0000000100000000
     0.01%     0.00%  (kubelet)        systemd                  [.] 0x00000000000d7e1a
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] do_open_execat
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] do_filp_open
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] path_openat
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] vfs_open
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] do_dentry_open
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000cbd94
     0.01%     0.00%  kubelet          [unknown]                [k] 0x030c40c708488948
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000003b5f460
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] cap_bprm_set_creds
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] prepare_binprm
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] security_bprm_set_creds
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [k] __indirect_thunk_start
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] bio_alloc_bioset
     0.01%     0.01%  rm               [kernel.kallsyms]        [k] _raw_spin_lock
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] sys_exit_group
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] do_group_exit
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] do_exit
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] exit_io_context
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] put_io_context_active
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] put_io_context
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] queue_work_on
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] __queue_work
     0.01%     0.00%  modprobe         [unknown]                [k] 0000000000000000
     0.01%     0.01%  rm               [kernel.kallsyms]        [k] bio_associate_blkcg
     0.01%     0.00%  rm               libc-2.23.so             [.] __GI___execve
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] do_syscall_64
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] sys_execve
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] search_binary_handler
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] load_elf_binary
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] clear_user
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] __clear_user
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] __do_fault
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] ext4_filemap_fault
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] filemap_fault
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] __do_page_cache_readahead
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] ext4_readpages
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] ext4_mpage_readpages
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] submit_bio
     0.01%     0.00%  rm               [kernel.kallsyms]        [k] generic_make_request
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000bc7f40
     0.01%     0.01%  perf             [kernel.kallsyms]        [k] ext4_data_block_valid
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] open_exec
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] ext4_ext_check_inode
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] __ext4_ext_check
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] __mod_node_page_state
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] unmap_region
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] lru_add_drain
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] lru_add_drain_cpu
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] pagevec_lru_move_fn
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __pagevec_lru_add_fn
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000f0fb
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000236692
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] eventpoll_release_file
     0.01%     0.00%  dockerd          [unknown]                [k] 0x0000000000000020
     0.01%     0.01%  zabbix_agentd    [kernel.kallsyms]        [k] __radix_tree_lookup
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000060d42
     0.01%     0.00%  kubelet          [unknown]                [.] 0x244c8908244c8b18
     0.01%     0.00%  kubelet          [unknown]                [.] 0x000000005dd1f1cc
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] ktime_get_update_offsets_now
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] cfq_insert_request
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] __do_page_cache_readahead
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] blk_finish_plug
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] blk_flush_plug_list
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] __elv_add_request
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x0000000000040339
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] sys_epoll_pwait
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] ep_poll
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] schedule_hrtimeout_range
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] from_kuid_munged
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] __fxstat64
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_newfstat
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] SYSC_newfstat
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] cp_new_stat
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*controlBuffer).get
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000f2568
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] tick_check_oneshot_broadcast_this_cpu
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] mutex_spin_on_owner
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000016e2a
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] unmap_page_range
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000032391
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000002d979
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000038a81
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000028661
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] _raw_spin_trylock
     0.01%     0.00%  systemd-udevd    libpthread-2.23.so       [.] __waitpid
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_wait4
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] SYSC_wait4
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kernel_wait4
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_wait
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] wait_consider_task
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] release_task
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_flush_task
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] d_invalidate
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] shrink_dentry_list
     0.01%     0.01%  systemd-udevd    [unknown]                [k] 0xfffffe00001be025
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] __x86_indirect_thunk_rdx
     0.01%     0.00%  ib_send_bw       [unknown]                [k] 0x7274636570732000
     0.01%     0.00%  rcu_sched        [kernel.kallsyms]        [k] schedule
     0.01%     0.00%  rcu_sched        [kernel.kallsyms]        [k] __sched_text_start
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] mlx5e_post_rx_wqes
     0.01%     0.00%  systemd-udevd    [unknown]                [k] 0x3228746375727473
     0.01%     0.00%  containerd       containerd               [.] runtime.ready
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*controlBuffer).put
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*controlBuffer).executeAndPut
     0.01%     0.00%  containerd       containerd               [.] runtime.selectnbsend
     0.01%     0.00%  containerd       containerd               [.] runtime.chansend
     0.01%     0.00%  containerd       containerd               [.] runtime.send
     0.01%     0.00%  containerd       containerd               [.] runtime.goready.func1
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [.] native_irq_return_iret
     0.01%     0.00%  ifconfig         ld-2.23.so               [.] dl_main
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] css_next_descendant_pre
     0.01%     0.00%  systemd          systemd                  [.] 0x00000000000e2698
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000060dce
     0.01%     0.01%  systemd-udevd    [unknown]                [k] 0x000055ccdb13cb1e
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000000e5d8
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000028b1e
     0.01%     0.00%  modprobe         [unknown]                [k] 0xc4b0a06504003062
     0.01%     0.00%  modprobe         kmod                     [.] 0x0000000000011070
     0.01%     0.01%  rm               [kernel.kallsyms]        [k] filemap_map_pages
     0.01%     0.00%  rm               [unknown]                [k] 0x5f346573735f706d
     0.01%     0.00%  rm               [unknown]                [k] 0000000000000000
     0.01%     0.00%  rm               libc-2.23.so             [.] 0x00000000003c0ee0
     0.01%     0.00%  rm               libc-2.23.so             [.] setlocale
     0.01%     0.00%  rm               libc-2.23.so             [.] _nl_find_locale
     0.01%     0.00%  rm               libc-2.23.so             [.] _nl_load_locale_from_archive
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002fbd31
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] pick_next_entity
     0.01%     0.01%  kworker/25:2     [kernel.kallsyms]        [k] get_random_u64
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] ret_from_fork
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] kthread
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] worker_thread
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] process_one_work
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] memcg_kmem_cache_create_func
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] memcg_create_kmem_cache
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] create_cache
     0.01%     0.00%  kworker/25:2     [kernel.kallsyms]        [k] __kmem_cache_create
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] reweight_entity
     0.01%     0.01%  (ifconfig)       [unknown]                [k] 0x00007f375ea97d17
     0.01%     0.00%  (ifconfig)       libc-2.23.so             [.] __dprintf_chk
     0.01%     0.00%  (ifconfig)       libc-2.23.so             [.] _IO_link_in
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] rwsem_down_write_failed
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] down_write
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] call_rwsem_down_write_failed
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000053a2
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000b2054b
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] legitimize_links
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021a6d
     0.01%     0.01%  systemd-journal  systemd-journald         [.] 0x000000000001f900
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x0000000000008421
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000036512
     0.01%     0.00%  zabbix_agentd    zabbix_agentd            [.] 0x000000000000ffd9
     0.01%     0.00%  zabbix_agentd    zabbix_agentd            [.] zbx_tcp_connect
     0.01%     0.00%  zabbix_agentd    zabbix_agentd            [.] 0x0000000000041ccd
     0.01%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] set_next_entity
     0.01%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] pick_next_task_fair
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x00000000000169d8
     0.01%     0.00%  systemd          [unknown]                [k] 0x00005626e24b62b0
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000384544
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] css_set_move_task
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] cgroup_exit
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] hrtimer_active
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] clear_user
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __clear_user
     0.01%     0.01%  modprobe         [unknown]                [k] 0x00007fd22a62b671
     0.01%     0.00%  irqbalance       [unknown]                [k] 0x7374707572726574
     0.01%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).handle
     0.01%     0.01%  ifconfig         [kernel.kallsyms]        [k] inode_init_once
     0.01%     0.00%  ifconfig         [unknown]                [k] 0x4100000000000000
     0.01%     0.00%  ifconfig         libc-2.23.so             [.] __socket
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sys_socket
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __sock_create
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sock_alloc
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] new_inode_pseudo
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] alloc_inode
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] sock_alloc_inode
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] __slab_alloc
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] ___slab_alloc
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] new_slab
     0.01%     0.00%  ifconfig         [kernel.kallsyms]        [k] init_once
     0.01%     0.01%  systemd-cgroups  ld-2.23.so               [.] _dl_fini
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] 0x00000000003c6060
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] __run_exit_handlers
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000016ad9
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] tsc_verify_tsc_adjust
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] arch_cpu_idle_enter
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000002f2973
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000002f2853
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] cmpxchg_double_slab.isra.61
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] remove_vma
     0.01%     0.01%  systemd-udevd    [unknown]                [k] 0x00007fb111ea3315
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x000000000002d1b0
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x7478652f62616c73
     0.01%     0.01%  (kubelet)        [kernel.kallsyms]        [k] filename_lookup
     0.01%     0.00%  (kubelet)        libc-2.23.so             [.] __fxstatat64
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] sys_newfstatat
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] SYSC_newfstatat
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] vfs_statx
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] user_path_at_empty
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] memcpy_erms
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] __GI___libc_read
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_read
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] vfs_read
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __vfs_read
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] proc_reg_read
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] seq_read
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] filesystems_proc_show
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] seq_printf
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] seq_vprintf
     0.01%     0.01%  systemd-udevd    [unknown]                [k] 0xfffffe000037601b
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] _cond_resched
     0.01%     0.01%  kubelet          kubelet                  [.] 0x00000000000230d3
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000022ed1
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000e806
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_update_ref_count.part.1
     0.01%     0.00%  systemd          [unknown]                [.] 0x9c3bf764d6cf00fe
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] put_cpu_partial
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] __rb_erase_color
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002200b
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d12
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000037084
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000036d84
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000035a58
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000035629
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000034f49
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] megasas_clear_intr_fusion
     0.01%     0.01%  systemd-udevd    libc-2.23.so             [.] _IO_fgets
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x0000000000000017
     0.01%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] ret_from_fork
     0.01%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] kthread
     0.01%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] smpboot_thread_fn
     0.01%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] run_ksoftirqd
     0.01%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] __softirqentry_text_start
     0.01%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] rcu_process_callbacks
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] kstat_irqs
     0.01%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2/hpack.(*Decoder).callEmit
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] consume_skb
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] skb_release_all
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] skb_release_head_state
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] unix_destruct_scm
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] sock_wfree
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] unix_write_space
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] __wake_up_sync_key
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] __wake_up_common_lock
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] __wake_up_common
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] ep_poll_callback
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] __wake_up_locked
     0.01%     0.00%  containerd       [kernel.kallsyms]        [k] default_wake_function
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] x86_pmu_enable
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] intel_pmu_enable_all
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] __intel_pmu_enable_all.constprop.19
     0.01%     0.01%  systemd-cgroups  [kernel.kallsyms]        [k] sys_getsockopt
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] page_counter_try_charge
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] memcg_kmem_charge_memcg
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb735d90
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] _raw_spin_trylock
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] free_unref_page_commit
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] free_unref_page_list
     0.01%     0.01%  kworker/25:1H    [kernel.kallsyms]        [k] blk_rq_map_sg
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] ret_from_fork
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] kthread
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] worker_thread
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] process_one_work
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] cfq_kick_queue
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] __blk_run_queue
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] scsi_request_fn
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] blk_peek_request
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] scsi_prep_fn
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] scsi_setup_cmnd
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] sd_init_command
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] sd_setup_read_write_cmnd
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] scsi_init_io
     0.01%     0.00%  kworker/25:1H    [kernel.kallsyms]        [k] scsi_init_sgtable
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] fopen@@GLIBC_2.2.5
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] intel_pstate_update_util
     0.01%     0.01%  perf             [kernel.kallsyms]        [k] percpu_counter_add_batch
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] __blk_run_queue
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] scsi_request_fn
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] blk_peek_request
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] cfq_dispatch_requests
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] cfq_dispatch_insert
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] cfq_remove_request
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000012367
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000023914a
     0.01%     0.00%  (ifconfig)       libc-2.23.so             [.] 0x000000000001f8a0
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] sync_regs
     0.01%     0.00%  (ifconfig)       systemd                  [.] 0x0000000000069650
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] entry_SYSCALL_64_stage2
     0.01%     0.00%  (ifconfig)       libc-2.23.so             [.] __opendir
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __put_anon_vma
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000461064
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000419d5d
     0.01%     0.01%  (kubelet)        [kernel.kallsyms]        [k] unmap_page_range
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] search_binary_handler
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] load_elf_binary
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] flush_old_exec
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] mmput
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] exit_mmap
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] unmap_vmas
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] unmap_single_vma
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005ee27
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000cab99c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000c48cfb
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000bf2265
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000be24d0
     0.01%     0.01%  ifconfig         libc-2.23.so             [.] _int_malloc
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] insert_work
     0.01%     0.01%  sleep            libc-2.23.so             [.] getenv
     0.01%     0.00%  sleep            [unknown]                [.] 0x6f722f3d454d4f48
     0.01%     0.00%  sleep            [unknown]                [.] 0x00007fff850ceda3
     0.01%     0.01%  ib_send_bw       libc-2.23.so             [.] _IO_setb
     0.01%     0.00%  ib_send_bw       [unknown]                [.] 0x202952286e6f6500
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] poll_health
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] __errno_location@plt
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000020f5a
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] finish_task_switch
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __GI___setsockopt
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.growslice
     0.01%     0.00%  dockerd          [unknown]                [k] 0x058b000003d98f0f
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x0000000000040975
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] mntput_no_expire
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] mntput
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.makechan
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] futex_wait_queue_me
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] _raw_spin_lock_irq
     0.01%     0.00%  (rm)             libc-2.23.so             [.] __GI___libc_write
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] sys_write
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] vfs_write
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __vfs_write
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] kernfs_fop_write
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] cgroup_file_write
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] cgroup1_procs_write
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] __cgroup1_procs_write.constprop.12
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] cgroup_attach_task
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] cgroup_migrate_prepare_dst
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] find_css_set
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] ktime_get
     0.01%     0.01%  swapper          [kernel.kallsyms]        [k] decay_load
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] anon_vma_clone
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000816348
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] syscall_return_via_sysret
     0.01%     0.01%  systemd-udevd    libc-2.23.so             [.] getpid
     0.01%     0.01%  systemd-journal  systemd-journald         [.] 0x000000000002c73c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003b8e66
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000003bc0bf
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000d0ed
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x00000000216bf26c
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __clock_gettime
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000000f3d2
     0.01%     0.01%  kworker/0:1      [kernel.kallsyms]        [k] strncpy
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_init_op
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb7217f0
     0.01%     0.01%  kubelet          [kernel.kallsyms]        [k] __default_send_IPI_dest_field
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] native_smp_send_reschedule
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] default_send_IPI_single_phys
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] __follow_mount_rcu.isra.27
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] link_path_walk
     0.01%     0.01%  systemd-cgroups  libc-2.23.so             [.] free
     0.01%     0.01%  kworker/u66:3    [kernel.kallsyms]        [k] find_busiest_group
     0.01%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] schedule
     0.01%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] __sched_text_start
     0.01%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] pick_next_task_fair
     0.01%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] load_balance
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000002153e
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005ea09
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] find_busiest_group
     0.01%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] load_balance
     0.01%     0.00%  systemd-udevd    [unknown]                [k] 0x3139323532383228
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb7195a0
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000029a0d
     0.01%     0.01%  irqbalance       [kernel.kallsyms]        [k] _find_next_bit
     0.01%     0.00%  irqbalance       [kernel.kallsyms]        [k] cpumask_next
     0.01%     0.01%  systemd-journal  [kernel.kallsyms]        [k] _raw_spin_lock
     0.01%     0.00%  systemd-journal  libc-2.23.so             [.] readlinkat
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] sys_readlinkat
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] pid_revalidate
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] task_dump_owner
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000060b38a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000606238
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000603fca
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000016abc5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000cdd8
     0.01%     0.00%  containerd       containerd               [.] syscall.Read
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] __srcu_read_lock
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] fsnotify_destroy_marks
     0.01%     0.00%  dbus-daemon      [unknown]                [k] 0x00005603374ad508
     0.01%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_do_implicit_return
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000011bed
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] ep_free
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ep_eventpoll_release
     0.01%     0.00%  systemd-cgroups  [unknown]                [.] 0x6e735f5f00352e32
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d57
     0.01%     0.00%  dockerd          [unknown]                [k] 0x0000c8888b48c320
     0.01%     0.00%  dockerd          [unknown]                [k] 0x000000c420459e38
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021e02
     0.01%     0.01%  (kubelet)        [kernel.kallsyms]        [k] mutex_spin_on_owner
     0.01%     0.00%  (kubelet)        libc-2.23.so             [.] __GI___libc_write
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] sys_write
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] vfs_write
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] __vfs_write
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] kernfs_fop_write
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] cgroup_file_write
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] cgroup1_procs_write
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] __cgroup1_procs_write.constprop.12
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] cgroup_kn_lock_live
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] mutex_lock
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] __mutex_lock_slowpath
     0.01%     0.00%  (kubelet)        [kernel.kallsyms]        [k] __mutex_lock.isra.2
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] kfree
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_fop_release
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] seq_release
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kvfree
     0.01%     0.01%  perf             [kernel.kallsyms]        [k] __ext4_journal_get_write_access
     0.01%     0.01%  perf             [kernel.kallsyms]        [k] free_page_and_swap_cache
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] flush_old_exec
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] mmput
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] exit_mmap
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] free_pgtables
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] free_pgd_range
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] ___pte_free_tlb
     0.01%     0.00%  perf             [kernel.kallsyms]        [k] tlb_remove_table
     0.01%     0.01%  zabbix_agentd    [unknown]                [k] 0xfffffe00001ea01b
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __breadahead
     0.01%     0.01%  systemd          systemd                  [.] 0x00000000000a79e0
     0.01%     0.00%  systemd          [unknown]                [.] 0x00005626e247caa0
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d27
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000004a433
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] syscall
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_getrandom
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] urandom_read
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x646e65765f737973
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x000000000003131a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000fc18b
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000fbcfc
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000fc39d
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000f498
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x000000000001daa3
     0.01%     0.01%  systemd-journal  [kernel.kallsyms]        [k] inode_init_always
     0.01%     0.00%  systemd-journal  libc-2.23.so             [.] 0x00000000001766e0
     0.01%     0.00%  systemd-journal  libc-2.23.so             [.] __GI___libc_open
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] sys_open
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] do_sys_open
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] do_filp_open
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] path_openat
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] shmem_create
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] shmem_mknod
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] shmem_get_inode
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] new_inode
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] new_inode_pseudo
     0.01%     0.00%  systemd-journal  [kernel.kallsyms]        [k] alloc_inode
     0.01%     0.01%  dockerd          [kernel.kallsyms]        [k] native_sched_clock
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] update_rq_clock
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] sched_clock_cpu
     0.01%     0.00%  dockerd          [kernel.kallsyms]        [k] sched_clock
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] _find_next_bit
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_task_dead
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __sched_text_start
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] pick_next_task_fair
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] load_balance
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] find_busiest_group
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] cpumask_next_and
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] mutex_unlock
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_iop_lookup
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_get_active.part.13
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_read
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vfs_read
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __vfs_read
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_fop_read
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] seq_read
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_seq_start
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] kernfs_get_active
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000000f18f
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000049f7f
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __libc_calloc
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] __init_rwsem
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_iop_lookup
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] kernfs_get_inode
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] iget_locked
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] inode_init_always
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] pte_alloc_one
     0.01%     0.00%  systemd-cgroups  libc-2.23.so             [.] __brk
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_brk
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_brk_flags
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __vma_link_rb
     0.01%     0.01%  sleep            [kernel.kallsyms]        [k] __vma_adjust
     0.01%     0.00%  sleep            ld-2.23.so               [.] _dl_sysdep_start
     0.01%     0.00%  sleep            ld-2.23.so               [.] dl_main
     0.01%     0.00%  sleep            ld-2.23.so               [.] mprotect
     0.01%     0.00%  sleep            [kernel.kallsyms]        [k] sys_mprotect
     0.01%     0.00%  sleep            [kernel.kallsyms]        [k] do_mprotect_pkey
     0.01%     0.00%  sleep            [kernel.kallsyms]        [k] mprotect_fixup
     0.01%     0.00%  sleep            [kernel.kallsyms]        [k] split_vma
     0.01%     0.00%  sleep            [kernel.kallsyms]        [k] __split_vma
     0.01%     0.00%  dockerd          [unknown]                [k] 0x484d75c08548008b
     0.01%     0.00%  dockerd          dockerd-ce               [.] runtime.notetsleep_internal
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x000000000002c363
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x20676e697070696b
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] tlb_finish_mmu
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] arch_tlb_finish_mmu
     0.01%     0.01%  kubelet          kubelet                  [.] 0x000000000005edc7
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000d5837b
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000cbb437
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ret_from_intr
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_IRQ
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] irq_exit
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __softirqentry_text_start
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] blk_done_softirq
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] scsi_softirq_done
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] scsi_finish_command
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] scsi_io_completion
     0.01%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] scsi_end_request
     0.01%     0.01%  sleep            ld-2.23.so               [.] _dl_start
     0.01%     0.01%  ib_send_bw       [kernel.kallsyms]        [k] copy_user_enhanced_fast_string
     0.01%     0.00%  ib_send_bw       [unknown]                [k] 0x2065737020656420
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021d99
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000021de2
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000060396
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000006b6107
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000056fc7c
     0.01%     0.01%  modprobe         [kernel.kallsyms]        [k] vma_interval_tree_remove
     0.01%     0.00%  modprobe         ld-2.23.so               [.] dl_main
     0.01%     0.00%  modprobe         ld-2.23.so               [.] mprotect
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] sys_mprotect
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] do_mprotect_pkey
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] mprotect_fixup
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] split_vma
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] __split_vma
     0.01%     0.00%  modprobe         [kernel.kallsyms]        [k] __vma_adjust
     0.01%     0.00%  modprobe         ld-2.23.so               [.] _dl_start_user
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_gp_kthread_wake
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] swake_up
     0.01%     0.01%  kubelet          kubelet                  [.] 0x0000000000017f0c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000f251e
     0.01%     0.00%  systemd-udevd    [unknown]                [k] 0x3532383228656863
     0.01%     0.01%  (ifconfig)       [kernel.kallsyms]        [k] copy_user_enhanced_fast_string
     0.01%     0.00%  (ifconfig)       libc-2.23.so             [.] __fxstatat64
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] sys_newfstatat
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] SYSC_newfstatat
     0.01%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] cp_new_stat
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000238d40
     0.01%     0.01%  (rm)             [kernel.kallsyms]        [k] __cmpxchg_double_slab.isra.51
     0.01%     0.00%  (rm)             [unknown]                [k] 0000000000000000
     0.01%     0.00%  (rm)             libc-2.23.so             [.] __GI___setsid
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] sys_setsid
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] sched_autogroup_create_attach
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] sched_create_group
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] alloc_fair_sched_group
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] kmem_cache_alloc_node_trace
     0.01%     0.00%  (rm)             [kernel.kallsyms]        [k] deactivate_slab.isra.70
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000036086
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000002b5de
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000002b867
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000000244ec
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] down_read
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] call_rwsem_down_read_failed
     0.01%     0.00%  kubelet          [kernel.kallsyms]        [k] rwsem_down_read_failed
     0.01%     0.01%  systemd          [kernel.kallsyms]        [k] native_flush_tlb_single
     0.01%     0.00%  systemd          systemd                  [.] 0x000000000004bd82
     0.01%     0.00%  systemd          libaudit.so.1.0.0        [.] audit_log_user_comm_message
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] ptep_clear_flush
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] flush_tlb_mm_range
     0.01%     0.00%  systemd          [kernel.kallsyms]        [k] flush_tlb_func_common.constprop.10
     0.01%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000004a24d
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] putname
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000a6b96c
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000056f9e5
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000000f407
     0.01%     0.00%  swapper          [kernel.kallsyms]        [k] cpumask_next_and
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x000000000002bf57
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x0000000000000010
     0.01%     0.00%  kubelet          [unknown]                [k] 0x000000000000000c
     0.01%     0.01%  kworker/24:0     [kernel.kallsyms]        [k] kfree
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] sysfs_slab_remove_workfn
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] kobject_del
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] kobject_del.part.5
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] sysfs_remove_dir
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] kernfs_remove
     0.01%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] __kernfs_remove.part.12
     0.01%     0.00%  dockerd          dockerd-ce               [.] syscall.Read
     0.01%     0.01%  systemd-udevd    [kernel.kallsyms]        [k] ep_send_events_proc
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] __epoll_wait_nocancel
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_epoll_wait
     0.01%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ep_poll
     0.01%     0.01%  kubelet          ld-2.23.so               [.] _dl_map_object_from_fd
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x0000000000016f40
     0.01%     0.01%  systemd-udevd    [unknown]                [k] 0x00007fb111ea303e
     0.01%     0.00%  systemd-udevd    libc-2.23.so             [.] vfprintf
     0.01%     0.00%  kubelet          kubelet                  [.] 0x00000000004470aa
     0.01%     0.00%  kubelet          kubelet                  [.] 0x0000000000446631
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000043d18a
     0.01%     0.00%  kubelet          kubelet                  [.] 0x000000000043d305
     0.01%     0.01%  containerd       containerd               [.] runtime.runqempty
     0.01%     0.01%  systemd-udevd    systemd-udevd            [.] 0x0000000000031f1c
     0.01%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb72ae80
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000677d1e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000628730
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x00000000000308a3
     0.00%     0.00%  systemd-udevd    libpthread-2.23.so       [.] __nptl_set_robust
     0.00%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] __alloc_percpu
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005d16a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003f2dc
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] _IO_getdelim
     0.00%     0.00%  systemd-cgroups  [unknown]                [.] 0x0000564f30af73a8
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5e_update_sw_counters
     0.00%     0.00%  kubelet          kubelet                  [.] main
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000601c5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000cb48c7
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000caf9d9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000cae9f9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000bcda97
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000bcd38d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000007a19
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000cb808e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000cb6043
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000cb5895
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000060c2d
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_nanosleep
     0.00%     0.00%  systemd-cgroups  [unknown]                [k] 0xfffffe000052e03a
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] ext4_do_update_inode
     0.00%     0.00%  rs:main Q:Reg    [unknown]                [k] 0000000000000000
     0.00%     0.00%  rs:main Q:Reg    libpthread-2.23.so       [.] 0x00000000000104bd
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] sys_write
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] vfs_write
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] __vfs_write
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] ext4_file_write_iter
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] __generic_file_write_iter
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] generic_perform_write
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] ext4_da_write_end
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] ext4_mark_inode_dirty
     0.00%     0.00%  rs:main Q:Reg    [kernel.kallsyms]        [k] ext4_mark_iloc_dirty
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] hash_futex
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] arch_align_stack
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000033744
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0xdb735ff0bdddbbd1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000002e8a7
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000239345
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __es_insert_extent
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] ext4_es_insert_extent
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ioread32be
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] in_fatal
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] ext4_map_blocks
     0.00%     0.00%  sleep            ld-2.23.so               [.] _dl_map_object
     0.00%     0.00%  sleep            ld-2.23.so               [.] open64
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] sys_open
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] do_sys_open
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] do_filp_open
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] path_openat
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] ext4_lookup
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] ext4_find_entry
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] ext4_dx_find_entry
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] __ext4_read_dirblock
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] ext4_bread
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] ext4_getblk
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] do_semtimedop
     0.00%     0.00%  zabbix_agentd    zabbix_agentd            [.] 0x0000000000038a74
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] scsi_dma_map
     0.00%     0.00%  zabbix_agentd    libc-2.23.so             [.] __GI_____strtoll_l_internal
     0.00%     0.00%  systemd-journal  libc-2.23.so             [.] _IO_str_underflow
     0.00%     0.00%  systemd-journal  [unknown]                [.] 0x257568253a6e2575
     0.00%     0.00%  systemd-journal  libc-2.23.so             [.] _IO_vsscanf
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021d35
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000041ad0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000060b323
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000060a5d2
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000060aa62
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000005fdc9e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000005fe016
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000a60f3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000a6178
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0xfffffe00001ea025
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] list_sort.part.0
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] list_sort
     0.00%     0.00%  kubelet          ld-2.23.so               [.] do_lookup_x
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] __GI___sigsetjmp
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] free_unref_page
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] free_pgd_range
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] tlb_remove_table
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] free_page_and_swap_cache
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __put_page
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000001185d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021d87
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000262df
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000032059
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000026687
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000268b7
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __init_rwsem
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_root_lookup
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_lookup
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_lookup_de
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_get_inode
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] new_inode_pseudo
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] alloc_inode
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] inode_init_always
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000602e1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003d6354
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000090578
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000007ba0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000002201a
     0.00%     0.00%  zabbix_agentd    zabbix_agentd            [.] 0x0000000000012dca
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000602dc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000626e5c
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] security_file_open
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vfs_open
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] do_dentry_open
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000004c5a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000a2e5dc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000459c68
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000010d03
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_decide_disposition
     0.00%     0.00%  (kubelet)        libc-2.23.so             [.] strlen
     0.00%     0.00%  (kubelet)        [unknown]                [.] 0x69642d6663752e00
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] pick_next_task_fair
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] sys_nanosleep
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] hrtimer_nanosleep
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] do_nanosleep
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] tick_sched_do_timer
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000bfd9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000019668
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000036ad9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000022ecc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000377c7
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000375f5
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] copy_user_enhanced_fast_string
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] sys_newfstat
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] SYSC_newfstat
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] cp_new_stat
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00007fc6e7eadcb0
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] dequeue_task_fair
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] dequeue_entity
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb719bd0
     0.00%     0.00%  ib_send_bw       [unknown]                [k] 0x6c61636f6c5f6d62
     0.00%     0.00%  ib_send_bw       libc-2.23.so             [.] __memcpy_sse2
     0.00%     0.00%  ib_send_bw       [unknown]                [.] 0x3a096469000a7365
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] iov_iter_fault_in_readable
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.closechan
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x00000000000312ab
     0.00%     0.00%  systemd-logind   libc-2.23.so             [.] memchr
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] sock_read_iter
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] sock_recvmsg
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] unix_stream_recvmsg
     0.00%     0.00%  systemd          systemd                  [.] 0x00000000000c94d7
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] __d_alloc
     0.00%     0.00%  (rm)             libc-2.23.so             [.] __getdents64
     0.00%     0.00%  (rm)             libc-2.23.so             [.] __opendir
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] sys_getdents
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] iterate_dir
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] sys_open
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] do_sys_open
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] proc_readfd
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] do_filp_open
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] proc_readfd_common
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] path_openat
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] proc_fill_cache
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] d_alloc_parallel
     0.00%     0.00%  (rm)             [kernel.kallsyms]        [k] d_alloc
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vfs_statx_fd
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] next_zone
     0.00%     0.00%  kubelet          kubelet                  [.] crosscall_amd64
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000033e9e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000033f5d
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] cpu_load_update_active
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] swapgs_restore_regs_and_return_to_usermode
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __wake_up_sync_key
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __wake_up_common_lock
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __wake_up_common
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] ep_poll_callback
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __wake_up_locked
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] default_wake_function
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] select_task_rq_fair
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] copy_page
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000002c1e6
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x000000000000726f
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_os_write_port
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ev_asynch_enable_gpe
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ev_finish_gpe
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_hw_low_set_gpe
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_hw_write
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_hw_write_port
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/github.com/golang/protobuf/proto.(*InternalMessageInfo).Size
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).SendMsg
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).sendMsg
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.encode
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/encoding/proto.(*codec).Marshal
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/encoding/proto.codec.Marshal
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/encoding/proto.marshal
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/github.com/golang/protobuf/proto.(*Buffer).Marshal
     0.00%     0.00%  kworker/dying    [kernel.kallsyms]        [k] kfree
     0.00%     0.00%  kworker/dying    [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/dying    [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/dying    [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] security_mmap_file
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000032047
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] hrtimer_cancel
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024e9e1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024dbd4
     0.00%     0.00%  dockerd          [unknown]                [k] 0x5024448b482575c0
     0.00%     0.00%  dockerd          [unknown]                [k] 0x000000c42079aa80
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] entry_SYSCALL_64_after_hwframe
     0.00%     0.00%  modprobe         libc-2.23.so             [.] __GI___libc_open
     0.00%     0.00%  systemd-journal  systemd-journald         [.] 0x000000000002c620
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] __strcmp_sse2_unaligned
     0.00%     0.00%  systemd-journal  libc-2.23.so             [.] __xstat64
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] sys_newstat
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] SYSC_newstat
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] vfs_statx
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] futex_wait
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] futex_wait_queue_me
     0.00%     0.00%  kubelet          ld-2.23.so               [.] mprotect
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_mprotect
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] do_mprotect_pkey
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb722380
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] __pagevec_lru_add_fn
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] ext4_da_write_begin
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] grab_cache_page_write_begin
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] pagecache_get_page
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] add_to_page_cache_lru
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] lru_cache_add
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] __lru_cache_add
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] pagevec_lru_move_fn
     0.00%     0.00%  zabbix_agentd    libc-2.23.so             [.] atoi
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] __memcpy_avx_unaligned
     0.00%     0.00%  (ifconfig)       libc-2.23.so             [.] __GI___setsid
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] sys_setsid
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] sched_autogroup_create_attach
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] sched_create_group
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] alloc_fair_sched_group
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] kmem_cache_alloc_node_trace
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] read_tsc
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] cfq_kick_queue
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] __blk_run_queue
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] scsi_request_fn
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] blk_peek_request
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] cfq_dispatch_requests
     0.00%     0.00%  kworker/19:1H    [kernel.kallsyms]        [k] ktime_get
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sd_setup_read_write_cmnd
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] scsi_setup_cmnd
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sd_init_command
     0.00%     0.00%  snapd            snapd                    [.] 0x00000000006165a7
     0.00%     0.00%  snapd            [unknown]                [.] 0xd024448d48098b48
     0.00%     0.00%  snapd            [unknown]                [.] 0x00000000000fd750
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] page_remove_rmap
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] munmap
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_munmap
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] vm_munmap
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] try_to_del_timer_sync
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] mem_cgroup_css_free
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] wb_domain_exit
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] del_timer_sync
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000016e38
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x00000064692f696d
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] lockref_put_or_lock
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] css_release_work_fn
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] cpu_cgroup_css_released
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] sched_offline_group
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] unregister_fair_sched_group
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] find_next_bit
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).ForEach
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x00007fb111e2e671
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] seq_put_decimal_ull
     0.00%     0.00%  kworker/9:1H     [kernel.kallsyms]        [k] pick_next_task_fair
     0.00%     0.00%  kworker/9:1H     [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/9:1H     [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/9:1H     [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/9:1H     [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  kworker/9:1H     [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000001cd21
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000001c447
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008d3ad0
     0.00%     0.00%  snapd            snapd                    [.] 0x000000000081a7b6
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] prefetch_freepointer
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000360f7
     0.00%     0.00%  systemd          [unknown]                [k] 0x00005626e24d23f0
     0.00%     0.00%  containerd       containerd               [.] runtime.retake
     0.00%     0.00%  containerd       [unknown]                [.] 0xb60fc031000001b9
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] mutex_spin_on_owner
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] mutex_lock
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] __mutex_lock_slowpath
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] __mutex_lock.isra.2
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ns_search_one_scope
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ns_search_and_enter
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] arp_rcv
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] arp_process
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] init_page_buffers
     0.00%     0.00%  modprobe         libc-2.23.so             [.] __fxstatat64
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] sys_newfstatat
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] SYSC_newfstatat
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] ext4_iget_normal
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] ext4_iget
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] __ext4_get_inode_loc
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] __breadahead
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] __getblk_gfp
     0.00%     0.00%  containerd       containerd               [.] runtime.notetsleep_internal
     0.00%     0.00%  systemd-journal  systemd-journald         [.] 0x000000000003241e
     0.00%     0.00%  containerd       [unknown]                [.] 0x484d75c08548008b
     0.00%     0.00%  containerd       [unknown]                [.] 0x0000562739c097c0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000d7f8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000677d28
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000006768be
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000671586
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000015776
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000edce
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000001cdcc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000001ea63
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] dyntick_save_progress_counter
     0.00%     0.00%  modprobe         ld-2.23.so               [.] check_match
     0.00%     0.00%  modprobe         [unknown]                [.] 0x42494c4700342e33
     0.00%     0.00%  modprobe         [unknown]                [.] 0x0000000009691974
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] fsnotify_grab_connector
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000157e6
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000005285ac
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000005063f7
     0.00%     0.00%  systemd-cgroups  [unknown]                [.] 0x4700656572660036
     0.00%     0.00%  containerd       containerd               [.] runtime.newobject
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] vsnprintf
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] seq_printf
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] seq_vprintf
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] up_write
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] __ctype_b_loc
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] ktime_get
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] __elv_add_request
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] cfq_insert_request
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] __cfq_update_io_thinktime
     0.00%     0.00%  systemd-cgroups  [unknown]                [.] 0x6e696c2d646c0036
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] cmd_work_handler
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] trailing_symlink
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000016e30
     0.00%     0.00%  containerd       containerd               [.] runtime.acquirep
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000215eb
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000002bfa6
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x00007fb11217004d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000578445
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004ee279
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] __statfs
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __blk_run_queue
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_request_fn
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_pident_instantiate
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] proc_pident_lookup
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.runqsteal
     0.00%     0.00%  containerd       containerd               [.] runtime.(*fixalloc).alloc
     0.00%     0.00%  containerd       containerd               [.] runtime.makechan
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mcache).nextFree.func1
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mcache).refill
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mcentral).cacheSpan
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mcentral).grow
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mheap).alloc
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mheap).alloc.func1
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mheap).alloc_m
     0.00%     0.00%  containerd       containerd               [.] runtime.(*mheap).allocSpanLocked
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002daef8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000254540
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024fce0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024af28
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] try_to_wake_up
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] wake_up_process
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] copy_user_generic_unrolled
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __mod_node_page_state
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] __register_atfork
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] page_add_new_anon_rmap
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] pick_next_task_idle
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000045a0ec
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000684c41
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000006849c2
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000681b6b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000068019d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021b9e
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000000ed16
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x0000000000003950
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] cpumask_next_and
     0.00%     0.00%  containerd       containerd               [.] runtime.newdefer
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*bufWriter).Flush
     0.00%     0.00%  containerd       containerd               [.] net.(*conn).Write
     0.00%     0.00%  containerd       containerd               [.] net.(*netFD).Write
     0.00%     0.00%  containerd       containerd               [.] internal/poll.(*FD).Write
     0.00%     0.00%  containerd       containerd               [.] runtime.deferproc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002fb0ff
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_init_request_from_bio
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005d180
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003eb99
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000002d47f
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000002d6bd
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000059e1c
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000059a5b
     0.00%     0.00%  systemd-logind   systemd-logind           [.] 0x000000000000daaf
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] syscall_trace_enter
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003205e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000031f34
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000002d32
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000003013
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] down_read_trylock
     0.00%     0.00%  modprobe         libc-2.23.so             [.] __GI___uname
     0.00%     0.00%  kubelet          libc-2.23.so             [.] new_heap
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __anon_vma_prepare
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000004c447
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005d199
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000038b8e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005c6ef
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000038ffe
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000038af0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005c671
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000470ed
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000046cf4
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000046a0b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000983d68
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] signalfd
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000000efcd
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] pci_conf1_read
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_pci_config_space_handler
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_os_read_pci_configuration
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] raw_pci_read
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000040f7c4
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008920ea
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000088f2fe
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] resched_curr
     0.00%     0.00%  containerd       [unknown]                [k] 0x0000000000000020
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] ttwu_do_wakeup
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] check_preempt_curr
     0.00%     0.00%  systemd-journal  systemd-journald         [.] 0x0000000000029996
     0.00%     0.00%  systemd-journal  [unknown]                [.] 0x000000d91b129375
     0.00%     0.00%  containerd       [unknown]                [.] 0x00020b5861b35b70
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] unix_stream_read_generic
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] consume_skb
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] skb_release_all
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] skb_release_head_state
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] unix_destruct_scm
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] sock_wfree
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] unix_write_space
     0.00%     0.00%  systemd-udevd    libpthread-2.23.so       [.] __GI___libc_close
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000061263
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000891e7f
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00007fc600000000
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] mem_cgroup_try_charge
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000001c8f5
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000001c321
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000001a846
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000005062d
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000002ccd6
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000001b82b
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000004a23e
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000005077d
     0.00%     0.00%  containerd-shim  containerd-shim          [.] 0x000000000001bdab
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] sys_semop
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] radix_tree_next_chunk
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021d21
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000447361
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __legitimize_mnt
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000eeec
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000030a7ee
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000307503
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000306e01
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] kmem_cache_free
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_delete_object_desc
     0.00%     0.00%  dockerd          dockerd-ce               [.] internal/poll.runtime_pollWait
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.newServerReporter
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002daed5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000242ac1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000ee15d
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] lru_cache_add_active_or_unevictable
     0.00%     0.00%  dbus-daemon      libc-2.23.so             [.] __libc_calloc
     0.00%     0.00%  dbus-daemon      libpthread-2.23.so       [.] pthread_mutex_unlock
     0.00%     0.00%  kubelet          [unknown]                [.] 0x4800034d0ae8c328
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00000000071d7324
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000484d3
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http/httpguts.ValidHeaderFieldValue
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] dput.part.22
     0.00%     0.00%  (kubelet)        [unknown]                [k] 0x0000000000000001
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] sys_getdents
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] iterate_dir
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] proc_readfd
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] proc_readfd_common
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] proc_fill_cache
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] dput
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] perf_event_pid_type
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] perf_iterate_sb
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] ext4_find_entry
     0.00%     0.00%  modprobe         [unknown]                [k] 0x00642e65626f7270
     0.00%     0.00%  modprobe         libc-2.23.so             [.] __xstat64
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] sys_newstat
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] SYSC_newstat
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] delayed_put_task_struct
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] __put_task_struct
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __fget_light
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_create_operands
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000002bec0
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] intel_pstate_update_util
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] deactivate_task
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] dequeue_task_fair
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] dequeue_entity
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] update_load_avg
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_start_plug
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008eac50
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000157c1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008450e1
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] cpumask_next
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.WithDeadline.func3
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.(*timerCtx).cancel
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vma_interval_tree_insert_after
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000057b02a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000053a5a1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003f9915
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005ee73
     0.00%     0.00%  systemd          systemd                  [.] 0x00000000000c95c4
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] __GI___basename
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x841f0f2e66001f0f
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] address_space_init_once
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] proc_alloc_inode
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] kmem_cache_alloc
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] init_once
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] inode_init_once
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_os_signal_semaphore
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ns_get_node
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_release_mutex
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000031cb0
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rb_erase_cached
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_status_exit
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] schedule_timeout
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] wait_for_completion_timeout
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000004c434
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021cf3
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x696c61766e69004d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000f219d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000386156
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003864e9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000385da7
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000385b1d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000385b8b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000f23bb
     0.00%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] _raw_spin_lock
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sock_poll
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] epoll_ctl
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_epoll_ctl
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ep_item_poll.isra.10
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000015621
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000b95185
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000a4c6c1
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] dequeue_entity
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] deactivate_task
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] dequeue_task_fair
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00007fc640ff8bf0
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000000000000000c
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000028bfc
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] cfq_activate_request
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000269082
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002560df
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000255b7e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000254b3d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000254e71
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] _cond_resched
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000009ea4f0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000b22830
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000084e000
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021d09
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] trailing_symlink
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000032068
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000004c26b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000046206
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000091f711
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000ad1240
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008d3b3b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008d18be
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008aaa78
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008aa5b4
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __x86_indirect_thunk_r12
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000032021
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000a31040
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] __radix_tree_lookup
     0.00%     0.00%  snapd            snapd                    [.] 0x000000000065e3ca
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] do_syscall_64
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] sys_openat
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] do_sys_open
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] do_filp_open
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] path_openat
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] link_path_walk
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] walk_component
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] lookup_slow
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] ext4_lookup
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] ext4_find_entry
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] ext4_bread_batch
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] ll_rw_block
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] submit_bh_wbc
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] submit_bio
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] generic_make_request
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] generic_make_request_checks
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] blkg_lookup_slowpath
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] radix_tree_lookup
     0.00%     0.00%  snapd            snapd                    [.] 0x0000000000617480
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002ddb85
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000028a7ad
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000006f734
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000006f6b9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000006fd02
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024c6e1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003c0849
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000251f68
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000252115
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000252a22
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024f1b4
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024ec59
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000024fc26
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000009a9dd0
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] delete_node
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000aea660
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] event_function
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000b520a8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000b951bc
     0.00%     0.00%  containerd       containerd               [.] reflect.flag.mustBe
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] init_cpu_features.constprop.0
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] deactivate_task
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] dequeue_task_fair
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] dequeue_entity
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000036165
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002dae30
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000a0e360
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000eb0b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004f89e2
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000d535
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000245a1
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_cblist_dequeue
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] common_file_perm
     0.00%     0.00%  containerd       [unknown]                [k] 0x000000c420646000
     0.00%     0.00%  containerd       [unknown]                [k] 0x0000562700000000
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] rw_verify_area
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] security_file_permission
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016cec4
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000677e20
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000603683
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000060434e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016abdd
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016a461
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_method_data_init_args
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] timer_clear_idle
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.(*mSpanList).remove
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*loopyWriter).handle
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*loopyWriter).headerHandler
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000009891a0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000bfea
     0.00%     0.00%  dockerd          [vdso]                   [.] __vdso_clock_gettime
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] 0x0000000000215490
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] task_work_add
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_close
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __close_fd
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] filp_close
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] fput
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] run_local_timers
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003c5d1c
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000038607d
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] pick_link
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000040d3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000008d99cc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000010de3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003100f4
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000030b1f8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000030ee4b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000030f04b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016a4a9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016a343
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016a2aa
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016a1e8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000169ac5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000001579f
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000b951d5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000b8190a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000b4c1c
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000677bca
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000656fd6
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] _raw_spin_unlock_irqrestore
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000004c44f
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] copy_user_enhanced_fast_string
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] cp_new_stat
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] swiotlb_unmap_sg_attrs
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] filemap_map_pages
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __getblk_gfp
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] pagecache_get_page
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000684b90
     0.00%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] update_cfs_rq_h_load
     0.00%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] select_task_rq_fair
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] error_entry
     0.00%     0.00%  containerd       containerd               [.] runtime.mapaccess2_fast32
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).preprocessData
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x00000000000312dd
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb719f30
     0.00%     0.00%  containerd       containerd               [.] unicode.IsSpace
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/filters.ParseAll
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/filters.Parse
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/filters.(*parser).parse
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/filters.(*parser).selectors
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/filters.(*parser).selector
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/filters.(*parser).fieldpath
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/filters.(*parser).field
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __note_gp_changes
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021484
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [.] native_irq_return_iret
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000085221b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000006c6b56
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003f99ea
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5e_update_q_counter
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5_core_query_q_counter
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002ddb7b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002ab3a0
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] release_pages
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] sys_exit_group
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] do_group_exit
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] do_exit
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] mmput
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] exit_mmap
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] tlb_finish_mmu
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] arch_tlb_finish_mmu
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] tlb_flush_mmu_free
     0.00%     0.00%  sleep            [kernel.kallsyms]        [k] free_pages_and_swap_cache
     0.00%     0.00%  systemd          systemd                  [.] 0x00000000000589d6
     0.00%     0.00%  systemd          [unknown]                [.] 0xfff877f8fff87808
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002fc6a1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000263342
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000026908c
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] try_charge
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000c8020
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] mutex_lock
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] intel_pstate_update_util
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_create_generic_state
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] ixgbe_ptp_rx_hang
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003b960
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] hrtimer_nanosleep
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] do_nanosleep
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_relocate_object
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] call_usermodehelper_exec_async
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] do_execve
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] do_execveat_common.isra.35
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] search_binary_handler
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] load_elf_binary
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] clear_user
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] __clear_user
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000001fe49
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021b20
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000022148
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000002432f
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000016192
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004f9bdc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004f9757
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000124c3
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] net_tx_action
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] alloc_pages_vma
     0.00%     0.00%  modprobe         [unknown]                [k] 0x2d0065626f727064
     0.00%     0.00%  modprobe         [unknown]                [k] 0x00007ffd071b3fa3
     0.00%     0.00%  modprobe         libc-2.23.so             [.] _init
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] down_read_trylock
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_start_user
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_start
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] radix_tree_maybe_preload
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000326280
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] smp_call_function_many
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] kmemcg_deactivate_workfn
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] kmemcg_cache_deact_after_rcu
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] __kmem_cache_shrink
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] on_each_cpu_cond
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] on_each_cpu_mask
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000184e3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003a63c
     0.00%     0.00%  kubelet          [unknown]                [k] 0x00020b5a14f9c4f0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000367c3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000035451
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000005d9200
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] syscall_return_via_sysret
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __queue_work
     0.00%     0.00%  kubelet          [unknown]                [k] 0x00020b5a00000014
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] get_pid_task
     0.00%     0.00%  iscsid           [kernel.kallsyms]        [k] syscall_return_via_sysret
     0.00%     0.00%  iscsid           [unknown]                [k] 0000000000000000
     0.00%     0.00%  iscsid           libc-2.23.so             [.] __poll_nocancel
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.selparkcommit
     0.00%     0.00%  dockerd          [unknown]                [.] 0x000000c420c5ad80
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000677fa0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000006779b1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000060b976
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000007bce
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] end_bio_bh_io_sync
     0.00%     0.00%  systemd          [unknown]                [.] 0x0000000000000741
     0.00%     0.00%  systemd          systemd                  [.] 0x00000000000c8090
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000003e83e
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).Bucket
     0.00%     0.00%  containerd       containerd               [.] runtime.findfunc
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] aa_file_perm
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/metadata.(*containerStore).List.func1
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/metadata.readContainer
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/metadata.readContainer.func1
     0.00%     0.00%  systemd-journal  [unknown]                [k] 0x00000000ffffffff
     0.00%     0.00%  systemd-journal  systemd-journald         [.] 0x0000000000009cae
     0.00%     0.00%  systemd-journal  libpthread-2.23.so       [.] __read_nocancel
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] rw_verify_area
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] security_file_permission
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] apparmor_file_permission
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] common_file_perm
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021b4e
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] find_busiest_group
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] __lock_page_or_retry
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] io_schedule
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] pick_next_task_fair
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] load_balance
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_advance_cbs
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_accelerate_cbs
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] alloc_set_pte
     0.00%     0.00%  (ifconfig)       [kernel.kallsyms]        [k] inode_init_always
     0.00%     0.00%  (ifconfig)       libc-2.23.so             [.] __strncpy_sse2_unaligned
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] ext4_get_group_desc
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] core_sys_select
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] do_select
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] poll_schedule_timeout
     0.00%     0.00%  dbus-daemon      libc-2.23.so             [.] __strcmp_sse2_unaligned
     0.00%     0.00%  dbus-daemon      [unknown]                [.] 0x00005603374a1490
     0.00%     0.00%  irqbalance       [kernel.kallsyms]        [k] string
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] kmem_cache_alloc
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] get_request
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] ioc_create_icq
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] __radix_tree_insert
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] __radix_tree_create
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] radix_tree_node_alloc.constprop.19
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000168fe6
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003ae0c7
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003a7ea3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003a2265
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003a576e
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] kjournald2
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] jbd2_journal_commit_transaction
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] update_rq_clock
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __fget
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00007fc6e76acd70
     0.00%     0.00%  dockerd          [unknown]                [k] 0x000000c42102b6e0
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __fdget
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003ae08d
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_add_reference
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] memcg_kmem_get_cache
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] memcg_drain_all_list_lrus
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] mem_cgroup_css_offline
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] memcg_offline_kmem.part.45
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] irq_chip_ack_parent
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ll_rw_block
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] submit_bh_wbc
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] submit_bio
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] generic_make_request
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] blk_queue_bio
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] add_disk_randomness
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] add_timer_randomness
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] mix_pool_bytes
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] _raw_spin_unlock_irqrestore
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] handle_irq
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.jmpdefer
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*bufWriter).Flush
     0.00%     0.00%  dockerd          dockerd-ce               [.] net.(*conn).Write
     0.00%     0.00%  dockerd          dockerd-ce               [.] net.(*netFD).Write
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_delete_parse_tree
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_complete_this_op
     0.00%     0.00%  kubelet          libpthread-2.23.so       [.] create_thread
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00007f14ac0008f0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000400000
     0.00%     0.00%  kubelet          kubelet                  [.] _cgo_try_pthread_create
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb719f90
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] update_load_avg
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] put_dec_trunc8
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] seq_put_decimal_ull
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] num_to_str
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] put_dec
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000d83b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000001969b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000677b7e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000045a174
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000b49d8
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sync_regs
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] time
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] queue_work_on
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000157df
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000351ac0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000033fca2
     0.00%     0.00%  containerd       containerd               [.] runtime.runqsteal
     0.00%     0.00%  containerd       containerd               [.] time.Now
     0.00%     0.00%  containerd       [unknown]                [.] 0x001258888b480084
     0.00%     0.00%  containerd       [unknown]                [.] 0x0000557021d6b590
     0.00%     0.00%  containerd       containerd               [.] runtime.gcBgMarkWorker
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] do_mmap
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] mmap_region
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000e153a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000c807a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000ada14
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] proc_i_callback
     0.00%     0.00%  containerd       containerd               [.] runtime.sysmon
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004f89a1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021d41
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] blkg_destroy
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] cfq_pd_offline
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] blkg_rwstat_add_aux
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] blkg_rwstat_read
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] __percpu_counter_sum
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb750d70
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_ev_fixed_event_detect
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_hw_register_read
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_hw_read_multiple
     0.00%     0.00%  (kubelet)        [unknown]                [k] 0x00007f375eae58e9
     0.00%     0.00%  (kubelet)        libc-2.23.so             [.] _int_malloc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002daf2a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000026ed63
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000026e8f7
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] vfs_open
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] do_dentry_open
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000474340
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_next_parse_state
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000320a5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003ec91
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x00007fb111e2c067
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb795740
     0.00%     0.00%  systemd          [kernel.kallsyms]        [k] anon_vma_fork
     0.00%     0.00%  systemd          [kernel.kallsyms]        [k] anon_vma_clone
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] pick_next_task_fair
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] update_blocked_averages
     0.00%     0.00%  systemd          systemd                  [.] 0x00000000000cc367
     0.00%     0.00%  systemd          [unknown]                [.] 0x00005626e247e850
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] select_idle_sibling
     0.00%     0.00%  in:imuxsock      [unknown]                [k] 0x0000000002097ae5
     0.00%     0.00%  in:imuxsock      libpthread-2.23.so       [.] pthread_cond_signal@@GLIBC_2.3.2
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] sys_futex
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] do_futex
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] futex_wake_op
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] wake_up_q
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] try_to_wake_up
     0.00%     0.00%  in:imuxsock      [kernel.kallsyms]        [k] select_task_rq_fair
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000004637d
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] _find_next_bit
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.goready.func1
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.ready
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000005f1190
     0.00%     0.00%  dockerd          dockerd-ce               [.] sync.(*Mutex).Lock
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] swapgs_restore_regs_and_return_to_usermode
     0.00%     0.00%  modprobe         ld-2.23.so               [.] _dl_debug_initialize
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).newStream
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] file_free_rcu
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] native_sched_clock
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] queue_unplugged
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_queue_start_tag
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_start_request
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sched_clock
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000465eb
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031dbdd
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031d8a3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031d5d4
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000323868
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003236f2
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031c825
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] native_read_msr
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000039c2dc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003bf7d3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003a09ac
     0.00%     0.00%  kubelet          [unknown]                [k] 0xfffffe000005e01b
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] lock_hrtimer_base.isra.20
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_alloc_op
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003bca0c
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000bd47
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000578b6
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] scsi_release_buffers
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] sg_free_table_chained
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __sg_free_table
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] sg_pool_free
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] nr_iowait_cpu
     0.00%     0.00%  zabbix_agentd    libc-2.23.so             [.] __xstat64
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] sys_newstat
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] SYSC_newstat
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] vfs_statx
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] cpuacct_charge
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] cursor_timer_handler
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000eedb
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000002dbdd7
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_terminate_control_method
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_stop_trace_method
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000016dac
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] MR_TargetIdToLdGet
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x0000000000000021
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_opcode_1A_1T_1R
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_store
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_store_object_to_node
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_write_data_to_field
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_insert_into_field
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ex_write_with_update_rule
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] update_load_avg
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] ttwu_do_activate
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] activate_task
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] enqueue_task_fair
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] enqueue_entity
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000001262e
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004bf759
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000b7449
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000007469d
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __init_rwsem
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] anon_vma_ctor
     0.00%     0.00%  kubelet          [unknown]                [k] 0x000000000045f3df
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000034958
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005f3cc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005f3df
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021543
     0.00%     0.00%  containerd       containerd               [.] unicode/utf8.ValidString
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __fget
     0.00%     0.00%  containerd       containerd               [.] runtime.adjustpointers
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*serverReporter).Handled
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __fdget_pos
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/prometheus/client_golang/prometheus.(*metricVec).hashLabelValues
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __fget_light
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/github.com/prometheus/client_golang/prometheus.validateLabelValues
     0.00%     0.00%  containerd       containerd               [.] runtime.adjustframe
     0.00%     0.00%  modprobe         [unknown]                [k] 0x00007f7e44c5f5a8
     0.00%     0.00%  modprobe         [unknown]                [k] 0x0000000000001000
     0.00%     0.00%  modprobe         ld-2.23.so               [.] mmap64
     0.00%     0.00%  modprobe         ld-2.23.so               [.] _dl_allocate_tls_storage
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003a645
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] tcp_keepalive_timer
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_exit
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sg_next
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000003b4c5
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __virt_addr_valid
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __vfs_write
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] sock_write_iter
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] sock_sendmsg
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] unix_stream_sendmsg
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] skb_copy_datagram_from_iter
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __check_object_size
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] kmem_cache_alloc_node
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] mempool_alloc
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ext4_readpages
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ext4_mpage_readpages
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] bio_alloc_bioset
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sysfs_kf_seq_show
     0.00%     0.00%  kubelet          libc-2.23.so             [.] arena_get2.part.3
     0.00%     0.00%  kubelet          libc-2.23.so             [.] next_line
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] kernfs_fop_read
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] kernfs_seq_show
     0.00%     0.00%  containerd       [unknown]                [k] 0x058b000003d98f0f
     0.00%     0.00%  containerd       [unknown]                [k] 0x00020b5800000014
     0.00%     0.00%  containerd       containerd               [.] runtime.usleep
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] sys_pselect6
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] memset_erms
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00020b5700000014
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] getname_flags
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] getname
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000602fa0
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] dm_make_request
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __split_and_process_bio
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] mempool_alloc
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] mempool_alloc_slab
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000036d400
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000017f53
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003bf954
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031db7d
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000315401
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __radix_tree_lookup
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] 0x00000000001766e0
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] __xstat64
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] sys_newstat
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] SYSC_newstat
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ext4_lookup
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ext4_find_entry
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ext4_bread_batch
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] ext4_getblk
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __getblk_gfp
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __find_get_block
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] pagecache_get_page
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] find_get_entry
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] radix_tree_lookup_slot
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] alloc_set_pte
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] _dl_vdso_vsym
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_lookup_symbol_x
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001d3f30
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] del_timer_sync
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] try_to_del_timer_sync
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] lock_timer_base
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] cpumask_any_but
     0.00%     0.00%  modprobe         libc-2.23.so             [.] __GI___munmap
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] sys_munmap
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] vm_munmap
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] do_munmap
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] unmap_region
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] tlb_flush_mmu_tlbonly
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] flush_tlb_mm_range
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] raise_softirq
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.(*cancelCtx).cancel
     0.00%     0.00%  ifconfig         [kernel.kallsyms]        [k] __rb_erase_color
     0.00%     0.00%  ifconfig         [kernel.kallsyms]        [k] mmput
     0.00%     0.00%  ifconfig         [kernel.kallsyms]        [k] exit_mmap
     0.00%     0.00%  ifconfig         [kernel.kallsyms]        [k] free_pgtables
     0.00%     0.00%  ifconfig         [kernel.kallsyms]        [k] unlink_file_vma
     0.00%     0.00%  ifconfig         [kernel.kallsyms]        [k] __remove_shared_vm_struct
     0.00%     0.00%  ifconfig         [kernel.kallsyms]        [k] vma_interval_tree_remove
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] timerqueue_del
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] smp_call_function_many
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000001ceb0
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] perf_sched_delayed
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] static_key_disable
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] static_key_disable_cpuslocked
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] jump_label_update
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] __jump_label_update
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] arch_jump_label_transform
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] __jump_label_transform.isra.0
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] text_poke_bp
     0.00%     0.00%  kworker/11:2     [kernel.kallsyms]        [k] on_each_cpu
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] update_curr
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003b5700
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] cap_capable
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] __GI___setsockopt
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sys_setsockopt
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] sock_setsockopt
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] capable
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ns_capable_common
     0.00%     0.00%  modprobe         [unknown]                [k] 0xfffffe000005e01b
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] megasas_cmd_type
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] __blk_run_queue
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] scsi_request_fn
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] scsi_dispatch_cmd
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] megasas_queue_command
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] check_match
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __blk_run_queue
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] scsi_request_fn
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] _raw_spin_lock_irq
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] do_fcntl
     0.00%     0.00%  modprobe         libc-2.23.so             [.] __GI___libc_fcntl
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] put_dec_trunc8
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] num_to_str
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000613b5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000e2fc1
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000002d4a0fa
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000e321d
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_map_object_deps
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004b83d0
     0.00%     0.00%  containerd       containerd               [.] runtime.reentersyscall
     0.00%     0.00%  containerd       containerd               [.] runtime.entersyscall
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001fa4a0
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] perf_event_task_tick
     0.00%     0.00%  containerd       [unknown]                [k] 0x00007f8db0ff8ef0
     0.00%     0.00%  containerd       [unknown]                [k] 0x000000c4208e8960
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] finish_task_switch
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] apic_timer_interrupt
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] smp_apic_timer_interrupt
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] hrtimer_interrupt
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __hrtimer_run_queues
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] tick_sched_timer
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] tick_sched_handle
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] update_process_times
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.(*waitq).dequeue
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000230d80
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000e3a57
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000046106
     0.00%     0.00%  irqbalance       [kernel.kallsyms]        [k] kmem_cache_alloc_trace
     0.00%     0.00%  irqbalance       [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.00%     0.00%  irqbalance       [kernel.kallsyms]        [k] do_syscall_64
     0.00%     0.00%  irqbalance       [kernel.kallsyms]        [k] vfs_open
     0.00%     0.00%  irqbalance       [kernel.kallsyms]        [k] do_dentry_open
     0.00%     0.00%  irqbalance       irqbalance               [.] 0x00000000000060ff
     0.00%     0.00%  irqbalance       [kernel.kallsyms]        [k] proc_reg_open
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] vmacache_find
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_create_operand
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_create_internal_object_dbg
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_allocate_object_desc_dbg
     0.00%     0.00%  kubelet          [unknown]                [k] 0x000000000000000d
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_clear_implicit_return
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] _raw_spin_lock_irq
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003bccfb
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000bd2d
     0.00%     0.00%  dockerd          dockerd-ce               [.] time.After
     0.00%     0.00%  dockerd          dockerd-ce               [.] time.NewTimer
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] update_ts_time_stats
     0.00%     0.00%  gmain            libgio-2.0.so.0.4800.2   [.] 0x0000000000103eef
     0.00%     0.00%  dockerd          [unknown]                [k] 0x480000557d653e91
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00007fc6a97f9d10
     0.00%     0.00%  dockerd          [unknown]                [.] 0x244c8908244c8b18
     0.00%     0.00%  dockerd          [unknown]                [.] 0x0000557d681f5e00
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.walltime
     0.00%     0.00%  gmain            [unknown]                [.] 0x737561727478652f
     0.00%     0.00%  gmain            [unknown]                [.] 0x0000000000a07550
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/github.com/golang/protobuf/proto.(*marshalInfo).size
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/health/grpc_health_v1.(*HealthCheckRequest).XXX_Size
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] cfq_find_next_rq
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] cfq_dispatch_insert
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] _find_next_bit
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] swapgs_restore_regs_and_return_to_usermode
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021a90
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] idle_cpu
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] select_idle_sibling
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] reweight_entity
     0.00%     0.00%  containerd       containerd               [.] runtime.readvarint
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] update_cfs_group
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_get_next_simple_arg
     0.00%     0.00%  watchdog/12      [kernel.kallsyms]        [k] kthread_should_stop
     0.00%     0.00%  watchdog/12      [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  watchdog/12      [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000384619
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000036dd4f
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000158b7
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000015c91
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000015e7c
     0.00%     0.00%  snapd            snapd                    [.] 0x000000000062cd8d
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] sys_select
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] core_sys_select
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] do_select
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] poll_schedule_timeout
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] schedule_hrtimeout_range
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] schedule_hrtimeout_range_clock
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] refresh_cpu_vm_stats
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] ext4_ext_map_blocks
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000004c437
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000ea02
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004e7f80
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031dc0e
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*MaxRecvMsgSizeCallOption).before
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ixgbe_msix_clean_rings
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000bfc7
     0.00%     0.00%  systemd          [kernel.kallsyms]        [k] filemap_map_pages
     0.00%     0.00%  (kubelet)        [kernel.kallsyms]        [k] swapgs_restore_regs_and_return_to_usermode
     0.00%     0.00%  (kubelet)        libpthread-2.23.so       [.] __sigaction
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001fa4eb
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000016f42a
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000165ea0
     0.00%     0.00%  containerd       containerd               [.] runtime.adjustdefers
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] bio_add_page
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000dfa3d0
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x00000000000312cc
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x247c894800000008
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000036b30
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x000055ccdb773ac0
     0.00%     0.00%  snapd            snapd                    [.] 0x000000000066aed0
     0.00%     0.00%  snapd            [unknown]                [.] 0x0000000000000008
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] finish_task_switch
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] release_pages
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] tlb_flush_mmu_free
     0.00%     0.00%  modprobe         [kernel.kallsyms]        [k] free_pages_and_swap_cache
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] ttwu_do_activate
     0.00%     0.00%  dockerd          [unknown]                [k] 0xfffd7d96e8240489
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb122a6d
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb722070
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x00000000000287e5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000005783c0
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] lookup_undo
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __switch_to
     0.00%     0.00%  systemd-udevd    libpthread-2.23.so       [.] __reclaim_stacks
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb795830
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_needs_cpu
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rb_next
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] _raw_spin_lock_irq
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000038abd3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000b4b56
     0.00%     0.00%  rs:main Q:Reg    libc-2.23.so             [.] strlen
     0.00%     0.00%  modprobe         kmod                     [.] 0x000000000000cd67
     0.00%     0.00%  modprobe         [unknown]                [.] 0x000055c9fd87983f
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/golang.org/x/net/http2/hpack.(*Encoder).WriteField
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).headerHandler
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).writeHeader
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] megasas_cmd_type
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] map_cmd_status
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000260b0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003bf828
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031dba9
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000315630
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003160ee
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001fa52c
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000173ef5
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000015811
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003483d
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ns_opens_scope
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.propagateCancel
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00020b5800000014
     0.00%     0.00%  perf             [unknown]                [k] 0xfffffe000003201b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000bfce
     0.00%     0.00%  dbus-daemon      libdbus-1.so.3.14.6      [.] _dbus_watch_unref
     0.00%     0.00%  dbus-daemon      [unknown]                [.] 0x0000000000000001
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001b60e0
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] add_interrupt_randomness
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003d20b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000005ad60
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003d357
     0.00%     0.00%  dockerd          dockerd-ce               [.] time.(*Timer).Stop
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] copy_user_generic_unrolled
     0.00%     0.00%  dbus-daemon      libdbus-1.so.3.14.6      [.] 0x0000000000028e78
     0.00%     0.00%  dbus-daemon      libpthread-2.23.so       [.] __GI___libc_recvmsg
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] sys_recvmsg
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] __sys_recvmsg
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] ___sys_recvmsg
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] copy_msghdr_from_user
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] import_iovec
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] rw_copy_check_uvector
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] pick_next_task_idle
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ds_exec_begin_op
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] __x86_indirect_thunk_rdx
     0.00%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] sys_imageblit
     0.00%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] bit_cursor
     0.00%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] soft_cursor
     0.00%     0.00%  kworker/u65:1    [kernel.kallsyms]        [k] drm_fb_helper_sys_imageblit
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] unregister_fair_sched_group
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] sched_autogroup_exit
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] sched_offline_group
     0.00%     0.00%  dockerd          [unknown]                [.] 0x0000000000000002
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x00007fb111f1e970
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] cfq_dispatch_requests
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] cfq_kick_queue
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] __blk_run_queue
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] scsi_request_fn
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] blk_peek_request
     0.00%     0.00%  kubelet          [unknown]                [k] 0x00020b5a00002710
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] hrtimer_try_to_cancel
     0.00%     0.00%  perf             libc-2.23.so             [.] __memcpy_sse2
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001c87fb
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001a58c0
     0.00%     0.00%  kworker/1:2      [kernel.kallsyms]        [k] _raw_spin_lock
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003046ea
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000003bf8be
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000031dbb7
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000001596f0
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] add_to_page_cache_lru
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __add_to_page_cache_locked
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000033a52
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003d880
     0.00%     0.00%  kubelet          [unknown]                [.] 0x018706dba7980d8d
     0.00%     0.00%  kubelet          [unknown]                [.] 0x06dd806d0d8d48c0
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000003d46e48
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00000000071d5480
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003ac9a
     0.00%     0.00%  dbus-daemon      [kernel.kallsyms]        [k] update_cfs_rq_h_load
     0.00%     0.00%  zabbix_agentd    zabbix_agentd            [.] 0x0000000000038aa7
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] br_netif_receive_skb
     0.00%     0.00%  zabbix_agentd    libc-2.23.so             [.] __GI___libc_connect
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] sys_connect
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] SYSC_connect
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] inet_stream_connect
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __inet_stream_connect
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] tcp_v4_connect
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] tcp_connect
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] tcp_transmit_skb
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] ip_queue_xmit
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] ip_local_out
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] ip_output
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] ip_finish_output
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] ip_finish_output2
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __local_bh_enable_ip
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] do_softirq.part.17
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] do_softirq_own_stack
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __softirqentry_text_start
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] net_rx_action
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] process_backlog
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __netif_receive_skb
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __netif_receive_skb_core
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] br_handle_frame
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] nf_hook_slow
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] br_nf_pre_routing
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] br_nf_pre_routing_finish
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] br_nf_hook_thresh
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] br_handle_frame_finish
     0.00%     0.00%  dbus-daemon      libdbus-1.so.3.14.6      [.] 0x000000000002b110
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __inc_numa_state
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] wp_page_copy
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] alloc_pages_vma
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __alloc_pages_nodemask
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] get_page_from_freelist
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] blk_rq_map_sg
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] scsi_init_io
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] scsi_init_sgtable
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000f1ac
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000578436
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004db865
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000671dd
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] swiotlb_map_sg_attrs
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] next_online_pgdat
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] update_cfs_rq_h_load
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] call_usermodehelper_exec_work
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] umh_complete
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] complete
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] __wake_up_locked
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] __wake_up_common
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] default_wake_function
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] try_to_wake_up
     0.00%     0.00%  kworker/u65:0    [kernel.kallsyms]        [k] select_task_rq_fair
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] decay_load
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).handleWindowUpdate
     0.00%     0.00%  containerd       containerd               [.] runtime.wakep
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] delayed_work_timer_fn
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] find_next_bit
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] acct_collect
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] _exit
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.netpollblock
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000016b1b
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] cgroup1_pidlist_destroy_all
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] flush_workqueue
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*Stream).RecvCompress
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*Stream).waitOnHeader
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] __bitmap_and
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] pick_next_task_fair
     0.00%     0.00%  kthreadd         [kernel.kallsyms]        [k] load_balance
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] mempool_alloc
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] cfq_kick_queue
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] __blk_run_queue
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] scsi_request_fn
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] blk_peek_request
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] scsi_prep_fn
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] scsi_setup_cmnd
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] sd_init_command
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] sd_setup_read_write_cmnd
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] scsi_init_io
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] scsi_init_sgtable
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] sg_alloc_table_chained
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] __sg_alloc_table
     0.00%     0.00%  kworker/31:1H    [kernel.kallsyms]        [k] sg_pool_alloc
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ps_push_scope
     0.00%     0.00%  kworker/0:1      [kernel.kallsyms]        [k] acpi_ut_create_thread_state
     0.00%     0.00%  ifconfig         libc-2.23.so             [.] buffered_vfprintf
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] mutex_unlock
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] memcg_kmem_cache_create_func
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] memcg_create_kmem_cache
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] create_cache
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] __kmem_cache_create
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] sysfs_slab_add
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] sysfs_create_group
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] internal_create_group
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] sysfs_add_file_mode_ns
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] __kernfs_create_file
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000060da4
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x0000000000037a22
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x44455a494c414954
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000021d2b
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000015a2fc
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000230da3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000015b404
     0.00%     0.00%  systemd-journal  libc-2.23.so             [.] __uflow
     0.00%     0.00%  systemd-journal  [unknown]                [.] 0x3d5954494c494341
     0.00%     0.00%  systemd-journal  [unknown]                [.] 0x00007ffd1210abf1
     0.00%     0.00%  systemd-journal  [unknown]                [.] 0x00007ffd1210a8c0
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] sg_next
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] blk_finish_plug
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] blk_flush_plug_list
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] queue_unplugged
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] __blk_run_queue
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] scsi_request_fn
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] scsi_dispatch_cmd
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] megasas_queue_command
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] megasas_build_and_issue_cmd_fusion
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] megasas_build_io_fusion
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] scsi_dma_map
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] swiotlb_map_sg_attrs
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] end_buffer_read_sync
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __end_buffer_read_notouch
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] unlock_buffer
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] wake_up_bit
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __wake_up_bit
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __wake_up
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __wake_up_common_lock
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] wake_bit_function
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] intel_pstate_update_pstate
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] select_task_rq_fair
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] wake_up_new_task
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] read_tsc
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_allocate_tls_storage
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] blk_finish_request
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __blk_put_request
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] elv_completed_request
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] cfq_completed_request
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ktime_get
     0.00%     0.00%  systemd          [kernel.kallsyms]        [k] security_file_alloc
     0.00%     0.00%  systemd          [unknown]                [k] 0x00005626e24a2140
     0.00%     0.00%  systemd          systemd                  [.] 0x00000000000d7a9e
     0.00%     0.00%  systemd          [unknown]                [.] 0x000000085eaed6c9
     0.00%     0.00%  systemd          [kernel.kallsyms]        [k] get_empty_filp
     0.00%     0.00%  systemd-cgroups  [unknown]                [k] 0x642a2520642a2520
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] _IO_vsscanf
     0.00%     0.00%  systemd-cgroups  libc-2.23.so             [.] _IO_vfscanf
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] resched_curr
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] ttwu_do_wakeup
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] check_preempt_curr
     0.00%     0.00%  kworker/1:1H     [kernel.kallsyms]        [k] update_curr
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/encoding/proto.codec.Unmarshal
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC.func4
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/google.golang.org/grpc/encoding/proto.(*codec).Unmarshal
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003ba6b
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] calloc@plt
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] lru_add_drain
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] lru_add_drain_cpu
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] pagevec_lru_move_fn
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] ttwu_do_wakeup
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000004e7e86
     0.00%     0.00%  kubelet          [unknown]                [k] 0xfffffe000016601b
     0.00%     0.00%  kubelet          libpthread-2.23.so       [.] pthread_sigmask
     0.00%     0.00%  kubelet          [unknown]                [k] 0x000000000040bfb7
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] fold_diff
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] mlx5_cmd_comp_handler
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).readMetaFrame.func1
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] __slab_free
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] cgroup_free
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] put_css_set_locked
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] kfree
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __indirect_thunk_start
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] 0x0000000000216ea8
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] native_set_pte_at
     0.00%     0.00%  systemd-udevd    systemd-udevd            [.] 0x000000000000edca
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] strlen
     0.00%     0.00%  systemd-journal  [unknown]                [k] 0000000000000000
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] cgroup_path_ns_locked
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] kernfs_path_from_node
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] kernfs_path_from_node_locked
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/daemon.(*Daemon).execCommandGC
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] netif_skb_features
     0.00%     0.00%  sshd             [unknown]                [k] 0x00007fb3894dab98
     0.00%     0.00%  sshd             [unknown]                [k] 0x0000557b9e7e4380
     0.00%     0.00%  sshd             libc-2.23.so             [.] __GI___libc_write
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] sys_write
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] vfs_write
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] __vfs_write
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] sock_write_iter
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] sock_sendmsg
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] inet_sendmsg
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] tcp_sendmsg
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] tcp_sendmsg_locked
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] tcp_push
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] __tcp_push_pending_frames
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] tcp_write_xmit
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] tcp_transmit_skb
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] ip_queue_xmit
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] ip_local_out
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] ip_output
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] ip_finish_output
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] ip_finish_output2
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] pipe_poll
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] dev_queue_xmit
     0.00%     0.00%  sshd             libc-2.23.so             [.] __select
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] __dev_queue_xmit
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] sch_direct_xmit
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] sys_select
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] core_sys_select
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] validate_xmit_skb_list
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] do_select
     0.00%     0.00%  sshd             [kernel.kallsyms]        [k] validate_xmit_skb
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] native_flush_tlb_single
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] cfq_dispatch_requests
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] blk_peek_request
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000198a3
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000019f0a
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __delayacct_tsk_init
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] check_preempt_curr
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ttwu_do_wakeup
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] cfq_close_cooperator
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.lock
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ir_ack_apic_edge
     0.00%     0.00%  dockerd          [unknown]                [.] 0x00020b5900000014
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] prepend_path
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] file_path
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] d_path
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] __unsetenv
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] trigger_load_balance
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.runqempty
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] _raw_spin_lock
     0.00%     0.00%  dockerd          [unknown]                [.] 0xe83024448b48c358
     0.00%     0.00%  dockerd          [unknown]                [k] 0x000000c400000000
     0.00%     0.00%  dockerd          [unknown]                [.] 0x0000557d681fa340
     0.00%     0.00%  dockerd          dockerd-ce               [.] time.Sleep
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.goparkunlock
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] native_load_tls
     0.00%     0.00%  dockerd          [unknown]                [.] 0x0000557d681fa4e0
     0.00%     0.00%  dockerd          [unknown]                [k] 0x00007fc6e8eafc90
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] _cond_resched
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] wait_for_completion
     0.00%     0.00%  zabbix_agentd    zabbix_agentd            [.] setproctitle_set_status
     0.00%     0.00%  zabbix_agentd    zabbix_agentd            [.] __zbx_zbx_setproctitle
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] sched_avg_update
     0.00%     0.00%  kworker/u64:2    [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.00%     0.00%  kworker/u64:2    [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/u64:2    [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/u64:2    [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/u64:2    [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/u64:2    [kernel.kallsyms]        [k] cmd_work_handler
     0.00%     0.00%  kworker/u64:2    [kernel.kallsyms]        [k] down
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] rcu_all_qs
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] update_blocked_averages
     0.00%     0.00%  dbus-daemon      libdbus-1.so.3.14.6      [.] _dbus_validate_path
     0.00%     0.00%  dbus-daemon      [unknown]                [.] 0x00007ffe14266460
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] nr_iowait_cpu
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] get_idle_time
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd.(*remoteNamespaces).List
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/namespaces.(*local).List
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/namespaces.(*local).withStoreView
     0.00%     0.00%  systemd-udevd    libc-2.23.so             [.] __strchr_sse2
     0.00%     0.00%  systemd-udevd    [unknown]                [.] 0x70756f7267633d4d
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] switch_mm_irqs_off
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] enter_lazy_tlb
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] switch_mm
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] vfs_getattr
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] futex_wait_setup
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000022ebf
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000031353
     0.00%     0.00%  containerd       [unknown]                [.] 0x82e8ffffff43e9ff
     0.00%     0.00%  containerd       containerd               [.] runtime.timerproc
     0.00%     0.00%  containerd       containerd               [.] runtime.notetsleepg
     0.00%     0.00%  containerd       containerd               [.] runtime.netpoll
     0.00%     0.00%  containerd       containerd               [.] internal/poll.(*pollDesc).waitRead
     0.00%     0.00%  containerd       containerd               [.] internal/poll.(*pollDesc).wait
     0.00%     0.00%  containerd       containerd               [.] internal/poll.runtime_pollWait
     0.00%     0.00%  containerd       containerd               [.] runtime.netpollblock
     0.00%     0.00%  containerd       [unknown]                [k] 0xfffffe0000192025
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] account_process_tick
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] put_cred_rcu
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] copy_user_generic_unrolled
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] get_timespec64
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] kmem_cache_alloc_trace
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] mlx5_alloc_cmd_msg
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __radix_tree_lookup
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] __find_get_block
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] find_get_entry
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] radix_tree_lookup_slot
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] _find_next_bit
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] user_path_at_empty
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003d1c7
     0.00%     0.00%  perf             ld-2.23.so               [.] do_lookup_x
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.netpoll
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*inFlow).onData
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).handleData
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] memset_erms
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] core_sys_select
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] do_select
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] poll_schedule_timeout
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] call_function_interrupt
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] smp_call_function_interrupt
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.WithTimeout
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.WithDeadline
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] lockref_get_not_zero
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] 0x00000000002159b0
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] ext4_file_open
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] fscrypt_file_open
     0.00%     0.00%  systemd-cgroups  [kernel.kallsyms]        [k] dget_parent
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.exitsyscall
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_os_write_port
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_ev_gpe_dispatch
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_hw_low_set_gpe
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_hw_write
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_hw_write_port
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/golang.org/x/net/context.WithCancel
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.WithCancel
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __task_pid_nr_ns
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000060c47
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] acpi_os_read_memory
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x00007fb1123977b7
     0.00%     0.00%  systemd-udevd    [unknown]                [k] 0x000055ccdb735b60
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] _raw_spin_lock
     0.00%     0.00%  jbd2/dm-0-8      [kernel.kallsyms]        [k] jbd2_journal_write_metadata_buffer
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __update_idle_core
     0.00%     0.00%  kubelet          [unknown]                [k] 0x00020b5a00000050
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] pick_next_task_idle
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] idle_cpu
     0.00%     0.00%  dockerd          [unknown]                [k] 0x0000000000000001
     0.00%     0.00%  dockerd          [unknown]                [k] 0x0000557d675d3910
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).getReadyTransport
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*ClientConn).getTransport
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*pickerWrapper).pick
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] load_new_mm_cr3
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] enter_lazy_tlb
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] switch_mm
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] switch_mm_irqs_off
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] load_new_mm_cr3
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] _copy_to_user
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __switch_to
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000f039
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00007f14e22dcee8
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] sys_sigaltstack
     0.00%     0.00%  dockerd          dockerd-ce               [.] strconv.FormatInt
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).createHeaderFields
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.encodeTimeout
     0.00%     0.00%  snapd            [kernel.kallsyms]        [k] finish_task_switch
     0.00%     0.00%  systemd-cgroups  ld-2.23.so               [.] _dl_setup_hash
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] complete
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __wake_up_locked
     0.00%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] security_cred_free
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] apparmor_cred_free
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] aa_free_task_context
     0.00%     0.00%  kubelet          [unknown]                [k] 0x00007f14cbffeee8
     0.00%     0.00%  kubelet          [unknown]                [k] 0x00000000071d7a80
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003343b
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Cursor).first
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/namespaces.(*local).withStore.func1
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/services/namespaces.(*local).List.func1
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/metadata.(*namespaceStore).List
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/gc/scheduler.(*gcScheduler).run
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.00%     0.00%  perf             [unknown]                [.] 0x0000000000920858
     0.00%     0.00%  perf             libc-2.23.so             [.] __poll_nocancel
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] sys_poll
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] do_sys_poll
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] poll_freewait
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] remove_wait_queue
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] has_cpu_slab
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] sysfs_slab_remove_workfn
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] css_free_work_fn
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] kobject_put
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] kobject_release
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] mem_cgroup_css_free
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] kmem_cache_release
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] memcg_destroy_kmem_caches
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] shutdown_cache
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] slab_kmem_cache_release
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] __kmem_cache_release
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] __kmem_cache_shutdown
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] free_percpu
     0.00%     0.00%  kworker/30:0     [kernel.kallsyms]        [k] on_each_cpu_cond
     0.00%     0.00%  containerd       containerd               [.] runtime.selparkcommit
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] clear_buddies
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] put_prev_entity
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] put_prev_task_fair
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_idle_enter
     0.00%     0.00%  systemd-journal  libc-2.23.so             [.] __strftime_internal
     0.00%     0.00%  systemd-journal  [unknown]                [.] 0x129a21f89b82d3b1
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] find_next_bit
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] _raw_spin_lock_irq
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] common_interrupt
     0.00%     0.00%  kubelet          [unknown]                [k] 0x00007f14e42e0ee8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000367ae
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] kfree
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __switch_to
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] kvm_check_and_clear_guest_paused
     0.00%     0.00%  dockerd          [unknown]                [.] 0x8502e1f14a0d8b28
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_dynticks_eqs_exit
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_irq_enter
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] hrtimer_try_to_cancel
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003bded
     0.00%     0.00%  kubelet          [unknown]                [.] 0x0000b90008247c83
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00020b5a14cac4f6
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000003babe
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] raise_softirq
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] common_file_perm
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] rw_verify_area
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] security_file_permission
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] apparmor_file_permission
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] _extract_crng
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] extract_crng
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg.func1
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).finish
     0.00%     0.00%  dockerd          dockerd-ce               [.] context.WithCancel.func1
     0.00%     0.00%  dockerd          [unknown]                [k] 0x4800029ba0e85024
     0.00%     0.00%  dockerd          [unknown]                [k] 0x000000c420092501
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] napi_schedule_prep
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000023f0e
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00007f14e117aee8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000032413
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/stats.OutgoingTags
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] process_timeout
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.nanotime
     0.00%     0.00%  dockerd          [unknown]                [.] 0xff6e8c0fc8394818
     0.00%     0.00%  dockerd          [unknown]                [.] 0x0000557d681fa360
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] update_group_capacity
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] rcu_all_qs
     0.00%     0.00%  containerd       containerd               [.] runtime.mput
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] swake_up_locked.part.2
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.interhash
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.mapassign
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] update_curr
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] __lookup_mnt
     0.00%     0.00%  systemd-journal  systemd-journald         [.] realloc@plt
     0.00%     0.00%  ntpd             [unknown]                [k] 0x0000000000007f20
     0.00%     0.00%  ntpd             libc-2.23.so             [.] __select
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] entry_SYSCALL_64_fastpath
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] sys_select
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] core_sys_select
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] do_select
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] poll_schedule_timeout
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] schedule_hrtimeout_range
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] schedule_hrtimeout_range_clock
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] deactivate_task
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] dequeue_task_fair
     0.00%     0.00%  ntpd             [kernel.kallsyms]        [k] dequeue_entity
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] it_real_fn
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] kill_pid_info
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] group_send_sig_info
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] do_send_sig_info
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] send_signal
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __send_signal
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] complete_signal
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] signal_wake_up_state
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] wake_up_state
     0.00%     0.00%  systemd-journal  libpthread-2.23.so       [.] __sendmsg_nocancel
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] sys_sendmsg
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] __sys_sendmsg
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] ___sys_sendmsg
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] sock_sendmsg
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] unix_dgram_sendmsg
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] unix_find_other
     0.00%     0.00%  systemd-journal  [kernel.kallsyms]        [k] kern_path
     0.00%     0.00%  systemd-journal  [unknown]                [.] 0x0000000100000001
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] up
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] update_load_avg
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] first_online_pgdat
     0.00%     0.00%  containerd       [unknown]                [.] 0x0000557022548440
     0.00%     0.00%  containerd       containerd               [.] runtime.exitsyscall
     0.00%     0.00%  containerd       containerd               [.] runtime.exitsyscallfast.func1
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ksize
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] apei_check_gar
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] flush_cpu_slab
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] ext4_getattr
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] vfs_getattr_nosec
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] ext4_file_getattr
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] irq_work_run
     0.00%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] gc_worker
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/daemon/stats.(*Collector).Run
     0.00%     0.00%  ksoftirqd/24     [kernel.kallsyms]        [k] note_gp_changes
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] sched_free_group_rcu
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] sched_free_group
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] free_fair_sched_group
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.notewakeup
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] mod_timer
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] get_nohz_timer_target
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] cpumask_next
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] __update_load_avg_se.isra.38
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] deactivate_task
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] dequeue_task_fair
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] dequeue_entity
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] strncpy_from_user
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] getname_flags
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] aa_dfa_match
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] security_inode_getattr
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] apparmor_inode_getattr
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] common_perm_cond
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] common_perm
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] aa_path_perm
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] profile_path_perm.part.6
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] __aa_path_perm.part.5
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] aa_str_perms
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.runqgrab
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] link_path_walk
     0.00%     0.00%  zabbix_agentd    [kernel.kallsyms]        [k] filename_lookup
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] native_apic_mem_write
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __accumulate_pelt_segments
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] free_pcppages_bulk
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] __x86_indirect_thunk_r15
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] vmstat_update
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] refresh_cpu_vm_stats
     0.00%     0.00%  kworker/10:2     [kernel.kallsyms]        [k] drain_zone_pages
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] smpboot_thread_fn
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] deactivate_task
     0.00%     0.00%  watchdog/10      [kernel.kallsyms]        [k] dequeue_task_rt
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] end_repeat_nmi
     0.00%     0.00%  dockerd          [unknown]                [k] 0x0000557d00000000
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __fdget_pos
     0.00%     0.00%  kubelet          [kernel.kallsyms]        [k] __switch_to
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] __lru_cache_add
     0.00%     0.00%  systemd-udevd    [kernel.kallsyms]        [k] lru_cache_add_active_or_unevictable
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).handleWindowUpdate
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*controlBuffer).put
     0.00%     0.00%  kubelet          kubelet                  [.] 0x00000000000360f5
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00007f14ca7bcee8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x000000000000ea78
     0.00%     0.00%  kubelet          [unknown]                [.] 0x00007f14bb7fdee8
     0.00%     0.00%  kubelet          kubelet                  [.] 0x0000000000034811
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] run_posix_cpu_timers
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.releasep
     0.00%     0.00%  dockerd          [unknown]                [.] 0x8b48fffc506ee824
     0.00%     0.00%  dockerd          [unknown]                [.] 0x0000557d6718e9c0
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.timerproc
     0.00%     0.00%  dockerd          dockerd-ce               [.] time.sendTime
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.selectnbsend
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.chansend
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] free_msg
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] account_entity_enqueue
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] skb_unlink
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] second_overflow
     0.00%     0.00%  perf             [unknown]                [k] 0x0000000003c2ac90
     0.00%     0.00%  perf             [unknown]                [k] 0x0000000003c37900
     0.00%     0.00%  perf             perf                     [.] perf_event__synthesize_threads
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] wq_worker_sleeping
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ctnetlink_conntrack_event
     0.00%     0.00%  containerd       [unknown]                [.] 0x0000562739c09440
     0.00%     0.00%  containerd       containerd               [.] runtime.entersyscallblock_handoff
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] tcp_v4_rcv
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] tcp_v4_send_reset
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ip_send_unicast_reply
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ip_push_pending_frames
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ip_send_skb
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] tcp_packet
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] nf_ct_kill_acct
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] nf_ct_delete
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] nf_conntrack_eventmask_report
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] pvclock_gtod_notify
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] timekeeping_update
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] raw_notifier_call_chain
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] _cond_resched
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] resched_curr
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] rcu_check_callbacks
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] __switch_to
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] select_idle_sibling
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] select_task_rq_fair
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] set_pte_vaddr
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] _raw_spin_lock
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __percpu_counter_sum
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] writeout_period
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] fprop_new_period
     0.00%     0.00%  dockerd          dockerd-ce               [.] github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*loopyWriter).processData
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ghes_copy_tofrom_phys
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] sched_clock
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] neigh_timer_handler
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] read_hpet
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] clocksource_watchdog
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __update_load_avg_se.isra.38
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] sys_write
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] vfs_write
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] __vfs_write
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] sock_write_iter
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] sock_sendmsg
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] unix_stream_sendmsg
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] sock_def_readable
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] __sched_text_start
     0.00%     0.00%  kworker/4:1      [kernel.kallsyms]        [k] schedule
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] update_cfs_group
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] deactivate_task
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] dequeue_task_fair
     0.00%     0.00%  kworker/12:0     [kernel.kallsyms]        [k] refresh_cpu_vm_stats
     0.00%     0.00%  kworker/12:0     [kernel.kallsyms]        [k] ret_from_fork
     0.00%     0.00%  kworker/12:0     [kernel.kallsyms]        [k] kthread
     0.00%     0.00%  kworker/12:0     [kernel.kallsyms]        [k] worker_thread
     0.00%     0.00%  kworker/12:0     [kernel.kallsyms]        [k] process_one_work
     0.00%     0.00%  kworker/12:0     [kernel.kallsyms]        [k] vmstat_update
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] account_idle_ticks
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.acquirep
     0.00%     0.00%  dockerd          [unknown]                [.] 0x240c8948000000b8
     0.00%     0.00%  dockerd          [unknown]                [.] 0x000000c42004c500
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] __update_load_avg_se.isra.38
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] activate_task
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] enqueue_task_fair
     0.00%     0.00%  kworker/u64:0    [kernel.kallsyms]        [k] enqueue_entity
     0.00%     0.00%  dockerd          [unknown]                [.] 0x47000000c42070bf
     0.00%     0.00%  dockerd          [unknown]                [.] 0x000000c42094e960
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath
     0.00%     0.00%  kworker/u66:3    [kernel.kallsyms]        [k] _cond_resched
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] _raw_spin_lock
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] dst_destroy
     0.00%     0.00%  dockerd          [unknown]                [k] 0xc4834848246c8b48
     0.00%     0.00%  dockerd          dockerd-ce               [.] time.AfterFunc
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] dst_destroy_rcu
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] insert_work
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] _raw_spin_lock_irq
     0.00%     0.00%  kworker/24:0     [kernel.kallsyms]        [k] flush_workqueue_prep_pwqs
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ipv4_conntrack_in
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] irq_work_run_list
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] perf_ctx_unlock
     0.00%     0.00%  containerd       containerd               [.] runtime.heapBitsSetType
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).Begin
     0.00%     0.00%  containerd       containerd               [.] github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).beginTx
     0.00%     0.00%  containerd       [kernel.kallsyms]        [k] __calc_delta
     0.00%     0.00%  containerd       [unknown]                [k] 0x000000c420952b47
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] build_skb
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __build_skb
     0.00%     0.00%  containerd       [unknown]                [.] 0x00020b581cd6a250
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] kzfree
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] nmi_restore
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] finish_swait
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ntp_get_next_leap
     0.00%     0.00%  dockerd          dockerd-ce               [.] runtime.resetspinning
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] return_from_SYSCALL_64
     0.00%     0.00%  dockerd          [kernel.kallsyms]        [k] do_syscall_64
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __intel_pmu_disable_all
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] x86_pmu_disable
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] intel_pmu_disable_all
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ghes_read_estatus
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ixgbe_service_timer
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ixgbe_service_event_schedule
     0.00%     0.00%  containerd       containerd               [.] runtime.stopm
     0.00%     0.00%  containerd       containerd               [.] runtime.notesleep
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] prepare_to_swait_event
     0.00%     0.00%  rcu_sched        [kernel.kallsyms]        [k] prepare_to_swait
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ntp_tick_length
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] fill_pud
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] __perf_event_enable
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ctx_resched
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] call_function_single_interrupt
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] smp_call_function_single_interrupt
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] generic_smp_call_function_single_interrupt
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] flush_smp_call_function_queue
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] remote_function
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] native_flush_tlb_single
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] nmi_handle
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] generic_exec_single
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] remote_function
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] event_function
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] __perf_event_enable
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] ctx_resched
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] irq_work_needs_cpu
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] native_flush_tlb_single
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] ghes_notify_nmi
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] nmi_cpu_backtrace
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] native_apic_mem_write
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] native_sched_clock
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] intel_bts_enable_local
     0.00%     0.00%  swapper          [kernel.kallsyms]        [k] intel_pmu_handle_irq
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] native_apic_mem_write
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] event_function
     0.00%     0.00%  ib_send_bw       [kernel.kallsyms]        [k] intel_bts_enable_local
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] intel_bts_enable_local
     0.00%     0.00%  perf             [kernel.kallsyms]        [k] x86_pmu_enable


#
# (Tip: Compare performance results with: perf diff [<old file> <new file>])
#


[-- Attachment #5: vm_tcp_vs_softroce_performance.txt --]
[-- Type: text/plain, Size: 2892 bytes --]

root@test-vm1:~# iperf -c 192.168.122.228
------------------------------------------------------------
Client connecting to 192.168.122.228, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.122.129 port 43148 connected with 192.168.122.228 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  30.9 GBytes  26.5 Gbits/sec
root@test-vm1:~# ib_send_bw -a 192.168.122.228
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF          Device         : rxe0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0011 PSN 0xabf3a0
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:122:129
 remote address: LID 0000 QPN 0x0011 PSN 0x896f1a
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:122:228
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000             0.23               0.21               0.108269
 4          1000             0.74               0.51               0.134786
 8          1000             1.03               0.96               0.125966
 16         1000             3.68               2.02               0.132067
 32         1000             4.18               4.18               0.136819
 64         1000             14.32              8.98               0.147210
 128        1000             16.12              15.98              0.130897
 256        1000             42.41              29.36              0.120244
 512        1000             54.31              50.48              0.103384
 1024       1000             183.71             84.58              0.086611
 2048       1000             204.69             179.39             0.091849
 4096       1000             226.88             201.48             0.051578
 8192       1000             271.09             218.00             0.027905
 16384      1000             271.99             263.34             0.016854
 32768      1000             267.45             259.51             0.008304
 65536      1000             252.69             252.69             0.004043
 131072     1000             266.48             266.48             0.002132
 262144     1000             257.79             256.26             0.001025
 524288     1000             255.60             255.54             0.000511
^C
root@test-vm1:~#


[-- Attachment #6: vm_CONFIG_CRYPTO_result.txt --]
[-- Type: text/plain, Size: 4678 bytes --]

root@wq-vm0:~/5.4/linux-5.4-rc7/tools/perf# cat /boot/config-5.4.0-rc7 | grep CONFIG_CRYPTO_
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=y
CONFIG_CRYPTO_AKCIPHER2=y
CONFIG_CRYPTO_AKCIPHER=y
CONFIG_CRYPTO_KPP2=y
CONFIG_CRYPTO_KPP=y
CONFIG_CRYPTO_ACOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=y
CONFIG_CRYPTO_NULL2=y
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_SIMD=m
CONFIG_CRYPTO_GLUE_HELPER_X86=m
CONFIG_CRYPTO_ENGINE=m
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_DH=y
CONFIG_CRYPTO_ECC=m
CONFIG_CRYPTO_ECDH=m
CONFIG_CRYPTO_ECRDSA=m
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=y
CONFIG_CRYPTO_CHACHA20POLY1305=m
CONFIG_CRYPTO_AEGIS128=m
CONFIG_CRYPTO_AEGIS128_AESNI_SSE2=m
CONFIG_CRYPTO_SEQIV=y
CONFIG_CRYPTO_ECHAINIV=m
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CFB=m
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_OFB=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=y
CONFIG_CRYPTO_KEYWRAP=m
CONFIG_CRYPTO_NHPOLY1305=m
CONFIG_CRYPTO_NHPOLY1305_SSE2=m
CONFIG_CRYPTO_NHPOLY1305_AVX2=m
CONFIG_CRYPTO_ADIANTUM=m
CONFIG_CRYPTO_ESSIV=m
CONFIG_CRYPTO_CMAC=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=y
CONFIG_CRYPTO_CRC32=m
CONFIG_CRYPTO_CRC32_PCLMUL=m
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_CRCT10DIF=y
CONFIG_CRYPTO_CRCT10DIF_PCLMUL=m
CONFIG_CRYPTO_GHASH=y
CONFIG_CRYPTO_POLY1305=m
CONFIG_CRYPTO_POLY1305_X86_64=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m
CONFIG_CRYPTO_LIB_SHA256=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_SHA3=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_STREEBOG=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m
CONFIG_CRYPTO_LIB_AES=y
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_TI=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_LIB_ARC4=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAMELLIA_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST5_AVX_X86_64=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_CAST6_AVX_X86_64=m
CONFIG_CRYPTO_LIB_DES=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_DES3_EDE_X86_64=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_CHACHA20=m
CONFIG_CRYPTO_CHACHA20_X86_64=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX2_X86_64=m
CONFIG_CRYPTO_SM4=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_842=m
CONFIG_CRYPTO_LZ4=m
CONFIG_CRYPTO_LZ4HC=m
CONFIG_CRYPTO_ZSTD=m
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_DRBG_MENU=y
CONFIG_CRYPTO_DRBG_HMAC=y
CONFIG_CRYPTO_DRBG_HASH=y
CONFIG_CRYPTO_DRBG_CTR=y
CONFIG_CRYPTO_DRBG=y
CONFIG_CRYPTO_JITTERENTROPY=y
CONFIG_CRYPTO_USER_API=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_STATS=y
CONFIG_CRYPTO_HASH_INFO=y
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=y
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_CRYPTO_DEV_ATMEL_I2C=m
CONFIG_CRYPTO_DEV_ATMEL_ECC=m
CONFIG_CRYPTO_DEV_ATMEL_SHA204A=m
CONFIG_CRYPTO_DEV_CCP=y
CONFIG_CRYPTO_DEV_CCP_DD=m
CONFIG_CRYPTO_DEV_SP_CCP=y
CONFIG_CRYPTO_DEV_CCP_CRYPTO=m
CONFIG_CRYPTO_DEV_SP_PSP=y
# CONFIG_CRYPTO_DEV_CCP_DEBUGFS is not set
CONFIG_CRYPTO_DEV_QAT=m
CONFIG_CRYPTO_DEV_QAT_DH895xCC=m
# CONFIG_CRYPTO_DEV_QAT_C3XXX is not set
# CONFIG_CRYPTO_DEV_QAT_C62X is not set
CONFIG_CRYPTO_DEV_QAT_DH895xCCVF=m
# CONFIG_CRYPTO_DEV_QAT_C3XXXVF is not set
# CONFIG_CRYPTO_DEV_QAT_C62XVF is not set
CONFIG_CRYPTO_DEV_NITROX=m
CONFIG_CRYPTO_DEV_NITROX_CNN55XX=m
CONFIG_CRYPTO_DEV_CHELSIO=m
CONFIG_CRYPTO_DEV_CHELSIO_TLS=m
CONFIG_CRYPTO_DEV_VIRTIO=m
CONFIG_CRYPTO_DEV_SAFEXCEL=m

[-- Attachment #7: vm_perf.txt --]
[-- Type: text/plain, Size: 219499 bytes --]

# ========
# captured on    : Mon Nov 18 01:34:10 2019
# header version : 1
# data offset    : 264
# data size      : 1310192
# feat offset    : 1310456
# hostname : wq-vm0
# os release : 5.4.0-rc7
# perf version : 5.4.0-rc7
# arch : x86_64
# nrcpus online : 4
# nrcpus avail : 4
# cpudesc : Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz
# cpuid : GenuineIntel,6,85,4
# total memory : 4030284 kB
# cmdline : /root/5.4/linux-5.4-rc7/tools/perf/perf record -ags sleep 10 
# event : name = cycles, , id = { 78, 79, 80, 81 }, size = 112, { sample_period, sample_freq } = 4000, sample_type = IP|TID|TIME|CALLCHAIN|CPU|PERIOD, read_format = TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING|ID, disabled = 1, inherit = 1, mmap = 1, comm = 1, freq = 1, inherit_stat = 1, task = 1, sample_id_all = 1, exclude_guest = 1, mmap2 = 1, comm_exec = 1, ksymbol = 1, bpf_event = 1
# CPU_TOPOLOGY info available, use -I to display
# NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: software = 1, uprobe = 7, cpu = 4, breakpoint = 5, tracepoint = 2, kprobe = 6, msr = 8
# CACHE info available, use -I to display
# time of first sample : 138227.333859
# time of last sample : 138237.339264
# sample duration :  10005.405 ms
# MEM_TOPOLOGY info available, use -I to display
# bpf_prog_info 75: bpf_prog_7be49e3934a125ba addr 0xffffffffc03e2c58 size 200
# bpf_prog_info 76: bpf_prog_2a142ef67aaad174 addr 0xffffffffc0801d5c size 200
# bpf_prog_info 79: bpf_prog_7be49e3934a125ba addr 0xffffffffc0ae73dc size 200
# bpf_prog_info 80: bpf_prog_2a142ef67aaad174 addr 0xffffffffc0b23918 size 200
# bpf_prog_info 81: bpf_prog_7be49e3934a125ba addr 0xffffffffc0a88964 size 200
# bpf_prog_info 82: bpf_prog_2a142ef67aaad174 addr 0xffffffffc0af3cf9 size 200
# missing features: TRACING_DATA BRANCH_STACK GROUP_DESC AUXTRACE STAT CLOCKID DIR_FORMAT COMPRESSED 
# ========
#
#
# Total Lost Samples: 0
#
# Samples: 12K of event 'cycles'
# Event count (approx.): 7361225122
#
# Children      Self  Command          Shared Object               Symbol                                        
# ........  ........  ...............  ..........................  ..............................................
#
    36.12%     0.00%  ib_send_bw       [unknown]                   [k] 0000000000000000
            |
            ---0
               rxe_query_device
               0
               |          
               |--29.58%--pthread_spin_lock
               |          |          
               |           --1.98%--common_interrupt
               |                     |          
               |                      --1.84%--do_IRQ
               |                                |          
               |                                 --1.37%--irq_exit
               |                                           |          
               |                                            --1.36%--__do_softirq
               |                                                      |          
               |                                                       --1.15%--net_rx_action
               |                                                                 |          
               |                                                                  --1.01%--virtnet_poll_tx
               |                                                                            |          
               |                                                                             --0.92%--free_old_xmit_skbs
               |                                                                                       |          
               |                                                                                        --0.68%--napi_consume_skb
               |                                                                                                  |          
               |                                                                                                   --0.64%--skb_release_all
               |          
               |--4.84%--rxe_poll_cq
               |          
                --1.70%--pthread_spin_lock@plt

    36.12%     0.00%  ib_send_bw       librxe-rdmav25.so           [.] rxe_query_device
            |
            ---rxe_query_device
               0
               |          
               |--29.58%--pthread_spin_lock
               |          |          
               |           --1.98%--common_interrupt
               |                     |          
               |                      --1.84%--do_IRQ
               |                                |          
               |                                 --1.37%--irq_exit
               |                                           |          
               |                                            --1.36%--__do_softirq
               |                                                      |          
               |                                                       --1.15%--net_rx_action
               |                                                                 |          
               |                                                                  --1.01%--virtnet_poll_tx
               |                                                                            |          
               |                                                                             --0.92%--free_old_xmit_skbs
               |                                                                                       |          
               |                                                                                        --0.68%--napi_consume_skb
               |                                                                                                  |          
               |                                                                                                   --0.64%--skb_release_all
               |          
               |--4.84%--rxe_poll_cq
               |          
                --1.70%--pthread_spin_lock@plt

    29.59%    27.29%  ib_send_bw       libpthread-2.27.so          [.] pthread_spin_lock
            |          
            |--27.29%--0
            |          rxe_query_device
            |          0
            |          pthread_spin_lock
            |          
             --2.30%--pthread_spin_lock
                       |          
                        --1.98%--common_interrupt
                                  |          
                                   --1.84%--do_IRQ
                                             |          
                                              --1.37%--irq_exit
                                                        |          
                                                         --1.36%--__do_softirq
                                                                   |          
                                                                    --1.15%--net_rx_action
                                                                              |          
                                                                               --1.01%--virtnet_poll_tx
                                                                                         |          
                                                                                          --0.92%--free_old_xmit_skbs
                                                                                                    |          
                                                                                                     --0.68%--napi_consume_skb
                                                                                                               |          
                                                                                                                --0.64%--skb_release_all

    24.03%    22.03%  ib_send_bw       ib_send_bw                  [.] run_iter_bw
            |
            ---run_iter_bw
               |          
                --1.78%--common_interrupt
                          |          
                           --1.74%--do_IRQ
                                     |          
                                      --1.29%--irq_exit
                                                |          
                                                 --1.26%--__do_softirq
                                                           |          
                                                            --1.03%--net_rx_action
                                                                      |          
                                                                       --0.93%--virtnet_poll_tx
                                                                                 |          
                                                                                  --0.76%--free_old_xmit_skbs
                                                                                            |          
                                                                                             --0.52%--napi_consume_skb

    17.17%    17.17%  ib_send_bw       ib_send_bw                  [.] print_report_bw
            |
            ---0x55ec0956f480
               print_report_bw

    17.17%     0.00%  ib_send_bw       [unknown]                   [.] 0x000055ec0956f480
            |
            ---0x55ec0956f480
               print_report_bw

    15.77%    14.63%  ib_send_bw       librxe-rdmav25.so           [.] rxe_poll_cq
            |          
            |--11.25%--rxe_poll_cq
            |          |          
            |           --1.02%--common_interrupt
            |                     |          
            |                      --0.94%--do_IRQ
            |                                |          
            |                                 --0.73%--irq_exit
            |                                           |          
            |                                            --0.72%--__do_softirq
            |                                                      |          
            |                                                       --0.61%--net_rx_action
            |                                                                 |          
            |                                                                  --0.56%--virtnet_poll_tx
            |                                                                            |          
            |                                                                             --0.52%--free_old_xmit_skbs
            |          
             --4.53%--0
                       rxe_query_device
                       0
                       rxe_poll_cq

     7.73%     0.46%  ib_send_bw       ib_send_bw                  [.] sample_get_cpu_mhz
            |          
             --7.28%--sample_get_cpu_mhz
                       |          
                        --7.01%--__vdso_gettimeofday
                                  |          
                                   --3.59%--0x7fff709297db

     7.11%     0.98%  ib_send_bw       [vdso]                      [.] __vdso_gettimeofday
            |          
             --6.13%--__vdso_gettimeofday
                       |          
                        --3.59%--0x7fff709297db

     5.10%     0.20%  ib_send_bw       [kernel.vmlinux]            [k] common_interrupt
            |          
             --4.90%--common_interrupt
                       |          
                        --4.82%--do_IRQ
                                  |          
                                  |--3.58%--irq_exit
                                  |          |          
                                  |           --3.53%--__do_softirq
                                  |                     |          
                                  |                     |--2.94%--net_rx_action
                                  |                     |          |          
                                  |                     |           --2.63%--virtnet_poll_tx
                                  |                     |                     |          
                                  |                     |                      --2.32%--free_old_xmit_skbs
                                  |                     |                                |          
                                  |                     |                                 --1.59%--napi_consume_skb
                                  |                     |                                           |          
                                  |                     |                                            --1.51%--skb_release_all
                                  |                     |                                                      |          
                                  |                     |                                                       --1.05%--skb_release_data
                                  |                     |                                                                 |          
                                  |                     |                                                                  --0.77%--skb_free_head
                                  |                     |                                                                            |          
                                  |                     |                                                                             --0.75%--kfree
                                  |                     |                                                                                       |          
                                  |                     |                                                                                        --0.64%--__slab_free
                                  |                     |          
                                  |                      --0.51%--tasklet_action
                                  |                                tasklet_action_common.isra.22
                                  |                                rxe_do_task
                                  |                                rxe_requester
                                  |          
                                   --1.18%--handle_edge_irq
                                             |          
                                             |--0.59%--irq_chip_ack_parent
                                             |          |          
                                             |           --0.57%--apic_ack_edge
                                             |                     |          
                                             |                      --0.51%--apic_ack_irq
                                             |          
                                              --0.51%--handle_irq_event

     4.83%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] do_IRQ
            |          
             --4.82%--do_IRQ
                       |          
                       |--3.58%--irq_exit
                       |          |          
                       |           --3.53%--__do_softirq
                       |                     |          
                       |                     |--2.94%--net_rx_action
                       |                     |          |          
                       |                     |           --2.63%--virtnet_poll_tx
                       |                     |                     |          
                       |                     |                      --2.32%--free_old_xmit_skbs
                       |                     |                                |          
                       |                     |                                 --1.59%--napi_consume_skb
                       |                     |                                           |          
                       |                     |                                            --1.51%--skb_release_all
                       |                     |                                                      |          
                       |                     |                                                       --1.05%--skb_release_data
                       |                     |                                                                 |          
                       |                     |                                                                  --0.77%--skb_free_head
                       |                     |                                                                            |          
                       |                     |                                                                             --0.75%--kfree
                       |                     |                                                                                       |          
                       |                     |                                                                                        --0.64%--__slab_free
                       |                     |          
                       |                      --0.51%--tasklet_action
                       |                                tasklet_action_common.isra.22
                       |                                rxe_do_task
                       |                                rxe_requester
                       |          
                        --1.18%--handle_edge_irq
                                  |          
                                  |--0.59%--irq_chip_ack_parent
                                  |          |          
                                  |           --0.57%--apic_ack_edge
                                  |                     |          
                                  |                      --0.51%--apic_ack_irq
                                  |          
                                   --0.51%--handle_irq_event

     3.60%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] irq_exit
            |          
             --3.58%--irq_exit
                       |          
                        --3.55%--__do_softirq
                                  |          
                                  |--2.96%--net_rx_action
                                  |          |          
                                  |           --2.63%--virtnet_poll_tx
                                  |                     |          
                                  |                      --2.33%--free_old_xmit_skbs
                                  |                                |          
                                  |                                 --1.60%--napi_consume_skb
                                  |                                           |          
                                  |                                            --1.51%--skb_release_all
                                  |                                                      |          
                                  |                                                       --1.06%--skb_release_data
                                  |                                                                 |          
                                  |                                                                  --0.77%--skb_free_head
                                  |                                                                            |          
                                  |                                                                             --0.75%--kfree
                                  |                                                                                       |          
                                  |                                                                                        --0.64%--__slab_free
                                  |          
                                   --0.51%--tasklet_action
                                             tasklet_action_common.isra.22
                                             rxe_do_task
                                             rxe_requester

     3.59%     3.59%  ib_send_bw       [vdso]                      [.] 0x00000000000007db
     3.59%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297db
            |
            ---0x7fff709297db

     3.55%     0.04%  ib_send_bw       [kernel.vmlinux]            [k] __do_softirq
            |          
             --3.51%--__do_softirq
                       |          
                       |--2.96%--net_rx_action
                       |          |          
                       |           --2.63%--virtnet_poll_tx
                       |                     |          
                       |                      --2.33%--free_old_xmit_skbs
                       |                                |          
                       |                                 --1.60%--napi_consume_skb
                       |                                           |          
                       |                                            --1.51%--skb_release_all
                       |                                                      |          
                       |                                                       --1.06%--skb_release_data
                       |                                                                 |          
                       |                                                                  --0.77%--skb_free_head
                       |                                                                            |          
                       |                                                                             --0.75%--kfree
                       |                                                                                       |          
                       |                                                                                        --0.64%--__slab_free
                       |          
                        --0.51%--tasklet_action
                                  tasklet_action_common.isra.22
                                  rxe_do_task
                                  rxe_requester

     2.98%     0.07%  ib_send_bw       [kernel.vmlinux]            [k] net_rx_action
            |          
             --2.92%--net_rx_action
                       |          
                        --2.63%--virtnet_poll_tx
                                  |          
                                   --2.33%--free_old_xmit_skbs
                                             |          
                                              --1.60%--napi_consume_skb
                                                        |          
                                                         --1.51%--skb_release_all
                                                                   |          
                                                                    --1.06%--skb_release_data
                                                                              |          
                                                                               --0.77%--skb_free_head
                                                                                         |          
                                                                                          --0.75%--kfree
                                                                                                    |          
                                                                                                     --0.64%--__slab_free

     2.65%     0.04%  ib_send_bw       [virtio_net]                [k] virtnet_poll_tx
            |          
             --2.61%--virtnet_poll_tx
                       |          
                        --2.33%--free_old_xmit_skbs
                                  |          
                                   --1.60%--napi_consume_skb
                                             |          
                                              --1.51%--skb_release_all
                                                        |          
                                                         --1.06%--skb_release_data
                                                                   |          
                                                                    --0.77%--skb_free_head
                                                                              |          
                                                                               --0.75%--kfree
                                                                                         |          
                                                                                          --0.64%--__slab_free

     2.42%     0.38%  ib_send_bw       [virtio_net]                [k] free_old_xmit_skbs
            |          
             --2.04%--free_old_xmit_skbs
                       |          
                        --1.65%--napi_consume_skb
                                  |          
                                   --1.51%--skb_release_all
                                             |          
                                              --1.06%--skb_release_data
                                                        |          
                                                         --0.77%--skb_free_head
                                                                   |          
                                                                    --0.75%--kfree
                                                                              |          
                                                                               --0.64%--__slab_free

     1.70%     1.54%  ib_send_bw       librxe-rdmav25.so           [.] pthread_spin_lock@plt
            |          
             --1.54%--0
                       rxe_query_device
                       0
                       pthread_spin_lock@plt

     1.65%     0.07%  ib_send_bw       [kernel.vmlinux]            [k] napi_consume_skb
            |          
             --1.58%--napi_consume_skb
                       |          
                        --1.51%--skb_release_all
                                  |          
                                   --1.06%--skb_release_data
                                             |          
                                              --0.77%--skb_free_head
                                                        |          
                                                         --0.75%--kfree
                                                                   |          
                                                                    --0.64%--__slab_free

     1.63%     0.00%  swapper          [kernel.vmlinux]            [k] 0xffffffff97e000d4
            |
            ---0xffffffff97e000d4
               |          
                --1.59%--start_secondary
                          |          
                           --1.59%--cpu_startup_entry
                                     |          
                                      --1.59%--do_idle
                                                |          
                                                 --1.56%--default_idle_call
                                                           arch_cpu_idle
                                                           default_idle
                                                           |          
                                                            --1.55%--native_safe_halt
                                                                      |          
                                                                       --1.50%--common_interrupt
                                                                                 do_IRQ
                                                                                 |          
                                                                                  --1.48%--irq_exit
                                                                                            __do_softirq
                                                                                            |          
                                                                                             --1.46%--tasklet_action
                                                                                                       tasklet_action_common.isra.22
                                                                                                       rxe_do_task
                                                                                                       rxe_requester
                                                                                                       |          
                                                                                                        --0.52%--rxe_send

     1.63%     0.00%  swapper          [kernel.vmlinux]            [k] cpu_startup_entry
            |
            ---cpu_startup_entry
               |          
                --1.62%--do_idle
                          |          
                           --1.58%--default_idle_call
                                     arch_cpu_idle
                                     default_idle
                                     |          
                                      --1.57%--native_safe_halt
                                                |          
                                                 --1.50%--common_interrupt
                                                           do_IRQ
                                                           |          
                                                            --1.48%--irq_exit
                                                                      __do_softirq
                                                                      |          
                                                                       --1.46%--tasklet_action
                                                                                 tasklet_action_common.isra.22
                                                                                 rxe_do_task
                                                                                 rxe_requester
                                                                                 |          
                                                                                  --0.52%--rxe_send

     1.62%     0.00%  swapper          [kernel.vmlinux]            [k] do_idle
            |          
             --1.62%--do_idle
                       |          
                        --1.58%--default_idle_call
                                  arch_cpu_idle
                                  default_idle
                                  |          
                                   --1.57%--native_safe_halt
                                             |          
                                              --1.50%--common_interrupt
                                                        do_IRQ
                                                        |          
                                                         --1.48%--irq_exit
                                                                   __do_softirq
                                                                   |          
                                                                    --1.46%--tasklet_action
                                                                              tasklet_action_common.isra.22
                                                                              rxe_do_task
                                                                              rxe_requester
                                                                              |          
                                                                               --0.52%--rxe_send

     1.59%     0.00%  swapper          [kernel.vmlinux]            [k] start_secondary
            |
            ---start_secondary
               |          
                --1.59%--cpu_startup_entry
                          |          
                           --1.59%--do_idle
                                     |          
                                      --1.56%--default_idle_call
                                                arch_cpu_idle
                                                default_idle
                                                |          
                                                 --1.55%--native_safe_halt
                                                           |          
                                                            --1.50%--common_interrupt
                                                                      do_IRQ
                                                                      |          
                                                                       --1.48%--irq_exit
                                                                                 __do_softirq
                                                                                 |          
                                                                                  --1.46%--tasklet_action
                                                                                            tasklet_action_common.isra.22
                                                                                            rxe_do_task
                                                                                            rxe_requester
                                                                                            |          
                                                                                             --0.52%--rxe_send

     1.59%     1.41%  ib_send_bw       libpthread-2.27.so          [.] pthread_spin_unlock
            |
            ---pthread_spin_unlock

     1.58%     0.00%  swapper          [kernel.vmlinux]            [k] default_idle_call
            |
            ---default_idle_call
               arch_cpu_idle
               default_idle
               |          
                --1.57%--native_safe_halt
                          |          
                           --1.50%--common_interrupt
                                     do_IRQ
                                     |          
                                      --1.48%--irq_exit
                                                __do_softirq
                                                |          
                                                 --1.46%--tasklet_action
                                                           tasklet_action_common.isra.22
                                                           rxe_do_task
                                                           rxe_requester
                                                           |          
                                                            --0.52%--rxe_send

     1.58%     0.00%  swapper          [kernel.vmlinux]            [k] arch_cpu_idle
            |
            ---arch_cpu_idle
               default_idle
               |          
                --1.57%--native_safe_halt
                          |          
                           --1.50%--common_interrupt
                                     do_IRQ
                                     |          
                                      --1.48%--irq_exit
                                                __do_softirq
                                                |          
                                                 --1.46%--tasklet_action
                                                           tasklet_action_common.isra.22
                                                           rxe_do_task
                                                           rxe_requester
                                                           |          
                                                            --0.52%--rxe_send

     1.58%     0.00%  swapper          [kernel.vmlinux]            [k] default_idle
            |
            ---default_idle
               |          
                --1.57%--native_safe_halt
                          |          
                           --1.50%--common_interrupt
                                     do_IRQ
                                     |          
                                      --1.48%--irq_exit
                                                __do_softirq
                                                |          
                                                 --1.46%--tasklet_action
                                                           tasklet_action_common.isra.22
                                                           rxe_do_task
                                                           rxe_requester
                                                           |          
                                                            --0.52%--rxe_send

     1.57%     0.00%  swapper          [kernel.vmlinux]            [k] native_safe_halt
            |
            ---native_safe_halt
               |          
                --1.50%--common_interrupt
                          do_IRQ
                          |          
                           --1.48%--irq_exit
                                     __do_softirq
                                     |          
                                      --1.46%--tasklet_action
                                                tasklet_action_common.isra.22
                                                rxe_do_task
                                                rxe_requester
                                                |          
                                                 --0.52%--rxe_send

     1.56%     0.03%  ib_send_bw       [kernel.vmlinux]            [k] skb_release_all
            |          
             --1.53%--skb_release_all
                       |          
                        --1.07%--skb_release_data
                                  |          
                                   --0.77%--skb_free_head
                                             |          
                                              --0.75%--kfree
                                                        |          
                                                         --0.64%--__slab_free

     1.53%     0.00%  swapper          [kernel.vmlinux]            [k] irq_exit
            |          
             --1.53%--irq_exit
                       |          
                        --1.53%--__do_softirq
                                  |          
                                   --1.46%--tasklet_action
                                             tasklet_action_common.isra.22
                                             rxe_do_task
                                             rxe_requester
                                             |          
                                              --0.52%--rxe_send

     1.53%     0.00%  swapper          [kernel.vmlinux]            [k] __do_softirq
            |
            ---__do_softirq
               |          
                --1.46%--tasklet_action
                          tasklet_action_common.isra.22
                          rxe_do_task
                          rxe_requester
                          |          
                           --0.52%--rxe_send

     1.50%     0.00%  swapper          [kernel.vmlinux]            [k] common_interrupt
            |
            ---common_interrupt
               do_IRQ
               |          
                --1.48%--irq_exit
                          __do_softirq
                          |          
                           --1.46%--tasklet_action
                                     tasklet_action_common.isra.22
                                     rxe_do_task
                                     rxe_requester
                                     |          
                                      --0.52%--rxe_send

     1.50%     0.00%  swapper          [kernel.vmlinux]            [k] do_IRQ
            |
            ---do_IRQ
               |          
                --1.48%--irq_exit
                          __do_softirq
                          |          
                           --1.46%--tasklet_action
                                     tasklet_action_common.isra.22
                                     rxe_do_task
                                     rxe_requester
                                     |          
                                      --0.52%--rxe_send

     1.46%     0.00%  swapper          [kernel.vmlinux]            [k] tasklet_action
            |
            ---tasklet_action
               tasklet_action_common.isra.22
               rxe_do_task
               rxe_requester
               |          
                --0.52%--rxe_send

     1.46%     0.00%  swapper          [kernel.vmlinux]            [k] tasklet_action_common.isra.22
            |
            ---tasklet_action_common.isra.22
               rxe_do_task
               rxe_requester
               |          
                --0.52%--rxe_send

     1.46%     0.00%  swapper          [rdma_rxe]                  [k] rxe_do_task
            |
            ---rxe_do_task
               rxe_requester
               |          
                --0.52%--rxe_send

     1.46%     0.05%  swapper          [rdma_rxe]                  [k] rxe_requester
            |          
             --1.40%--rxe_requester
                       |          
                        --0.52%--rxe_send

     1.25%     0.08%  ib_send_bw       [kernel.vmlinux]            [k] handle_edge_irq
            |          
             --1.16%--handle_edge_irq
                       |          
                       |--0.60%--irq_chip_ack_parent
                       |          |          
                       |           --0.58%--apic_ack_edge
                       |                     |          
                       |                      --0.52%--apic_ack_irq
                       |                                |          
                       |                                 --0.50%--kvm_guest_apic_eoi_write
                       |          
                        --0.52%--handle_irq_event

     1.09%     0.30%  ib_send_bw       [kernel.vmlinux]            [k] skb_release_data
            |          
             --0.79%--skb_release_data
                       |          
                        --0.77%--skb_free_head
                                  |          
                                   --0.75%--kfree
                                             |          
                                              --0.64%--__slab_free

     0.79%     0.66%  ib_send_bw       [kernel.vmlinux]            [k] __slab_free
     0.78%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] skb_free_head
            |          
             --0.76%--skb_free_head
                       |          
                        --0.75%--kfree
                                  |          
                                   --0.64%--__slab_free

     0.76%     0.12%  ib_send_bw       [kernel.vmlinux]            [k] kfree
            |          
             --0.64%--kfree
                       __slab_free

     0.64%     0.06%  ib_send_bw       [kernel.vmlinux]            [k] irq_chip_ack_parent
            |          
             --0.59%--irq_chip_ack_parent
                       |          
                        --0.58%--apic_ack_edge
                                  |          
                                   --0.52%--apic_ack_irq
                                             |          
                                              --0.50%--kvm_guest_apic_eoi_write

     0.61%     0.00%  ib_send_bw       [rdma_rxe]                  [k] rxe_do_task
            |
            ---rxe_do_task
               rxe_requester

     0.61%     0.03%  ib_send_bw       [rdma_rxe]                  [k] rxe_requester
            |          
             --0.58%--rxe_requester

     0.60%     0.03%  ib_send_bw       [kernel.vmlinux]            [k] apic_ack_edge
            |          
             --0.58%--apic_ack_edge
                       |          
                        --0.52%--apic_ack_irq
                                  |          
                                   --0.50%--kvm_guest_apic_eoi_write

     0.54%     0.07%  ib_send_bw       [kernel.vmlinux]            [k] kvm_guest_apic_eoi_write
     0.52%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] handle_irq_event
            |          
             --0.50%--handle_irq_event

     0.52%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] apic_ack_irq
            |          
             --0.52%--apic_ack_irq
                       |          
                        --0.50%--kvm_guest_apic_eoi_write

     0.52%     0.02%  swapper          [rdma_rxe]                  [k] rxe_send
     0.51%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] tasklet_action
            |
            ---tasklet_action
               tasklet_action_common.isra.22
               rxe_do_task
               rxe_requester

     0.51%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] tasklet_action_common.isra.22
            |
            ---tasklet_action_common.isra.22
               rxe_do_task
               rxe_requester

     0.49%     0.00%  swapper          [kernel.vmlinux]            [k] ip_local_out
     0.49%     0.49%  ib_send_bw       [kernel.vmlinux]            [k] native_apic_msr_eoi_write
     0.48%     0.00%  swapper          [kernel.vmlinux]            [k] ip_output
     0.48%     0.01%  swapper          [kernel.vmlinux]            [k] ip_finish_output
     0.45%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] handle_irq_event_percpu
     0.45%     0.01%  swapper          [kernel.vmlinux]            [k] __ip_finish_output
     0.44%     0.43%  ib_send_bw       [rdma_rxe]                  [k] rxe_skb_tx_dtor
     0.44%     0.01%  swapper          [kernel.vmlinux]            [k] ip_finish_output2
     0.43%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] skb_release_head_state
     0.42%     0.01%  swapper          [kernel.vmlinux]            [k] __dev_queue_xmit
     0.42%     0.00%  swapper          [kernel.vmlinux]            [k] dev_queue_xmit
     0.41%     0.41%  ib_send_bw       [kernel.vmlinux]            [k] irq_entries_start
     0.39%     0.24%  ib_send_bw       [kernel.vmlinux]            [k] vring_interrupt
     0.39%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_get_buf
     0.38%     0.28%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_get_buf_ctx
     0.38%     0.01%  swapper          [kernel.vmlinux]            [k] sch_direct_xmit
     0.37%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] __handle_irq_event_percpu
     0.34%     0.03%  ib_send_bw       [kernel.vmlinux]            [k] entry_SYSCALL_64
     0.32%     0.00%  ib_send_bw       libpthread-2.27.so          [.] __libc_write
     0.31%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] do_syscall_64
     0.29%     0.29%  ib_send_bw       [vdso]                      [.] 0x0000000000000813
     0.29%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929813
     0.29%     0.29%  ib_send_bw       [vdso]                      [.] 0x0000000000000818
     0.29%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929818
     0.29%     0.00%  swapper          [rdma_rxe]                  [k] copy_data
     0.28%     0.02%  swapper          [rdma_rxe]                  [k] rxe_init_packet
     0.27%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] __kfree_skb_flush
     0.27%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __x64_sys_write
     0.27%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] ksys_write
     0.27%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] vfs_write
     0.26%     0.01%  swapper          [kernel.vmlinux]            [k] dev_hard_start_xmit
     0.26%     0.11%  ib_send_bw       [kernel.vmlinux]            [k] kmem_cache_free_bulk
     0.25%     0.00%  ib_send_bw       [rdma_rxe]                  [k] rxe_send
     0.25%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] ip_local_out
     0.25%     0.02%  swapper          [virtio_net]                [k] start_xmit
     0.25%     0.25%  ib_send_bw       [vdso]                      [.] 0x00000000000007fc
     0.25%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297fc
     0.25%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __vfs_write
     0.25%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] ip_output
     0.25%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] ip_finish_output
     0.23%     0.01%  ib_send_bw       [ib_uverbs]                 [k] ib_uverbs_write
     0.23%     0.15%  ib_send_bw       [kernel.vmlinux]            [k] _raw_spin_lock
     0.23%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] __ip_finish_output
     0.23%     0.23%  ib_send_bw       [kernel.vmlinux]            [k] interrupt_entry
     0.22%     0.22%  ib_send_bw       [vdso]                      [.] 0x0000000000000756
     0.22%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929756
     0.22%     0.02%  ib_send_bw       [ib_uverbs]                 [k] ib_uverbs_post_send
     0.22%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] ip_finish_output2
     0.21%     0.21%  ib_send_bw       [vdso]                      [.] 0x00000000000007dd
     0.21%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297dd
     0.20%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] dev_queue_xmit
     0.19%     0.01%  swapper          [crc32_pclmul]              [k] crc32_pclmul_update
     0.19%     0.06%  swapper          [rdma_rxe]                  [k] rxe_prepare
     0.19%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __dev_queue_xmit
     0.19%     0.00%  swapper          [kernel.vmlinux]            [k] crypto_shash_update
     0.19%     0.19%  swapper          [kernel.vmlinux]            [k] memset_erms
     0.19%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] sch_direct_xmit
     0.18%     0.18%  ib_send_bw       [vdso]                      [.] 0x00000000000007b0
     0.18%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297b0
     0.17%     0.00%  swapper          [virtio_net]                [k] free_old_xmit_skbs
     0.17%     0.03%  swapper          [kernel.vmlinux]            [k] __alloc_skb
     0.17%     0.01%  swapper          [rdma_rxe]                  [k] rxe_mem_copy
     0.15%     0.01%  ib_send_bw       [virtio_net]                [k] skb_xmit_done
     0.15%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] dev_hard_start_xmit
     0.15%     0.00%  swapper          [rdma_rxe]                  [k] rxe_crc32.isra.4
     0.15%     0.05%  ib_send_bw       [rdma_rxe]                  [k] rxe_post_send
     0.14%     0.00%  ib_send_bw       [virtio_net]                [k] start_xmit
     0.14%     0.00%  ib_send_bw       [virtio_net]                [k] virtqueue_napi_schedule
     0.13%     0.01%  ib_send_bw       [rdma_rxe]                  [k] rxe_init_packet
     0.13%     0.13%  ib_send_bw       [vdso]                      [.] 0x00000000000006ea
     0.13%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709296ea
     0.12%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] put_cpu_partial
     0.11%     0.11%  ib_send_bw       [vdso]                      [.] 0x0000000000000775
     0.11%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929775
     0.10%     0.00%  swapper          [crc32_pclmul]              [k] crc32_pclmul_le
     0.10%     0.01%  ib_send_bw       [rdma_rxe]                  [k] rxe_run_task
     0.10%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] ret_from_fork
     0.10%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] kthread
     0.10%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] worker_thread
     0.10%     0.10%  ib_send_bw       [kernel.vmlinux]            [k] detach_buf_split
     0.10%     0.00%  perf             [kernel.vmlinux]            [k] entry_SYSCALL_64
     0.10%     0.00%  perf             [kernel.vmlinux]            [k] do_syscall_64
     0.10%     0.10%  ib_send_bw       [vdso]                      [.] 0x00000000000006d0
     0.10%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709296d0
     0.10%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] process_one_work
     0.10%     0.09%  swapper          [kernel.vmlinux]            [k] virtqueue_get_buf_ctx
     0.10%     0.00%  swapper          [kernel.vmlinux]            [k] virtqueue_get_buf
     0.09%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_fb_helper_dirty_work
     0.09%     0.01%  ib_send_bw       [rdma_rxe]                  [k] copy_data
     0.09%     0.01%  ib_send_bw       [virtio_net]                [k] virtqueue_napi_complete
     0.09%     0.09%  ib_send_bw       [vdso]                      [.] 0x00000000000007b8
     0.09%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297b8
     0.09%     0.09%  ib_send_bw       [vdso]                      [.] 0x0000000000000810
     0.09%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929810
     0.09%     0.05%  ib_send_bw       [kernel.vmlinux]            [k] unfreeze_partials.isra.81
     0.09%     0.00%  swapper          [kernel.vmlinux]            [k] __kmalloc_reserve.isra.61
     0.08%     0.08%  ib_send_bw       [kernel.vmlinux]            [k] __napi_schedule
     0.08%     0.08%  ib_send_bw       [vdso]                      [.] 0x00000000000006d8
     0.08%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709296d8
     0.08%     0.08%  ib_send_bw       [vdso]                      [.] 0x00000000000006c5
     0.08%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709296c5
     0.08%     0.08%  ib_send_bw       [vdso]                      [.] 0x0000000000000801
     0.08%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929801
     0.08%     0.08%  ib_send_bw       [vdso]                      [.] 0x0000000000000760
     0.08%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929760
     0.08%     0.07%  ib_send_bw       [kernel.vmlinux]            [k] __pv_queued_spin_lock_slowpath
     0.08%     0.08%  swapper          [kernel.vmlinux]            [k] crc32_le_base
     0.08%     0.01%  ib_send_bw       [rdma_rxe]                  [k] rxe_mem_copy
     0.07%     0.02%  ib_send_bw       [crc32_pclmul]              [k] crc32_pclmul_update
     0.07%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] crypto_shash_update
     0.07%     0.01%  swapper          [kernel.vmlinux]            [k] napi_consume_skb
     0.07%     0.01%  swapper          [rdma_rxe]                  [k] rxe_icrc_hdr
     0.07%     0.07%  ib_send_bw       [kernel.vmlinux]            [k] napi_complete_done
     0.07%     0.07%  ib_send_bw       [vdso]                      [.] 0x0000000000000786
     0.07%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929786
     0.07%     0.07%  ib_send_bw       [vdso]                      [.] 0x00000000000007cb
     0.07%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297cb
     0.07%     0.07%  ib_send_bw       [vdso]                      [.] 0x000000000000082a
     0.07%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff7092982a
     0.07%     0.01%  ib_send_bw       [rdma_rxe]                  [k] rxe_prepare
     0.07%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] apic_timer_interrupt
     0.07%     0.00%  swapper          [kernel.vmlinux]            [k] __dev_kfree_skb_any
     0.07%     0.00%  swapper          [kernel.vmlinux]            [k] apic_timer_interrupt
     0.07%     0.00%  ib_send_bw       [rdma_rxe]                  [k] rxe_crc32.isra.4
     0.06%     0.00%  swapper          [kernel.vmlinux]            [k] smp_apic_timer_interrupt
     0.06%     0.06%  ib_send_bw       [vdso]                      [.] 0x0000000000000768
     0.06%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929768
     0.06%     0.06%  ib_send_bw       [vdso]                      [.] 0x00000000000006df
     0.06%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709296df
     0.06%     0.06%  ib_send_bw       [vdso]                      [.] 0x0000000000000763
     0.06%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff70929763
     0.06%     0.06%  ib_send_bw       [vdso]                      [.] 0x00000000000007a5
     0.06%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297a5
     0.06%     0.00%  swapper          [kernel.vmlinux]            [k] consume_skb
     0.06%     0.06%  ib_send_bw       [vdso]                      [.] 0x00000000000007f2
     0.06%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297f2
     0.06%     0.01%  swapper          [rdma_rxe]                  [k] rxe_crc32.isra.1
     0.06%     0.06%  swapper          [kernel.vmlinux]            [k] _raw_read_unlock_irqrestore
     0.06%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] smp_apic_timer_interrupt
     0.06%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_client_buffer_vmap
     0.06%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_vmap
     0.06%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_shmem_vmap
     0.06%     0.06%  swapper          [kernel.vmlinux]            [k] memcpy_erms
     0.06%     0.06%  ib_send_bw       [kernel.vmlinux]            [k] __raw_callee_save___pv_queued_spin_unlock
     0.06%     0.02%  swapper          [kernel.vmlinux]            [k] validate_xmit_skb
     0.06%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] __alloc_skb
     0.06%     0.00%  swapper          [kernel.vmlinux]            [k] skb_release_all
     0.05%     0.00%  perf             libc-2.27.so                [.] __libc_start_main
     0.05%     0.00%  perf             perf                        [.] main
     0.05%     0.00%  perf             perf                        [.] handle_internal_command
     0.05%     0.00%  perf             perf                        [.] run_builtin
     0.05%     0.00%  perf             perf                        [.] cmd_record
     0.05%     0.05%  swapper          [crc32_pclmul]              [k] loop_64
     0.05%     0.05%  ib_send_bw       [kernel.vmlinux]            [k] napi_schedule_prep
     0.05%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_shmem_get_pages
     0.05%     0.00%  perf             [unknown]                   [k] 0x3a06258d4c544155
     0.05%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_get_pages
     0.05%     0.00%  swapper          [kernel.vmlinux]            [k] validate_xmit_skb_list
     0.05%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __dev_kfree_skb_any
     0.05%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] consume_skb
     0.05%     0.02%  swapper          [rdma_rxe]                  [k] rxe_pool_get_index
     0.05%     0.00%  perf             perf                        [.] evlist__enable
     0.05%     0.00%  perf             perf                        [.] evsel__enable
     0.05%     0.00%  perf             perf                        [.] perf_evsel__enable
     0.05%     0.05%  perf             [kernel.vmlinux]            [k] smp_call_function_single
     0.05%     0.00%  perf             libc-2.27.so                [.] __GI___ioctl
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] __x64_sys_ioctl
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] ksys_ioctl
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] do_vfs_ioctl
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] perf_ioctl
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] _perf_ioctl
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] perf_event_for_each_child
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] _perf_event_enable
     0.05%     0.00%  perf             [kernel.vmlinux]            [k] event_function_call
     0.05%     0.00%  swapper          [rdma_rxe]                  [k] lookup_mem
     0.05%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] hrtimer_interrupt
     0.04%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] shmem_read_mapping_page_gfp
     0.04%     0.04%  ib_send_bw       [kernel.vmlinux]            [k] netif_tx_wake_queue
     0.04%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] add_interrupt_randomness
     0.04%     0.02%  swapper          [ib_core]                   [k] rdma_get_gid_attr
     0.04%     0.00%  perf             libpthread-2.27.so          [.] __libc_write
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] __x64_sys_write
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] ksys_write
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] vfs_write
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] __vfs_write
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] new_sync_write
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] ext4_file_write_iter
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] __generic_file_write_iter
     0.04%     0.00%  perf             [kernel.vmlinux]            [k] generic_perform_write
     0.04%     0.00%  swapper          [kernel.vmlinux]            [k] skb_release_data
     0.04%     0.04%  swapper          [kernel.vmlinux]            [k] __raw_callee_save___pv_queued_spin_unlock
     0.04%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] shmem_getpage_gfp
     0.04%     0.01%  ib_send_bw       [rdma_rxe]                  [k] rxe_icrc_hdr
     0.04%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __hrtimer_run_queues
     0.03%     0.00%  swapper          [kernel.vmlinux]            [k] skb_free_head
     0.03%     0.03%  swapper          [kernel.vmlinux]            [k] _raw_read_lock_irqsave
     0.03%     0.01%  kworker/1:1-eve  [kernel.vmlinux]            [k] find_lock_entry
     0.03%     0.03%  ib_send_bw       librxe-rdmav25.so           [.] rxe_post_send
     0.03%     0.03%  ib_send_bw       [kernel.vmlinux]            [k] memset_erms
     0.03%     0.00%  ib_send_bw       [ib_core]                   [k] rdma_get_gid_attr
     0.03%     0.03%  swapper          [kernel.vmlinux]            [k] _raw_spin_lock
     0.03%     0.00%  swapper          [kernel.vmlinux].init.text  [k] x86_64_start_kernel
     0.03%     0.00%  swapper          [kernel.vmlinux].init.text  [k] x86_64_start_reservations
     0.03%     0.00%  swapper          [kernel.vmlinux].init.text  [k] start_kernel
     0.03%     0.00%  swapper          [kernel.vmlinux].init.text  [k] arch_call_rest_init
     0.03%     0.00%  swapper          [kernel.vmlinux]            [k] rest_init
     0.03%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] prepare_exit_to_usermode
     0.03%     0.03%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_kick_prepare
     0.03%     0.02%  swapper          [kernel.vmlinux]            [k] kfree
     0.03%     0.03%  ib_send_bw       [kernel.vmlinux]            [k] __x86_indirect_thunk_rax
     0.03%     0.00%  swapper          [kernel.vmlinux]            [k] net_rx_action
     0.03%     0.00%  swapper          [virtio_net]                [k] virtnet_poll
     0.03%     0.01%  swapper          [virtio_net]                [k] receive_buf
     0.03%     0.00%  swapper          [ib_core]                   [k] rdma_read_gid_attr_ndev_rcu
     0.03%     0.02%  swapper          [kernel.vmlinux]            [k] kmem_cache_alloc_node
     0.03%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_client_buffer_vunmap
     0.03%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_vunmap
     0.03%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_shmem_vunmap
     0.03%     0.03%  swapper          [kernel.vmlinux]            [k] netif_skb_features
     0.03%     0.00%  ib_send_bw       [rdma_rxe]                  [k] rxe_crc32.isra.1
     0.03%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] _raw_read_unlock_irqrestore
     0.02%     0.01%  swapper          [kernel.vmlinux]            [k] __kmalloc_node_track_caller
     0.02%     0.01%  swapper          [kernel.vmlinux]            [k] run_timer_softirq
     0.02%     0.00%  perf             [unknown]                   [k] 0x00007efe80af5c23
     0.02%     0.02%  ib_send_bw       [crc32_pclmul]              [k] loop_64
     0.02%     0.02%  swapper          [kernel.vmlinux]            [k] kernel_fpu_begin
     0.02%     0.02%  swapper          [kernel.vmlinux]            [k] __ksize
     0.02%     0.01%  swapper          [kernel.vmlinux]            [k] ___slab_alloc
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] __slab_alloc
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] tick_nohz_idle_stop_tick
     0.02%     0.02%  swapper          [kernel.vmlinux]            [k] dst_release
     0.02%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_put_pages
     0.02%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_shmem_put_pages
     0.02%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_gem_shmem_put_pages_locked
     0.02%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] timerqueue_add
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] enqueue_hrtimer
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] run_rebalance_domains
     0.02%     0.02%  swapper          [kernel.vmlinux]            [k] __cgroup_bpf_run_filter_skb
     0.02%     0.02%  swapper          [kernel.vmlinux]            [k] ip_idents_reserve
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] __ip_select_ident
     0.02%     0.02%  swapper          [kernel.vmlinux]            [k] ip_send_check
     0.02%     0.01%  kworker/1:1-eve  [kernel.vmlinux]            [k] find_get_entry
     0.02%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] validate_xmit_skb
     0.02%     0.02%  ib_send_bw       [ib_uverbs]                 [k] rdma_lookup_put_uobject
     0.02%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] native_write_msr
     0.02%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] ret_from_fork
     0.02%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] kthread
     0.02%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] smpboot_thread_fn
     0.02%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] run_ksoftirqd
     0.02%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] __do_softirq
     0.02%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] tasklet_action
     0.02%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] tasklet_action_common.isra.22
     0.02%     0.00%  ksoftirqd/2      [rdma_rxe]                  [k] rxe_do_task
     0.02%     0.00%  ksoftirqd/2      [rdma_rxe]                  [k] rxe_requester
     0.02%     0.02%  swapper          [rdma_rxe]                  [k] rxe_skb_tx_dtor
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] tick_sched_timer
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] tick_sched_handle
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] napi_gro_receive
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] netif_receive_skb_internal
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] __netif_receive_skb
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] __netif_receive_skb_one_core
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] ip_rcv
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] ip_rcv_finish
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] ip_local_deliver
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] ip_local_deliver_finish
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] ip_protocol_deliver_rcu
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] udp_rcv
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] __udp4_lib_rcv
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] udp_unicast_rcv_skb
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] udp_queue_rcv_skb
     0.02%     0.00%  swapper          [kernel.vmlinux]            [k] udp_queue_rcv_one_skb
     0.02%     0.00%  swapper          [rdma_rxe]                  [k] rxe_udp_encap_recv
     0.02%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ret_from_fork
     0.02%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] kthread
     0.02%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] worker_thread
     0.02%     0.00%  perf             [kernel.vmlinux]            [k] ext4_da_write_begin
     0.02%     0.02%  swapper          [kernel.vmlinux]            [k] virtqueue_kick_prepare
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] discard_slab
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __free_slab
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __free_pages
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __kmalloc_reserve.isra.61
     0.02%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] __radix_tree_lookup
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] radix_tree_lookup
     0.02%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] __ksize
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] memdup_user
     0.02%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] ip_send_check
     0.02%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] __get_user_8
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] 0xffffffff97e9f348
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] do_group_exit
     0.02%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] do_exit
     0.02%     0.02%  ib_send_bw       [kernel.vmlinux]            [k] _raw_read_lock_irqsave
     0.02%     0.01%  ib_send_bw       [crc32_pclmul]              [k] crc32_pclmul_le
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] virtqueue_add_outbuf
     0.01%     0.01%  swapper          [sch_fq_codel]              [k] fq_codel_dequeue
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] irq_enter
     0.01%     0.01%  swapper          [ib_core]                   [k] rdma_put_gid_attr
     0.01%     0.01%  swapper          [rdma_rxe]                  [k] rxe_find_route
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] tick_irq_enter
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] call_timer_fn
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] __slab_free
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_interrupt
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __pagevec_release
     0.01%     0.00%  swapper          [rdma_rxe]                  [k] rxe_rcv
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] update_blocked_averages
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] __local_bh_enable_ip
     0.01%     0.01%  kworker/1:1-eve  [kernel.vmlinux]            [k] release_pages
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] get_partial_node.isra.82
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] reschedule_interrupt
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] smp_reschedule_interrupt
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] scheduler_ipi
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] process_one_work
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] try_to_wake_up
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] wake_up_process
     0.01%     0.01%  ib_send_bw       [crc32_pclmul]              [k] loop_16
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] validate_xmit_skb_list
     0.01%     0.01%  perf             [kernel.vmlinux]            [k] clear_page_erms
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] grab_cache_page_write_begin
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] pagecache_get_page
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] __page_cache_alloc
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] alloc_pages_current
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] __alloc_pages_nodemask
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] get_page_from_freelist
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] virtqueue_notify
     0.01%     0.00%  perf             [unknown]                   [k] 0x00007efe7fe59d56
     0.01%     0.01%  swapper          [crc32_pclmul]              [k] less_64
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] pvclock_clocksource_read
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] __hrtimer_run_queues
     0.01%     0.00%  perf             libc-2.27.so                [.] __GI___execve
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] __x64_sys_execve
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b793
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004adc7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022732
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e8b1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025613
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000488b2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a7e6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000459d2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004acc4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042aef
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003fc0e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e2c7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042124
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b3e5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027b39
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003cd2e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042021
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d31a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c970
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f851
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047905
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039e4e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036f6c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028504
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002aa1a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035fbe
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001bfb6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027a36
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d8fb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b689
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021d77
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000429e6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ec61
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000487a8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027558
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003fb05
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b9d0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001beb2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047ee7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ee97
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f244
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e1bc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047de4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025519
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018aef
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c363
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a439
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041b43
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a917
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024c58
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018135
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a6e9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039482
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d7f8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e7b7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047808
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022638
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004abcb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024673
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021c75
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003cc24
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001def7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f758
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024b55
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b2dc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c260
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000458c7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020dd8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d21c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c877
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021793
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ed94
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041f28
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ddf4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000283fb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f141
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018031
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000026b9a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020cd5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046f48
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039d43
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000301fc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036e61
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b58f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001bdb9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b8d7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003937f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c95c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035ec1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049e29
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002793d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046e45
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003408e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003eb65
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047ceb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000189f6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044f04
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000428ec
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003649d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000365a1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000026a97
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a33c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000486af
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e0c3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003fa0b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d6ff
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045007
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035601
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025420
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002253f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021b7b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002457a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002745b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003bd82
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044a25
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004aad2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000019a90
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a81e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003cb2a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041a46
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000040ba5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024a5c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000038ea0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021699
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b016
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d912
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044069
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004770f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000300fe
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b1e2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000265b8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a5f0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f65e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046966
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003bc85
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000457ce
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e2a5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e6be
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ec9a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041186
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c166
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049d26
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029a7c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020bdb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c859
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028301
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c77d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001dcfa
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017b50
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000311ab
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f048
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001bcbf
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d123
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000354fe
     0.01%     0.01%  ib_send_bw       [vdso]                      [.] 0x00000000000007b3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017f38
     0.01%     0.00%  ib_send_bw       [vdso]                      [.] 0x00007fff709297b3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043f64
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027844
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000306db
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041e2f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001aa31
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001af13
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039c49
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039285
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f83e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000335bd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000363a4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e1a2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036d68
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000207f3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000023cba
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000330de
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b496
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000336c0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000307de
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000033f80
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035dc8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046d4b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a243
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000188fd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b7de
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003109f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003dcc3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c37b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c729
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029979
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000427f3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ea6c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041083
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047bf1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037f09
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b3c4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000236d8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049848
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d606
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002699e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a725
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f912
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044928
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000215a0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022446
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035020
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000019997
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021a82
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044e0a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024481
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000305e1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003bb8b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025327
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000023bb7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027362
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002dfca
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004194d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000264bc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024963
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a4f6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000485b5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000030005
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a9d8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f565
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043a86
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000040aa7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ca31
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000038da4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001eba1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b0e9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047615
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000362aa
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d819
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004686a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000456d4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e5c5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049c2d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f73b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001bbc6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e0a8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c760
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b2c1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ef4e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e973
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001dc01
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002774a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003918c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ae1a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017a57
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a938
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f25e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020ae2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028208
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c684
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000235db
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017e3f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043e6b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041d36
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c06d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039b50
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035404
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000033e87
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a8df
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004482f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d02a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035cce
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001bacd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000268a4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018803
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b6e4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032fe1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003261d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002949b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a149
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b39d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000334c3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027269
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000361b1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004974b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000214a6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000206fa
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000426fa
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032720
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000030fa6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001989e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ba92
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046c52
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021989
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047af8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036c6f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e4cc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044d11
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039093
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003dbc7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c27e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000040f8a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027651
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000304e8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a62c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041c3c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002486a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029880
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d50d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c62c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f819
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002234d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000360b7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024388
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000023abe
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000263c3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f642
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049b33
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f46c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037e04
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041854
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001eaa8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ee55
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034f23
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ed5c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ff0c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002522e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ded1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002aff0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000479ff
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a532
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b1c8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000038cab
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003dfaf
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ade4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002188f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d413
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004751c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000484bc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c938
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a3fd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d720
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000409ae
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043989
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e9af
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ad21
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000267ab
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046770
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000333ca
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001795e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000033d8e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c667
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001db08
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032140
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f160
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017d46
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000455db
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a83f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b5eb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000209e9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000234e2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e879
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024771
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003bf74
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000038f99
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002cf31
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c58b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002810f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035bd5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b2a4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044736
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032524
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000190d8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000030eac
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000239c5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044c18
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043d72
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049652
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002716f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039a57
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032ee8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003530b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020601
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001870a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046b59
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002939e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017c4d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000213ad
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001da0f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000303ef
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029787
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] strlen
     0.01%     0.00%  ib_send_bw       libc-2.27.so                [.] read
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __x64_sys_read
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] ksys_read
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] vfs_read
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __vfs_read
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] proc_reg_read
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] seq_read
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] show_cpuinfo
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000197a4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000266b2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b999
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c185
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000040e91
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003dace
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002a050
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042601
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b0cf
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003deb6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000208f0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036b76
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e3d2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000262ca
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002fe13
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f549
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046a60
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044b1f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003be7b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002428f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f720
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049a3a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ddd8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037d0b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004175b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000302f6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022254
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f373
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034e2a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002aef7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018fd5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ac28
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c533
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000483c3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025135
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d627
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ace7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000038bb2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c83f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f067
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000408b5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c56e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000033c95
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a304
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c822
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000332d1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043890
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017865
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046677
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047423
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032043
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028016
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000233e9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000238cb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ce38
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a746
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ab2f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ddbd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000454e3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b4f2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027076
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000030db3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c492
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049941
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018611
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043c79
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e780
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032def
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000292a6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003242b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003995e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004463d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020508
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035213
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000035add
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b1ab
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049559
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000331d8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002968e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000212b4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029f58
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d9d5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000237d2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000196ab
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003afd6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018edc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000040d98
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c475
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002fd1a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b8a0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041662
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042508
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002c08c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f450
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f627
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003511a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000040c9f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000038ab9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000261d1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036a7d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043b80
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002503c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024196
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034d31
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000482ca
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037c12
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002dcdf
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002adfe
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e2da
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ef6e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003abee
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002215b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000033b9c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029595
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f27a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c746
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000043797
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c43b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003aedd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000407bc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d52e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a20c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f357
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000384e6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004732b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a64d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027f1d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039865
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001776c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c399
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004657f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000232f0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000030cba
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032cf6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b3f9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031f4a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018518
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002cd3f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000026f7d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000453ea
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044544
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020410
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000195b3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032332
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000291ad
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049460
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002fc21
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000211bc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e688
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000359e4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d8dc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b7a7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000401ea
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b0b2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029e5f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022d1e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000430ce
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018de3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002bf94
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041569
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004bd71
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028ae3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037b19
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000389c0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000048e8e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036984
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f52e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004240f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002409d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ee75
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000481d2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000383e4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025bff
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024f43
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003aaf5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034c38
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ad05
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000033aa3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022062
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003223a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f181
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002dbe6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000260d8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b9c1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001cf5c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e1e1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c2a0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001719a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003976c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045fad
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a07c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c64e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027e24
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004369f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000406c4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047232
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d435
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c342
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a113
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000026e85
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001fe3e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000231f7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046486
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002cc47
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000030bc2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017673
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031e51
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a555
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000194ba
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032bfd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d30a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000400e8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b301
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018420
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000210c3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000452f1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a42c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e8a3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020317
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004444b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b6ae
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022c1c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037548
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d7e3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049367
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000290b4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000358eb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004afba
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042fc9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e58f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002fb28
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002be9b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004bc6d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000289de
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000048d8c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018cea
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a9fc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042317
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029d66
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] skb_put
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025afd
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004aec1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018bf2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000339aa
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f436
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041471
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b8bf
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003688c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000480d9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021f6a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ce5a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002daed
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000023fa5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ed7d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000388c7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024e4b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ac0d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f089
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037a21
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045eab
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004221e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000382ec
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c1a8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039674
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001fd3c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027c33
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_disable_cb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002d9f5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ab14
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034b40
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001e0e9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000027d2c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000019f7a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d208
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047fe1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000021e71
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004713a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003f33d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025fe0
     0.01%     0.01%  ib_send_bw       [rdma_rxe]                  [k] rxe_find_route
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001c0af
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000026d8c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ef90
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020fca
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000018327
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000030ac9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c555
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000024d52
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002cb4e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004a01b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a327
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001dff0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000435a6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000405cb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003178a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e497
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e7a1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001822e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c24a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000230ff
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d33d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031d59
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004638e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028fbc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003957b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000451f9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037446
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001757b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b208
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032b05
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000193c1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020ed2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000357f3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003c45d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000047041
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000026c94
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a45c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034668
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002fa30
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049f22
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ca56
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a904
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b5b6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022b23
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002021e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003fff0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d6eb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004926f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002bda2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001b110
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044353
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003e39e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000338b2
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000356fa
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045101
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042ed1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029c6e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041378
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000048c93
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000036793
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004bb74
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000288e6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002f937
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b7c7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000029b75
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003669b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003b4be
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ec84
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000023eac
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037929
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025a05
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001cd62
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000337b9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000041280
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] sg_init_table
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000387cf
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] native_flush_tlb_one_user
     0.01%     0.00%  ib_send_bw       libc-2.27.so                [.] __munmap
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __x64_sys_munmap
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __vm_munmap
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __do_munmap
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] unmap_region
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] tlb_finish_mmu
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] tlb_flush_mmu
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] flush_tlb_mm_range
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] flush_tlb_func_common.constprop.10
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000381f3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001fc43
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d110
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000309d1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031684
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045db3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000023db4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000308d8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034a48
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000019e82
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034566
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] credit_entropy_bits
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000404d3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025ee8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e6a9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000023007
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a22f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c152
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000434ae
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003734d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031c61
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000192c9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d245
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028ec3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032a0d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000046296
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] vmap
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017483
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a364
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004425b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000020126
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d5f3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000049177
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a80c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022a2b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002bcaa
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003fef8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042dd9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000191d1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004ba7c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000287ee
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000048b9b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000044163
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b6cf
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002eb8c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037830
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000380fb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000386d7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001cc6a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002590d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d018
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001fb4c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003158c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000038003
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045cbb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034950
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_poll
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e5b1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022f0f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000019d8a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000403db
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003446e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000032915
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a137
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025df0
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000433b6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004c05a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004619e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d14d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000385df
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031b69
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037256
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a26c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001738b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028dcb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022934
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004907f
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] kmem_cache_alloc_node
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003281d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002002f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d4fb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002bbb3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003fe00
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002283c
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] free_pcp_prepare
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a714
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042ce1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003fd08
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000048aa4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000402e3
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000022e17
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000286f6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b5d7
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b985
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037738
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000048f87
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000042be9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000489ac
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025cf8
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002ea94
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ca7b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000285fe
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004b88d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000025815
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002b4df
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001cb73
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001d055
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000460a6
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001a174
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002571e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000017293
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002babb
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001f95c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003cf20
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001fa54
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003d403
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000001ff37
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003ce28
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e4b9
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000019c92
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034858
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031494
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045bc4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e3c1
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000019b9a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000002e99c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a03f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000045acc
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037641
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000039f48
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000432bf
     0.01%     0.01%  ib_send_bw       [ib_uverbs]                 [k] lookup_get_idr_uobject
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004bf62
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034376
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003715e
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031a71
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028cd4
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000037067
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] update_process_times
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] scheduler_tick
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] perf_event_task_tick
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] x86_pmu_disable
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] intel_pmu_disable_all
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000004be6b
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a61c
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034760
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000431c7
     0.01%     0.01%  ib_send_bw       [rdma_rxe]                  [k] rxe_pool_get_index
     0.01%     0.00%  ib_send_bw       [rdma_rxe]                  [k] lookup_mem
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000028bdc
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] copy_user_enhanced_fast_string
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] dst_release
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003139d
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003a525
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x00000000000312a5
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003427f
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x000000000003197a
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000034188
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] ksoftirqd_running
     0.01%     0.00%  ib_send_bw       [unknown]                   [.] 0x0000000000031883
     0.01%     0.01%  ib_send_bw       [ib_core]                   [k] rdma_put_gid_attr
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] __irq_complete_move
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] __cgroup_bpf_run_filter_skb
     0.01%     0.00%  ib_send_bw       [ib_core]                   [k] rdma_read_gid_attr_ndev_rcu
     0.01%     0.01%  ksoftirqd/3      [kernel.vmlinux]            [k] detach_buf_split
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] ret_from_fork
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] kthread
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] smpboot_thread_fn
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] run_ksoftirqd
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] __do_softirq
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] net_rx_action
     0.01%     0.00%  ksoftirqd/3      [virtio_net]                [k] virtnet_poll_tx
     0.01%     0.00%  ksoftirqd/3      [virtio_net]                [k] free_old_xmit_skbs
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] virtqueue_get_buf
     0.01%     0.00%  ksoftirqd/3      [kernel.vmlinux]            [k] virtqueue_get_buf_ctx
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] tick_program_event
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] clockevents_program_event
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] apic_chip_data.part.18
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_enable_cb_prepare
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] __ip_local_out
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] passthru_features_check
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] ext4_da_write_end
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] detach_buf_split
     0.01%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_atomic_helper_dirtyfb
     0.01%     0.00%  ksoftirqd/2      [rdma_rxe]                  [k] rxe_init_packet
     0.01%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] ret_from_fork
     0.01%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] kthread
     0.01%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] worker_thread
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] run_posix_cpu_timers
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] mmput
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] exit_mmap
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] unmap_vmas
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] unmap_single_vma
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] unmap_page_range
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] irq_entries_start
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] __do_execve_file.isra.38
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] apparmor_file_free_security
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] task_work_run
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] ____fput
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __fput
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] tick_nohz_next_event
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] get_next_timer_interrupt
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] skb_release_head_state
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] handle_edge_irq
     0.01%     0.01%  ib_send_bw       [crc32_pclmul]              [k] less_64
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] fpregs_assert_state_consistent
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] __raise_softirq_irqoff
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] crc32_le_base
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] rcu_irq_exit
     0.01%     0.01%  ib_send_bw       [rdma_rxe]                  [k] next_opcode.isra.17
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] _mix_pool_bytes
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __mix_pool_bytes
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] free_unref_page_prepare.part.73
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] free_unref_page
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __ip_local_out
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] kmem_cache_free
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] kfree_skbmem
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] __fget_light
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __fdget_pos
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_add
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] virtqueue_add_outbuf
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] vunmap
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __vunmap
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] remove_vm_area
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] syscall_return_via_sysret
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] irq_may_run
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] common_file_perm
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] rw_verify_area
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] security_file_permission
     0.01%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] apparmor_file_permission
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] _copy_from_user
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] async_page_fault
     0.01%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] process_one_work
     0.01%     0.00%  irqbalance       libc-2.27.so                [.] read
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] entry_SYSCALL_64
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] do_syscall_64
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] __x64_sys_read
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] ksys_read
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] vfs_read
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] __vfs_read
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] proc_reg_read
     0.01%     0.00%  irqbalance       [kernel.vmlinux]            [k] seq_read
     0.01%     0.01%  swapper          [crc32_pclmul]              [k] loop_16
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] search_binary_handler
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] load_elf_binary
     0.01%     0.00%  sleep            [kernel.vmlinux]            [k] entry_SYSCALL_64
     0.01%     0.00%  sleep            [kernel.vmlinux]            [k] do_syscall_64
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] tick_nohz_idle_exit
     0.01%     0.01%  ib_send_bw       [kernel.vmlinux]            [k] netif_skb_features
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] generic_write_end
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] __mark_inode_dirty
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] ext4_dirty_inode
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __get_vm_area_node
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] __remove_hrtimer
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] rebalance_domains
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] update_load_avg
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] native_apic_msr_eoi_write
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] kvm_guest_apic_eoi_write
     0.01%     0.01%  kworker/1:1-eve  [kernel.vmlinux]            [k] page_mapping
     0.01%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_atomic_commit
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] ktime_get
     0.01%     0.00%  kworker/3:1-mm_  [drm_kms_helper]            [k] drm_fb_helper_dirty_work
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] __tick_nohz_idle_restart_tick
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] flush_old_exec
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] mmput
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] exit_mmap
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] tick_do_update_jiffies64.part.14
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] check_move_unevictable_pages
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] disk_events_workfn
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] disk_check_events
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] sr_block_check_events
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] cdrom_check_events
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] sr_check_events
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __scsi_execute
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] kmem_cache_free
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_start_range_ns
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] kvm_sched_clock_read
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] update_rq_clock
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] sched_clock_cpu
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] tick_sched_timer
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] tick_sched_handle
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] _nohz_idle_balance
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] update_nohz_stats
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] schedule
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __schedule
     0.01%     0.01%  kworker/1:1-eve  [kernel.vmlinux]            [k] vunmap_page_range
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] process_timeout
     0.01%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] bit_cursor
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] tick_program_event
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] clockevents_program_event
     0.01%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] alloc_vmap_area
     0.01%     0.01%  perf             [kernel.vmlinux]            [k] copy_user_enhanced_fast_string
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] iov_iter_copy_from_user_atomic
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] irq_chip_ack_parent
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] apic_ack_edge
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] apic_ack_irq
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] rcu_dynticks_eqs_exit
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] rcu_idle_exit
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] quiet_vmstat
     0.01%     0.00%  swapper          [kernel.vmlinux]            [k] need_update
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] siphash_3u32
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] virtqueue_enable_cb_delayed
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] vring_map_one_sg
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] kfree_skbmem
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] ext4_block_write_begin
     0.01%     0.00%  perf             [kernel.vmlinux]            [k] create_empty_buffers
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] virtqueue_add
     0.01%     0.01%  swapper          [rdma_rxe]                  [k] rxe_get_av
     0.01%     0.01%  swapper          [kernel.vmlinux]            [k] ipv4_dst_check
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __schedule
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] schedule_idle
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] kvm_clock_get_cycles
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] native_write_msr
     0.00%     0.00%  swapper          [rdma_rxe]                  [k] lookup_iova
     0.00%     0.00%  ksoftirqd/2      [ib_core]                   [k] rdma_put_gid_attr
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] load_balance
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] iowrite16
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] vp_notify
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __skb_to_sgvec
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] skb_to_sgvec
     0.00%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] __ip_local_out
     0.00%     0.00%  ksoftirqd/2      [rdma_rxe]                  [k] rxe_send
     0.00%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] ip_local_out
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] schedule
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] sg_next
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] ipv4_mtu
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] sched_clock
     0.00%     0.00%  bash             bash                        [.] 0x0000000000056c20
     0.00%     0.00%  bash             bash                        [.] 0x0000561dcf890c20
     0.00%     0.00%  swapper          [crc32_pclmul]              [k] fold_64
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] skb_put
     0.00%     0.00%  ksoftirqd/2      [rdma_rxe]                  [k] rxe_mem_copy
     0.00%     0.00%  ksoftirqd/2      [rdma_rxe]                  [k] copy_data
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] fb_flashcursor
     0.00%     0.00%  kworker/u8:0-ev  [drm_kms_helper]            [k] drm_fb_helper_sys_imageblit
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] irq_fpu_usable
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] psi_task_change
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] do_softirq.part.20
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] do_softirq_own_stack
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] wake_up_process
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] try_to_wake_up
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] ttwu_do_activate
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] activate_task
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] enqueue_task
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_atomic_helper_commit
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] validate_xmit_xfrm
     0.00%     0.00%  swapper          [ib_core]                   [k] ib_device_put
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] pick_next_task_fair
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] newidle_balance
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] update_process_times
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] ttwu_do_activate
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] activate_task
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] enqueue_task
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __queue_work
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] insert_work
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] show_interrupts
     0.00%     0.00%  sshd             [unknown]                   [k] 0000000000000000
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] entry_SYSCALL_64
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] do_syscall_64
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] find_first_bit
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] interrupt_entry
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] queue_work_on
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] cursor_timer_handler
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_execute_rq
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_execute_rq_nowait
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_mq_sched_insert_request
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_mq_run_hw_queue
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __blk_mq_delay_run_hw_queue
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __blk_mq_run_hw_queue
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_mq_sched_dispatch_requests
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_mq_do_dispatch_sched
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_mq_dispatch_rq_list
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] scsi_queue_rq
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_scsi_queuecmd
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_scsi_translate
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_qc_issue
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_bmdma_qc_issue
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_sff_qc_issue
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] unlock_page
     0.00%     0.00%  swapper          [crc32_pclmul]              [k] crc32_pclmul_le_16
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] find_next_bit
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] lapic_next_deadline
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] update_sd_lb_stats
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] find_busiest_group
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __schedule
     0.00%     0.00%  perf             perf                        [.] perf_evlist__prepare_workload
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_cancel
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] crc32c_pcl_intel_update
     0.00%     0.00%  perf             [unknown]                   [k] 0xffffffffc0463068
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] ext4_mark_inode_dirty
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] ext4_mark_iloc_dirty
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] ext4_inode_csum_set
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] ext4_inode_csum.isra.62
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] find_lock_entry
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_client_buffer_vmap
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_vmap
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_shmem_vmap
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_shmem_get_pages
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_get_pages
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] shmem_read_mapping_page_gfp
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] shmem_getpage_gfp
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] memcpy_toio
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] commit_tail
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_atomic_helper_commit_tail
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_atomic_helper_commit_planes
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_simple_kms_plane_atomic_update
     0.00%     0.00%  kworker/1:1-eve  [cirrus]                    [k] cirrus_pipe_update
     0.00%     0.00%  kworker/1:1-eve  [cirrus]                    [k] cirrus_fb_blit_rect
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __ext4_journal_start_sb
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] rcu_core_si
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] update_blocked_averages
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] 0xffffffff97e9f348
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] do_group_exit
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] do_exit
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] mmput
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] exit_mmap
     0.00%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] memset_erms
     0.00%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] __alloc_skb
     0.00%     0.00%  ksoftirqd/2      [kernel.vmlinux]            [k] __kmalloc_reserve.isra.61
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] soft_cursor
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __raw_callee_save___pv_queued_spin_unlock
     0.00%     0.00%  perf             [unknown]                   [k] 0x00007db7e8c73827
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] load_balance
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] find_busiest_group
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] update_sd_lb_stats
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_force_reprogram
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] num_to_str
     0.00%     0.00%  irqbalance       [unknown]                   [k] 0x3220202020202020
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] show_stat
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] seq_put_decimal_ull
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] seq_put_decimal_ull_width
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] enqueue_task_fair
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] usb_hcd_poll_rh_status
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] rh_timer_func
     0.00%     0.00%  ib_send_bw       [ib_uverbs]                 [k] uverbs_response
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] _raw_spin_unlock_irqrestore
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] delay_tsc
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __const_udelay
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] timekeeping_advance
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] iov_iter_fault_in_readable
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] rb_erase
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] psi_task_change
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] ret_from_fork
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] kthread
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] smpboot_thread_fn
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] schedule
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] __schedule
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] deactivate_task
     0.00%     0.00%  migration/0      [kernel.vmlinux]            [k] dequeue_task
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] xas_load
     0.00%     0.00%  sshd             libc-2.27.so                [.] __GI___libc_write
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] __x64_sys_write
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] ksys_write
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] vfs_write
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] __vfs_write
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] new_sync_write
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] sock_write_iter
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] sock_sendmsg
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] inet_sendmsg
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] tcp_sendmsg
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] tcp_sendmsg_locked
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] tcp_push
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] __tcp_push_pending_frames
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] tcp_write_xmit
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] __tcp_transmit_skb
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] unmap_page_range
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] unmap_vmas
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] unmap_single_vma
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_fb_memcpy_dstclip
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] do_async_page_fault
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_client_buffer_vunmap
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_vunmap
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_shmem_vunmap
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] insert_vmap_area.constprop.51
     0.00%     0.00%  irqbalance       [unknown]                   [k] 0x2030555043202020
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __queue_work
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] insert_work
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] wake_up_process
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] try_to_wake_up
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] update_wall_time
     0.00%     0.00%  kworker/u8:0-ev  [drm_kms_helper]            [k] drm_fb_helper_dirty.isra.13
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_try_to_cancel
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] timerqueue_del
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] update_nohz_stats
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] mem_cgroup_page_lruvec
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __purge_vmap_area_lazy
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] free_vmap_area_noflush
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __next_timer_interrupt
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] _raw_spin_lock
     0.00%     0.00%  sleep            [unknown]                   [k] 0x0000000000000040
     0.00%     0.00%  sleep            ld-2.27.so                  [.] _dl_sysdep_start
     0.00%     0.00%  sleep            ld-2.27.so                  [.] dl_main
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] async_page_fault
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] do_async_page_fault
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] do_page_fault
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] __do_page_fault
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] handle_mm_fault
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] __handle_mm_fault
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] PageHuge
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] error_entry
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] enqueue_entity
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] __d_lookup_rcu
     0.00%     0.00%  sleep            [unknown]                   [k] 0x6f74207367756220
     0.00%     0.00%  sleep            sleep                       [.] 0x000055d220bebc21
     0.00%     0.00%  sleep            libc-2.27.so                [.] setlocale
     0.00%     0.00%  sleep            libc-2.27.so                [.] _nl_find_locale
     0.00%     0.00%  sleep            libc-2.27.so                [.] __GI___open64_nocancel
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] __x64_sys_openat
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] do_sys_open
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] do_filp_open
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] path_openat
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] lookup_fast
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_dev_select.constprop.22
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] pick_next_task_fair
     0.00%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_atomic_check_only
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] blk_account_io_start
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] ret_from_fork
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] kthread
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] kjournald2
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] jbd2_journal_commit_transaction
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] journal_submit_commit_record.part.22
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] submit_bh
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] submit_bh_wbc
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] submit_bio
     0.00%     0.00%  jbd2/vda1-8      [kernel.vmlinux]            [k] generic_make_request
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] idle_cpu
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] perf_event_mmap
     0.00%     0.00%  sleep            libc-2.27.so                [.] __GI___execve
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] __x64_sys_execve
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] __do_execve_file.isra.38
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] search_binary_handler
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] load_elf_binary
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] vm_brk_flags
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] do_brk_flags
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] rcu_core
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] rcu_report_qs_rnp
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] kvmalloc_node
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] rcu_sched_clock_irq
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] sched_ttwu_pending
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] kmem_cache_free
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] remove_vma
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] vm_area_free
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] kmem_cache_alloc
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] alloc_page_buffers
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] alloc_buffer_head
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] _cond_resched
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] tlb_finish_mmu
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] tlb_flush_mmu
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] irq_work_needs_cpu
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] __radix_tree_lookup
     0.00%     0.00%  irqbalance       [unknown]                   [k] 0x3020302030203020
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] irq_to_desc
     0.00%     0.00%  irqbalance       [kernel.vmlinux]            [k] radix_tree_lookup
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] _raw_spin_unlock_irqrestore
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __ext4_journal_stop
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] jbd2_journal_stop
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __wake_up
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __wake_up_common_lock
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] irq_work_tick
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] pick_next_task_fair
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] unmap_page_range
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] unmap_vmas
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] unmap_single_vma
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] down_write
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] free_pgtables
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] unlink_file_vma
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] update_curr
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ttwu_do_activate
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] activate_task
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] enqueue_task
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] enqueue_task_fair
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] enqueue_entity
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_reprogram
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_wakeup
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] file_free_rcu
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] kmem_cache_alloc_node_trace
     0.00%     0.00%  perf             libc-2.27.so                [.] __poll
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __x64_sys_poll
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] do_sys_poll
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] handle_irq_event
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] ktime_get_update_offsets_now
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] page_remove_rmap
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] queue_work_on
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] skb_clone
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] tsc_verify_tsc_adjust
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] check_move_unevictable_pages
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_shmem_put_pages
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_shmem_put_pages_locked
     0.00%     0.00%  kworker/3:1-mm_  [drm]                       [k] drm_gem_put_pages
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] lock_hrtimer_base
     0.00%     0.00%  sshd             [sch_fq_codel]              [k] fq_codel_enqueue
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] ip_queue_xmit
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] core_sys_select
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] __ip_queue_xmit
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] ip_local_out
     0.00%     0.00%  sshd             [unknown]                   [k] 0x000055da430880b0
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] ip_output
     0.00%     0.00%  sshd             libc-2.27.so                [.] __select
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] ip_finish_output
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] __ip_finish_output
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] __x64_sys_select
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] ip_finish_output2
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] dev_queue_xmit
     0.00%     0.00%  sshd             [kernel.vmlinux]            [k] kern_select
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] account_process_tick
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] do_page_fault
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __do_page_fault
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] handle_mm_fault
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __handle_mm_fault
     0.00%     0.00%  perf             [unknown]                   [k] 0x00007db7be93fd20
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] put_task_stack
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] finish_task_switch
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] _raw_spin_lock_irq
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] memchr_inv
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] psi_task_change
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] deactivate_task
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] dequeue_task
     0.00%     0.00%  irqbalance       irqbalance                  [.] 0x000000000000757f
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] nr_iowait_cpu
     0.00%     0.00%  irqbalance       [unknown]                   [.] 0x0000000000000002
     0.00%     0.00%  irqbalance       irqbalance                  [.] 0x000055c4f0e1e57f
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] tick_check_oneshot_broadcast_this_cpu
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] sched_clock_cpu
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __update_load_avg_cfs_rq
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] hrtimer_get_next_event
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] vunmap_page_range
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] vunmap
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] __vunmap
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] remove_vm_area
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] kfree
     0.00%     0.00%  kworker/1:1-eve  [drm]                       [k] __drm_atomic_state_free
     0.00%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_atomic_state_default_release
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] update_fast_timekeeper
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] timekeeping_update
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] tick_nohz_idle_enter
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] enqueue_hrtimer
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] may_open
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] open_exec
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] do_open_execat
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] do_filp_open
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] path_openat
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] update_curr
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] ret_from_fork
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] kthread
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] worker_thread
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] schedule
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] __schedule
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] deactivate_task
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] dequeue_task
     0.00%     0.00%  kworker/2:0-mm_  [kernel.vmlinux]            [k] dequeue_task_fair
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] find_next_bit
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] vmstat_update
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] refresh_cpu_vm_stats
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] next_zone
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __switch_to
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] psi_task_change
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] vmap_page_range_noflush
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] map_vm_area
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] free_unref_page_commit
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] free_pages_and_swap_cache
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] release_pages
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] free_unref_page_list
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] free_unref_page_list
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_sff_tf_load
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] ww_mutex_lock
     0.00%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_atomic_get_plane_state
     0.00%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_modeset_lock
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] lru_add_drain_cpu
     0.00%     0.00%  gmain            libglib-2.0.so.0.5600.4     [.] g_source_set_ready_time
     0.00%     0.00%  gmain            libgio-2.0.so.0.5600.4      [.] 0x00007f20c7eefc60
     0.00%     0.00%  gmain            [unknown]                   [.] 0x00007f20c7b84280
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __bitmap_and
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] delayed_work_timer_fn
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] x86_pmu_enable
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] intel_pmu_enable_all
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __intel_pmu_enable_all.constprop.29
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] memcpy_erms
     0.00%     0.00%  kworker/u8:0-ev  [sysimgblt]                 [k] sys_imageblit
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] xas_start
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] security_vm_enough_memory_mm
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] insert_vm_struct
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] xas_start
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] kthread_data
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] find_get_entry
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] schedule
     0.00%     0.00%  kworker/3:1-mm_  [kernel.vmlinux]            [k] xas_load
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] wq_worker_running
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __update_load_avg_cfs_rq
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] cpumask_next_and
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] page_frag_alloc
     0.00%     0.00%  swapper          [virtio_net]                [k] page_to_skb
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __napi_alloc_skb
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] touch_softlockup_watchdog_sched
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] apic_timer_interrupt
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] uhci_hub_status_data
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] get_nohz_timer_target
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] mod_timer
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] kmem_cache_alloc_trace
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_mq_free_request
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_put_request
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] cpuacct_charge
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] get_page_from_freelist
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] ata_sff_queue_pio_task
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] queue_delayed_work_on
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __queue_delayed_work
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] blk_rq_map_kern
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] bio_copy_kern
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] alloc_pages_current
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] __alloc_pages_nodemask
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] kthread_data
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] generic_exec_single
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __kmalloc_node
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] __raw_callee_save___pv_queued_spin_unlock
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] vmalloc_sync_all
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] sync_global_pgds
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] sync_global_pgds_l4
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] kmalloc_slab
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] set_next_entity
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] _raw_spin_lock_irqsave
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] delay_tsc
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] ret_from_fork
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] kthread
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] worker_thread
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] process_one_work
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] disk_events_workfn
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] disk_check_events
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] sr_block_check_events
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] cdrom_check_events
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] sr_check_events
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] __scsi_execute
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] blk_execute_rq
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] blk_execute_rq_nowait
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] blk_mq_sched_insert_request
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] blk_mq_run_hw_queue
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] __blk_mq_delay_run_hw_queue
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] __blk_mq_run_hw_queue
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] blk_mq_sched_dispatch_requests
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] blk_mq_do_dispatch_sched
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] blk_mq_dispatch_rq_list
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] scsi_queue_rq
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] ata_scsi_queuecmd
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] ata_scsi_translate
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] ata_qc_issue
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] ata_bmdma_qc_issue
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] ata_sff_qc_issue
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] ata_dev_select.constprop.22
     0.00%     0.00%  kworker/u8:2-ev  [kernel.vmlinux]            [k] __const_udelay
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] scheduler_tick
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] perf_event_task_tick
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] smp_apic_timer_interrupt
     0.00%     0.00%  kworker/1:1-eve  [kernel.vmlinux]            [k] setup_vmalloc_vm
     0.00%     0.00%  gmain            [kernel.vmlinux]            [k] __x64_sys_poll
     0.00%     0.00%  gmain            libc-2.27.so                [.] __poll
     0.00%     0.00%  gmain            [kernel.vmlinux]            [k] entry_SYSCALL_64
     0.00%     0.00%  gmain            [kernel.vmlinux]            [k] do_syscall_64
     0.00%     0.00%  kworker/1:1-eve  [drm]                       [k] drm_rect_clip_scaled
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_atomic_helper_check
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_atomic_helper_check_planes
     0.00%     0.00%  kworker/1:1-eve  [drm_kms_helper]            [k] drm_simple_kms_plane_atomic_check
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] add_interrupt_randomness
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] _raw_spin_unlock_irqrestore
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] dequeue_task_fair
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] deactivate_task
     0.00%     0.00%  kworker/u8:0-ev  [kernel.vmlinux]            [k] dequeue_task
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] clear_buddies
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] pick_next_entity
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] vm_normal_page
     0.00%     0.00%  khungtaskd       [kernel.vmlinux]            [k] watchdog
     0.00%     0.00%  khungtaskd       [kernel.vmlinux]            [k] ret_from_fork
     0.00%     0.00%  khungtaskd       [kernel.vmlinux]            [k] kthread
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] _raw_spin_lock
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] account_entity_enqueue
     0.00%     0.00%  perf             libc-2.27.so                [.] __restore_rt
     0.00%     0.00%  perf             perf                        [.] sig_handler
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] flush_tlb_func_common.constprop.10
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] ldt_arch_exit_mmap
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] free_ldt_pgtables.part.3
     0.00%     0.00%  sleep            [kernel.vmlinux]            [k] flush_tlb_mm_range
     0.00%     0.00%  swapper          [virtio_net]                [k] skb_recv_done
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] handle_irq_event_percpu
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __handle_irq_event_percpu
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] vring_interrupt
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __x86_indirect_thunk_r14
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __hrtimer_next_event_base
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __hrtimer_get_next_event
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __hrtimer_run_queues
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] hrtimer_interrupt
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] timerqueue_add
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] filemap_map_pages
     0.00%     0.00%  perf             [unknown]                   [k] 0x786562696c2f746f
     0.00%     0.00%  perf             [unknown]                   [k] 0x00007ffdcb81bf9e
     0.00%     0.00%  perf             libc-2.27.so                [.] __strncmp_sse42
     0.00%     0.00%  sleep            sleep                       [.] 0x0000000000001c41
     0.00%     0.00%  sleep            libc-2.27.so                [.] __run_exit_handlers
     0.00%     0.00%  sleep            sleep                       [.] 0x000055d220be7c41
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] mutex_unlock
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] rq_unlock
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] pvclock_clocksource_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] poll_schedule_timeout.constprop.11
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] schedule_hrtimeout_range
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] schedule_hrtimeout_range_clock
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] schedule
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __schedule
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] deactivate_task
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] dequeue_task
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] psi_task_change
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] record_times
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] sched_clock_cpu
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] sched_clock
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] kvm_sched_clock_read
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] call_function_single_interrupt
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __fget
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __fdget
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __fget_light
     0.00%     0.00%  sleep            sleep                       [.] 0x0000000000004a67
     0.00%     0.00%  sleep            sleep                       [.] 0x000055d220beaa67
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] sync_regs
     0.00%     0.00%  perf             libc-2.27.so                [.] __strchr_avx2
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] generic_smp_call_function_single_interrupt
     0.00%     0.00%  perf             libc-2.27.so                [.] __execvpe
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] ktime_get_update_offsets_now
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] ctx_resched
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] smp_call_function_single_interrupt
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] flush_smp_call_function_queue
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] remote_function
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] event_function
     0.00%     0.00%  swapper          [kernel.vmlinux]            [k] __perf_event_enable
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] mutex_lock
     0.00%     0.00%  perf             libpthread-2.27.so          [.] __libc_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __x64_sys_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] ksys_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] vfs_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __vfs_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] new_sync_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] pipe_read
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] pipe_wait
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] call_function_single_interrupt
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] smp_call_function_single_interrupt
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] event_function
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] generic_smp_call_function_single_interrupt
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] flush_smp_call_function_queue
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] remote_function
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] native_write_msr
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] remote_function
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] event_function
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __perf_event_enable
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] ctx_resched
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] x86_pmu_enable
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] intel_pmu_enable_all
     0.00%     0.00%  perf             [kernel.vmlinux]            [k] __intel_pmu_enable_all.constprop.29
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __perf_event_enable
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] ctx_resched
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] x86_pmu_enable
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] intel_pmu_enable_all
     0.00%     0.00%  ib_send_bw       [kernel.vmlinux]            [k] __intel_pmu_enable_all.constprop.29


#
# (Tip: Sample related events with: perf record -e '{cycles,instructions}:S')
#


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-18  6:38   ` [question]Why " wangqi
@ 2019-11-18  9:49     ` Leon Romanovsky
  2019-11-18 10:13       ` wangqi
  0 siblings, 1 reply; 10+ messages in thread
From: Leon Romanovsky @ 2019-11-18  9:49 UTC (permalink / raw)
  To: wangqi; +Cc: linux-rdma

On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
>
> > On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
> >> Dear experts on RDMA,
> >>       We are sorry to disturb you. Because of a project, we need to
> >> integrate soft-RoCE in our system. However ,we are very confused by our
> >> soft-RoCE throughput results, which are quite low compared with TCP
> >> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
> >> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
> >> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
> >> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
> >> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
> >> respectively. But in the figure 6 in the attached paper "A Performance
> >> Comparison of Container Networking Alternatives", the throughput of
> >> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
> >> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
> >> we get such high bandwidth? Do we need to configure some OS system settings?
> >>       We find that in 2017, someone finds the same problem and he posts all
> >> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
> >> . But it remains unsolved. His results are nearly the same with our's. For
> >> simplicity,  we do not post our results in this email. You can get very
> >> detailed information in the web page listed above.
> >>       We are very confused by our results. We will very appreciate it if we
> >> can receive your early reply. Best wishes,
> >> Wang Qi
> > Can you please fix your email client?
> > The email text looks like one big sentence.
> >
> > From the perf report attached to this bugzilla, looks like RXE does a
> > lot of CRC32 calculations and it is consistent with what Matan said
> > a long time ago, RXE "stuck" in ICRC calculations required by spec.
> >
> > I'm curios what are your CONFIG_CRYPTO_* configs?
> >
> > ThanksCONFIG_CRYPTO_* configs
> >
> >
>
>
> I'm sorry for the editor problem in my last email. Now I use another editor.

Now your email has extra line between lines.

>
> We get our rdma-core and perftest from
>
> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
>
> We attach five files to clarify our problem.
>
> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
>
> and softroce throughput in our two servers (connected via back to back).
>
> * The second file "server_CONFIG_CRYPTO_result.txt" is the
>
> CONFIG_CRYPTO_* config results in the two servers..
>
> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
>
> & perf record -ags sleep 10 & wait" results in our two servers (we use
>
> "perf report --header >perf" to make the file).
>
> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
>
> and softroce throughput in two virtual machines with the latest linux kernel
>
> 5.4.0-rc7
>
> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
>
> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
>
> machines.
>
> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
>
> & perf record -ags sleep 10 & wait " result in the two virtual machines.
>
> On the other side, we tried to use the rxe command "rxe_cfg crc disable"

I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
and there is no such module parameter in the kernel.

Thanks

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-18  9:49     ` Leon Romanovsky
@ 2019-11-18 10:13       ` wangqi
  2019-11-18 12:28         ` Leon Romanovsky
  0 siblings, 1 reply; 10+ messages in thread
From: wangqi @ 2019-11-18 10:13 UTC (permalink / raw)
  To: Leon Romanovsky, linux-rdma


On 2019/11/18 下午5:49, Leon Romanovsky wrote:
> On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
>> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
>>
>>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
>>>> Dear experts on RDMA,
>>>>       We are sorry to disturb you. Because of a project, we need to
>>>> integrate soft-RoCE in our system. However ,we are very confused by our
>>>> soft-RoCE throughput results, which are quite low compared with TCP
>>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
>>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
>>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
>>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
>>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
>>>> respectively. But in the figure 6 in the attached paper "A Performance
>>>> Comparison of Container Networking Alternatives", the throughput of
>>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
>>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
>>>> we get such high bandwidth? Do we need to configure some OS system settings?
>>>>       We find that in 2017, someone finds the same problem and he posts all
>>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
>>>> . But it remains unsolved. His results are nearly the same with our's. For
>>>> simplicity,  we do not post our results in this email. You can get very
>>>> detailed information in the web page listed above.
>>>>       We are very confused by our results. We will very appreciate it if we
>>>> can receive your early reply. Best wishes,
>>>> Wang Qi
>>> Can you please fix your email client?
>>> The email text looks like one big sentence.
>>>
>>> From the perf report attached to this bugzilla, looks like RXE does a
>>> lot of CRC32 calculations and it is consistent with what Matan said
>>> a long time ago, RXE "stuck" in ICRC calculations required by spec.
>>>
>>> I'm curios what are your CONFIG_CRYPTO_* configs?
>>>
>>> ThanksCONFIG_CRYPTO_* configs
>>>
>>>
>>
>> I'm sorry for the editor problem in my last email. Now I use another editor.
> Now your email has extra line between lines.
>
>> We get our rdma-core and perftest from
>>
>> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
>> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
>>
>> We attach five files to clarify our problem.
>>
>> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
>>
>> and softroce throughput in our two servers (connected via back to back).
>>
>> * The second file "server_CONFIG_CRYPTO_result.txt" is the
>>
>> CONFIG_CRYPTO_* config results in the two servers..
>>
>> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
>>
>> & perf record -ags sleep 10 & wait" results in our two servers (we use
>>
>> "perf report --header >perf" to make the file).
>>
>> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
>>
>> and softroce throughput in two virtual machines with the latest linux kernel
>>
>> 5.4.0-rc7
>>
>> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
>>
>> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
>>
>> machines.
>>
>> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
>>
>> & perf record -ags sleep 10 & wait " result in the two virtual machines.
>>
>> On the other side, we tried to use the rxe command "rxe_cfg crc disable"
> I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
> and there is no such module parameter in the kernel.
>
> Thanks


We get the command "rxe_cfg crc disable" from the following webpages:

https://www.systutorials.com/docs/linux/man/8-rxe_cfg/

https://www.reflectionsofthevoid.com/2011/08/

It may be removed in the present soft-roce edition.

Can you figure out why our softroce throughput is so low from the six

files in our last email? We hope to get a much higher softroce throughput,

like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be

up to 100 Gbps in our system).

Qi



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-18 10:13       ` wangqi
@ 2019-11-18 12:28         ` Leon Romanovsky
  2019-11-18 12:56           ` wangqi
  0 siblings, 1 reply; 10+ messages in thread
From: Leon Romanovsky @ 2019-11-18 12:28 UTC (permalink / raw)
  To: wangqi; +Cc: linux-rdma

On Mon, Nov 18, 2019 at 06:13:07PM +0800, wangqi wrote:
>
> On 2019/11/18 下午5:49, Leon Romanovsky wrote:
> > On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
> >> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
> >>
> >>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
> >>>> Dear experts on RDMA,
> >>>>       We are sorry to disturb you. Because of a project, we need to
> >>>> integrate soft-RoCE in our system. However ,we are very confused by our
> >>>> soft-RoCE throughput results, which are quite low compared with TCP
> >>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
> >>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
> >>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
> >>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
> >>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
> >>>> respectively. But in the figure 6 in the attached paper "A Performance
> >>>> Comparison of Container Networking Alternatives", the throughput of
> >>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
> >>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
> >>>> we get such high bandwidth? Do we need to configure some OS system settings?
> >>>>       We find that in 2017, someone finds the same problem and he posts all
> >>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
> >>>> . But it remains unsolved. His results are nearly the same with our's. For
> >>>> simplicity,  we do not post our results in this email. You can get very
> >>>> detailed information in the web page listed above.
> >>>>       We are very confused by our results. We will very appreciate it if we
> >>>> can receive your early reply. Best wishes,
> >>>> Wang Qi
> >>> Can you please fix your email client?
> >>> The email text looks like one big sentence.
> >>>
> >>> From the perf report attached to this bugzilla, looks like RXE does a
> >>> lot of CRC32 calculations and it is consistent with what Matan said
> >>> a long time ago, RXE "stuck" in ICRC calculations required by spec.
> >>>
> >>> I'm curios what are your CONFIG_CRYPTO_* configs?
> >>>
> >>> ThanksCONFIG_CRYPTO_* configs
> >>>
> >>>
> >>
> >> I'm sorry for the editor problem in my last email. Now I use another editor.
> > Now your email has extra line between lines.
> >
> >> We get our rdma-core and perftest from
> >>
> >> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
> >> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
> >>
> >> We attach five files to clarify our problem.
> >>
> >> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
> >>
> >> and softroce throughput in our two servers (connected via back to back).
> >>
> >> * The second file "server_CONFIG_CRYPTO_result.txt" is the
> >>
> >> CONFIG_CRYPTO_* config results in the two servers..
> >>
> >> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
> >>
> >> & perf record -ags sleep 10 & wait" results in our two servers (we use
> >>
> >> "perf report --header >perf" to make the file).
> >>
> >> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
> >>
> >> and softroce throughput in two virtual machines with the latest linux kernel
> >>
> >> 5.4.0-rc7
> >>
> >> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
> >>
> >> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
> >>
> >> machines.
> >>
> >> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
> >>
> >> & perf record -ags sleep 10 & wait " result in the two virtual machines.
> >>
> >> On the other side, we tried to use the rxe command "rxe_cfg crc disable"
> > I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
> > and there is no such module parameter in the kernel.
> >
> > Thanks
>
>
> We get the command "rxe_cfg crc disable" from the following webpages:
>
> https://www.systutorials.com/docs/linux/man/8-rxe_cfg/
>
> https://www.reflectionsofthevoid.com/2011/08/
>
> It may be removed in the present soft-roce edition.

It was never existed in upstream variant and in the kernel you are testing.

>
> Can you figure out why our softroce throughput is so low from the six

According to the logs, it is ICRC.

>
> files in our last email? We hope to get a much higher softroce throughput,
>
> like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be
>
> up to 100 Gbps in our system).
>
> Qi
>
>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-18 12:28         ` Leon Romanovsky
@ 2019-11-18 12:56           ` wangqi
  2019-11-18 14:41             ` Leon Romanovsky
  0 siblings, 1 reply; 10+ messages in thread
From: wangqi @ 2019-11-18 12:56 UTC (permalink / raw)
  To: Leon Romanovsky, linux-rdma


On 2019/11/18 下午8:28, Leon Romanovsky wrote:
> On Mon, Nov 18, 2019 at 06:13:07PM +0800, wangqi wrote:
>> On 2019/11/18 下午5:49, Leon Romanovsky wrote:
>>> On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
>>>> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
>>>>
>>>>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
>>>>>> Dear experts on RDMA,
>>>>>>       We are sorry to disturb you. Because of a project, we need to
>>>>>> integrate soft-RoCE in our system. However ,we are very confused by our
>>>>>> soft-RoCE throughput results, which are quite low compared with TCP
>>>>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
>>>>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
>>>>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
>>>>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
>>>>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
>>>>>> respectively. But in the figure 6 in the attached paper "A Performance
>>>>>> Comparison of Container Networking Alternatives", the throughput of
>>>>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
>>>>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
>>>>>> we get such high bandwidth? Do we need to configure some OS system settings?
>>>>>>       We find that in 2017, someone finds the same problem and he posts all
>>>>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
>>>>>> . But it remains unsolved. His results are nearly the same with our's. For
>>>>>> simplicity,  we do not post our results in this email. You can get very
>>>>>> detailed information in the web page listed above.
>>>>>>       We are very confused by our results. We will very appreciate it if we
>>>>>> can receive your early reply. Best wishes,
>>>>>> Wang Qi
>>>>> Can you please fix your email client?
>>>>> The email text looks like one big sentence.
>>>>>
>>>>> From the perf report attached to this bugzilla, looks like RXE does a
>>>>> lot of CRC32 calculations and it is consistent with what Matan said
>>>>> a long time ago, RXE "stuck" in ICRC calculations required by spec.
>>>>>
>>>>> I'm curios what are your CONFIG_CRYPTO_* configs?
>>>>>
>>>>> ThanksCONFIG_CRYPTO_* configs
>>>>>
>>>>>
>>>> I'm sorry for the editor problem in my last email. Now I use another editor.
>>> Now your email has extra line between lines.
>>>
>>>> We get our rdma-core and perftest from
>>>>
>>>> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
>>>> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
>>>>
>>>> We attach five files to clarify our problem.
>>>>
>>>> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
>>>>
>>>> and softroce throughput in our two servers (connected via back to back).
>>>>
>>>> * The second file "server_CONFIG_CRYPTO_result.txt" is the
>>>>
>>>> CONFIG_CRYPTO_* config results in the two servers..
>>>>
>>>> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
>>>>
>>>> & perf record -ags sleep 10 & wait" results in our two servers (we use
>>>>
>>>> "perf report --header >perf" to make the file).
>>>>
>>>> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
>>>>
>>>> and softroce throughput in two virtual machines with the latest linux kernel
>>>>
>>>> 5.4.0-rc7
>>>>
>>>> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
>>>>
>>>> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
>>>>
>>>> machines.
>>>>
>>>> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
>>>>
>>>> & perf record -ags sleep 10 & wait " result in the two virtual machines.
>>>>
>>>> On the other side, we tried to use the rxe command "rxe_cfg crc disable"
>>> I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
>>> and there is no such module parameter in the kernel.
>>>
>>> Thanks
>>
>> We get the command "rxe_cfg crc disable" from the following webpages:
>>
>> https://www.systutorials.com/docs/linux/man/8-rxe_cfg/
>>
>> https://www.reflectionsofthevoid.com/2011/08/
>>
>> It may be removed in the present soft-roce edition.
> It was never existed in upstream variant and in the kernel you are testing.
>
>> Can you figure out why our softroce throughput is so low from the six
> According to the logs, it is ICRC.
>
>> files in our last email? We hope to get a much higher softroce throughput,
>>
>> like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be
>>
>> up to 100 Gbps in our system).
>>
>> Qi
>>
>>
We try to use "rxe_cfg icrc disable" and "rxe_cfg ICRC disable", but it

seems that the performance doesn't change at all.

Qi




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-18 12:56           ` wangqi
@ 2019-11-18 14:41             ` Leon Romanovsky
  2019-11-19  0:22               ` wangqi
  0 siblings, 1 reply; 10+ messages in thread
From: Leon Romanovsky @ 2019-11-18 14:41 UTC (permalink / raw)
  To: wangqi; +Cc: linux-rdma

On Mon, Nov 18, 2019 at 08:56:35PM +0800, wangqi wrote:
>
> On 2019/11/18 下午8:28, Leon Romanovsky wrote:
> > On Mon, Nov 18, 2019 at 06:13:07PM +0800, wangqi wrote:
> >> On 2019/11/18 下午5:49, Leon Romanovsky wrote:
> >>> On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
> >>>> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
> >>>>
> >>>>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
> >>>>>> Dear experts on RDMA,
> >>>>>>       We are sorry to disturb you. Because of a project, we need to
> >>>>>> integrate soft-RoCE in our system. However ,we are very confused by our
> >>>>>> soft-RoCE throughput results, which are quite low compared with TCP
> >>>>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
> >>>>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
> >>>>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
> >>>>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
> >>>>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
> >>>>>> respectively. But in the figure 6 in the attached paper "A Performance
> >>>>>> Comparison of Container Networking Alternatives", the throughput of
> >>>>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
> >>>>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
> >>>>>> we get such high bandwidth? Do we need to configure some OS system settings?
> >>>>>>       We find that in 2017, someone finds the same problem and he posts all
> >>>>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
> >>>>>> . But it remains unsolved. His results are nearly the same with our's. For
> >>>>>> simplicity,  we do not post our results in this email. You can get very
> >>>>>> detailed information in the web page listed above.
> >>>>>>       We are very confused by our results. We will very appreciate it if we
> >>>>>> can receive your early reply. Best wishes,
> >>>>>> Wang Qi
> >>>>> Can you please fix your email client?
> >>>>> The email text looks like one big sentence.
> >>>>>
> >>>>> From the perf report attached to this bugzilla, looks like RXE does a
> >>>>> lot of CRC32 calculations and it is consistent with what Matan said
> >>>>> a long time ago, RXE "stuck" in ICRC calculations required by spec.
> >>>>>
> >>>>> I'm curios what are your CONFIG_CRYPTO_* configs?
> >>>>>
> >>>>> ThanksCONFIG_CRYPTO_* configs
> >>>>>
> >>>>>
> >>>> I'm sorry for the editor problem in my last email. Now I use another editor.
> >>> Now your email has extra line between lines.
> >>>
> >>>> We get our rdma-core and perftest from
> >>>>
> >>>> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
> >>>> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
> >>>>
> >>>> We attach five files to clarify our problem.
> >>>>
> >>>> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
> >>>>
> >>>> and softroce throughput in our two servers (connected via back to back).
> >>>>
> >>>> * The second file "server_CONFIG_CRYPTO_result.txt" is the
> >>>>
> >>>> CONFIG_CRYPTO_* config results in the two servers..
> >>>>
> >>>> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
> >>>>
> >>>> & perf record -ags sleep 10 & wait" results in our two servers (we use
> >>>>
> >>>> "perf report --header >perf" to make the file).
> >>>>
> >>>> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
> >>>>
> >>>> and softroce throughput in two virtual machines with the latest linux kernel
> >>>>
> >>>> 5.4.0-rc7
> >>>>
> >>>> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
> >>>>
> >>>> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
> >>>>
> >>>> machines.
> >>>>
> >>>> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
> >>>>
> >>>> & perf record -ags sleep 10 & wait " result in the two virtual machines.
> >>>>
> >>>> On the other side, we tried to use the rxe command "rxe_cfg crc disable"
> >>> I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
> >>> and there is no such module parameter in the kernel.
> >>>
> >>> Thanks
> >>
> >> We get the command "rxe_cfg crc disable" from the following webpages:
> >>
> >> https://www.systutorials.com/docs/linux/man/8-rxe_cfg/
> >>
> >> https://www.reflectionsofthevoid.com/2011/08/
> >>
> >> It may be removed in the present soft-roce edition.
> > It was never existed in upstream variant and in the kernel you are testing.
> >
> >> Can you figure out why our softroce throughput is so low from the six
> > According to the logs, it is ICRC.
> >
> >> files in our last email? We hope to get a much higher softroce throughput,
> >>
> >> like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be
> >>
> >> up to 100 Gbps in our system).
> >>
> >> Qi
> >>
> >>
> We try to use "rxe_cfg icrc disable" and "rxe_cfg ICRC disable", but it
>
> seems that the performance doesn't change at all.

Why are you continuing to try "disable" if your kernel and rdma-core don't support it?

Thanks

>
> Qi
>
>
>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-18 14:41             ` Leon Romanovsky
@ 2019-11-19  0:22               ` wangqi
  2019-11-19  7:38                 ` Leon Romanovsky
  0 siblings, 1 reply; 10+ messages in thread
From: wangqi @ 2019-11-19  0:22 UTC (permalink / raw)
  To: Leon Romanovsky; +Cc: linux-rdma


On 2019/11/18 下午10:41, Leon Romanovsky wrote:
> On Mon, Nov 18, 2019 at 08:56:35PM +0800, wangqi wrote:
>> On 2019/11/18 下午8:28, Leon Romanovsky wrote:
>>> On Mon, Nov 18, 2019 at 06:13:07PM +0800, wangqi wrote:
>>>> On 2019/11/18 下午5:49, Leon Romanovsky wrote:
>>>>> On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
>>>>>> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
>>>>>>
>>>>>>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
>>>>>>>> Dear experts on RDMA,
>>>>>>>>       We are sorry to disturb you. Because of a project, we need to
>>>>>>>> integrate soft-RoCE in our system. However ,we are very confused by our
>>>>>>>> soft-RoCE throughput results, which are quite low compared with TCP
>>>>>>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
>>>>>>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
>>>>>>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
>>>>>>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
>>>>>>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
>>>>>>>> respectively. But in the figure 6 in the attached paper "A Performance
>>>>>>>> Comparison of Container Networking Alternatives", the throughput of
>>>>>>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
>>>>>>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
>>>>>>>> we get such high bandwidth? Do we need to configure some OS system settings?
>>>>>>>>       We find that in 2017, someone finds the same problem and he posts all
>>>>>>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
>>>>>>>> . But it remains unsolved. His results are nearly the same with our's. For
>>>>>>>> simplicity,  we do not post our results in this email. You can get very
>>>>>>>> detailed information in the web page listed above.
>>>>>>>>       We are very confused by our results. We will very appreciate it if we
>>>>>>>> can receive your early reply. Best wishes,
>>>>>>>> Wang Qi
>>>>>>> Can you please fix your email client?
>>>>>>> The email text looks like one big sentence.
>>>>>>>
>>>>>>> From the perf report attached to this bugzilla, looks like RXE does a
>>>>>>> lot of CRC32 calculations and it is consistent with what Matan said
>>>>>>> a long time ago, RXE "stuck" in ICRC calculations required by spec.
>>>>>>>
>>>>>>> I'm curios what are your CONFIG_CRYPTO_* configs?
>>>>>>>
>>>>>>> ThanksCONFIG_CRYPTO_* configs
>>>>>>>
>>>>>>>
>>>>>> I'm sorry for the editor problem in my last email. Now I use another editor.
>>>>> Now your email has extra line between lines.
>>>>>
>>>>>> We get our rdma-core and perftest from
>>>>>>
>>>>>> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
>>>>>> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
>>>>>>
>>>>>> We attach five files to clarify our problem.
>>>>>>
>>>>>> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
>>>>>>
>>>>>> and softroce throughput in our two servers (connected via back to back).
>>>>>>
>>>>>> * The second file "server_CONFIG_CRYPTO_result.txt" is the
>>>>>>
>>>>>> CONFIG_CRYPTO_* config results in the two servers..
>>>>>>
>>>>>> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
>>>>>>
>>>>>> & perf record -ags sleep 10 & wait" results in our two servers (we use
>>>>>>
>>>>>> "perf report --header >perf" to make the file).
>>>>>>
>>>>>> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
>>>>>>
>>>>>> and softroce throughput in two virtual machines with the latest linux kernel
>>>>>>
>>>>>> 5.4.0-rc7
>>>>>>
>>>>>> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
>>>>>>
>>>>>> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
>>>>>>
>>>>>> machines.
>>>>>>
>>>>>> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
>>>>>>
>>>>>> & perf record -ags sleep 10 & wait " result in the two virtual machines.
>>>>>>
>>>>>> On the other side, we tried to use the rxe command "rxe_cfg crc disable"
>>>>> I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
>>>>> and there is no such module parameter in the kernel.
>>>>>
>>>>> Thanks
>>>> We get the command "rxe_cfg crc disable" from the following webpages:
>>>>
>>>> https://www.systutorials.com/docs/linux/man/8-rxe_cfg/
>>>>
>>>> https://www.reflectionsofthevoid.com/2011/08/
>>>>
>>>> It may be removed in the present soft-roce edition.
>>> It was never existed in upstream variant and in the kernel you are testing.
>>>
>>>> Can you figure out why our softroce throughput is so low from the six
>>> According to the logs, it is ICRC.
>>>
>>>> files in our last email? We hope to get a much higher softroce throughput,
>>>>
>>>> like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be
>>>>
>>>> up to 100 Gbps in our system).
>>>>
>>>> Qi
>>>>
>>>>
>> We try to use "rxe_cfg icrc disable" and "rxe_cfg ICRC disable", but it
>>
>> seems that the performance doesn't change at all.
> Why are you continuing to try "disable" if your kernel and rdma-core don't support it?
>
> Thanks
>
>> Qi
>>
>>
>>

Thank you.

Can you tell us the edition of OS, kernel and softroce in your system?

So we can install the same edition and make our softroce work well.

By the way, can you tell me the softroce throughput (measured by

ib_send_bw or ib_read_bw) and the TCP throughput (measured by

iperf) in your system?

Qi


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [question]Why our soft-RoCE throughput is quite low compared with TCP
  2019-11-19  0:22               ` wangqi
@ 2019-11-19  7:38                 ` Leon Romanovsky
  0 siblings, 0 replies; 10+ messages in thread
From: Leon Romanovsky @ 2019-11-19  7:38 UTC (permalink / raw)
  To: wangqi; +Cc: linux-rdma

On Tue, Nov 19, 2019 at 08:22:34AM +0800, wangqi wrote:
>
> On 2019/11/18 下午10:41, Leon Romanovsky wrote:
> > On Mon, Nov 18, 2019 at 08:56:35PM +0800, wangqi wrote:
> >> On 2019/11/18 下午8:28, Leon Romanovsky wrote:
> >>> On Mon, Nov 18, 2019 at 06:13:07PM +0800, wangqi wrote:
> >>>> On 2019/11/18 下午5:49, Leon Romanovsky wrote:
> >>>>> On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
> >>>>>> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
> >>>>>>
> >>>>>>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
> >>>>>>>> Dear experts on RDMA,
> >>>>>>>>       We are sorry to disturb you. Because of a project, we need to
> >>>>>>>> integrate soft-RoCE in our system. However ,we are very confused by our
> >>>>>>>> soft-RoCE throughput results, which are quite low compared with TCP
> >>>>>>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
> >>>>>>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
> >>>>>>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
> >>>>>>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
> >>>>>>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
> >>>>>>>> respectively. But in the figure 6 in the attached paper "A Performance
> >>>>>>>> Comparison of Container Networking Alternatives", the throughput of
> >>>>>>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
> >>>>>>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
> >>>>>>>> we get such high bandwidth? Do we need to configure some OS system settings?
> >>>>>>>>       We find that in 2017, someone finds the same problem and he posts all
> >>>>>>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
> >>>>>>>> . But it remains unsolved. His results are nearly the same with our's. For
> >>>>>>>> simplicity,  we do not post our results in this email. You can get very
> >>>>>>>> detailed information in the web page listed above.
> >>>>>>>>       We are very confused by our results. We will very appreciate it if we
> >>>>>>>> can receive your early reply. Best wishes,
> >>>>>>>> Wang Qi
> >>>>>>> Can you please fix your email client?
> >>>>>>> The email text looks like one big sentence.
> >>>>>>>
> >>>>>>> From the perf report attached to this bugzilla, looks like RXE does a
> >>>>>>> lot of CRC32 calculations and it is consistent with what Matan said
> >>>>>>> a long time ago, RXE "stuck" in ICRC calculations required by spec.
> >>>>>>>
> >>>>>>> I'm curios what are your CONFIG_CRYPTO_* configs?
> >>>>>>>
> >>>>>>> ThanksCONFIG_CRYPTO_* configs
> >>>>>>>
> >>>>>>>
> >>>>>> I'm sorry for the editor problem in my last email. Now I use another editor.
> >>>>> Now your email has extra line between lines.
> >>>>>
> >>>>>> We get our rdma-core and perftest from
> >>>>>>
> >>>>>> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
> >>>>>> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
> >>>>>>
> >>>>>> We attach five files to clarify our problem.
> >>>>>>
> >>>>>> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
> >>>>>>
> >>>>>> and softroce throughput in our two servers (connected via back to back).
> >>>>>>
> >>>>>> * The second file "server_CONFIG_CRYPTO_result.txt" is the
> >>>>>>
> >>>>>> CONFIG_CRYPTO_* config results in the two servers..
> >>>>>>
> >>>>>> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
> >>>>>>
> >>>>>> & perf record -ags sleep 10 & wait" results in our two servers (we use
> >>>>>>
> >>>>>> "perf report --header >perf" to make the file).
> >>>>>>
> >>>>>> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
> >>>>>>
> >>>>>> and softroce throughput in two virtual machines with the latest linux kernel
> >>>>>>
> >>>>>> 5.4.0-rc7
> >>>>>>
> >>>>>> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
> >>>>>>
> >>>>>> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
> >>>>>>
> >>>>>> machines.
> >>>>>>
> >>>>>> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
> >>>>>>
> >>>>>> & perf record -ags sleep 10 & wait " result in the two virtual machines.
> >>>>>>
> >>>>>> On the other side, we tried to use the rxe command "rxe_cfg crc disable"
> >>>>> I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
> >>>>> and there is no such module parameter in the kernel.
> >>>>>
> >>>>> Thanks
> >>>> We get the command "rxe_cfg crc disable" from the following webpages:
> >>>>
> >>>> https://www.systutorials.com/docs/linux/man/8-rxe_cfg/
> >>>>
> >>>> https://www.reflectionsofthevoid.com/2011/08/
> >>>>
> >>>> It may be removed in the present soft-roce edition.
> >>> It was never existed in upstream variant and in the kernel you are testing.
> >>>
> >>>> Can you figure out why our softroce throughput is so low from the six
> >>> According to the logs, it is ICRC.
> >>>
> >>>> files in our last email? We hope to get a much higher softroce throughput,
> >>>>
> >>>> like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be
> >>>>
> >>>> up to 100 Gbps in our system).
> >>>>
> >>>> Qi
> >>>>
> >>>>
> >> We try to use "rxe_cfg icrc disable" and "rxe_cfg ICRC disable", but it
> >>
> >> seems that the performance doesn't change at all.
> > Why are you continuing to try "disable" if your kernel and rdma-core don't support it?
> >
> > Thanks
> >
> >> Qi
> >>
> >>
> >>
>
> Thank you.
>
> Can you tell us the edition of OS, kernel and softroce in your system?
>
> So we can install the same edition and make our softroce work well.
>
> By the way, can you tell me the softroce throughput (measured by
>
> ib_send_bw or ib_read_bw) and the TCP throughput (measured by
>
> iperf) in your system?

I'm sorry, but I'm not running anything performance related on RXE,
so I imagine that my results will be aligned with yours.

RXE is used as my rapid development platform.

Thanks

>
> Qi
>

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-11-19  7:39 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-15 13:26 Why our soft-RoCE throughput is quite low compared with TCP QWang
2019-11-15 16:07 ` Leon Romanovsky
2019-11-18  6:38   ` [question]Why " wangqi
2019-11-18  9:49     ` Leon Romanovsky
2019-11-18 10:13       ` wangqi
2019-11-18 12:28         ` Leon Romanovsky
2019-11-18 12:56           ` wangqi
2019-11-18 14:41             ` Leon Romanovsky
2019-11-19  0:22               ` wangqi
2019-11-19  7:38                 ` Leon Romanovsky

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.