* any suggested configuration for rxe?
@ 2017-01-30 1:05 feu tche
[not found] ` <CAJ-uSQZcnQ+Mqj1XGuMSzP54WQi6KofDyp11dCog+bqGmL3MWA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: feu tche @ 2017-01-30 1:05 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi guys,
Is there any configuration (distro/kernel/rdma-core branch) you can
suggest to try out software rdma (rxe)?
I'm trying to get software rdma working to enable local testing on the
vm (vmware) without going proper hardware and so far can't get
pingpong tool working.
I've tried CentOS 7.2 first by building vanilla 4.9.5 with rxe enabled
Now I've tried Ubuntu 16.04.1, using built ubuntu 4.9.5 build with
rxe. I've noticed that rxe is actually enabled by default as module so
now tried just released 4.9.6 as well.
I've tried linux-rdma/rdma-core master and rdma-core-12 tag and lately
rleon/rdma-core debian-fixes-last
What I get is (in all the cases I've tried) that module is loaded and
looks ok, however serverside ibv_rc_pingpong fails when client
connects with:
root@xdev:~/rdma-core-rleon# ibv_rc_pingpong -d rxe0 -g 0 -i 1
local address: LID 0x0000, QPN 0x000012, PSN 0xaa7393, GID
fe80::20c:29ff:fe87:5082
remote address: LID 0x0000, QPN 0x000013, PSN 0xf70e00, GID
fe80::20c:29ff:fe87:5082
Completion for unknown wr_id 0
parse WC failed 2
Below is output of other checks, all looks good as far as I understand.
If you can suggest anything, please let me know.
Thanks!
Fedor
root@xdev:~# uname -a
Linux xdev 4.9.6-040906-generic #201701260330 SMP Thu Jan 26 08:32:10
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@xdev:~# lsmod | grep rxe
rdma_rxe 114688 1
ip6_udp_tunnel 16384 1 rdma_rxe
udp_tunnel 16384 1 rdma_rxe
ib_core 204800 6 rdma_rxe,ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm
root@xdev:~# lsmod | grep ib_uverbs
ib_uverbs 65536 4 rdma_ucm
ib_core 204800 6 rdma_rxe,ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm
root@xdev:~# rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
ens33 yes e1000 1500 192.168.237.132 rxe0 1024 (3)
root@xdev:~# ibv_devices
device node GUID
------ ----------------
rxe0 020c29fffe875082
root@xdev:~# ibv_devinfo -d rxe0
hca_id: rxe0
transport: InfiniBand (0)
fw_ver: 0.0.0
node_guid: 020c:29ff:fe87:5082
sys_image_guid: 0000:0000:0000:0000
vendor_id: 0x0000
vendor_part_id: 0
hw_ver: 0x0
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAJ-uSQZcnQ+Mqj1XGuMSzP54WQi6KofDyp11dCog+bqGmL3MWA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-01-30 9:35 ` Moni Shoua
[not found] ` <CAG9sBKPj9wJhwZOFDXoOwH0=tBZR1pWdS6chYn_VK6SLdDQ7Xg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: Moni Shoua @ 2017-01-30 9:35 UTC (permalink / raw)
To: feu tche; +Cc: linux-rdma
On Mon, Jan 30, 2017 at 3:05 AM, feu tche <feutche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi guys,
>
> Is there any configuration (distro/kernel/rdma-core branch) you can
> suggest to try out software rdma (rxe)?
>
> I'm trying to get software rdma working to enable local testing on the
> vm (vmware) without going proper hardware and so far can't get
> pingpong tool working.
> I've tried CentOS 7.2 first by building vanilla 4.9.5 with rxe enabled
> Now I've tried Ubuntu 16.04.1, using built ubuntu 4.9.5 build with
> rxe. I've noticed that rxe is actually enabled by default as module so
> now tried just released 4.9.6 as well.
> I've tried linux-rdma/rdma-core master and rdma-core-12 tag and lately
> rleon/rdma-core debian-fixes-last
>
> What I get is (in all the cases I've tried) that module is loaded and
> looks ok, however serverside ibv_rc_pingpong fails when client
> connects with:
>
> root@xdev:~/rdma-core-rleon# ibv_rc_pingpong -d rxe0 -g 0 -i 1
> local address: LID 0x0000, QPN 0x000012, PSN 0xaa7393, GID
> fe80::20c:29ff:fe87:5082
> remote address: LID 0x0000, QPN 0x000013, PSN 0xf70e00, GID
> fe80::20c:29ff:fe87:5082
> Completion for unknown wr_id 0
> parse WC failed 2
>
> Below is output of other checks, all looks good as far as I understand.
>
> If you can suggest anything, please let me know.
>
> Thanks!
>
> Fedor
>
> root@xdev:~# uname -a
> Linux xdev 4.9.6-040906-generic #201701260330 SMP Thu Jan 26 08:32:10
> UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> root@xdev:~# lsmod | grep rxe
> rdma_rxe 114688 1
> ip6_udp_tunnel 16384 1 rdma_rxe
> udp_tunnel 16384 1 rdma_rxe
> ib_core 204800 6 rdma_rxe,ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm
>
> root@xdev:~# lsmod | grep ib_uverbs
> ib_uverbs 65536 4 rdma_ucm
> ib_core 204800 6 rdma_rxe,ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm
>
> root@xdev:~# rxe_cfg status
> Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
> ens33 yes e1000 1500 192.168.237.132 rxe0 1024 (3)
>
> root@xdev:~# ibv_devices
> device node GUID
> ------ ----------------
> rxe0 020c29fffe875082
>
> root@xdev:~# ibv_devinfo -d rxe0
> hca_id: rxe0
> transport: InfiniBand (0)
> fw_ver: 0.0.0
> node_guid: 020c:29ff:fe87:5082
> sys_image_guid: 0000:0000:0000:0000
> vendor_id: 0x0000
> vendor_part_id: 0
> hw_ver: 0x0
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
please try ibv_rc_pingpong with -g 1 and not g 0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAG9sBKPj9wJhwZOFDXoOwH0=tBZR1pWdS6chYn_VK6SLdDQ7Xg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-01-30 15:30 ` feu tche
[not found] ` <CAJ-uSQY50gRrQe7TAoJq3rd_MFqX8o=y_CZ=hqFJ1=WFZSg+tw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: feu tche @ 2017-01-30 15:30 UTC (permalink / raw)
To: Moni Shoua; +Cc: linux-rdma
Thanks Moni,
Unfortunately getting the same with -g1:
root@xdev:~# ibv_rc_pingpong -d rxe0 -g 1 -i 1
local address: LID 0x0000, QPN 0x000018, PSN 0xdf8d29, GID
::ffff:192.168.237.132
remote address: LID 0x0000, QPN 0x000011, PSN 0xc5c58c, GID
::ffff:192.168.237.133
Completion for unknown wr_id 0
parse WC failed 2
Regards,
Fedor
On 30 January 2017 at 09:35, Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> On Mon, Jan 30, 2017 at 3:05 AM, feu tche <feutche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Hi guys,
>>
>> Is there any configuration (distro/kernel/rdma-core branch) you can
>> suggest to try out software rdma (rxe)?
>>
>> I'm trying to get software rdma working to enable local testing on the
>> vm (vmware) without going proper hardware and so far can't get
>> pingpong tool working.
>> I've tried CentOS 7.2 first by building vanilla 4.9.5 with rxe enabled
>> Now I've tried Ubuntu 16.04.1, using built ubuntu 4.9.5 build with
>> rxe. I've noticed that rxe is actually enabled by default as module so
>> now tried just released 4.9.6 as well.
>> I've tried linux-rdma/rdma-core master and rdma-core-12 tag and lately
>> rleon/rdma-core debian-fixes-last
>>
>> What I get is (in all the cases I've tried) that module is loaded and
>> looks ok, however serverside ibv_rc_pingpong fails when client
>> connects with:
>>
>> root@xdev:~/rdma-core-rleon# ibv_rc_pingpong -d rxe0 -g 0 -i 1
>> local address: LID 0x0000, QPN 0x000012, PSN 0xaa7393, GID
>> fe80::20c:29ff:fe87:5082
>> remote address: LID 0x0000, QPN 0x000013, PSN 0xf70e00, GID
>> fe80::20c:29ff:fe87:5082
>> Completion for unknown wr_id 0
>> parse WC failed 2
>>
>> Below is output of other checks, all looks good as far as I understand.
>>
>> If you can suggest anything, please let me know.
>>
>> Thanks!
>>
>> Fedor
>>
>> root@xdev:~# uname -a
>> Linux xdev 4.9.6-040906-generic #201701260330 SMP Thu Jan 26 08:32:10
>> UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>>
>> root@xdev:~# lsmod | grep rxe
>> rdma_rxe 114688 1
>> ip6_udp_tunnel 16384 1 rdma_rxe
>> udp_tunnel 16384 1 rdma_rxe
>> ib_core 204800 6 rdma_rxe,ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm
>>
>> root@xdev:~# lsmod | grep ib_uverbs
>> ib_uverbs 65536 4 rdma_ucm
>> ib_core 204800 6 rdma_rxe,ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm
>>
>> root@xdev:~# rxe_cfg status
>> Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
>> ens33 yes e1000 1500 192.168.237.132 rxe0 1024 (3)
>>
>> root@xdev:~# ibv_devices
>> device node GUID
>> ------ ----------------
>> rxe0 020c29fffe875082
>>
>> root@xdev:~# ibv_devinfo -d rxe0
>> hca_id: rxe0
>> transport: InfiniBand (0)
>> fw_ver: 0.0.0
>> node_guid: 020c:29ff:fe87:5082
>> sys_image_guid: 0000:0000:0000:0000
>> vendor_id: 0x0000
>> vendor_part_id: 0
>> hw_ver: 0x0
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE (4)
>> max_mtu: 4096 (5)
>> active_mtu: 1024 (3)
>> sm_lid: 0
>> port_lid: 0
>> port_lmc: 0x00
>> link_layer: Ethernet
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> please try ibv_rc_pingpong with -g 1 and not g 0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAJ-uSQY50gRrQe7TAoJq3rd_MFqX8o=y_CZ=hqFJ1=WFZSg+tw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-01-31 8:28 ` Moni Shoua
[not found] ` <CAG9sBKPGWNrR_ejm-LxNzy5iqXKwvxUD2-mLYsGJnVw7FkxKHg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: Moni Shoua @ 2017-01-31 8:28 UTC (permalink / raw)
To: feu tche; +Cc: linux-rdma
On Mon, Jan 30, 2017 at 5:30 PM, feu tche <feutche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Thanks Moni,
>
> Unfortunately getting the same with -g1:
>
> root@xdev:~# ibv_rc_pingpong -d rxe0 -g 1 -i 1
> local address: LID 0x0000, QPN 0x000018, PSN 0xdf8d29, GID
> ::ffff:192.168.237.132
> remote address: LID 0x0000, QPN 0x000011, PSN 0xc5c58c, GID
> ::ffff:192.168.237.133
> Completion for unknown wr_id 0
> parse WC failed 2
>
> Regards,
>
> Fedor
Hi
1. Can you ping from 192.168.237.132 to 192.168.237.133?
2. Which Ethernet NICs are you using (vendor)?
3. Can you please run the test with tracing on the server side (the
side you sent the log for) and attach the results
(/sys/kernel/debug/tracing/trace)
thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAG9sBKPGWNrR_ejm-LxNzy5iqXKwvxUD2-mLYsGJnVw7FkxKHg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-02-01 9:35 ` Moni Shoua
[not found] ` <CAG9sBKMVVtUdffzH8DbS7vRzYiyAPDBn+n2MMBYvpe6SQXYHjA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: Moni Shoua @ 2017-02-01 9:35 UTC (permalink / raw)
To: feu tche; +Cc: linux-rdma
>
> Hi
> 1. Can you ping from 192.168.237.132 to 192.168.237.133?
> 2. Which Ethernet NICs are you using (vendor)?
> 3. Can you please run the test with tracing on the server side (the
> side you sent the log for) and attach the results
> (/sys/kernel/debug/tracing/trace)
>
> thanks
I forgot to mention that in the trace filter I'd like to see functions
from ib_core, rdma_cm, ib_cm, ib_rxe
Also, please check if you have a firewall (iptables) that block UDP/4791
thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAG9sBKMVVtUdffzH8DbS7vRzYiyAPDBn+n2MMBYvpe6SQXYHjA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-02-01 23:07 ` feu tche
[not found] ` <CAJ-uSQb0jqewQL095i_+vtzCnPK70AnLE+XAgv_XkjxKW9jLFQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: feu tche @ 2017-02-01 23:07 UTC (permalink / raw)
To: Moni Shoua; +Cc: linux-rdma
[-- Attachment #1: Type: text/plain, Size: 5747 bytes --]
hi Moni, guys,
Sure yes I could ping, yes, no firewall or anything, in all cases that
was clean minimal install of Ubuntu 16.04.1 with updates and 4.9.6
kernel.
In my previous tests that was all on vmware vms on virtual network,
but as you asked I also decided to check on real hardware.
I've got same results (below) between two machines with dual Intel
I350 and I also checked same both directions against a machine with
dual Mellanox ConnectX-3 Pro.
=====================================================================================
Client:
root@rdma-client:~/rdma-core-master# rxe_cfg status
Argument "" isn't numeric in numeric eq (==) at
/root/rdma-core-master/providers/rxe/rxe_cfg line 140.
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
bond0 yes bonding 1500 147.75.100.109 rxe0 1024 (3)
p1p1 yes igb 1500
p1p2 yes igb 1500
root@rdma-client:~/rdma-core-master# ibv_devinfo -d rxe0
hca_id: rxe0
transport: InfiniBand (0)
fw_ver: 0.0.0
node_guid: 0ec4:7aff:fe80:5d8e
sys_image_guid: 0000:0000:0000:0000
vendor_id: 0x0000
vendor_part_id: 0
hw_ver: 0x0
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
root@rdma-client:~/rdma-core-master# ping 147.75.100.177
PING 147.75.100.177 (147.75.100.177) 56(84) bytes of data.
64 bytes from 147.75.100.177: icmp_seq=1 ttl=63 time=0.451 ms
64 bytes from 147.75.100.177: icmp_seq=2 ttl=63 time=0.548 ms
64 bytes from 147.75.100.177: icmp_seq=3 ttl=63 time=0.548 ms
64 bytes from 147.75.100.177: icmp_seq=4 ttl=63 time=0.552 ms
root@rdma-client:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 1 -i 1
-n 1 147.75.100.177
local address: LID 0x0000, QPN 0x000011, PSN 0x4af006, GID
::ffff:147.75.100.109
remote address: LID 0x0000, QPN 0x000011, PSN 0xf304fb, GID
::ffff:147.75.100.177
^C
root@rdma-client:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 0 -i 1
-n 1 147.75.100.177
local address: LID 0x0000, QPN 0x000011, PSN 0xe483d9, GID
fe80::ec4:7aff:fe80:5d8e
remote address: LID 0x0000, QPN 0x000011, PSN 0xda937e, GID
fe80::ec4:7aff:fe80:5c2e
^C
====================================================================================
Server:
H/W path Device Class Description
======================================================
system SYS-5039MS-H12TRF (To be
filled by O.E.M.)
/0 bus X11SSE-F
/0/100 bridge Sky Lake Host Bridge/DRAM
Registers
/0/100/1 bridge Sky Lake PCIe Controller
(x16)
/0/100/1/0 p1p1 network I350 Gigabit Network
Connection
/0/100/1/0.1 p1p2 network I350 Gigabit Network Connection
root@rdma-server:~/rdma-core-master# rxe_cfg status
Argument "" isn't numeric in numeric eq (==) at
/root/rdma-core-master/providers/rxe/rxe_cfg line 140.
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
bond0 yes bonding 1500 147.75.100.177 rxe0 1024 (3)
p1p1 yes igb 1500
p1p2 yes igb 1500
root@rdma-server:~/rdma-core-master# ibv_devinfo -d rxe0
hca_id: rxe0
transport: InfiniBand (0)
fw_ver: 0.0.0
node_guid: 0ec4:7aff:fe80:5c2e
sys_image_guid: 0000:0000:0000:0000
vendor_id: 0x0000
vendor_part_id: 0
hw_ver: 0x0
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
root@rdma-server:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 1 -i 1
local address: LID 0x0000, QPN 0x000011, PSN 0xf304fb, GID
::ffff:147.75.100.177
remote address: LID 0x0000, QPN 0x000011, PSN 0x4af006, GID
::ffff:147.75.100.109
Completion for unknown wr_id 0
parse WC failed 2
root@rdma-server:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 0 -i 1
local address: LID 0x0000, QPN 0x000011, PSN 0xddb87c, GID
fe80::ec4:7aff:fe80:5c2e
remote address: LID 0x0000, QPN 0x000011, PSN 0x415a75, GID
fe80::ec4:7aff:fe80:5d8e
^C
Trace is attached.
Thanks!
Fedor
On 1 Feb 2017 09:35, "Moni Shoua" <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>
> >
> > Hi
> > 1. Can you ping from 192.168.237.132 to 192.168.237.133?
> > 2. Which Ethernet NICs are you using (vendor)?
> > 3. Can you please run the test with tracing on the server side (the
> > side you sent the log for) and attach the results
> > (/sys/kernel/debug/tracing/trace)
> >
> > thanks
>
> I forgot to mention that in the trace filter I'd like to see functions
> from ib_core, rdma_cm, ib_cm, ib_rxe
>
> Also, please check if you have a firewall (iptables) that block UDP/4791
>
> thanks
[-- Attachment #2: trace.txt --]
[-- Type: text/plain, Size: 100440 bytes --]
# tracer: function
#
# entries-in-buffer/entries-written: 6379/6379 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<idle>-0 [000] ..s. 5679.003076: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.071068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.071070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.071071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.071071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.071073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.071074: rxe_pool_get_index <-rxe_rcv
ibv_rc_pingpong-10613 [002] .... 5679.125601: show_node_type <-dev_attr_show
ibv_rc_pingpong-10613 [002] .... 5679.125654: rxe_alloc_ucontext <-ib_uverbs_get_context
ibv_rc_pingpong-10613 [002] .... 5679.125655: rxe_alloc <-rxe_alloc_ucontext
ibv_rc_pingpong-10613 [002] .... 5679.125659: ib_register_event_handler <-ib_uverbs_alloc_event_file
ibv_rc_pingpong-10613 [002] .... 5679.125666: rxe_alloc_pd <-ib_uverbs_alloc_pd
ibv_rc_pingpong-10613 [002] .... 5679.125666: rxe_alloc <-rxe_alloc_pd
ibv_rc_pingpong-10613 [002] .... 5679.125674: rxe_reg_user_mr <-ib_uverbs_reg_mr
ibv_rc_pingpong-10613 [002] .... 5679.125675: rxe_alloc <-rxe_reg_user_mr
ibv_rc_pingpong-10613 [002] .... 5679.125676: rxe_add_index <-rxe_reg_user_mr
ibv_rc_pingpong-10613 [002] .... 5679.125677: rxe_mem_init_user <-rxe_reg_user_mr
ibv_rc_pingpong-10613 [002] .... 5679.125678: ib_umem_get <-rxe_mem_init_user
ibv_rc_pingpong-10613 [002] .... 5679.125683: rxe_map_sg_attrs <-ib_umem_get
ibv_rc_pingpong-10613 [002] .... 5679.125683: rxe_map_sg <-rxe_map_sg_attrs
ibv_rc_pingpong-10613 [002] .... 5679.125684: rxe_mem_init <-rxe_mem_init_user
ibv_rc_pingpong-10613 [002] .... 5679.125685: rxe_mem_alloc.isra.2 <-rxe_mem_init_user
ibv_rc_pingpong-10613 [002] .... 5679.125692: rxe_create_cq <-create_cq.constprop.22
ibv_rc_pingpong-10613 [002] .... 5679.125692: rxe_cq_chk_attr <-rxe_create_cq
ibv_rc_pingpong-10613 [002] .... 5679.125693: rxe_alloc <-rxe_create_cq
ibv_rc_pingpong-10613 [002] .... 5679.125694: rxe_cq_from_init <-rxe_create_cq
ibv_rc_pingpong-10613 [002] .... 5679.125694: rxe_queue_init <-rxe_cq_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125716: do_mmap_info <-rxe_cq_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125716: rxe_create_mmap_info <-do_mmap_info
ibv_rc_pingpong-10613 [002] .... 5679.125723: rxe_mmap <-ib_uverbs_mmap
ibv_rc_pingpong-10613 [002] .... 5679.125738: rxe_create_qp <-create_qp.isra.20
ibv_rc_pingpong-10613 [002] .... 5679.125738: rxe_qp_chk_init <-rxe_create_qp
ibv_rc_pingpong-10613 [002] .... 5679.125739: rxe_qp_chk_cap <-rxe_qp_chk_init
ibv_rc_pingpong-10613 [002] .... 5679.125739: rxe_alloc <-rxe_create_qp
ibv_rc_pingpong-10613 [002] .... 5679.125740: rxe_add_index <-rxe_create_qp
ibv_rc_pingpong-10613 [002] .... 5679.125741: rxe_qp_from_init <-rxe_create_qp
ibv_rc_pingpong-10613 [002] .... 5679.125747: rxe_queue_init <-rxe_qp_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125751: do_mmap_info <-rxe_qp_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125751: rxe_create_mmap_info <-do_mmap_info
ibv_rc_pingpong-10613 [002] .... 5679.125752: rxe_init_task <-rxe_qp_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125753: rxe_init_task <-rxe_qp_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125753: rxe_queue_init <-rxe_qp_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125763: do_mmap_info <-rxe_qp_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125764: rxe_create_mmap_info <-do_mmap_info
ibv_rc_pingpong-10613 [002] .... 5679.125765: rxe_init_task <-rxe_qp_from_init
ibv_rc_pingpong-10613 [002] .... 5679.125769: rxe_mmap <-ib_uverbs_mmap
ibv_rc_pingpong-10613 [002] .... 5679.125775: rxe_mmap <-ib_uverbs_mmap
ibv_rc_pingpong-10613 [002] .... 5679.125781: ib_query_qp <-ib_uverbs_query_qp
ibv_rc_pingpong-10613 [002] .... 5679.125781: rxe_query_qp <-ib_query_qp
ibv_rc_pingpong-10613 [002] .... 5679.125781: rxe_qp_to_init <-rxe_query_qp
ibv_rc_pingpong-10613 [002] .... 5679.125782: rxe_qp_to_attr <-rxe_query_qp
ibv_rc_pingpong-10613 [002] .... 5679.125782: rxe_av_to_attr <-rxe_qp_to_attr
ibv_rc_pingpong-10613 [002] .... 5679.125783: rxe_av_to_attr <-rxe_qp_to_attr
ibv_rc_pingpong-10613 [002] .... 5679.125787: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
ibv_rc_pingpong-10613 [002] .... 5679.125787: rxe_modify_qp <-ib_uverbs_modify_qp
ibv_rc_pingpong-10613 [002] .... 5679.125788: rxe_qp_chk_attr <-rxe_modify_qp
ibv_rc_pingpong-10613 [002] .... 5679.125788: ib_modify_qp_is_ok <-rxe_qp_chk_attr
ibv_rc_pingpong-10613 [002] .... 5679.125789: rxe_qp_from_attr <-rxe_modify_qp
ibv_rc_pingpong-10613 [002] .... 5679.125883: ib_query_port <-ib_uverbs_query_port
ibv_rc_pingpong-10613 [002] .... 5679.125884: rxe_query_port <-ib_query_port
ibv_rc_pingpong-10613 [002] .... 5679.125886: rdma_port_get_link_layer <-ib_query_port
ibv_rc_pingpong-10613 [002] .... 5679.125886: rxe_get_link_layer <-rdma_port_get_link_layer
ibv_rc_pingpong-10613 [002] .... 5679.125887: link_layer <-rxe_get_link_layer
ibv_rc_pingpong-10613 [002] .... 5679.125887: rdma_port_get_link_layer <-ib_uverbs_query_port
ibv_rc_pingpong-10613 [002] .... 5679.125888: rxe_get_link_layer <-rdma_port_get_link_layer
ibv_rc_pingpong-10613 [002] .... 5679.125888: link_layer <-rxe_get_link_layer
ibv_rc_pingpong-10613 [002] .... 5679.125914: port_attr_show <-sysfs_kf_seq_show
ibv_rc_pingpong-10613 [002] .... 5679.125915: show_port_gid <-port_attr_show
ibv_rc_pingpong-10613 [002] .... 5679.125915: ib_query_gid <-show_port_gid
ibv_rc_pingpong-10613 [002] .... 5679.125916: ib_get_cached_gid <-ib_query_gid
ibv_rc_pingpong-10613 [002] d... 5679.125916: __ib_cache_gid_get <-ib_get_cached_gid
<idle>-0 [000] ..s. 5679.138982: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.138984: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.138984: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.138985: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.138986: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.138988: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.138989: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.207010: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.207012: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.207013: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.207013: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.207014: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.207016: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.207017: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.274968: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.274970: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.274971: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.274972: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.274973: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.274974: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.274975: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.343071: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.343073: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.343075: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.343075: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.343076: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.343078: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.343079: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.411069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.411071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.411072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.411073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.411074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.411075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.411076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.479069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.479071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.479072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.479072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.479074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.479075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.479076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.547071: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.547073: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.547074: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.547075: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.547076: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.547077: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.547078: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.615072: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.615074: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.615075: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.615076: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.615077: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.615078: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.615080: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.683069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.683071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.683072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.683073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.683074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.683075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.683076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.751069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.751071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.751071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.751072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.751073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.751075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.751076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.819068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.819070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.819071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.819071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.819073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.819074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.819075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.887069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.887071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.887073: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.887073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.887074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.887076: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.887077: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5679.955068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5679.955070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.955071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5679.955072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5679.955073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5679.955074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5679.955075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.023067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.023069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.023070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.023071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.023072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.023073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.023074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.091070: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.091072: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.091073: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.091074: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.091075: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.091076: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.091077: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.159068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.159071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.159072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.159072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.159073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.159075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.159076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.227068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.227070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.227071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.227072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.227073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.227074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.227075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.295067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.295069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.295070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.295071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.295072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.295073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.295075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.363067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.363069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.363070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.363070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.363072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.363073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.363074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.431070: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.431072: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.431074: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.431074: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.431076: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.431078: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.431079: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.499067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.499069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.499070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.499070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.499071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.499073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.499074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.567069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.567071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.567072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.567073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.567074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.567075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.567076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.635068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.635070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.635071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.635072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.635073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.635074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.635075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.702978: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.702980: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.702981: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.702981: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.702982: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.702984: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.702985: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.771067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.771069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.771070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.771070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.771071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.771073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.771074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.839068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.839071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.839072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.839072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.839073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.839075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.839076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.907066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.907068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.907069: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.907069: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.907071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.907072: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.907073: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5680.975067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5680.975069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.975070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5680.975071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5680.975072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5680.975073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5680.975074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.043067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.043069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.043070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.043071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.043072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.043073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.043074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.111067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.111069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.111070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.111071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.111072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.111073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.111074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.179069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.179071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.179072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.179073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.179074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.179075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.179076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [004] ..s. 5681.246842: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [004] ..s. 5681.246844: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5681.246845: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5681.246846: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [004] d.s. 5681.246847: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [004] ..s. 5681.246849: rxe_pool_get_index <-rxe_rcv
<idle>-0 [004] ..s. 5681.246850: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.315068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.315070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.315071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.315072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.315073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.315074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.315075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.383066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.383067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.383068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.383069: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.383070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.383072: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.383073: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.451067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.451069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.451070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.451071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.451072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.451073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.451074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.519068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.519070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.519070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.519071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.519072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.519074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.519075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.587026: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.587028: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.587029: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.587029: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.587031: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.587032: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.587033: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.655072: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.655074: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.655074: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.655075: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.655076: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.655078: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.655079: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.723066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.723068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.723069: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.723070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.723071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.723072: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.723073: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.791069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.791072: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.791072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.791073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.791074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.791075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.791077: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.859069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.859071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.859072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.859072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.859074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.859075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.859076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.927069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.927071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.927072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.927073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.927074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.927076: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.927077: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5681.995065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5681.995067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.995068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5681.995068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5681.995070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5681.995071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5681.995072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.063065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.063066: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.063067: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.063068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.063069: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.063071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.063072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.131068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.131070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.131071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.131071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.131073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.131074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.131075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.199066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.199068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.199069: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.199070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.199071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.199073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.199074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.267065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.267067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.267068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.267069: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.267070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.267071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.267072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.335066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.335068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.335069: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.335069: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.335070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.335072: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.335073: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.403065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.403067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.403068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.403068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.403070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.403071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.403072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.470976: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.470978: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.470979: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.470979: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.470980: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.470982: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.470983: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.539066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.539068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.539069: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.539070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.539071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.539072: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.539073: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.607069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.607071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.607072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.607072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.607074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.607075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.607076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.675068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.675070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.675071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.675072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.675073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.675074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.675075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.743069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.743071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.743072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.743072: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.743073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.743075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.743076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.811066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.811068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.811070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.811070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.811071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.811073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.811074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.879065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.879067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.879068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.879068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.879069: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.879071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.879072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5682.947068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5682.947069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.947070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5682.947071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5682.947072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5682.947074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5682.947075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.015018: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.015020: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.015021: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.015021: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.015023: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.015024: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.015025: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.083067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.083070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.083071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.083071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.083072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.083074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.083075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.151066: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.151068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.151069: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.151070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.151071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.151073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.151074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.219067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.219069: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.219070: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.219071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.219072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.219073: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.219074: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.287064: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.287066: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.287067: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.287067: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.287069: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.287070: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.287071: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.355067: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.355068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.355069: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.355070: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.355071: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.355072: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.355073: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.423064: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.423066: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.423068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.423068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.423069: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.423071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.423072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.491065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.491067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.491068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.491068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.491070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.491071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.491072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [004] ..s. 5683.558831: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [004] ..s. 5683.558833: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5683.558834: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5683.558835: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [004] d.s. 5683.558836: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [004] ..s. 5683.558838: rxe_pool_get_index <-rxe_rcv
<idle>-0 [004] ..s. 5683.558839: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [004] ..s. 5683.626808: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [004] ..s. 5683.626809: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5683.626809: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5683.626809: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [004] d.s. 5683.626810: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [004] ..s. 5683.626810: rxe_pool_get_index <-rxe_rcv
<idle>-0 [004] ..s. 5683.626811: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [004] ..s. 5683.694808: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [004] ..s. 5683.694808: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5683.694809: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [004] ..s. 5683.694809: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [004] d.s. 5683.694809: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [004] ..s. 5683.694810: rxe_pool_get_index <-rxe_rcv
<idle>-0 [004] ..s. 5683.694810: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.763068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.763070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.763071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.763071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.763072: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.763074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.763075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.831069: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.831071: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.831072: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.831073: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.831074: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.831075: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.831076: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.899080: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.899082: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.899083: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.899083: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.899084: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.899086: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.899087: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5683.967065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5683.967068: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.967068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5683.967069: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5683.967070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5683.967071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5683.967072: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5684.034977: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5684.034979: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.034979: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.034980: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5684.034981: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5684.034982: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5684.034984: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5684.103064: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5684.103066: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.103067: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.103068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5684.103069: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5684.103070: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5684.103071: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5684.171063: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5684.171065: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.171065: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.171066: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5684.171067: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5684.171069: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5684.171070: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5684.238970: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5684.238972: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.238973: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.238974: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5684.238975: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5684.238976: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5684.238977: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5684.307068: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5684.307070: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.307071: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.307071: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5684.307073: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5684.307074: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5684.307075: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5684.375062: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5684.375064: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.375065: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.375065: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5684.375066: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5684.375068: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5684.375069: check_type_state.isra.7 <-rxe_rcv
<idle>-0 [000] ..s. 5684.443064: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5684.443066: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.443067: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5684.443068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5684.443069: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5684.443071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5684.443072: check_type_state.isra.7 <-rxe_rcv
ibv_rc_pingpong-10613 [000] .... 5685.384312: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
ibv_rc_pingpong-10613 [000] .... 5685.384315: ib_resolve_eth_dmac.part.10 <-ib_resolve_eth_dmac
ibv_rc_pingpong-10613 [000] .... 5685.384316: ib_query_gid <-ib_resolve_eth_dmac.part.10
ibv_rc_pingpong-10613 [000] .... 5685.384317: ib_get_cached_gid <-ib_query_gid
ibv_rc_pingpong-10613 [000] d... 5685.384318: __ib_cache_gid_get <-ib_get_cached_gid
ibv_rc_pingpong-10613 [000] .... 5685.384319: rdma_addr_find_l2_eth_by_grh <-ib_resolve_eth_dmac.part.10
ibv_rc_pingpong-10613 [000] .... 5685.384319: rdma_resolve_ip <-rdma_addr_find_l2_eth_by_grh
ibv_rc_pingpong-10613 [000] .... 5685.384320: addr_resolve <-rdma_resolve_ip
ibv_rc_pingpong-10613 [000] .... 5685.384322: addr_resolve_neigh <-addr_resolve
ibv_rc_pingpong-10613 [000] .... 5685.384323: rdma_copy_addr <-addr_resolve_neigh
ibv_rc_pingpong-10613 [000] .... 5685.384325: queue_req <-rdma_resolve_ip
ibv_rc_pingpong-10613 [000] .... 5685.384325: set_timeout <-queue_req
kworker/0:1-67 [000] .... 5685.384340: process_req <-process_one_work
kworker/0:1-67 [000] .... 5685.384341: resolve_cb <-process_req
ibv_rc_pingpong-10613 [001] .... 5685.384389: rxe_modify_qp <-ib_uverbs_modify_qp
ibv_rc_pingpong-10613 [001] .... 5685.384390: rxe_qp_chk_attr <-rxe_modify_qp
ibv_rc_pingpong-10613 [001] .... 5685.384391: ib_modify_qp_is_ok <-rxe_qp_chk_attr
ibv_rc_pingpong-10613 [001] .... 5685.384392: rxe_av_chk_attr <-rxe_qp_chk_attr
ibv_rc_pingpong-10613 [001] .... 5685.384392: rxe_qp_from_attr <-rxe_modify_qp
ibv_rc_pingpong-10613 [001] .... 5685.384393: free_rd_atomic_resources <-rxe_qp_from_attr
ibv_rc_pingpong-10613 [001] .... 5685.384395: ib_get_cached_gid <-rxe_qp_from_attr
ibv_rc_pingpong-10613 [001] d... 5685.384395: __ib_cache_gid_get <-ib_get_cached_gid
ibv_rc_pingpong-10613 [001] .... 5685.384396: rxe_av_from_attr <-rxe_qp_from_attr
ibv_rc_pingpong-10613 [001] .... 5685.384396: rxe_av_fill_ip_info <-rxe_qp_from_attr
ibv_rc_pingpong-10613 [001] .... 5685.384406: ib_resolve_eth_dmac <-ib_uverbs_modify_qp
ibv_rc_pingpong-10613 [001] .... 5685.384407: rxe_modify_qp <-ib_uverbs_modify_qp
ibv_rc_pingpong-10613 [001] .... 5685.384407: rxe_qp_chk_attr <-rxe_modify_qp
ibv_rc_pingpong-10613 [001] .... 5685.384407: ib_modify_qp_is_ok <-rxe_qp_chk_attr
ibv_rc_pingpong-10613 [001] .... 5685.384408: rxe_qp_from_attr <-rxe_modify_qp
ibv_rc_pingpong-10613 [000] ..s. 5685.384884: rxe_udp_encap_recv <-udp_queue_rcv_skb
ibv_rc_pingpong-10613 [000] ..s. 5685.384886: net_to_rxe <-rxe_udp_encap_recv
ibv_rc_pingpong-10613 [000] ..s. 5685.384887: rxe_rcv <-rxe_udp_encap_recv
ibv_rc_pingpong-10613 [000] ..s. 5685.384888: ib_find_cached_gid_by_port <-rxe_rcv
ibv_rc_pingpong-10613 [000] d.s. 5685.384889: find_gid <-ib_find_cached_gid_by_port
ibv_rc_pingpong-10613 [000] ..s. 5685.384891: rxe_pool_get_index <-rxe_rcv
ibv_rc_pingpong-10613 [000] ..s. 5685.384892: check_type_state.isra.7 <-rxe_rcv
ibv_rc_pingpong-10613 [000] ..s. 5685.384893: check_keys <-rxe_rcv
ibv_rc_pingpong-10613 [000] ..s. 5685.384894: rxe_icrc_hdr <-rxe_rcv
ibv_rc_pingpong-10613 [000] ..s. 5685.384896: rxe_resp_queue_pkt <-rxe_rcv
ibv_rc_pingpong-10613 [000] ..s. 5685.384897: rxe_run_task <-rxe_resp_queue_pkt
ibv_rc_pingpong-10613 [000] ..s. 5685.384897: rxe_do_task <-rxe_run_task
ibv_rc_pingpong-10613 [000] ..s. 5685.384898: rxe_responder <-rxe_do_task
ibv_rc_pingpong-10613 [000] ..s. 5685.384906: rxe_qp_error <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384906: rxe_run_task <-rxe_qp_error
ibv_rc_pingpong-10613 [000] ..s. 5685.384907: rxe_run_task <-rxe_qp_error
ibv_rc_pingpong-10613 [000] ..s. 5685.384908: rxe_run_task <-rxe_qp_error
ibv_rc_pingpong-10613 [000] ..s. 5685.384910: rxe_do_task <-tasklet_action
ibv_rc_pingpong-10613 [000] ..s. 5685.384910: rxe_responder <-rxe_do_task
ibv_rc_pingpong-10613 [000] ..s. 5685.384912: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384913: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384914: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384914: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384915: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384915: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384916: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384916: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384916: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384917: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384917: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384918: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384918: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384919: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384920: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384920: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384921: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384921: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384922: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384922: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384923: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384923: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384923: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384924: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384924: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384925: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384925: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384926: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384926: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384926: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384927: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384927: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384928: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384928: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384929: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384929: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384930: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384930: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384930: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384931: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384931: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384932: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384932: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384933: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384933: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384933: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384934: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384934: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384935: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384935: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384936: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384936: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384936: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384937: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384937: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384938: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384938: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384939: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384939: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384939: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384940: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384940: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384941: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384941: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384941: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384942: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384942: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384943: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384943: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384944: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384944: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384944: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384945: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384945: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384946: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384946: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384947: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384947: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384947: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384948: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384948: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384949: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384949: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384949: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384950: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384950: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384951: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384951: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384952: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384952: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384952: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384953: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384953: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384954: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384954: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384954: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384955: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384955: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384956: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384956: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384957: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384957: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384958: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384958: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384958: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384959: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384959: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384960: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384960: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384961: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384961: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384961: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384962: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384962: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384963: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384963: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384963: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384964: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384964: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384965: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384965: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384966: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384966: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384966: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384967: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384967: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384968: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384968: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384969: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384969: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384969: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384970: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384970: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384971: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384971: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384972: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384972: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384972: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384973: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384973: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384974: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384974: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384974: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384975: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384975: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384976: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384976: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384977: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384977: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384977: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384978: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384978: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384979: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384979: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384980: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384980: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384980: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384981: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384981: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384982: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384983: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384983: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384984: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384984: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384984: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384985: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384985: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384986: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384986: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384986: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384987: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384987: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384988: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384988: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384989: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384989: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384989: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384990: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384990: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384991: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384991: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384992: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384992: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384992: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384993: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384993: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384994: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384994: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384995: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384995: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384995: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384996: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384996: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384997: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384997: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384998: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384998: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384998: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384999: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.384999: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385000: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385000: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385000: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385001: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385001: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385002: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385002: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385003: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385003: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385003: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385004: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385004: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385005: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385005: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385005: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385006: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385006: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385007: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385007: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385008: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385008: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385008: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385009: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385009: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385010: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385010: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385010: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385011: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385011: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385012: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385012: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385013: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385013: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385013: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385014: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385014: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385015: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385015: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385015: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385016: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385016: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385017: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385017: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385018: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385018: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385018: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385019: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385019: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385020: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385020: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385021: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385021: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385022: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385022: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385022: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385023: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385023: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385024: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385024: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385024: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385025: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385025: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385026: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385026: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385026: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385027: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385027: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385028: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385028: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385029: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385029: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385029: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385030: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385030: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385031: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385031: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385031: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385032: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385032: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385033: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385033: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385034: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385034: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385034: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385035: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385035: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385036: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385036: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385036: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385037: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385037: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385038: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385038: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385039: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385039: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385039: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385040: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385040: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385041: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385041: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385042: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385042: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385042: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385043: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385044: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385044: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385044: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385045: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385045: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385046: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385046: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385046: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385047: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385047: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385048: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385048: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385049: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385049: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385049: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385050: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385050: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385051: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385051: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385052: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385052: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385052: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385053: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385053: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385054: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385054: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385054: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385055: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385055: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385056: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385056: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385056: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385057: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385057: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385058: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385058: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385059: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385059: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385059: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385060: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385060: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385061: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385061: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385062: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385062: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385062: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385063: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385063: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385064: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385064: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385065: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385065: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385065: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385066: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385066: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385067: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385067: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385068: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385068: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385068: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385069: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385069: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385070: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385070: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385070: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385071: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385071: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385072: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385072: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385073: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385073: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385073: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385074: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385074: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385075: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385075: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385076: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385076: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385076: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385077: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385077: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385078: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385078: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385079: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385079: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385079: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385080: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385080: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385081: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385081: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385082: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385082: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385082: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385083: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385083: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385084: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385084: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385085: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385085: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385085: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385086: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385086: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385087: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385087: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385088: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385088: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385088: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385089: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385089: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385090: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385090: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385090: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385091: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385091: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385092: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385092: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385093: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385093: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385094: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385094: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385094: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385095: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385095: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385096: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385096: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385096: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385097: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385097: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385098: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385098: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385099: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385099: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385099: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385100: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385100: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385101: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385101: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385101: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385102: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385102: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385103: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385103: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385104: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385104: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385104: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385105: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385106: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385106: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385107: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385107: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385107: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385108: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385108: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385109: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385109: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385110: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385110: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385110: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385111: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385111: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385112: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385112: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385113: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385113: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385113: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385114: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385114: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385115: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385115: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385116: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385116: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385116: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385117: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385117: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385118: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385118: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385118: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385119: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385119: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385120: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385120: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385121: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385121: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385121: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385122: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385122: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385123: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385123: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385124: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385124: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385124: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385125: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385125: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385126: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385126: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385126: rxe_cq_post <-rxe_responder
ibv_rc_pingpong-10613 [000] ..s. 5685.385127: rxe_do_task <-tasklet_action
ibv_rc_pingpong-10613 [000] ..s. 5685.385128: rxe_completer <-rxe_do_task
ibv_rc_pingpong-10613 [000] ..s. 5685.385129: rxe_do_task <-tasklet_action
ibv_rc_pingpong-10613 [000] ..s. 5685.385129: rxe_requester <-rxe_do_task
ibv_rc_pingpong-10613 [000] .... 5685.385680: rxe_vma_close <-remove_vma
ibv_rc_pingpong-10613 [000] .... 5685.385681: rxe_vma_close <-remove_vma
ibv_rc_pingpong-10613 [000] .... 5685.385682: rxe_vma_close <-remove_vma
ibv_rc_pingpong-10613 [000] .... 5685.385693: ib_unregister_event_handler <-ib_uverbs_event_close
ibv_rc_pingpong-10613 [000] .... 5685.385698: ib_destroy_qp <-ib_uverbs_cleanup_ucontext
ibv_rc_pingpong-10613 [000] .... 5685.385700: rxe_destroy_qp <-ib_destroy_qp
ibv_rc_pingpong-10613 [000] .... 5685.385700: rxe_qp_destroy <-rxe_destroy_qp
ibv_rc_pingpong-10613 [000] .... 5685.385700: rxe_cleanup_task <-rxe_qp_destroy
ibv_rc_pingpong-10613 [000] .... 5685.385702: rxe_cleanup_task <-rxe_qp_destroy
ibv_rc_pingpong-10613 [000] .... 5685.385702: rxe_cleanup_task <-rxe_qp_destroy
ibv_rc_pingpong-10613 [000] .... 5685.385703: __rxe_do_task <-rxe_qp_destroy
ibv_rc_pingpong-10613 [000] .... 5685.385703: rxe_requester <-__rxe_do_task
ibv_rc_pingpong-10613 [000] .... 5685.385703: __rxe_do_task <-rxe_qp_destroy
ibv_rc_pingpong-10613 [000] .... 5685.385704: rxe_completer <-__rxe_do_task
ibv_rc_pingpong-10613 [000] .... 5685.385704: __rxe_do_task <-rxe_qp_destroy
ibv_rc_pingpong-10613 [000] .... 5685.385705: rxe_requester <-__rxe_do_task
ibv_rc_pingpong-10613 [000] .... 5685.385705: rxe_drop_index <-rxe_destroy_qp
ibv_rc_pingpong-10613 [000] .... 5685.385706: rxe_elem_release <-rxe_destroy_qp
ibv_rc_pingpong-10613 [000] .... 5685.385706: rxe_qp_cleanup <-rxe_elem_release
ibv_rc_pingpong-10613 [000] .... 5685.385707: rxe_drop_all_mcast_groups <-rxe_qp_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385707: rxe_queue_cleanup <-rxe_qp_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385708: rxe_mmap_release <-rxe_queue_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385712: rxe_queue_cleanup <-rxe_qp_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385712: rxe_mmap_release <-rxe_queue_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385721: free_rd_atomic_resources <-rxe_qp_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385721: free_rd_atomic_resource <-free_rd_atomic_resources
ibv_rc_pingpong-10613 [000] .... 5685.385729: rxe_dev_put <-rxe_elem_release
ibv_rc_pingpong-10613 [000] .... 5685.385731: ib_destroy_cq <-ib_uverbs_cleanup_ucontext
ibv_rc_pingpong-10613 [000] .... 5685.385731: rxe_destroy_cq <-ib_destroy_cq
ibv_rc_pingpong-10613 [000] .... 5685.385732: rxe_elem_release <-rxe_destroy_cq
ibv_rc_pingpong-10613 [000] .... 5685.385732: rxe_cq_cleanup <-rxe_elem_release
ibv_rc_pingpong-10613 [000] .... 5685.385732: rxe_queue_cleanup <-rxe_cq_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385733: rxe_mmap_release <-rxe_queue_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385736: rxe_dev_put <-rxe_elem_release
ibv_rc_pingpong-10613 [000] .... 5685.385738: ib_dereg_mr <-ib_uverbs_cleanup_ucontext
ibv_rc_pingpong-10613 [000] .... 5685.385739: rxe_dereg_mr <-ib_dereg_mr
ibv_rc_pingpong-10613 [000] .... 5685.385739: rxe_drop_index <-rxe_dereg_mr
ibv_rc_pingpong-10613 [000] .... 5685.385740: rxe_elem_release <-rxe_dereg_mr
ibv_rc_pingpong-10613 [000] .... 5685.385740: rxe_mem_cleanup <-rxe_elem_release
ibv_rc_pingpong-10613 [000] .... 5685.385740: ib_umem_release <-rxe_mem_cleanup
ibv_rc_pingpong-10613 [000] .... 5685.385741: __ib_umem_release <-ib_umem_release
ibv_rc_pingpong-10613 [000] .... 5685.385742: rxe_unmap_sg <-__ib_umem_release
ibv_rc_pingpong-10613 [000] .... 5685.385745: rxe_dev_put <-rxe_elem_release
ibv_rc_pingpong-10613 [000] .... 5685.385746: ib_dealloc_pd <-ib_uverbs_cleanup_ucontext
ibv_rc_pingpong-10613 [000] .... 5685.385747: rxe_dealloc_pd <-ib_dealloc_pd
ibv_rc_pingpong-10613 [000] .... 5685.385747: rxe_elem_release <-rxe_dealloc_pd
ibv_rc_pingpong-10613 [000] .... 5685.385748: rxe_dev_put <-rxe_elem_release
ibv_rc_pingpong-10613 [000] .... 5685.385748: rxe_dealloc_ucontext <-ib_uverbs_cleanup_ucontext
ibv_rc_pingpong-10613 [000] .... 5685.385749: rxe_elem_release <-rxe_dealloc_ucontext
ibv_rc_pingpong-10613 [000] .... 5685.385749: rxe_dev_put <-rxe_elem_release
<idle>-0 [000] ..s. 5685.451065: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5685.451067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.451068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.451069: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5685.451070: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5685.451071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5685.519063: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5685.519065: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.519066: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.519067: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5685.519068: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5685.519069: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5685.587064: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5685.587067: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.587068: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.587068: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5685.587069: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5685.587071: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5685.655064: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5685.655066: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.655067: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.655067: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5685.655068: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5685.655070: rxe_pool_get_index <-rxe_rcv
<idle>-0 [000] ..s. 5685.723062: rxe_udp_encap_recv <-udp_queue_rcv_skb
<idle>-0 [000] ..s. 5685.723065: net_to_rxe <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.723066: rxe_rcv <-rxe_udp_encap_recv
<idle>-0 [000] ..s. 5685.723066: ib_find_cached_gid_by_port <-rxe_rcv
<idle>-0 [000] d.s. 5685.723067: find_gid <-ib_find_cached_gid_by_port
<idle>-0 [000] ..s. 5685.723069: rxe_pool_get_index <-rxe_rcv
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAJ-uSQb0jqewQL095i_+vtzCnPK70AnLE+XAgv_XkjxKW9jLFQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-02-02 6:56 ` Moni Shoua
[not found] ` <CAG9sBKPERZ-8f2N-S7B4+HmgrF+1=VGmOXTTM7urQ4+DZSAmFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-02-06 8:15 ` Yonatan Cohen
1 sibling, 1 reply; 11+ messages in thread
From: Moni Shoua @ 2017-02-02 6:56 UTC (permalink / raw)
To: feu tche; +Cc: linux-rdma
On Thu, Feb 2, 2017 at 1:07 AM, feu tche <feutche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> hi Moni, guys,
>
> Sure yes I could ping, yes, no firewall or anything, in all cases that
> was clean minimal install of Ubuntu 16.04.1 with updates and 4.9.6
> kernel.
> In my previous tests that was all on vmware vms on virtual network,
> but as you asked I also decided to check on real hardware.
>
> I've got same results (below) between two machines with dual Intel
> I350 and I also checked same both directions against a machine with
> dual Mellanox ConnectX-3 Pro.
>
I will try to bring up the same setup here and see how it goes.
thanks for the report
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAG9sBKPERZ-8f2N-S7B4+HmgrF+1=VGmOXTTM7urQ4+DZSAmFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-02-02 8:18 ` Yigal Korman
0 siblings, 0 replies; 11+ messages in thread
From: Yigal Korman @ 2017-02-02 8:18 UTC (permalink / raw)
To: Moni Shoua; +Cc: feu tche, linux-rdma
On Thu, Feb 2, 2017 at 8:56 AM, Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> On Thu, Feb 2, 2017 at 1:07 AM, feu tche <feutche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> hi Moni, guys,
>>
>> Sure yes I could ping, yes, no firewall or anything, in all cases that
>> was clean minimal install of Ubuntu 16.04.1 with updates and 4.9.6
>> kernel.
>> In my previous tests that was all on vmware vms on virtual network,
>> but as you asked I also decided to check on real hardware.
>>
>> I've got same results (below) between two machines with dual Intel
>> I350 and I also checked same both directions against a machine with
>> dual Mellanox ConnectX-3 Pro.
>>
> I will try to bring up the same setup here and see how it goes.
> thanks for the report
[resend in plaintext]
Hi, I have the same issue on 4.9 with libvirt using qemu/kvm (nic is
virtio_net).
Let me know if you need any info from me on reproducing this.
Thanks,
Yigal
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAJ-uSQb0jqewQL095i_+vtzCnPK70AnLE+XAgv_XkjxKW9jLFQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-02-02 6:56 ` Moni Shoua
@ 2017-02-06 8:15 ` Yonatan Cohen
[not found] ` <c50d9d3d-f696-ed0c-f23f-1c5a8e1f0f2d-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
1 sibling, 1 reply; 11+ messages in thread
From: Yonatan Cohen @ 2017-02-06 8:15 UTC (permalink / raw)
To: feu tche, Moni Shoua; +Cc: linux-rdma
On 2/2/2017 1:07 AM, feu tche wrote:
> hi Moni, guys,
>
> Sure yes I could ping, yes, no firewall or anything, in all cases that
> was clean minimal install of Ubuntu 16.04.1 with updates and 4.9.6
> kernel.
> In my previous tests that was all on vmware vms on virtual network,
> but as you asked I also decided to check on real hardware.
>
> I've got same results (below) between two machines with dual Intel
> I350 and I also checked same both directions against a machine with
> dual Mellanox ConnectX-3 Pro.
>
> =====================================================================================
> Client:
>
> root@rdma-client:~/rdma-core-master# rxe_cfg status
> Argument "" isn't numeric in numeric eq (==) at
> /root/rdma-core-master/providers/rxe/rxe_cfg line 140.
> Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
> bond0 yes bonding 1500 147.75.100.109 rxe0 1024 (3)
> p1p1 yes igb 1500
> p1p2 yes igb 1500
> root@rdma-client:~/rdma-core-master# ibv_devinfo -d rxe0
> hca_id: rxe0
> transport: InfiniBand (0)
> fw_ver: 0.0.0
> node_guid: 0ec4:7aff:fe80:5d8e
> sys_image_guid: 0000:0000:0000:0000
> vendor_id: 0x0000
> vendor_part_id: 0
> hw_ver: 0x0
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
>
> root@rdma-client:~/rdma-core-master# ping 147.75.100.177
> PING 147.75.100.177 (147.75.100.177) 56(84) bytes of data.
> 64 bytes from 147.75.100.177: icmp_seq=1 ttl=63 time=0.451 ms
> 64 bytes from 147.75.100.177: icmp_seq=2 ttl=63 time=0.548 ms
> 64 bytes from 147.75.100.177: icmp_seq=3 ttl=63 time=0.548 ms
> 64 bytes from 147.75.100.177: icmp_seq=4 ttl=63 time=0.552 ms
>
> root@rdma-client:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 1 -i 1
> -n 1 147.75.100.177
> local address: LID 0x0000, QPN 0x000011, PSN 0x4af006, GID
> ::ffff:147.75.100.109
> remote address: LID 0x0000, QPN 0x000011, PSN 0xf304fb, GID
> ::ffff:147.75.100.177
> ^C
>
> root@rdma-client:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 0 -i 1
> -n 1 147.75.100.177
> local address: LID 0x0000, QPN 0x000011, PSN 0xe483d9, GID
> fe80::ec4:7aff:fe80:5d8e
> remote address: LID 0x0000, QPN 0x000011, PSN 0xda937e, GID
> fe80::ec4:7aff:fe80:5c2e
> ^C
> ====================================================================================
>
> Server:
>
> H/W path Device Class Description
> ======================================================
> system SYS-5039MS-H12TRF (To be
> filled by O.E.M.)
> /0 bus X11SSE-F
> /0/100 bridge Sky Lake Host Bridge/DRAM
> Registers
> /0/100/1 bridge Sky Lake PCIe Controller
> (x16)
> /0/100/1/0 p1p1 network I350 Gigabit Network
> Connection
> /0/100/1/0.1 p1p2 network I350 Gigabit Network Connection
>
>
> root@rdma-server:~/rdma-core-master# rxe_cfg status
> Argument "" isn't numeric in numeric eq (==) at
> /root/rdma-core-master/providers/rxe/rxe_cfg line 140.
> Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
> bond0 yes bonding 1500 147.75.100.177 rxe0 1024 (3)
> p1p1 yes igb 1500
> p1p2 yes igb 1500
we never tried coupling rxe with a bond/bridge
please retry with a real device. like the p1p1 you have there.
thanks.
> root@rdma-server:~/rdma-core-master# ibv_devinfo -d rxe0
> hca_id: rxe0
> transport: InfiniBand (0)
> fw_ver: 0.0.0
> node_guid: 0ec4:7aff:fe80:5c2e
> sys_image_guid: 0000:0000:0000:0000
> vendor_id: 0x0000
> vendor_part_id: 0
> hw_ver: 0x0
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 0
> port_lmc: 0x00
> link_layer: Ethernet
>
> root@rdma-server:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 1 -i 1
> local address: LID 0x0000, QPN 0x000011, PSN 0xf304fb, GID
> ::ffff:147.75.100.177
> remote address: LID 0x0000, QPN 0x000011, PSN 0x4af006, GID
> ::ffff:147.75.100.109
> Completion for unknown wr_id 0
> parse WC failed 2
>
> root@rdma-server:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 0 -i 1
> local address: LID 0x0000, QPN 0x000011, PSN 0xddb87c, GID
> fe80::ec4:7aff:fe80:5c2e
> remote address: LID 0x0000, QPN 0x000011, PSN 0x415a75, GID
> fe80::ec4:7aff:fe80:5d8e
> ^C
>
> Trace is attached.
>
> Thanks!
>
> Fedor
>
>
> On 1 Feb 2017 09:35, "Moni Shoua" <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>>
>>>
>>> Hi
>>> 1. Can you ping from 192.168.237.132 to 192.168.237.133?
>>> 2. Which Ethernet NICs are you using (vendor)?
>>> 3. Can you please run the test with tracing on the server side (the
>>> side you sent the log for) and attach the results
>>> (/sys/kernel/debug/tracing/trace)
>>>
>>> thanks
>>
>> I forgot to mention that in the trace filter I'd like to see functions
>> from ib_core, rdma_cm, ib_cm, ib_rxe
>>
>> Also, please check if you have a firewall (iptables) that block UDP/4791
>>
>> thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <c50d9d3d-f696-ed0c-f23f-1c5a8e1f0f2d-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2017-02-06 9:44 ` feu tche
[not found] ` <CAJ-uSQbrS3MDz0NyWv1ctKQdXG9-e633mKc5cPe5fKOuL8Bj3Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: feu tche @ 2017-02-06 9:44 UTC (permalink / raw)
To: Yonatan Cohen; +Cc: Moni Shoua, linux-rdma
hi Yonatan,
Yes I can try single physical device, just need some time to configure.
When initially trying on vmware that was a single device, with all the
same results, so I don't think it will make any difference to be
honest. As I remember from diagrams, rxe fits just before bonding so
should work with it as well?
Thanks!
Fedor
On 6 February 2017 at 08:15, Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> On 2/2/2017 1:07 AM, feu tche wrote:
>>
>> hi Moni, guys,
>>
>> Sure yes I could ping, yes, no firewall or anything, in all cases that
>> was clean minimal install of Ubuntu 16.04.1 with updates and 4.9.6
>> kernel.
>> In my previous tests that was all on vmware vms on virtual network,
>> but as you asked I also decided to check on real hardware.
>>
>> I've got same results (below) between two machines with dual Intel
>> I350 and I also checked same both directions against a machine with
>> dual Mellanox ConnectX-3 Pro.
>>
>>
>> =====================================================================================
>> Client:
>>
>> root@rdma-client:~/rdma-core-master# rxe_cfg status
>> Argument "" isn't numeric in numeric eq (==) at
>> /root/rdma-core-master/providers/rxe/rxe_cfg line 140.
>> Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
>> bond0 yes bonding 1500 147.75.100.109 rxe0 1024 (3)
>> p1p1 yes igb 1500
>> p1p2 yes igb 1500
>> root@rdma-client:~/rdma-core-master# ibv_devinfo -d rxe0
>> hca_id: rxe0
>> transport: InfiniBand (0)
>> fw_ver: 0.0.0
>> node_guid: 0ec4:7aff:fe80:5d8e
>> sys_image_guid: 0000:0000:0000:0000
>> vendor_id: 0x0000
>> vendor_part_id: 0
>> hw_ver: 0x0
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE (4)
>> max_mtu: 4096 (5)
>> active_mtu: 1024 (3)
>> sm_lid: 0
>> port_lid: 0
>> port_lmc: 0x00
>> link_layer: Ethernet
>>
>> root@rdma-client:~/rdma-core-master# ping 147.75.100.177
>> PING 147.75.100.177 (147.75.100.177) 56(84) bytes of data.
>> 64 bytes from 147.75.100.177: icmp_seq=1 ttl=63 time=0.451 ms
>> 64 bytes from 147.75.100.177: icmp_seq=2 ttl=63 time=0.548 ms
>> 64 bytes from 147.75.100.177: icmp_seq=3 ttl=63 time=0.548 ms
>> 64 bytes from 147.75.100.177: icmp_seq=4 ttl=63 time=0.552 ms
>>
>> root@rdma-client:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 1 -i 1
>> -n 1 147.75.100.177
>> local address: LID 0x0000, QPN 0x000011, PSN 0x4af006, GID
>> ::ffff:147.75.100.109
>> remote address: LID 0x0000, QPN 0x000011, PSN 0xf304fb, GID
>> ::ffff:147.75.100.177
>> ^C
>>
>> root@rdma-client:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 0 -i 1
>> -n 1 147.75.100.177
>> local address: LID 0x0000, QPN 0x000011, PSN 0xe483d9, GID
>> fe80::ec4:7aff:fe80:5d8e
>> remote address: LID 0x0000, QPN 0x000011, PSN 0xda937e, GID
>> fe80::ec4:7aff:fe80:5c2e
>> ^C
>>
>> ====================================================================================
>>
>> Server:
>>
>> H/W path Device Class Description
>> ======================================================
>> system SYS-5039MS-H12TRF (To be
>> filled by O.E.M.)
>> /0 bus X11SSE-F
>> /0/100 bridge Sky Lake Host Bridge/DRAM
>> Registers
>> /0/100/1 bridge Sky Lake PCIe Controller
>> (x16)
>> /0/100/1/0 p1p1 network I350 Gigabit Network
>> Connection
>> /0/100/1/0.1 p1p2 network I350 Gigabit Network Connection
>>
>>
>> root@rdma-server:~/rdma-core-master# rxe_cfg status
>> Argument "" isn't numeric in numeric eq (==) at
>> /root/rdma-core-master/providers/rxe/rxe_cfg line 140.
>> Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
>> bond0 yes bonding 1500 147.75.100.177 rxe0 1024 (3)
>> p1p1 yes igb 1500
>> p1p2 yes igb 1500
>
> we never tried coupling rxe with a bond/bridge
> please retry with a real device. like the p1p1 you have there.
> thanks.
>
>> root@rdma-server:~/rdma-core-master# ibv_devinfo -d rxe0
>> hca_id: rxe0
>> transport: InfiniBand (0)
>> fw_ver: 0.0.0
>> node_guid: 0ec4:7aff:fe80:5c2e
>> sys_image_guid: 0000:0000:0000:0000
>> vendor_id: 0x0000
>> vendor_part_id: 0
>> hw_ver: 0x0
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE (4)
>> max_mtu: 4096 (5)
>> active_mtu: 1024 (3)
>> sm_lid: 0
>> port_lid: 0
>> port_lmc: 0x00
>> link_layer: Ethernet
>>
>> root@rdma-server:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 1 -i 1
>> local address: LID 0x0000, QPN 0x000011, PSN 0xf304fb, GID
>> ::ffff:147.75.100.177
>> remote address: LID 0x0000, QPN 0x000011, PSN 0x4af006, GID
>> ::ffff:147.75.100.109
>> Completion for unknown wr_id 0
>> parse WC failed 2
>>
>> root@rdma-server:~/rdma-core-master# ibv_rc_pingpong -d rxe0 -g 0 -i 1
>> local address: LID 0x0000, QPN 0x000011, PSN 0xddb87c, GID
>> fe80::ec4:7aff:fe80:5c2e
>> remote address: LID 0x0000, QPN 0x000011, PSN 0x415a75, GID
>> fe80::ec4:7aff:fe80:5d8e
>> ^C
>>
>> Trace is attached.
>>
>> Thanks!
>>
>> Fedor
>>
>>
>> On 1 Feb 2017 09:35, "Moni Shoua" <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>>>
>>>
>>>>
>>>> Hi
>>>> 1. Can you ping from 192.168.237.132 to 192.168.237.133?
>>>> 2. Which Ethernet NICs are you using (vendor)?
>>>> 3. Can you please run the test with tracing on the server side (the
>>>> side you sent the log for) and attach the results
>>>> (/sys/kernel/debug/tracing/trace)
>>>>
>>>> thanks
>>>
>>>
>>> I forgot to mention that in the trace filter I'd like to see functions
>>> from ib_core, rdma_cm, ib_cm, ib_rxe
>>>
>>> Also, please check if you have a firewall (iptables) that block UDP/4791
>>>
>>> thanks
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: any suggested configuration for rxe?
[not found] ` <CAJ-uSQbrS3MDz0NyWv1ctKQdXG9-e633mKc5cPe5fKOuL8Bj3Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2017-02-06 12:38 ` Yonatan Cohen
0 siblings, 0 replies; 11+ messages in thread
From: Yonatan Cohen @ 2017-02-06 12:38 UTC (permalink / raw)
To: feu tche; +Cc: Moni Shoua, linux-rdma
On 2/6/2017 11:44 AM, feu tche wrote:
> hi Yonatan,
>
> Yes I can try single physical device, just need some time to configure.
> When initially trying on vmware that was a single device, with all the
> same results, so I don't think it will make any difference to be
> honest. As I remember from diagrams, rxe fits just before bonding so
> should work with it as well?
yes it should. but we have seen issues with it before.
we did not investigate yet.
>
> Thanks!
>
> Fedor
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2017-02-06 12:38 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-30 1:05 any suggested configuration for rxe? feu tche
[not found] ` <CAJ-uSQZcnQ+Mqj1XGuMSzP54WQi6KofDyp11dCog+bqGmL3MWA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-01-30 9:35 ` Moni Shoua
[not found] ` <CAG9sBKPj9wJhwZOFDXoOwH0=tBZR1pWdS6chYn_VK6SLdDQ7Xg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-01-30 15:30 ` feu tche
[not found] ` <CAJ-uSQY50gRrQe7TAoJq3rd_MFqX8o=y_CZ=hqFJ1=WFZSg+tw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-01-31 8:28 ` Moni Shoua
[not found] ` <CAG9sBKPGWNrR_ejm-LxNzy5iqXKwvxUD2-mLYsGJnVw7FkxKHg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-02-01 9:35 ` Moni Shoua
[not found] ` <CAG9sBKMVVtUdffzH8DbS7vRzYiyAPDBn+n2MMBYvpe6SQXYHjA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-02-01 23:07 ` feu tche
[not found] ` <CAJ-uSQb0jqewQL095i_+vtzCnPK70AnLE+XAgv_XkjxKW9jLFQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-02-02 6:56 ` Moni Shoua
[not found] ` <CAG9sBKPERZ-8f2N-S7B4+HmgrF+1=VGmOXTTM7urQ4+DZSAmFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-02-02 8:18 ` Yigal Korman
2017-02-06 8:15 ` Yonatan Cohen
[not found] ` <c50d9d3d-f696-ed0c-f23f-1c5a8e1f0f2d-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-02-06 9:44 ` feu tche
[not found] ` <CAJ-uSQbrS3MDz0NyWv1ctKQdXG9-e633mKc5cPe5fKOuL8Bj3Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-02-06 12:38 ` Yonatan Cohen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.