* ibv_rc_pingpong fails to create a completion queue
@ 2021-02-22 16:19 Filip Krakowski
2021-02-23 3:08 ` Zhu Yanjun
0 siblings, 1 reply; 6+ messages in thread
From: Filip Krakowski @ 2021-02-22 16:19 UTC (permalink / raw)
To: linux-rdma
Hi,
whenever I try to test a reliable connection using "ibv_rc_pingpong -d
mlx5_0" to start the server side the test immediately stops with
"Couldn't create CQ". Since I couldn't find a solution for this problem
in one week I would like to ask if someone has encountered this error
before or knows a way of troubleshooting it. Just to be sure I updated
the controller to its latest firmware (16.29.2002) today, but the error
remained the same.
System Information
====
* CentOS Linux release 8.1.1911 (Core)
* Linux 4.18.0-151.el8.x86_64
* ConnectX-5 (MCX555A-ECA)
Installed Packages
====
* rdma-core-32.0-4.el8.x86_64
* libibverbs-32.0-4.el8.x86_64
Loaded Kernel Modules (lsmod | grep -E 'rdma|mlx')
====
rpcrdma 274432 0
sunrpc 454656 22
rpcrdma,nfsv4,auth_rpcgss,lockd,nfsv3,rpcsec_gss_krb5,nfs_acl,nfs
rdma_ucm 32768 0
rdma_cm 69632 5
rpcrdma,ib_srpt,ib_iser,ib_isert,rdma_ucm
iw_cm 53248 1 rdma_cm
ib_cm 57344 3 rdma_cm,ib_ipoib,ib_srpt
mlx5_ib 327680 0
ib_uverbs 147456 3 i40iw,rdma_ucm,mlx5_ib
ib_core 356352 14
rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,i40iw,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx5_core 798720 1 mlx5_ib
mlxfw 24576 1 mlx5_core
Infiniband Device Info (ibv_devinfo)
====
hca_id: i40iw0
transport: iWARP (1)
fw_ver: 0.2
node_guid: 3cec:ef0d:51c3:0000
sys_image_guid: 3cec:ef0d:51c3:0000
vendor_id: 0x8086
vendor_part_id: 14290
hw_ver: 0x0
board_id: I40IW Board ID
phys_port_cnt: 1
port: 1
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 1
port_lmc: 0x00
link_layer: Ethernet
hca_id: i40iw1
transport: iWARP (1)
fw_ver: 0.2
node_guid: 3cec:ef0d:51c2:0000
sys_image_guid: 3cec:ef0d:51c2:0000
vendor_id: 0x8086
vendor_part_id: 14290
hw_ver: 0x0
board_id: I40IW Board ID
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 1
port_lmc: 0x00
link_layer: Ethernet
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 16.29.2002
node_guid: 0c42:a103:0054:74ca
sys_image_guid: 0c42:a103:0054:74ca
vendor_id: 0x02c9
vendor_part_id: 4119
hw_ver: 0x0
board_id: MT_0000000010
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 8
port_lid: 196
port_lmc: 0x00
link_layer: InfiniBand
Best regards
Filip
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ibv_rc_pingpong fails to create a completion queue
2021-02-22 16:19 ibv_rc_pingpong fails to create a completion queue Filip Krakowski
@ 2021-02-23 3:08 ` Zhu Yanjun
2021-02-23 8:56 ` Leon Romanovsky
0 siblings, 1 reply; 6+ messages in thread
From: Zhu Yanjun @ 2021-02-23 3:08 UTC (permalink / raw)
To: Filip Krakowski; +Cc: linux-rdma
On Tue, Feb 23, 2021 at 12:21 AM Filip Krakowski <krakowski@hhu.de> wrote:
>
> Hi,
>
> whenever I try to test a reliable connection using "ibv_rc_pingpong -d
> mlx5_0"
ibv_rc_pingpong -d rxe0 -g 1 > /dev/null &
ibv_rc_pingpong -d rxe0 -g 1 192.168.1.2
I made tests with the above. It can work well.
Normally "-g" is needed.
Before directly using mlx5, please make tests with softroce firstly.
Zhu Yanjun
to start the server side the test immediately stops with
> "Couldn't create CQ". Since I couldn't find a solution for this problem
> in one week I would like to ask if someone has encountered this error
> before or knows a way of troubleshooting it. Just to be sure I updated
> the controller to its latest firmware (16.29.2002) today, but the error
> remained the same.
>
> System Information
> ====
>
> * CentOS Linux release 8.1.1911 (Core)
> * Linux 4.18.0-151.el8.x86_64
> * ConnectX-5 (MCX555A-ECA)
>
>
> Installed Packages
> ====
>
> * rdma-core-32.0-4.el8.x86_64
> * libibverbs-32.0-4.el8.x86_64
>
>
> Loaded Kernel Modules (lsmod | grep -E 'rdma|mlx')
> ====
>
> rpcrdma 274432 0
> sunrpc 454656 22
> rpcrdma,nfsv4,auth_rpcgss,lockd,nfsv3,rpcsec_gss_krb5,nfs_acl,nfs
> rdma_ucm 32768 0
> rdma_cm 69632 5
> rpcrdma,ib_srpt,ib_iser,ib_isert,rdma_ucm
> iw_cm 53248 1 rdma_cm
> ib_cm 57344 3 rdma_cm,ib_ipoib,ib_srpt
> mlx5_ib 327680 0
> ib_uverbs 147456 3 i40iw,rdma_ucm,mlx5_ib
> ib_core 356352 14
> rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,i40iw,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
> mlx5_core 798720 1 mlx5_ib
> mlxfw 24576 1 mlx5_core
>
>
>
> Infiniband Device Info (ibv_devinfo)
> ====
>
> hca_id: i40iw0
> transport: iWARP (1)
> fw_ver: 0.2
> node_guid: 3cec:ef0d:51c3:0000
> sys_image_guid: 3cec:ef0d:51c3:0000
> vendor_id: 0x8086
> vendor_part_id: 14290
> hw_ver: 0x0
> board_id: I40IW Board ID
> phys_port_cnt: 1
> port: 1
> state: PORT_DOWN (1)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 1
> port_lmc: 0x00
> link_layer: Ethernet
>
> hca_id: i40iw1
> transport: iWARP (1)
> fw_ver: 0.2
> node_guid: 3cec:ef0d:51c2:0000
> sys_image_guid: 3cec:ef0d:51c2:0000
> vendor_id: 0x8086
> vendor_part_id: 14290
> hw_ver: 0x0
> board_id: I40IW Board ID
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 1024 (3)
> sm_lid: 0
> port_lid: 1
> port_lmc: 0x00
> link_layer: Ethernet
>
> hca_id: mlx5_0
> transport: InfiniBand (0)
> fw_ver: 16.29.2002
> node_guid: 0c42:a103:0054:74ca
> sys_image_guid: 0c42:a103:0054:74ca
> vendor_id: 0x02c9
> vendor_part_id: 4119
> hw_ver: 0x0
> board_id: MT_0000000010
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 4096 (5)
> active_mtu: 4096 (5)
> sm_lid: 8
> port_lid: 196
> port_lmc: 0x00
> link_layer: InfiniBand
>
>
> Best regards
> Filip
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ibv_rc_pingpong fails to create a completion queue
2021-02-23 3:08 ` Zhu Yanjun
@ 2021-02-23 8:56 ` Leon Romanovsky
2021-02-23 9:50 ` Zhu Yanjun
0 siblings, 1 reply; 6+ messages in thread
From: Leon Romanovsky @ 2021-02-23 8:56 UTC (permalink / raw)
To: Zhu Yanjun; +Cc: Filip Krakowski, linux-rdma
On Tue, Feb 23, 2021 at 11:08:18AM +0800, Zhu Yanjun wrote:
> On Tue, Feb 23, 2021 at 12:21 AM Filip Krakowski <krakowski@hhu.de> wrote:
> >
> > Hi,
> >
> > whenever I try to test a reliable connection using "ibv_rc_pingpong -d
> > mlx5_0"
>
> ibv_rc_pingpong -d rxe0 -g 1 > /dev/null &
>
> ibv_rc_pingpong -d rxe0 -g 1 192.168.1.2
>
> I made tests with the above. It can work well.
>
> Normally "-g" is needed.
"-g" is needed because you are running RoCE, while Filip is running IB.
>
> Before directly using mlx5, please make tests with softroce firstly.
Are you sure that RXE works in 4.18.0-151.el8.x86_64 kernel?
>
> Zhu Yanjun
>
> to start the server side the test immediately stops with
> > "Couldn't create CQ". Since I couldn't find a solution for this problem
> > in one week I would like to ask if someone has encountered this error
> > before or knows a way of troubleshooting it. Just to be sure I updated
> > the controller to its latest firmware (16.29.2002) today, but the error
> > remained the same.
> >
> > System Information
> > ====
> >
> > * CentOS Linux release 8.1.1911 (Core)
> > * Linux 4.18.0-151.el8.x86_64
> > * ConnectX-5 (MCX555A-ECA)
> >
> >
> > Installed Packages
> > ====
> >
> > * rdma-core-32.0-4.el8.x86_64
> > * libibverbs-32.0-4.el8.x86_64
> >
> >
> > Loaded Kernel Modules (lsmod | grep -E 'rdma|mlx')
> > ====
> >
> > rpcrdma 274432 0
> > sunrpc 454656 22
> > rpcrdma,nfsv4,auth_rpcgss,lockd,nfsv3,rpcsec_gss_krb5,nfs_acl,nfs
> > rdma_ucm 32768 0
> > rdma_cm 69632 5
> > rpcrdma,ib_srpt,ib_iser,ib_isert,rdma_ucm
> > iw_cm 53248 1 rdma_cm
> > ib_cm 57344 3 rdma_cm,ib_ipoib,ib_srpt
> > mlx5_ib 327680 0
> > ib_uverbs 147456 3 i40iw,rdma_ucm,mlx5_ib
> > ib_core 356352 14
> > rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,i40iw,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
> > mlx5_core 798720 1 mlx5_ib
> > mlxfw 24576 1 mlx5_core
> >
> >
> >
> > Infiniband Device Info (ibv_devinfo)
> > ====
> >
> > hca_id: i40iw0
> > transport: iWARP (1)
> > fw_ver: 0.2
> > node_guid: 3cec:ef0d:51c3:0000
> > sys_image_guid: 3cec:ef0d:51c3:0000
> > vendor_id: 0x8086
> > vendor_part_id: 14290
> > hw_ver: 0x0
> > board_id: I40IW Board ID
> > phys_port_cnt: 1
> > port: 1
> > state: PORT_DOWN (1)
> > max_mtu: 4096 (5)
> > active_mtu: 1024 (3)
> > sm_lid: 0
> > port_lid: 1
> > port_lmc: 0x00
> > link_layer: Ethernet
> >
> > hca_id: i40iw1
> > transport: iWARP (1)
> > fw_ver: 0.2
> > node_guid: 3cec:ef0d:51c2:0000
> > sys_image_guid: 3cec:ef0d:51c2:0000
> > vendor_id: 0x8086
> > vendor_part_id: 14290
> > hw_ver: 0x0
> > board_id: I40IW Board ID
> > phys_port_cnt: 1
> > port: 1
> > state: PORT_ACTIVE (4)
> > max_mtu: 4096 (5)
> > active_mtu: 1024 (3)
> > sm_lid: 0
> > port_lid: 1
> > port_lmc: 0x00
> > link_layer: Ethernet
> >
> > hca_id: mlx5_0
> > transport: InfiniBand (0)
> > fw_ver: 16.29.2002
> > node_guid: 0c42:a103:0054:74ca
> > sys_image_guid: 0c42:a103:0054:74ca
> > vendor_id: 0x02c9
> > vendor_part_id: 4119
> > hw_ver: 0x0
> > board_id: MT_0000000010
> > phys_port_cnt: 1
> > port: 1
> > state: PORT_ACTIVE (4)
> > max_mtu: 4096 (5)
> > active_mtu: 4096 (5)
> > sm_lid: 8
> > port_lid: 196
> > port_lmc: 0x00
> > link_layer: InfiniBand
> >
> >
> > Best regards
> > Filip
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ibv_rc_pingpong fails to create a completion queue
2021-02-23 8:56 ` Leon Romanovsky
@ 2021-02-23 9:50 ` Zhu Yanjun
2021-02-23 13:45 ` Filip Krakowski
0 siblings, 1 reply; 6+ messages in thread
From: Zhu Yanjun @ 2021-02-23 9:50 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: Filip Krakowski, linux-rdma
On Tue, Feb 23, 2021 at 4:56 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Feb 23, 2021 at 11:08:18AM +0800, Zhu Yanjun wrote:
> > On Tue, Feb 23, 2021 at 12:21 AM Filip Krakowski <krakowski@hhu.de> wrote:
> > >
> > > Hi,
> > >
> > > whenever I try to test a reliable connection using "ibv_rc_pingpong -d
> > > mlx5_0"
> >
> > ibv_rc_pingpong -d rxe0 -g 1 > /dev/null &
> >
> > ibv_rc_pingpong -d rxe0 -g 1 192.168.1.2
> >
> > I made tests with the above. It can work well.
> >
> > Normally "-g" is needed.
>
> "-g" is needed because you are running RoCE, while Filip is running IB.
>
> >
> > Before directly using mlx5, please make tests with softroce firstly.
>
> Are you sure that RXE works in 4.18.0-151.el8.x86_64 kernel?
I have no 4.18.0-151.el8.x86_64 kernel at hand.
I use the kernel 4.18.0-240.10.1.el8_3.x86_64 to make tests.
SoftRoCE can work well.
Zhu Yanjun
>
> >
> > Zhu Yanjun
> >
> > to start the server side the test immediately stops with
> > > "Couldn't create CQ". Since I couldn't find a solution for this problem
> > > in one week I would like to ask if someone has encountered this error
> > > before or knows a way of troubleshooting it. Just to be sure I updated
> > > the controller to its latest firmware (16.29.2002) today, but the error
> > > remained the same.
> > >
> > > System Information
> > > ====
> > >
> > > * CentOS Linux release 8.1.1911 (Core)
> > > * Linux 4.18.0-151.el8.x86_64
> > > * ConnectX-5 (MCX555A-ECA)
> > >
> > >
> > > Installed Packages
> > > ====
> > >
> > > * rdma-core-32.0-4.el8.x86_64
> > > * libibverbs-32.0-4.el8.x86_64
> > >
> > >
> > > Loaded Kernel Modules (lsmod | grep -E 'rdma|mlx')
> > > ====
> > >
> > > rpcrdma 274432 0
> > > sunrpc 454656 22
> > > rpcrdma,nfsv4,auth_rpcgss,lockd,nfsv3,rpcsec_gss_krb5,nfs_acl,nfs
> > > rdma_ucm 32768 0
> > > rdma_cm 69632 5
> > > rpcrdma,ib_srpt,ib_iser,ib_isert,rdma_ucm
> > > iw_cm 53248 1 rdma_cm
> > > ib_cm 57344 3 rdma_cm,ib_ipoib,ib_srpt
> > > mlx5_ib 327680 0
> > > ib_uverbs 147456 3 i40iw,rdma_ucm,mlx5_ib
> > > ib_core 356352 14
> > > rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,i40iw,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
> > > mlx5_core 798720 1 mlx5_ib
> > > mlxfw 24576 1 mlx5_core
> > >
> > >
> > >
> > > Infiniband Device Info (ibv_devinfo)
> > > ====
> > >
> > > hca_id: i40iw0
> > > transport: iWARP (1)
> > > fw_ver: 0.2
> > > node_guid: 3cec:ef0d:51c3:0000
> > > sys_image_guid: 3cec:ef0d:51c3:0000
> > > vendor_id: 0x8086
> > > vendor_part_id: 14290
> > > hw_ver: 0x0
> > > board_id: I40IW Board ID
> > > phys_port_cnt: 1
> > > port: 1
> > > state: PORT_DOWN (1)
> > > max_mtu: 4096 (5)
> > > active_mtu: 1024 (3)
> > > sm_lid: 0
> > > port_lid: 1
> > > port_lmc: 0x00
> > > link_layer: Ethernet
> > >
> > > hca_id: i40iw1
> > > transport: iWARP (1)
> > > fw_ver: 0.2
> > > node_guid: 3cec:ef0d:51c2:0000
> > > sys_image_guid: 3cec:ef0d:51c2:0000
> > > vendor_id: 0x8086
> > > vendor_part_id: 14290
> > > hw_ver: 0x0
> > > board_id: I40IW Board ID
> > > phys_port_cnt: 1
> > > port: 1
> > > state: PORT_ACTIVE (4)
> > > max_mtu: 4096 (5)
> > > active_mtu: 1024 (3)
> > > sm_lid: 0
> > > port_lid: 1
> > > port_lmc: 0x00
> > > link_layer: Ethernet
> > >
> > > hca_id: mlx5_0
> > > transport: InfiniBand (0)
> > > fw_ver: 16.29.2002
> > > node_guid: 0c42:a103:0054:74ca
> > > sys_image_guid: 0c42:a103:0054:74ca
> > > vendor_id: 0x02c9
> > > vendor_part_id: 4119
> > > hw_ver: 0x0
> > > board_id: MT_0000000010
> > > phys_port_cnt: 1
> > > port: 1
> > > state: PORT_ACTIVE (4)
> > > max_mtu: 4096 (5)
> > > active_mtu: 4096 (5)
> > > sm_lid: 8
> > > port_lid: 196
> > > port_lmc: 0x00
> > > link_layer: InfiniBand
> > >
> > >
> > > Best regards
> > > Filip
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ibv_rc_pingpong fails to create a completion queue
2021-02-23 9:50 ` Zhu Yanjun
@ 2021-02-23 13:45 ` Filip Krakowski
2021-02-24 10:27 ` Zhu Yanjun
0 siblings, 1 reply; 6+ messages in thread
From: Filip Krakowski @ 2021-02-23 13:45 UTC (permalink / raw)
To: Zhu Yanjun, Leon Romanovsky; +Cc: linux-rdma
Hi,
"I use the kernel 4.18.0-240.10.1.el8_3.x86_64 to make tests." was the
line that solved this problem.
I never thought I would be stuck with a problem for a week caused by the
kernel...
That said, updating the kernel to "4.18.0-277.el8.x86_64" solved the
problem.
Thanks for answering this fast and sorry for taking your time
considering the trivial solution.
Best regards
Filip
On 2/23/21 10:50 AM, Zhu Yanjun wrote:
> On Tue, Feb 23, 2021 at 4:56 PM Leon Romanovsky <leon@kernel.org> wrote:
>> On Tue, Feb 23, 2021 at 11:08:18AM +0800, Zhu Yanjun wrote:
>>> On Tue, Feb 23, 2021 at 12:21 AM Filip Krakowski <krakowski@hhu.de> wrote:
>>>> Hi,
>>>>
>>>> whenever I try to test a reliable connection using "ibv_rc_pingpong -d
>>>> mlx5_0"
>>> ibv_rc_pingpong -d rxe0 -g 1 > /dev/null &
>>>
>>> ibv_rc_pingpong -d rxe0 -g 1 192.168.1.2
>>>
>>> I made tests with the above. It can work well.
>>>
>>> Normally "-g" is needed.
>> "-g" is needed because you are running RoCE, while Filip is running IB.
>>
>>> Before directly using mlx5, please make tests with softroce firstly.
>> Are you sure that RXE works in 4.18.0-151.el8.x86_64 kernel?
> I have no 4.18.0-151.el8.x86_64 kernel at hand.
> I use the kernel 4.18.0-240.10.1.el8_3.x86_64 to make tests.
>
> SoftRoCE can work well.
>
> Zhu Yanjun
>
>>> Zhu Yanjun
>>>
>>> to start the server side the test immediately stops with
>>>> "Couldn't create CQ". Since I couldn't find a solution for this problem
>>>> in one week I would like to ask if someone has encountered this error
>>>> before or knows a way of troubleshooting it. Just to be sure I updated
>>>> the controller to its latest firmware (16.29.2002) today, but the error
>>>> remained the same.
>>>>
>>>> System Information
>>>> ====
>>>>
>>>> * CentOS Linux release 8.1.1911 (Core)
>>>> * Linux 4.18.0-151.el8.x86_64
>>>> * ConnectX-5 (MCX555A-ECA)
>>>>
>>>>
>>>> Installed Packages
>>>> ====
>>>>
>>>> * rdma-core-32.0-4.el8.x86_64
>>>> * libibverbs-32.0-4.el8.x86_64
>>>>
>>>>
>>>> Loaded Kernel Modules (lsmod | grep -E 'rdma|mlx')
>>>> ====
>>>>
>>>> rpcrdma 274432 0
>>>> sunrpc 454656 22
>>>> rpcrdma,nfsv4,auth_rpcgss,lockd,nfsv3,rpcsec_gss_krb5,nfs_acl,nfs
>>>> rdma_ucm 32768 0
>>>> rdma_cm 69632 5
>>>> rpcrdma,ib_srpt,ib_iser,ib_isert,rdma_ucm
>>>> iw_cm 53248 1 rdma_cm
>>>> ib_cm 57344 3 rdma_cm,ib_ipoib,ib_srpt
>>>> mlx5_ib 327680 0
>>>> ib_uverbs 147456 3 i40iw,rdma_ucm,mlx5_ib
>>>> ib_core 356352 14
>>>> rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,i40iw,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
>>>> mlx5_core 798720 1 mlx5_ib
>>>> mlxfw 24576 1 mlx5_core
>>>>
>>>>
>>>>
>>>> Infiniband Device Info (ibv_devinfo)
>>>> ====
>>>>
>>>> hca_id: i40iw0
>>>> transport: iWARP (1)
>>>> fw_ver: 0.2
>>>> node_guid: 3cec:ef0d:51c3:0000
>>>> sys_image_guid: 3cec:ef0d:51c3:0000
>>>> vendor_id: 0x8086
>>>> vendor_part_id: 14290
>>>> hw_ver: 0x0
>>>> board_id: I40IW Board ID
>>>> phys_port_cnt: 1
>>>> port: 1
>>>> state: PORT_DOWN (1)
>>>> max_mtu: 4096 (5)
>>>> active_mtu: 1024 (3)
>>>> sm_lid: 0
>>>> port_lid: 1
>>>> port_lmc: 0x00
>>>> link_layer: Ethernet
>>>>
>>>> hca_id: i40iw1
>>>> transport: iWARP (1)
>>>> fw_ver: 0.2
>>>> node_guid: 3cec:ef0d:51c2:0000
>>>> sys_image_guid: 3cec:ef0d:51c2:0000
>>>> vendor_id: 0x8086
>>>> vendor_part_id: 14290
>>>> hw_ver: 0x0
>>>> board_id: I40IW Board ID
>>>> phys_port_cnt: 1
>>>> port: 1
>>>> state: PORT_ACTIVE (4)
>>>> max_mtu: 4096 (5)
>>>> active_mtu: 1024 (3)
>>>> sm_lid: 0
>>>> port_lid: 1
>>>> port_lmc: 0x00
>>>> link_layer: Ethernet
>>>>
>>>> hca_id: mlx5_0
>>>> transport: InfiniBand (0)
>>>> fw_ver: 16.29.2002
>>>> node_guid: 0c42:a103:0054:74ca
>>>> sys_image_guid: 0c42:a103:0054:74ca
>>>> vendor_id: 0x02c9
>>>> vendor_part_id: 4119
>>>> hw_ver: 0x0
>>>> board_id: MT_0000000010
>>>> phys_port_cnt: 1
>>>> port: 1
>>>> state: PORT_ACTIVE (4)
>>>> max_mtu: 4096 (5)
>>>> active_mtu: 4096 (5)
>>>> sm_lid: 8
>>>> port_lid: 196
>>>> port_lmc: 0x00
>>>> link_layer: InfiniBand
>>>>
>>>>
>>>> Best regards
>>>> Filip
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ibv_rc_pingpong fails to create a completion queue
2021-02-23 13:45 ` Filip Krakowski
@ 2021-02-24 10:27 ` Zhu Yanjun
0 siblings, 0 replies; 6+ messages in thread
From: Zhu Yanjun @ 2021-02-24 10:27 UTC (permalink / raw)
To: Filip Krakowski; +Cc: Leon Romanovsky, linux-rdma
On Tue, Feb 23, 2021 at 9:45 PM Filip Krakowski <krakowski@hhu.de> wrote:
>
> Hi,
>
> "I use the kernel 4.18.0-240.10.1.el8_3.x86_64 to make tests." was the
> line that solved this problem.
> I never thought I would be stuck with a problem for a week caused by the
> kernel...
>
> That said, updating the kernel to "4.18.0-277.el8.x86_64" solved the
> problem.
Glad to hear this.
Zhu Yanjun
> Thanks for answering this fast and sorry for taking your time
> considering the trivial solution.
>
> Best regards
> Filip
>
> On 2/23/21 10:50 AM, Zhu Yanjun wrote:
> > On Tue, Feb 23, 2021 at 4:56 PM Leon Romanovsky <leon@kernel.org> wrote:
> >> On Tue, Feb 23, 2021 at 11:08:18AM +0800, Zhu Yanjun wrote:
> >>> On Tue, Feb 23, 2021 at 12:21 AM Filip Krakowski <krakowski@hhu.de> wrote:
> >>>> Hi,
> >>>>
> >>>> whenever I try to test a reliable connection using "ibv_rc_pingpong -d
> >>>> mlx5_0"
> >>> ibv_rc_pingpong -d rxe0 -g 1 > /dev/null &
> >>>
> >>> ibv_rc_pingpong -d rxe0 -g 1 192.168.1.2
> >>>
> >>> I made tests with the above. It can work well.
> >>>
> >>> Normally "-g" is needed.
> >> "-g" is needed because you are running RoCE, while Filip is running IB.
> >>
> >>> Before directly using mlx5, please make tests with softroce firstly.
> >> Are you sure that RXE works in 4.18.0-151.el8.x86_64 kernel?
> > I have no 4.18.0-151.el8.x86_64 kernel at hand.
> > I use the kernel 4.18.0-240.10.1.el8_3.x86_64 to make tests.
> >
> > SoftRoCE can work well.
> >
> > Zhu Yanjun
> >
> >>> Zhu Yanjun
> >>>
> >>> to start the server side the test immediately stops with
> >>>> "Couldn't create CQ". Since I couldn't find a solution for this problem
> >>>> in one week I would like to ask if someone has encountered this error
> >>>> before or knows a way of troubleshooting it. Just to be sure I updated
> >>>> the controller to its latest firmware (16.29.2002) today, but the error
> >>>> remained the same.
> >>>>
> >>>> System Information
> >>>> ====
> >>>>
> >>>> * CentOS Linux release 8.1.1911 (Core)
> >>>> * Linux 4.18.0-151.el8.x86_64
> >>>> * ConnectX-5 (MCX555A-ECA)
> >>>>
> >>>>
> >>>> Installed Packages
> >>>> ====
> >>>>
> >>>> * rdma-core-32.0-4.el8.x86_64
> >>>> * libibverbs-32.0-4.el8.x86_64
> >>>>
> >>>>
> >>>> Loaded Kernel Modules (lsmod | grep -E 'rdma|mlx')
> >>>> ====
> >>>>
> >>>> rpcrdma 274432 0
> >>>> sunrpc 454656 22
> >>>> rpcrdma,nfsv4,auth_rpcgss,lockd,nfsv3,rpcsec_gss_krb5,nfs_acl,nfs
> >>>> rdma_ucm 32768 0
> >>>> rdma_cm 69632 5
> >>>> rpcrdma,ib_srpt,ib_iser,ib_isert,rdma_ucm
> >>>> iw_cm 53248 1 rdma_cm
> >>>> ib_cm 57344 3 rdma_cm,ib_ipoib,ib_srpt
> >>>> mlx5_ib 327680 0
> >>>> ib_uverbs 147456 3 i40iw,rdma_ucm,mlx5_ib
> >>>> ib_core 356352 14
> >>>> rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,i40iw,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
> >>>> mlx5_core 798720 1 mlx5_ib
> >>>> mlxfw 24576 1 mlx5_core
> >>>>
> >>>>
> >>>>
> >>>> Infiniband Device Info (ibv_devinfo)
> >>>> ====
> >>>>
> >>>> hca_id: i40iw0
> >>>> transport: iWARP (1)
> >>>> fw_ver: 0.2
> >>>> node_guid: 3cec:ef0d:51c3:0000
> >>>> sys_image_guid: 3cec:ef0d:51c3:0000
> >>>> vendor_id: 0x8086
> >>>> vendor_part_id: 14290
> >>>> hw_ver: 0x0
> >>>> board_id: I40IW Board ID
> >>>> phys_port_cnt: 1
> >>>> port: 1
> >>>> state: PORT_DOWN (1)
> >>>> max_mtu: 4096 (5)
> >>>> active_mtu: 1024 (3)
> >>>> sm_lid: 0
> >>>> port_lid: 1
> >>>> port_lmc: 0x00
> >>>> link_layer: Ethernet
> >>>>
> >>>> hca_id: i40iw1
> >>>> transport: iWARP (1)
> >>>> fw_ver: 0.2
> >>>> node_guid: 3cec:ef0d:51c2:0000
> >>>> sys_image_guid: 3cec:ef0d:51c2:0000
> >>>> vendor_id: 0x8086
> >>>> vendor_part_id: 14290
> >>>> hw_ver: 0x0
> >>>> board_id: I40IW Board ID
> >>>> phys_port_cnt: 1
> >>>> port: 1
> >>>> state: PORT_ACTIVE (4)
> >>>> max_mtu: 4096 (5)
> >>>> active_mtu: 1024 (3)
> >>>> sm_lid: 0
> >>>> port_lid: 1
> >>>> port_lmc: 0x00
> >>>> link_layer: Ethernet
> >>>>
> >>>> hca_id: mlx5_0
> >>>> transport: InfiniBand (0)
> >>>> fw_ver: 16.29.2002
> >>>> node_guid: 0c42:a103:0054:74ca
> >>>> sys_image_guid: 0c42:a103:0054:74ca
> >>>> vendor_id: 0x02c9
> >>>> vendor_part_id: 4119
> >>>> hw_ver: 0x0
> >>>> board_id: MT_0000000010
> >>>> phys_port_cnt: 1
> >>>> port: 1
> >>>> state: PORT_ACTIVE (4)
> >>>> max_mtu: 4096 (5)
> >>>> active_mtu: 4096 (5)
> >>>> sm_lid: 8
> >>>> port_lid: 196
> >>>> port_lmc: 0x00
> >>>> link_layer: InfiniBand
> >>>>
> >>>>
> >>>> Best regards
> >>>> Filip
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-02-24 10:28 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-22 16:19 ibv_rc_pingpong fails to create a completion queue Filip Krakowski
2021-02-23 3:08 ` Zhu Yanjun
2021-02-23 8:56 ` Leon Romanovsky
2021-02-23 9:50 ` Zhu Yanjun
2021-02-23 13:45 ` Filip Krakowski
2021-02-24 10:27 ` Zhu Yanjun
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.