Re: [PATCH for-rc v3 0/6] RDMA/rxe: Various bug fixes.

From: Bart Van Assche <bvanassche@acm.org>
To: Bob Pearson <rpearsonhpe@gmail.com>,
	"Pearson, Robert B" <robert.pearson2@hpe.com>,
	"jgg@nvidia.com" <jgg@nvidia.com>,
	"zyjzyj2000@gmail.com" <zyjzyj2000@gmail.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"mie@igel.co.jp" <mie@igel.co.jp>
Subject: Re: [PATCH for-rc v3 0/6] RDMA/rxe: Various bug fixes.
Date: Fri, 10 Sep 2021 15:07:52 -0700	[thread overview]
Message-ID: <918787c7-de06-ef67-80ac-ae2e7643dd61@acm.org> (raw)
In-Reply-To: <cf358367-5cb8-0482-28e6-993a4f6bb047@gmail.com>

On 9/10/21 2:47 PM, Bob Pearson wrote:
> OK I checked out the kernel with the SHA number above and applied the patch series
> and rebuilt and reinstalled the kernel. I checked out v36.0 of rdma-core and rebuilt
> that. rdma is version 5.9.0 but I doubt that will have any effect. My startup script
> is
> 
>      export LD_LIBRARY_PATH=/home/bob/src/rdma-core/build/lib/:/usr/local/lib:/usr/lib
> 
> 
> 
>      sudo ip link set dev enp0s3 mtu 8500
> 
>      sudo ip addr add dev enp0s3 fe80::0a00:27ff:fe94:8a69/64
> 
>      sudo rdma link add rxe0 type rxe netdev enp0s3
> 
> 
> I am running on a Virtualbox VM instance of Ubuntu 21.04 with 20 cores and 8GB of RAM.
> 
> The test looks like
> 
>      sudo ./check -q srp/001
> 
>      srp/001 (Create and remove LUNs)                             [passed]
> 
>          runtime  1.174s  ...  1.236s
> 
> There were no issues.
> 
> Any guesses what else to look at?

The test I ran is different. I did not run any of the ip link / ip addr /
rdma link commands since the blktests scripts already run the rdma link
command. The bug I reported in my previous email is reproducible and
triggers a VM halt.

Are we using the same kernel config? I attached my kernel config to my
previous email. The source code location of the crash address is as
follows:

(gdb) list *(rxe_completer+0x96d)
0x228d is in rxe_completer (drivers/infiniband/sw/rxe/rxe_comp.c:149).
144              */
145             wqe = queue_head(qp->sq.queue, QUEUE_TYPE_FROM_CLIENT);
146             *wqe_p = wqe;
147
148             /* no WQE or requester has not started it yet */
149             if (!wqe || wqe->state == wqe_state_posted)
150                     return pkt ? COMPST_DONE : COMPST_EXIT;
151
152             /* WQE does not require an ack */
153             if (wqe->state == wqe_state_done)

The disassembly output is as follows:

drivers/infiniband/sw/rxe/rxe_comp.c:
149             if (!wqe || wqe->state == wqe_state_posted)
    0x0000000000002277 <+2391>:  test   %r12,%r12
    0x000000000000227a <+2394>:  je     0x2379 <rxe_completer+2649>
    0x0000000000002280 <+2400>:  lea    0x94(%r12),%rdi
    0x0000000000002288 <+2408>:  call   0x228d <rxe_completer+2413>
    0x000000000000228d <+2413>:  mov    0x94(%r12),%eax
    0x0000000000002295 <+2421>:  test   %eax,%eax
    0x0000000000002297 <+2423>:  je     0x237c <rxe_completer+2652>

So the instruction that triggers the crash is "mov 0x94(%r12),%eax".
Does consumer_addr() perhaps return an invalid address under certain
circumstances?

Thanks,

Bart.