Re: v5.14 RXE driver broken?

From: Bart Van Assche <bvanassche@acm.org>
To: Jason Gunthorpe <jgg@ziepe.ca>, Zhu Yanjun <zyjzyj2000@gmail.com>
Cc: Bob Pearson <rpearsonhpe@gmail.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>
Subject: Re: v5.14 RXE driver broken?
Date: Wed, 25 Aug 2021 11:22:19 -0700	[thread overview]
Message-ID: <423ed740-96e7-aebb-3e6f-7416108f5a62@acm.org> (raw)
In-Reply-To: <20210825163219.GY543798@ziepe.ca>

On 8/25/21 9:32 AM, Jason Gunthorpe wrote:
> On Wed, Aug 25, 2021 at 11:02:14AM +0800, Zhu Yanjun wrote:
>> On Tue, Aug 24, 2021 at 11:02 AM Bart Van Assche <bvanassche@acm.org> wrote:
>>>
>>> Hi Bob,
>>>
>>> If I run the following test against Linus' master branch then that test
>>> passes (commit d5ae8d7f85b7 ("Revert "media: dvb header files: move some
>>> headers to staging"")):
>>>
>>> # export use_siw=1 && modprobe brd && (cd blktests && ./check -q srp/002)
>>> srp/002 (File I/O on top of multipath concurrently with logout and login (mq)) [passed]
>>>      runtime    ...  48.849s
>>>
>>> The following test fails:
>>>
>>> # export use_siw= && modprobe brd && (cd blktests && ./check -q srp/002)
>>> srp/002 (File I/O on top of multipath concurrently with logout and login (mq)) [failed]
>>>      runtime  48.849s  ...  15.024s
>>>      +++ /home/bart/software/blktests/results/nodev/srp/002.out.bad      2021-08-23 19:51:05.182958728 -0700
>>>      @@ -1,2 +1 @@
>>>       Configured SRP target driver
>>>      -Passed
>>
>> Can this commit "RDMA/rxe: Zero out index member of struct rxe_queue"
>> in the link https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/commit/?h=wip/jgg-for-rc
>> fix this problem?
>>
>> And the commit will be merged into linux upstream very soon.
> 
> Please let me know Bart, if the rxe driver is still broken I will
> definitely punt all the changes for RXE to the next cycle until it can
> be fixed.

Hi Jason,

Thanks for having offered to revert the RXE changes from this merge window.
Unfortunately that wouldn't be sufficient. My test results so far for test
srp/002 in combination with the rdma_rxe driver are as follows:
* Kernel v5.12: test passes.
* Kernel v5.13: test fails.
* Kernel v5.14-rc7: test fails.

For the rdma_rxe tests for kernel v5.14-rc7 I found the following in the kernel
log:

ib_srp:add_target_store: ib_srp: max_sectors = 1024; max_pages_per_mr = 512; mr_page_size = 4096; max_sectors_per_mr = 4096; mr_per_cmd = 2
ib_srp: enp1s0_rxe: ib_alloc_mr() failed. Try to reduce max_cmd_per_lun, max_sect or ch_count

There is sufficient memory available in the VM in which I ran the tests. It is
not clear to me why ib_alloc_mr() fails with these parameters when using the
rdma_rxe driver? As one can see in srp_alloc_fr_pool() the SRP initiator driver
respects the max_pages_per_mr RDMA driver limit.

Thanks,

Bart.