Re: RDMA (smbdirect) testing

From: Namjae Jeon <linkinjeon@kernel.org>
To: Tom Talpey <tom@talpey.com>
Cc: David Howells <dhowells@redhat.com>,
	Long Li <longli@microsoft.com>, Hyunchul Lee <hyc.lee@gmail.com>,
	Steve French <smfrench@gmail.com>,
	CIFS <linux-cifs@vger.kernel.org>
Subject: Re: RDMA (smbdirect) testing
Date: Mon, 23 May 2022 08:06:50 +0900	[thread overview]
Message-ID: <CAKYAXd-AKPyDQCYbQw+eA32MsMqFTFE8Z=iUvb4JOK+pbdiZjA@mail.gmail.com> (raw)
In-Reply-To: <dc1dff3e-d19e-4300-41b8-ccb7459eacde@talpey.com>

2022-05-21 20:54 GMT+09:00, Tom Talpey <tom@talpey.com>:
>
> On 5/20/2022 2:12 PM, David Howells wrote:
>> Tom Talpey <tom@talpey.com> wrote:
>>
>>> SoftROCE is a bit of a hot mess in upstream right now. It's
>>> getting a lot of attention, but it's still pretty shaky.
>>> If you're testing, I'd STRONGLY recommend SoftiWARP.
>>
>> I'm having problems getting that working.  I'm setting the client up
>> with:
>>
>> rdma link add siw0 type siw netdev enp6s0
>> mount //192.168.6.1/scratch /xfstest.scratch -o rdma,user=shares,pass=...
>>
>> and then see:
>>
>> CIFS: Attempting to mount \\192.168.6.1\scratch
>> CIFS: VFS: _smbd_get_connection:1513 warning: device max_send_sge = 6 too
>> small
>> CIFS: VFS: _smbd_get_connection:1516 Queue Pair creation may fail
>> CIFS: VFS: _smbd_get_connection:1519 warning: device max_recv_sge = 6 too
>> small
>> CIFS: VFS: _smbd_get_connection:1522 Queue Pair creation may fail
>> CIFS: VFS: _smbd_get_connection:1559 rdma_create_qp failed -22
>> CIFS: VFS: _smbd_get_connection:1513 warning: device max_send_sge = 6 too
>> small
>> CIFS: VFS: _smbd_get_connection:1516 Queue Pair creation may fail
>> CIFS: VFS: _smbd_get_connection:1519 warning: device max_recv_sge = 6 too
>> small
>> CIFS: VFS: _smbd_get_connection:1522 Queue Pair creation may fail
>> CIFS: VFS: _smbd_get_connection:1559 rdma_create_qp failed -22
>> CIFS: VFS: cifs_mount failed w/return code = -2
>>
>> in dmesg.
>>
>> Problem is, I don't know what to do about it:-/
>
> It looks like the client is hardcoding 16 sge's, and has no option to
> configure a smaller value, or reduce its requested number. That's bad,
> because providers all have their own limits - and SIW_MAX_SGE is 6. I
> thought I'd seen this working (metze?), but either the code changed or
> someone built a custom version.
I also fully agree that we should provide users with the path to
configure this value.
>
> Namjae/Long, have you used siw successfully?
No. I was able to reproduce the same problem that David reported. I
and Hyunchul will take a look. I also confirmed that RDMA work well
without any problems with soft-ROCE. Until this problem is fixed, I'd
like to say David to use soft-ROCE.

> Why does the code require
> 16 sge's, regardless of other size limits? Normally, if the lower layer
> supports fewer, the upper layer will simply reduce its operation sizes.
This should be answered by Long Li. It seems that he set the optimized
value for the NICs he used to implement RDMA in cifs.
>
> Tom.
>