Re: [PATCH v2 4/6] Reduce server smbdirect max send/receive segment sizes

From: Namjae Jeon <linkinjeon@kernel.org>
To: Tom Talpey <tom@talpey.com>
Cc: smfrench@gmail.com, linux-cifs@vger.kernel.org,
	senozhatsky@chromium.org, bmt@zurich.ibm.com,
	longli@microsoft.com, dhowells@redhat.com
Subject: Re: [PATCH v2 4/6] Reduce server smbdirect max send/receive segment sizes
Date: Mon, 26 Sep 2022 10:13:41 +0900	[thread overview]
Message-ID: <CAKYAXd8JiF5N_Ve65=wHPyW+twRT5WOoHH=j6+u0YAAjCV-n2Q@mail.gmail.com> (raw)
In-Reply-To: <11e7b888-460e-efd1-0a8c-3dbf594d9429@talpey.com>

2022-09-26 0:41 GMT+09:00, Tom Talpey <tom@talpey.com>:
> On 9/24/2022 11:40 PM, Namjae Jeon wrote:
>> 2022-09-24 6:53 GMT+09:00, Tom Talpey <tom@talpey.com>:
>>> Reduce ksmbd smbdirect max segment send and receive size to 1364
>>> to match protocol norms. Larger buffers are unnecessary and add
>>> significant memory overhead.
>>>
>>> Signed-off-by: Tom Talpey <tom@talpey.com>
>>> ---
>>>   fs/ksmbd/transport_rdma.c | 4 ++--
>>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/fs/ksmbd/transport_rdma.c b/fs/ksmbd/transport_rdma.c
>>> index 494b8e5af4b3..0315bca3d53b 100644
>>> --- a/fs/ksmbd/transport_rdma.c
>>> +++ b/fs/ksmbd/transport_rdma.c
>>> @@ -62,13 +62,13 @@ static int smb_direct_receive_credit_max = 255;
>>>   static int smb_direct_send_credit_target = 255;
>>>
>>>   /* The maximum single message size can be sent to remote peer */
>>> -static int smb_direct_max_send_size = 8192;
>>> +static int smb_direct_max_send_size = 1364;
>>>
>>>   /*  The maximum fragmented upper-layer payload receive size supported
>>> */
>>>   static int smb_direct_max_fragmented_recv_size = 1024 * 1024;
>>>
>>>   /*  The maximum single-message size which can be received */
>>> -static int smb_direct_max_receive_size = 8192;
>>> +static int smb_direct_max_receive_size = 1364;
>> Can I know what value windows server set to ?
>>
>> I can see the following settings for them in MS-SMBD.pdf
>> Connection.MaxSendSize is set to 1364.
>> Connection.MaxReceiveSize is set to 8192.
>
> Glad you asked, it's an interesting situation IMO.
>
> In MS-SMBD, the following are documented as behavior notes:
>
> Client-side (active connect):
>   Connection.MaxSendSize is set to 1364.
>   Connection.MaxReceiveSize is set to 8192.
>
> Server-side (passive listen):
>   Connection.MaxSendSize is set to 1364.
>   Connection.MaxReceiveSize is set to 8192.
>
> However, these are only the initial values. During SMBD
> negotiation, the two sides adjust downward to the other's
> maximum. Therefore, Windows connecting to Windows will use
> 1364 on both sides.
>
> In cifs and ksmbd, the choices were messier:
>
> Client-side smbdirect.c:
>   int smbd_max_send_size = 1364;
>   int smbd_max_receive_size = 8192;
>
> Server-side transport_rdma.c:
>   static int smb_direct_max_send_size = 8192;
>   static int smb_direct_max_receive_size = 8192;
>
> Therefore, peers connecting to ksmbd would typically end up
> negotiating 1364 for send and 8192 for receive.
>
> There is almost no good reason to use larger buffers. Because
> RDMA is highly efficient, and the smbdirect protocol trivially
> fragments longer messages, there is no significant performance
> penalty.
>
> And, because not many SMB3 messages require 8192 bytes over
> smbdirect, it's a colossal waste of virtually contiguous kernel
> memory to allocate 8192 to all receives.
>
> By setting all four to the practical reality of 1364, it's a
> consistent and efficient default, and aligns Linux smbdirect
> with Windows.
Thanks for your detailed explanation!  Agree to set both to 1364 by
default, Is there any usage to increase it? I wonder if users need any
configuration parameters to adjust them.

>
> Tom.
>
>>
>> Why does the specification describe setting it to 8192?
>>>
>>>   static int smb_direct_max_read_write_size = SMBD_DEFAULT_IOSIZE;
>>>
>>> --
>>> 2.34.1
>>>
>>>
>>
>