Re: [PATCH net-next] net/smc: Use kvzalloc for allocating smc_link_group

From: Karsten Graul <kgraul@linux.ibm.com>
To: Tony Lu <tonylu@linux.alibaba.com>
Cc: kuba@kernel.org, davem@davemloft.net, netdev@vger.kernel.org,
	linux-s390@vger.kernel.org
Subject: Re: [PATCH net-next] net/smc: Use kvzalloc for allocating smc_link_group
Date: Fri, 21 Jan 2022 12:06:56 +0100	[thread overview]
Message-ID: <c5873d85-d791-319b-e3a1-86abda204b45@linux.ibm.com> (raw)
In-Reply-To: <YeoncJZoa3ELWyxM@TonyMac-Alibaba>

On 21/01/2022 04:24, Tony Lu wrote:
> On Thu, Jan 20, 2022 at 03:50:26PM +0100, Karsten Graul wrote:
>> On 20/01/2022 15:09, Tony Lu wrote:
>>> When analyzed memory usage of SMC, we found that the size of struct
>>> smc_link_group is 16048 bytes, which is too big for a busy machine to
>>> allocate contiguous memory. Using kvzalloc instead that falls back to
>>> vmalloc if there has not enough contiguous memory.
>>
>> I am wondering where the needed contiguous memory for the required RMB buffers should come from when 
>> you don't even get enough storage for the initial link group?
> 
> Yes, this is what I want to talking about. The RMB buffers size inherits
> from TCP, we cannot assume that RMB is always larger than 16k bytes, the
> tcp_mem can be changed on the fly, and it can be tuned to very small for
> saving memory. Also, If we freed existed link group or somewhere else,
> we can allocate enough contiguous memory for the new link group.

The lowest size for an RMB is 16kb, smaller inherited tcp sizes do not apply here.
> 
>> The idea is that when the system is so low on contiguous memory then a link group creation should fail 
>> early, because most of the later buffer allocations will also fail then later.
> 
> IMHO, it is not a "pre-checker" for allocating buffer, it is a reminder
> for us to save contiguous memory, this is a precious resource, and a
> possible way to do this. This patch is not the best approach to solve
> this problem, but the simplest one. A possible approach to allocate
> link array in link group with a pointer to another memory. Glad to hear
> your advice.

I am still not fully convinced of this change. It does not harm and the overhead of
a vmalloc() is acceptable because a link group is not created so often. But since
kvzmalloc() will first try to use normal kmalloc() and if that fails switch to the
(more expensive) vmalloc() this will not _save_ any contiguous memory.
And for the subsequent required allocations of at least one RMB we need another 16KB.

Did this change had any measurable advantages in your tests?