From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: [PATCH net] mlx4_core: restore optimal ICM memory allocation Date: Wed, 30 May 2018 17:30:18 -0400 Message-ID: References: <20180530041152.113393-1-edumazet@google.com> <2cd40c9e-16a5-ba96-db7c-3aafc5af0957@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: In-Reply-To: <2cd40c9e-16a5-ba96-db7c-3aafc5af0957@oracle.com> Sender: netdev-owner@vger.kernel.org To: Qing Huang Cc: David Miller , netdev , Eric Dumazet , John Sperbeck , Tarick Bedeir , Daniel Jurgens , Zhu Yanjun , Tariq Toukan , linux-rdma , Santosh Shilimkar List-Id: linux-rdma@vger.kernel.org On Wed, May 30, 2018 at 5:08 PM Qing Huang wrote: > > > > On 5/30/2018 1:50 PM, Eric Dumazet wrote: > > On Wed, May 30, 2018 at 4:30 PM Qing Huang wrote: > >> > >> > >> On 5/29/2018 9:11 PM, Eric Dumazet wrote: > >>> Commit 1383cb8103bb ("mlx4_core: allocate ICM memory in page size chunks") > >>> brought a regression caught in our regression suite, thanks to KASAN. > >> If KASAN reported issue was really caused by smaller chunk sizes, > >> changing allocation > >> order dynamically will eventually hit the same issue. > > Sigh, you have little idea of what your patch really did... > > > > The KASAN part only shows the tip of the iceberg, but our main concern > > is an increase of memory overhead. > > Well, the commit log only mentioned KASAN and but the change here didn't > seem to solve > the issue. Can you elaborate ? My patch solves our problems. Both the memory overhead and KASAN splats are gone. > > > > > Alternative is to revert your patch, since we are now very late in 4.17 cycle. > > > > Memory usage has grown a lot with your patch, since each 4KB page needs a full > > struct mlx4_icm_chunk (256 bytes of overhead !) > > Going to smaller chunks will have some overhead. It depends on the > application though. > What's the total increased memory consumption in your env? As I explained, your patch adds 256 bytes of overhead per 4KB. Your changelog did not mentioned that at all, and we discovered this the hard way. That is pretty intolerable, and is a blocker for us, memory is precious.