Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt update

From: Aruna Ramakrishna <aruna.ramakrishna@oracle.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>,
	leon@kernel.org, dledford@redhat.com, linux-rdma@vger.kernel.org,
	linux-kernel@vger.kernel.org,
	Rajesh Sivaramasubramaniom 
	<rajesh.sivaramasubramaniom@oracle.com>,
	Rama Nichanamatlu <rama.nichanamatlu@oracle.com>,
	Jeffery Yoder <jeffery.yoder@oracle.com>
Subject: Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt update
Date: Tue, 23 Mar 2021 21:27:38 -0700	[thread overview]
Message-ID: <0DFF7518-8818-445B-94AC-8EB2096446BE@oracle.com> (raw)
In-Reply-To: <20210323231321.GF2710221@ziepe.ca>

> On Mar 23, 2021, at 4:13 PM, Jason Gunthorpe <jgg@ziepe.ca> wrote:
> 
> On Tue, Mar 23, 2021 at 12:41:51PM -0700, Aruna Ramakrishna wrote:
>>   There is a far greater possibility of an order-8 allocation failing,
>>   esp. with the addition of __GFP_NORETRY , and the code would have to
>>   fall back to a lower order allocation more often than not (esp. on a
>>   long running system). Unless the performance gains from using order-8
>>   pages is significant (and it does not seem that way to me), we can just
>>   skip this step and directly go to the lower order allocation.
> 
> Do not send HTML mails.

I apologize; I’ve fixed the setting now.

> 
> Do you have benchmarks that show the performance of the high order
> pages is not relavent? I'm a bit surprised to hear that
> 

I guess my point was more to the effect that an order-8 alloc will fail more often than not, in this flow. For instance, when we were debugging the latency spikes here, this was the typical buddyinfo output on that system:

Node 0, zone      DMA      0      1      1      2      3      0      1      0      1      1      3 
Node 0, zone    DMA32      7      7      7      6     10      2      6      7      6      2    306 
Node 0, zone   Normal   3390  51354  17574   6556   1586     26      2      1      0      0      0 
Node 1, zone   Normal  11519  23315  23306   9738     73      2      0      1      0      0      0 

I think this level of fragmentation is pretty normal on long running systems. Here, in the reg_mr flow, the first try (order-8) alloc will probably fail 9 times out of 10 (esp. after the addition of GFP_NORETRY flag), and then as fallback, the code tries to allocate a lower order, and if that too fails, it allocates a page. I think it makes sense to just avoid trying an order-8 alloc here.

Thanks,
Aruna

> This code really needs some attention to use a proper
> scatter/gather. I understand the chip can do it, just some of the
> software layers need to be stripped away so it can form the right SGL
> in the HW.
> 
> Jason