Re: [PATCH rdma-next 8/9] RDMA: Globally allocate and release QP memory

From: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
To: Leon Romanovsky <leon@kernel.org>, Gal Pressman <galpress@amazon.com>
Cc: Doug Ledford <dledford@redhat.com>,
	Jason Gunthorpe <jgg@nvidia.com>,
	Adit Ranadive <aditr@vmware.com>,
	Ariel Elior <aelior@marvell.com>,
	Bernard Metzler <bmt@zurich.ibm.com>,
	Christian Benvenuti <benve@cisco.com>,
	linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
	Michal Kalderon <mkalderon@marvell.com>,
	Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>,
	Mustafa Ismail <mustafa.ismail@intel.com>,
	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>,
	Nelson Escobar <neescoba@cisco.com>,
	Potnuri Bharat Teja <bharat@chelsio.com>,
	Selvin Xavier <selvin.xavier@broadcom.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	Steve Wise <larrystevenwise@gmail.com>,
	VMware PV-Drivers <pv-drivers@vmware.com>,
	Weihang Li <liweihang@huawei.com>,
	Wenpeng Liang <liangwenpeng@huawei.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Zhu Yanjun <zyjzyj2000@gmail.com>
Subject: Re: [PATCH rdma-next 8/9] RDMA: Globally allocate and release QP memory
Date: Wed, 21 Jul 2021 14:05:34 -0400	[thread overview]
Message-ID: <5539c203-0d3b-6296-7554-143e7afb6e34@cornelisnetworks.com> (raw)
In-Reply-To: <YPaKu4ppS0Bz6fW1@unreal>

On 7/20/21 4:35 AM, Leon Romanovsky wrote:
> On Mon, Jul 19, 2021 at 04:42:11PM +0300, Gal Pressman wrote:
>> On 18/07/2021 15:00, Leon Romanovsky wrote:
>>> From: Leon Romanovsky <leonro@nvidia.com>
>>>
>>> Convert QP object to follow IB/core general allocation scheme.
>>> That change allows us to make sure that restrack properly kref
>>> the memory.
>>>
>>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
>>
>> EFA and core parts look good to me.
>> Reviewed-by: Gal Pressman <galpress@amazon.com>
>> Tested-by: Gal Pressman <galpress@amazon.com>

Leon, I pulled your tree and tested, things look good so far.

For rdmavt and core:
Reviewed-by: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Tested-by: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>

> Thanks a lot.
> 
>>
>>> +static inline void *rdma_zalloc_obj(struct ib_device *dev, size_t size,
>>> +				    gfp_t gfp, bool is_numa_aware)
>>> +{
>>> +	if (is_numa_aware && dev->ops.get_numa_node)
>>
>> Honestly I think it's better to return an error if a numa aware allocation is
>> requested and get_numa_node is not provided.
> 
> We don't want any driver to use and implement ".get_numa_node()" callback.
> 
> Initially, I thought about adding WARN_ON(driver_id != HFI && .get_numa_node)
> to the device.c, but decided to stay with comment in ib_verbs.h only.

Maybe you could update that comment and add that it's for performance? This way
its clear we are different for a reason. I'd be fine adding a WARN_ON_ONCE like
you mention here. I don't think we need to fail the call but drawing attention
to it would not necessarily be a bad thing. Either way, RB/TB for me stands.

-Denny