Re: 答复: 答复: 答复: [PATCH] page_pool: mark unbound node page as reusable pages

[Yunsheng Lin <linyunsheng@huawei.com>]

>> -----邮件原件-----
>> 发件人: Yunsheng Lin [mailto:linyunsheng@huawei.com]
>> 发送时间: 2019年12月5日 10:06
>> 收件人: Li,Rongqing <lirongqing@baidu.com>; netdev@vger.kernel.org;
>> saeedm@mellanox.com
>> 主题: Re: 答复: 答复: [PATCH] page_pool: mark unbound node page as
>> reusable pages
>>
>> On 2019/12/5 9:55, Li,Rongqing wrote:
>>>
>>>
>>>> -----邮件原件-----
>>>> 发件人: Yunsheng Lin [mailto:linyunsheng@huawei.com]
>>>> 发送时间: 2019年12月5日 9:44
>>>> 收件人: Li,Rongqing <lirongqing@baidu.com>; netdev@vger.kernel.org;
>>>> saeedm@mellanox.com
>>>> 主题: Re: 答复: [PATCH] page_pool: mark unbound node page as reusable
>>>> pages
>>>>
>>>> On 2019/12/5 9:08, Li,Rongqing wrote:
>>>>>
>>>>>
>>>>>> -----邮件原件-----
>>>>>> 发件人: Yunsheng Lin [mailto:linyunsheng@huawei.com]
>>>>>> 发送时间: 2019年12月5日 8:55
>>>>>> 收件人: Li,Rongqing <lirongqing@baidu.com>; netdev@vger.kernel.org;
>>>>>> saeedm@mellanox.com
>>>>>> 主题: Re: [PATCH] page_pool: mark unbound node page as reusable
>> pages
>>>>>>
>>>>>> On 2019/12/4 18:14, Li RongQing wrote:
>>>>>>> some drivers uses page pool, but not require to allocate page from
>>>>>>> bound node, so pool.p.nid is NUMA_NO_NODE, and this fixed patch
>>>>>>> will block this kind of driver to recycle
>>>>>>>
>>>>>>> Fixes: d5394610b1ba ("page_pool: Don't recycle non-reusable
>>>>>>> pages")
>>>>>>> Signed-off-by: Li RongQing <lirongqing@baidu.com>
>>>>>>> Cc: Saeed Mahameed <saeedm@mellanox.com>
>>>>>>> ---
>>>>>>>  net/core/page_pool.c | 4 +++-
>>>>>>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/net/core/page_pool.c b/net/core/page_pool.c index
>>>>>>> a6aefe989043..4054db683178 100644
>>>>>>> --- a/net/core/page_pool.c
>>>>>>> +++ b/net/core/page_pool.c
>>>>>>> @@ -317,7 +317,9 @@ static bool __page_pool_recycle_direct(struct
>>>>>>> page
>>>>>> *page,
>>>>>>>   */
>>>>>>>  static bool pool_page_reusable(struct page_pool *pool, struct
>>>>>>> page
>>>>>>> *page)  {
>>>>>>> -	return !page_is_pfmemalloc(page) && page_to_nid(page) ==
>>>> pool->p.nid;
>>>>>>> +	return !page_is_pfmemalloc(page) &&
>>>>>>> +		(page_to_nid(page) == pool->p.nid ||
>>>>>>> +		 pool->p.nid == NUMA_NO_NODE);
>>>>>>
>>>>>> If I understand it correctly, you are allowing recycling when
>>>>>> pool->p.nid is NUMA_NO_NODE, which does not seems match the
>> commit
>>>>>> log: "this fixed patch will block this kind of driver to recycle".
>>>>>>
>>>>>> Maybe you mean "commit d5394610b1ba" by this fixed patch?
>>>>>
>>>>> yes
>>>>>
>>>>>>
>>>>>> Also, maybe it is better to allow recycling if the below condition is
>> matched:
>>>>>>
>>>>>> 	pool->p.nid == NUMA_NO_NODE && page_to_nid(page) ==
>>>>>> numa_mem_id()
>>>>>
>>>>> If driver uses NUMA_NO_NODE, it does not care numa node, and maybe
>>>>> its platform Only has a node, so not need to compare like
>>>>> "page_to_nid(page) ==
>>>> numa_mem_id()"
>>>>
>>>> Normally, driver does not care if the node of a device is
>>>> NUMA_NO_NODE or not, it just uses the node that returns from
>> dev_to_node().
>>>>
>>>> Even for multi node system, the node of a device may be NUMA_NO_NODE
>>>> when BIOS/FW has not specified it through ACPI/DT, see [1].
>>>>
>>>>
>>>> [1] https://lore.kernel.org/patchwork/patch/1141952/
>>>>
>>>
>>> at this condition, page can be allocated from any node from driver
>>> boot, why need to check "page_to_nid(page) == numa_mem_id()" at recycle?
>>
>> For performance, the performance is better when the rx page is on the same
>> node as the rx process is running.
>>
>> We want the node of rx page is close to the node of device/cpu to achive better
>> performance, since the node of device is unknown, maybe we choose the node
>> of memory that is close to the cpu that is running to handle the rx cleaning.
>>
> 
> if the driver takes care about numa node, it should not assign NUMA_NO_NODE, it should
> assign a detail numa node at starting step. Not depend on recycle to decide the numa
> node

How and where we should handle the NUMA_NO_NODE has been discussed before,
see [1].
but the driver has not been considered the place to handle it.

For driver that has not using page pool, the numa_mem_id() checking
is how they decide to recycle or not, see [2] [3].

I think it is better to be consistent with the page pool too.


[1] https://lore.kernel.org/patchwork/patch/1125789/
[2] https://elixir.bootlin.com/linux/v5.4.2/source/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c#L2437
[3] https://elixir.bootlin.com/linux/v5.4.2/source/drivers/net/ethernet/intel/i40e/i40e_txrx.c#L1858
> 
> -RongQing
> 
> 
>>>
>>> -Li
>>>
>>>>>
>>>>>
>>>>> -RongQing
>>>>>
>>>>>
>>>>>>
>>>>>>>  }
>>>>>>>
>>>>>>>  void __page_pool_put_page(struct page_pool *pool, struct page
>>>>>>> *page,
>>>>>>>
>>>>>
>>>
>