Re: [RFC Patch v1 1/3] net: ena: implement local page cache (LPC) system

[Shay Agroskin <shayagr@amazon.com>]

Eric Dumazet <eric.dumazet@gmail.com> writes:

> On 3/9/21 6:10 PM, Shay Agroskin wrote:
>> The page cache holds pages we allocated in the past during napi 
>> cycle,
>> and tracks their availability status using page ref count.
>> 
>> The cache can hold up to 2048 pages. Upon allocating a page, we 
>> check
>> whether the next entry in the cache contains an unused page, 
>> and if so
>> fetch it. If the next page is already used by another entity or 
>> if it
>> belongs to a different NUMA core than the napi routine, we 
>> allocate a
>> page in the regular way (page from a different NUMA core is 
>> replaced by
>> the newly allocated page).
>> 
>> This system can help us reduce the contention between different 
>> cores
>> when allocating page since every cache is unique to a queue.
>
> For reference, many drivers already use a similar strategy.
>
>> +
>> +/* Fetch the cached page (mark the page as used and pass it to 
>> the caller).
>> + * If the page belongs to a different NUMA than the current 
>> one, free the cache
>> + * page and allocate another one instead.
>> + */
>> +static struct page *ena_fetch_cache_page(struct ena_ring 
>> *rx_ring,
>> +					 struct ena_page 
>> *ena_page,
>> +					 dma_addr_t *dma,
>> +					 int current_nid)
>> +{
>> +	/* Remove pages belonging to different node than 
>> current_nid from cache */
>> +	if (unlikely(page_to_nid(ena_page->page) != current_nid)) 
>> {
>> + 
>> ena_increase_stat(&rx_ring->rx_stats.lpc_wrong_numa, 1, 
>> &rx_ring->syncp);
>> +		ena_replace_cache_page(rx_ring, ena_page);
>> +	}
>> +
>> 
>
> And they use dev_page_is_reusable() instead of copy/pasting this 
> logic.
>
> As a bonus, they properly deal with pfmemalloc

Thanks for reviewing it, and sorry for the time it took me to 
reply, I wanted to test some of the suggestions given me here 
before replying.

Providing that this patchset would still be necessary for our 
driver after testing the patchset Saeed suggested, I will switch 
to using dev_page_is_reusable() instead of this expression.

Shay