Re: [RFC 2/2] page_frag_cache: Store metadata in struct page

[Alexander Duyck <alexander.duyck@gmail.com>]

On Thu, Mar 15, 2018 at 12:53 PM, Matthew Wilcox <willy@infradead.org> wrote:
> From: Matthew Wilcox <mawilcox@microsoft.com>
>
> Shrink page_frag_cache from 24 to 8 bytes (a single pointer to the
> currently-in-use struct page) by using the page's refcount directly
> (instead of maintaining a bias) and storing our current progress through
> the page in the same bits currently used for page->index.  We no longer
> need to reflect the page pfmemalloc state if we're storing the page
> directly.
>
> On the downside, we now call page_address() on every allocation, and we
> do an atomic_inc() rather than a non-atomic decrement, but we should
> touch the same number of cachelines and there is far less code (and
> the code is less complex).
>
> Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>

So my concern with this patch is that it is essentially trading off
CPU performance for reduced size. One of the reasons for getting away
from using the page pointer is because it is expensive to access the
page when the ref_count is bouncing on multiple cache lines. In
addition converting from a page to a virtual address is actually more
expensive then you would think it should be. I went through and
optimized that the best I could, but there is still a pretty
significant cost to the call.

I won't be able to test the patches until next week, but I expect I
will probably see a noticeable regression when performing a small
packet routing test.

- Alex