From: Dongli Zhang <dongli.zhang@oracle.com>
To: Matthew Wilcox <willy@infradead.org>
Cc: linux-mm@kvack.org, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, akpm@linux-foundation.org,
davem@davemloft.net, kuba@kernel.org,
aruna.ramakrishna@oracle.com, bert.barbe@oracle.com,
rama.nichanamatlu@oracle.com, venkat.x.venkatsubra@oracle.com,
manjunath.b.patil@oracle.com, joe.jin@oracle.com,
srinivas.eeda@oracle.com
Subject: Re: [PATCH 1/1] mm: avoid re-using pfmemalloc page in page_frag_alloc()
Date: Tue, 3 Nov 2020 13:37:45 -0800 [thread overview]
Message-ID: <7fc532c3-8780-4484-6932-6bfabecda189@oracle.com> (raw)
In-Reply-To: <20201103211541.GH27442@casper.infradead.org>
On 11/3/20 1:15 PM, Matthew Wilcox wrote:
> On Tue, Nov 03, 2020 at 12:57:33PM -0800, Dongli Zhang wrote:
>> On 11/3/20 12:35 PM, Matthew Wilcox wrote:
>>> On Tue, Nov 03, 2020 at 11:32:39AM -0800, Dongli Zhang wrote:
>>>> However, once kernel is not under memory pressure any longer (suppose large
>>>> amount of memory pages are just reclaimed), the page_frag_alloc() may still
>>>> re-use the prior pfmemalloc page_frag_cache->va to allocate skb->data. As a
>>>> result, the skb->pfmemalloc is always true unless page_frag_cache->va is
>>>> re-allocated, even the kernel is not under memory pressure any longer.
>>>> + /*
>>>> + * Try to avoid re-using pfmemalloc page because kernel may already
>>>> + * run out of the memory pressure situation at any time.
>>>> + */
>>>> + if (unlikely(nc->va && nc->pfmemalloc)) {
>>>> + page = virt_to_page(nc->va);
>>>> + __page_frag_cache_drain(page, nc->pagecnt_bias);
>>>> + nc->va = NULL;
>>>> + }
>>>
>>> I think this is the wrong way to solve this problem. Instead, we should
>>> use up this page, but refuse to recycle it. How about something like this (not even compile tested):
>>
>> Thank you very much for the feedback. Yes, the option is to use the same page
>> until it is used up (offset < 0). Instead of recycling it, the kernel free it
>> and allocate new one.
>>
>> This depends on whether we will tolerate the packet drop until this page is used up.
>>
>> For virtio-net, the payload (skb->data) is of size 128-byte. The padding and
>> alignment will finally make it as 512-byte.
>>
>> Therefore, for virtio-net, we will have at most 4096/512-1=7 packets dropped
>> before the page is used up.
>
> My thinking is that if the kernel is under memory pressure then freeing
> the page and allocating a new one is likely to put even more strain
> on the memory allocator, so we want to do this "soon", rather than at
> each allocation.
>
> Thanks for providing the numbers. Do you think that dropping (up to)
> 7 packets is acceptable?>
> We could also do something like ...
>
> if (unlikely(nc->pfmemalloc)) {
> page = alloc_page(GFP_NOWAIT | __GFP_NOWARN);
> if (page)
> nc->pfmemalloc = 0;
> put_page(page);
> }
>
> to test if the memory allocator has free pages at the moment. Not sure
> whether that's a good idea or not -- hopefully you have a test environment
> set up where you can reproduce this condition on demand and determine
> which of these three approaches is best!
>
From mm's perspective, we expect to reduce the number of page allocation
(especially under memory pressure).
From networking's perspective, we expect to reduce the number of skb drop.
That's why I CCed netdev folks (including David and Jakub), although the patch
is for mm/page_alloc.c. The page_frag_alloc() is primarily used by networking
and nvme-tcp.
Unfortunately, so far I do not have the env to reproduce. I reproduced with a
patch to fail page allocation and set nc->pfmemalloc on purpose.
From mm's perspective, I think to use up the page is a good option. Indeed tt is
system administrator's duty to avoid memory pressure, in order to avoid the
extra packet drops.
Dongli Zhang
next prev parent reply other threads:[~2020-11-03 21:38 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-03 19:32 [PATCH 1/1] mm: avoid re-using pfmemalloc page in page_frag_alloc() Dongli Zhang
2020-11-03 20:35 ` Matthew Wilcox
2020-11-03 20:57 ` Dongli Zhang
2020-11-03 21:15 ` Matthew Wilcox
2020-11-03 21:37 ` Dongli Zhang [this message]
2020-11-04 1:16 ` Rama Nichanamatlu
2020-11-04 8:50 ` Eric Dumazet
2020-11-04 12:36 ` Matthew Wilcox
2020-11-04 12:51 ` Eric Dumazet
2020-11-04 16:41 ` Dongli Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7fc532c3-8780-4484-6932-6bfabecda189@oracle.com \
--to=dongli.zhang@oracle.com \
--cc=akpm@linux-foundation.org \
--cc=aruna.ramakrishna@oracle.com \
--cc=bert.barbe@oracle.com \
--cc=davem@davemloft.net \
--cc=joe.jin@oracle.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=manjunath.b.patil@oracle.com \
--cc=netdev@vger.kernel.org \
--cc=rama.nichanamatlu@oracle.com \
--cc=srinivas.eeda@oracle.com \
--cc=venkat.x.venkatsubra@oracle.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).