From: Nitesh Narayan Lal <nitesh@redhat.com>
To: David Hildenbrand <david@redhat.com>, Michal Hocko <mhocko@kernel.org>
Cc: "Michael S. Tsirkin" <mst@redhat.com>,
Alexander Duyck <alexander.h.duyck@linux.intel.com>,
Alexander Duyck <alexander.duyck@gmail.com>,
virtio-dev@lists.oasis-open.org, kvm list <kvm@vger.kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>,
Dave Hansen <dave.hansen@intel.com>,
LKML <linux-kernel@vger.kernel.org>,
Matthew Wilcox <willy@infradead.org>,
linux-mm <linux-mm@kvack.org>,
Andrew Morton <akpm@linux-foundation.org>,
will@kernel.org, linux-arm-kernel@lists.infradead.org,
Oscar Salvador <osalvador@suse.de>,
Yang Zhang <yang.zhang.wz@gmail.com>,
Pankaj Gupta <pagupta@redhat.com>,
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>,
Rik van Riel <riel@surriel.com>,
lcapitulino@redhat.com, "Wang, Wei W" <wei.w.wang@intel.com>,
Andrea Arcangeli <aarcange@redhat.com>,
ying.huang@intel.com, Paolo Bonzini <pbonzini@redhat.com>,
Dan Williams <dan.j.williams@intel.com>,
Fengguang Wu <fengguang.wu@intel.com>,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Subject: Re: [PATCH v9 0/8] stg mail -e --version=v9 \
Date: Thu, 12 Sep 2019 08:00:45 -0400 [thread overview]
Message-ID: <880cb776-a5c0-026c-6c50-3d8d3c2bb2df@redhat.com> (raw)
In-Reply-To: <ef460202-cebd-c6d2-19f3-e8a82a3d3cbd@redhat.com>
On 9/12/19 3:47 AM, David Hildenbrand wrote:
> On 12.09.19 09:16, Michal Hocko wrote:
>> On Wed 11-09-19 18:09:18, David Hildenbrand wrote:
>>> On 11.09.19 15:51, Michal Hocko wrote:
>>>> On Wed 11-09-19 15:20:02, Michal Hocko wrote:
>>>> [...]
>>>>>> 4. Continuously report, not the "one time report everything" approach.
>>>>> So you mean the allocator reporting this rather than an external code to
>>>>> poll right? I do not know, how much this is nice to have than must have?
>>>> Another idea that I haven't really thought through so it might turned
>>>> out to be completely bogus but let's try anyway. Your "report everything"
>>>> just made me look and realize that free_pages_prepare already performs
>>>> stuff that actually does something similar yet unrelated.
>>>>
>>>> We do report to special page poisoning, zeroying or
>>>> CONFIG_DEBUG_PAGEALLOC to unmap the address from the kernel address
>>>> space. This sounds like something fitting your model no?
>>>>
>>> AFAIKS, the poisoning/unmapping is done whenever a page is freed. I
>>> don't quite see yet how that would help to remember if a page was
>>> already reported.
>> Do you still have to differ that state when each page is reported?
> Ah, very good point. I can see that the reason for this was not
> discussed in this thread so far. (Alexander, Nitesh, please correct me
> if I am wrong). It's buried in the long history of free page
> hinting/reporting.
>
> Some early patch sets tried to report during every free synchronously.
> Free a page, report them to the hypervisor. This resulted in some issues
> (especially, locking-related and the virtio + the hypervisor being
> involved, resulting in unpredictable delays, quite some overhead ...).
> It was no good.
+1
If I remember correctly then Alexander had posted a patch-set
prior to this series where he was reporting every page of a fixed
order from __free_one_page(). But as you said it will be costly as
it will involve one hypercall per page of reporting_order.
>
> One design decision then was to not report single pages, but a bunch of
> pages at once. This made it necessary to "remember" the pages to be
> reported and to temporarily block them from getting allocated while
> reporting.
Until my v7 posting [1] I was doing this. We did not proceed with
this as blocking allocation was not recommended for reporting.
>
> Nitesh implemented (at least) two "capture PFNs of free pages in an
> array when freeing" approaches. One being synchronous from the freeing
> CPU once the list was full (having similar issues as plain synchronous
> reporting) and one being asynchronous by a separate thread (which solved
> many locking issues).
One issue with asynchronous + array approach was that it could have lead
to false OOMs due to several pages being isolated at the same time.
>
> Turned out the a simple array can quickly lead to us having to drop
> "reports" to the hypervisor because the array is full and the reporting
> thread was not able to keep up. Not good as well. Especially, if some
> process frees a lot of memory this can happen quickly and Nitesh wa
> sable to trigger this scenario frequently.
+1
>
> Finally, Nitesh decided to use the bitmap instead to keep track of pages
> to report. I'd like to note that this approach could still be combined
> with an "array of potentially free PFNs". Only when the array/circular
> buffer runs out of entries ("reporting thread cannot keep up"), we would
> have to go back to scanning the bitmap.
I will have to think about it.
> That was also the point where Alexander decided to look into integrating
> tracking/handling reported/unreported pages directly in the buddy.
>
>>> After reporting the page we would have to switch some
>>> state (Nitesh: bitmap bit, Alexander: page flag) to identify that.
>> Yes, you can either store the state somewhere.
>>
>>> Of course, we could map the page and treat that as "the state" when we
>>> reported it, but I am not sure that's such a good idea :)
>>>
>>> As always, I might be very wrong ...
>> I still do not fully understand the usecase so I might be equally wrong.
>> My thinking is along these lines. Why should you scan free pages when
>> you can effectively capture each freed page? If you go one step further
>> then post_alloc_hook would be the counterpart to know that your page has
>> been allocated.
> I'd like to note that Nitesh's patch set contains the following hunk,
> which is roughly what you were thinking :)
>
>
> -static inline void __free_one_page(struct page *page,
> +inline void __free_one_page(struct page *page,
> unsigned long pfn,
> struct zone *zone, unsigned int order,
> - int migratetype)
> + int migratetype, bool hint)
> {
> unsigned long combined_pfn;
> unsigned long uninitialized_var(buddy_pfn);
> @@ -980,7 +981,8 @@ static inline void __free_one_page(struct page *page,
> migratetype);
> else
> add_to_free_area(page, &zone->free_area[order], migratetype);
> -
> + if (hint)
> + page_hinting_enqueue(page, order);
> }
>
>
> (ignore the hint parameter, when he would switch to a isolate vs.
> alloc/free, that can go away and all we left is the enqueue part)
Precisely.
Although, there would be a scenario where the allocation will
take place for a page whose order would be < REPORTING_ORDER.
In that case, if I decide to ignore all the remaining pages and clear
the previously head free page bit, I might end
up losing the reporting opportunity.
But I can certainly look into this.
>
>
> Inside that callback we can remember the pages any way we want. Right
> now in a bitmap. Maybe later in a array + bitmap (as discussed above).
> Another idea I had was to simply go over all pages and report them when
> running into this "array full" condition. But I am not yet sure about
> the performance implications on rather large machines. So the bitmap
> idea might have some other limitations but seems to do its job.
That's correct, I was actually trying to come up with a basic framework.
Which is acceptable in terms of benefits and performance and that can fit into
most of the use-cases (if not all).
After which my plan was to further optimize it.
>
> Hoe that makes things clearer and am not missing something.
Thanks for explaining.
>
--
Thanks
Nitesh
next prev parent reply other threads:[~2019-09-12 12:00 UTC|newest]
Thread overview: 82+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-07 17:25 [PATCH v9 0/8] stg mail -e --version=v9 \ Alexander Duyck
2019-09-07 17:25 ` [PATCH v9 1/8] mm: Add per-cpu logic to page shuffling Alexander Duyck
2019-09-09 8:14 ` David Hildenbrand
2019-09-09 15:11 ` Alexander Duyck
2019-09-10 12:11 ` Michal Hocko
2019-09-10 22:14 ` Alexander Duyck
2019-09-10 22:11 ` Alexander Duyck
2019-09-09 9:07 ` Kirill A. Shutemov
2019-09-09 15:12 ` Alexander Duyck
2019-09-07 17:25 ` [PATCH v9 2/8] mm: Adjust shuffle code to allow for future coalescing Alexander Duyck
2019-09-09 8:19 ` David Hildenbrand
2019-09-09 9:47 ` Kirill A. Shutemov
2019-09-09 15:22 ` Alexander Duyck
2019-09-09 15:35 ` Kirill A. Shutemov
2019-09-09 15:37 ` Alexander Duyck
2019-09-09 16:43 ` Alexander Duyck
2019-09-09 17:00 ` Kirill A. Shutemov
2019-09-10 12:20 ` Michal Hocko
2019-09-10 14:48 ` Alexander Duyck
2019-09-07 17:25 ` [PATCH v9 3/8] mm: Move set/get_pcppage_migratetype to mmzone.h Alexander Duyck
2019-09-09 8:22 ` David Hildenbrand
2019-09-09 9:56 ` Kirill A. Shutemov
2019-09-09 18:01 ` Alexander Duyck
2019-09-09 18:12 ` Alexander Duyck
2019-09-10 12:23 ` Michal Hocko
2019-09-10 14:46 ` Alexander Duyck
2019-09-10 17:45 ` Michal Hocko
2019-09-10 20:26 ` Alexander Duyck
2019-09-07 17:25 ` [PATCH v9 4/8] mm: Use zone and order instead of free area in free_list manipulators Alexander Duyck
2019-09-10 12:27 ` Michal Hocko
2019-09-07 17:25 ` [PATCH v9 5/8] arm64: Move hugetlb related definitions out of pgtable.h to page-defs.h Alexander Duyck
2019-09-09 8:52 ` David Hildenbrand
2019-09-09 15:27 ` Alexander Duyck
2019-09-17 17:48 ` Will Deacon
2019-09-17 20:07 ` Alexander Duyck
2019-09-07 17:25 ` [PATCH v9 6/8] mm: Introduce Reported pages Alexander Duyck
2019-09-09 14:42 ` Kirill A. Shutemov
2019-09-09 16:25 ` Alexander Duyck
2019-09-09 16:33 ` Kirill A. Shutemov
2019-09-07 17:26 ` [PATCH v9 7/8] virtio-balloon: Pull page poisoning config out of free page hinting Alexander Duyck
2019-09-09 8:59 ` David Hildenbrand
2019-09-09 15:31 ` Alexander Duyck
2019-09-07 17:26 ` [PATCH v9 8/8] virtio-balloon: Add support for providing unused page reports to host Alexander Duyck
2019-09-07 17:34 ` [PATCH v9 0/8] mm / virtio: Provide support for unused page reporting Alexander Duyck
2019-09-10 12:42 ` [PATCH v9 0/8] stg mail -e --version=v9 \ Michal Hocko
2019-09-10 14:42 ` Alexander Duyck
2019-09-10 14:47 ` Michal Hocko
2019-09-10 16:05 ` Alexander Duyck
2019-09-10 16:18 ` [virtio-dev] " Dr. David Alan Gilbert
2019-09-10 16:22 ` David Hildenbrand
2019-09-11 9:23 ` Michael S. Tsirkin
2019-09-11 9:50 ` David Hildenbrand
2019-09-10 17:52 ` Michal Hocko
2019-09-10 18:00 ` Michal Hocko
2019-09-10 20:37 ` Alexander Duyck
2019-09-10 21:23 ` Alexander Duyck
2019-09-11 11:36 ` Michal Hocko
2019-09-11 11:47 ` David Hildenbrand
2019-09-11 12:08 ` Michael S. Tsirkin
2019-09-11 12:19 ` Michal Hocko
2019-09-11 12:25 ` Michal Hocko
2019-09-11 12:42 ` David Hildenbrand
2019-09-11 12:54 ` Michal Hocko
2019-09-11 13:03 ` David Hildenbrand
2019-09-11 13:20 ` Michal Hocko
2019-09-11 13:51 ` Michal Hocko
2019-09-11 16:09 ` David Hildenbrand
2019-09-12 7:16 ` Michal Hocko
2019-09-12 7:47 ` David Hildenbrand
2019-09-12 9:26 ` Michal Hocko
2019-09-12 12:00 ` Nitesh Narayan Lal [this message]
2019-09-11 14:03 ` Nitesh Narayan Lal
2019-09-11 16:02 ` David Hildenbrand
2019-09-11 13:19 ` Nitesh Narayan Lal
2019-09-11 12:55 ` Nitesh Narayan Lal
2019-09-11 15:12 ` Alexander Duyck
2019-09-12 9:19 ` Michal Hocko
2019-09-12 10:24 ` Kirill A. Shutemov
2019-09-12 11:11 ` Michal Hocko
2019-09-12 15:42 ` Alexander Duyck
2019-09-12 16:35 ` Mel Gorman
2019-09-12 17:48 ` Alexander Duyck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=880cb776-a5c0-026c-6c50-3d8d3c2bb2df@redhat.com \
--to=nitesh@redhat.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=alexander.duyck@gmail.com \
--cc=alexander.h.duyck@linux.intel.com \
--cc=catalin.marinas@arm.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@intel.com \
--cc=david@redhat.com \
--cc=fengguang.wu@intel.com \
--cc=kirill.shutemov@linux.intel.com \
--cc=konrad.wilk@oracle.com \
--cc=kvm@vger.kernel.org \
--cc=lcapitulino@redhat.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=mst@redhat.com \
--cc=osalvador@suse.de \
--cc=pagupta@redhat.com \
--cc=pbonzini@redhat.com \
--cc=riel@surriel.com \
--cc=virtio-dev@lists.oasis-open.org \
--cc=wei.w.wang@intel.com \
--cc=will@kernel.org \
--cc=willy@infradead.org \
--cc=yang.zhang.wz@gmail.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).