From: David Hildenbrand <firstname.lastname@example.org> To: Alexander Duyck <email@example.com> Cc: Michal Hocko <firstname.lastname@example.org>, email@example.com, kvm list <firstname.lastname@example.org>, "Michael S. Tsirkin" <email@example.com>, Dave Hansen <firstname.lastname@example.org>, LKML <email@example.com>, Matthew Wilcox <firstname.lastname@example.org>, linux-mm <email@example.com>, Vlastimil Babka <firstname.lastname@example.org>, Andrew Morton <email@example.com>, Mel Gorman <firstname.lastname@example.org>, email@example.com, Oscar Salvador <firstname.lastname@example.org>, Yang Zhang <email@example.com>, Pankaj Gupta <firstname.lastname@example.org>, Konrad Rzeszutek Wilk <email@example.com>, Nitesh Narayan Lal <firstname.lastname@example.org>, Rik van Riel <email@example.com>, firstname.lastname@example.org, "Wang, Wei W" <email@example.com>, Andrea Arcangeli <firstname.lastname@example.org>, Paolo Bonzini <email@example.com>, Dan Williams <firstname.lastname@example.org>, Alexander Duyck <email@example.com> Subject: Re: [PATCH v10 0/6] mm / virtio: Provide support for unused page reporting Date: Tue, 24 Sep 2019 19:28:19 +0200 [thread overview] Message-ID: <firstname.lastname@example.org> (raw) In-Reply-To: <CAKgT0UedoNBk3cp64SpCzXJqjtqBWZQSB7QzF7R_jhTDXbzNPg@mail.gmail.com> On 24.09.19 19:07, Alexander Duyck wrote: > On Tue, Sep 24, 2019 at 8:32 AM David Hildenbrand <email@example.com> wrote: >> >> On 24.09.19 16:23, Michal Hocko wrote: >>> On Wed 18-09-19 10:52:25, Alexander Duyck wrote: >>> [...] >>>> In order to try and keep the time needed to find a non-reported page to >>>> a minimum we maintain a "reported_boundary" pointer. This pointer is used >>>> by the get_unreported_pages iterator to determine at what point it should >>>> resume searching for non-reported pages. In order to guarantee pages do >>>> not get past the scan I have modified add_to_free_list_tail so that it >>>> will not insert pages behind the reported_boundary. >>>> >>>> If another process needs to perform a massive manipulation of the free >>>> list, such as compaction, it can either reset a given individual boundary >>>> which will push the boundary back to the list_head, or it can clear the >>>> bit indicating the zone is actively processing which will result in the >>>> reporting process resetting all of the boundaries for a given zone. >>> >>> Is this any different from the previous version? The last review >>> feedback (both from me and Mel) was that we are not happy to have an >>> externally imposed constrains on how the page allocator is supposed to >>> maintain its free lists. >>> >>> If this is really the only way to go forward then I would like to hear >>> very convincing arguments about other approaches not being feasible. >> >> Adding to what Alexander said, I don't consider the other approaches >> (especially the bitmap-based approach Nitesh is currently working on) >> infeasible. There might be more rough edges (e.g., sparse zones) and >> eventually sometimes a little more work to be done, but definitely >> feasible. Incorporating stuff into the buddy might make some tasks >> (e.g., identify free pages) more efficient. >> >> I still somewhat like the idea of capturing hints of free pages (in >> whatever data structure) and then going over the hints, seeing if the >> pages are still free. Then only temporarily isolating the still-free >> pages, reporting them, and un-isolating them after they were reported. I >> like the idea that the pages are not fake-allocated but only temporarily >> blocked. That works nicely e.g., with the movable zone (contain only >> movable data). > > One other change in this patch set is that I split the headers so that > there is an internal header that resides in the mm tree and an > external one that provides the page reporting device structure and the > register/unregister functions. All that virtio-balloon knows is that > it is registering a notifier and will be called with scatter gather > lists for memory that is not currently in use by the kernel. It has no > visibility into the internal free_areas or the current state of the > buddy allocator. Rather than having two blocks that are both trying to > maintain that state, I have consolidated it all into the buddy > allocator with page reporting. > >> But anyhow, after decades of people working on free page >> hinting/reporting, I am happy with anything that gets accepted upstream :D > > Agreed. After working on this for 9 months I would be happy to get > something upstream that addresses this. IBM upstreamed their proprietary solution - 45e576b1c3d0 ("S390] guest page hinting light") - in 2008. Rik has presented a generic approach in 2011 (!) https://www.linux-kvm.org/images/f/ff/2011-forum-memory-overcommit.pdf I think Nitesh has been working on this (initially as an Intern) since Mid 2017. So yeah, this stuff has quite some history :) > > - Alex > -- Thanks, David / dhildenb
prev parent reply other threads:[~2019-09-24 17:28 UTC|newest] Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-09-18 17:52 Alexander Duyck 2019-09-18 17:52 ` [PATCH v10 1/6] mm: Adjust shuffle code to allow for future coalescing Alexander Duyck 2019-09-18 17:52 ` [PATCH v10 2/6] mm: Use zone and order instead of free area in free_list manipulators Alexander Duyck 2019-09-18 17:52 ` [PATCH v10 3/6] mm: Introduce Reported pages Alexander Duyck 2019-09-21 15:25 ` [mm] 0f5b256b2c: will-it-scale.per_process_ops -1.2% regression kernel test robot 2019-09-23 8:15 ` [PATCH v10 3/6] mm: Introduce Reported pages Michael S. Tsirkin 2019-09-23 14:50 ` Alexander Duyck 2019-09-23 15:00 ` Michael S. Tsirkin 2019-09-23 15:28 ` Alexander Duyck 2019-09-23 15:37 ` Michael S. Tsirkin 2019-09-23 15:45 ` David Hildenbrand 2019-09-23 15:47 ` David Hildenbrand 2019-09-23 15:50 ` Michael S. Tsirkin 2019-09-23 15:53 ` David Hildenbrand 2019-09-23 15:49 ` Michael S. Tsirkin 2019-09-23 16:39 ` Alexander Duyck 2019-09-18 17:52 ` [PATCH v10 4/6] mm: Add device side and notifier for unused page reporting Alexander Duyck 2019-09-18 17:53 ` [PATCH v10 5/6] virtio-balloon: Pull page poisoning config out of free page hinting Alexander Duyck 2019-09-18 17:58 ` Michael S. Tsirkin 2019-09-18 18:05 ` Alexander Duyck 2019-09-18 17:53 ` [PATCH v10 6/6] virtio-balloon: Add support for providing unused page reports to host Alexander Duyck 2019-09-18 17:53 ` [PATCH v10 QEMU 1/3] virtio-ballon: Implement support for page poison tracking feature Alexander Duyck 2019-09-18 17:53 ` [PATCH v10 QEMU 2/3] virtio-balloon: Add bit to notify guest of unused page reporting Alexander Duyck 2019-09-18 17:53 ` [PATCH v10 QEMU 3/3] virtio-balloon: Provide a interface for " Alexander Duyck 2019-09-24 14:23 ` [PATCH v10 0/6] mm / virtio: Provide support " Michal Hocko 2019-09-24 15:20 ` Alexander Duyck 2019-09-26 12:22 ` Michal Hocko 2019-09-26 15:13 ` Alexander Duyck 2019-09-24 15:32 ` David Hildenbrand 2019-09-24 15:51 ` Nitesh Narayan Lal 2019-09-24 17:07 ` Alexander Duyck 2019-09-24 17:28 ` David Hildenbrand [this message]
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --subject='Re: [PATCH v10 0/6] mm / virtio: Provide support for unused page reporting' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).