From: David Hildenbrand <email@example.com> To: Alexander Duyck <firstname.lastname@example.org>, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com Cc: firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org Subject: Re: [PATCH v14 0/6] mm / virtio: Provide support for unused page reporting Date: Tue, 26 Nov 2019 13:20:32 +0100 [thread overview] Message-ID: <email@example.com> (raw) In-Reply-To: <firstname.lastname@example.org> On 19.11.19 22:46, Alexander Duyck wrote: > This series provides an asynchronous means of reporting unused guest > pages to a hypervisor so that the memory associated with those pages can > be dropped and reused by other processes and/or guests on the host. Using > this it is possible to avoid unnecessary I/O to disk and greatly improve > performance in the case of memory overcommit on the host. > > When enabled it will allocate a set of statistics to track the number of > reported pages. When the nr_free for a given free area is greater than > this by the high water mark we will schedule a worker to begin pulling the > non-reported memory and to provide it to the reporting interface via a > scatterlist. > > Currently this is only in use by virtio-balloon however there is the hope > that at some point in the future other hypervisors might be able to make > use of it. In the virtio-balloon/QEMU implementation the hypervisor is > currently using MADV_DONTNEED to indicate to the host kernel that the page > is currently unused. It will be zeroed and faulted back into the guest the > next time the page is accessed. Remind me why we are using MADV_DONTNEED? Mostly for debugging purposes right now, right? Did you do any measurements with MADV_FREE? I guess there should be quite a performance increase in some scenarios. > > To track if a page is reported or not the Uptodate flag was repurposed and > used as a Reported flag for Buddy pages. We walk though the free list > isolating pages and adding them to the scatterlist until we either > encounter the end of the list or have filled the scatterlist with pages to > be reported. If we fill the scatterlist before we reach the end of the > list we rotate the list so that the first unreported page we encounter is > moved to the head of the list as that is where we will resume after we > have freed the reported pages back into the tail of the list. So the boundary pointer didn't actually provide that big of a benefit I assume (IOW, worst thing is you have to re-scan the whole list)? > > Below are the results from various benchmarks. I primarily focused on two > tests. The first is the will-it-scale/page_fault2 test, and the other is > a modified version of will-it-scale/page_fault1 that was enabled to use > THP. I did this as it allows for better visibility into different parts > of the memory subsystem. The guest is running with 32G for RAM on one > node of a E5-2630 v3. The host has had some power saving features disabled > by setting the /dev/cpu_dma_latency value to 10ms. > > Test page_fault1 (THP) page_fault2 > Name tasks Process Iter STDEV Process Iter STDEV > Baseline 1 1203934.75 0.04% 379940.75 0.11% > 16 8828217.00 0.85% 3178653.00 1.28% > > Patches applied 1 1207961.25 0.10% 380852.25 0.25% > 16 8862373.00 0.98% 3246397.25 0.68% > > Patches enabled 1 1207758.75 0.17% 373079.25 0.60% > MADV disabled 16 8870373.75 0.29% 3204989.75 1.08% > > Patches enabled 1 1261183.75 0.39% 373201.50 0.50% > 16 8371359.75 0.65% 3233665.50 0.84% > > Patches enabled 1 1090201.50 0.25% 376967.25 0.29% > page shuffle 16 8108719.75 0.58% 3218450.25 1.07% > > The results above are for a baseline with a linux-next-20191115 kernel, > that kernel with this patch set applied but page reporting disabled in > virtio-balloon, patches applied but the madvise disabled by direct > assigning a device, the patches applied and page reporting fully > enabled, and the patches enabled with page shuffling enabled. These > results include the deviation seen between the average value reported here > versus the high and/or low value. I observed that during the test memory > usage for the first three tests never dropped whereas with the patches > fully enabled the VM would drop to using only a few GB of the host's > memory when switching from memhog to page fault tests. > > Most of the overhead seen with this patch set enabled seems due to page > faults caused by accessing the reported pages and the host zeroing the page > before giving it back to the guest. This overhead is much more visible when > using THP than with standard 4K pages. In addition page shuffling seemed to > increase the amount of faults generated due to an increase in memory churn. MADV_FREE would be interesting. > > The overall guest size is kept fairly small to only a few GB while the test > is running. If the host memory were oversubscribed this patch set should > result in a performance improvement as swapping memory in the host can be > avoided. > > A brief history on the background of unused page reporting can be found at: > https://email@example.com/ > > Changes from v12: > https://firstname.lastname@example.org/ > Rebased on linux-next 20191031 > Renamed page_is_reported to page_reported > Renamed add_page_to_reported_list to mark_page_reported > Dropped unused definition of add_page_to_reported_list for non-reporting case > Split free_area_reporting out from get_unreported_tail > Minor updates to cover page > > Changes from v13: > https://email@example.com/ > Rewrote core reporting functionality > Merged patches 3 & 4 > Dropped boundary list and related code > Folded get_reported_page into page_reporting_fill > Folded page_reporting_fill into page_reporting_cycle > Pulled reporting functionality out of free_reported_page > Renamed it to __free_isolated_page > Moved page reporting specific bits to page_reporting_drain > Renamed phdev to prdev since we aren't "hinting" we are "reporting" > Added documentation to describe the usage of unused page reporting > Updated cover page and patch descriptions to avoid mention of boundary > > > --- > > Alexander Duyck (6): > mm: Adjust shuffle code to allow for future coalescing > mm: Use zone and order instead of free area in free_list manipulators > mm: Introduce Reported pages > mm: Add unused page reporting documentation > virtio-balloon: Pull page poisoning config out of free page hinting > virtio-balloon: Add support for providing unused page reports to host > > > Documentation/vm/unused_page_reporting.rst | 44 ++++ > drivers/virtio/Kconfig | 1 > drivers/virtio/virtio_balloon.c | 88 +++++++ > include/linux/mmzone.h | 56 +---- > include/linux/page-flags.h | 11 + > include/linux/page_reporting.h | 31 +++ > include/uapi/linux/virtio_balloon.h | 1 > mm/Kconfig | 11 + > mm/Makefile | 1 > mm/memory_hotplug.c | 2 > mm/page_alloc.c | 181 +++++++++++---- > mm/page_reporting.c | 337 ++++++++++++++++++++++++++++ > mm/page_reporting.h | 125 ++++++++++ > mm/shuffle.c | 12 - > mm/shuffle.h | 6 > 15 files changed, 805 insertions(+), 102 deletions(-) So roughly 100 LOC less added, that's nice to see :) I'm planning to look into the details soon, just fairly busy lately. I hope Mel Et al. can also comment. -- Thanks, David / dhildenb
next prev parent reply other threads:[~2019-11-26 12:20 UTC|newest] Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-11-19 21:46 Alexander Duyck 2019-11-19 21:46 ` [PATCH v14 1/6] mm: Adjust shuffle code to allow for future coalescing Alexander Duyck 2019-11-27 13:52 ` Mel Gorman 2019-11-19 21:46 ` [PATCH v14 2/6] mm: Use zone and order instead of free area in free_list manipulators Alexander Duyck 2019-11-27 13:54 ` Mel Gorman 2019-11-19 21:46 ` [PATCH v14 3/6] mm: Introduce Reported pages Alexander Duyck 2019-11-27 15:24 ` Mel Gorman 2019-11-27 17:22 ` Alexander Duyck 2019-11-27 18:35 ` Mel Gorman 2019-11-27 21:55 ` Alexander Duyck 2019-11-28 9:22 ` Mel Gorman 2019-11-29 19:25 ` Alexander Duyck 2019-11-19 21:46 ` [PATCH v14 4/6] mm: Add unused page reporting documentation Alexander Duyck 2019-11-19 21:46 ` [PATCH v14 5/6] virtio-balloon: Pull page poisoning config out of free page hinting Alexander Duyck 2019-11-19 21:46 ` [PATCH v14 6/6] virtio-balloon: Add support for providing unused page reports to host Alexander Duyck 2019-11-28 15:25 ` David Hildenbrand 2019-11-28 17:00 ` Michael S. Tsirkin 2019-12-04 17:48 ` Alexander Duyck 2019-12-04 17:53 ` Alexander Duyck 2019-11-29 21:13 ` Alexander Duyck 2019-12-01 11:46 ` Michael S. Tsirkin 2019-12-01 18:25 ` Alexander Duyck 2019-12-02 10:43 ` David Hildenbrand 2019-11-19 21:54 ` [PATCH v14 QEMU 1/3] virtio-ballon: Implement support for page poison tracking feature Alexander Duyck 2019-11-19 21:54 ` [PATCH v14 QEMU 2/3] virtio-balloon: Add bit to notify guest of unused page reporting Alexander Duyck 2019-11-19 21:54 ` [PATCH v14 QEMU 3/3] virtio-balloon: Provide a interface for " Alexander Duyck 2019-11-26 12:20 ` David Hildenbrand [this message] 2019-11-26 16:45 ` [PATCH v14 0/6] mm / virtio: Provide support " Alexander Duyck 2019-11-27 10:01 ` David Hildenbrand 2019-11-27 17:36 ` Alexander Duyck 2019-11-27 17:37 ` David Hildenbrand
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --subject='Re: [PATCH v14 0/6] mm / virtio: Provide support for unused page reporting' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).