From: Andrew Morton <firstname.lastname@example.org> To: Alexander Duyck <email@example.com> Cc: firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com Subject: Re: [PATCH v17 0/9] mm / virtio: Provide support for free page reporting Date: Tue, 11 Feb 2020 15:05:10 -0800 [thread overview] Message-ID: <firstname.lastname@example.org> (raw) In-Reply-To: <email@example.com> On Tue, 11 Feb 2020 14:45:51 -0800 Alexander Duyck <firstname.lastname@example.org> wrote: > This series provides an asynchronous means of reporting free guest pages > to a hypervisor so that the memory associated with those pages can be > dropped and reused by other processes and/or guests on the host. Using > this it is possible to avoid unnecessary I/O to disk and greatly improve > performance in the case of memory overcommit on the host. "greatly improve" sounds nice. > When enabled we will be performing a scan of free memory every 2 seconds > while pages of sufficiently high order are being freed. In each pass at > least one sixteenth of each free list will be reported. By doing this we > avoid racing against other threads that may be causing a high amount of > memory churn. > > The lowest page order currently scanned when reporting pages is > pageblock_order so that this feature will not interfere with the use of > Transparent Huge Pages in the case of virtualization. > > Currently this is only in use by virtio-balloon however there is the hope > that at some point in the future other hypervisors might be able to make > use of it. In the virtio-balloon/QEMU implementation the hypervisor is > currently using MADV_DONTNEED to indicate to the host kernel that the page > is currently free. It will be zeroed and faulted back into the guest the > next time the page is accessed. > > To track if a page is reported or not the Uptodate flag was repurposed and > used as a Reported flag for Buddy pages. We walk though the free list > isolating pages and adding them to the scatterlist until we either > encounter the end of the list or have processed at least one sixteenth of > the pages that were listed in nr_free prior to us starting. If we fill the > scatterlist before we reach the end of the list we rotate the list so that > the first unreported page we encounter is moved to the head of the list as > that is where we will resume after we have freed the reported pages back > into the tail of the list. > > Below are the results from various benchmarks. I primarily focused on two > tests. The first is the will-it-scale/page_fault2 test, and the other is > a modified version of will-it-scale/page_fault1 that was enabled to use > THP. I did this as it allows for better visibility into different parts > of the memory subsystem. The guest is running with 32G for RAM on one > node of a E5-2630 v3. The host has had some features such as CPU turbo > disabled in the BIOS. > > Test page_fault1 (THP) page_fault2 > Name tasks Process Iter STDEV Process Iter STDEV > Baseline 1 1012402.50 0.14% 361855.25 0.81% > 16 8827457.25 0.09% 3282347.00 0.34% > > Patches Applied 1 1007897.00 0.23% 361887.00 0.26% > 16 8784741.75 0.39% 3240669.25 0.48% > > Patches Enabled 1 1010227.50 0.39% 359749.25 0.56% > 16 8756219.00 0.24% 3226608.75 0.97% > > Patches Enabled 1 1050982.00 4.26% 357966.25 0.14% > page shuffle 16 8672601.25 0.49% 3223177.75 0.40% > > Patches enabled 1 1003238.00 0.22% 360211.00 0.22% > shuffle w/ RFC 16 8767010.50 0.32% 3199874.00 0.71% But these differences seem really small - around 1%? I think we're just showing not much harm was caused? > The results above are for a baseline with a linux-next-20191219 kernel, > that kernel with this patch set applied but page reporting disabled in > virtio-balloon, the patches applied and page reporting fully enabled, the > patches enabled with page shuffling enabled, and the patches applied with > page shuffling enabled and an RFC patch that makes used of MADV_FREE in > QEMU. These results include the deviation seen between the average value > reported here versus the high and/or low value. I observed that during the > test memory usage for the first three tests never dropped whereas with the > patches fully enabled the VM would drop to using only a few GB of the > host's memory when switching from memhog to page fault tests. And this is the "great improvement", yes? Is it possible to measure the end-user-visible benefits of this? > Any of the overhead visible with this patch set enabled seems due to page > faults caused by accessing the reported pages and the host zeroing the page > before giving it back to the guest. This overhead is much more visible when > using THP than with standard 4K pages. In addition page shuffling seemed to > increase the amount of faults generated due to an increase in memory churn. > The overehad is reduced when using MADV_FREE as we can avoid the extra > zeroing of the pages when they are reintroduced to the host, as can be seen > when the RFC is applied with shuffling enabled. > > The overall guest size is kept fairly small to only a few GB while the test > is running. If the host memory were oversubscribed this patch set should > result in a performance improvement as swapping memory in the host can be > avoided. "should result". Can we firm this up a lot?
next prev parent reply other threads:[~2020-02-11 23:05 UTC|newest] Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-02-11 22:45 Alexander Duyck 2020-02-11 22:46 ` [PATCH v17 1/9] mm: Adjust shuffle code to allow for future coalescing Alexander Duyck 2020-02-11 22:46 ` [PATCH v17 2/9] mm: Use zone and order instead of free area in free_list manipulators Alexander Duyck 2020-02-11 22:46 ` [PATCH v17 3/9] mm: Add function __putback_isolated_page Alexander Duyck 2020-02-19 14:33 ` Mel Gorman 2020-02-11 22:46 ` [PATCH v17 4/9] mm: Introduce Reported pages Alexander Duyck 2020-02-19 14:55 ` Mel Gorman 2020-02-20 18:44 ` Alexander Duyck 2020-02-20 22:35 ` Mel Gorman 2020-02-21 19:25 ` Alexander Duyck 2020-02-21 20:19 ` Mel Gorman 2020-02-11 22:46 ` [PATCH v17 5/9] virtio-balloon: Pull page poisoning config out of free page hinting Alexander Duyck 2020-02-11 22:46 ` [PATCH v17 6/9] virtio-balloon: Add support for providing free page reports to host Alexander Duyck 2020-02-11 22:47 ` [PATCH v17 7/9] mm/page_reporting: Rotate reported pages to the tail of the list Alexander Duyck 2020-02-19 14:59 ` Mel Gorman 2020-02-11 22:47 ` [PATCH v17 8/9] mm/page_reporting: Add budget limit on how many pages can be reported per pass Alexander Duyck 2020-02-19 15:02 ` Mel Gorman 2020-02-11 22:47 ` [PATCH v17 9/9] mm/page_reporting: Add free page reporting documentation Alexander Duyck 2020-02-11 22:51 ` [PATCH v17 QEMU 1/3] virtio-ballon: Implement support for page poison tracking feature Alexander Duyck 2020-02-11 22:51 ` [PATCH v17 QEMU 2/3] virtio-balloon: Add support for providing free page reports to host Alexander Duyck 2020-02-11 22:51 ` [PATCH v17 QEMU 3/3] virtio-balloon: Provide a interface for free page reporting Alexander Duyck 2020-02-11 22:53 ` [PATCH v17 QEMU 4/3 RFC] memory: Add support for MADV_FREE as mechanism to lazy discard pages Alexander Duyck 2020-02-11 23:05 ` Andrew Morton [this message] 2020-02-11 23:55 ` [PATCH v17 0/9] mm / virtio: Provide support for free page reporting Alexander Duyck 2020-02-12 0:19 ` Andrew Morton 2020-02-12 1:19 ` Alexander Duyck 2020-02-18 16:37 ` Alexander Duyck 2020-02-19 8:49 ` Mel Gorman 2020-02-19 15:06 ` Mel Gorman
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [PATCH v17 0/9] mm / virtio: Provide support for free page reporting' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).