From: Nitesh Narayan Lal <nitesh@redhat.com> To: Alexander Duyck <alexander.duyck@gmail.com> Cc: kvm list <kvm@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>, linux-mm <linux-mm@kvack.org>, virtio-dev@lists.oasis-open.org, Paolo Bonzini <pbonzini@redhat.com>, lcapitulino@redhat.com, Pankaj Gupta <pagupta@redhat.com>, "Wang, Wei W" <wei.w.wang@intel.com>, Yang Zhang <yang.zhang.wz@gmail.com>, Rik van Riel <riel@surriel.com>, David Hildenbrand <david@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, dodgen@google.com, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, dhildenb@redhat.com, Andrea Arcangeli <aarcange@redhat.com>, john.starks@microsoft.com, Dave Hansen <dave.hansen@intel.com>, Michal Hocko <mhocko@suse.com>, cohuck@redhat.com Subject: Re: [RFC][Patch v12 1/2] mm: page_reporting: core infrastructure Date: Thu, 15 Aug 2019 09:15:37 -0400 Message-ID: <09c6fbef-fa53-3a25-d3d6-460b9b6b2020@redhat.com> (raw) In-Reply-To: <CAKgT0UfavuUT4ZvfxVdm3h25qc86ksxPO=GFpFkf8zbGAjHPvg@mail.gmail.com> On 8/14/19 12:11 PM, Alexander Duyck wrote: > On Wed, Aug 14, 2019 at 8:49 AM Nitesh Narayan Lal <nitesh@redhat.com> wrote: >> >> On 8/12/19 2:47 PM, Alexander Duyck wrote: >>> On Mon, Aug 12, 2019 at 6:13 AM Nitesh Narayan Lal <nitesh@redhat.com> wrote: >>>> This patch introduces the core infrastructure for free page reporting in >>>> virtual environments. It enables the kernel to track the free pages which >>>> can be reported to its hypervisor so that the hypervisor could >>>> free and reuse that memory as per its requirement. >>>> >>>> While the pages are getting processed in the hypervisor (e.g., >>>> via MADV_DONTNEED), the guest must not use them, otherwise, data loss >>>> would be possible. To avoid such a situation, these pages are >>>> temporarily removed from the buddy. The amount of pages removed >>>> temporarily from the buddy is governed by the backend(virtio-balloon >>>> in our case). >>>> >>>> To efficiently identify free pages that can to be reported to the >>>> hypervisor, bitmaps in a coarse granularity are used. Only fairly big >>>> chunks are reported to the hypervisor - especially, to not break up THP >>>> in the hypervisor - "MAX_ORDER - 2" on x86, and to save space. The bits >>>> in the bitmap are an indication whether a page *might* be free, not a >>>> guarantee. A new hook after buddy merging sets the bits. >>>> >>>> Bitmaps are stored per zone, protected by the zone lock. A workqueue >>>> asynchronously processes the bitmaps, trying to isolate and report pages >>>> that are still free. The backend (virtio-balloon) is responsible for >>>> reporting these batched pages to the host synchronously. Once reporting/ >>>> freeing is complete, isolated pages are returned back to the buddy. >>>> >>>> Signed-off-by: Nitesh Narayan Lal <nitesh@redhat.com> >> [...] >>>> +} >>>> + >>>> +/** >>>> + * __page_reporting_enqueue - tracks the freed page in the respective zone's >>>> + * bitmap and enqueues a new page reporting job to the workqueue if possible. >>>> + */ >>>> +void __page_reporting_enqueue(struct page *page) >>>> +{ >>>> + struct page_reporting_config *phconf; >>>> + struct zone *zone; >>>> + >>>> + rcu_read_lock(); >>>> + /* >>>> + * We should not process this page if either page reporting is not >>>> + * yet completely enabled or it has been disabled by the backend. >>>> + */ >>>> + phconf = rcu_dereference(page_reporting_conf); >>>> + if (!phconf) >>>> + return; >>>> + >>>> + zone = page_zone(page); >>>> + bitmap_set_bit(page, zone); >>>> + >>>> + /* >>>> + * We should not enqueue a job if a previously enqueued reporting work >>>> + * is in progress or we don't have enough free pages in the zone. >>>> + */ >>>> + if (atomic_read(&zone->free_pages) >= phconf->max_pages && >>>> + !atomic_cmpxchg(&phconf->refcnt, 0, 1)) >>> This doesn't make any sense to me. Why are you only incrementing the >>> refcount if it is zero? Combining this with the assignment above, this >>> isn't really a refcnt. It is just an oversized bitflag. >> >> The intent for having an extra variable was to ensure that at a time only one >> reporting job is enqueued. I do agree that for that purpose I really don't need >> a reference counter and I should have used something like bool >> 'page_hinting_active'. But with bool, I think there could be a possible chance >> of race. Maybe I should rename this variable and keep it as atomic. >> Any thoughts? > You could just use a bitflag to achieve what you are doing here. That > is the primary use case for many of the test_and_set_bit type > operations. However one issue with doing it as a bitflag is that you > have no way of telling that you took care of all requesters. I think you are right, I might end up missing on certain reporting opportunities in some special cases. Specifically when the pages which are part of this new reporting request belongs to a section of the bitmap which has already been scanned. Although, I have failed to reproduce this kind of situation in an actual setup. > That is > where having an actual reference count comes in handy as you know > exactly how many zones are requesting to be reported on. True. > >>> Also I am pretty sure this results in the opportunity to miss pages >>> because there is nothing to prevent you from possibly missing a ton of >>> pages you could hint on if a large number of pages are pushed out all >>> at once and then the system goes idle in terms of memory allocation >>> and freeing. >> >> I was looking at how you are enqueuing/processing reporting jobs for each zone. >> I am wondering if I should also consider something on similar lines as having >> that I might be able to address the concern which you have raised above. But it >> would also mean that I have to add an additional flag in the zone_flags. :) > You could do it either in the zone or outside the zone as yet another > bitmap. I decided to put the flags inside the zone because there was a > number of free bits there and it should be faster since we were > already using the zone structure. There are two possibilities which could happen while I am reporting: 1. Another request might come in for a different zone. 2. Another request could come in for the same zone and the pages belong to a section of the bitmap which has already been scanned. Having a per zone flag to indicate reporting status will solve the first issue and to an extent the second as well. Having refcnt will possibly solve both of them. What I am wondering about is that in my case I could easily impact the performance negatively by performing more bitmap scanning. -- Thanks Nitesh
next prev parent reply index Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-08-12 13:12 [RFC][PATCH v12 0/2] mm: Support for page reporting Nitesh Narayan Lal 2019-08-12 13:12 ` [RFC][Patch v12 1/2] mm: page_reporting: core infrastructure Nitesh Narayan Lal 2019-08-12 18:47 ` Alexander Duyck 2019-08-12 20:04 ` Nitesh Narayan Lal 2019-08-20 14:11 ` Nitesh Narayan Lal 2019-08-12 20:05 ` David Hildenbrand 2019-08-13 10:30 ` Nitesh Narayan Lal 2019-08-13 10:34 ` David Hildenbrand 2019-08-13 10:42 ` Nitesh Narayan Lal 2019-08-13 10:44 ` David Hildenbrand 2019-08-13 23:14 ` Alexander Duyck 2019-08-14 7:07 ` David Hildenbrand 2019-08-14 12:49 ` [virtio-dev] " Nitesh Narayan Lal 2019-08-14 15:49 ` Nitesh Narayan Lal 2019-08-14 16:11 ` Alexander Duyck 2019-08-15 13:15 ` Nitesh Narayan Lal [this message] 2019-08-15 19:22 ` Nitesh Narayan Lal 2019-08-15 23:00 ` Alexander Duyck 2019-08-16 18:35 ` Nitesh Narayan Lal 2019-08-30 15:15 ` Nitesh Narayan Lal 2019-08-30 15:31 ` Alexander Duyck 2019-08-30 16:05 ` Nitesh Narayan Lal 2019-09-04 8:40 ` [virtio-dev] " David Hildenbrand 2019-10-10 20:36 ` Alexander Duyck 2019-10-11 11:02 ` Nitesh Narayan Lal 2019-08-12 13:12 ` [RFC][Patch v12 2/2] virtio-balloon: interface to support free page reporting Nitesh Narayan Lal 2019-08-14 10:29 ` Cornelia Huck 2019-08-14 11:47 ` Nitesh Narayan Lal 2019-08-14 13:42 ` Cornelia Huck 2019-08-14 14:01 ` Nitesh Narayan Lal 2019-08-12 13:13 ` [QEMU Patch 1/2] virtio-balloon: adding bit for page reporting support Nitesh Narayan Lal 2019-08-12 13:13 ` [QEMU Patch 2/2] virtio-balloon: support for handling page reporting Nitesh Narayan Lal 2019-08-12 15:18 ` Alexander Duyck 2019-08-12 15:26 ` Nitesh Narayan Lal 2019-09-11 12:30 ` [RFC][PATCH v12 0/2] mm: Support for " David Hildenbrand
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=09c6fbef-fa53-3a25-d3d6-460b9b6b2020@redhat.com \ --to=nitesh@redhat.com \ --cc=aarcange@redhat.com \ --cc=alexander.duyck@gmail.com \ --cc=cohuck@redhat.com \ --cc=dave.hansen@intel.com \ --cc=david@redhat.com \ --cc=dhildenb@redhat.com \ --cc=dodgen@google.com \ --cc=john.starks@microsoft.com \ --cc=konrad.wilk@oracle.com \ --cc=kvm@vger.kernel.org \ --cc=lcapitulino@redhat.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@suse.com \ --cc=mst@redhat.com \ --cc=pagupta@redhat.com \ --cc=pbonzini@redhat.com \ --cc=riel@surriel.com \ --cc=virtio-dev@lists.oasis-open.org \ --cc=wei.w.wang@intel.com \ --cc=yang.zhang.wz@gmail.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
KVM Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/kvm/0 kvm/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 kvm kvm/ https://lore.kernel.org/kvm \ kvm@vger.kernel.org public-inbox-index kvm Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.kvm AGPL code for this site: git clone https://public-inbox.org/public-inbox.git