From: Nitesh Narayan Lal <nitesh@redhat.com>
To: Alexander Duyck <alexander.h.duyck@linux.intel.com>,
Alexander Duyck <alexander.duyck@gmail.com>
Cc: LKML <linux-kernel@vger.kernel.org>,
linux-mm <linux-mm@kvack.org>,
David Hildenbrand <david@redhat.com>,
virtio-dev@lists.oasis-open.org, kvm list <kvm@vger.kernel.org>,
"Michael S. Tsirkin" <mst@redhat.com>,
Dave Hansen <dave.hansen@intel.com>,
Matthew Wilcox <willy@infradead.org>,
Michal Hocko <mhocko@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Mel Gorman <mgorman@techsingularity.net>,
Vlastimil Babka <vbabka@suse.cz>,
Oscar Salvador <osalvador@suse.de>,
Yang Zhang <yang.zhang.wz@gmail.com>,
Pankaj Gupta <pagupta@redhat.com>,
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>,
Rik van Riel <riel@surriel.com>,
lcapitulino@redhat.com, "Wang, Wei W" <wei.w.wang@intel.com>,
Andrea Arcangeli <aarcange@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>,
Dan Williams <dan.j.williams@intel.com>
Subject: Re: [PATCH v11 0/6] mm / virtio: Provide support for unused page reporting
Date: Wed, 9 Oct 2019 13:08:28 -0400 [thread overview]
Message-ID: <90ab549c-2155-7a57-5dc9-f5a785049e8c@redhat.com> (raw)
In-Reply-To: <4b6a9bda0a19c20b04338fd1d9b4f96086480355.camel@linux.intel.com>
On 10/9/19 12:50 PM, Alexander Duyck wrote:
> On Wed, 2019-10-09 at 12:25 -0400, Nitesh Narayan Lal wrote:
>> On 10/7/19 1:20 PM, Alexander Duyck wrote:
>>> On Mon, Oct 7, 2019 at 10:07 AM Nitesh Narayan Lal <nitesh@redhat.com> wrote:
>>>> On 10/7/19 12:27 PM, Alexander Duyck wrote:
>>>>> On Mon, 2019-10-07 at 12:19 -0400, Nitesh Narayan Lal wrote:
>>>>>> On 10/7/19 11:33 AM, Alexander Duyck wrote:
>>>>>>> On Mon, 2019-10-07 at 08:29 -0400, Nitesh Narayan Lal wrote:
>>>>>>>> On 10/2/19 10:25 AM, Alexander Duyck wrote:
>>> <snip>
>>>
>>>>>>>> page_reporting.c change:
>>>>>>>> @@ -101,8 +101,12 @@ static void scan_zone_bitmap(struct page_reporting_config
>>>>>>>> *phconf,
>>>>>>>> /* Process only if the page is still online */
>>>>>>>> page = pfn_to_online_page((setbit << PAGE_REPORTING_MIN_ORDER) +
>>>>>>>> zone->base_pfn);
>>>>>>>> - if (!page)
>>>>>>>> + if (!page || !PageBuddy(page)) {
>>>>>>>> + clear_bit(setbit, zone->bitmap);
>>>>>>>> + atomic_dec(&zone->free_pages);
>>>>>>>> continue;
>>>>>>>> + }
>>>>>>>>
>>>>>>> I suspect the zone->free_pages is going to be expensive for you to deal
>>>>>>> with. It is a global atomic value and is going to have the cacheline
>>>>>>> bouncing that it is contained in. As a result thinks like setting the
>>>>>>> bitmap with be more expensive as every tome a CPU increments free_pages it
>>>>>>> will likely have to take the cache line containing the bitmap pointer as
>>>>>>> well.
>>>>>> I see I will have to explore this more. I am wondering if there is a way to
>>>>>> measure this If its effect is not visible in will-it-scale/page_fault1. If
>>>>>> there is a noticeable amount of degradation, I will have to address this.
>>>>> If nothing else you might look at seeing if you can split up the
>>>>> structures so that the bitmap and nr_bits is in a different region
>>>>> somewhere since those are read-mostly values.
>>>> ok, I will try to understand the issue and your suggestion.
>>>> Thank you for bringing this up.
>>>>
>>>>> Also you are now updating the bitmap and free_pages both inside and
>>>>> outside of the zone lock so that will likely have some impact.
>>>> So as per your previous suggestion, I have made the bitmap structure
>>>> object as a rcu protected pointer. So we are safe from that side.
>>>> The other downside which I can think of is a race where one page
>>>> trying to increment free_pages and other trying to decrements it.
>>>> However, being an atomic variable that should not be a problem.
>>>> Did I miss anything?
>>> I'm not so much worried about a race as the cache line bouncing
>>> effect. Basically your notifier combined within this hinting thread
>>> will likely result in more time spent by the thread that holds the
>>> lock since it will be trying to access the bitmap to set the bit and
>>> the free_pages to report the bit, but at the same time you will have
>>> this thread clearing bits and decrementing the free_pages values.
>>>
>>> One thing you could consider in your worker thread would be to do
>>> reallocate and replace the bitmap every time you plan to walk it. By
>>> doing that you would avoid the cacheline bouncing on the bitmap since
>>> you would only have to read it, and you would no longer have another
>>> thread dirtying it. You could essentially reset the free_pages at the
>>> same time you replace the bitmap. It would need to all happen with the
>>> zone lock held though when you swap it out.
>> If I am not mistaken then from what you are suggesting, I will have to hold
>> the zone lock for the entire duration of swap & scan which would be costly if
>> the bitmap is large, isn't? Also, we might end up missing free pages that are
>> getting
>> freed while we are scanning.
> You would only need to hold the zone lock when you swap the bitmap. Once
> it is swapped you wouldn't need to worry about the locking again for
> bitmap access since your worker thread would be the only one holding the
> current bitmap. Think of it as a batch clearing of the bits.
I see.
>
> You already end up missing pages freed while scanning since you are doing
> it linearly.
I was referring to free pages for whom bits will not be set while we
are doing the batch clearing of the bits.
>
>> As far as free_pages count is concerned, I am thinking if I should
>> replace it with zone->free_area[REPORTING_ORDER].nr_free which is already there
>> (I still need to explore this in a bit more depth).
>>
>>> - Alex
> So there ends up being two ways you could use nr_free. One is to track it
> the way I did with the number of reported pages being tracked, however
> that requires reducing the count when reported pages are pulled from the
> free_area and identifying reported pages vs unreported ones.
>
> The other option would be to look at converting nr_free into a pair of
> free running counters, one tracking frees, and another tracking
> allocations. Then you just need to record a snapshot of the nr_free values
> when you do something like the bitmap swap, and then you would be able to
> track churn, but it wouldn't give you an exact count of unreported pages
> since it is possible to just alloc/free a single page multiple times to
> make it look like you have freed a number of pages even though you really
> haven't.
Yeah possibly. I will think about it a little bit more to see what
is the best way to do it.
--
Thanks
Nitesh
next prev parent reply other threads:[~2019-10-09 17:08 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-01 15:29 [PATCH v11 0/6] mm / virtio: Provide support for unused page reporting Alexander Duyck
2019-10-01 15:29 ` [PATCH v11 1/6] mm: Adjust shuffle code to allow for future coalescing Alexander Duyck
2019-10-01 15:29 ` [PATCH v11 2/6] mm: Use zone and order instead of free area in free_list manipulators Alexander Duyck
2019-10-15 0:42 ` [mm] 2eca680594: will-it-scale.per_process_ops -2.5% regression kernel test robot
2019-10-01 15:29 ` [PATCH v11 3/6] mm: Introduce Reported pages Alexander Duyck
2019-10-01 15:29 ` [PATCH v11 4/6] mm: Add device side and notifier for unused page reporting Alexander Duyck
2019-10-01 15:29 ` [PATCH v11 5/6] virtio-balloon: Pull page poisoning config out of free page hinting Alexander Duyck
2019-10-01 15:29 ` [PATCH v11 6/6] virtio-balloon: Add support for providing unused page reports to host Alexander Duyck
2019-10-01 15:31 ` [PATCH v11 QEMU 1/3] virtio-ballon: Implement support for page poison tracking feature Alexander Duyck
2019-10-01 15:31 ` [PATCH v11 QEMU 2/3] virtio-balloon: Add bit to notify guest of unused page reporting Alexander Duyck
2019-10-01 15:31 ` [PATCH v11 QEMU 3/3] virtio-balloon: Provide a interface for " Alexander Duyck
2019-10-01 15:35 ` [PATCH v11 0/6] mm / virtio: Provide support " David Hildenbrand
2019-10-01 16:21 ` Alexander Duyck
2019-10-01 18:41 ` David Hildenbrand
2019-10-01 19:17 ` Nitesh Narayan Lal
2019-10-01 19:08 ` Michael S. Tsirkin
2019-10-01 19:16 ` Nitesh Narayan Lal
2019-10-01 20:25 ` Alexander Duyck
2019-10-01 20:49 ` Alexander Duyck
2019-10-01 20:51 ` Dave Hansen
2019-10-02 15:04 ` Nitesh Narayan Lal
2019-10-02 14:41 ` [virtio-dev] " Nitesh Narayan Lal
2019-10-02 0:55 ` Alexander Duyck
2019-10-02 7:13 ` David Hildenbrand
2019-10-02 10:44 ` Nitesh Narayan Lal
2019-10-02 10:36 ` Nitesh Narayan Lal
2019-10-02 14:25 ` Alexander Duyck
2019-10-02 14:36 ` Nitesh Narayan Lal
2019-10-07 12:29 ` Nitesh Narayan Lal
2019-10-07 15:33 ` Alexander Duyck
2019-10-07 16:19 ` Nitesh Narayan Lal
2019-10-07 16:27 ` Alexander Duyck
2019-10-07 17:06 ` Nitesh Narayan Lal
2019-10-07 17:20 ` Alexander Duyck
2019-10-09 16:25 ` Nitesh Narayan Lal
2019-10-09 16:50 ` Alexander Duyck
2019-10-09 17:08 ` Nitesh Narayan Lal [this message]
2019-10-09 17:26 ` Alexander Duyck
2019-10-09 15:21 ` Nitesh Narayan Lal
2019-10-09 16:35 ` Alexander Duyck
2019-10-09 19:46 ` Nitesh Narayan Lal
2019-10-10 7:36 ` David Hildenbrand
2019-10-10 10:27 ` Nitesh Narayan Lal
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=90ab549c-2155-7a57-5dc9-f5a785049e8c@redhat.com \
--to=nitesh@redhat.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=alexander.duyck@gmail.com \
--cc=alexander.h.duyck@linux.intel.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@intel.com \
--cc=david@redhat.com \
--cc=konrad.wilk@oracle.com \
--cc=kvm@vger.kernel.org \
--cc=lcapitulino@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@kernel.org \
--cc=mst@redhat.com \
--cc=osalvador@suse.de \
--cc=pagupta@redhat.com \
--cc=pbonzini@redhat.com \
--cc=riel@surriel.com \
--cc=vbabka@suse.cz \
--cc=virtio-dev@lists.oasis-open.org \
--cc=wei.w.wang@intel.com \
--cc=willy@infradead.org \
--cc=yang.zhang.wz@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).