From: Muhammad Usama Anjum <usama.anjum@collabora.com>
To: Peter Xu <peterx@redhat.com>
Cc: Muhammad Usama Anjum <usama.anjum@collabora.com>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Andrea Arcangeli <aarcange@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>,
Mike Rapoport <rppt@linux.vnet.ibm.com>,
Axel Rasmussen <axelrasmussen@google.com>,
Nadav Amit <nadav.amit@gmail.com>,
David Hildenbrand <david@redhat.com>,
"kernel@collabora.com" <kernel@collabora.com>
Subject: Re: [PATCH v2] mm/uffd: UFFD_FEATURE_WP_UNPOPULATED
Date: Wed, 1 Mar 2023 12:55:51 +0500 [thread overview]
Message-ID: <640319be-ddb6-d74f-b731-eee5ceab3d01@collabora.com> (raw)
In-Reply-To: <fb7ec372-2b16-14e1-a8cd-a90f4449661f@collabora.com>
Hi Peter,
Finally I've found the bug.
On 2/28/23 9:24 PM, Muhammad Usama Anjum wrote:
> On 2/28/23 8:58 PM, Peter Xu wrote:
>> On Tue, Feb 28, 2023 at 12:21:45PM +0500, Muhammad Usama Anjum wrote:
>>> Hi Peter,
>>
>> Hi, Muhammad,
>>
>>>
>>> Thank you so much for sending.
>>>
>>> On 2/28/23 5:36 AM, Peter Xu wrote:
>>>> On Mon, Feb 27, 2023 at 06:00:44PM -0500, Peter Xu wrote:
>>>>> This is a new feature that controls how uffd-wp handles none ptes. When
>>>>> it's set, the kernel will handle anonymous memory the same way as file
>>>>> memory, by allowing the user to wr-protect unpopulated ptes.
>>>>>
>>>>> File memories handles none ptes consistently by allowing wr-protecting of
>>>>> none ptes because of the unawareness of page cache being exist or not. For
>>>>> anonymous it was not as persistent because we used to assume that we don't
>>>>> need protections on none ptes or known zero pages.
>>>>>
>>>>> One use case of such a feature bit was VM live snapshot, where if without
>>>>> wr-protecting empty ptes the snapshot can contain random rubbish in the
>>>>> holes of the anonymous memory, which can cause misbehave of the guest when
>>>>> the guest OS assumes the pages should be all zeros.
>>>>>
>>>>> QEMU worked it around by pre-populate the section with reads to fill in
>>>>> zero page entries before starting the whole snapshot process [1].
>>>>>
>>>>> Recently there's another need raised on using userfaultfd wr-protect for
>>>>> detecting dirty pages (to replace soft-dirty in some cases) [2]. In that
>>>>> case if without being able to wr-protect none ptes by default, the dirty
>>>>> info can get lost, since we cannot treat every none pte to be dirty (the
>>>>> current design is identify a page dirty based on uffd-wp bit being cleared).
>>>>>
>>>>> In general, we want to be able to wr-protect empty ptes too even for
>>>>> anonymous.
>>>>>
>>>>> This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make
>>>>> uffd-wp handling on none ptes being consistent no matter what the memory
>>>>> type is underneath. It doesn't have any impact on file memories so far
>>>>> because we already have pte markers taking care of that. So it only
>>>>> affects anonymous.
>>>>>
>>>>> The feature bit is by default off, so the old behavior will be maintained.
>>>>> Sometimes it may be wanted because the wr-protect of none ptes will contain
>>>>> overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to
>>>>> anonymous), but also on creating the pgtables to store the pte markers. So
>>>>> there's potentially less chance of using thp on the first fault for a none
>>>>> pmd or larger than a pmd.
>>>>>
>>>>> The major implementation part is teaching the whole kernel to understand
>>>>> pte markers even for anonymously mapped ranges, meanwhile allowing the
>>>>> UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the
>>>>> new feature bit is set.
>>>>>
>>>>> Note that even if the patch subject starts with mm/uffd, there're a few
>>>>> small refactors to major mm path of handling anonymous page faults. But
>>>>> they should be straightforward.
>>>>>
>>>>> So far, add a very light smoke test within the userfaultfd kselftest
>>>>> pagemap unit test to make sure anon pte markers work.
>>>>>
>>>>> [1] https://lore.kernel.org/all/20210401092226.102804-4-andrey.gruzdev@virtuozzo.com/
>>>>> [1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/
>>>>>
>>>>> Signed-off-by: Peter Xu <peterx@redhat.com>
>>>>> ---
>>>>> v1->v2:
>>>>> - Use pte markers rather than populate zero pages when protect [David]
>>>>> - Rename WP_ZEROPAGE to WP_UNPOPULATED [David]
>>>>
>>>> Some very initial performance numbers (I only ran in a VM but it should be
>>>> similar, unit is "us") below as requested. The measurement is about time
>>>> spent when wr-protecting 10G range of empty but mapped memory. It's done
>>>> in a VM, assuming we'll get similar results on bare metal.
>>>>
>>>> Four test cases:
>>>>
>>>> - default UFFDIO_WP
>>>> - pre-read the memory, then UFFDIO_WP (what QEMU does right now)
>>>> - pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
>>>> - UFFDIO_WP with WP_UNPOPULATED
>>>>
>>>> Results:
>>>>
>>>> Test DEFAULT: 2
>>>> Test PRE-READ: 3277099 (pre-fault 3253826)
>>>> Test MADVISE: 2250361 (pre-fault 2226310)
>>>> Test WP-UNPOPULATE: 20850
>>>>
>>>> I'll add these information into the commit message when there's a new
>>>> version.
>>> I'm hitting a bug where I'm unable to write to the memory after adding this
>>> patch and wp the memory. I'm hitting this case in your test and my tests as
>>> well. Please apply the following diff to your test to reproduce on your end:
>>>
>>> --- uffd_wp_perf.c.orig 2023-02-28 12:09:38.971820791 +0500
>>> +++ uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500
>>> @@ -114,6 +114,7 @@
>>> start1 = get_usec();
>>> }
>>> wp_range(uffd, buffer, SIZE, true);
>>> + buffer[0] = 'a';
While using WP_UNPOPULATED, we get stuck if newly allocated memory is read
without initialization. This can be reproduced by either of the following
statements:
printf("%c", buffer[0]);
buffer[0]++;
This bug has start to appear on this patch. How are you handling reading
newly allocated memory when WP_UNPOPULATED is defined?
Running my pagemap_ioctl selftest as benchmark in a VM:
without zeropage / wp_unpopulated (decide from pte_none() if page is dirty
or not, buggy and wrong implementation, just for reference)
26.608 seconds
with zeropage
39.203 seconds
with wp_unpopulated
62.907 seconds
136% worse performance overall
60% worse performance of unpopulated than zeropage
>>> if (start1 == 0)
>>> printf("%"PRIu64"\n", get_usec() - start);
>>> else
>>
>> This is expected, because the test didn't start any fault resolving thread,
>> so the write will block until someone unprotects the page.
> Ohh.. sorry. Wrong reproducer.
>
>>
>> But it shouldn't happen to your use case if you applied both WP_UNPOPULATED
>> & WP_ASYNC.
> I'm using both WP_UNPOPULATED and ASYNC. The program gets stuck at right time:
>
>
> 1..57
> ok 1 sanity_tests_sd wrong flag specified
> ok 2 sanity_tests_sd wrong mask specified
> ok 3 sanity_tests_sd wrong return mask specified
> ok 4 sanity_tests_sd mixture of correct and wrong flag
> ok 5 sanity_tests_sd PM_SCAN_OP_WP cannot be used without get
> ok 6 sanity_tests_sd Clear area with larger vec size
> ^C
> Program received signal SIGINT, Interrupt.
> 0x000000000040220c in sanity_tests_sd () at pagemap_ioctl.c:198
> 198 mem[i]++;
> (gdb) bt
> #0 0x000000000040220c in sanity_tests_sd () at pagemap_ioctl.c:198
> #1 0x0000000000404e14 in main () at pagemap_ioctl.c:846
> ()
>
> /proc/$PID/stack is empty. Not sure why. I can see stack trace of other
> applications, but not this one's.
>
> Let me send better reproducer for you.
>
>>
>> Could you try "cat /proc/$PID/stack" to see where does your thread stuck
>> at?
>>
>
--
BR,
Muhammad Usama Anjum
next prev parent reply other threads:[~2023-03-01 7:56 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-27 23:00 [PATCH v2] mm/uffd: UFFD_FEATURE_WP_UNPOPULATED Peter Xu
2023-02-28 0:36 ` Peter Xu
2023-02-28 7:21 ` Muhammad Usama Anjum
2023-02-28 15:58 ` Peter Xu
2023-02-28 16:24 ` Muhammad Usama Anjum
2023-03-01 7:55 ` Muhammad Usama Anjum [this message]
2023-03-01 15:19 ` Peter Xu
2023-03-01 17:13 ` Muhammad Usama Anjum
2023-03-02 9:37 ` David Hildenbrand
2023-03-02 13:57 ` Peter Xu
2023-03-02 14:01 ` David Hildenbrand
2023-03-02 15:14 ` Muhammad Usama Anjum
2023-03-02 22:00 ` Peter Xu
2023-03-02 17:19 ` Muhammad Usama Anjum
2023-03-02 17:38 ` David Hildenbrand
2023-03-02 22:21 ` Peter Xu
2023-03-03 6:42 ` Muhammad Usama Anjum
2023-03-03 16:47 ` Peter Xu
2023-03-06 9:03 ` Muhammad Usama Anjum
2023-03-06 16:09 ` Muhammad Usama Anjum
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=640319be-ddb6-d74f-b731-eee5ceab3d01@collabora.com \
--to=usama.anjum@collabora.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=david@redhat.com \
--cc=kernel@collabora.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=nadav.amit@gmail.com \
--cc=peterx@redhat.com \
--cc=rppt@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).