* [PATCH RESEND v15 0/5] Implement IOCTL to get and optionally clear info about PTEs
@ 2023-04-20 6:01 Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 1/5] userfaultfd: UFFD_FEATURE_WP_ASYNC Muhammad Usama Anjum
` (4 more replies)
0 siblings, 5 replies; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-20 6:01 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Mike Rapoport, Nadav Amit
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-mm,
linux-kselftest, Greg KH, kernel
*Changes in v15*
- Build fix (Add missed build fix in RESEND)
*Changes in v14*
- Fix build error caused by #ifdef added at last minute in some configs
*Changes in v13*
- Rebase on top of next-20230414
- Give-up on using uffd_wp_range() and write new helpers, flush tlb only
once
*Changes in v12*
- Update and other memory types to UFFD_FEATURE_WP_ASYNC
- Rebaase on top of next-20230406
- Review updates
*Changes in v11*
- Rebase on top of next-20230307
- Base patches on UFFD_FEATURE_WP_UNPOPULATED
- Do a lot of cosmetic changes and review updates
- Remove ENGAGE_WP + !GET operation as it can be performed with
UFFDIO_WRITEPROTECT
*Changes in v10*
- Add specific condition to return error if hugetlb is used with wp
async
- Move changes in tools/include/uapi/linux/fs.h to separate patch
- Add documentation
*Changes in v9:*
- Correct fault resolution for userfaultfd wp async
- Fix build warnings and errors which were happening on some configs
- Simplify pagemap ioctl's code
*Changes in v8:*
- Update uffd async wp implementation
- Improve PAGEMAP_IOCTL implementation
*Changes in v7:*
- Add uffd wp async
- Update the IOCTL to use uffd under the hood instead of soft-dirty
flags
*Motivation*
The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows
GetWriteWatch() syscall [1]. The GetWriteWatch{} retrieves the addresses of
the pages that are written to in a region of virtual memory.
This syscall is used in Windows applications and games etc. This syscall is
being emulated in pretty slow manner in userspace. Our purpose is to
enhance the kernel such that we translate it efficiently in a better way.
Currently some out of tree hack patches are being used to efficiently
emulate it in some kernels. We intend to replace those with these patches.
So the whole gaming on Linux can effectively get benefit from this. It
means there would be tons of users of this code.
CRIU use case [2] was mentioned by Andrei and Danylo:
> Use cases for migrating sparse VMAs are binaries sanitized with ASAN,
> MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of
> shadow memory [4]. Being able to migrate such binaries allows to highly
> reduce the amount of work needed to identify and fix post-migration
> crashes, which happen constantly.
Andrei's defines the following uses of this code:
* it is more granular and allows us to track changed pages more
effectively. The current interface can clear dirty bits for the entire
process only. In addition, reading info about pages is a separate
operation. It means we must freeze the process to read information
about all its pages, reset dirty bits, only then we can start dumping
pages. The information about pages becomes more and more outdated,
while we are processing pages. The new interface solves both these
downsides. First, it allows us to read pte bits and clear the
soft-dirty bit atomically. It means that CRIU will not need to freeze
processes to pre-dump their memory. Second, it clears soft-dirty bits
for a specified region of memory. It means CRIU will have actual info
about pages to the moment of dumping them.
* The new interface has to be much faster because basic page filtering
is happening in the kernel. With the old interface, we have to read
pagemap for each page.
*Implementation Evolution (Short Summary)*
From the definition of GetWriteWatch(), we feel like kernel's soft-dirty
feature can be used under the hood with some additions like:
* reset soft-dirty flag for only a specific region of memory instead of
clearing the flag for the entire process
* get and clear soft-dirty flag for a specific region atomically
So we decided to use ioctl on pagemap file to read or/and reset soft-dirty
flag. But using soft-dirty flag, sometimes we get extra pages which weren't
even written. They had become soft-dirty because of VMA merging and
VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were
able to by-pass this short coming by ignoring VM_SOFTDIRTY until David
reported that mprotect etc messes up the soft-dirty flag while ignoring
VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We
discussed if we can revert these patches. But we could not reach to any
conclusion. So at this point, I made couple of tries to solve this whole
VM_SOFTDIRTY issue by correcting the soft-dirty implementation:
* [7] Correct the bug fixed wrongly back in 2014. It had potential to cause
regression. We left it behind.
* [8] Keep a list of soft-dirty part of a VMA across splits and merges. I
got the reply don't increase the size of the VMA by 8 bytes.
At this point, we left soft-dirty considering it is too much delicate and
userfaultfd [9] seemed like the only way forward. From there onward, we
have been basing soft-dirty emulation on userfaultfd wp feature where
kernel resolves the faults itself when WP_ASYNC feature is used. It was
straight forward to add WP_ASYNC feature in userfautlfd. Now we get only
those pages dirty or written-to which are really written in reality. (PS
There is another WP_UNPOPULATED userfautfd feature is required which is
needed to avoid pre-faulting memory before write-protecting [9].)
All the different masks were added on the request of CRIU devs to create
interface more generic and better.
[1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getwritewatch
[2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com
[3] https://github.com/google/sanitizers
[4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit
[5] https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@redhat.com
[6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@redhat.com/
[7] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
[8] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
[9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com
[10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@google.com
* Original Cover letter from v8*
Hello,
Note:
Soft-dirty pages and pages which have been written-to are synonyms. As
kernel already has soft-dirty feature inside which we have given up to
use, we are using written-to terminology while using UFFD async WP under
the hood.
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are
supported in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
pages have been written-to.
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE)
It is possible to find and clear soft-dirty pages entirely in userspace.
But it isn't efficient:
- The mprotect and SIGSEGV handler for bookkeeping
- The userfaultfd wp (synchronous) with the handler for bookkeeping
Some benchmarks can be seen here[1]. This series adds features that weren't
present earlier:
- There is no atomic get soft-dirty/Written-to status and clear present in
the kernel.
- The pages which have been written-to can not be found in accurate way.
(Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty
pages than there actually are.)
Historically, soft-dirty PTE bit tracking has been used in the CRIU
project. The procfs interface is enough for finding the soft-dirty bit
status and clearing the soft-dirty bit of all the pages of a process.
We have the use case where we need to track the soft-dirty PTE bit for
only specific pages on-demand. We need this tracking and clear mechanism
of a region of memory while the process is running to emulate the
getWriteWatch() syscall of Windows.
*(Moved to using UFFD instead of soft-dirtyi feature to find pages which
have been written-to from v7 patch series)*:
Stop using the soft-dirty flags for finding which pages have been
written to. It is too delicate and wrong as it shows more soft-dirty
pages than the actual soft-dirty pages. There is no interest in
correcting it [2][3] as this is how the feature was written years ago.
It shouldn't be updated to changed behaviour. Peter Xu has suggested
using the async version of the UFFD WP [4] as it is based inherently
on the PTEs.
So in this patch series, I've added a new mode to the UFFD which is
asynchronous version of the write protect. When this variant of the
UFFD WP is used, the page faults are resolved automatically by the
kernel. The pages which have been written-to can be found by reading
pagemap file (!PM_UFFD_WP). This feature can be used successfully to
find which pages have been written to from the time the pages were
write protected. This works just like the soft-dirty flag without
showing any extra pages which aren't soft-dirty in reality.
The information related to pages if the page is file mapped, present and
swapped is required for the CRIU project [5][6]. The addition of the
required mask, any mask, excluded mask and return masks are also required
for the CRIU project [5].
The IOCTL returns the addresses of the pages which match the specific
masks. The page addresses are returned in struct page_region in a compact
form. The max_pages is needed to support a use case where user only wants
to get a specific number of pages. So there is no need to find all the
pages of interest in the range when max_pages is specified. The IOCTL
returns when the maximum number of the pages are found. The max_pages is
optional. If max_pages is specified, it must be equal or greater than the
vec_size. This restriction is needed to handle worse case when one
page_region only contains info of one page and it cannot be compacted.
This is needed to emulate the Windows getWriteWatch() syscall.
The patch series include the detailed selftest which can be used as an
example for the uffd async wp test and PAGEMAP_IOCTL. It shows the
interface usages as well.
[1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/
[2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
[3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
[4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n
[5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/
[6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/
Regards,
Muhammad Usama Anjum
Muhammad Usama Anjum (4):
fs/proc/task_mmu: Implement IOCTL to get and optionally clear info
about PTEs
tools headers UAPI: Update linux/fs.h with the kernel sources
mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL
selftests: mm: add pagemap ioctl tests
Peter Xu (1):
userfaultfd: UFFD_FEATURE_WP_ASYNC
Documentation/admin-guide/mm/pagemap.rst | 56 +
Documentation/admin-guide/mm/userfaultfd.rst | 35 +
fs/proc/task_mmu.c | 481 +++++++
fs/userfaultfd.c | 26 +-
include/linux/userfaultfd_k.h | 21 +-
include/uapi/linux/fs.h | 53 +
include/uapi/linux/userfaultfd.h | 9 +-
mm/hugetlb.c | 32 +-
mm/memory.c | 27 +-
tools/include/uapi/linux/fs.h | 53 +
tools/testing/selftests/mm/.gitignore | 1 +
tools/testing/selftests/mm/Makefile | 3 +-
tools/testing/selftests/mm/config | 1 +
tools/testing/selftests/mm/pagemap_ioctl.c | 1326 ++++++++++++++++++
tools/testing/selftests/mm/run_vmtests.sh | 4 +
15 files changed, 2105 insertions(+), 23 deletions(-)
create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh
--
2.39.2
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH RESEND v15 1/5] userfaultfd: UFFD_FEATURE_WP_ASYNC
2023-04-20 6:01 [PATCH RESEND v15 0/5] Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
@ 2023-04-20 6:01 ` Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
` (3 subsequent siblings)
4 siblings, 0 replies; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-20 6:01 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Mike Rapoport, Nadav Amit
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-mm,
linux-kselftest, Greg KH, kernel
From: Peter Xu <peterx@redhat.com>
This patch adds a new userfaultfd-wp feature UFFD_FEATURE_WP_ASYNC, that
allows userfaultfd wr-protect faults to be resolved by the kernel directly.
It can be used like a high accuracy version of soft-dirty, without vma
modifications during tracking, and also with ranged support by default
rather than for a whole mm when reset the protections due to existence of
ioctl(UFFDIO_WRITEPROTECT).
Several goals of such a dirty tracking interface:
1. All types of memory should be supported and tracable. This is nature
for soft-dirty but should mention when the context is userfaultfd,
because it used to only support anon/shmem/hugetlb. The problem is for
a dirty tracking purpose these three types may not be enough, and it's
legal to track anything e.g. any page cache writes from mmap.
2. Protections can be applied to partial of a memory range, without vma
split/merge fuss. The hope is that the tracking itself should not
affect any vma layout change. It also helps when reset happens because
the reset will not need mmap write lock which can block the tracee.
3. Accuracy needs to be maintained. This means we need pte markers to work
on any type of VMA.
One could question that, the whole concept of async dirty tracking is not
really close to fundamentally what userfaultfd used to be: it's not "a
fault to be serviced by userspace" anymore. However, using userfaultfd-wp
here as a framework is convenient for us in at least:
1. VM_UFFD_WP vma flag, which has a very good name to suite something like
this, so we don't need VM_YET_ANOTHER_SOFT_DIRTY. Just use a new
feature bit to identify from a sync version of uffd-wp registration.
2. PTE markers logic can be leveraged across the whole kernel to maintain
the uffd-wp bit as long as an arch supports, this also applies to this
case where uffd-wp bit will be a hint to dirty information and it will
not go lost easily (e.g. when some page cache ptes got zapped).
3. Reuse ioctl(UFFDIO_WRITEPROTECT) interface for either starting or
resetting a range of memory, while there's no counterpart in the old
soft-dirty world, hence if this is wanted in a new design we'll need a
new interface otherwise.
We can somehow understand that commonality because uffd-wp was
fundamentally a similar idea of write-protecting pages just like
soft-dirty.
This implementation allows WP_ASYNC to imply WP_UNPOPULATED, because so far
WP_ASYNC seems to not usable if without WP_UNPOPULATE. This also gives us
chance to modify impl of WP_ASYNC just in case it could be not depending on
WP_UNPOPULATED anymore in the future kernels. It's also fine to imply that
because both features will rely on PTE_MARKER_UFFD_WP config option, so
they'll show up together (or both missing) in an UFFDIO_API probe.
vma_can_userfault() now allows any VMA if the userfaultfd registration is
only about async uffd-wp. So we can track dirty for all kinds of memory
including generic file systems (like XFS, EXT4 or BTRFS).
One trick worth mention in do_wp_page() is that we need to manually update
vmf->orig_pte here because it can be used later with a pte_same() check -
this path always has FAULT_FLAG_ORIG_PTE_VALID set in the flags.
The major defect of this approach of dirty tracking is we need to populate
the pgtables when tracking starts. Soft-dirty doesn't do it like that.
It's unwanted in the case where the range of memory to track is huge and
unpopulated (e.g., tracking updates on a 10G file with mmap() on top,
without having any page cache installed yet). One way to improve this is
to allow pte markers exist for larger than PTE level for PMD+. That will
not change the interface if to implemented, so we can leave that for later.
Co-developed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
Changes in v12:
- Peter added the hugetlb support and revamped some other implementation
- Transferred the authorship to Peter
- Merge documentation to this patch
Changes in v11:
- Fix return code in userfaultfd_register() and minor changes here and
there
- Rebase on top of next-20230307
- Base patches on UFFD_FEATURE_WP_UNPOPULATED https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com
- UFFD_FEATURE_WP_ASYNC depends on UFFD_FEATURE_WP_UNPOPULATED to work
(correctly)
Changes in v10:
- Build fix
- Update comments and add error condition to return error from uffd
register if hugetlb pages are present when wp async flag is set
Changes in v9:
- Correct the fault resolution with code contributed by Peter
Changes in v7:
- Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC
- Handle automatic page fault resolution in better way (thanks to Peter)
---
Documentation/admin-guide/mm/userfaultfd.rst | 35 ++++++++++++++++++++
fs/userfaultfd.c | 26 ++++++++++++---
include/linux/userfaultfd_k.h | 21 +++++++++++-
include/uapi/linux/userfaultfd.h | 9 ++++-
mm/hugetlb.c | 32 ++++++++++--------
mm/memory.c | 27 +++++++++++++--
6 files changed, 128 insertions(+), 22 deletions(-)
diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst
index 7c304e432205..4b7f43fbbe18 100644
--- a/Documentation/admin-guide/mm/userfaultfd.rst
+++ b/Documentation/admin-guide/mm/userfaultfd.rst
@@ -244,6 +244,41 @@ write-protected (so future writes will also result in a WP fault). These ioctls
support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP``
respectively) to configure the mapping this way.
+If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set,
+any vma registered with write-protection will work in async mode rather
+than the default sync mode.
+
+In async mode, there will be no message generated when a write operation
+happens, meanwhile the write-protection will be resolved automatically by
+the kernel. It can be seen as a more accurate version of soft-dirty
+tracking and it can be different in a few ways:
+
+ - The dirty result will not be affected by vma changes (e.g. vma
+ merging) because the dirty is only tracked by the pte.
+
+ - It supports range operations by default, so one can enable tracking on
+ any range of memory as long as page aligned.
+
+ - Dirty information will not get lost if the pte was zapped due to
+ various reasons (e.g. during split of a shmem transparent huge page).
+
+ - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit
+ set; dirty when uffd-wp bit cleared), it has different semantics on
+ some of the memory operations. For example: ``MADV_DONTNEED`` on
+ anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as
+ dirtying of memory by dropping uffd-wp bit during the procedure.
+
+The user app can collect the "written/dirty" status by looking up the
+uffd-wp bit for the pages being interested in /proc/pagemap.
+
+The page will not be under track of uffd-wp async mode until the page is
+explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode
+flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault
+that was tracked by async mode userfaultfd-wp is invalid.
+
+When userfaultfd-wp async mode is used alone, it can be applied to all
+kinds of memory.
+
QEMU/KVM
========
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 0fd96d6e39ce..3ab021521a96 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -123,6 +123,11 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
return ctx->features & UFFD_FEATURE_INITIALIZED;
}
+static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
+{
+ return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
+}
+
/*
* Whether WP_UNPOPULATED is enabled on the uffd context. It is only
* meaningful when userfaultfd_wp()==true on the vma and when it's
@@ -1332,6 +1337,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
bool basic_ioctls;
unsigned long start, end, vma_end;
struct vma_iterator vmi;
+ bool wp_async = userfaultfd_wp_async_ctx(ctx);
user_uffdio_register = (struct uffdio_register __user *) arg;
@@ -1405,7 +1411,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
/* check not compatible vmas */
ret = -EINVAL;
- if (!vma_can_userfault(cur, vm_flags))
+ if (!vma_can_userfault(cur, vm_flags, wp_async))
goto out_unlock;
/*
@@ -1464,7 +1470,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
for_each_vma_range(vmi, vma, end) {
cond_resched();
- BUG_ON(!vma_can_userfault(vma, vm_flags));
+ BUG_ON(!vma_can_userfault(vma, vm_flags, wp_async));
BUG_ON(vma->vm_userfaultfd_ctx.ctx &&
vma->vm_userfaultfd_ctx.ctx != ctx);
WARN_ON(!(vma->vm_flags & VM_MAYWRITE));
@@ -1563,6 +1569,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
unsigned long start, end, vma_end;
const void __user *buf = (void __user *)arg;
struct vma_iterator vmi;
+ bool wp_async = userfaultfd_wp_async_ctx(ctx);
ret = -EFAULT;
if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
@@ -1616,7 +1623,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
* provides for more strict behavior to notice
* unregistration errors.
*/
- if (!vma_can_userfault(cur, cur->vm_flags))
+ if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
goto out_unlock;
found = true;
@@ -1629,7 +1636,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
for_each_vma_range(vmi, vma, end) {
cond_resched();
- BUG_ON(!vma_can_userfault(vma, vma->vm_flags));
+ BUG_ON(!vma_can_userfault(vma, vma->vm_flags, wp_async));
/*
* Nothing to do: this vma is already registered into this
@@ -1966,6 +1973,11 @@ static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
return ret;
}
+bool userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+ return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
+}
+
static inline unsigned int uffd_ctx_features(__u64 user_features)
{
/*
@@ -1999,6 +2011,11 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
ret = -EPERM;
if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
goto err_out;
+
+ /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
+ if (features & UFFD_FEATURE_WP_ASYNC)
+ features |= UFFD_FEATURE_WP_UNPOPULATED;
+
/* report all available features and ioctls to userland */
uffdio_api.features = UFFD_API_FEATURES;
#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
@@ -2011,6 +2028,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
#ifndef CONFIG_PTE_MARKER_UFFD_WP
uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
+ uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
#endif
uffdio_api.ioctls = UFFD_API_IOCTLS;
ret = -EFAULT;
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index d78b01524349..ce4f9d18de3e 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -157,11 +157,22 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
}
static inline bool vma_can_userfault(struct vm_area_struct *vma,
- unsigned long vm_flags)
+ unsigned long vm_flags,
+ bool wp_async)
{
+ vm_flags &= __VM_UFFD_FLAGS;
+
if ((vm_flags & VM_UFFD_MINOR) &&
(!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
return false;
+
+ /*
+ * If wp async enabled, and WP is the only mode enabled, allow any
+ * memory type.
+ */
+ if (wp_async && (vm_flags == VM_UFFD_WP))
+ return true;
+
#ifndef CONFIG_PTE_MARKER_UFFD_WP
/*
* If user requested uffd-wp but not enabled pte markers for
@@ -171,6 +182,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma,
if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma))
return false;
#endif
+
+ /* By default, allow any of anon|shmem|hugetlb */
return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
vma_is_shmem(vma);
}
@@ -193,6 +206,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
extern void userfaultfd_unmap_complete(struct mm_struct *mm,
struct list_head *uf);
extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
+extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
#else /* CONFIG_USERFAULTFD */
@@ -293,6 +307,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
return false;
}
+static inline bool userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+ return false;
+}
+
#endif /* CONFIG_USERFAULTFD */
static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
index 66dd4cd277bd..cfb87a112a9f 100644
--- a/include/uapi/linux/userfaultfd.h
+++ b/include/uapi/linux/userfaultfd.h
@@ -39,7 +39,8 @@
UFFD_FEATURE_MINOR_SHMEM | \
UFFD_FEATURE_EXACT_ADDRESS | \
UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
- UFFD_FEATURE_WP_UNPOPULATED)
+ UFFD_FEATURE_WP_UNPOPULATED | \
+ UFFD_FEATURE_WP_ASYNC)
#define UFFD_API_IOCTLS \
((__u64)1 << _UFFDIO_REGISTER | \
(__u64)1 << _UFFDIO_UNREGISTER | \
@@ -210,6 +211,11 @@ struct uffdio_api {
* (i.e. empty ptes). This will be the default behavior for shmem
* & hugetlbfs, so this flag only affects anonymous memory behavior
* when userfault write-protection mode is registered.
+ *
+ * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
+ * asynchronous mode is supported in which the write fault is
+ * automatically resolved and write-protection is un-set.
+ * It implies UFFD_FEATURE_WP_UNPOPULATED.
*/
#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
#define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -225,6 +231,7 @@ struct uffdio_api {
#define UFFD_FEATURE_EXACT_ADDRESS (1<<11)
#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
#define UFFD_FEATURE_WP_UNPOPULATED (1<<13)
+#define UFFD_FEATURE_WP_ASYNC (1<<14)
__u64 features;
__u64 ioctls;
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 155bb6712be1..0d0efd96449e 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -6150,21 +6150,27 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
/* Handle userfault-wp first, before trying to lock more pages */
if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) &&
(flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {
- struct vm_fault vmf = {
- .vma = vma,
- .address = haddr,
- .real_address = address,
- .flags = flags,
- };
+ if (!userfaultfd_wp_async(vma)) {
+ struct vm_fault vmf = {
+ .vma = vma,
+ .address = haddr,
+ .real_address = address,
+ .flags = flags,
+ };
- spin_unlock(ptl);
- if (pagecache_folio) {
- folio_unlock(pagecache_folio);
- folio_put(pagecache_folio);
+ spin_unlock(ptl);
+ if (pagecache_folio) {
+ folio_unlock(pagecache_folio);
+ folio_put(pagecache_folio);
+ }
+ hugetlb_vma_unlock_read(vma);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ return handle_userfault(&vmf, VM_UFFD_WP);
}
- hugetlb_vma_unlock_read(vma);
- mutex_unlock(&hugetlb_fault_mutex_table[hash]);
- return handle_userfault(&vmf, VM_UFFD_WP);
+
+ entry = huge_pte_clear_uffd_wp(entry);
+ set_huge_pte_at(mm, haddr, ptep, entry);
+ /* Fallthrough to CoW */
}
/*
diff --git a/mm/memory.c b/mm/memory.c
index 1c5b231fe6e3..60f1a4e348b5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3331,11 +3331,28 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
struct vm_area_struct *vma = vmf->vma;
struct folio *folio = NULL;
+ pte_t pte;
if (likely(!unshare)) {
if (userfaultfd_pte_wp(vma, *vmf->pte)) {
- pte_unmap_unlock(vmf->pte, vmf->ptl);
- return handle_userfault(vmf, VM_UFFD_WP);
+ if (!userfaultfd_wp_async(vma)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return handle_userfault(vmf, VM_UFFD_WP);
+ }
+
+ /*
+ * Nothing needed (cache flush, TLB invalidations,
+ * etc.) because we're only removing the uffd-wp bit,
+ * which is completely invisible to the user.
+ */
+ pte = pte_clear_uffd_wp(*vmf->pte);
+
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ /*
+ * Update this to be prepared for following up CoW
+ * handling
+ */
+ vmf->orig_pte = pte;
}
/*
@@ -4824,8 +4841,11 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
if (vma_is_anonymous(vmf->vma)) {
if (likely(!unshare) &&
- userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
+ if (userfaultfd_wp_async(vmf->vma))
+ goto split;
return handle_userfault(vmf, VM_UFFD_WP);
+ }
return do_huge_pmd_wp_page(vmf);
}
@@ -4837,6 +4857,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
}
}
+split:
/* COW or write-notify handled on pte level: split pmd. */
__split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
--
2.39.2
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-04-20 6:01 [PATCH RESEND v15 0/5] Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 1/5] userfaultfd: UFFD_FEATURE_WP_ASYNC Muhammad Usama Anjum
@ 2023-04-20 6:01 ` Muhammad Usama Anjum
2023-04-26 7:06 ` Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 3/5] tools headers UAPI: Update linux/fs.h with the kernel sources Muhammad Usama Anjum
` (2 subsequent siblings)
4 siblings, 1 reply; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-20 6:01 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Mike Rapoport, Nadav Amit
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-mm,
linux-kselftest, Greg KH, kernel
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are supported
in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE)
This IOCTL can be extended to get information about more PTE bits.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v15:
- Build fix:
- Use generic tlb flush function in pagemap_scan_pmd_entry() instead of
using x86 specific flush function in do_pagemap_scan()
- Remove #ifdef from pagemap_scan_hugetlb_entry()
- Use mm instead of undefined vma->vm_mm
Changes in v14:
- Fix build error caused by #ifdef added at last minute in some configs
Changes in v13:
- Review updates
- mmap_read_lock_killable() instead of mmap_read_lock()
- Replace uffd_wp_range() with helpers which increases performance
drastically for OP_WP operations by reducing the number of tlb
flushing etc
- Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range
Changes in v12:
- Add hugetlb support to cover all memory types
- Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
- Review updates to the code
Changes in v11:
- Find written pages in a better way
- Fix a corner case (thanks Paul)
- Improve the code/comments
- remove ENGAGE_WP + ! GET operation
- shorten the commit message in favour of moving documentation to
pagemap.rst
Changes in v10:
- move changes in tools/include/uapi/linux/fs.h to separate patch
- update commit message
Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code
Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
Changes in v6:
- Rename variables and update comments
- Make IOCTL independent of soft_dirty config
- Change masks and bitmap type to _u64
- Improve code quality
Changes in v5:
- Remove tlb flushing even for clear operation
Changes in v4:
- Update the interface and implementation
Changes in v3:
- Tighten the user-kernel interface by using explicit types and add more
error checking
Changes in v2:
- Convert the interface from syscall to ioctl
- Remove pidfd support as it doesn't make sense in ioctl
task_mmu
task_mmu
task_mmu.c
---
fs/proc/task_mmu.c | 481 ++++++++++++++++++++++++++++++++++++++++
include/uapi/linux/fs.h | 53 +++++
2 files changed, 534 insertions(+)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 38b19a757281..e119e13a9ba5 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -19,6 +19,7 @@
#include <linux/shmem_fs.h>
#include <linux/uaccess.h>
#include <linux/pkeys.h>
+#include <linux/minmax.h>
#include <asm/elf.h>
#include <asm/tlb.h>
@@ -1767,11 +1768,491 @@ static int pagemap_release(struct inode *inode, struct file *file)
return 0;
}
+#define PM_SCAN_FOUND_MAX_PAGES (1)
+#define PM_SCAN_BITS_ALL (PAGE_IS_WRITTEN | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PM_SCAN_OPS (PM_SCAN_OP_GET | PM_SCAN_OP_WP)
+#define PM_SCAN_DO_UFFD_WP(a) (a->flags & PM_SCAN_OP_WP)
+#define PM_SCAN_BITMAP(wt, file, present, swap) \
+ ((wt) | ((file) << 1) | ((present) << 2) | ((swap) << 3))
+
+struct pagemap_scan_private {
+ struct page_region *vec;
+ struct page_region cur;
+ unsigned long vec_len, vec_index;
+ unsigned int max_pages, found_pages, flags;
+ unsigned long required_mask, anyof_mask, excluded_mask, return_mask;
+};
+
+static inline bool is_pte_uffd_wp(pte_t pte)
+{
+ return (pte_present(pte) && pte_uffd_wp(pte)) ||
+ pte_swp_uffd_wp_any(pte);
+}
+
+static inline void make_uffd_wp_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *pte)
+{
+ pte_t ptent = *pte;
+
+ if (pte_present(ptent)) {
+ pte_t old_pte;
+
+ old_pte = ptep_modify_prot_start(vma, addr, pte);
+ ptent = pte_mkuffd_wp(ptent);
+ ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
+ } else if (is_swap_pte(ptent)) {
+ ptent = pte_swp_mkuffd_wp(ptent);
+ set_pte_at(vma->vm_mm, addr, pte, ptent);
+ }
+}
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static inline bool is_pmd_uffd_wp(pmd_t pmd)
+{
+ return (pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
+ (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd));
+}
+
+static inline void make_uffd_wp_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmdp)
+{
+ pmd_t old, pmd = *pmdp;
+
+ if (pmd_present(pmd)) {
+ old = pmdp_invalidate_ad(vma, addr, pmdp);
+ pmd = pmd_mkuffd_wp(old);
+ set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+ } else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
+ pmd = pmd_swp_mkuffd_wp(pmd);
+ set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+ }
+}
+#endif
+
+#ifdef CONFIG_HUGETLB_PAGE
+static inline bool is_huge_pte_uffd_wp(pte_t pte)
+{
+ return ((pte_present(pte) && huge_pte_uffd_wp(pte)) ||
+ pte_swp_uffd_wp_any(pte));
+}
+
+static inline void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ pte_t ptent)
+{
+ pte_t old_pte;
+
+ if (!huge_pte_none(ptent)) {
+ old_pte = huge_ptep_modify_prot_start(vma, addr, ptep);
+ ptent = huge_pte_mkuffd_wp(old_pte);
+ ptep_modify_prot_commit(vma, addr, ptep, old_pte, ptent);
+ } else {
+ set_huge_pte_at(vma->vm_mm, addr, ptep,
+ make_pte_marker(PTE_MARKER_UFFD_WP));
+ }
+}
+#endif
+
+static inline bool pagemap_scan_check_page_written(struct pagemap_scan_private *p)
+{
+ return (p->required_mask | p->anyof_mask | p->excluded_mask) &
+ PAGE_IS_WRITTEN;
+}
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
+ struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+
+ if (pagemap_scan_check_page_written(p) && (!userfaultfd_wp(vma) ||
+ !userfaultfd_wp_async(vma)))
+ return -EPERM;
+
+ if (vma->vm_flags & VM_PFNMAP)
+ return 1;
+
+ return 0;
+}
+
+static int pagemap_scan_output(bool wt, bool file, bool pres, bool swap,
+ struct pagemap_scan_private *p,
+ unsigned long addr, unsigned int n_pages)
+{
+ unsigned long bitmap = PM_SCAN_BITMAP(wt, file, pres, swap);
+ struct page_region *cur = &p->cur;
+
+ if (!n_pages)
+ return -EINVAL;
+
+ if ((p->required_mask & bitmap) != p->required_mask)
+ return 0;
+ if (p->anyof_mask && !(p->anyof_mask & bitmap))
+ return 0;
+ if (p->excluded_mask & bitmap)
+ return 0;
+
+ bitmap &= p->return_mask;
+ if (!bitmap)
+ return 0;
+
+ if (cur->bitmap == bitmap &&
+ cur->start + cur->len * PAGE_SIZE == addr) {
+ cur->len += n_pages;
+ p->found_pages += n_pages;
+
+ if (p->max_pages && (p->found_pages == p->max_pages))
+ return PM_SCAN_FOUND_MAX_PAGES;
+
+ return 0;
+ }
+
+ /*
+ * All data is copied to cur first. When more data is found, we push
+ * cur to vec and copy new data to cur. The vec_index represents the
+ * current index of vec array. We add 1 to the vec_index while
+ * performing checks to account for data in cur.
+ */
+ if (p->vec_index && (p->vec_index + 1) >= p->vec_len)
+ return -ENOSPC;
+
+ if (cur->len) {
+ memcpy(&p->vec[p->vec_index], cur, sizeof(*p->vec));
+ p->vec_index++;
+ }
+
+ cur->start = addr;
+ cur->len = n_pages;
+ cur->bitmap = bitmap;
+ p->found_pages += n_pages;
+
+ if (p->max_pages && (p->found_pages == p->max_pages))
+ return PM_SCAN_FOUND_MAX_PAGES;
+
+ return 0;
+}
+
+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+ unsigned long end, struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ unsigned long addr = end;
+ pte_t *pte, *orig_pte;
+ spinlock_t *ptl;
+ bool is_written;
+ int ret = 0;
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ unsigned long n_pages = (end - start)/PAGE_SIZE;
+
+ if (p->max_pages && n_pages > p->max_pages - p->found_pages)
+ n_pages = p->max_pages - p->found_pages;
+
+ is_written = !is_pmd_uffd_wp(*pmd);
+
+ /*
+ * Break huge page into small pages if the WP operation need to
+ * be performed is on a portion of the huge page.
+ */
+ if (is_written && PM_SCAN_DO_UFFD_WP(p) &&
+ n_pages < HPAGE_SIZE/PAGE_SIZE) {
+ spin_unlock(ptl);
+ split_huge_pmd(vma, pmd, start);
+ goto process_smaller_pages;
+ }
+
+ ret = pagemap_scan_output(is_written, vma->vm_file,
+ pmd_present(*pmd), is_swap_pmd(*pmd),
+ p, start, n_pages);
+
+ if (ret >= 0 && is_written && PM_SCAN_DO_UFFD_WP(p))
+ make_uffd_wp_pmd(vma, addr, pmd);
+
+ spin_unlock(ptl);
+
+ if (PM_SCAN_DO_UFFD_WP(p))
+ flush_tlb_range(vma, start, end);
+
+ return ret;
+ }
+process_smaller_pages:
+ if (pmd_trans_unstable(pmd))
+ return 0;
+#endif
+
+ orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+ for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE) {
+ is_written = !is_pte_uffd_wp(*pte);
+
+ ret = pagemap_scan_output(is_written, vma->vm_file,
+ pte_present(*pte), is_swap_pte(*pte),
+ p, addr, 1);
+
+ if (ret >= 0 && is_written && PM_SCAN_DO_UFFD_WP(p))
+ make_uffd_wp_pte(vma, addr, pte);
+ }
+ pte_unmap_unlock(orig_pte, ptl);
+
+ if (PM_SCAN_DO_UFFD_WP(p))
+ flush_tlb_range(vma, start, addr);
+
+ cond_resched();
+ return ret;
+}
+
+#ifdef CONFIG_HUGETLB_PAGE
+static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
+ unsigned long start, unsigned long end,
+ struct mm_walk *walk)
+{
+ unsigned long n_pages = (end - start)/PAGE_SIZE;
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ struct hstate *h = hstate_vma(vma);
+ int ret = -EPERM;
+ spinlock_t *ptl;
+ bool is_written;
+ pte_t pte;
+
+ if (p->max_pages && n_pages > p->max_pages - p->found_pages)
+ n_pages = p->max_pages - p->found_pages;
+
+ if (PM_SCAN_DO_UFFD_WP(p)) {
+ i_mmap_lock_write(vma->vm_file->f_mapping);
+ ptl = huge_pte_lock(h, vma->vm_mm, ptep);
+ }
+
+ pte = huge_ptep_get(ptep);
+ is_written = !is_huge_pte_uffd_wp(pte);
+
+ /*
+ * Partial hugetlb page clear isn't supported
+ */
+ if (is_written && PM_SCAN_DO_UFFD_WP(p) &&
+ n_pages < HPAGE_SIZE/PAGE_SIZE)
+ goto unlock_and_return;
+
+ ret = pagemap_scan_output(is_written, vma->vm_file, pte_present(pte),
+ is_swap_pte(pte), p, start, n_pages);
+ if (ret < 0)
+ goto unlock_and_return;
+
+ if (is_written && PM_SCAN_DO_UFFD_WP(p)) {
+ make_uffd_wp_huge_pte(vma, start, ptep, pte);
+ flush_hugetlb_tlb_range(vma, start, end);
+ }
+
+unlock_and_return:
+ if (PM_SCAN_DO_UFFD_WP(p)) {
+ spin_unlock(ptl);
+ i_mmap_unlock_write(vma->vm_file->f_mapping);
+ }
+
+ return ret;
+}
+#else
+#define pagemap_scan_hugetlb_entry NULL
+#endif
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
+ int depth, struct mm_walk *walk)
+{
+ unsigned long n_pages = (end - addr)/PAGE_SIZE;
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ int ret = 0;
+
+ if (!vma)
+ return 0;
+
+ if (p->max_pages && p->found_pages + n_pages > p->max_pages)
+ n_pages = p->max_pages - p->found_pages;
+
+ ret = pagemap_scan_output(false, vma->vm_file, false, false, p, addr,
+ n_pages);
+ return ret;
+}
+
+static const struct mm_walk_ops pagemap_scan_ops = {
+ .test_walk = pagemap_scan_test_walk,
+ .pmd_entry = pagemap_scan_pmd_entry,
+ .pte_hole = pagemap_scan_pte_hole,
+ .hugetlb_entry = pagemap_scan_hugetlb_entry,
+};
+
+static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start,
+ struct page_region __user *vec)
+{
+ /* Detect illegal size, flags, len and masks */
+ if (arg->size != sizeof(struct pm_scan_arg))
+ return -EINVAL;
+ if (arg->flags & ~PM_SCAN_OPS)
+ return -EINVAL;
+ if (!arg->len)
+ return -EINVAL;
+ if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask |
+ arg->return_mask) & ~PM_SCAN_BITS_ALL)
+ return -EINVAL;
+ if (!arg->required_mask && !arg->anyof_mask &&
+ !arg->excluded_mask)
+ return -EINVAL;
+ if (!arg->return_mask)
+ return -EINVAL;
+
+ /* Validate memory ranges */
+ if (!(arg->flags & PM_SCAN_OP_GET))
+ return -EINVAL;
+ if (!arg->vec)
+ return -EINVAL;
+ if (arg->vec_len == 0)
+ return -EINVAL;
+
+ if (!IS_ALIGNED(start, PAGE_SIZE))
+ return -EINVAL;
+ if (!access_ok((void __user *)start, arg->len))
+ return -EFAULT;
+
+ if (!PM_SCAN_DO_UFFD_WP(arg))
+ return 0;
+
+ if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask) &
+ ~PAGE_IS_WRITTEN)
+ return -EINVAL;
+
+ return 0;
+}
+
+static long do_pagemap_scan(struct mm_struct *mm,
+ struct pm_scan_arg __user *uarg)
+{
+ unsigned long start, end, walk_start, walk_end;
+ unsigned long empty_slots, vec_index = 0;
+ struct mmu_notifier_range range;
+ struct page_region __user *vec;
+ struct pagemap_scan_private p;
+ struct pm_scan_arg arg;
+ int ret = 0;
+
+ if (copy_from_user(&arg, uarg, sizeof(arg)))
+ return -EFAULT;
+
+ start = untagged_addr((unsigned long)arg.start);
+ vec = (struct page_region *)untagged_addr((unsigned long)arg.vec);
+
+ ret = pagemap_scan_args_valid(&arg, start, vec);
+ if (ret)
+ return ret;
+
+ end = start + arg.len;
+ p.max_pages = arg.max_pages;
+ p.found_pages = 0;
+ p.flags = arg.flags;
+ p.required_mask = arg.required_mask;
+ p.anyof_mask = arg.anyof_mask;
+ p.excluded_mask = arg.excluded_mask;
+ p.return_mask = arg.return_mask;
+ p.cur.len = 0;
+ p.cur.start = 0;
+ p.vec = NULL;
+ p.vec_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
+
+ /*
+ * Allocate smaller buffer to get output from inside the page walk
+ * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
+ * we want to return output to user in compact form where no two
+ * consecutive regions should be continuous and have the same flags.
+ * So store the latest element in p.cur between different walks and
+ * store the p.cur at the end of the walk to the user buffer.
+ */
+ p.vec = kmalloc_array(p.vec_len, sizeof(*p.vec), GFP_KERNEL);
+ if (!p.vec)
+ return -ENOMEM;
+
+ if (p.flags & PM_SCAN_OP_WP) {
+ mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
+ mm, start, end);
+ mmu_notifier_invalidate_range_start(&range);
+ }
+
+ walk_start = walk_end = start;
+ while (walk_end < end && !ret) {
+ p.vec_index = 0;
+
+ empty_slots = arg.vec_len - vec_index;
+ p.vec_len = min(p.vec_len, empty_slots);
+
+ walk_end = (walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
+ if (walk_end > end)
+ walk_end = end;
+
+ ret = mmap_read_lock_killable(mm);
+ if (ret)
+ goto free_data;
+ ret = walk_page_range(mm, walk_start, walk_end,
+ &pagemap_scan_ops, &p);
+ mmap_read_unlock(mm);
+
+ if (ret && ret != -ENOSPC && ret != PM_SCAN_FOUND_MAX_PAGES)
+ goto free_data;
+
+ walk_start = walk_end;
+ if (p.vec_index) {
+ if (copy_to_user(&vec[vec_index], p.vec,
+ p.vec_index * sizeof(*p.vec))) {
+ /*
+ * Return error even though the OP succeeded
+ */
+ ret = -EFAULT;
+ goto free_data;
+ }
+ vec_index += p.vec_index;
+ }
+ }
+
+ if (p.flags & PM_SCAN_OP_WP)
+ mmu_notifier_invalidate_range_end(&range);
+
+ if (p.cur.len) {
+ if (copy_to_user(&vec[vec_index], &p.cur, sizeof(*p.vec))) {
+ ret = -EFAULT;
+ goto free_data;
+ }
+ vec_index++;
+ }
+
+ ret = vec_index;
+
+free_data:
+ kfree(p.vec);
+ return ret;
+}
+
+static long do_pagemap_cmd(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)arg;
+ struct mm_struct *mm = file->private_data;
+
+ switch (cmd) {
+ case PAGEMAP_SCAN:
+ return do_pagemap_scan(mm, uarg);
+
+ default:
+ return -EINVAL;
+ }
+}
+
const struct file_operations proc_pagemap_operations = {
.llseek = mem_lseek, /* borrow this */
.read = pagemap_read,
.open = pagemap_open,
.release = pagemap_release,
+ .unlocked_ioctl = do_pagemap_cmd,
+ .compat_ioctl = do_pagemap_cmd,
};
#endif /* CONFIG_PROC_PAGE_MONITOR */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index b7b56871029c..47879c38ce2f 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -305,4 +305,57 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WRITTEN (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region in pages
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size: Size of the structure
+ * @flags: Flags for the IOCTL
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pm_scan_arg {
+ __u64 size;
+ __u64 flags;
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u64 max_pages;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Supported flags */
+#define PM_SCAN_OP_GET (1 << 0)
+#define PM_SCAN_OP_WP (1 << 1)
+
#endif /* _UAPI_LINUX_FS_H */
--
2.39.2
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH RESEND v15 3/5] tools headers UAPI: Update linux/fs.h with the kernel sources
2023-04-20 6:01 [PATCH RESEND v15 0/5] Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 1/5] userfaultfd: UFFD_FEATURE_WP_ASYNC Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
@ 2023-04-20 6:01 ` Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 4/5] mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 5/5] selftests: mm: add pagemap ioctl tests Muhammad Usama Anjum
4 siblings, 0 replies; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-20 6:01 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Mike Rapoport, Nadav Amit
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-mm,
linux-kselftest, Greg KH, kernel
New IOCTL and macros has been added in the kernel sources. Update the
tools header file as well.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
tools/include/uapi/linux/fs.h | 53 +++++++++++++++++++++++++++++++++++
1 file changed, 53 insertions(+)
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index b7b56871029c..47879c38ce2f 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -305,4 +305,57 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WRITTEN (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region in pages
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size: Size of the structure
+ * @flags: Flags for the IOCTL
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pm_scan_arg {
+ __u64 size;
+ __u64 flags;
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u64 max_pages;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Supported flags */
+#define PM_SCAN_OP_GET (1 << 0)
+#define PM_SCAN_OP_WP (1 << 1)
+
#endif /* _UAPI_LINUX_FS_H */
--
2.39.2
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH RESEND v15 4/5] mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL
2023-04-20 6:01 [PATCH RESEND v15 0/5] Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
` (2 preceding siblings ...)
2023-04-20 6:01 ` [PATCH RESEND v15 3/5] tools headers UAPI: Update linux/fs.h with the kernel sources Muhammad Usama Anjum
@ 2023-04-20 6:01 ` Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 5/5] selftests: mm: add pagemap ioctl tests Muhammad Usama Anjum
4 siblings, 0 replies; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-20 6:01 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Mike Rapoport, Nadav Amit
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-mm,
linux-kselftest, Greg KH, kernel
Add some explanation and method to use write-protection and written-to
on memory range.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v11:
- Add more documentation
---
Documentation/admin-guide/mm/pagemap.rst | 56 ++++++++++++++++++++++++
1 file changed, 56 insertions(+)
diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst
index c8f380271cad..a3e08f15b433 100644
--- a/Documentation/admin-guide/mm/pagemap.rst
+++ b/Documentation/admin-guide/mm/pagemap.rst
@@ -227,3 +227,59 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is
always 12 at most architectures). Since Linux 3.11 their meaning changes
after first clear of soft-dirty bits. Since Linux 4.2 they are used for
flags unconditionally.
+
+Pagemap Scan IOCTL
+==================
+
+The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally
+clear the info about page table entries. The following operations are supported
+in this IOCTL:
+- Get the information if the pages have been written-to (``PAGE_IS_WRITTEN``),
+ file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``) or swapped
+ (``PAGE_IS_SWAPPED``).
+- Find pages which have been written-to and write protect the pages atomically
+ (atomic ``PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE``)
+
+The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
+ 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size``
+ field. This field will be helpful in recognizing the structure if extensions
+ are done later.
+ 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_OP_GET``
+ and ``PM_SCAN_OP_WP`` are the only added flag at this time.
+ 3. The range is specified through ``start`` and ``len``.
+ 4. The output buffer of ``struct page_region`` array and size is specified in
+ ``vec`` and ``vec_len``.
+ 5. The optional maximum requested pages are specified in the ``max_pages``.
+ 6. The masks are specified in ``required_mask``, ``anyof_mask``,
+ ``excluded_ mask`` and ``return_mask``.
+ 1. To find if ``PAGE_IS_WRITTEN`` flag is set for pages which have
+ ``PAGE_IS_FILE`` set and ``PAGE_IS_SWAPPED`` un-set, ``required_mask``
+ is set to ``PAGE_IS_FILE``, ``exclude_mask`` is set to
+ ``PAGE_IS_SWAPPED`` and ``return_mask`` is set to ``PAGE_IS_WRITTEN``.
+ The output buffer in ``vec`` and length must be specified in ``vec_len``.
+ 2. To find pages which have either ``PAGE_IS_FILE`` or ``PAGE_IS_SWAPPED``
+ set, ``anyof_masks`` is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``.
+ 3. To find written pages and engage write protect, ``PAGE_IS_WRITTEN`` is
+ specified in ``required_mask`` and ``return_mask``. In addition to
+ specifying the output buffer in ``vec`` and length in ``vec_len``, the
+ ``PAGEMAP_WP_ENGAGE`` is specified in ``flags`` to perform write protect
+ on the range as well.
+
+The ``PAGE_IS_WRITTEN`` flag can be considered as the better and correct
+alternative of soft-dirty flag. It doesn't get affected by household chores (VMA
+merging) of the kernel and hence the user can find the true soft-dirty pages
+only. This IOCTL adds the atomic way to find which pages have been written and
+write protect those pages again. This kind of operation is needed to efficiently
+find out which pages have changed in the memory.
+
+To get information about which pages have been written-to or optionally write
+protect the pages, following must be performed first in order:
+ 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall.
+ 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features
+ are set by ``UFFDIO_API`` IOCTL.
+ 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode
+ through ``UFFDIO_REGISTER`` IOCTL.
+ 4. Then the any part of the registered memory or the whole memory region must
+ be write protected using the ``UFFDIO_WRITEPROTECT`` IOCTL.
+ 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which
+ have been written-to or optionally write protect the pages as well.
--
2.39.2
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH RESEND v15 5/5] selftests: mm: add pagemap ioctl tests
2023-04-20 6:01 [PATCH RESEND v15 0/5] Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
` (3 preceding siblings ...)
2023-04-20 6:01 ` [PATCH RESEND v15 4/5] mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL Muhammad Usama Anjum
@ 2023-04-20 6:01 ` Muhammad Usama Anjum
4 siblings, 0 replies; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-20 6:01 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Mike Rapoport, Nadav Amit
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-mm,
linux-kselftest, Greg KH, kernel
Add pagemap ioctl tests. Add several different types of tests to judge
the correction of the interface.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v13:
- Update tests and rebase Makefile
Changes in v12:
- Updates and add more memory type tests
Changes in v11:
- Rebase on top of next-20230216 and update tests
Chages in v7:
- Add and update all test cases
Changes in v6:
- Rename variables
Changes in v4:
- Updated all the tests to conform to new IOCTL
Changes in v3:
- Add another test to do sanity of flags
Changes in v2:
- Update the tests to use the ioctl interface instead of syscall
selftests
---
tools/testing/selftests/mm/.gitignore | 1 +
tools/testing/selftests/mm/Makefile | 3 +-
tools/testing/selftests/mm/config | 1 +
tools/testing/selftests/mm/pagemap_ioctl.c | 1326 ++++++++++++++++++++
tools/testing/selftests/mm/run_vmtests.sh | 4 +
5 files changed, 1334 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh
diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore
index 8917455f4f51..f1a06f842d55 100644
--- a/tools/testing/selftests/mm/.gitignore
+++ b/tools/testing/selftests/mm/.gitignore
@@ -17,6 +17,7 @@ mremap_dontunmap
mremap_test
on-fault-limit
transhuge-stress
+pagemap_ioctl
protection_keys
protection_keys_32
protection_keys_64
diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index dda8598bf5ef..f28c10c3dc8b 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -30,7 +30,7 @@ MACHINE ?= $(shell echo $(uname_M) | sed -e 's/aarch64.*/arm64/' -e 's/ppc64.*/p
MAKEFLAGS += --no-builtin-rules
CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES)
-LDLIBS = -lrt -lpthread
+LDLIBS = -lrt -lpthread -lm
TEST_GEN_PROGS = cow
TEST_GEN_PROGS += compaction_test
@@ -55,6 +55,7 @@ TEST_GEN_PROGS += mrelease_test
TEST_GEN_PROGS += mremap_dontunmap
TEST_GEN_PROGS += mremap_test
TEST_GEN_PROGS += on-fault-limit
+TEST_GEN_PROGS += pagemap_ioctl
TEST_GEN_PROGS += thuge-gen
TEST_GEN_PROGS += transhuge-stress
TEST_GEN_PROGS += uffd-stress
diff --git a/tools/testing/selftests/mm/config b/tools/testing/selftests/mm/config
index be087c4bc396..4309916f629e 100644
--- a/tools/testing/selftests/mm/config
+++ b/tools/testing/selftests/mm/config
@@ -1,5 +1,6 @@
CONFIG_SYSVIPC=y
CONFIG_USERFAULTFD=y
+CONFIG_PTE_MARKER_UFFD_WP=y
CONFIG_TEST_VMALLOC=m
CONFIG_DEVICE_PRIVATE=y
CONFIG_TEST_HMM=m
diff --git a/tools/testing/selftests/mm/pagemap_ioctl.c b/tools/testing/selftests/mm/pagemap_ioctl.c
new file mode 100644
index 000000000000..deee7c618d6b
--- /dev/null
+++ b/tools/testing/selftests/mm/pagemap_ioctl.c
@@ -0,0 +1,1326 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <fcntl.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <malloc.h>
+#include "vm_util.h"
+#include "../kselftest.h"
+#include <linux/types.h>
+#include <linux/memfd.h>
+#include <linux/userfaultfd.h>
+#include <linux/fs.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <math.h>
+#include <asm/unistd.h>
+#include <pthread.h>
+#include <sys/resource.h>
+#include <assert.h>
+#include <sys/ipc.h>
+#include <sys/shm.h>
+
+#define PAGEMAP_BITS_ALL (PAGE_IS_WRITTEN | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PAGEMAP_NON_WRITTEN_BITS (PAGE_IS_FILE | PAGE_IS_PRESENT | \
+ PAGE_IS_SWAPPED)
+
+#define TEST_ITERATIONS 10
+#define PAGEMAP "/proc/self/pagemap"
+int pagemap_fd;
+int uffd;
+int page_size;
+int hpage_size;
+
+static long pagemap_ioctl(void *start, int len, void *vec, int vec_len, int flag,
+ int max_pages, long required_mask, long anyof_mask, long excluded_mask,
+ long return_mask)
+{
+ struct pm_scan_arg arg;
+
+ arg.start = (uintptr_t)start;
+ arg.len = len;
+ arg.vec = (uintptr_t)vec;
+ arg.vec_len = vec_len;
+ arg.flags = flag;
+ arg.size = sizeof(struct pm_scan_arg);
+ arg.max_pages = max_pages;
+ arg.required_mask = required_mask;
+ arg.anyof_mask = anyof_mask;
+ arg.excluded_mask = excluded_mask;
+ arg.return_mask = return_mask;
+
+ return ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+}
+
+int init_uffd(void)
+{
+ struct uffdio_api uffdio_api;
+
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd == -1)
+ ksft_exit_fail_msg("uffd syscall failed\n");
+
+ uffdio_api.api = UFFD_API;
+ uffdio_api.features = UFFD_FEATURE_WP_UNPOPULATED | UFFD_FEATURE_WP_ASYNC |
+ UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
+ if (ioctl(uffd, UFFDIO_API, &uffdio_api))
+ ksft_exit_fail_msg("UFFDIO_API\n");
+
+ if (!(uffdio_api.api & UFFDIO_REGISTER_MODE_WP) ||
+ !(uffdio_api.features & UFFD_FEATURE_WP_UNPOPULATED) ||
+ !(uffdio_api.features & UFFD_FEATURE_WP_ASYNC) ||
+ !(uffdio_api.features & UFFD_FEATURE_WP_HUGETLBFS_SHMEM))
+ ksft_exit_fail_msg("UFFDIO_API error %llu\n", uffdio_api.api);
+
+ return 0;
+}
+
+int wp_init(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_register uffdio_register;
+ struct uffdio_writeprotect wp;
+
+ uffdio_register.range.start = (unsigned long)lpBaseAddress;
+ uffdio_register.range.len = dwRegionSize;
+ uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+ if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
+ ksft_exit_fail_msg("ioctl(UFFDIO_REGISTER) %d %s\n", errno, strerror(errno));
+
+ if (!(uffdio_register.ioctls & UFFDIO_WRITEPROTECT))
+ ksft_exit_fail_msg("ioctl set is incorrect\n");
+
+ wp.range.start = (unsigned long)lpBaseAddress;
+ wp.range.len = dwRegionSize;
+ wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+ if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp))
+ ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+
+ return 0;
+}
+
+int wp_free(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_register uffdio_register;
+
+ uffdio_register.range.start = (unsigned long)lpBaseAddress;
+ uffdio_register.range.len = dwRegionSize;
+ uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+ if (ioctl(uffd, UFFDIO_UNREGISTER, &uffdio_register.range))
+ ksft_exit_fail_msg("ioctl unregister failure\n");
+ return 0;
+}
+
+int wp_addr_range(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_writeprotect wp;
+
+ wp.range.start = (unsigned long)lpBaseAddress;
+ wp.range.len = dwRegionSize;
+ wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+ if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp))
+ ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+
+ return 0;
+}
+
+void *gethugetlb_mem(int size, int *shmid)
+{
+ char *mem;
+
+ if (shmid) {
+ *shmid = shmget(2, size, SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W);
+ if (*shmid < 0)
+ return NULL;
+
+ mem = shmat(*shmid, 0, 0);
+ if (mem == (char *)-1) {
+ shmctl(*shmid, IPC_RMID, NULL);
+ ksft_exit_fail_msg("Shared memory attach failure\n");
+ }
+ } else {
+ mem = mmap(NULL, size, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_HUGETLB | MAP_PRIVATE, -1, 0);
+ if (mem == MAP_FAILED)
+ return NULL;
+ }
+
+ return mem;
+}
+
+int userfaultfd_tests(void)
+{
+ int mem_size, vec_size, written, num_pages = 16;
+ char *mem, *vec;
+
+ mem_size = num_pages * page_size;
+ mem = mmap(NULL, mem_size, PROT_NONE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(mem, mem_size);
+
+ /* Change protection of pages differently */
+ mprotect(mem, mem_size/8, PROT_READ|PROT_WRITE);
+ mprotect(mem + 1 * mem_size/8, mem_size/8, PROT_READ);
+ mprotect(mem + 2 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE);
+ mprotect(mem + 3 * mem_size/8, mem_size/8, PROT_READ);
+ mprotect(mem + 4 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE);
+ mprotect(mem + 5 * mem_size/8, mem_size/8, PROT_NONE);
+ mprotect(mem + 6 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE);
+ mprotect(mem + 7 * mem_size/8, mem_size/8, PROT_READ);
+
+ wp_addr_range(mem + (mem_size/16), mem_size - 2 * (mem_size/8));
+ wp_addr_range(mem, mem_size);
+
+ vec_size = mem_size/page_size;
+ vec = malloc(sizeof(struct page_region) * vec_size);
+
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ vec_size - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", __func__);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+ free(vec);
+ return 0;
+}
+
+int sanity_tests_sd(void)
+{
+ char *mem, *m[2];
+ int mem_size, vec_size, ret, ret2, ret3, i, num_pages = 10;
+ struct page_region *vec, *vec2;
+
+ vec_size = 100;
+ mem_size = num_pages * page_size;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ if (!vec)
+ ksft_exit_fail_msg("error nomem\n");
+
+ vec2 = malloc(sizeof(struct page_region) * vec_size);
+ if (!vec2)
+ ksft_exit_fail_msg("error nomem\n");
+
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ /* 1. wrong operation */
+ ksft_test_result(pagemap_ioctl(mem, 0, vec, vec_size, PM_SCAN_OP_GET,
+ 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0,
+ "%s memory size must be valid\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, NULL, vec_size, PM_SCAN_OP_GET,
+ 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0,
+ "%s output buffer must be specified\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, 0, PM_SCAN_OP_GET,
+ 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0,
+ "%s output buffer size must be valid\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, -1,
+ 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0,
+ "%s wrong flag specified\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_WP,
+ 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0,
+ "%s PM_SCAN_OP_WP cannot be used without PM_SCAN_OP_GET\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size,
+ PM_SCAN_OP_GET | PM_SCAN_OP_WP | 0xFF,
+ 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0,
+ "%s flag has extra bits specified\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET,
+ 0, 0, 0, 0, PAGE_IS_WRITTEN) < 0,
+ "%s no selection mask is specified\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET,
+ 0, PAGE_IS_WRITTEN, PAGE_IS_WRITTEN, 0, 0) < 0,
+ "%s no return mask is specified\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET,
+ 0, PAGE_IS_WRITTEN, 0, 0, 0x1000) < 0,
+ "%s wrong return mask specified\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ 0, 0xFFF, PAGE_IS_WRITTEN, 0, PAGE_IS_WRITTEN) < 0,
+ "%s mixture of correct and wrong flag\n", __func__);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ 0, 0, 0, PAGEMAP_BITS_ALL, PAGE_IS_WRITTEN) < 0,
+ "%s PAGEMAP_BITS_ALL cannot be specified with PM_SCAN_OP_WP\n", __func__);
+
+ /* 2. Clear area with larger vec size */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ ksft_test_result(ret >= 0, "%s Clear area with larger vec size\n", __func__);
+
+ /* 3. Repeated pattern of written and non-written pages */
+ for (i = 0; i < mem_size; i += 2 * page_size)
+ mem[i]++;
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0,
+ 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == mem_size/(page_size * 2),
+ "%s Repeated pattern of written and non-written pages\n", __func__);
+
+ /* 4. Repeated pattern of written and non-written pages in parts */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ num_pages/2 - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret2 = pagemap_ioctl(mem, mem_size, vec, 2, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0,
+ PAGE_IS_WRITTEN);
+ if (ret2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+ ret3 = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret3 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret3, errno, strerror(errno));
+
+ ksft_test_result((ret + ret3) == num_pages/2 && ret2 == 2,
+ "%s Repeated pattern of written and non-written pages in parts\n",
+ __func__);
+
+ /* 5. only get 2 dirty pages and clear them as well */
+ vec_size = mem_size/page_size;
+ memset(mem, -1, mem_size);
+
+ /* get and clear second and third pages */
+ ret = pagemap_ioctl(mem + page_size, 2 * page_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret2 = pagemap_ioctl(mem, mem_size, vec2, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == 2 &&
+ vec[0].start == (uintptr_t)(mem + page_size) &&
+ ret2 == 2 && vec2[0].len == 1 && vec2[0].start == (uintptr_t)mem &&
+ vec2[1].len == vec_size - 3 &&
+ vec2[1].start == (uintptr_t)(mem + 3 * page_size),
+ "%s only get 2 written pages and clear them as well\n", __func__);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 6. Two regions */
+ m[0] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (m[0] == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ m[1] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (m[1] == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(m[0], mem_size);
+ wp_init(m[1], mem_size);
+ wp_addr_range(m[0], mem_size);
+ wp_addr_range(m[1], mem_size);
+
+ memset(m[0], 'a', mem_size);
+ memset(m[1], 'b', mem_size);
+
+ wp_addr_range(m[0], mem_size);
+
+ ret = pagemap_ioctl(m[1], mem_size, vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0,
+ PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == mem_size/page_size,
+ "%s Two regions\n", __func__);
+
+ wp_free(m[0], mem_size);
+ wp_free(m[1], mem_size);
+ munmap(m[0], mem_size);
+ munmap(m[1], mem_size);
+
+ free(vec);
+ free(vec2);
+ return 0;
+}
+
+int base_tests(char *prefix, char *mem, int mem_size, int skip)
+{
+ int vec_size, written;
+ struct page_region *vec, *vec2;
+
+ if (skip) {
+ ksft_test_result_skip("%s all new pages must not be written (dirty)\n", prefix);
+ ksft_test_result_skip("%s all pages must be written (dirty)\n", prefix);
+ ksft_test_result_skip("%s all pages dirty other than first and the last one\n",
+ prefix);
+ ksft_test_result_skip("%s PM_SCAN_OP_WP\n", prefix);
+ ksft_test_result_skip("%s only middle page dirty\n", prefix);
+ ksft_test_result_skip("%s only two middle pages dirty\n", prefix);
+ return 0;
+ }
+
+ vec_size = mem_size/page_size;
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ vec2 = malloc(sizeof(struct page_region) * vec_size);
+
+ /* 1. all new pages must be not be written (dirty) */
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, vec_size - 2,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", prefix);
+
+ /* 2. all pages must be written */
+ memset(mem, -1, mem_size);
+
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0,
+ PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written == 1 && vec[0].len == mem_size/page_size,
+ "%s all pages must be written (dirty)\n", prefix);
+
+ /* 3. all pages dirty other than first and the last one */
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ memset(mem + page_size, 0, mem_size - (2 * page_size));
+
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written == 1 && vec[0].len >= vec_size - 2 && vec[0].len <= vec_size,
+ "%s all pages dirty other than first and the last one\n", prefix);
+
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written == 0,
+ "%s PM_SCAN_OP_WP\n", prefix);
+
+ /* 4. only middle page dirty */
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ mem[vec_size/2 * page_size]++;
+
+ written = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+ 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written == 1 && vec[0].len >= 1,
+ "%s only middle page dirty\n", prefix);
+
+ /* 5. only two middle pages dirty and walk over only middle pages */
+ written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ mem[vec_size/2 * page_size]++;
+ mem[(vec_size/2 + 1) * page_size]++;
+
+ written = pagemap_ioctl(&mem[vec_size/2 * page_size], 2 * page_size, vec, 1, PM_SCAN_OP_GET,
+ 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written == 1 && vec[0].start == (uintptr_t)(&mem[vec_size/2 * page_size])
+ && vec[0].len == 2,
+ "%s only two middle pages dirty\n", prefix);
+
+ free(vec);
+ free(vec2);
+ return 0;
+}
+
+void *gethugepage(int map_size)
+{
+ int ret;
+ char *map;
+
+ map = memalign(hpage_size, map_size);
+ if (!map)
+ ksft_exit_fail_msg("memalign failed %d %s\n", errno, strerror(errno));
+
+ ret = madvise(map, map_size, MADV_HUGEPAGE);
+ if (ret)
+ return NULL;
+
+ memset(map, 0, map_size);
+
+ return map;
+}
+
+int hpage_unit_tests(void)
+{
+ char *map;
+ int ret, ret2;
+ size_t num_pages = 10;
+ int map_size = hpage_size * num_pages;
+ int vec_size = map_size/page_size;
+ struct page_region *vec, *vec2;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ vec2 = malloc(sizeof(struct page_region) * vec_size);
+ if (!vec || !vec2)
+ ksft_exit_fail_msg("malloc failed\n");
+
+ map = gethugepage(map_size);
+ if (map) {
+ wp_init(map, map_size);
+ wp_addr_range(map, map_size);
+
+ /* 1. all new huge page must not be written (dirty) */
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 0, "%s all new huge page must not be written (dirty)\n",
+ __func__);
+
+ /* 2. all the huge page must not be written */
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 0, "%s all the huge page must not be written\n", __func__);
+
+ /* 3. all the huge page must be written and clear dirty as well */
+ memset(map, -1, map_size);
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].start == (uintptr_t)map &&
+ vec[0].len == vec_size && vec[0].bitmap == PAGE_IS_WRITTEN,
+ "%s all the huge page must be written and clear\n", __func__);
+
+ /* 4. only middle page written */
+ wp_free(map, map_size);
+ free(map);
+ map = gethugepage(map_size);
+ wp_init(map, map_size);
+ wp_addr_range(map, map_size);
+ map[vec_size/2 * page_size]++;
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len > 0,
+ "%s only middle page written\n", __func__);
+
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("%s all new huge page must be written\n", __func__);
+ ksft_test_result_skip("%s all the huge page must not be written\n", __func__);
+ ksft_test_result_skip("%s all the huge page must be written and clear\n", __func__);
+ ksft_test_result_skip("%s only middle page written\n", __func__);
+ }
+
+ /* 5. clear first half of huge page */
+ map = gethugepage(map_size);
+ if (map) {
+ wp_init(map, map_size);
+ wp_addr_range(map, map_size);
+
+ memset(map, 0, map_size);
+
+ wp_addr_range(map, map_size/2);
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+ vec[0].start == (uintptr_t)(map + map_size/2),
+ "%s clear first half of huge page\n", __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("%s clear first half of huge page\n", __func__);
+ }
+
+ /* 6. clear first half of huge page with limited buffer */
+ map = gethugepage(map_size);
+ if (map) {
+ wp_init(map, map_size);
+ wp_addr_range(map, map_size);
+
+ memset(map, 0, map_size);
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ vec_size/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+ vec[0].start == (uintptr_t)(map + map_size/2),
+ "%s clear first half of huge page with limited buffer\n",
+ __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("%s clear first half of huge page with limited buffer\n",
+ __func__);
+ }
+
+ /* 7. clear second half of huge page */
+ map = gethugepage(map_size);
+ if (map) {
+ wp_init(map, map_size);
+ wp_addr_range(map, map_size);
+
+ memset(map, -1, map_size);
+
+ ret = pagemap_ioctl(map + map_size/2, map_size/2, vec, vec_size,
+ PM_SCAN_OP_GET | PM_SCAN_OP_WP, vec_size/2, PAGE_IS_WRITTEN, 0,
+ 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2,
+ "%s clear second half huge page\n", __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("%s clear second half huge page\n", __func__);
+ }
+
+ /* 8. get half huge page */
+ map = gethugepage(map_size);
+ if (map) {
+ wp_init(map, map_size);
+ wp_addr_range(map, map_size);
+
+ memset(map, -1, map_size);
+ usleep(100);
+
+ ret = pagemap_ioctl(map, map_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+ hpage_size/(2*page_size), PAGE_IS_WRITTEN, 0, 0,
+ PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == hpage_size/(2*page_size),
+ "%s get half huge page\n", __func__);
+
+ ret2 = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+ if (ret2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+ ksft_test_result(ret2 == 1 && vec[0].len == (map_size - hpage_size/2)/page_size,
+ "%s get half huge page\n", __func__);
+
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("%s get half huge page\n", __func__);
+ ksft_test_result_skip("%s get half huge page\n", __func__);
+ }
+
+ free(vec);
+ free(vec2);
+ return 0;
+}
+
+int unmapped_region_tests(void)
+{
+ void *start = (void *)0x10000000;
+ int written, len = 0x00040000;
+ int vec_size = len / page_size;
+ struct page_region *vec = malloc(sizeof(struct page_region) * vec_size);
+
+ /* 1. Get written pages */
+ written = pagemap_ioctl(start, len, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGEMAP_NON_WRITTEN_BITS, 0, 0, PAGEMAP_NON_WRITTEN_BITS);
+ if (written < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+ ksft_test_result(written >= 0, "%s Get status of pages\n", __func__);
+
+ free(vec);
+ return 0;
+}
+
+static void test_simple(void)
+{
+ int i;
+ char *map;
+ struct page_region vec;
+
+ map = aligned_alloc(page_size, page_size);
+ if (!map)
+ ksft_exit_fail_msg("aligned_alloc failed\n");
+
+ wp_init(map, page_size);
+ wp_addr_range(map, page_size);
+
+ for (i = 0 ; i < TEST_ITERATIONS; i++) {
+ if (pagemap_ioctl(map, page_size, &vec, 1, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 1) {
+ ksft_print_msg("written bit was 1, but should be 0 (i=%d)\n", i);
+ break;
+ }
+
+ wp_addr_range(map, page_size);
+ /* Write something to the page to get the written bit enabled on the page */
+ map[0]++;
+
+ if (pagemap_ioctl(map, page_size, &vec, 1, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0) {
+ ksft_print_msg("written bit was 0, but should be 1 (i=%d)\n", i);
+ break;
+ }
+
+ wp_addr_range(map, page_size);
+ }
+ wp_free(map, page_size);
+ free(map);
+
+ ksft_test_result(i == TEST_ITERATIONS, "Test %s\n", __func__);
+}
+
+int sanity_tests(void)
+{
+ int mem_size, vec_size, ret, fd, i, buf_size;
+ struct page_region *vec;
+ char *mem, *fmem;
+ struct stat sbuf;
+
+ /* 1. wrong operation */
+ mem_size = 10 * page_size;
+ vec_size = mem_size / page_size;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED || vec == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size,
+ PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, PAGEMAP_BITS_ALL, 0, 0,
+ PAGEMAP_BITS_ALL) < 0,
+ "%s clear op can only be specified with PAGE_IS_WRITTEN\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) >= 0,
+ "%s required_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ 0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL) >= 0,
+ "%s anyof_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ 0, 0, PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL) >= 0,
+ "%s excluded_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL, 0,
+ PAGEMAP_BITS_ALL) >= 0,
+ "%s required_mask and anyof_mask specified\n", __func__);
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 2. Get sd and present pages with anyof_mask */
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ memset(mem, 0, mem_size);
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ 0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT),
+ "%s Get sd and present pages with anyof_mask\n", __func__);
+
+ /* 3. Get sd and present pages with required_mask */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT),
+ "%s Get all the pages with required_mask\n", __func__);
+
+ /* 4. Get sd and present pages with required_mask and anyof_mask */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, PAGE_IS_PRESENT, 0, PAGEMAP_BITS_ALL);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT),
+ "%s Get sd and present pages with required_mask and anyof_mask\n",
+ __func__);
+
+ /* 5. Don't get sd pages */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ 0, 0, PAGE_IS_WRITTEN, PAGEMAP_BITS_ALL);
+ ksft_test_result(ret == 0, "%s Don't get sd pages\n", __func__);
+
+ /* 6. Don't get present pages */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ 0, 0, PAGE_IS_PRESENT, PAGEMAP_BITS_ALL);
+ ksft_test_result(ret == 0, "%s Don't get present pages\n", __func__);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 8. Find written present pages with return mask */
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ memset(mem, 0, mem_size);
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ 0, PAGEMAP_BITS_ALL, 0, PAGE_IS_WRITTEN);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == PAGE_IS_WRITTEN,
+ "%s Find written present pages with return mask\n", __func__);
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 9. Memory mapped file */
+ fd = open(__FILE__, O_RDONLY);
+ if (fd < 0)
+ ksft_exit_fail_msg("%s Memory mapped file\n");
+
+ ret = stat(__FILE__, &sbuf);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ fmem = mmap(NULL, sbuf.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
+ if (fmem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+ ret = pagemap_ioctl(fmem, sbuf.st_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ 0, PAGEMAP_NON_WRITTEN_BITS, 0, PAGEMAP_NON_WRITTEN_BITS);
+
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem &&
+ vec[0].len == ceilf((float)sbuf.st_size/page_size) &&
+ vec[0].bitmap == PAGE_IS_FILE,
+ "%s Memory mapped file\n", __func__);
+
+ munmap(fmem, sbuf.st_size);
+ close(fd);
+
+ /* 10. Create and read/write to a memory mapped file*/
+ buf_size = page_size * 10;
+
+ fd = open(__FILE__".tmp2", O_RDWR | O_CREAT, 0777);
+ if (fd < 0)
+ ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n",
+ strerror(errno));
+
+ for (i = 0; i < buf_size; i++)
+ if (write(fd, "c", 1) < 0)
+ ksft_exit_fail_msg("Create and read/write to a memory mapped file\n");
+
+ fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+ if (fmem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+ wp_init(fmem, buf_size);
+ wp_addr_range(fmem, buf_size);
+
+ for (i = 0; i < buf_size; i++)
+ fmem[i] = i;
+
+ ret = pagemap_ioctl(fmem, buf_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN | PAGE_IS_FILE, PAGE_IS_PRESENT | PAGE_IS_SWAPPED, 0,
+ PAGEMAP_BITS_ALL);
+
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem &&
+ vec[0].len == (buf_size/page_size) &&
+ (vec[0].bitmap | PAGE_IS_WRITTEN) && (vec[0].bitmap | PAGE_IS_FILE),
+ "%s Read/write to private memory mapped file\n", __func__);
+
+ wp_free(fmem, buf_size);
+ munmap(fmem, buf_size);
+ close(fd);
+
+ free(vec);
+ return 0;
+}
+
+int mprotect_tests(void)
+{
+ int ret;
+ char *mem, *mem2;
+ struct page_region vec;
+ int pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+
+ if (pagemap_fd < 0) {
+ fprintf(stderr, "open() failed\n");
+ exit(1);
+ }
+
+ /* 1. Map two pages */
+ mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, 2 * page_size);
+ wp_addr_range(mem, 2 * page_size);
+
+ /* Populate both pages. */
+ memset(mem, 1, 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+ 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2, "%s Both pages written\n", __func__);
+
+ /* 2. Start tracking */
+ wp_addr_range(mem, 2 * page_size);
+
+ ksft_test_result(pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0,
+ PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0,
+ "%s Both pages are not written (dirty)\n", __func__);
+
+ /* 3. Remap the second page */
+ mem2 = mmap(mem + page_size, page_size, PROT_READ|PROT_WRITE,
+ MAP_PRIVATE|MAP_ANON|MAP_FIXED, -1, 0);
+ if (mem2 == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem2, page_size);
+ wp_addr_range(mem2, page_size);
+
+ /* Protect + unprotect. */
+ mprotect(mem, page_size, PROT_NONE);
+ mprotect(mem, 2 * page_size, PROT_READ);
+ mprotect(mem, 2 * page_size, PROT_READ|PROT_WRITE);
+
+ /* Modify both pages. */
+ memset(mem, 2, 2 * page_size);
+
+ /* Protect + unprotect. */
+ mprotect(mem, page_size, PROT_NONE);
+ mprotect(mem, page_size, PROT_READ);
+ mprotect(mem, page_size, PROT_READ|PROT_WRITE);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+ 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2,
+ "%s Both pages written after remap and mprotect\n", __func__);
+
+ /* 4. Clear and make the pages written */
+ wp_addr_range(mem, 2 * page_size);
+
+ memset(mem, 'A', 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+ 0, 0, PAGE_IS_WRITTEN);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2,
+ "%s Clear and make the pages written\n", __func__);
+
+ wp_free(mem, 2 * page_size);
+ munmap(mem, 2 * page_size);
+ return 0;
+}
+
+/* transact test */
+static const unsigned int nthreads = 6, pages_per_thread = 32, access_per_thread = 8;
+static pthread_barrier_t start_barrier, end_barrier;
+static unsigned int extra_thread_faults;
+static unsigned int iter_count = 1000;
+static volatile int finish;
+
+static ssize_t get_dirty_pages_reset(char *mem, unsigned int count,
+ int reset, int page_size)
+{
+ struct pm_scan_arg arg = {0};
+ struct page_region rgns[256];
+ int i, j, cnt, ret;
+
+ arg.size = sizeof(struct pm_scan_arg);
+ arg.start = (uintptr_t)mem;
+ arg.max_pages = count;
+ arg.len = count * page_size;
+ arg.vec = (uintptr_t)rgns;
+ arg.vec_len = sizeof(rgns) / sizeof(*rgns);
+ arg.flags = PM_SCAN_OP_GET;
+ if (reset)
+ arg.flags |= PM_SCAN_OP_WP;
+ arg.required_mask = PAGE_IS_WRITTEN;
+ arg.return_mask = PAGE_IS_WRITTEN;
+
+ ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+ if (ret < 0)
+ ksft_exit_fail_msg("ioctl failed\n");
+
+ cnt = 0;
+ for (i = 0; i < ret; ++i) {
+ if (rgns[i].bitmap != PAGE_IS_WRITTEN)
+ ksft_exit_fail_msg("wrong bitmap\n");
+
+ for (j = 0; j < rgns[i].len; ++j)
+ cnt++;
+ }
+
+ return cnt;
+}
+
+void *thread_proc(void *mem)
+{
+ int *m = mem;
+ long curr_faults, faults;
+ struct rusage r;
+ unsigned int i;
+ int ret;
+
+ if (getrusage(RUSAGE_THREAD, &r))
+ ksft_exit_fail_msg("getrusage\n");
+
+ curr_faults = r.ru_minflt;
+
+ while (!finish) {
+ ret = pthread_barrier_wait(&start_barrier);
+ if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+ ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+ for (i = 0; i < access_per_thread; ++i)
+ __atomic_add_fetch(m + i * (0x1000 / sizeof(*m)), 1, __ATOMIC_SEQ_CST);
+
+ ret = pthread_barrier_wait(&end_barrier);
+ if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+ ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+ if (getrusage(RUSAGE_THREAD, &r))
+ ksft_exit_fail_msg("getrusage\n");
+
+ faults = r.ru_minflt - curr_faults;
+ if (faults < access_per_thread)
+ ksft_exit_fail_msg("faults < access_per_thread");
+
+ __atomic_add_fetch(&extra_thread_faults, faults - access_per_thread,
+ __ATOMIC_SEQ_CST);
+ curr_faults = r.ru_minflt;
+ }
+
+ return NULL;
+}
+
+static void transact_test(int page_size)
+{
+ unsigned int i, count, extra_pages;
+ pthread_t th;
+ char *mem;
+ int ret, c;
+
+ if (pthread_barrier_init(&start_barrier, NULL, nthreads + 1))
+ ksft_exit_fail_msg("pthread_barrier_init\n");
+
+ if (pthread_barrier_init(&end_barrier, NULL, nthreads + 1))
+ ksft_exit_fail_msg("pthread_barrier_init\n");
+
+ mem = mmap(NULL, 0x1000 * nthreads * pages_per_thread, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("Error mmap %s.\n", strerror(errno));
+
+ wp_init(mem, 0x1000 * nthreads * pages_per_thread);
+ wp_addr_range(mem, 0x1000 * nthreads * pages_per_thread);
+
+ memset(mem, 0, 0x1000 * nthreads * pages_per_thread);
+
+ count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size);
+ ksft_test_result(count > 0, "%s count %d\n", __func__, count);
+ count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size);
+ ksft_test_result(count == 0, "%s count %d\n", __func__, count);
+
+ finish = 0;
+ for (i = 0; i < nthreads; ++i)
+ pthread_create(&th, NULL, thread_proc, mem + 0x1000 * i * pages_per_thread);
+
+ extra_pages = 0;
+ for (i = 0; i < iter_count; ++i) {
+ count = 0;
+
+ ret = pthread_barrier_wait(&start_barrier);
+ if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+ ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+ count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1,
+ page_size);
+
+ ret = pthread_barrier_wait(&end_barrier);
+ if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+ ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+ if (count > nthreads * access_per_thread)
+ ksft_exit_fail_msg("Too big count %d expected %d, iter %d\n",
+ count, nthreads * access_per_thread, i);
+
+ c = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size);
+ count += c;
+
+ if (c > nthreads * access_per_thread) {
+ ksft_test_result_fail(" %s count > nthreads\n", __func__);
+ return;
+ }
+
+ if (count != nthreads * access_per_thread) {
+ /*
+ * The purpose of the test is to make sure that no page updates are lost
+ * when the page updates and read-resetting soft dirty flags are performed
+ * in parallel. However, it is possible that the application will get the
+ * soft dirty flags twice on the two consecutive read-resets. This seems
+ * unavoidable as soft dirty flag is handled in software through page faults
+ * in kernel. While the updating the flags is supposed to be synchronized
+ * between page fault handling and read-reset, it is possible that
+ * read-reset happens after page fault PTE update but before the application
+ * re-executes write instruction. So read-reset gets the flag, clears write
+ * access and application gets page fault again for the same write.
+ */
+ if (count < nthreads * access_per_thread) {
+ ksft_test_result_fail("Lost update, iter %d, %d vs %d.\n", i, count,
+ nthreads * access_per_thread);
+ return;
+ }
+
+ extra_pages += count - nthreads * access_per_thread;
+ }
+ }
+
+ pthread_barrier_wait(&start_barrier);
+ finish = 1;
+ pthread_barrier_wait(&end_barrier);
+
+ ksft_test_result_pass("%s Extra pages %u (%.1lf%%), extra thread faults %d.\n", __func__,
+ extra_pages,
+ 100.0 * extra_pages / (iter_count * nthreads * access_per_thread),
+ extra_thread_faults);
+}
+
+int main(void)
+{
+ int mem_size, shmid, buf_size, fd, i, ret;
+ char *mem, *map, *fmem;
+ struct stat sbuf;
+
+ ksft_print_header();
+ ksft_set_plan(90);
+
+ page_size = getpagesize();
+ hpage_size = read_pmd_pagesize();
+
+ pagemap_fd = open(PAGEMAP, O_RDWR);
+ if (pagemap_fd < 0)
+ return -EINVAL;
+
+ if (init_uffd())
+ ksft_exit_fail_msg("uffd init failed\n");
+
+ /*
+ * Written (dirty) PTE bit tests
+ */
+
+ /* 1. Sanity testing */
+ sanity_tests_sd();
+
+ /* 2. Normal page testing */
+ mem_size = 10 * page_size;
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ base_tests("Page testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 3. Large page testing */
+ mem_size = 512 * 10 * page_size;
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ base_tests("Large Page testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 4. Huge page testing */
+ map = gethugepage(hpage_size);
+ if (map) {
+ wp_init(map, hpage_size);
+ wp_addr_range(map, hpage_size);
+ base_tests("Huge page testing:", map, hpage_size, 0);
+ wp_free(map, hpage_size);
+ free(map);
+ } else {
+ base_tests("Huge page testing:", NULL, 0, 1);
+ }
+
+ /* 5. Hugetlb page testing */
+ mem_size = 2*1024*1024;
+ mem = gethugetlb_mem(mem_size, &shmid);
+ if (mem) {
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ base_tests("Hugetlb shmem testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ shmctl(shmid, IPC_RMID, NULL);
+ } else {
+ base_tests("Hugetlb shmem testing:", NULL, 0, 1);
+ }
+
+ /* 6. Hugetlb page testing */
+ mem = gethugetlb_mem(mem_size, NULL);
+ if (mem) {
+ wp_init(mem, mem_size);
+ wp_addr_range(mem, mem_size);
+
+ base_tests("Hugetlb mem testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ } else {
+ base_tests("Hugetlb mem testing:", NULL, 0, 1);
+ }
+
+ /* 7. file memory testing */
+ buf_size = page_size * 10;
+
+ fd = open(__FILE__".tmp0", O_RDWR | O_CREAT, 0777);
+ if (fd < 0)
+ ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n",
+ strerror(errno));
+
+ for (i = 0; i < buf_size; i++)
+ if (write(fd, "c", 1) < 0)
+ ksft_exit_fail_msg("Create and read/write to a memory mapped file\n");
+
+ ret = stat(__FILE__".tmp0", &sbuf);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ fmem = mmap(NULL, sbuf.st_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+ if (fmem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+ wp_init(fmem, sbuf.st_size);
+ wp_addr_range(fmem, sbuf.st_size);
+
+ base_tests("File memory testing:", fmem, sbuf.st_size, 0);
+
+ wp_free(fmem, sbuf.st_size);
+ munmap(fmem, sbuf.st_size);
+ close(fd);
+
+ /* 8. file memory testing */
+ buf_size = page_size * 10;
+
+ fd = memfd_create(__FILE__".tmp00", MFD_NOEXEC_SEAL);
+ if (fd < 0)
+ ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n",
+ strerror(errno));
+
+ if (ftruncate(fd, buf_size))
+ ksft_exit_fail_msg("Error ftruncate\n");
+
+ for (i = 0; i < buf_size; i++)
+ if (write(fd, "c", 1) < 0)
+ ksft_exit_fail_msg("Create and read/write to a memory mapped file\n");
+
+ fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+ if (fmem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+ wp_init(fmem, buf_size);
+ wp_addr_range(fmem, buf_size);
+
+ base_tests("File anonymous memory testing:", fmem, buf_size, 0);
+
+ wp_free(fmem, buf_size);
+ munmap(fmem, buf_size);
+ close(fd);
+
+ /* 9. Huge page tests */
+ hpage_unit_tests();
+
+ /* 10. Iterative test */
+ test_simple();
+
+ /* 11. Mprotect test */
+ mprotect_tests();
+
+ /* 12. Transact test */
+ transact_test(page_size);
+
+ /*
+ * Other PTE bit tests
+ */
+
+ /* 1. Sanity testing */
+ sanity_tests();
+
+ /* 2. Unmapped address test */
+ unmapped_region_tests();
+
+ /* 3. Userfaultfd tests */
+ userfaultfd_tests();
+
+ close(pagemap_fd);
+ return ksft_exit_pass();
+}
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
old mode 100644
new mode 100755
index eb25f238f5b8..fa77c8af2b0c
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -53,6 +53,8 @@ separated by spaces:
memory protection key tests
- soft_dirty
test soft dirty page bit semantics
+- pagemap
+ test pagemap_scan IOCTL
- cow
test copy-on-write semantics
example: ./run_vmtests.sh -t "hmm mmap ksm"
@@ -282,6 +284,8 @@ fi
CATEGORY="soft_dirty" run_test ./soft-dirty
+CATEGORY="pagemap" run_test ./pagemap_ioctl
+
# COW tests
CATEGORY="cow" run_test ./cow
--
2.39.2
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-04-20 6:01 ` [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
@ 2023-04-26 7:06 ` Muhammad Usama Anjum
2023-04-26 14:13 ` Peter Xu
0 siblings, 1 reply; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-26 7:06 UTC (permalink / raw)
To: Peter Xu, Paul Gofman
Cc: Muhammad Usama Anjum, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Cyrill Gorcunov, Michał Mirosław,
Andrew Morton, Suren Baghdasaryan, Andrei Vagin, Alex Sierra,
Matthew Wilcox, Pasha Tatashin, Danylo Mocherniuk,
Axel Rasmussen, Gustavo A . R . Silva, David Hildenbrand,
Dan Williams, linux-kernel, Mike Rapoport, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel, Nadav Amit
On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
> +/* Supported flags */
> +#define PM_SCAN_OP_GET (1 << 0)
> +#define PM_SCAN_OP_WP (1 << 1)
We have only these flag options available in PAGEMAP_SCAN IOCTL.
PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
be specified as need. But PM_SCAN_OP_WP cannot be specified without
PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
functionality which can be achieved by UFFDIO_WRITEPROTECT.)
1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
vs
2) UFFDIO_WRITEPROTECT
After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
getting really good performance which is comparable just like we are
depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
performance and behavior wise.
I've got the results from someone else that UFFDIO_WRITEPROTECT block
pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
as I don't have tests comparing them one-to-one.
What are your thoughts about it? Have you thought about making
UFFDIO_WRITEPROTECT perform better?
I'm sorry to mention the word "performance" here. Actually we want better
performance to emulate Windows syscall. That is why we are adding this
functionality. So either we need to see what can be improved in
UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
pagemap_ioctl?
Thank you so much for the help.
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-04-26 7:06 ` Muhammad Usama Anjum
@ 2023-04-26 14:13 ` Peter Xu
2023-04-27 15:31 ` Muhammad Usama Anjum
2023-05-22 10:24 ` Muhammad Usama Anjum
0 siblings, 2 replies; 15+ messages in thread
From: Peter Xu @ 2023-04-26 14:13 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: Paul Gofman, Alexander Viro, Shuah Khan, Christian Brauner,
Yang Shi, Vlastimil Babka, Liam R . Howlett, Yun Zhou,
Cyrill Gorcunov, Michał Mirosław, Andrew Morton,
Suren Baghdasaryan, Andrei Vagin, Alex Sierra, Matthew Wilcox,
Pasha Tatashin, Danylo Mocherniuk, Axel Rasmussen,
Gustavo A . R . Silva, David Hildenbrand, Dan Williams,
linux-kernel, Mike Rapoport, linux-fsdevel, linux-mm,
linux-kselftest, Greg KH, kernel, Nadav Amit
Hi, Muhammad,
On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote:
> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
> > +/* Supported flags */
> > +#define PM_SCAN_OP_GET (1 << 0)
> > +#define PM_SCAN_OP_WP (1 << 1)
> We have only these flag options available in PAGEMAP_SCAN IOCTL.
> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
> be specified as need. But PM_SCAN_OP_WP cannot be specified without
> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
> functionality which can be achieved by UFFDIO_WRITEPROTECT.)
>
> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
> vs
> 2) UFFDIO_WRITEPROTECT
>
> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
> getting really good performance which is comparable just like we are
> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
> performance and behavior wise.
>
> I've got the results from someone else that UFFDIO_WRITEPROTECT block
> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
> as I don't have tests comparing them one-to-one.
>
> What are your thoughts about it? Have you thought about making
> UFFDIO_WRITEPROTECT perform better?
>
> I'm sorry to mention the word "performance" here. Actually we want better
> performance to emulate Windows syscall. That is why we are adding this
> functionality. So either we need to see what can be improved in
> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
> pagemap_ioctl?
I'm fine if you want to add it back if it works for you. Though before
that, could you remind me why there can be a difference on performance?
Thanks,
--
Peter Xu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-04-26 14:13 ` Peter Xu
@ 2023-04-27 15:31 ` Muhammad Usama Anjum
2023-05-22 10:24 ` Muhammad Usama Anjum
1 sibling, 0 replies; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-04-27 15:31 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, Paul Gofman, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Cyrill Gorcunov, Michał Mirosław,
Andrew Morton, Suren Baghdasaryan, Andrei Vagin, Alex Sierra,
Matthew Wilcox, Pasha Tatashin, Danylo Mocherniuk,
Axel Rasmussen, Gustavo A . R . Silva, David Hildenbrand,
Dan Williams, linux-kernel, Mike Rapoport, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel, Nadav Amit
Hi Peter,
Thank you for your reply.
On 4/26/23 7:13 PM, Peter Xu wrote:
> Hi, Muhammad,
>
> On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote:
>> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
>>> +/* Supported flags */
>>> +#define PM_SCAN_OP_GET (1 << 0)
>>> +#define PM_SCAN_OP_WP (1 << 1)
>> We have only these flag options available in PAGEMAP_SCAN IOCTL.
>> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
>> be specified as need. But PM_SCAN_OP_WP cannot be specified without
>> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
>> functionality which can be achieved by UFFDIO_WRITEPROTECT.)
>>
>> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
>> vs
>> 2) UFFDIO_WRITEPROTECT
>>
>> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
>> getting really good performance which is comparable just like we are
>> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
>> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
>> performance and behavior wise.
>>
>> I've got the results from someone else that UFFDIO_WRITEPROTECT block
>> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
>> as I don't have tests comparing them one-to-one.
>>
>> What are your thoughts about it? Have you thought about making
>> UFFDIO_WRITEPROTECT perform better?
>>
>> I'm sorry to mention the word "performance" here. Actually we want better
>> performance to emulate Windows syscall. That is why we are adding this
>> functionality. So either we need to see what can be improved in
>> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
>> pagemap_ioctl?
>
> I'm fine if you want to add it back if it works for you. Though before
> that, could you remind me why there can be a difference on performance?
The only difference can be that UFFDIO_WRITEPROTECT acquires read mm lock
once for entire duration. But for PAGEMAP_SCAN IOCTL, we acquire and
release for each PMD to keep intermediate buffer short.
This must be hard to convince you. So I'll write some test to see what is
the exact difference and show you the numbers.
>
> Thanks,
>
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-04-26 14:13 ` Peter Xu
2023-04-27 15:31 ` Muhammad Usama Anjum
@ 2023-05-22 10:24 ` Muhammad Usama Anjum
2023-05-22 11:26 ` Muhammad Usama Anjum
1 sibling, 1 reply; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-05-22 10:24 UTC (permalink / raw)
To: Peter Xu, linux-mm
Cc: Muhammad Usama Anjum, Paul Gofman, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Cyrill Gorcunov, Michał Mirosław,
Andrew Morton, Suren Baghdasaryan, Andrei Vagin, Alex Sierra,
Matthew Wilcox, Pasha Tatashin, Danylo Mocherniuk,
Axel Rasmussen, Gustavo A . R . Silva, David Hildenbrand,
Dan Williams, linux-kernel, Mike Rapoport, linux-fsdevel,
linux-kselftest, Greg KH, kernel, Nadav Amit
[-- Attachment #1: Type: text/plain, Size: 4734 bytes --]
On 4/26/23 7:13 PM, Peter Xu wrote:
> Hi, Muhammad,
>
> On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote:
>> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
>>> +/* Supported flags */
>>> +#define PM_SCAN_OP_GET (1 << 0)
>>> +#define PM_SCAN_OP_WP (1 << 1)
>> We have only these flag options available in PAGEMAP_SCAN IOCTL.
>> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
>> be specified as need. But PM_SCAN_OP_WP cannot be specified without
>> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
>> functionality which can be achieved by UFFDIO_WRITEPROTECT.)
>>
>> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
>> vs
>> 2) UFFDIO_WRITEPROTECT
>>
>> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
>> getting really good performance which is comparable just like we are
>> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
>> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
>> performance and behavior wise.
>>
>> I've got the results from someone else that UFFDIO_WRITEPROTECT block
>> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
>> as I don't have tests comparing them one-to-one.
>>
>> What are your thoughts about it? Have you thought about making
>> UFFDIO_WRITEPROTECT perform better?
>>
>> I'm sorry to mention the word "performance" here. Actually we want better
>> performance to emulate Windows syscall. That is why we are adding this
>> functionality. So either we need to see what can be improved in
>> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
>> pagemap_ioctl?
>
> I'm fine if you want to add it back if it works for you. Though before
> that, could you remind me why there can be a difference on performance?
I've looked at the code again and I think I've found something. Lets look
at exact performance numbers:
I've run 2 different tests. In first test UFFDIO_WRITEPROTECT is being used
for engaging WP. In second test PM_SCAN_OP_WP is being used. I've measured
the average write time to the same memory which is being WP-ed and total
time of execution of these APIs:
**avg write time:**
| No of pages | 2000 | 8192 | 100000 | 500000 |
|------------------------|------|------|--------|--------|
| UFFDIO_WRITEPROTECT | 2200 | 2300 | 4100 | 4200 |
| PM_SCAN_OP_WP | 2000 | 2300 | 2500 | 2800 |
**Execution time measured in rdtsc:**
| No of pages | 2000 | 8192 | 100000 | 500000 |
|------------------------|------|-------|--------|--------|
| UFFDIO_WRITEPROTECT | 3200 | 14000 | 59000 | 58000 |
| PM_SCAN_OP_WP | 1900 | 7000 | 38000 | 40000 |
Avg write time for UFFDIO_WRITEPROTECT is 1.3 times slow. The execution
time is 1.5 times slower in the case of UFFDIO_WRITEPROTECT. So
UFFDIO_WRITEPROTECT is making writes slower to the pages and execution time
is also slower.
This proves that PM_SCAN_OP_WP is better than UFFDIO_WRITEPROTECT. Although
PM_SCAN_OP_WP and UFFDIO_WRITEPROTECT have been implemented differently. We
should have seen no difference in performance. But we have quite a lot of
difference in performance here. PM_SCAN_OP_WP takes read mm lock, uses
walk_page_range() to walk over pages which finds VMAs from address ranges
to walk over them and pagemap_scan_pmd_entry() is handling most of the work
including tlb flushing. UFFDIO_WRITEPROTECT is also taking the mm lock and
iterating from all the different page directories until a pte is found and
then flags are updated there and tlb is flushed for every pte.
My next deduction would be that we are getting worse performance as we are
flushing tlb for one page at a time in case of UFFDIO_WRITEPROTECT. While
we flush tlb for 512 pages (moslty) at a time in case of PM_SCAN_OP_WP.
I've just verified this by adding some logs to the change_pte_range() and
pagemap_scan_pmd_entry(). Logs are attached. I've allocated memory of 1000
pages and write-protected it with UFFDIO_WRITEPROTECT and PM_SCAN_OP_WP.
The logs show that UFFDIO_WRITEPROTECT has flushed tlb 1000 times of size 1
page each time. While PM_SCAN_OP_WP has flushed only 3 times of bigger
sizes. I've learned over my last experience that tlb flush is very
expensive. Probably this is what we need to improve if we don't want to add
PM_SCAN_OP_WP?
The UFFDIO_WRITEPROTECT uses change_pte_range() which is very generic
function and I'm not sure if can try to not do tlb flushes if uffd_wp is
true. We can try to do flush somewhere else and hopefully we should do only
one flush if possible. It will not be so straight forward to move away from
generic fundtion. Thoughts?
> Thanks,
>
--
BR,
Muhammad Usama Anjum
[-- Attachment #2: log --]
[-- Type: text/plain, Size: 50192 bytes --]
[ 59.108902] change_pte_range 7FAB8ED41000 1000
[ 59.109228] change_pte_range 7FAB8ED42000 1000
[ 59.109538] change_pte_range 7FAB8ED43000 1000
[ 59.109854] change_pte_range 7FAB8ED44000 1000
[ 59.110158] change_pte_range 7FAB8ED45000 1000
[ 59.110460] change_pte_range 7FAB8ED46000 1000
[ 59.110783] change_pte_range 7FAB8ED47000 1000
[ 59.111113] change_pte_range 7FAB8ED48000 1000
[ 59.111463] change_pte_range 7FAB8ED49000 1000
[ 59.111777] change_pte_range 7FAB8ED4A000 1000
[ 59.112084] change_pte_range 7FAB8ED4B000 1000
[ 59.112408] change_pte_range 7FAB8ED4C000 1000
[ 59.112719] change_pte_range 7FAB8ED4D000 1000
[ 59.113025] change_pte_range 7FAB8ED4E000 1000
[ 59.113331] change_pte_range 7FAB8ED4F000 1000
[ 59.113634] change_pte_range 7FAB8ED50000 1000
[ 59.113952] change_pte_range 7FAB8ED51000 1000
[ 59.114261] change_pte_range 7FAB8ED52000 1000
[ 59.114563] change_pte_range 7FAB8ED53000 1000
[ 59.114878] change_pte_range 7FAB8ED54000 1000
[ 59.115225] change_pte_range 7FAB8ED55000 1000
[ 59.115533] change_pte_range 7FAB8ED56000 1000
[ 59.115855] change_pte_range 7FAB8ED57000 1000
[ 59.116167] change_pte_range 7FAB8ED58000 1000
[ 59.116484] change_pte_range 7FAB8ED59000 1000
[ 59.116807] change_pte_range 7FAB8ED5A000 1000
[ 59.117124] change_pte_range 7FAB8ED5B000 1000
[ 59.117436] change_pte_range 7FAB8ED5C000 1000
[ 59.117756] change_pte_range 7FAB8ED5D000 1000
[ 59.118072] change_pte_range 7FAB8ED5E000 1000
[ 59.118385] change_pte_range 7FAB8ED5F000 1000
[ 59.118702] change_pte_range 7FAB8ED60000 1000
[ 59.119015] change_pte_range 7FAB8ED61000 1000
[ 59.119404] change_pte_range 7FAB8ED62000 1000
[ 59.119737] change_pte_range 7FAB8ED63000 1000
[ 59.120049] change_pte_range 7FAB8ED64000 1000
[ 59.120369] change_pte_range 7FAB8ED65000 1000
[ 59.120682] change_pte_range 7FAB8ED66000 1000
[ 59.121002] change_pte_range 7FAB8ED67000 1000
[ 59.121317] change_pte_range 7FAB8ED68000 1000
[ 59.121628] change_pte_range 7FAB8ED69000 1000
[ 59.121948] change_pte_range 7FAB8ED6A000 1000
[ 59.122261] change_pte_range 7FAB8ED6B000 1000
[ 59.122574] change_pte_range 7FAB8ED6C000 1000
[ 59.122893] change_pte_range 7FAB8ED6D000 1000
[ 59.123219] change_pte_range 7FAB8ED6E000 1000
[ 59.123569] change_pte_range 7FAB8ED6F000 1000
[ 59.123893] change_pte_range 7FAB8ED70000 1000
[ 59.124208] change_pte_range 7FAB8ED71000 1000
[ 59.124524] change_pte_range 7FAB8ED72000 1000
[ 59.124844] change_pte_range 7FAB8ED73000 1000
[ 59.125159] change_pte_range 7FAB8ED74000 1000
[ 59.125479] change_pte_range 7FAB8ED75000 1000
[ 59.125797] change_pte_range 7FAB8ED76000 1000
[ 59.126114] change_pte_range 7FAB8ED77000 1000
[ 59.126431] change_pte_range 7FAB8ED78000 1000
[ 59.126752] change_pte_range 7FAB8ED79000 1000
[ 59.127081] change_pte_range 7FAB8ED7A000 1000
[ 59.127396] change_pte_range 7FAB8ED7B000 1000
[ 59.127735] change_pte_range 7FAB8ED7C000 1000
[ 59.128051] change_pte_range 7FAB8ED7D000 1000
[ 59.128370] change_pte_range 7FAB8ED7E000 1000
[ 59.128683] change_pte_range 7FAB8ED7F000 1000
[ 59.129008] change_pte_range 7FAB8ED80000 1000
[ 59.129317] change_pte_range 7FAB8ED81000 1000
[ 59.129630] change_pte_range 7FAB8ED82000 1000
[ 59.129950] change_pte_range 7FAB8ED83000 1000
[ 59.130260] change_pte_range 7FAB8ED84000 1000
[ 59.130566] change_pte_range 7FAB8ED85000 1000
[ 59.130894] change_pte_range 7FAB8ED86000 1000
[ 59.131216] change_pte_range 7FAB8ED87000 1000
[ 59.131523] change_pte_range 7FAB8ED88000 1000
[ 59.131837] change_pte_range 7FAB8ED89000 1000
[ 59.132149] change_pte_range 7FAB8ED8A000 1000
[ 59.132455] change_pte_range 7FAB8ED8B000 1000
[ 59.132776] change_pte_range 7FAB8ED8C000 1000
[ 59.133090] change_pte_range 7FAB8ED8D000 1000
[ 59.133395] change_pte_range 7FAB8ED8E000 1000
[ 59.133727] change_pte_range 7FAB8ED8F000 1000
[ 59.134040] change_pte_range 7FAB8ED90000 1000
[ 59.134352] change_pte_range 7FAB8ED91000 1000
[ 59.134663] change_pte_range 7FAB8ED92000 1000
[ 59.134996] change_pte_range 7FAB8ED93000 1000
[ 59.135322] change_pte_range 7FAB8ED94000 1000
[ 59.135628] change_pte_range 7FAB8ED95000 1000
[ 59.135950] change_pte_range 7FAB8ED96000 1000
[ 59.136263] change_pte_range 7FAB8ED97000 1000
[ 59.136572] change_pte_range 7FAB8ED98000 1000
[ 59.136890] change_pte_range 7FAB8ED99000 1000
[ 59.137199] change_pte_range 7FAB8ED9A000 1000
[ 59.137511] change_pte_range 7FAB8ED9B000 1000
[ 59.137826] change_pte_range 7FAB8ED9C000 1000
[ 59.138135] change_pte_range 7FAB8ED9D000 1000
[ 59.138443] change_pte_range 7FAB8ED9E000 1000
[ 59.138768] change_pte_range 7FAB8ED9F000 1000
[ 59.139084] change_pte_range 7FAB8EDA0000 1000
[ 59.139451] change_pte_range 7FAB8EDA1000 1000
[ 59.139811] change_pte_range 7FAB8EDA2000 1000
[ 59.140123] change_pte_range 7FAB8EDA3000 1000
[ 59.140437] change_pte_range 7FAB8EDA4000 1000
[ 59.140755] change_pte_range 7FAB8EDA5000 1000
[ 59.141065] change_pte_range 7FAB8EDA6000 1000
[ 59.141376] change_pte_range 7FAB8EDA7000 1000
[ 59.141685] change_pte_range 7FAB8EDA8000 1000
[ 59.142004] change_pte_range 7FAB8EDA9000 1000
[ 59.142315] change_pte_range 7FAB8EDAA000 1000
[ 59.142624] change_pte_range 7FAB8EDAB000 1000
[ 59.142954] change_pte_range 7FAB8EDAC000 1000
[ 59.143310] change_pte_range 7FAB8EDAD000 1000
[ 59.143668] change_pte_range 7FAB8EDAE000 1000
[ 59.144067] change_pte_range 7FAB8EDAF000 1000
[ 59.144453] change_pte_range 7FAB8EDB0000 1000
[ 59.144840] change_pte_range 7FAB8EDB1000 1000
[ 59.145225] change_pte_range 7FAB8EDB2000 1000
[ 59.145599] change_pte_range 7FAB8EDB3000 1000
[ 59.145988] change_pte_range 7FAB8EDB4000 1000
[ 59.146376] change_pte_range 7FAB8EDB5000 1000
[ 59.146779] change_pte_range 7FAB8EDB6000 1000
[ 59.147171] change_pte_range 7FAB8EDB7000 1000
[ 59.147560] change_pte_range 7FAB8EDB8000 1000
[ 59.147954] change_pte_range 7FAB8EDB9000 1000
[ 59.148329] change_pte_range 7FAB8EDBA000 1000
[ 59.148706] change_pte_range 7FAB8EDBB000 1000
[ 59.149082] change_pte_range 7FAB8EDBC000 1000
[ 59.149449] change_pte_range 7FAB8EDBD000 1000
[ 59.149833] change_pte_range 7FAB8EDBE000 1000
[ 59.150228] change_pte_range 7FAB8EDBF000 1000
[ 59.150587] change_pte_range 7FAB8EDC0000 1000
[ 59.150927] change_pte_range 7FAB8EDC1000 1000
[ 59.151257] change_pte_range 7FAB8EDC2000 1000
[ 59.151567] change_pte_range 7FAB8EDC3000 1000
[ 59.151887] change_pte_range 7FAB8EDC4000 1000
[ 59.152194] change_pte_range 7FAB8EDC5000 1000
[ 59.152503] change_pte_range 7FAB8EDC6000 1000
[ 59.152822] change_pte_range 7FAB8EDC7000 1000
[ 59.153136] change_pte_range 7FAB8EDC8000 1000
[ 59.153444] change_pte_range 7FAB8EDC9000 1000
[ 59.153764] change_pte_range 7FAB8EDCA000 1000
[ 59.154073] change_pte_range 7FAB8EDCB000 1000
[ 59.154383] change_pte_range 7FAB8EDCC000 1000
[ 59.154701] change_pte_range 7FAB8EDCD000 1000
[ 59.155042] change_pte_range 7FAB8EDCE000 1000
[ 59.155370] change_pte_range 7FAB8EDCF000 1000
[ 59.155688] change_pte_range 7FAB8EDD0000 1000
[ 59.156012] change_pte_range 7FAB8EDD1000 1000
[ 59.156328] change_pte_range 7FAB8EDD2000 1000
[ 59.156642] change_pte_range 7FAB8EDD3000 1000
[ 59.157029] change_pte_range 7FAB8EDD4000 1000
[ 59.157403] change_pte_range 7FAB8EDD5000 1000
[ 59.157796] change_pte_range 7FAB8EDD6000 1000
[ 59.158174] change_pte_range 7FAB8EDD7000 1000
[ 59.158567] change_pte_range 7FAB8EDD8000 1000
[ 59.158970] change_pte_range 7FAB8EDD9000 1000
[ 59.159360] change_pte_range 7FAB8EDDA000 1000
[ 59.159744] change_pte_range 7FAB8EDDB000 1000
[ 59.160118] change_pte_range 7FAB8EDDC000 1000
[ 59.160439] change_pte_range 7FAB8EDDD000 1000
[ 59.160780] change_pte_range 7FAB8EDDE000 1000
[ 59.161099] change_pte_range 7FAB8EDDF000 1000
[ 59.161430] change_pte_range 7FAB8EDE0000 1000
[ 59.161761] change_pte_range 7FAB8EDE1000 1000
[ 59.162082] change_pte_range 7FAB8EDE2000 1000
[ 59.162399] change_pte_range 7FAB8EDE3000 1000
[ 59.162744] change_pte_range 7FAB8EDE4000 1000
[ 59.163071] change_pte_range 7FAB8EDE5000 1000
[ 59.163388] change_pte_range 7FAB8EDE6000 1000
[ 59.163715] change_pte_range 7FAB8EDE7000 1000
[ 59.164036] change_pte_range 7FAB8EDE8000 1000
[ 59.164355] change_pte_range 7FAB8EDE9000 1000
[ 59.164672] change_pte_range 7FAB8EDEA000 1000
[ 59.164999] change_pte_range 7FAB8EDEB000 1000
[ 59.165319] change_pte_range 7FAB8EDEC000 1000
[ 59.165638] change_pte_range 7FAB8EDED000 1000
[ 59.165964] change_pte_range 7FAB8EDEE000 1000
[ 59.166335] change_pte_range 7FAB8EDEF000 1000
[ 59.166709] change_pte_range 7FAB8EDF0000 1000
[ 59.167089] change_pte_range 7FAB8EDF1000 1000
[ 59.167468] change_pte_range 7FAB8EDF2000 1000
[ 59.167837] change_pte_range 7FAB8EDF3000 1000
[ 59.168219] change_pte_range 7FAB8EDF4000 1000
[ 59.168617] change_pte_range 7FAB8EDF5000 1000
[ 59.169010] change_pte_range 7FAB8EDF6000 1000
[ 59.169401] change_pte_range 7FAB8EDF7000 1000
[ 59.169786] change_pte_range 7FAB8EDF8000 1000
[ 59.170165] change_pte_range 7FAB8EDF9000 1000
[ 59.170523] change_pte_range 7FAB8EDFA000 1000
[ 59.170911] change_pte_range 7FAB8EDFB000 1000
[ 59.171288] change_pte_range 7FAB8EDFC000 1000
[ 59.171669] change_pte_range 7FAB8EDFD000 1000
[ 59.172042] change_pte_range 7FAB8EDFE000 1000
[ 59.172410] change_pte_range 7FAB8EDFF000 1000
[ 59.172853] change_pte_range 7FAB8EE00000 1000
[ 59.173308] change_pte_range 7FAB8EE01000 1000
[ 59.173760] change_pte_range 7FAB8EE02000 1000
[ 59.174301] change_pte_range 7FAB8EE03000 1000
[ 59.174761] change_pte_range 7FAB8EE04000 1000
[ 59.175202] change_pte_range 7FAB8EE05000 1000
[ 59.175523] change_pte_range 7FAB8EE06000 1000
[ 59.175851] change_pte_range 7FAB8EE07000 1000
[ 59.176158] change_pte_range 7FAB8EE08000 1000
[ 59.176465] change_pte_range 7FAB8EE09000 1000
[ 59.176788] change_pte_range 7FAB8EE0A000 1000
[ 59.177099] change_pte_range 7FAB8EE0B000 1000
[ 59.177408] change_pte_range 7FAB8EE0C000 1000
[ 59.177726] change_pte_range 7FAB8EE0D000 1000
[ 59.178036] change_pte_range 7FAB8EE0E000 1000
[ 59.178341] change_pte_range 7FAB8EE0F000 1000
[ 59.178654] change_pte_range 7FAB8EE10000 1000
[ 59.178972] change_pte_range 7FAB8EE11000 1000
[ 59.179307] change_pte_range 7FAB8EE12000 1000
[ 59.179620] change_pte_range 7FAB8EE13000 1000
[ 59.179944] change_pte_range 7FAB8EE14000 1000
[ 59.180258] change_pte_range 7FAB8EE15000 1000
[ 59.180567] change_pte_range 7FAB8EE16000 1000
[ 59.180886] change_pte_range 7FAB8EE17000 1000
[ 59.181201] change_pte_range 7FAB8EE18000 1000
[ 59.181510] change_pte_range 7FAB8EE19000 1000
[ 59.181828] change_pte_range 7FAB8EE1A000 1000
[ 59.182139] change_pte_range 7FAB8EE1B000 1000
[ 59.182451] change_pte_range 7FAB8EE1C000 1000
[ 59.182769] change_pte_range 7FAB8EE1D000 1000
[ 59.183094] change_pte_range 7FAB8EE1E000 1000
[ 59.183410] change_pte_range 7FAB8EE1F000 1000
[ 59.183731] change_pte_range 7FAB8EE20000 1000
[ 59.184046] change_pte_range 7FAB8EE21000 1000
[ 59.184359] change_pte_range 7FAB8EE22000 1000
[ 59.184669] change_pte_range 7FAB8EE23000 1000
[ 59.184982] change_pte_range 7FAB8EE24000 1000
[ 59.185296] change_pte_range 7FAB8EE25000 1000
[ 59.185605] change_pte_range 7FAB8EE26000 1000
[ 59.185921] change_pte_range 7FAB8EE27000 1000
[ 59.186237] change_pte_range 7FAB8EE28000 1000
[ 59.186549] change_pte_range 7FAB8EE29000 1000
[ 59.186861] change_pte_range 7FAB8EE2A000 1000
[ 59.187185] change_pte_range 7FAB8EE2B000 1000
[ 59.187500] change_pte_range 7FAB8EE2C000 1000
[ 59.187821] change_pte_range 7FAB8EE2D000 1000
[ 59.188132] change_pte_range 7FAB8EE2E000 1000
[ 59.188450] change_pte_range 7FAB8EE2F000 1000
[ 59.188763] change_pte_range 7FAB8EE30000 1000
[ 59.189077] change_pte_range 7FAB8EE31000 1000
[ 59.189391] change_pte_range 7FAB8EE32000 1000
[ 59.189709] change_pte_range 7FAB8EE33000 1000
[ 59.190021] change_pte_range 7FAB8EE34000 1000
[ 59.190330] change_pte_range 7FAB8EE35000 1000
[ 59.190638] change_pte_range 7FAB8EE36000 1000
[ 59.190958] change_pte_range 7FAB8EE37000 1000
[ 59.191269] change_pte_range 7FAB8EE38000 1000
[ 59.191582] change_pte_range 7FAB8EE39000 1000
[ 59.191900] change_pte_range 7FAB8EE3A000 1000
[ 59.192218] change_pte_range 7FAB8EE3B000 1000
[ 59.192525] change_pte_range 7FAB8EE3C000 1000
[ 59.192845] change_pte_range 7FAB8EE3D000 1000
[ 59.193157] change_pte_range 7FAB8EE3E000 1000
[ 59.193466] change_pte_range 7FAB8EE3F000 1000
[ 59.193784] change_pte_range 7FAB8EE40000 1000
[ 59.194097] change_pte_range 7FAB8EE41000 1000
[ 59.194405] change_pte_range 7FAB8EE42000 1000
[ 59.194720] change_pte_range 7FAB8EE43000 1000
[ 59.195028] change_pte_range 7FAB8EE44000 1000
[ 59.195350] change_pte_range 7FAB8EE45000 1000
[ 59.195659] change_pte_range 7FAB8EE46000 1000
[ 59.195977] change_pte_range 7FAB8EE47000 1000
[ 59.196290] change_pte_range 7FAB8EE48000 1000
[ 59.196604] change_pte_range 7FAB8EE49000 1000
[ 59.196932] change_pte_range 7FAB8EE4A000 1000
[ 59.197260] change_pte_range 7FAB8EE4B000 1000
[ 59.197581] change_pte_range 7FAB8EE4C000 1000
[ 59.197917] change_pte_range 7FAB8EE4D000 1000
[ 59.198243] change_pte_range 7FAB8EE4E000 1000
[ 59.198563] change_pte_range 7FAB8EE4F000 1000
[ 59.198888] change_pte_range 7FAB8EE50000 1000
[ 59.199214] change_pte_range 7FAB8EE51000 1000
[ 59.199534] change_pte_range 7FAB8EE52000 1000
[ 59.199861] change_pte_range 7FAB8EE53000 1000
[ 59.200185] change_pte_range 7FAB8EE54000 1000
[ 59.200506] change_pte_range 7FAB8EE55000 1000
[ 59.200837] change_pte_range 7FAB8EE56000 1000
[ 59.201160] change_pte_range 7FAB8EE57000 1000
[ 59.201487] change_pte_range 7FAB8EE58000 1000
[ 59.201814] change_pte_range 7FAB8EE59000 1000
[ 59.202136] change_pte_range 7FAB8EE5A000 1000
[ 59.202463] change_pte_range 7FAB8EE5B000 1000
[ 59.202788] change_pte_range 7FAB8EE5C000 1000
[ 59.203130] change_pte_range 7FAB8EE5D000 1000
[ 59.203457] change_pte_range 7FAB8EE5E000 1000
[ 59.203775] change_pte_range 7FAB8EE5F000 1000
[ 59.204088] change_pte_range 7FAB8EE60000 1000
[ 59.204398] change_pte_range 7FAB8EE61000 1000
[ 59.204724] change_pte_range 7FAB8EE62000 1000
[ 59.205043] change_pte_range 7FAB8EE63000 1000
[ 59.205358] change_pte_range 7FAB8EE64000 1000
[ 59.205674] change_pte_range 7FAB8EE65000 1000
[ 59.206000] change_pte_range 7FAB8EE66000 1000
[ 59.206369] change_pte_range 7FAB8EE67000 1000
[ 59.206737] change_pte_range 7FAB8EE68000 1000
[ 59.207050] change_pte_range 7FAB8EE69000 1000
[ 59.207432] change_pte_range 7FAB8EE6A000 1000
[ 59.207822] change_pte_range 7FAB8EE6B000 1000
[ 59.208211] change_pte_range 7FAB8EE6C000 1000
[ 59.208599] change_pte_range 7FAB8EE6D000 1000
[ 59.208946] change_pte_range 7FAB8EE6E000 1000
[ 59.209267] change_pte_range 7FAB8EE6F000 1000
[ 59.209588] change_pte_range 7FAB8EE70000 1000
[ 59.209910] change_pte_range 7FAB8EE71000 1000
[ 59.210223] change_pte_range 7FAB8EE72000 1000
[ 59.210529] change_pte_range 7FAB8EE73000 1000
[ 59.210847] change_pte_range 7FAB8EE74000 1000
[ 59.211182] change_pte_range 7FAB8EE75000 1000
[ 59.211494] change_pte_range 7FAB8EE76000 1000
[ 59.211813] change_pte_range 7FAB8EE77000 1000
[ 59.212120] change_pte_range 7FAB8EE78000 1000
[ 59.212430] change_pte_range 7FAB8EE79000 1000
[ 59.212746] change_pte_range 7FAB8EE7A000 1000
[ 59.213056] change_pte_range 7FAB8EE7B000 1000
[ 59.213365] change_pte_range 7FAB8EE7C000 1000
[ 59.213675] change_pte_range 7FAB8EE7D000 1000
[ 59.213992] change_pte_range 7FAB8EE7E000 1000
[ 59.214300] change_pte_range 7FAB8EE7F000 1000
[ 59.214616] change_pte_range 7FAB8EE80000 1000
[ 59.214929] change_pte_range 7FAB8EE81000 1000
[ 59.215252] change_pte_range 7FAB8EE82000 1000
[ 59.215564] change_pte_range 7FAB8EE83000 1000
[ 59.215881] change_pte_range 7FAB8EE84000 1000
[ 59.216194] change_pte_range 7FAB8EE85000 1000
[ 59.216512] change_pte_range 7FAB8EE86000 1000
[ 59.216835] change_pte_range 7FAB8EE87000 1000
[ 59.217149] change_pte_range 7FAB8EE88000 1000
[ 59.217461] change_pte_range 7FAB8EE89000 1000
[ 59.217779] change_pte_range 7FAB8EE8A000 1000
[ 59.218089] change_pte_range 7FAB8EE8B000 1000
[ 59.218401] change_pte_range 7FAB8EE8C000 1000
[ 59.218713] change_pte_range 7FAB8EE8D000 1000
[ 59.219027] change_pte_range 7FAB8EE8E000 1000
[ 59.219367] change_pte_range 7FAB8EE8F000 1000
[ 59.219686] change_pte_range 7FAB8EE90000 1000
[ 59.220010] change_pte_range 7FAB8EE91000 1000
[ 59.220322] change_pte_range 7FAB8EE92000 1000
[ 59.220634] change_pte_range 7FAB8EE93000 1000
[ 59.220954] change_pte_range 7FAB8EE94000 1000
[ 59.221268] change_pte_range 7FAB8EE95000 1000
[ 59.221580] change_pte_range 7FAB8EE96000 1000
[ 59.221904] change_pte_range 7FAB8EE97000 1000
[ 59.222221] change_pte_range 7FAB8EE98000 1000
[ 59.222538] change_pte_range 7FAB8EE99000 1000
[ 59.222859] change_pte_range 7FAB8EE9A000 1000
[ 59.223232] change_pte_range 7FAB8EE9B000 1000
[ 59.223577] change_pte_range 7FAB8EE9C000 1000
[ 59.223902] change_pte_range 7FAB8EE9D000 1000
[ 59.224214] change_pte_range 7FAB8EE9E000 1000
[ 59.224526] change_pte_range 7FAB8EE9F000 1000
[ 59.224849] change_pte_range 7FAB8EEA0000 1000
[ 59.225160] change_pte_range 7FAB8EEA1000 1000
[ 59.225473] change_pte_range 7FAB8EEA2000 1000
[ 59.225793] change_pte_range 7FAB8EEA3000 1000
[ 59.226115] change_pte_range 7FAB8EEA4000 1000
[ 59.226432] change_pte_range 7FAB8EEA5000 1000
[ 59.226753] change_pte_range 7FAB8EEA6000 1000
[ 59.227091] change_pte_range 7FAB8EEA7000 1000
[ 59.227403] change_pte_range 7FAB8EEA8000 1000
[ 59.227728] change_pte_range 7FAB8EEA9000 1000
[ 59.228041] change_pte_range 7FAB8EEAA000 1000
[ 59.228350] change_pte_range 7FAB8EEAB000 1000
[ 59.228663] change_pte_range 7FAB8EEAC000 1000
[ 59.228984] change_pte_range 7FAB8EEAD000 1000
[ 59.229298] change_pte_range 7FAB8EEAE000 1000
[ 59.229608] change_pte_range 7FAB8EEAF000 1000
[ 59.229926] change_pte_range 7FAB8EEB0000 1000
[ 59.230239] change_pte_range 7FAB8EEB1000 1000
[ 59.230555] change_pte_range 7FAB8EEB2000 1000
[ 59.230876] change_pte_range 7FAB8EEB3000 1000
[ 59.231199] change_pte_range 7FAB8EEB4000 1000
[ 59.231512] change_pte_range 7FAB8EEB5000 1000
[ 59.231833] change_pte_range 7FAB8EEB6000 1000
[ 59.232146] change_pte_range 7FAB8EEB7000 1000
[ 59.232456] change_pte_range 7FAB8EEB8000 1000
[ 59.232776] change_pte_range 7FAB8EEB9000 1000
[ 59.233090] change_pte_range 7FAB8EEBA000 1000
[ 59.233413] change_pte_range 7FAB8EEBB000 1000
[ 59.233733] change_pte_range 7FAB8EEBC000 1000
[ 59.234058] change_pte_range 7FAB8EEBD000 1000
[ 59.234378] change_pte_range 7FAB8EEBE000 1000
[ 59.234705] change_pte_range 7FAB8EEBF000 1000
[ 59.235028] change_pte_range 7FAB8EEC0000 1000
[ 59.235354] change_pte_range 7FAB8EEC1000 1000
[ 59.235675] change_pte_range 7FAB8EEC2000 1000
[ 59.236006] change_pte_range 7FAB8EEC3000 1000
[ 59.236329] change_pte_range 7FAB8EEC4000 1000
[ 59.236652] change_pte_range 7FAB8EEC5000 1000
[ 59.236983] change_pte_range 7FAB8EEC6000 1000
[ 59.237309] change_pte_range 7FAB8EEC7000 1000
[ 59.237630] change_pte_range 7FAB8EEC8000 1000
[ 59.237962] change_pte_range 7FAB8EEC9000 1000
[ 59.238285] change_pte_range 7FAB8EECA000 1000
[ 59.238604] change_pte_range 7FAB8EECB000 1000
[ 59.238933] change_pte_range 7FAB8EECC000 1000
[ 59.239267] change_pte_range 7FAB8EECD000 1000
[ 59.239594] change_pte_range 7FAB8EECE000 1000
[ 59.239946] change_pte_range 7FAB8EECF000 1000
[ 59.240279] change_pte_range 7FAB8EED0000 1000
[ 59.240667] change_pte_range 7FAB8EED1000 1000
[ 59.241156] change_pte_range 7FAB8EED2000 1000
[ 59.241612] change_pte_range 7FAB8EED3000 1000
[ 59.242040] change_pte_range 7FAB8EED4000 1000
[ 59.242359] change_pte_range 7FAB8EED5000 1000
[ 59.242675] change_pte_range 7FAB8EED6000 1000
[ 59.242990] change_pte_range 7FAB8EED7000 1000
[ 59.243309] change_pte_range 7FAB8EED8000 1000
[ 59.243627] change_pte_range 7FAB8EED9000 1000
[ 59.243944] change_pte_range 7FAB8EEDA000 1000
[ 59.244256] change_pte_range 7FAB8EEDB000 1000
[ 59.244568] change_pte_range 7FAB8EEDC000 1000
[ 59.244892] change_pte_range 7FAB8EEDD000 1000
[ 59.245206] change_pte_range 7FAB8EEDE000 1000
[ 59.245518] change_pte_range 7FAB8EEDF000 1000
[ 59.245839] change_pte_range 7FAB8EEE0000 1000
[ 59.246152] change_pte_range 7FAB8EEE1000 1000
[ 59.246460] change_pte_range 7FAB8EEE2000 1000
[ 59.246782] change_pte_range 7FAB8EEE3000 1000
[ 59.247103] change_pte_range 7FAB8EEE4000 1000
[ 59.247421] change_pte_range 7FAB8EEE5000 1000
[ 59.247739] change_pte_range 7FAB8EEE6000 1000
[ 59.248045] change_pte_range 7FAB8EEE7000 1000
[ 59.248363] change_pte_range 7FAB8EEE8000 1000
[ 59.248674] change_pte_range 7FAB8EEE9000 1000
[ 59.248996] change_pte_range 7FAB8EEEA000 1000
[ 59.249312] change_pte_range 7FAB8EEEB000 1000
[ 59.249618] change_pte_range 7FAB8EEEC000 1000
[ 59.249940] change_pte_range 7FAB8EEED000 1000
[ 59.250253] change_pte_range 7FAB8EEEE000 1000
[ 59.250563] change_pte_range 7FAB8EEEF000 1000
[ 59.250879] change_pte_range 7FAB8EEF0000 1000
[ 59.251203] change_pte_range 7FAB8EEF1000 1000
[ 59.251512] change_pte_range 7FAB8EEF2000 1000
[ 59.251830] change_pte_range 7FAB8EEF3000 1000
[ 59.252140] change_pte_range 7FAB8EEF4000 1000
[ 59.252457] change_pte_range 7FAB8EEF5000 1000
[ 59.252778] change_pte_range 7FAB8EEF6000 1000
[ 59.253092] change_pte_range 7FAB8EEF7000 1000
[ 59.253404] change_pte_range 7FAB8EEF8000 1000
[ 59.253727] change_pte_range 7FAB8EEF9000 1000
[ 59.254039] change_pte_range 7FAB8EEFA000 1000
[ 59.254346] change_pte_range 7FAB8EEFB000 1000
[ 59.254656] change_pte_range 7FAB8EEFC000 1000
[ 59.254980] change_pte_range 7FAB8EEFD000 1000
[ 59.255301] change_pte_range 7FAB8EEFE000 1000
[ 59.255617] change_pte_range 7FAB8EEFF000 1000
[ 59.255939] change_pte_range 7FAB8EF00000 1000
[ 59.256312] change_pte_range 7FAB8EF01000 1000
[ 59.256835] change_pte_range 7FAB8EF02000 1000
[ 59.257221] change_pte_range 7FAB8EF03000 1000
[ 59.257600] change_pte_range 7FAB8EF04000 1000
[ 59.257986] change_pte_range 7FAB8EF05000 1000
[ 59.258368] change_pte_range 7FAB8EF06000 1000
[ 59.258752] change_pte_range 7FAB8EF07000 1000
[ 59.259142] change_pte_range 7FAB8EF08000 1000
[ 59.259528] change_pte_range 7FAB8EF09000 1000
[ 59.259918] change_pte_range 7FAB8EF0A000 1000
[ 59.260300] change_pte_range 7FAB8EF0B000 1000
[ 59.260679] change_pte_range 7FAB8EF0C000 1000
[ 59.261064] change_pte_range 7FAB8EF0D000 1000
[ 59.261449] change_pte_range 7FAB8EF0E000 1000
[ 59.261835] change_pte_range 7FAB8EF0F000 1000
[ 59.262214] change_pte_range 7FAB8EF10000 1000
[ 59.262592] change_pte_range 7FAB8EF11000 1000
[ 59.262978] change_pte_range 7FAB8EF12000 1000
[ 59.263375] change_pte_range 7FAB8EF13000 1000
[ 59.263725] change_pte_range 7FAB8EF14000 1000
[ 59.264039] change_pte_range 7FAB8EF15000 1000
[ 59.264346] change_pte_range 7FAB8EF16000 1000
[ 59.264654] change_pte_range 7FAB8EF17000 1000
[ 59.264970] change_pte_range 7FAB8EF18000 1000
[ 59.265282] change_pte_range 7FAB8EF19000 1000
[ 59.265589] change_pte_range 7FAB8EF1A000 1000
[ 59.265908] change_pte_range 7FAB8EF1B000 1000
[ 59.266216] change_pte_range 7FAB8EF1C000 1000
[ 59.266530] change_pte_range 7FAB8EF1D000 1000
[ 59.266849] change_pte_range 7FAB8EF1E000 1000
[ 59.267170] change_pte_range 7FAB8EF1F000 1000
[ 59.267484] change_pte_range 7FAB8EF20000 1000
[ 59.267806] change_pte_range 7FAB8EF21000 1000
[ 59.268125] change_pte_range 7FAB8EF22000 1000
[ 59.268466] change_pte_range 7FAB8EF23000 1000
[ 59.268826] change_pte_range 7FAB8EF24000 1000
[ 59.269143] change_pte_range 7FAB8EF25000 1000
[ 59.269460] change_pte_range 7FAB8EF26000 1000
[ 59.269780] change_pte_range 7FAB8EF27000 1000
[ 59.270096] change_pte_range 7FAB8EF28000 1000
[ 59.270407] change_pte_range 7FAB8EF29000 1000
[ 59.270733] change_pte_range 7FAB8EF2A000 1000
[ 59.271046] change_pte_range 7FAB8EF2B000 1000
[ 59.271378] change_pte_range 7FAB8EF2C000 1000
[ 59.271693] change_pte_range 7FAB8EF2D000 1000
[ 59.272023] change_pte_range 7FAB8EF2E000 1000
[ 59.272363] change_pte_range 7FAB8EF2F000 1000
[ 59.272685] change_pte_range 7FAB8EF30000 1000
[ 59.273014] change_pte_range 7FAB8EF31000 1000
[ 59.273337] change_pte_range 7FAB8EF32000 1000
[ 59.273655] change_pte_range 7FAB8EF33000 1000
[ 59.273998] change_pte_range 7FAB8EF34000 1000
[ 59.274310] change_pte_range 7FAB8EF35000 1000
[ 59.274752] change_pte_range 7FAB8EF36000 1000
[ 59.275230] change_pte_range 7FAB8EF37000 1000
[ 59.275688] change_pte_range 7FAB8EF38000 1000
[ 59.276088] change_pte_range 7FAB8EF39000 1000
[ 59.276413] change_pte_range 7FAB8EF3A000 1000
[ 59.276742] change_pte_range 7FAB8EF3B000 1000
[ 59.277067] change_pte_range 7FAB8EF3C000 1000
[ 59.277393] change_pte_range 7FAB8EF3D000 1000
[ 59.277716] change_pte_range 7FAB8EF3E000 1000
[ 59.278040] change_pte_range 7FAB8EF3F000 1000
[ 59.278350] change_pte_range 7FAB8EF40000 1000
[ 59.278658] change_pte_range 7FAB8EF41000 1000
[ 59.278975] change_pte_range 7FAB8EF42000 1000
[ 59.279292] change_pte_range 7FAB8EF43000 1000
[ 59.279604] change_pte_range 7FAB8EF44000 1000
[ 59.279921] change_pte_range 7FAB8EF45000 1000
[ 59.280233] change_pte_range 7FAB8EF46000 1000
[ 59.280545] change_pte_range 7FAB8EF47000 1000
[ 59.280863] change_pte_range 7FAB8EF48000 1000
[ 59.281175] change_pte_range 7FAB8EF49000 1000
[ 59.281483] change_pte_range 7FAB8EF4A000 1000
[ 59.281801] change_pte_range 7FAB8EF4B000 1000
[ 59.282110] change_pte_range 7FAB8EF4C000 1000
[ 59.282434] change_pte_range 7FAB8EF4D000 1000
[ 59.282752] change_pte_range 7FAB8EF4E000 1000
[ 59.283071] change_pte_range 7FAB8EF4F000 1000
[ 59.283383] change_pte_range 7FAB8EF50000 1000
[ 59.283704] change_pte_range 7FAB8EF51000 1000
[ 59.284012] change_pte_range 7FAB8EF52000 1000
[ 59.284325] change_pte_range 7FAB8EF53000 1000
[ 59.284636] change_pte_range 7FAB8EF54000 1000
[ 59.284953] change_pte_range 7FAB8EF55000 1000
[ 59.285269] change_pte_range 7FAB8EF56000 1000
[ 59.285586] change_pte_range 7FAB8EF57000 1000
[ 59.285904] change_pte_range 7FAB8EF58000 1000
[ 59.286217] change_pte_range 7FAB8EF59000 1000
[ 59.286529] change_pte_range 7FAB8EF5A000 1000
[ 59.286850] change_pte_range 7FAB8EF5B000 1000
[ 59.287170] change_pte_range 7FAB8EF5C000 1000
[ 59.287478] change_pte_range 7FAB8EF5D000 1000
[ 59.287789] change_pte_range 7FAB8EF5E000 1000
[ 59.288105] change_pte_range 7FAB8EF5F000 1000
[ 59.288419] change_pte_range 7FAB8EF60000 1000
[ 59.288737] change_pte_range 7FAB8EF61000 1000
[ 59.289046] change_pte_range 7FAB8EF62000 1000
[ 59.289363] change_pte_range 7FAB8EF63000 1000
[ 59.289682] change_pte_range 7FAB8EF64000 1000
[ 59.289999] change_pte_range 7FAB8EF65000 1000
[ 59.290375] change_pte_range 7FAB8EF66000 1000
[ 59.290774] change_pte_range 7FAB8EF67000 1000
[ 59.291161] change_pte_range 7FAB8EF68000 1000
[ 59.291541] change_pte_range 7FAB8EF69000 1000
[ 59.291933] change_pte_range 7FAB8EF6A000 1000
[ 59.292322] change_pte_range 7FAB8EF6B000 1000
[ 59.292660] change_pte_range 7FAB8EF6C000 1000
[ 59.292982] change_pte_range 7FAB8EF6D000 1000
[ 59.293298] change_pte_range 7FAB8EF6E000 1000
[ 59.293611] change_pte_range 7FAB8EF6F000 1000
[ 59.293930] change_pte_range 7FAB8EF70000 1000
[ 59.294247] change_pte_range 7FAB8EF71000 1000
[ 59.294556] change_pte_range 7FAB8EF72000 1000
[ 59.294874] change_pte_range 7FAB8EF73000 1000
[ 59.295200] change_pte_range 7FAB8EF74000 1000
[ 59.295511] change_pte_range 7FAB8EF75000 1000
[ 59.295825] change_pte_range 7FAB8EF76000 1000
[ 59.296135] change_pte_range 7FAB8EF77000 1000
[ 59.296448] change_pte_range 7FAB8EF78000 1000
[ 59.296763] change_pte_range 7FAB8EF79000 1000
[ 59.297071] change_pte_range 7FAB8EF7A000 1000
[ 59.297383] change_pte_range 7FAB8EF7B000 1000
[ 59.297706] change_pte_range 7FAB8EF7C000 1000
[ 59.298021] change_pte_range 7FAB8EF7D000 1000
[ 59.298329] change_pte_range 7FAB8EF7E000 1000
[ 59.298636] change_pte_range 7FAB8EF7F000 1000
[ 59.298951] change_pte_range 7FAB8EF80000 1000
[ 59.299280] change_pte_range 7FAB8EF81000 1000
[ 59.299599] change_pte_range 7FAB8EF82000 1000
[ 59.299917] change_pte_range 7FAB8EF83000 1000
[ 59.300232] change_pte_range 7FAB8EF84000 1000
[ 59.300546] change_pte_range 7FAB8EF85000 1000
[ 59.300861] change_pte_range 7FAB8EF86000 1000
[ 59.301170] change_pte_range 7FAB8EF87000 1000
[ 59.301487] change_pte_range 7FAB8EF88000 1000
[ 59.301803] change_pte_range 7FAB8EF89000 1000
[ 59.302111] change_pte_range 7FAB8EF8A000 1000
[ 59.302424] change_pte_range 7FAB8EF8B000 1000
[ 59.302740] change_pte_range 7FAB8EF8C000 1000
[ 59.303064] change_pte_range 7FAB8EF8D000 1000
[ 59.303375] change_pte_range 7FAB8EF8E000 1000
[ 59.303691] change_pte_range 7FAB8EF8F000 1000
[ 59.304013] change_pte_range 7FAB8EF90000 1000
[ 59.304347] change_pte_range 7FAB8EF91000 1000
[ 59.304660] change_pte_range 7FAB8EF92000 1000
[ 59.304984] change_pte_range 7FAB8EF93000 1000
[ 59.305299] change_pte_range 7FAB8EF94000 1000
[ 59.305619] change_pte_range 7FAB8EF95000 1000
[ 59.305935] change_pte_range 7FAB8EF96000 1000
[ 59.306249] change_pte_range 7FAB8EF97000 1000
[ 59.306575] change_pte_range 7FAB8EF98000 1000
[ 59.307022] change_pte_range 7FAB8EF99000 1000
[ 59.307546] change_pte_range 7FAB8EF9A000 1000
[ 59.307951] change_pte_range 7FAB8EF9B000 1000
[ 59.308268] change_pte_range 7FAB8EF9C000 1000
[ 59.308583] change_pte_range 7FAB8EF9D000 1000
[ 59.308905] change_pte_range 7FAB8EF9E000 1000
[ 59.309216] change_pte_range 7FAB8EF9F000 1000
[ 59.309536] change_pte_range 7FAB8EFA0000 1000
[ 59.309866] change_pte_range 7FAB8EFA1000 1000
[ 59.310191] change_pte_range 7FAB8EFA2000 1000
[ 59.310512] change_pte_range 7FAB8EFA3000 1000
[ 59.310844] change_pte_range 7FAB8EFA4000 1000
[ 59.311173] change_pte_range 7FAB8EFA5000 1000
[ 59.311496] change_pte_range 7FAB8EFA6000 1000
[ 59.311827] change_pte_range 7FAB8EFA7000 1000
[ 59.312151] change_pte_range 7FAB8EFA8000 1000
[ 59.312473] change_pte_range 7FAB8EFA9000 1000
[ 59.312804] change_pte_range 7FAB8EFAA000 1000
[ 59.313128] change_pte_range 7FAB8EFAB000 1000
[ 59.313452] change_pte_range 7FAB8EFAC000 1000
[ 59.313779] change_pte_range 7FAB8EFAD000 1000
[ 59.314100] change_pte_range 7FAB8EFAE000 1000
[ 59.314429] change_pte_range 7FAB8EFAF000 1000
[ 59.314758] change_pte_range 7FAB8EFB0000 1000
[ 59.315087] change_pte_range 7FAB8EFB1000 1000
[ 59.315412] change_pte_range 7FAB8EFB2000 1000
[ 59.315739] change_pte_range 7FAB8EFB3000 1000
[ 59.316064] change_pte_range 7FAB8EFB4000 1000
[ 59.316384] change_pte_range 7FAB8EFB5000 1000
[ 59.316696] change_pte_range 7FAB8EFB6000 1000
[ 59.317010] change_pte_range 7FAB8EFB7000 1000
[ 59.317326] change_pte_range 7FAB8EFB8000 1000
[ 59.317642] change_pte_range 7FAB8EFB9000 1000
[ 59.317960] change_pte_range 7FAB8EFBA000 1000
[ 59.318269] change_pte_range 7FAB8EFBB000 1000
[ 59.318579] change_pte_range 7FAB8EFBC000 1000
[ 59.318893] change_pte_range 7FAB8EFBD000 1000
[ 59.319217] change_pte_range 7FAB8EFBE000 1000
[ 59.319532] change_pte_range 7FAB8EFBF000 1000
[ 59.319851] change_pte_range 7FAB8EFC0000 1000
[ 59.320161] change_pte_range 7FAB8EFC1000 1000
[ 59.320480] change_pte_range 7FAB8EFC2000 1000
[ 59.320796] change_pte_range 7FAB8EFC3000 1000
[ 59.321109] change_pte_range 7FAB8EFC4000 1000
[ 59.321418] change_pte_range 7FAB8EFC5000 1000
[ 59.321743] change_pte_range 7FAB8EFC6000 1000
[ 59.322055] change_pte_range 7FAB8EFC7000 1000
[ 59.322367] change_pte_range 7FAB8EFC8000 1000
[ 59.322683] change_pte_range 7FAB8EFC9000 1000
[ 59.323049] change_pte_range 7FAB8EFCA000 1000
[ 59.323384] change_pte_range 7FAB8EFCB000 1000
[ 59.323753] change_pte_range 7FAB8EFCC000 1000
[ 59.324072] change_pte_range 7FAB8EFCD000 1000
[ 59.324389] change_pte_range 7FAB8EFCE000 1000
[ 59.324711] change_pte_range 7FAB8EFCF000 1000
[ 59.325027] change_pte_range 7FAB8EFD0000 1000
[ 59.325341] change_pte_range 7FAB8EFD1000 1000
[ 59.325656] change_pte_range 7FAB8EFD2000 1000
[ 59.325985] change_pte_range 7FAB8EFD3000 1000
[ 59.326297] change_pte_range 7FAB8EFD4000 1000
[ 59.326618] change_pte_range 7FAB8EFD5000 1000
[ 59.326938] change_pte_range 7FAB8EFD6000 1000
[ 59.327256] change_pte_range 7FAB8EFD7000 1000
[ 59.327567] change_pte_range 7FAB8EFD8000 1000
[ 59.327886] change_pte_range 7FAB8EFD9000 1000
[ 59.328196] change_pte_range 7FAB8EFDA000 1000
[ 59.328512] change_pte_range 7FAB8EFDB000 1000
[ 59.328831] change_pte_range 7FAB8EFDC000 1000
[ 59.329145] change_pte_range 7FAB8EFDD000 1000
[ 59.329457] change_pte_range 7FAB8EFDE000 1000
[ 59.329776] change_pte_range 7FAB8EFDF000 1000
[ 59.330083] change_pte_range 7FAB8EFE0000 1000
[ 59.330400] change_pte_range 7FAB8EFE1000 1000
[ 59.330720] change_pte_range 7FAB8EFE2000 1000
[ 59.331032] change_pte_range 7FAB8EFE3000 1000
[ 59.331355] change_pte_range 7FAB8EFE4000 1000
[ 59.331664] change_pte_range 7FAB8EFE5000 1000
[ 59.331982] change_pte_range 7FAB8EFE6000 1000
[ 59.332296] change_pte_range 7FAB8EFE7000 1000
[ 59.332610] change_pte_range 7FAB8EFE8000 1000
[ 59.332928] change_pte_range 7FAB8EFE9000 1000
[ 59.333250] change_pte_range 7FAB8EFEA000 1000
[ 59.333567] change_pte_range 7FAB8EFEB000 1000
[ 59.333891] change_pte_range 7FAB8EFEC000 1000
[ 59.334206] change_pte_range 7FAB8EFED000 1000
[ 59.334522] change_pte_range 7FAB8EFEE000 1000
[ 59.334839] change_pte_range 7FAB8EFEF000 1000
[ 59.335167] change_pte_range 7FAB8EFF0000 1000
[ 59.335480] change_pte_range 7FAB8EFF1000 1000
[ 59.335798] change_pte_range 7FAB8EFF2000 1000
[ 59.336111] change_pte_range 7FAB8EFF3000 1000
[ 59.336424] change_pte_range 7FAB8EFF4000 1000
[ 59.336746] change_pte_range 7FAB8EFF5000 1000
[ 59.337069] change_pte_range 7FAB8EFF6000 1000
[ 59.337383] change_pte_range 7FAB8EFF7000 1000
[ 59.337705] change_pte_range 7FAB8EFF8000 1000
[ 59.338018] change_pte_range 7FAB8EFF9000 1000
[ 59.338329] change_pte_range 7FAB8EFFA000 1000
[ 59.338641] change_pte_range 7FAB8EFFB000 1000
[ 59.338962] change_pte_range 7FAB8EFFC000 1000
[ 59.339278] change_pte_range 7FAB8EFFD000 1000
[ 59.339587] change_pte_range 7FAB8EFFE000 1000
[ 59.339905] change_pte_range 7FAB8EFFF000 1000
[ 59.340236] change_pte_range 7FAB8F000000 1000
[ 59.340599] change_pte_range 7FAB8F001000 1000
[ 59.340932] change_pte_range 7FAB8F002000 1000
[ 59.341248] change_pte_range 7FAB8F003000 1000
[ 59.341557] change_pte_range 7FAB8F004000 1000
[ 59.341877] change_pte_range 7FAB8F005000 1000
[ 59.342191] change_pte_range 7FAB8F006000 1000
[ 59.342507] change_pte_range 7FAB8F007000 1000
[ 59.342828] change_pte_range 7FAB8F008000 1000
[ 59.343149] change_pte_range 7FAB8F009000 1000
[ 59.343463] change_pte_range 7FAB8F00A000 1000
[ 59.343787] change_pte_range 7FAB8F00B000 1000
[ 59.344102] change_pte_range 7FAB8F00C000 1000
[ 59.344421] change_pte_range 7FAB8F00D000 1000
[ 59.344745] change_pte_range 7FAB8F00E000 1000
[ 59.345063] change_pte_range 7FAB8F00F000 1000
[ 59.345381] change_pte_range 7FAB8F010000 1000
[ 59.345692] change_pte_range 7FAB8F011000 1000
[ 59.346016] change_pte_range 7FAB8F012000 1000
[ 59.346333] change_pte_range 7FAB8F013000 1000
[ 59.346644] change_pte_range 7FAB8F014000 1000
[ 59.346971] change_pte_range 7FAB8F015000 1000
[ 59.347295] change_pte_range 7FAB8F016000 1000
[ 59.347617] change_pte_range 7FAB8F017000 1000
[ 59.347949] change_pte_range 7FAB8F018000 1000
[ 59.348277] change_pte_range 7FAB8F019000 1000
[ 59.348598] change_pte_range 7FAB8F01A000 1000
[ 59.348932] change_pte_range 7FAB8F01B000 1000
[ 59.349257] change_pte_range 7FAB8F01C000 1000
[ 59.349584] change_pte_range 7FAB8F01D000 1000
[ 59.349918] change_pte_range 7FAB8F01E000 1000
[ 59.350243] change_pte_range 7FAB8F01F000 1000
[ 59.350562] change_pte_range 7FAB8F020000 1000
[ 59.350888] change_pte_range 7FAB8F021000 1000
[ 59.351220] change_pte_range 7FAB8F022000 1000
[ 59.351540] change_pte_range 7FAB8F023000 1000
[ 59.351867] change_pte_range 7FAB8F024000 1000
[ 59.352189] change_pte_range 7FAB8F025000 1000
[ 59.352509] change_pte_range 7FAB8F026000 1000
[ 59.352835] change_pte_range 7FAB8F027000 1000
[ 59.353156] change_pte_range 7FAB8F028000 1000
[ 59.353477] change_pte_range 7FAB8F029000 1000
[ 59.353807] change_pte_range 7FAB8F02A000 1000
[ 59.354118] change_pte_range 7FAB8F02B000 1000
[ 59.354430] change_pte_range 7FAB8F02C000 1000
[ 59.354747] change_pte_range 7FAB8F02D000 1000
[ 59.355059] change_pte_range 7FAB8F02E000 1000
[ 59.355378] change_pte_range 7FAB8F02F000 1000
[ 59.355704] change_pte_range 7FAB8F030000 1000
[ 59.356020] change_pte_range 7FAB8F031000 1000
[ 59.356330] change_pte_range 7FAB8F032000 1000
[ 59.356661] change_pte_range 7FAB8F033000 1000
[ 59.357013] change_pte_range 7FAB8F034000 1000
[ 59.357348] change_pte_range 7FAB8F035000 1000
[ 59.357658] change_pte_range 7FAB8F036000 1000
[ 59.357986] change_pte_range 7FAB8F037000 1000
[ 59.358304] change_pte_range 7FAB8F038000 1000
[ 59.358619] change_pte_range 7FAB8F039000 1000
[ 59.358940] change_pte_range 7FAB8F03A000 1000
[ 59.359258] change_pte_range 7FAB8F03B000 1000
[ 59.359570] change_pte_range 7FAB8F03C000 1000
[ 59.359890] change_pte_range 7FAB8F03D000 1000
[ 59.360206] change_pte_range 7FAB8F03E000 1000
[ 59.360517] change_pte_range 7FAB8F03F000 1000
[ 59.360840] change_pte_range 7FAB8F040000 1000
[ 59.361155] change_pte_range 7FAB8F041000 1000
[ 59.361470] change_pte_range 7FAB8F042000 1000
[ 59.361783] change_pte_range 7FAB8F043000 1000
[ 59.362097] change_pte_range 7FAB8F044000 1000
[ 59.362409] change_pte_range 7FAB8F045000 1000
[ 59.362721] change_pte_range 7FAB8F046000 1000
[ 59.363036] change_pte_range 7FAB8F047000 1000
[ 59.363354] change_pte_range 7FAB8F048000 1000
[ 59.363666] change_pte_range 7FAB8F049000 1000
[ 59.363982] change_pte_range 7FAB8F04A000 1000
[ 59.364293] change_pte_range 7FAB8F04B000 1000
[ 59.364609] change_pte_range 7FAB8F04C000 1000
[ 59.364930] change_pte_range 7FAB8F04D000 1000
[ 59.365239] change_pte_range 7FAB8F04E000 1000
[ 59.365555] change_pte_range 7FAB8F04F000 1000
[ 59.365873] change_pte_range 7FAB8F050000 1000
[ 59.366189] change_pte_range 7FAB8F051000 1000
[ 59.366495] change_pte_range 7FAB8F052000 1000
[ 59.366816] change_pte_range 7FAB8F053000 1000
[ 59.367134] change_pte_range 7FAB8F054000 1000
[ 59.367450] change_pte_range 7FAB8F055000 1000
[ 59.367775] change_pte_range 7FAB8F056000 1000
[ 59.368091] change_pte_range 7FAB8F057000 1000
[ 59.368411] change_pte_range 7FAB8F058000 1000
[ 59.368738] change_pte_range 7FAB8F059000 1000
[ 59.369059] change_pte_range 7FAB8F05A000 1000
[ 59.369372] change_pte_range 7FAB8F05B000 1000
[ 59.369684] change_pte_range 7FAB8F05C000 1000
[ 59.370011] change_pte_range 7FAB8F05D000 1000
[ 59.370324] change_pte_range 7FAB8F05E000 1000
[ 59.370637] change_pte_range 7FAB8F05F000 1000
[ 59.370959] change_pte_range 7FAB8F060000 1000
[ 59.371276] change_pte_range 7FAB8F061000 1000
[ 59.371594] change_pte_range 7FAB8F062000 1000
[ 59.371910] change_pte_range 7FAB8F063000 1000
[ 59.372225] change_pte_range 7FAB8F064000 1000
[ 59.372538] change_pte_range 7FAB8F065000 1000
[ 59.372866] change_pte_range 7FAB8F066000 1000
[ 59.373181] change_pte_range 7FAB8F067000 1000
[ 59.373499] change_pte_range 7FAB8F068000 1000
[ 59.373858] change_pte_range 7FAB8F069000 1000
[ 59.374192] change_pte_range 7FAB8F06A000 1000
[ 59.374519] change_pte_range 7FAB8F06B000 1000
[ 59.374840] change_pte_range 7FAB8F06C000 1000
[ 59.375163] change_pte_range 7FAB8F06D000 1000
[ 59.375475] change_pte_range 7FAB8F06E000 1000
[ 59.375799] change_pte_range 7FAB8F06F000 1000
[ 59.376113] change_pte_range 7FAB8F070000 1000
[ 59.376431] change_pte_range 7FAB8F071000 1000
[ 59.376756] change_pte_range 7FAB8F072000 1000
[ 59.377069] change_pte_range 7FAB8F073000 1000
[ 59.377383] change_pte_range 7FAB8F074000 1000
[ 59.377707] change_pte_range 7FAB8F075000 1000
[ 59.378020] change_pte_range 7FAB8F076000 1000
[ 59.378342] change_pte_range 7FAB8F077000 1000
[ 59.378659] change_pte_range 7FAB8F078000 1000
[ 59.378980] change_pte_range 7FAB8F079000 1000
[ 59.379297] change_pte_range 7FAB8F07A000 1000
[ 59.379614] change_pte_range 7FAB8F07B000 1000
[ 59.379935] change_pte_range 7FAB8F07C000 1000
[ 59.380249] change_pte_range 7FAB8F07D000 1000
[ 59.380561] change_pte_range 7FAB8F07E000 1000
[ 59.380882] change_pte_range 7FAB8F07F000 1000
[ 59.381198] change_pte_range 7FAB8F080000 1000
[ 59.381515] change_pte_range 7FAB8F081000 1000
[ 59.381835] change_pte_range 7FAB8F082000 1000
[ 59.382146] change_pte_range 7FAB8F083000 1000
[ 59.382460] change_pte_range 7FAB8F084000 1000
[ 59.382783] change_pte_range 7FAB8F085000 1000
[ 59.383103] change_pte_range 7FAB8F086000 1000
[ 59.383420] change_pte_range 7FAB8F087000 1000
[ 59.383739] change_pte_range 7FAB8F088000 1000
[ 59.384057] change_pte_range 7FAB8F089000 1000
[ 59.384371] change_pte_range 7FAB8F08A000 1000
[ 59.384685] change_pte_range 7FAB8F08B000 1000
[ 59.385013] change_pte_range 7FAB8F08C000 1000
[ 59.385337] change_pte_range 7FAB8F08D000 1000
[ 59.385664] change_pte_range 7FAB8F08E000 1000
[ 59.385994] change_pte_range 7FAB8F08F000 1000
[ 59.386318] change_pte_range 7FAB8F090000 1000
[ 59.386641] change_pte_range 7FAB8F091000 1000
[ 59.386971] change_pte_range 7FAB8F092000 1000
[ 59.387303] change_pte_range 7FAB8F093000 1000
[ 59.387628] change_pte_range 7FAB8F094000 1000
[ 59.387951] change_pte_range 7FAB8F095000 1000
[ 59.388265] change_pte_range 7FAB8F096000 1000
[ 59.388588] change_pte_range 7FAB8F097000 1000
[ 59.388923] change_pte_range 7FAB8F098000 1000
[ 59.389246] change_pte_range 7FAB8F099000 1000
[ 59.389579] change_pte_range 7FAB8F09A000 1000
[ 59.389917] change_pte_range 7FAB8F09B000 1000
[ 59.390265] change_pte_range 7FAB8F09C000 1000
[ 59.390585] change_pte_range 7FAB8F09D000 1000
[ 59.390911] change_pte_range 7FAB8F09E000 1000
[ 59.391259] change_pte_range 7FAB8F09F000 1000
[ 59.391575] change_pte_range 7FAB8F0A0000 1000
[ 59.391903] change_pte_range 7FAB8F0A1000 1000
[ 59.392215] change_pte_range 7FAB8F0A2000 1000
[ 59.392532] change_pte_range 7FAB8F0A3000 1000
[ 59.392845] change_pte_range 7FAB8F0A4000 1000
[ 59.393159] change_pte_range 7FAB8F0A5000 1000
[ 59.393476] change_pte_range 7FAB8F0A6000 1000
[ 59.393797] change_pte_range 7FAB8F0A7000 1000
[ 59.394111] change_pte_range 7FAB8F0A8000 1000
[ 59.394425] change_pte_range 7FAB8F0A9000 1000
[ 59.394746] change_pte_range 7FAB8F0AA000 1000
[ 59.395069] change_pte_range 7FAB8F0AB000 1000
[ 59.395384] change_pte_range 7FAB8F0AC000 1000
[ 59.395713] change_pte_range 7FAB8F0AD000 1000
[ 59.396028] change_pte_range 7FAB8F0AE000 1000
[ 59.396354] change_pte_range 7FAB8F0AF000 1000
[ 59.396675] change_pte_range 7FAB8F0B0000 1000
[ 59.396997] change_pte_range 7FAB8F0B1000 1000
[ 59.397309] change_pte_range 7FAB8F0B2000 1000
[ 59.397621] change_pte_range 7FAB8F0B3000 1000
[ 59.397937] change_pte_range 7FAB8F0B4000 1000
[ 59.398247] change_pte_range 7FAB8F0B5000 1000
[ 59.398560] change_pte_range 7FAB8F0B6000 1000
[ 59.398884] change_pte_range 7FAB8F0B7000 1000
[ 59.399211] change_pte_range 7FAB8F0B8000 1000
[ 59.399526] change_pte_range 7FAB8F0B9000 1000
[ 59.399848] change_pte_range 7FAB8F0BA000 1000
[ 59.400163] change_pte_range 7FAB8F0BB000 1000
[ 59.400485] change_pte_range 7FAB8F0BC000 1000
[ 59.400809] change_pte_range 7FAB8F0BD000 1000
[ 59.401121] change_pte_range 7FAB8F0BE000 1000
[ 59.401447] change_pte_range 7FAB8F0BF000 1000
[ 59.401769] change_pte_range 7FAB8F0C0000 1000
[ 59.402081] change_pte_range 7FAB8F0C1000 1000
[ 59.402392] change_pte_range 7FAB8F0C2000 1000
[ 59.402715] change_pte_range 7FAB8F0C3000 1000
[ 59.403032] change_pte_range 7FAB8F0C4000 1000
[ 59.403357] change_pte_range 7FAB8F0C5000 1000
[ 59.403673] change_pte_range 7FAB8F0C6000 1000
[ 59.403993] change_pte_range 7FAB8F0C7000 1000
[ 59.404308] change_pte_range 7FAB8F0C8000 1000
[ 59.404627] change_pte_range 7FAB8F0C9000 1000
[ 59.404950] change_pte_range 7FAB8F0CA000 1000
[ 59.405267] change_pte_range 7FAB8F0CB000 1000
[ 59.405578] change_pte_range 7FAB8F0CC000 1000
[ 59.405903] change_pte_range 7FAB8F0CD000 1000
[ 59.406268] change_pte_range 7FAB8F0CE000 1000
[ 59.406576] change_pte_range 7FAB8F0CF000 1000
[ 59.406903] change_pte_range 7FAB8F0D0000 1000
[ 59.407226] change_pte_range 7FAB8F0D1000 1000
[ 59.407534] change_pte_range 7FAB8F0D2000 1000
[ 59.407853] change_pte_range 7FAB8F0D3000 1000
[ 59.408170] change_pte_range 7FAB8F0D4000 1000
[ 59.408488] change_pte_range 7FAB8F0D5000 1000
[ 59.408812] change_pte_range 7FAB8F0D6000 1000
[ 59.409127] change_pte_range 7FAB8F0D7000 1000
[ 59.409447] change_pte_range 7FAB8F0D8000 1000
[ 59.409772] change_pte_range 7FAB8F0D9000 1000
[ 59.410091] change_pte_range 7FAB8F0DA000 1000
[ 59.410412] change_pte_range 7FAB8F0DB000 1000
[ 59.410733] change_pte_range 7FAB8F0DC000 1000
[ 59.411052] change_pte_range 7FAB8F0DD000 1000
[ 59.411386] change_pte_range 7FAB8F0DE000 1000
[ 59.411709] change_pte_range 7FAB8F0DF000 1000
[ 59.412025] change_pte_range 7FAB8F0E0000 1000
[ 59.412334] change_pte_range 7FAB8F0E1000 1000
[ 59.412658] change_pte_range 7FAB8F0E2000 1000
[ 59.412987] change_pte_range 7FAB8F0E3000 1000
[ 59.413313] change_pte_range 7FAB8F0E4000 1000
[ 59.413626] change_pte_range 7FAB8F0E5000 1000
[ 59.413948] change_pte_range 7FAB8F0E6000 1000
[ 59.414264] change_pte_range 7FAB8F0E7000 1000
[ 59.414579] change_pte_range 7FAB8F0E8000 1000
[ 59.414895] change_pte_range 7FAB8F0E9000 1000
[ 59.415242] change_pte_range 7FAB8F0EA000 1000
[ 59.415553] change_pte_range 7FAB8F0EB000 1000
[ 59.415877] change_pte_range 7FAB8F0EC000 1000
[ 59.416195] change_pte_range 7FAB8F0ED000 1000
[ 59.416508] change_pte_range 7FAB8F0EE000 1000
[ 59.416831] change_pte_range 7FAB8F0EF000 1000
[ 59.417140] change_pte_range 7FAB8F0F0000 1000
[ 59.417451] change_pte_range 7FAB8F0F1000 1000
[ 59.417769] change_pte_range 7FAB8F0F2000 1000
[ 59.418083] change_pte_range 7FAB8F0F3000 1000
[ 59.418396] change_pte_range 7FAB8F0F4000 1000
[ 59.418711] change_pte_range 7FAB8F0F5000 1000
[ 59.419028] change_pte_range 7FAB8F0F6000 1000
[ 59.419355] change_pte_range 7FAB8F0F7000 1000
[ 59.419665] change_pte_range 7FAB8F0F8000 1000
[ 59.419982] change_pte_range 7FAB8F0F9000 1000
[ 59.420298] change_pte_range 7FAB8F0FA000 1000
[ 59.420611] change_pte_range 7FAB8F0FB000 1000
[ 59.420934] change_pte_range 7FAB8F0FC000 1000
[ 59.421252] change_pte_range 7FAB8F0FD000 1000
[ 59.421566] change_pte_range 7FAB8F0FE000 1000
[ 59.421890] change_pte_range 7FAB8F0FF000 1000
[ 59.422214] change_pte_range 7FAB8F100000 1000
[ 59.422541] change_pte_range 7FAB8F101000 1000
[ 59.422869] change_pte_range 7FAB8F102000 1000
[ 59.423202] change_pte_range 7FAB8F103000 1000
[ 59.423562] change_pte_range 7FAB8F104000 1000
[ 59.424080] change_pte_range 7FAB8F105000 1000
[ 59.424524] change_pte_range 7FAB8F106000 1000
[ 59.424904] change_pte_range 7FAB8F107000 1000
[ 59.425234] change_pte_range 7FAB8F108000 1000
[ 59.425617] change_pte_range 7FAB8F109000 1000
[ 59.426026] change_pte_range 7FAB8F10A000 1000
[ 59.426430] change_pte_range 7FAB8F10B000 1000
[ 59.426844] change_pte_range 7FAB8F10C000 1000
[ 59.427227] change_pte_range 7FAB8F10D000 1000
[ 59.427633] change_pte_range 7FAB8F10E000 1000
[ 59.428049] change_pte_range 7FAB8F10F000 1000
[ 59.428456] change_pte_range 7FAB8F110000 1000
[ 59.428870] change_pte_range 7FAB8F111000 1000
[ 59.429235] change_pte_range 7FAB8F112000 1000
[ 59.429611] change_pte_range 7FAB8F113000 1000
[ 59.430010] change_pte_range 7FAB8F114000 1000
[ 59.430384] change_pte_range 7FAB8F115000 1000
[ 59.430776] change_pte_range 7FAB8F116000 1000
[ 59.431167] change_pte_range 7FAB8F117000 1000
[ 59.431556] change_pte_range 7FAB8F118000 1000
[ 59.431953] change_pte_range 7FAB8F119000 1000
[ 59.432330] change_pte_range 7FAB8F11A000 1000
[ 59.432704] change_pte_range 7FAB8F11B000 1000
[ 59.433071] change_pte_range 7FAB8F11C000 1000
[ 59.433443] change_pte_range 7FAB8F11D000 1000
[ 59.433826] change_pte_range 7FAB8F11E000 1000
[ 59.434222] change_pte_range 7FAB8F11F000 1000
[ 59.434594] change_pte_range 7FAB8F120000 1000
[ 59.435006] change_pte_range 7FAB8F121000 1000
[ 59.435418] change_pte_range 7FAB8F122000 1000
[ 59.435811] change_pte_range 7FAB8F123000 1000
[ 59.436191] change_pte_range 7FAB8F124000 1000
[ 59.436583] change_pte_range 7FAB8F125000 1000
[ 59.436972] change_pte_range 7FAB8F126000 1000
[ 59.437374] change_pte_range 7FAB8F127000 1000
[ 59.437774] change_pte_range 7FAB8F128000 1000
[ 59.438934] pagemap_scan_pmd_entry 7FAB8ED41000 7FAB8EE00000
[ 59.439443] pagemap_scan_pmd_entry 7FAB8EE00000 7FAB8F000000
[ 59.439955] pagemap_scan_pmd_entry 7FAB8F000000 7FAB8F129000
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-05-22 10:24 ` Muhammad Usama Anjum
@ 2023-05-22 11:26 ` Muhammad Usama Anjum
2023-05-23 19:43 ` Peter Xu
0 siblings, 1 reply; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-05-22 11:26 UTC (permalink / raw)
To: Peter Xu, linux-mm
Cc: Muhammad Usama Anjum, Paul Gofman, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Cyrill Gorcunov, Michał Mirosław,
Andrew Morton, Suren Baghdasaryan, Andrei Vagin, Alex Sierra,
Matthew Wilcox, Pasha Tatashin, Danylo Mocherniuk,
Axel Rasmussen, Gustavo A . R . Silva, David Hildenbrand,
Dan Williams, linux-kernel, Mike Rapoport, linux-fsdevel,
linux-kselftest, Greg KH, kernel, Nadav Amit
On 5/22/23 3:24 PM, Muhammad Usama Anjum wrote:
> On 4/26/23 7:13 PM, Peter Xu wrote:
>> Hi, Muhammad,
>>
>> On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote:
>>> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
>>>> +/* Supported flags */
>>>> +#define PM_SCAN_OP_GET (1 << 0)
>>>> +#define PM_SCAN_OP_WP (1 << 1)
>>> We have only these flag options available in PAGEMAP_SCAN IOCTL.
>>> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
>>> be specified as need. But PM_SCAN_OP_WP cannot be specified without
>>> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
>>> functionality which can be achieved by UFFDIO_WRITEPROTECT.)
>>>
>>> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
>>> vs
>>> 2) UFFDIO_WRITEPROTECT
>>>
>>> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
>>> getting really good performance which is comparable just like we are
>>> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
>>> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
>>> performance and behavior wise.
>>>
>>> I've got the results from someone else that UFFDIO_WRITEPROTECT block
>>> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
>>> as I don't have tests comparing them one-to-one.
>>>
>>> What are your thoughts about it? Have you thought about making
>>> UFFDIO_WRITEPROTECT perform better?
>>>
>>> I'm sorry to mention the word "performance" here. Actually we want better
>>> performance to emulate Windows syscall. That is why we are adding this
>>> functionality. So either we need to see what can be improved in
>>> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
>>> pagemap_ioctl?
>>
>> I'm fine if you want to add it back if it works for you. Though before
>> that, could you remind me why there can be a difference on performance?
> I've looked at the code again and I think I've found something. Lets look
> at exact performance numbers:
>
> I've run 2 different tests. In first test UFFDIO_WRITEPROTECT is being used
> for engaging WP. In second test PM_SCAN_OP_WP is being used. I've measured
> the average write time to the same memory which is being WP-ed and total
> time of execution of these APIs:
>
> **avg write time:**
> | No of pages | 2000 | 8192 | 100000 | 500000 |
> |------------------------|------|------|--------|--------|
> | UFFDIO_WRITEPROTECT | 2200 | 2300 | 4100 | 4200 |
> | PM_SCAN_OP_WP | 2000 | 2300 | 2500 | 2800 |
>
> **Execution time measured in rdtsc:**
> | No of pages | 2000 | 8192 | 100000 | 500000 |
> |------------------------|------|-------|--------|--------|
> | UFFDIO_WRITEPROTECT | 3200 | 14000 | 59000 | 58000 |
> | PM_SCAN_OP_WP | 1900 | 7000 | 38000 | 40000 |
>
> Avg write time for UFFDIO_WRITEPROTECT is 1.3 times slow. The execution
> time is 1.5 times slower in the case of UFFDIO_WRITEPROTECT. So
> UFFDIO_WRITEPROTECT is making writes slower to the pages and execution time
> is also slower.
>
> This proves that PM_SCAN_OP_WP is better than UFFDIO_WRITEPROTECT. Although
> PM_SCAN_OP_WP and UFFDIO_WRITEPROTECT have been implemented differently. We
> should have seen no difference in performance. But we have quite a lot of
> difference in performance here. PM_SCAN_OP_WP takes read mm lock, uses
> walk_page_range() to walk over pages which finds VMAs from address ranges
> to walk over them and pagemap_scan_pmd_entry() is handling most of the work
> including tlb flushing. UFFDIO_WRITEPROTECT is also taking the mm lock and
> iterating from all the different page directories until a pte is found and
> then flags are updated there and tlb is flushed for every pte.
>
> My next deduction would be that we are getting worse performance as we are
> flushing tlb for one page at a time in case of UFFDIO_WRITEPROTECT. While
> we flush tlb for 512 pages (moslty) at a time in case of PM_SCAN_OP_WP.
> I've just verified this by adding some logs to the change_pte_range() and
> pagemap_scan_pmd_entry(). Logs are attached. I've allocated memory of 1000
> pages and write-protected it with UFFDIO_WRITEPROTECT and PM_SCAN_OP_WP.
> The logs show that UFFDIO_WRITEPROTECT has flushed tlb 1000 times of size 1
> page each time. While PM_SCAN_OP_WP has flushed only 3 times of bigger
> sizes. I've learned over my last experience that tlb flush is very
> expensive. Probably this is what we need to improve if we don't want to add
> PM_SCAN_OP_WP?
>
> The UFFDIO_WRITEPROTECT uses change_pte_range() which is very generic
> function and I'm not sure if can try to not do tlb flushes if uffd_wp is
> true. We can try to do flush somewhere else and hopefully we should do only
> one flush if possible. It will not be so straight forward to move away from
> generic fundtion. Thoughts?
I've just tested this theory of not doing per pte flushes and only did one
flush on entire range in uffd_wp_range(). But it didn't improve the
situation either. I was wrong that tlb flushes may be the cause.
>
>
>> Thanks,
>>
>
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-05-22 11:26 ` Muhammad Usama Anjum
@ 2023-05-23 19:43 ` Peter Xu
2023-05-24 11:26 ` Muhammad Usama Anjum
0 siblings, 1 reply; 15+ messages in thread
From: Peter Xu @ 2023-05-23 19:43 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: linux-mm, Paul Gofman, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Cyrill Gorcunov, Michał Mirosław,
Andrew Morton, Suren Baghdasaryan, Andrei Vagin, Alex Sierra,
Matthew Wilcox, Pasha Tatashin, Danylo Mocherniuk,
Axel Rasmussen, Gustavo A . R . Silva, David Hildenbrand,
Dan Williams, linux-kernel, Mike Rapoport, linux-fsdevel,
linux-kselftest, Greg KH, kernel, Nadav Amit
Hi, Muhammad,
On Mon, May 22, 2023 at 04:26:07PM +0500, Muhammad Usama Anjum wrote:
> On 5/22/23 3:24 PM, Muhammad Usama Anjum wrote:
> > On 4/26/23 7:13 PM, Peter Xu wrote:
> >> Hi, Muhammad,
> >>
> >> On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote:
> >>> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
> >>>> +/* Supported flags */
> >>>> +#define PM_SCAN_OP_GET (1 << 0)
> >>>> +#define PM_SCAN_OP_WP (1 << 1)
> >>> We have only these flag options available in PAGEMAP_SCAN IOCTL.
> >>> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
> >>> be specified as need. But PM_SCAN_OP_WP cannot be specified without
> >>> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
> >>> functionality which can be achieved by UFFDIO_WRITEPROTECT.)
> >>>
> >>> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
> >>> vs
> >>> 2) UFFDIO_WRITEPROTECT
> >>>
> >>> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
> >>> getting really good performance which is comparable just like we are
> >>> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
> >>> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
> >>> performance and behavior wise.
> >>>
> >>> I've got the results from someone else that UFFDIO_WRITEPROTECT block
> >>> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
> >>> as I don't have tests comparing them one-to-one.
> >>>
> >>> What are your thoughts about it? Have you thought about making
> >>> UFFDIO_WRITEPROTECT perform better?
> >>>
> >>> I'm sorry to mention the word "performance" here. Actually we want better
> >>> performance to emulate Windows syscall. That is why we are adding this
> >>> functionality. So either we need to see what can be improved in
> >>> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
> >>> pagemap_ioctl?
> >>
> >> I'm fine if you want to add it back if it works for you. Though before
> >> that, could you remind me why there can be a difference on performance?
> > I've looked at the code again and I think I've found something. Lets look
> > at exact performance numbers:
> >
> > I've run 2 different tests. In first test UFFDIO_WRITEPROTECT is being used
> > for engaging WP. In second test PM_SCAN_OP_WP is being used. I've measured
> > the average write time to the same memory which is being WP-ed and total
> > time of execution of these APIs:
What is the steps of the test? Is it as simple as "writeprotect",
"unprotect", then write all pages in a single thread?
Is UFFDIO_WRITEPROTECT sent in one range covering all pages?
Maybe you can attach the test program here too.
> >
> > **avg write time:**
> > | No of pages | 2000 | 8192 | 100000 | 500000 |
> > |------------------------|------|------|--------|--------|
> > | UFFDIO_WRITEPROTECT | 2200 | 2300 | 4100 | 4200 |
> > | PM_SCAN_OP_WP | 2000 | 2300 | 2500 | 2800 |
> >
> > **Execution time measured in rdtsc:**
> > | No of pages | 2000 | 8192 | 100000 | 500000 |
> > |------------------------|------|-------|--------|--------|
> > | UFFDIO_WRITEPROTECT | 3200 | 14000 | 59000 | 58000 |
> > | PM_SCAN_OP_WP | 1900 | 7000 | 38000 | 40000 |
> >
> > Avg write time for UFFDIO_WRITEPROTECT is 1.3 times slow. The execution
> > time is 1.5 times slower in the case of UFFDIO_WRITEPROTECT. So
> > UFFDIO_WRITEPROTECT is making writes slower to the pages and execution time
> > is also slower.
> >
> > This proves that PM_SCAN_OP_WP is better than UFFDIO_WRITEPROTECT. Although
> > PM_SCAN_OP_WP and UFFDIO_WRITEPROTECT have been implemented differently. We
> > should have seen no difference in performance. But we have quite a lot of
> > difference in performance here. PM_SCAN_OP_WP takes read mm lock, uses
> > walk_page_range() to walk over pages which finds VMAs from address ranges
> > to walk over them and pagemap_scan_pmd_entry() is handling most of the work
> > including tlb flushing. UFFDIO_WRITEPROTECT is also taking the mm lock and
> > iterating from all the different page directories until a pte is found and
> > then flags are updated there and tlb is flushed for every pte.
> >
> > My next deduction would be that we are getting worse performance as we are
> > flushing tlb for one page at a time in case of UFFDIO_WRITEPROTECT. While
> > we flush tlb for 512 pages (moslty) at a time in case of PM_SCAN_OP_WP.
> > I've just verified this by adding some logs to the change_pte_range() and
> > pagemap_scan_pmd_entry(). Logs are attached. I've allocated memory of 1000
> > pages and write-protected it with UFFDIO_WRITEPROTECT and PM_SCAN_OP_WP.
> > The logs show that UFFDIO_WRITEPROTECT has flushed tlb 1000 times of size 1
> > page each time. While PM_SCAN_OP_WP has flushed only 3 times of bigger
> > sizes. I've learned over my last experience that tlb flush is very
> > expensive. Probably this is what we need to improve if we don't want to add
> > PM_SCAN_OP_WP?
> >
> > The UFFDIO_WRITEPROTECT uses change_pte_range() which is very generic
> > function and I'm not sure if can try to not do tlb flushes if uffd_wp is
> > true. We can try to do flush somewhere else and hopefully we should do only
> > one flush if possible. It will not be so straight forward to move away from
> > generic fundtion. Thoughts?
> I've just tested this theory of not doing per pte flushes and only did one
> flush on entire range in uffd_wp_range(). But it didn't improve the
> situation either. I was wrong that tlb flushes may be the cause.
I had a feeling that you were trapping tlb_flush_pte_range(), which is
actually not really sending any TLB flushes but updating mmu_gather object
for the addr range for future invalidations.
That's probably why it didn't show an effect when you comment it out.
I am not sure whether the wr-protect path difference can be caused by the
arch hooks, namely arch_enter_lazy_mmu_mode() / arch_leave_lazy_mmu_mode().
On x86 I saw that it's actually hooked onto some PV calls. I had a feeling
that this is for optimization only, but maybe it's still a good idea you
also take that into your new code:
static inline void arch_enter_lazy_mmu_mode(void)
{
PVOP_VCALL0(mmu.lazy_mode.enter);
}
The other thing is I think you're flushing tlb outside pgtable lock in your
new code. IIUC that's racy, see:
commit 6ce64428d62026a10cb5d80138ff2f90cc21d367
Author: Nadav Amit <namit@vmware.com>
Date: Fri Mar 12 21:08:17 2021 -0800
mm/userfaultfd: fix memory corruption due to writeprotect
So you may want to put it at least into pgtable lock critical section, or
IIUC you can also do inc_tlb_flush_pending() then dec_tlb_flush_pending()
just like __tlb_gather_mmu(), to make sure do_wp_page() will properly flush
the page when unluckily hit some of the page.
That's also the spot (the flush_tlb_page() in 6ce64428d) that made me think
on whether it caused the slowness on writting to those pages. But it
really depends on your test program, e.g. if it's a single threaded I don't
think it'll trigger because when writting mm_tlb_flush_pending() should
start to return 0 already, so the tlb should logically not be needed. If
you want maybe you can double check that.
So in short, I had a feeling that the new PM_SCAN_OP_WP just misses
something here and there so it's faster - it means even if it's faster it
may also be prone to race conditions etc so we'd better figure it out..
Thanks,
--
Peter Xu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-05-23 19:43 ` Peter Xu
@ 2023-05-24 11:26 ` Muhammad Usama Anjum
2023-05-24 13:55 ` Peter Xu
0 siblings, 1 reply; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-05-24 11:26 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, linux-mm, Paul Gofman, Alexander Viro,
Shuah Khan, Christian Brauner, Yang Shi, Vlastimil Babka,
Liam R . Howlett, Yun Zhou, Cyrill Gorcunov,
Michał Mirosław, Andrew Morton, Suren Baghdasaryan,
Andrei Vagin, Alex Sierra, Matthew Wilcox, Pasha Tatashin,
Danylo Mocherniuk, Axel Rasmussen, Gustavo A . R . Silva,
David Hildenbrand, Dan Williams, linux-kernel, Mike Rapoport,
linux-fsdevel, linux-kselftest, Greg KH, kernel, Nadav Amit
[-- Attachment #1: Type: text/plain, Size: 8901 bytes --]
On 5/24/23 12:43 AM, Peter Xu wrote:
> Hi, Muhammad,
>
> On Mon, May 22, 2023 at 04:26:07PM +0500, Muhammad Usama Anjum wrote:
>> On 5/22/23 3:24 PM, Muhammad Usama Anjum wrote:
>>> On 4/26/23 7:13 PM, Peter Xu wrote:
>>>> Hi, Muhammad,
>>>>
>>>> On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote:
>>>>> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
>>>>>> +/* Supported flags */
>>>>>> +#define PM_SCAN_OP_GET (1 << 0)
>>>>>> +#define PM_SCAN_OP_WP (1 << 1)
>>>>> We have only these flag options available in PAGEMAP_SCAN IOCTL.
>>>>> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
>>>>> be specified as need. But PM_SCAN_OP_WP cannot be specified without
>>>>> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
>>>>> functionality which can be achieved by UFFDIO_WRITEPROTECT.)
>>>>>
>>>>> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
>>>>> vs
>>>>> 2) UFFDIO_WRITEPROTECT
>>>>>
>>>>> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
>>>>> getting really good performance which is comparable just like we are
>>>>> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
>>>>> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
>>>>> performance and behavior wise.
>>>>>
>>>>> I've got the results from someone else that UFFDIO_WRITEPROTECT block
>>>>> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
>>>>> as I don't have tests comparing them one-to-one.
>>>>>
>>>>> What are your thoughts about it? Have you thought about making
>>>>> UFFDIO_WRITEPROTECT perform better?
>>>>>
>>>>> I'm sorry to mention the word "performance" here. Actually we want better
>>>>> performance to emulate Windows syscall. That is why we are adding this
>>>>> functionality. So either we need to see what can be improved in
>>>>> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
>>>>> pagemap_ioctl?
>>>>
>>>> I'm fine if you want to add it back if it works for you. Though before
>>>> that, could you remind me why there can be a difference on performance?
>>> I've looked at the code again and I think I've found something. Lets look
>>> at exact performance numbers:
>>>
>>> I've run 2 different tests. In first test UFFDIO_WRITEPROTECT is being used
>>> for engaging WP. In second test PM_SCAN_OP_WP is being used. I've measured
>>> the average write time to the same memory which is being WP-ed and total
>>> time of execution of these APIs:
>
> What is the steps of the test? Is it as simple as "writeprotect",
> "unprotect", then write all pages in a single thread?
>
> Is UFFDIO_WRITEPROTECT sent in one range covering all pages?
>
> Maybe you can attach the test program here too.
I'd not attached the test earlier as I thought that you wouldn't be
interested in running the test. I've attached it now. The test has multiple
threads where one thread tries to get status of flags and reset them, while
other threads write to that memory. In main(), we call the pagemap_scan
ioctl to get status of flags and reset the memory area as well. While in N
threads, the memory is written.
I usually run the test by following where memory area is of 100000 * pages:
./win2_linux 8 100000 1 1 0
I'm running tests on real hardware. The results are pretty consistent. I'm
also testing only on x86_64. PM_SCAN_OP_WP wins every time as compared to
UFFDIO_WRITEPROTECT.
The PM_SCAN_OP_WP op doesn't work exclusively on v15. So please find the
updated WIP code here:
https://gitlab.collabora.com/usama.anjum/linux-mainline/-/commits/memwatchv16/
>
>>>
>>> **avg write time:**
>>> | No of pages | 2000 | 8192 | 100000 | 500000 |
>>> |------------------------|------|------|--------|--------|
>>> | UFFDIO_WRITEPROTECT | 2200 | 2300 | 4100 | 4200 |
>>> | PM_SCAN_OP_WP | 2000 | 2300 | 2500 | 2800 |
>>>
>>> **Execution time measured in rdtsc:**
>>> | No of pages | 2000 | 8192 | 100000 | 500000 |
>>> |------------------------|------|-------|--------|--------|
>>> | UFFDIO_WRITEPROTECT | 3200 | 14000 | 59000 | 58000 |
>>> | PM_SCAN_OP_WP | 1900 | 7000 | 38000 | 40000 |
>>>
>>> Avg write time for UFFDIO_WRITEPROTECT is 1.3 times slow. The execution
>>> time is 1.5 times slower in the case of UFFDIO_WRITEPROTECT. So
>>> UFFDIO_WRITEPROTECT is making writes slower to the pages and execution time
>>> is also slower.
>>>
>>> This proves that PM_SCAN_OP_WP is better than UFFDIO_WRITEPROTECT. Although
>>> PM_SCAN_OP_WP and UFFDIO_WRITEPROTECT have been implemented differently. We
>>> should have seen no difference in performance. But we have quite a lot of
>>> difference in performance here. PM_SCAN_OP_WP takes read mm lock, uses
>>> walk_page_range() to walk over pages which finds VMAs from address ranges
>>> to walk over them and pagemap_scan_pmd_entry() is handling most of the work
>>> including tlb flushing. UFFDIO_WRITEPROTECT is also taking the mm lock and
>>> iterating from all the different page directories until a pte is found and
>>> then flags are updated there and tlb is flushed for every pte.
>>>
>>> My next deduction would be that we are getting worse performance as we are
>>> flushing tlb for one page at a time in case of UFFDIO_WRITEPROTECT. While
>>> we flush tlb for 512 pages (moslty) at a time in case of PM_SCAN_OP_WP.
>>> I've just verified this by adding some logs to the change_pte_range() and
>>> pagemap_scan_pmd_entry(). Logs are attached. I've allocated memory of 1000
>>> pages and write-protected it with UFFDIO_WRITEPROTECT and PM_SCAN_OP_WP.
>>> The logs show that UFFDIO_WRITEPROTECT has flushed tlb 1000 times of size 1
>>> page each time. While PM_SCAN_OP_WP has flushed only 3 times of bigger
>>> sizes. I've learned over my last experience that tlb flush is very
>>> expensive. Probably this is what we need to improve if we don't want to add
>>> PM_SCAN_OP_WP?
>>>
>>> The UFFDIO_WRITEPROTECT uses change_pte_range() which is very generic
>>> function and I'm not sure if can try to not do tlb flushes if uffd_wp is
>>> true. We can try to do flush somewhere else and hopefully we should do only
>>> one flush if possible. It will not be so straight forward to move away from
>>> generic fundtion. Thoughts?
>> I've just tested this theory of not doing per pte flushes and only did one
>> flush on entire range in uffd_wp_range(). But it didn't improve the
>> situation either. I was wrong that tlb flushes may be the cause.
>
> I had a feeling that you were trapping tlb_flush_pte_range(), which is
> actually not really sending any TLB flushes but updating mmu_gather object
> for the addr range for future invalidations.
>
> That's probably why it didn't show an effect when you comment it out.
Yeah, probably.
>
> I am not sure whether the wr-protect path difference can be caused by the
> arch hooks, namely arch_enter_lazy_mmu_mode() / arch_leave_lazy_mmu_mode().
>
> On x86 I saw that it's actually hooked onto some PV calls. I had a feeling
> that this is for optimization only, but maybe it's still a good idea you
> also take that into your new code:
>
> static inline void arch_enter_lazy_mmu_mode(void)
> {
> PVOP_VCALL0(mmu.lazy_mode.enter);
> }
I've just looked into it. It isn't making any difference. But I think I
should include it in the code. It must be helpful for hypervisors etc.
>
> The other thing is I think you're flushing tlb outside pgtable lock in your
> new code. IIUC that's racy, see:
>
> commit 6ce64428d62026a10cb5d80138ff2f90cc21d367
> Author: Nadav Amit <namit@vmware.com>
> Date: Fri Mar 12 21:08:17 2021 -0800
>
> mm/userfaultfd: fix memory corruption due to writeprotect
>
> So you may want to put it at least into pgtable lock critical section, or
> IIUC you can also do inc_tlb_flush_pending() then dec_tlb_flush_pending()
> just like __tlb_gather_mmu(), to make sure do_wp_page() will properly flush
> the page when unluckily hit some of the page.
Good point. I'll release page table lock after tlb flushing. I've just
added it to next WIP v16.
>
> That's also the spot (the flush_tlb_page() in 6ce64428d) that made me think
> on whether it caused the slowness on writting to those pages. But it
> really depends on your test program, e.g. if it's a single threaded I don't
> think it'll trigger because when writting mm_tlb_flush_pending() should
> start to return 0 already, so the tlb should logically not be needed. If
> you want maybe you can double check that.
>
> So in short, I had a feeling that the new PM_SCAN_OP_WP just misses
> something here and there so it's faster - it means even if it's faster it
> may also be prone to race conditions etc so we'd better figure it out..
The test program is multi-threaded. The performance number cannot be
reproduced with single-threaded application.
>
> Thanks,
>
--
BR,
Muhammad Usama Anjum
[-- Attachment #2: win2_linux.c --]
[-- Type: text/x-csrc, Size: 13016 bytes --]
#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <sys/mman.h>
#define _GNU_SOURCE
#define _OPEN_THREADS
#include <pthread.h>
#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <sys/ioctl.h>
#include <linux/userfaultfd.h>
#include <sys/ioctl.h>
#define LOG(format, ...) {printf("%lx:%s: " format, getpid(), __func__ __VA_OPT__(,)__VA_ARGS__);}
#define MAX_THREAD_COUNT 64
#define PAGE_SIZE 0x1000
#define TEST_TIME (3.0 * 1000.0) * 10
static bool finish;
static int nthreads;
static volatile long long raw_writes_count, writes_time;
static char *mem;
static bool random_access, read_reset;
#if defined(__i386__)
static __inline__ unsigned long long rdtsc(void)
{
unsigned long long int x;
__asm__ volatile (".byte 0x0f, 0x31" : "=A" (x));
return x;
}
#elif defined(__x86_64__)
static __inline__ unsigned long long rdtsc(void)
{
unsigned hi, lo;
__asm__ __volatile__ ("rdtsc" : "=a"(lo), "=d"(hi));
return ( (unsigned long long)lo)|( ((unsigned long long)hi)<<32 );
}
#endif
struct thread_info
{
volatile void *mem;
size_t npages;
};
void *thread_proc(void *data)
{
struct thread_info *info = data;
volatile unsigned char *m = info->mem;
size_t page;
unsigned long long t1, t2;
page = 0;
while (!finish)
{
t1 = rdtsc();
if (random_access)
page = rand() % info->npages;
else
page = (page + 1) % info->npages;
++*(volatile unsigned int *)(m + page * PAGE_SIZE);
__atomic_add_fetch (&raw_writes_count, 1, __ATOMIC_RELAXED);
t2 = rdtsc();
__atomic_add_fetch (&writes_time, t2 - t1, __ATOMIC_RELAXED);
}
return 0;
}
double curr_time_ms(void)
{
struct timespec current_time;
clock_gettime(CLOCK_MONOTONIC, ¤t_time);
return current_time.tv_sec * 1000 +
current_time.tv_nsec / 1000000;
}
static double rdtsc_c;
//static double inline mcs_from_rdtsc_avg(void *tsc, unsigned long long count)
//{
// return (unsigned int)tsc * rdtsc_c / count;
//}
////////////////////////////////////////////////////////////////////////////////////////////////
#define UFFD_FEATURE_WP_UNPOPULATED (1<<13)
#define UFFD_FEATURE_WP_ASYNC (1<<14)
#ifndef PAGEMAP_SCAN
/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
#define PAGE_IS_WRITTEN (1 << 0)
#define PAGE_IS_FILE (1 << 1)
#define PAGE_IS_PRESENT (1 << 2)
#define PAGE_IS_SWAPPED (1 << 3)
/*
* struct page_region - Page region with bitmap flags
* @start: Start of the region
* @len: Length of the region
* bitmap: Bits sets for the region
*/
struct page_region {
unsigned long long start;
unsigned long long len;
unsigned long long bitmap;
};
/*
* struct pm_scan_arg - Pagemap ioctl argument
* @size: Size of the structure
* @flags: Flags for the IOCTL
* @start: Starting address of the region
* @len: Length of the region (All the pages in this length are included)
* @vec: Address of page_region struct array for output
* @vec_len: Length of the page_region struct array
* @max_pages: Optional max return pages
* @required_mask: Required mask - All of these bits have to be set in the PTE
* @anyof_mask: Any mask - Any of these bits are set in the PTE
* @excluded_mask: Exclude mask - None of these bits are set in the PTE
* @return_mask: Bits that are to be reported in page_region
*/
struct pm_scan_arg {
unsigned long long size;
unsigned long long flags;
unsigned long long start;
unsigned long long len;
unsigned long long vec;
unsigned long long vec_len;
unsigned long long max_pages;
unsigned long long required_mask;
unsigned long long anyof_mask;
unsigned long long excluded_mask;
unsigned long long return_mask;
};
#define PM_SCAN_OP_GET (1 << 0)
#define PM_SCAN_OP_WP (1 << 1)
/* Pagemap ioctl */
#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
#endif
#define __NR_userfaultfd 323
#define PAGEMAP "/proc/self/pagemap"
int pagemap_fd;
int uffd;
static long pagemap_ioctl(void *start, int len, void *vec, int vec_len, int flag,
int max_pages, long required_mask, long anyof_mask, long excluded_mask,
long return_mask)
{
struct pm_scan_arg arg;
arg.start = (uintptr_t)start;
arg.len = len;
arg.vec = (uintptr_t)vec;
arg.vec_len = vec_len;
arg.flags = flag;
arg.size = sizeof(struct pm_scan_arg);
arg.max_pages = max_pages;
arg.required_mask = required_mask;
arg.anyof_mask = anyof_mask;
arg.excluded_mask = excluded_mask;
arg.return_mask = return_mask;
return ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
}
int init_uffd(void)
{
struct uffdio_api uffdio_api;
uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
if (uffd == -1)
exit(1);
uffdio_api.api = UFFD_API;
uffdio_api.features = UFFD_FEATURE_WP_UNPOPULATED | UFFD_FEATURE_WP_ASYNC |
UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
if (ioctl(uffd, UFFDIO_API, &uffdio_api))
exit(1);
if (!(uffdio_api.api & UFFDIO_REGISTER_MODE_WP) ||
!(uffdio_api.features & UFFD_FEATURE_WP_UNPOPULATED) ||
!(uffdio_api.features & UFFD_FEATURE_WP_ASYNC) ||
!(uffdio_api.features & UFFD_FEATURE_WP_HUGETLBFS_SHMEM))
exit(1);
return 0;
}
int wp_init(void *lpBaseAddress, int dwRegionSize)
{
struct uffdio_register uffdio_register;
struct uffdio_writeprotect wp;
uffdio_register.range.start = (unsigned long)lpBaseAddress;
uffdio_register.range.len = dwRegionSize;
uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
exit(1);
if (!(uffdio_register.ioctls & UFFDIO_WRITEPROTECT))
exit(1);
wp.range.start = (unsigned long)lpBaseAddress;
wp.range.len = dwRegionSize;
wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp))
exit(1);
return 0;
}
int wp_free(void *lpBaseAddress, int dwRegionSize)
{
struct uffdio_register uffdio_register;
uffdio_register.range.start = (unsigned long)lpBaseAddress;
uffdio_register.range.len = dwRegionSize;
uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
if (ioctl(uffd, UFFDIO_UNREGISTER, &uffdio_register.range))
exit(1);
return 0;
}
int wp_addr_range(void *lpBaseAddress, int dwRegionSize)
{
struct uffdio_writeprotect wp;
wp.range.start = (unsigned long)lpBaseAddress;
wp.range.len = dwRegionSize;
wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp))
exit(1);
return 0;
}
//int wp_addr_range_(void *lpBaseAddress, int dwRegionSize)
//{
// struct page_region *vec;
// int ret;
//
// vec = malloc(sizeof(struct page_region) * dwRegionSize);
//
// ret = pagemap_ioctl(lpBaseAddress, dwRegionSize, vec, dwRegionSize, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
// 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
//
// if (ret < 0)
// exit(1);
//
// free(vec);
// return 0;
//}
int wp_addr_range__(void *lpBaseAddress, int dwRegionSize)
{
int ret;
ret = pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, PM_SCAN_OP_WP,
0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
if (ret < 0)
exit(1);
return 0;
}
#define WRITE_WATCH_FLAG_RESET 1
struct page_region *buf;
int GetWriteWatch(int flag, char *start, long len, long **vec, unsigned int *ww_count,
unsigned int *granularity)
{
struct pm_scan_arg arg;
int i, j, ind = 0, ret;
arg.start = (uintptr_t)start;
arg.len = len;
arg.vec = (uintptr_t)buf; //(uintptr_t)vec;
arg.vec_len = *ww_count;
arg.flags = PM_SCAN_OP_GET;
if (flag == WRITE_WATCH_FLAG_RESET)
arg.flags |= PM_SCAN_OP_WP;
arg.size = sizeof(struct pm_scan_arg);
arg.max_pages = *ww_count;
arg.required_mask = PAGE_IS_WRITTEN;
arg.anyof_mask = 0;
arg.excluded_mask = 0;
arg.return_mask = PAGE_IS_WRITTEN;
if (granularity)
*granularity = 4096;
ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
if (ret < 0)
goto free_and_return;
*ww_count = 0;
for (i = 0; i < ret; i++) {
*ww_count += buf[i].len;
for (j = 0; j < buf[i].len; j++)
vec[ind++] = (long int *)(buf[i].start + j * 4096);
}
ret = 0;
free_and_return:
return ret;
}
//#define USE_PAGEMAP_GET_WP
//#define USE_PAGEMAP_WP
unsigned long long reset_total;
int ResetWriteWatch(void *lpBaseAddress, int dwRegionSize)
{
unsigned long long t = rdtsc();
int ret;
#ifdef USE_PAGEMAP_GET_WP
ret = wp_addr_range_(lpBaseAddress, dwRegionSize);
#elif defined USE_PAGEMAP_WP
ret = wp_addr_range__(lpBaseAddress, dwRegionSize);
#else
ret = wp_addr_range(lpBaseAddress, dwRegionSize);
#endif
reset_total += rdtsc() - t;
return ret;
}
int main(int argc, char *argv[])
{
unsigned int ww_count, ww_total, cycle_count, fault_count;
struct thread_info info[MAX_THREAD_COUNT];
unsigned long long t1, t2, rdtsc_start, rdtsc_end;
double start, curr, cycle_start;
static long **ww_addr;
long long writes_count;
unsigned long long wwread_time;
double rw_delay_ms;
unsigned int granularity;
unsigned int old_prot;
unsigned int count;
unsigned int i;
int get_count;
size_t npages;
pagemap_fd = open(PAGEMAP, O_RDWR);
if (pagemap_fd < 0) {
perror("pagemapfd");
return -EINVAL;
}
if (init_uffd())
return -1;
if (argc < 6) {
puts("Usage: win.exe <nthreads> <npages> <watch_delay_ms> <random_access> <read_reset>\n");
return -1;
}
nthreads = atoi(argv[1]);
if (nthreads > MAX_THREAD_COUNT) {
LOG("Maximum of %u threads supported.\n", MAX_THREAD_COUNT);
return -1;
}
npages = atoi(argv[2]);
if (npages < nthreads || npages % nthreads) {
LOG("npages should be > nthreads and evenly divisible by nthreads.\n");
return -1;
}
rw_delay_ms = atof(argv[3]);
random_access = atoi(argv[4]);
read_reset = atoi(argv[5]);
ww_addr = malloc(sizeof(*ww_addr) * npages);
mem = mmap(NULL, npages * PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
if (!mem) {
LOG("Failed to allocate virtual memory.\n");
return -1;
}
wp_init(mem, npages * PAGE_SIZE);
wp_addr_range(mem, npages * PAGE_SIZE);
mem[0] = 0x28;
mem[0x1000] = 0x29;
buf = malloc(100000 * sizeof(struct page_region));
ww_count = 100;
if (GetWriteWatch(0, mem, 0x1000 * npages, ww_addr, &ww_count, &granularity)) {
LOG("GetWriteWatch() failed, GetLastError() %lu.\n", errno);
return -1;
}
// LOG("count %llu, %p, %p.\n", ww_count, mem, ww_addr[0]);
mem = mmap(NULL, npages * 0x1000, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
if (!mem) {
perror("Error commit");
return -1;
}
wp_init(mem, npages * 0x1000);
wp_addr_range(mem, npages * 0x1000);
ww_count = 100;
if (GetWriteWatch(0, mem, 0x1000 * npages, ww_addr, &ww_count, &granularity)) {
LOG("GetWriteWatch() failed, GetLastError() %lu.\n", errno);
return -1;
}
// LOG("count %llu, %p, %p.\n", ww_count, mem, ww_addr[0]);
memset(mem, 0, npages * PAGE_SIZE);
ResetWriteWatch(mem, npages * PAGE_SIZE);
ww_count = 100;
if (GetWriteWatch(0, mem, 0x1000 * npages, ww_addr, &ww_count, &granularity)) {
LOG("GetWriteWatch() failed, GetLastError() %lu.\n", errno);
return -1;
}
// LOG("read bytes %lu, count after read %llu, %p, %p.\n", count, ww_count, mem, ww_addr[0]);
memset(mem, 0, npages * PAGE_SIZE);
ResetWriteWatch(mem, npages * PAGE_SIZE);
pthread_t th;
for (i = 0; i < nthreads; ++i) {
info[i].mem = mem + 0x1000 * i * npages / nthreads;
info[i].npages = npages / nthreads;
pthread_create(&th, NULL, thread_proc, &info[i]);
}
get_count = npages;
wwread_time = 0;
curr = start = curr_time_ms();
ww_total = 0;
cycle_count = 0;
rdtsc_start = rdtsc();
while (curr - start < TEST_TIME) {
char *addr, *end;
cycle_start = curr;
t1 = rdtsc();
addr = mem;
end = mem + npages * PAGE_SIZE;
// LOG("cycle %I64u.\n", cycle_count);
while (addr < end) {
ww_count = get_count;
if (GetWriteWatch(read_reset ? WRITE_WATCH_FLAG_RESET : 0, addr, end - addr,
ww_addr, &ww_count, &granularity)) {
LOG("GetWriteWatch() failed, GetLastError() %lu.\n", errno);
return -1;
}
ww_total += ww_count;
if (ww_count < get_count)
break;
addr = (char *)ww_addr[ww_count - 1] + 0x1000;
LOG("addr %p, end %p, ww_count %I64u.\n", addr, end, ww_count);
}
if (!read_reset)
ResetWriteWatch(mem, end - mem);
t2 = rdtsc();
wwread_time += t2 - t1;
curr = curr_time_ms();
while (curr - start < TEST_TIME && curr - cycle_start < rw_delay_ms) {
sched_yield();
curr = curr_time_ms();
}
++cycle_count;
}
// rdtsc_end = rdtsc();
writes_count = raw_writes_count;
finish = true;
// rdtsc_c = 1000.0 * (curr - start) / (rdtsc_end - rdtsc_start);
// LOG("rdtsc_c %lf.\n", rdtsc_c);
sleep(1);
LOG("Elapsed %.1lf, cycle_count %llu, writes_count %lld, writes watched %llu.\n",
curr - start, cycle_count, writes_count, ww_total);
LOG("writes per thread * msec %.3lf, avg. write time %.1lf, GetWriteWatch() avg %.1lf.\n",
writes_count / (TEST_TIME * nthreads), (float)writes_time/writes_count,
wwread_time / cycle_count);
printf("ResetWriteWatch() time rdtsc --> %llu M\n", reset_total/1000000);
free(buf);
return 0;
}
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-05-24 11:26 ` Muhammad Usama Anjum
@ 2023-05-24 13:55 ` Peter Xu
2023-05-24 14:16 ` Muhammad Usama Anjum
0 siblings, 1 reply; 15+ messages in thread
From: Peter Xu @ 2023-05-24 13:55 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: linux-mm, Paul Gofman, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Cyrill Gorcunov, Michał Mirosław,
Andrew Morton, Suren Baghdasaryan, Andrei Vagin, Alex Sierra,
Matthew Wilcox, Pasha Tatashin, Danylo Mocherniuk,
Axel Rasmussen, Gustavo A . R . Silva, David Hildenbrand,
Dan Williams, linux-kernel, Mike Rapoport, linux-fsdevel,
linux-kselftest, Greg KH, kernel, Nadav Amit
On Wed, May 24, 2023 at 04:26:33PM +0500, Muhammad Usama Anjum wrote:
> On 5/24/23 12:43 AM, Peter Xu wrote:
> > Hi, Muhammad,
> >
> > On Mon, May 22, 2023 at 04:26:07PM +0500, Muhammad Usama Anjum wrote:
> >> On 5/22/23 3:24 PM, Muhammad Usama Anjum wrote:
> >>> On 4/26/23 7:13 PM, Peter Xu wrote:
> >>>> Hi, Muhammad,
> >>>>
> >>>> On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote:
> >>>>> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote:
> >>>>>> +/* Supported flags */
> >>>>>> +#define PM_SCAN_OP_GET (1 << 0)
> >>>>>> +#define PM_SCAN_OP_WP (1 << 1)
> >>>>> We have only these flag options available in PAGEMAP_SCAN IOCTL.
> >>>>> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can
> >>>>> be specified as need. But PM_SCAN_OP_WP cannot be specified without
> >>>>> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate
> >>>>> functionality which can be achieved by UFFDIO_WRITEPROTECT.)
> >>>>>
> >>>>> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP
> >>>>> vs
> >>>>> 2) UFFDIO_WRITEPROTECT
> >>>>>
> >>>>> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are
> >>>>> getting really good performance which is comparable just like we are
> >>>>> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp,
> >>>>> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT
> >>>>> performance and behavior wise.
> >>>>>
> >>>>> I've got the results from someone else that UFFDIO_WRITEPROTECT block
> >>>>> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this
> >>>>> as I don't have tests comparing them one-to-one.
> >>>>>
> >>>>> What are your thoughts about it? Have you thought about making
> >>>>> UFFDIO_WRITEPROTECT perform better?
> >>>>>
> >>>>> I'm sorry to mention the word "performance" here. Actually we want better
> >>>>> performance to emulate Windows syscall. That is why we are adding this
> >>>>> functionality. So either we need to see what can be improved in
> >>>>> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in
> >>>>> pagemap_ioctl?
> >>>>
> >>>> I'm fine if you want to add it back if it works for you. Though before
> >>>> that, could you remind me why there can be a difference on performance?
> >>> I've looked at the code again and I think I've found something. Lets look
> >>> at exact performance numbers:
> >>>
> >>> I've run 2 different tests. In first test UFFDIO_WRITEPROTECT is being used
> >>> for engaging WP. In second test PM_SCAN_OP_WP is being used. I've measured
> >>> the average write time to the same memory which is being WP-ed and total
> >>> time of execution of these APIs:
> >
> > What is the steps of the test? Is it as simple as "writeprotect",
> > "unprotect", then write all pages in a single thread?
> >
> > Is UFFDIO_WRITEPROTECT sent in one range covering all pages?
> >
> > Maybe you can attach the test program here too.
>
> I'd not attached the test earlier as I thought that you wouldn't be
> interested in running the test. I've attached it now. The test has multiple
Thanks. No plan to run it, just to make sure I understand why such a
difference.
> threads where one thread tries to get status of flags and reset them, while
> other threads write to that memory. In main(), we call the pagemap_scan
> ioctl to get status of flags and reset the memory area as well. While in N
> threads, the memory is written.
>
> I usually run the test by following where memory area is of 100000 * pages:
> ./win2_linux 8 100000 1 1 0
>
> I'm running tests on real hardware. The results are pretty consistent. I'm
> also testing only on x86_64. PM_SCAN_OP_WP wins every time as compared to
> UFFDIO_WRITEPROTECT.
If it's multi-threaded test especially when the ioctl runs together with
the writers, then I'd assume it's caused by writers frequently need to
flush tlb (when writes during UFFDIO_WRITEPROTECT), the flush target could
potentially also include the core running the main thread who is also
trying to reprotect because they run on the same mm.
This makes me think that your current test case probably is the worst case
of Nadav's patch 6ce64428d6 because (1) the UFFDIO_WRITEPROTECT covers a
super large range, and (2) there're a _lot_ of concurrent writers during
the ioctl, so all of them will need to trigger a tlb flush, and that tlb
flush will further slow down the ioctl sender.
While I think that's the optimal case sometimes, of having minimum tlb
flush on the ioctl(UFFDIO_WRITEPROTECT), so maybe it makes sense somewhere
else where concurrent writers are not that much. I'll need to rethink a bit
on all these to find out whether we can have a good way for both..
For now, if your workload is mostly exactly like your test case, maybe you
can have your pagemap version of WP-only op there, making sure tlb flush is
within the pgtable lock critical section (so you should be safe even
without Nadav's patch). If so, I'd appreciate you can add some comment
somewhere about such difference of using pagemap WP-only and
ioctl(UFFDIO_WRITEPROTECT), though. In short, functional-wise they should
be the same, but trivial detail difference on performance as TBD (maybe one
day we can have a good approach for all and make them aligned again, but
maybe that also doesn't need to block your work).
--
Peter Xu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs
2023-05-24 13:55 ` Peter Xu
@ 2023-05-24 14:16 ` Muhammad Usama Anjum
0 siblings, 0 replies; 15+ messages in thread
From: Muhammad Usama Anjum @ 2023-05-24 14:16 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, linux-mm, Paul Gofman, Alexander Viro,
Shuah Khan, Christian Brauner, Yang Shi, Vlastimil Babka,
Liam R . Howlett, Yun Zhou, Cyrill Gorcunov,
Michał Mirosław, Andrew Morton, Suren Baghdasaryan,
Andrei Vagin, Alex Sierra, Matthew Wilcox, Pasha Tatashin,
Danylo Mocherniuk, Axel Rasmussen, Gustavo A . R . Silva,
David Hildenbrand, Dan Williams, linux-kernel, Mike Rapoport,
linux-fsdevel, linux-kselftest, Greg KH, kernel, Nadav Amit
On 5/24/23 6:55 PM, Peter Xu wrote:
...
>>> What is the steps of the test? Is it as simple as "writeprotect",
>>> "unprotect", then write all pages in a single thread?
>>>
>>> Is UFFDIO_WRITEPROTECT sent in one range covering all pages?
>>>
>>> Maybe you can attach the test program here too.
>>
>> I'd not attached the test earlier as I thought that you wouldn't be
>> interested in running the test. I've attached it now. The test has multiple
>
> Thanks. No plan to run it, just to make sure I understand why such a
> difference.
>
>> threads where one thread tries to get status of flags and reset them, while
>> other threads write to that memory. In main(), we call the pagemap_scan
>> ioctl to get status of flags and reset the memory area as well. While in N
>> threads, the memory is written.
>>
>> I usually run the test by following where memory area is of 100000 * pages:
>> ./win2_linux 8 100000 1 1 0
>>
>> I'm running tests on real hardware. The results are pretty consistent. I'm
>> also testing only on x86_64. PM_SCAN_OP_WP wins every time as compared to
>> UFFDIO_WRITEPROTECT.
>
> If it's multi-threaded test especially when the ioctl runs together with
> the writers, then I'd assume it's caused by writers frequently need to
> flush tlb (when writes during UFFDIO_WRITEPROTECT), the flush target could
> potentially also include the core running the main thread who is also
> trying to reprotect because they run on the same mm.
>
> This makes me think that your current test case probably is the worst case
> of Nadav's patch 6ce64428d6 because (1) the UFFDIO_WRITEPROTECT covers a
> super large range, and (2) there're a _lot_ of concurrent writers during
> the ioctl, so all of them will need to trigger a tlb flush, and that tlb
> flush will further slow down the ioctl sender.
>
> While I think that's the optimal case sometimes, of having minimum tlb
> flush on the ioctl(UFFDIO_WRITEPROTECT), so maybe it makes sense somewhere
> else where concurrent writers are not that much. I'll need to rethink a bit
> on all these to find out whether we can have a good way for both..
>
> For now, if your workload is mostly exactly like your test case, maybe you
> can have your pagemap version of WP-only op there, making sure tlb flush is
> within the pgtable lock critical section (so you should be safe even
> without Nadav's patch). If so, I'd appreciate you can add some comment
> somewhere about such difference of using pagemap WP-only and
> ioctl(UFFDIO_WRITEPROTECT), though. In short, functional-wise they should
> be the same, but trivial detail difference on performance as TBD (maybe one
> day we can have a good approach for all and make them aligned again, but
> maybe that also doesn't need to block your work).
Thank you for understanding what I've been trying to convey. We are going
to translate Windows syscall to this new ioctl. So it is very difficult to
find out the exact use cases as application must be using this syscall in
several different ways. There is one thing for sure is that we want to get
best performance possible which we are getting by adding WP-only. I'll add
it and send v16. I think that we are almost there.
>
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2023-05-24 14:16 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-04-20 6:01 [PATCH RESEND v15 0/5] Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 1/5] userfaultfd: UFFD_FEATURE_WP_ASYNC Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs Muhammad Usama Anjum
2023-04-26 7:06 ` Muhammad Usama Anjum
2023-04-26 14:13 ` Peter Xu
2023-04-27 15:31 ` Muhammad Usama Anjum
2023-05-22 10:24 ` Muhammad Usama Anjum
2023-05-22 11:26 ` Muhammad Usama Anjum
2023-05-23 19:43 ` Peter Xu
2023-05-24 11:26 ` Muhammad Usama Anjum
2023-05-24 13:55 ` Peter Xu
2023-05-24 14:16 ` Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 3/5] tools headers UAPI: Update linux/fs.h with the kernel sources Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 4/5] mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL Muhammad Usama Anjum
2023-04-20 6:01 ` [PATCH RESEND v15 5/5] selftests: mm: add pagemap ioctl tests Muhammad Usama Anjum
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).