[PATCH v25 0/5] Implement IOCTL to get and optionally clear info about PTEs

[PATCH v25 0/5] Implement IOCTL to get and optionally clear info about PTEs

[Muhammad Usama Anjum]

*Changes in v25*:
- Do proper filtering on hole as well (hole got missed earlier)

*Changes in v24*:
- Rebase on top of next-20230710
- Place WP markers in case of hole as well

*Changes in v23*:
- Set vec_buf_index in loop only when vec_buf_index is set
- Return -EFAULT instead of -EINVAL if vec is NULL
- Correctly return the walk ending address to the page granularity

*Changes in v22*:
- Interface change:
  - Replace [start start + len) with [start, end)
  - Return the ending address of the address walk in start

*Changes in v21*:
- Abort walk instead of returning error if WP is to be performed on
  partial hugetlb

*Changes in v20*
- Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO

*Changes in v19*
- Minor changes and interface updates

*Changes in v18*
- Rebase on top of next-20230613
- Minor updates

*Changes in v17*
- Rebase on top of next-20230606
- Minor improvements in PAGEMAP_SCAN IOCTL patch

*Changes in v16*
- Fix a corner case
- Add exclusive PM_SCAN_OP_WP back

*Changes in v15*
- Build fix (Add missed build fix in RESEND)

*Changes in v14*
- Fix build error caused by #ifdef added at last minute in some configs

*Changes in v13*
- Rebase on top of next-20230414
- Give-up on using uffd_wp_range() and write new helpers, flush tlb only
  once

*Changes in v12*
- Update and other memory types to UFFD_FEATURE_WP_ASYNC
- Rebaase on top of next-20230406
- Review updates

*Changes in v11*
- Rebase on top of next-20230307
- Base patches on UFFD_FEATURE_WP_UNPOPULATED
- Do a lot of cosmetic changes and review updates
- Remove ENGAGE_WP + !GET operation as it can be performed with
  UFFDIO_WRITEPROTECT

*Changes in v10*
- Add specific condition to return error if hugetlb is used with wp
  async
- Move changes in tools/include/uapi/linux/fs.h to separate patch
- Add documentation

*Changes in v9:*
- Correct fault resolution for userfaultfd wp async
- Fix build warnings and errors which were happening on some configs
- Simplify pagemap ioctl's code

*Changes in v8:*
- Update uffd async wp implementation
- Improve PAGEMAP_IOCTL implementation

*Changes in v7:*
- Add uffd wp async
- Update the IOCTL to use uffd under the hood instead of soft-dirty
  flags

*Motivation*
The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows
GetWriteWatch() syscall [1]. The GetWriteWatch{} retrieves the addresses of
the pages that are written to in a region of virtual memory.

This syscall is used in Windows applications and games etc. This syscall is
being emulated in pretty slow manner in userspace. Our purpose is to
enhance the kernel such that we translate it efficiently in a better way.
Currently some out of tree hack patches are being used to efficiently
emulate it in some kernels. We intend to replace those with these patches.
So the whole gaming on Linux can effectively get benefit from this. It
means there would be tons of users of this code.

CRIU use case [2] was mentioned by Andrei and Danylo:
> Use cases for migrating sparse VMAs are binaries sanitized with ASAN,
> MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of
> shadow memory [4]. Being able to migrate such binaries allows to highly
> reduce the amount of work needed to identify and fix post-migration
> crashes, which happen constantly.

Andrei's defines the following uses of this code:
* it is more granular and allows us to track changed pages more
  effectively. The current interface can clear dirty bits for the entire
  process only. In addition, reading info about pages is a separate
  operation. It means we must freeze the process to read information
  about all its pages, reset dirty bits, only then we can start dumping
  pages. The information about pages becomes more and more outdated,
  while we are processing pages. The new interface solves both these
  downsides. First, it allows us to read pte bits and clear the
  soft-dirty bit atomically. It means that CRIU will not need to freeze
  processes to pre-dump their memory. Second, it clears soft-dirty bits
  for a specified region of memory. It means CRIU will have actual info
  about pages to the moment of dumping them.
* The new interface has to be much faster because basic page filtering
  is happening in the kernel. With the old interface, we have to read
  pagemap for each page.

*Implementation Evolution (Short Summary)*
From the definition of GetWriteWatch(), we feel like kernel's soft-dirty
feature can be used under the hood with some additions like:
* reset soft-dirty flag for only a specific region of memory instead of
clearing the flag for the entire process
* get and clear soft-dirty flag for a specific region atomically

So we decided to use ioctl on pagemap file to read or/and reset soft-dirty
flag. But using soft-dirty flag, sometimes we get extra pages which weren't
even written. They had become soft-dirty because of VMA merging and
VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were
able to by-pass this short coming by ignoring VM_SOFTDIRTY until David
reported that mprotect etc messes up the soft-dirty flag while ignoring
VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We
discussed if we can revert these patches. But we could not reach to any
conclusion. So at this point, I made couple of tries to solve this whole
VM_SOFTDIRTY issue by correcting the soft-dirty implementation:
* [7] Correct the bug fixed wrongly back in 2014. It had potential to cause
regression. We left it behind.
* [8] Keep a list of soft-dirty part of a VMA across splits and merges. I
got the reply don't increase the size of the VMA by 8 bytes.

At this point, we left soft-dirty considering it is too much delicate and
userfaultfd [9] seemed like the only way forward. From there onward, we
have been basing soft-dirty emulation on userfaultfd wp feature where
kernel resolves the faults itself when WP_ASYNC feature is used. It was
straight forward to add WP_ASYNC feature in userfautlfd. Now we get only
those pages dirty or written-to which are really written in reality. (PS
There is another WP_UNPOPULATED userfautfd feature is required which is
needed to avoid pre-faulting memory before write-protecting [9].)

All the different masks were added on the request of CRIU devs to create
interface more generic and better.

[1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getwritewatch
[2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com
[3] https://github.com/google/sanitizers
[4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit
[5] https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@redhat.com
[6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@redhat.com/
[7] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
[8] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
[9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com
[10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@google.com

* Original Cover letter from v8*
Hello,

Note:
Soft-dirty pages and pages which have been written-to are synonyms. As
kernel already has soft-dirty feature inside which we have given up to
use, we are using written-to terminology while using UFFD async WP under
the hood.

This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are
supported in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
  file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
  (PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
  pages have been written-to.
- Find pages which have been written-to and write protect the pages
  (atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE)

It is possible to find and clear soft-dirty pages entirely in userspace.
But it isn't efficient:
- The mprotect and SIGSEGV handler for bookkeeping
- The userfaultfd wp (synchronous) with the handler for bookkeeping

Some benchmarks can be seen here[1]. This series adds features that weren't
present earlier:
- There is no atomic get soft-dirty/Written-to status and clear present in
  the kernel.
- The pages which have been written-to can not be found in accurate way.
  (Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty
  pages than there actually are.)

Historically, soft-dirty PTE bit tracking has been used in the CRIU
project. The procfs interface is enough for finding the soft-dirty bit
status and clearing the soft-dirty bit of all the pages of a process.
We have the use case where we need to track the soft-dirty PTE bit for
only specific pages on-demand. We need this tracking and clear mechanism
of a region of memory while the process is running to emulate the
getWriteWatch() syscall of Windows.

*(Moved to using UFFD instead of soft-dirtyi feature to find pages which
have been written-to from v7 patch series)*:
Stop using the soft-dirty flags for finding which pages have been
written to. It is too delicate and wrong as it shows more soft-dirty
pages than the actual soft-dirty pages. There is no interest in
correcting it [2][3] as this is how the feature was written years ago.
It shouldn't be updated to changed behaviour. Peter Xu has suggested
using the async version of the UFFD WP [4] as it is based inherently
on the PTEs.

So in this patch series, I've added a new mode to the UFFD which is
asynchronous version of the write protect. When this variant of the
UFFD WP is used, the page faults are resolved automatically by the
kernel. The pages which have been written-to can be found by reading
pagemap file (!PM_UFFD_WP). This feature can be used successfully to
find which pages have been written to from the time the pages were
write protected. This works just like the soft-dirty flag without
showing any extra pages which aren't soft-dirty in reality.

The information related to pages if the page is file mapped, present and
swapped is required for the CRIU project [5][6]. The addition of the
required mask, any mask, excluded mask and return masks are also required
for the CRIU project [5].

The IOCTL returns the addresses of the pages which match the specific
masks. The page addresses are returned in struct page_region in a compact
form. The max_pages is needed to support a use case where user only wants
to get a specific number of pages. So there is no need to find all the
pages of interest in the range when max_pages is specified. The IOCTL
returns when the maximum number of the pages are found. The max_pages is
optional. If max_pages is specified, it must be equal or greater than the
vec_size. This restriction is needed to handle worse case when one
page_region only contains info of one page and it cannot be compacted.
This is needed to emulate the Windows getWriteWatch() syscall.

The patch series include the detailed selftest which can be used as an
example for the uffd async wp test and PAGEMAP_IOCTL. It shows the
interface usages as well.

[1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/
[2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
[3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
[4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n
[5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/
[6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/

Regards,
Muhammad Usama Anjum

Muhammad Usama Anjum (4):
  fs/proc/task_mmu: Implement IOCTL to get and optionally clear info
    about PTEs
  tools headers UAPI: Update linux/fs.h with the kernel sources
  mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL
  selftests: mm: add pagemap ioctl tests

Peter Xu (1):
  userfaultfd: UFFD_FEATURE_WP_ASYNC

 Documentation/admin-guide/mm/pagemap.rst     |   58 +
 Documentation/admin-guide/mm/userfaultfd.rst |   35 +
 fs/proc/task_mmu.c                           |  591 +++++++
 fs/userfaultfd.c                             |   26 +-
 include/linux/hugetlb.h                      |    1 +
 include/linux/userfaultfd_k.h                |   21 +-
 include/uapi/linux/fs.h                      |   55 +
 include/uapi/linux/userfaultfd.h             |    9 +-
 mm/hugetlb.c                                 |   34 +-
 mm/memory.c                                  |   27 +-
 tools/include/uapi/linux/fs.h                |   55 +
 tools/testing/selftests/mm/.gitignore        |    2 +
 tools/testing/selftests/mm/Makefile          |    3 +-
 tools/testing/selftests/mm/config            |    1 +
 tools/testing/selftests/mm/pagemap_ioctl.c   | 1464 ++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh    |    4 +
 16 files changed, 2362 insertions(+), 24 deletions(-)
 create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
 mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh

-- 
2.39.2

[PATCH v25 1/5] userfaultfd: UFFD_FEATURE_WP_ASYNC

[Muhammad Usama Anjum]

From: Peter Xu <peterx@redhat.com>

This patch adds a new userfaultfd-wp feature UFFD_FEATURE_WP_ASYNC, that
allows userfaultfd wr-protect faults to be resolved by the kernel directly.

It can be used like a high accuracy version of soft-dirty, without vma
modifications during tracking, and also with ranged support by default
rather than for a whole mm when reset the protections due to existence of
ioctl(UFFDIO_WRITEPROTECT).

Several goals of such a dirty tracking interface:

1. All types of memory should be supported and tracable. This is nature
   for soft-dirty but should mention when the context is userfaultfd,
   because it used to only support anon/shmem/hugetlb. The problem is for
   a dirty tracking purpose these three types may not be enough, and it's
   legal to track anything e.g. any page cache writes from mmap.

2. Protections can be applied to partial of a memory range, without vma
   split/merge fuss.  The hope is that the tracking itself should not
   affect any vma layout change.  It also helps when reset happens because
   the reset will not need mmap write lock which can block the tracee.

3. Accuracy needs to be maintained.  This means we need pte markers to work
   on any type of VMA.

One could question that, the whole concept of async dirty tracking is not
really close to fundamentally what userfaultfd used to be: it's not "a
fault to be serviced by userspace" anymore. However, using userfaultfd-wp
here as a framework is convenient for us in at least:

1. VM_UFFD_WP vma flag, which has a very good name to suite something like
   this, so we don't need VM_YET_ANOTHER_SOFT_DIRTY. Just use a new
   feature bit to identify from a sync version of uffd-wp registration.

2. PTE markers logic can be leveraged across the whole kernel to maintain
   the uffd-wp bit as long as an arch supports, this also applies to this
   case where uffd-wp bit will be a hint to dirty information and it will
   not go lost easily (e.g. when some page cache ptes got zapped).

3. Reuse ioctl(UFFDIO_WRITEPROTECT) interface for either starting or
   resetting a range of memory, while there's no counterpart in the old
   soft-dirty world, hence if this is wanted in a new design we'll need a
   new interface otherwise.

We can somehow understand that commonality because uffd-wp was
fundamentally a similar idea of write-protecting pages just like
soft-dirty.

This implementation allows WP_ASYNC to imply WP_UNPOPULATED, because so far
WP_ASYNC seems to not usable if without WP_UNPOPULATE.  This also gives us
chance to modify impl of WP_ASYNC just in case it could be not depending on
WP_UNPOPULATED anymore in the future kernels. It's also fine to imply that
because both features will rely on PTE_MARKER_UFFD_WP config option, so
they'll show up together (or both missing) in an UFFDIO_API probe.

vma_can_userfault() now allows any VMA if the userfaultfd registration is
only about async uffd-wp. So we can track dirty for all kinds of memory
including generic file systems (like XFS, EXT4 or BTRFS).

One trick worth mention in do_wp_page() is that we need to manually update
vmf->orig_pte here because it can be used later with a pte_same() check -
this path always has FAULT_FLAG_ORIG_PTE_VALID set in the flags.

The major defect of this approach of dirty tracking is we need to populate
the pgtables when tracking starts. Soft-dirty doesn't do it like that.
It's unwanted in the case where the range of memory to track is huge and
unpopulated (e.g., tracking updates on a 10G file with mmap() on top,
without having any page cache installed yet). One way to improve this is
to allow pte markers exist for larger than PTE level for PMD+. That will
not change the interface if to implemented, so we can leave that for later.

Co-developed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
Changes in v24:
- Rebase on top of next-20230710
- UFFD_FEATURE_WP_ASYNC has been defined to be (1 << 15) after the
  rebase

Changes in v18:
- Rebase on top of next-20230613 (use ptep_get)

Changes in v17:
- Rebase on top of next-20230525

Changes in v12:
- Peter added the hugetlb support and revamped some other implementation
- Transferred the authorship to Peter
- Merge documentation to this patch

Changes in v11:
- Fix return code in userfaultfd_register() and minor changes here and
  there
- Rebase on top of next-20230307
- Base patches on UFFD_FEATURE_WP_UNPOPULATED https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com
- UFFD_FEATURE_WP_ASYNC depends on UFFD_FEATURE_WP_UNPOPULATED to work
  (correctly)

Changes in v10:
- Build fix
- Update comments and add error condition to return error from uffd
  register if hugetlb pages are present when wp async flag is set

Changes in v9:
- Correct the fault resolution with code contributed by Peter

Changes in v7:
- Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC
- Handle automatic page fault resolution in better way (thanks to Peter)
---
 Documentation/admin-guide/mm/userfaultfd.rst | 35 ++++++++++++++++++++
 fs/userfaultfd.c                             | 26 ++++++++++++---
 include/linux/userfaultfd_k.h                | 21 +++++++++++-
 include/uapi/linux/userfaultfd.h             |  9 ++++-
 mm/hugetlb.c                                 | 32 ++++++++++--------
 mm/memory.c                                  | 27 +++++++++++++--
 6 files changed, 128 insertions(+), 22 deletions(-)

diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst
index 4349a8c2b978..203e26da5f92 100644
--- a/Documentation/admin-guide/mm/userfaultfd.rst
+++ b/Documentation/admin-guide/mm/userfaultfd.rst
@@ -244,6 +244,41 @@ write-protected (so future writes will also result in a WP fault). These ioctls
 support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP``
 respectively) to configure the mapping this way.
 
+If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set,
+any vma registered with write-protection will work in async mode rather
+than the default sync mode.
+
+In async mode, there will be no message generated when a write operation
+happens, meanwhile the write-protection will be resolved automatically by
+the kernel.  It can be seen as a more accurate version of soft-dirty
+tracking and it can be different in a few ways:
+
+  - The dirty result will not be affected by vma changes (e.g. vma
+    merging) because the dirty is only tracked by the pte.
+
+  - It supports range operations by default, so one can enable tracking on
+    any range of memory as long as page aligned.
+
+  - Dirty information will not get lost if the pte was zapped due to
+    various reasons (e.g. during split of a shmem transparent huge page).
+
+  - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit
+    set; dirty when uffd-wp bit cleared), it has different semantics on
+    some of the memory operations.  For example: ``MADV_DONTNEED`` on
+    anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as
+    dirtying of memory by dropping uffd-wp bit during the procedure.
+
+The user app can collect the "written/dirty" status by looking up the
+uffd-wp bit for the pages being interested in /proc/pagemap.
+
+The page will not be under track of uffd-wp async mode until the page is
+explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode
+flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set.  Trying to resolve a page fault
+that was tracked by async mode userfaultfd-wp is invalid.
+
+When userfaultfd-wp async mode is used alone, it can be applied to all
+kinds of memory.
+
 Memory Poisioning Emulation
 ---------------------------
 
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index db57d5d4fcb7..08be70c7a57e 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -123,6 +123,11 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
 	return ctx->features & UFFD_FEATURE_INITIALIZED;
 }
 
+static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
+{
+	return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
+}
+
 /*
  * Whether WP_UNPOPULATED is enabled on the uffd context.  It is only
  * meaningful when userfaultfd_wp()==true on the vma and when it's
@@ -1318,6 +1323,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 	bool basic_ioctls;
 	unsigned long start, end, vma_end;
 	struct vma_iterator vmi;
+	bool wp_async = userfaultfd_wp_async_ctx(ctx);
 	pgoff_t pgoff;
 
 	user_uffdio_register = (struct uffdio_register __user *) arg;
@@ -1392,7 +1398,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 
 		/* check not compatible vmas */
 		ret = -EINVAL;
-		if (!vma_can_userfault(cur, vm_flags))
+		if (!vma_can_userfault(cur, vm_flags, wp_async))
 			goto out_unlock;
 
 		/*
@@ -1453,7 +1459,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 	for_each_vma_range(vmi, vma, end) {
 		cond_resched();
 
-		BUG_ON(!vma_can_userfault(vma, vm_flags));
+		BUG_ON(!vma_can_userfault(vma, vm_flags, wp_async));
 		BUG_ON(vma->vm_userfaultfd_ctx.ctx &&
 		       vma->vm_userfaultfd_ctx.ctx != ctx);
 		WARN_ON(!(vma->vm_flags & VM_MAYWRITE));
@@ -1553,6 +1559,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 	unsigned long start, end, vma_end;
 	const void __user *buf = (void __user *)arg;
 	struct vma_iterator vmi;
+	bool wp_async = userfaultfd_wp_async_ctx(ctx);
 	pgoff_t pgoff;
 
 	ret = -EFAULT;
@@ -1607,7 +1614,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 		 * provides for more strict behavior to notice
 		 * unregistration errors.
 		 */
-		if (!vma_can_userfault(cur, cur->vm_flags))
+		if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
 			goto out_unlock;
 
 		found = true;
@@ -1623,7 +1630,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 	for_each_vma_range(vmi, vma, end) {
 		cond_resched();
 
-		BUG_ON(!vma_can_userfault(vma, vma->vm_flags));
+		BUG_ON(!vma_can_userfault(vma, vma->vm_flags, wp_async));
 
 		/*
 		 * Nothing to do: this vma is already registered into this
@@ -2005,6 +2012,11 @@ static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long
 	return ret;
 }
 
+bool userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+	return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
+}
+
 static inline unsigned int uffd_ctx_features(__u64 user_features)
 {
 	/*
@@ -2038,6 +2050,11 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
 	ret = -EPERM;
 	if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
 		goto err_out;
+
+	/* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
+	if (features & UFFD_FEATURE_WP_ASYNC)
+		features |= UFFD_FEATURE_WP_UNPOPULATED;
+
 	/* report all available features and ioctls to userland */
 	uffdio_api.features = UFFD_API_FEATURES;
 #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
@@ -2050,6 +2067,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
 #ifndef CONFIG_PTE_MARKER_UFFD_WP
 	uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
 	uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
+	uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
 #endif
 	uffdio_api.ioctls = UFFD_API_IOCTLS;
 	ret = -EFAULT;
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index ac8c6854097c..c98df391bfd8 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -161,11 +161,22 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
 }
 
 static inline bool vma_can_userfault(struct vm_area_struct *vma,
-				     unsigned long vm_flags)
+				     unsigned long vm_flags,
+				     bool wp_async)
 {
+	vm_flags &= __VM_UFFD_FLAGS;
+
 	if ((vm_flags & VM_UFFD_MINOR) &&
 	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
 		return false;
+
+	/*
+	 * If wp async enabled, and WP is the only mode enabled, allow any
+	 * memory type.
+	 */
+	if (wp_async && (vm_flags == VM_UFFD_WP))
+		return true;
+
 #ifndef CONFIG_PTE_MARKER_UFFD_WP
 	/*
 	 * If user requested uffd-wp but not enabled pte markers for
@@ -175,6 +186,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma,
 	if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma))
 		return false;
 #endif
+
+	/* By default, allow any of anon|shmem|hugetlb */
 	return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
 	    vma_is_shmem(vma);
 }
@@ -197,6 +210,7 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma,
 extern void userfaultfd_unmap_complete(struct mm_struct *mm,
 				       struct list_head *uf);
 extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
+extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
 
 #else /* CONFIG_USERFAULTFD */
 
@@ -297,6 +311,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
 	return false;
 }
 
+static inline bool userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+	return false;
+}
+
 #endif /* CONFIG_USERFAULTFD */
 
 static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
index 62151706c5a3..0dbc81015018 100644
--- a/include/uapi/linux/userfaultfd.h
+++ b/include/uapi/linux/userfaultfd.h
@@ -40,7 +40,8 @@
 			   UFFD_FEATURE_EXACT_ADDRESS |		\
 			   UFFD_FEATURE_WP_HUGETLBFS_SHMEM |	\
 			   UFFD_FEATURE_WP_UNPOPULATED |	\
-			   UFFD_FEATURE_POISON)
+			   UFFD_FEATURE_POISON |		\
+			   UFFD_FEATURE_WP_ASYNC)
 #define UFFD_API_IOCTLS				\
 	((__u64)1 << _UFFDIO_REGISTER |		\
 	 (__u64)1 << _UFFDIO_UNREGISTER |	\
@@ -216,6 +217,11 @@ struct uffdio_api {
 	 * (i.e. empty ptes).  This will be the default behavior for shmem
 	 * & hugetlbfs, so this flag only affects anonymous memory behavior
 	 * when userfault write-protection mode is registered.
+	 *
+	 * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
+	 * asynchronous mode is supported in which the write fault is
+	 * automatically resolved and write-protection is un-set.
+	 * It implies UFFD_FEATURE_WP_UNPOPULATED.
 	 */
 #define UFFD_FEATURE_PAGEFAULT_FLAG_WP		(1<<0)
 #define UFFD_FEATURE_EVENT_FORK			(1<<1)
@@ -232,6 +238,7 @@ struct uffdio_api {
 #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM		(1<<12)
 #define UFFD_FEATURE_WP_UNPOPULATED		(1<<13)
 #define UFFD_FEATURE_POISON			(1<<14)
+#define UFFD_FEATURE_WP_ASYNC			(1<<15)
 	__u64 features;
 
 	__u64 ioctls;
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index e4a28ce0667f..ee88a2a6649f 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -6165,21 +6165,27 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 	/* Handle userfault-wp first, before trying to lock more pages */
 	if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) &&
 	    (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {
-		struct vm_fault vmf = {
-			.vma = vma,
-			.address = haddr,
-			.real_address = address,
-			.flags = flags,
-		};
+		if (!userfaultfd_wp_async(vma)) {
+			struct vm_fault vmf = {
+				.vma = vma,
+				.address = haddr,
+				.real_address = address,
+				.flags = flags,
+			};
 
-		spin_unlock(ptl);
-		if (pagecache_folio) {
-			folio_unlock(pagecache_folio);
-			folio_put(pagecache_folio);
+			spin_unlock(ptl);
+			if (pagecache_folio) {
+				folio_unlock(pagecache_folio);
+				folio_put(pagecache_folio);
+			}
+			hugetlb_vma_unlock_read(vma);
+			mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+			return handle_userfault(&vmf, VM_UFFD_WP);
 		}
-		hugetlb_vma_unlock_read(vma);
-		mutex_unlock(&hugetlb_fault_mutex_table[hash]);
-		return handle_userfault(&vmf, VM_UFFD_WP);
+
+		entry = huge_pte_clear_uffd_wp(entry);
+		set_huge_pte_at(mm, haddr, ptep, entry);
+		/* Fallthrough to CoW */
 	}
 
 	/*
diff --git a/mm/memory.c b/mm/memory.c
index 0a265ac6246e..b9f8c32ddd28 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3352,11 +3352,28 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 	const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
 	struct vm_area_struct *vma = vmf->vma;
 	struct folio *folio = NULL;
+	pte_t pte;
 
 	if (likely(!unshare)) {
 		if (userfaultfd_pte_wp(vma, ptep_get(vmf->pte))) {
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			return handle_userfault(vmf, VM_UFFD_WP);
+			if (!userfaultfd_wp_async(vma)) {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				return handle_userfault(vmf, VM_UFFD_WP);
+			}
+
+			/*
+			 * Nothing needed (cache flush, TLB invalidations,
+			 * etc.) because we're only removing the uffd-wp bit,
+			 * which is completely invisible to the user.
+			 */
+			pte = pte_clear_uffd_wp(ptep_get(vmf->pte));
+
+			set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+			/*
+			 * Update this to be prepared for following up CoW
+			 * handling
+			 */
+			vmf->orig_pte = pte;
 		}
 
 		/*
@@ -4851,8 +4868,11 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
 
 	if (vma_is_anonymous(vmf->vma)) {
 		if (likely(!unshare) &&
-		    userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
+		    userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
+			if (userfaultfd_wp_async(vmf->vma))
+				goto split;
 			return handle_userfault(vmf, VM_UFFD_WP);
+		}
 		return do_huge_pmd_wp_page(vmf);
 	}
 
@@ -4864,6 +4884,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
 		}
 	}
 
+split:
 	/* COW or write-notify handled on pte level: split pmd. */
 	__split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
 
-- 
2.39.2

[PATCH v25 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs

[Muhammad Usama Anjum]

This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are supported
in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
  file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT), swapped
  (PAGE_IS_SWAPPED) or page has pfn zero (PAGE_IS_PFNZERO).
- Find pages which have been written-to and/or write protect the pages
  (atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP)

This IOCTL can be extended to get information about more PTE bits. The
entire address range passed by user [start, end) is scanned until either
the user provided buffer is full or max_pages have been found.

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v25:
- Do proper filtering on hole as well (hole got missed earlier)

Changes in v24:
- Place WP markers in case of hole as well

Changes in v23:
- Set vec_buf_index to 0 only when vec_buf_index is set
- Return -EFAULT instead of -EINVAL if vec is NULL
- Correctly return the walk ending address to the page granularity

Changes in v22:
- Interface change to return walk ending address to user:
  - Replace [start start + len) with [start, end)
  - Return the ending address of the address walk in start

Changes in v21:
- Abort walk instead of returning error if WP is to be performed on
  partial hugetlb
- Changed the data types of some variables in pagemap_scan_private to
  long

Changes in v20:
- Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO

Changes in v19:
- Interface changes such as renaming, return mask and WP can be used
  with any flags specified in masks
- Internal code changes

Changes in v18:
- Rebased on top of next-20230613
  - ptep_get() updates
  - remove pmd_trans_unstable() and add ACTION_AGAIN
- Review updates (Micheal)

Changes in v17:
- Rebased on next-20230606
- Made make_uffd_wp_*_pte() better and minor changes

Changes in v16:
- Fixed a corner case where kernel writes beyond user buffer by one
  element
- Bring back exclusive PM_SCAN_OP_WP
- Cosmetic changes

Changes in v15:
- Build fix:
  - Use generic tlb flush function in pagemap_scan_pmd_entry() instead of
    using x86 specific flush function in do_pagemap_scan()
  - Remove #ifdef from pagemap_scan_hugetlb_entry()
  - Use mm instead of undefined vma->vm_mm

Changes in v14:
- Fix build error caused by #ifdef added at last minute in some configs

Changes in v13:
- Review updates
- mmap_read_lock_killable() instead of mmap_read_lock()
- Replace uffd_wp_range() with helpers which increases performance
  drastically for OP_WP operations by reducing the number of tlb
  flushing etc
- Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range

Changes in v12:
- Add hugetlb support to cover all memory types
- Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
- Review updates to the code

Changes in v11:
- Find written pages in a better way
- Fix a corner case (thanks Paul)
- Improve the code/comments
- remove ENGAGE_WP + ! GET operation
- shorten the commit message in favour of moving documentation to
  pagemap.rst

Changes in v10:
- move changes in tools/include/uapi/linux/fs.h to separate patch
- update commit message

Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code

Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
---
 fs/proc/task_mmu.c      | 591 ++++++++++++++++++++++++++++++++++++++++
 include/linux/hugetlb.h |   1 +
 include/uapi/linux/fs.h |  55 ++++
 mm/hugetlb.c            |   2 +-
 4 files changed, 648 insertions(+), 1 deletion(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index cfab855fe7e9..596fdfdc4387 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -19,6 +19,7 @@
 #include <linux/shmem_fs.h>
 #include <linux/uaccess.h>
 #include <linux/pkeys.h>
+#include <linux/minmax.h>
 
 #include <asm/elf.h>
 #include <asm/tlb.h>
@@ -1768,11 +1769,601 @@ static int pagemap_release(struct inode *inode, struct file *file)
 	return 0;
 }
 
+#define PM_SCAN_REQUIRE_UFFD	(1ULL << 63)
+
+#define PM_SCAN_FOUND_MAX_PAGES	(1)
+#define PM_SCAN_END_WALK	(-256)
+
+#define PM_SCAN_BITS_ALL	(PAGE_IS_WRITTEN | PAGE_IS_FILE |	\
+				 PAGE_IS_PRESENT | PAGE_IS_SWAPPED |	\
+				 PAGE_IS_PFNZERO)
+#define PM_SCAN_OPS		(PM_SCAN_OP_GET | PM_SCAN_OP_WP)
+#define IS_PM_SCAN_GET(flags)	(flags & PM_SCAN_OP_GET)
+#define IS_PM_SCAN_WP(flags)	(flags & PM_SCAN_OP_WP)
+
+#define PM_SCAN_FLAGS(wt, file, present, swap, pfnzero)	\
+	((wt) | ((file) << 1) | ((present) << 2) |	\
+	 ((swap) << 3) | ((pfnzero) << 4))
+
+struct pagemap_scan_private {
+	struct page_region *vec_buf, cur_buf;
+	unsigned long vec_buf_len, vec_buf_index, max_pages, found_pages;
+	unsigned long long flags, required_mask, anyof_mask, excluded_mask, return_mask, end_addr;
+};
+
+static inline bool is_pte_uffd_wp(pte_t pte)
+{
+	return (pte_present(pte) && pte_uffd_wp(pte)) ||
+	       pte_swp_uffd_wp_any(pte);
+}
+
+static inline void make_uffd_wp_pte(struct vm_area_struct *vma,
+				    unsigned long addr, pte_t *pte)
+{
+	pte_t ptent = ptep_get(pte);
+
+	if (pte_present(ptent)) {
+		pte_t old_pte;
+
+		old_pte = ptep_modify_prot_start(vma, addr, pte);
+		ptent = pte_mkuffd_wp(ptent);
+		ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
+	} else if (is_swap_pte(ptent)) {
+		ptent = pte_swp_mkuffd_wp(ptent);
+		set_pte_at(vma->vm_mm, addr, pte, ptent);
+	} else {
+		set_pte_at(vma->vm_mm, addr, pte,
+			   make_pte_marker(PTE_MARKER_UFFD_WP));
+	}
+}
+
+static inline bool pagemap_scan_is_file(struct vm_area_struct *vma, pte_t ptent,
+					unsigned long addr)
+{
+	struct page *page = NULL;
+	swp_entry_t entry;
+
+	if (pte_present(ptent)) {
+		page = vm_normal_page(vma, addr, ptent);
+	} else {
+		entry = pte_to_swp_entry(ptent);
+		if (is_pfn_swap_entry(entry))
+			page = pfn_swap_entry_to_page(entry);
+	}
+
+	if (page && !PageAnon(page))
+		return true;
+
+	return false;
+}
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static inline bool is_pmd_uffd_wp(pmd_t pmd)
+{
+	return (pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
+	       (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd));
+}
+
+static inline void make_uffd_wp_pmd(struct vm_area_struct *vma,
+				    unsigned long addr, pmd_t *pmdp)
+{
+	pmd_t old, pmd = *pmdp;
+
+	if (pmd_present(pmd)) {
+		old = pmdp_invalidate_ad(vma, addr, pmdp);
+		pmd = pmd_mkuffd_wp(old);
+		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+	} else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
+		pmd = pmd_swp_mkuffd_wp(pmd);
+		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+	}
+}
+#endif
+
+#ifdef CONFIG_HUGETLB_PAGE
+static inline bool is_huge_pte_uffd_wp(pte_t pte)
+{
+	return (pte_present(pte) && huge_pte_uffd_wp(pte)) ||
+	       pte_swp_uffd_wp_any(pte);
+}
+
+static inline void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
+					 unsigned long addr, pte_t *ptep,
+					 pte_t ptent)
+{
+	if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
+		return;
+
+	if (is_hugetlb_entry_migration(ptent))
+		set_huge_pte_at(vma->vm_mm, addr, ptep,
+				pte_swp_mkuffd_wp(ptent));
+	else if (!huge_pte_none(ptent))
+		huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
+					     huge_pte_mkuffd_wp(ptent));
+	else
+		set_huge_pte_at(vma->vm_mm, addr, ptep,
+				make_pte_marker(PTE_MARKER_UFFD_WP));
+}
+
+static inline bool pagemap_scan_is_huge_file(pte_t pte)
+{
+	if (pte_present(pte) && (!PageAnon(pte_page(pte))))
+		return true;
+
+	return false;
+}
+#endif
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
+				  struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+
+	if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) ||
+	    !userfaultfd_wp_use_markers(vma)))
+		return -EPERM;
+
+	if (vma->vm_flags & VM_PFNMAP)
+		return 1;
+
+	return 0;
+}
+
+static int pagemap_scan_output(unsigned long bitmap,
+			       struct pagemap_scan_private *p,
+			       unsigned long addr, unsigned int n_pages)
+{
+	struct page_region *cur_buf = &p->cur_buf;
+
+	bitmap &= p->return_mask;
+
+	if (cur_buf->flags == bitmap &&
+	    cur_buf->start + cur_buf->len * PAGE_SIZE == addr) {
+		cur_buf->len += n_pages;
+		p->found_pages += n_pages;
+	} else {
+		if (cur_buf->len) {
+			if (p->vec_buf_index >= p->vec_buf_len) {
+				p->end_addr = addr;
+				return PM_SCAN_END_WALK;
+			}
+
+			memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
+			       sizeof(*p->vec_buf));
+			p->vec_buf_index++;
+		}
+
+		cur_buf->start = addr;
+		cur_buf->len = n_pages;
+		cur_buf->flags = bitmap;
+		p->found_pages += n_pages;
+	}
+
+	if (p->found_pages == p->max_pages) {
+		p->end_addr = addr + n_pages * PAGE_SIZE;
+		return PM_SCAN_FOUND_MAX_PAGES;
+	}
+
+	return 0;
+}
+
+static bool pagemap_scan_is_interesting_page(unsigned long bitmap,
+					     struct pagemap_scan_private *p)
+{
+	if ((p->required_mask & bitmap) != p->required_mask)
+		return false;
+	if (p->anyof_mask && !(p->anyof_mask & bitmap))
+		return false;
+	if (p->excluded_mask & bitmap)
+		return false;
+
+	return true;
+}
+
+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+				  unsigned long end, struct mm_walk *walk)
+{
+	bool is_written, flush = false, interesting = true;
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	unsigned long bitmap, addr = end;
+	pte_t *pte, *orig_pte, ptent;
+	spinlock_t *ptl;
+	int ret = 0;
+
+	arch_enter_lazy_mmu_mode();
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	ptl = pmd_trans_huge_lock(pmd, vma);
+	if (ptl) {
+		unsigned long n_pages = (end - start)/PAGE_SIZE;
+
+		if (n_pages > p->max_pages - p->found_pages)
+			n_pages = p->max_pages - p->found_pages;
+
+		is_written = !is_pmd_uffd_wp(*pmd);
+
+		bitmap = PM_SCAN_FLAGS(is_written, false,
+				       pmd_present(*pmd), is_swap_pmd(*pmd),
+				       pmd_present(*pmd) && is_zero_pfn(pmd_pfn(*pmd)));
+
+		if (IS_PM_SCAN_GET(p->flags))
+			interesting = pagemap_scan_is_interesting_page(bitmap, p);
+
+		if (interesting) {
+			/*
+			 * Break huge page into small pages if the WP operation
+			 * need to be performed is on a portion of the huge page.
+			 */
+			if (is_written && IS_PM_SCAN_WP(p->flags) &&
+			    n_pages < HPAGE_SIZE/PAGE_SIZE) {
+				spin_unlock(ptl);
+
+				split_huge_pmd(vma, pmd, start);
+				goto process_smaller_pages;
+			}
+
+			if (IS_PM_SCAN_GET(p->flags))
+				ret = pagemap_scan_output(bitmap, p, start, n_pages);
+
+			if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) {
+				make_uffd_wp_pmd(vma, start, pmd);
+				flush_tlb_range(vma, start, end);
+			}
+		}
+
+		spin_unlock(ptl);
+		arch_leave_lazy_mmu_mode();
+
+		return ret;
+	}
+
+process_smaller_pages:
+#endif
+
+	orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+	if (!pte) {
+		walk->action = ACTION_AGAIN;
+		return 0;
+	}
+
+	for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE) {
+		ptent = ptep_get(pte);
+		is_written = !is_pte_uffd_wp(ptent);
+
+		bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_file(vma, ptent, addr),
+				       pte_present(ptent), is_swap_pte(ptent),
+				       pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
+
+		if (IS_PM_SCAN_GET(p->flags)) {
+			interesting = pagemap_scan_is_interesting_page(bitmap, p);
+			if (interesting)
+				ret = pagemap_scan_output(bitmap, p, addr, 1);
+		}
+
+		if (IS_PM_SCAN_WP(p->flags) && is_written && interesting &&
+		    ret >= 0) {
+			make_uffd_wp_pte(vma, addr, pte);
+			flush = true;
+		}
+	}
+
+	if (flush)
+		flush_tlb_range(vma, start, addr);
+
+	pte_unmap_unlock(orig_pte, ptl);
+	arch_leave_lazy_mmu_mode();
+
+	cond_resched();
+	return ret;
+}
+
+#ifdef CONFIG_HUGETLB_PAGE
+static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
+				      unsigned long start, unsigned long end,
+				      struct mm_walk *walk)
+{
+	unsigned long n_pages = (end - start)/PAGE_SIZE;
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	bool is_written, interesting = true;
+	struct hstate *h = hstate_vma(vma);
+	unsigned long bitmap;
+	spinlock_t *ptl;
+	int ret = 0;
+	pte_t ptent;
+
+	if (IS_PM_SCAN_WP(p->flags) && n_pages < HPAGE_SIZE/PAGE_SIZE) {
+		p->end_addr = start;
+		return -EINVAL;
+	}
+
+	if (n_pages > p->max_pages - p->found_pages)
+		n_pages = p->max_pages - p->found_pages;
+
+	if (IS_PM_SCAN_WP(p->flags)) {
+		i_mmap_lock_write(vma->vm_file->f_mapping);
+		ptl = huge_pte_lock(h, vma->vm_mm, ptep);
+	}
+
+	ptent = huge_ptep_get(ptep);
+	is_written = !is_huge_pte_uffd_wp(ptent);
+
+	bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_huge_file(ptent),
+			       pte_present(ptent), is_swap_pte(ptent),
+			       pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
+
+	if (IS_PM_SCAN_GET(p->flags))
+		interesting = pagemap_scan_is_interesting_page(bitmap, p);
+
+	if (interesting) {
+		/*
+		 * Partial hugetlb page clear isn't supported
+		 */
+		if (is_written && IS_PM_SCAN_WP(p->flags) &&
+		    n_pages < HPAGE_SIZE/PAGE_SIZE) {
+			ret = PM_SCAN_END_WALK;
+			p->end_addr = start;
+			goto unlock_and_return;
+		}
+
+		if (IS_PM_SCAN_GET(p->flags))
+			ret = pagemap_scan_output(bitmap, p, start, n_pages);
+
+		if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) {
+			make_uffd_wp_huge_pte(vma, start, ptep, ptent);
+			flush_hugetlb_tlb_range(vma, start, end);
+		}
+	}
+
+unlock_and_return:
+	if (IS_PM_SCAN_WP(p->flags)) {
+		spin_unlock(ptl);
+		i_mmap_unlock_write(vma->vm_file->f_mapping);
+	}
+
+	return ret;
+}
+#else
+#define pagemap_scan_hugetlb_entry NULL
+#endif
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
+				 int depth, struct mm_walk *walk)
+{
+	unsigned long n_pages = (end - addr)/PAGE_SIZE;
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	bool interesting = true;
+	unsigned long bitmap;
+	int ret = 0;
+
+	if (!vma)
+		return 0;
+
+	bitmap = PM_SCAN_FLAGS(false, false, false, false, false);
+
+	if (IS_PM_SCAN_GET(p->flags))
+		interesting = pagemap_scan_is_interesting_page(bitmap, p);
+
+	if (interesting) {
+		if (IS_PM_SCAN_GET(p->flags)) {
+			if (n_pages > p->max_pages - p->found_pages)
+				n_pages = p->max_pages - p->found_pages;
+
+			ret = pagemap_scan_output(bitmap, p, addr, n_pages);
+		}
+
+		if (IS_PM_SCAN_WP(p->flags) && !ret &&
+		    uffd_wp_range(vma, addr, end - addr, true) < 0)
+			ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+static const struct mm_walk_ops pagemap_scan_ops = {
+	.test_walk = pagemap_scan_test_walk,
+	.pmd_entry = pagemap_scan_pmd_entry,
+	.pte_hole = pagemap_scan_pte_hole,
+	.hugetlb_entry = pagemap_scan_hugetlb_entry,
+};
+
+static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start,
+				   unsigned long end, struct page_region __user *vec)
+{
+	/* Detect illegal size, flags, len and masks */
+	if (arg->size != sizeof(struct pm_scan_arg))
+		return -EINVAL;
+	if (!arg->flags)
+		return -EINVAL;
+	if (arg->flags & ~PM_SCAN_OPS)
+		return -EINVAL;
+	if (!(end - start))
+		return -EINVAL;
+	if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask |
+	     arg->return_mask) & ~PM_SCAN_BITS_ALL)
+		return -EINVAL;
+	if (!arg->required_mask && !arg->anyof_mask &&
+	    !arg->excluded_mask)
+		return -EINVAL;
+	if (!arg->return_mask)
+		return -EINVAL;
+
+	/* Validate memory range */
+	if (!IS_ALIGNED(start, PAGE_SIZE))
+		return -EINVAL;
+	if (!access_ok((void __user *)start, end - start))
+		return -EFAULT;
+
+	if (IS_PM_SCAN_GET(arg->flags)) {
+		if (arg->vec_len == 0)
+			return -EINVAL;
+		if (!vec)
+			return -EFAULT;
+		if (!access_ok((void __user *)vec,
+			       arg->vec_len * sizeof(struct page_region)))
+			return -EFAULT;
+	}
+
+	if (IS_PM_SCAN_WP(arg->flags) && !IS_PM_SCAN_GET(arg->flags) &&
+	    arg->max_pages)
+		return -EINVAL;
+
+	return 0;
+}
+
+static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg)
+{
+	struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg;
+	unsigned long long start, end, walk_start, walk_end;
+	unsigned long empty_slots, vec_index = 0;
+	struct mmu_notifier_range range;
+	struct page_region __user *vec;
+	struct pagemap_scan_private p;
+	struct pm_scan_arg arg;
+	int ret = 0;
+
+	if (copy_from_user(&arg, uarg, sizeof(arg)))
+		return -EFAULT;
+
+	start = untagged_addr((unsigned long)arg.start);
+	end = untagged_addr((unsigned long)arg.end);
+	vec = (struct page_region __user *)untagged_addr((unsigned long)arg.vec);
+
+	ret = pagemap_scan_args_valid(&arg, start, end, vec);
+	if (ret)
+		return ret;
+
+	p.max_pages = (arg.max_pages) ? arg.max_pages : ULONG_MAX;
+	p.found_pages = 0;
+	p.required_mask = arg.required_mask;
+	p.anyof_mask = arg.anyof_mask;
+	p.excluded_mask = arg.excluded_mask;
+	p.return_mask = arg.return_mask;
+	p.flags = arg.flags;
+	p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) &
+		    PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0;
+	p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0;
+	p.vec_buf = NULL;
+	p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
+	p.vec_buf_index = 0;
+	p.end_addr = 0;
+
+	/*
+	 * Allocate smaller buffer to get output from inside the page walk
+	 * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
+	 * we want to return output to user in compact form where no two
+	 * consecutive regions should be continuous and have the same flags.
+	 * So store the latest element in p.cur_buf between different walks and
+	 * store the p.cur_buf at the end of the walk to the user buffer.
+	 */
+	if (IS_PM_SCAN_GET(p.flags)) {
+		p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf),
+					  GFP_KERNEL);
+		if (!p.vec_buf)
+			return -ENOMEM;
+	}
+
+	/*
+	 * Protection change for the range is going to happen.
+	 */
+	if (IS_PM_SCAN_WP(p.flags)) {
+		mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
+					mm, start, end);
+		mmu_notifier_invalidate_range_start(&range);
+	}
+
+	walk_start = walk_end = start;
+	while (walk_end < end && !ret) {
+		if (IS_PM_SCAN_GET(p.flags)) {
+			/*
+			 * All data is copied to cur_buf first. When more data
+			 * is found, we push cur_buf to vec_buf and copy new
+			 * data to cur_buf. Subtract 1 from length as the
+			 * index of cur_buf isn't counted in length.
+			 */
+			empty_slots = arg.vec_len - vec_index;
+			p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1);
+		}
+
+		ret = mmap_read_lock_killable(mm);
+		if (ret)
+			goto out;
+
+		walk_end = min((walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK, end);
+
+		ret = walk_page_range(mm, walk_start, walk_end,
+				      &pagemap_scan_ops, &p);
+		mmap_read_unlock(mm);
+
+		if (ret == PM_SCAN_FOUND_MAX_PAGES || ret == PM_SCAN_END_WALK)
+			arg.start = p.end_addr;
+
+		if (ret && ret != PM_SCAN_FOUND_MAX_PAGES &&
+		    ret != PM_SCAN_END_WALK)
+			goto out;
+
+		if (p.vec_buf_index) {
+			if (copy_to_user(&vec[vec_index], p.vec_buf,
+					 p.vec_buf_index * sizeof(*p.vec_buf))) {
+				/*
+				 * Return error even though the OP succeeded
+				 */
+				ret = -EFAULT;
+				goto out;
+			}
+			vec_index += p.vec_buf_index;
+			p.vec_buf_index = 0;
+		}
+		walk_start = walk_end;
+	}
+
+	if (p.cur_buf.len) {
+		if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) {
+			ret = -EFAULT;
+			goto out;
+		}
+		vec_index++;
+	}
+
+	ret = vec_index;
+
+out:
+	if (!p.end_addr)
+		arg.start = walk_start;
+	if (copy_to_user(&uarg->start, &arg.start, sizeof(arg.start)))
+		ret = -EFAULT;
+
+	if (IS_PM_SCAN_WP(p.flags))
+		mmu_notifier_invalidate_range_end(&range);
+
+	kfree(p.vec_buf);
+	return ret;
+}
+
+static long do_pagemap_cmd(struct file *file, unsigned int cmd,
+			   unsigned long arg)
+{
+	struct mm_struct *mm = file->private_data;
+
+	switch (cmd) {
+	case PAGEMAP_SCAN:
+		return do_pagemap_scan(mm, arg);
+
+	default:
+		return -EINVAL;
+	}
+}
+
 const struct file_operations proc_pagemap_operations = {
 	.llseek		= mem_lseek, /* borrow this */
 	.read		= pagemap_read,
 	.open		= pagemap_open,
 	.release	= pagemap_release,
+	.unlocked_ioctl = do_pagemap_cmd,
+	.compat_ioctl	= do_pagemap_cmd,
 };
 #endif /* CONFIG_PROC_PAGE_MONITOR */
 
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 623e98d62df3..c44e7bde0339 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 		unsigned long cp_flags);
 
 bool is_hugetlb_entry_migration(pte_t pte);
+bool is_hugetlb_entry_hwpoisoned(pte_t pte);
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
 
 #else /* !CONFIG_HUGETLB_PAGE */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index b7b56871029c..ac684d99e68f 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -305,4 +305,59 @@ typedef int __bitwise __kernel_rwf_t;
 #define RWF_SUPPORTED	(RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
 			 RWF_APPEND)
 
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN	_IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WRITTEN		(1 << 0)
+#define PAGE_IS_FILE		(1 << 1)
+#define PAGE_IS_PRESENT		(1 << 2)
+#define PAGE_IS_SWAPPED		(1 << 3)
+#define PAGE_IS_PFNZERO		(1 << 4)
+
+/*
+ * struct page_region - Page region with flags
+ * @start:	Start of the region
+ * @len:	Length of the region in pages
+ * @bitmap:	Bits sets for the region
+ */
+struct page_region {
+	__u64 start;
+	__u64 len;
+	__u64 flags;
+};
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size:		Size of the structure
+ * @flags:		Flags for the IOCTL
+ * @start:		Starting address of the region
+ *			(Ending address of the walk is also returned in it)
+ * @end:		Ending address of the region
+ * @vec:		Address of page_region struct array for output
+ * @vec_len:		Length of the page_region struct array
+ * @max_pages:		Optional max return pages
+ * @required_mask:	Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask:		Any mask - Any of these bits are set in the PTE
+ * @excluded_mask:	Exclude mask - None of these bits are set in the PTE
+ * @return_mask:	Bits that are to be reported in page_region
+ */
+struct pm_scan_arg {
+	__u64 size;
+	__u64 flags;
+	__u64 start;
+	__u64 end;
+	__u64 vec;
+	__u64 vec_len;
+	__u64 max_pages;
+	__u64 required_mask;
+	__u64 anyof_mask;
+	__u64 excluded_mask;
+	__u64 return_mask;
+};
+
+/* Supported flags */
+#define PM_SCAN_OP_GET	(1 << 0)
+#define PM_SCAN_OP_WP	(1 << 1)
+
 #endif /* _UAPI_LINUX_FS_H */
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index ee88a2a6649f..f7cefdcb114c 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4981,7 +4981,7 @@ bool is_hugetlb_entry_migration(pte_t pte)
 		return false;
 }
 
-static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
+bool is_hugetlb_entry_hwpoisoned(pte_t pte)
 {
 	swp_entry_t swp;
 
-- 
2.39.2

[PATCH v25 3/5] tools headers UAPI: Update linux/fs.h with the kernel sources

[Muhammad Usama Anjum]

New IOCTL and macros has been added in the kernel sources. Update the
tools header file as well.

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v21:
- Update tools/include/uapi/linux/fs.h

Changes in v19:
- Update fs.h accourding to precious patch
---
 tools/include/uapi/linux/fs.h | 55 +++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index b7b56871029c..ac684d99e68f 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -305,4 +305,59 @@ typedef int __bitwise __kernel_rwf_t;
 #define RWF_SUPPORTED	(RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
 			 RWF_APPEND)
 
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN	_IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WRITTEN		(1 << 0)
+#define PAGE_IS_FILE		(1 << 1)
+#define PAGE_IS_PRESENT		(1 << 2)
+#define PAGE_IS_SWAPPED		(1 << 3)
+#define PAGE_IS_PFNZERO		(1 << 4)
+
+/*
+ * struct page_region - Page region with flags
+ * @start:	Start of the region
+ * @len:	Length of the region in pages
+ * @bitmap:	Bits sets for the region
+ */
+struct page_region {
+	__u64 start;
+	__u64 len;
+	__u64 flags;
+};
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size:		Size of the structure
+ * @flags:		Flags for the IOCTL
+ * @start:		Starting address of the region
+ *			(Ending address of the walk is also returned in it)
+ * @end:		Ending address of the region
+ * @vec:		Address of page_region struct array for output
+ * @vec_len:		Length of the page_region struct array
+ * @max_pages:		Optional max return pages
+ * @required_mask:	Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask:		Any mask - Any of these bits are set in the PTE
+ * @excluded_mask:	Exclude mask - None of these bits are set in the PTE
+ * @return_mask:	Bits that are to be reported in page_region
+ */
+struct pm_scan_arg {
+	__u64 size;
+	__u64 flags;
+	__u64 start;
+	__u64 end;
+	__u64 vec;
+	__u64 vec_len;
+	__u64 max_pages;
+	__u64 required_mask;
+	__u64 anyof_mask;
+	__u64 excluded_mask;
+	__u64 return_mask;
+};
+
+/* Supported flags */
+#define PM_SCAN_OP_GET	(1 << 0)
+#define PM_SCAN_OP_WP	(1 << 1)
+
 #endif /* _UAPI_LINUX_FS_H */
-- 
2.39.2

[PATCH v25 4/5] mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL

[Muhammad Usama Anjum]

Add some explanation and method to use write-protection and written-to
on memory range.

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v21:
- Update according to changed interface

Changes in v20:
- Add new flag in documentation

Changes in v19:
- Improved documentatiom with punctuation marks

Changes in v16:
- Update the documentation

Changes in v11:
- Add more documentation

rst
---
 Documentation/admin-guide/mm/pagemap.rst | 58 ++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst
index c8f380271cad..4bdad8eff009 100644
--- a/Documentation/admin-guide/mm/pagemap.rst
+++ b/Documentation/admin-guide/mm/pagemap.rst
@@ -227,3 +227,61 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is
 always 12 at most architectures). Since Linux 3.11 their meaning changes
 after first clear of soft-dirty bits. Since Linux 4.2 they are used for
 flags unconditionally.
+
+Pagemap Scan IOCTL
+==================
+
+The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally
+clear the info about page table entries. The following operations are supported
+in this IOCTL:
+- Get the information if the pages have been written to (``PAGE_IS_WRITTEN``),
+  file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped
+  (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``).
+- Find pages which have been written to and/or write protect the pages atomically
+  (atomic ``PM_SCAN_OP_GET + PM_SCAN_OP_WP``)
+
+The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
+ 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size``
+    field. This field will be helpful in recognizing the structure if extensions
+    are done later.
+ 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_OP_GET``
+    and ``PM_SCAN_OP_WP`` are the only added flags at this time.
+ 3. The range is specified through ``start`` and ``end``.
+ 4. The output buffer of ``struct page_region`` array and size is specified in
+    ``vec`` and ``vec_len``.
+ 5. The optional maximum requested pages are specified in the ``max_pages``.
+ 6. The masks are specified in ``required_mask``, ``anyof_mask``,
+    ``excluded_ mask`` and ``return_mask``.
+    1.  To find if ``PAGE_IS_WRITTEN`` flag is set for pages which have
+        ``PAGE_IS_FILE`` set and ``PAGE_IS_SWAPPED`` unset, ``required_mask``
+        is set to ``PAGE_IS_FILE``, ``exclude_mask`` is set to
+        ``PAGE_IS_SWAPPED`` and ``return_mask`` is set to ``PAGE_IS_WRITTEN``.
+        The output buffer in ``vec`` and length must be specified in ``vec_len``.
+    2. To find pages which have either ``PAGE_IS_FILE`` or ``PAGE_IS_SWAPPED``
+       set, ``anyof_masks`` is set to  ``PAGE_IS_FILE | PAGE_IS_SWAPPED``.
+    3. To find written pages and engage write protect, ``PAGE_IS_WRITTEN`` is
+       specified in ``required_mask`` and ``return_mask``. In addition to
+       specifying the output buffer in ``vec`` and length in ``vec_len``, the
+       ``PM_SCAN_OP_WP`` is specified in ``flags`` to perform write protect
+       on the range as well.
+
+The ``PAGE_IS_WRITTEN`` flag can be considered as the better and correct
+alternative of soft-dirty flag. It doesn't get affected by housekeeping chores
+(VMA merging) of the kernel and hence the user can find the true soft-dirty pages
+only. This IOCTL adds the atomic way to find which pages have been written and
+write protect those pages again. This kind of operation is needed to efficiently
+find out which pages have changed in the memory.
+
+To get information about which pages have been written to or optionally write
+protect the pages, following must be performed first in order:
+ 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall.
+ 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features
+    are set by ``UFFDIO_API`` IOCTL.
+ 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode
+    through ``UFFDIO_REGISTER`` IOCTL.
+ 4. Then any part of the registered memory or the whole memory region must
+    be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_OP_WP``
+    or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the
+    same operation. The former is better in terms of performance.
+ 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which
+    have been written to and/or optionally write protect the pages as well.
-- 
2.39.2

[PATCH v25 5/5] selftests: mm: add pagemap ioctl tests

[Muhammad Usama Anjum]

Add pagemap ioctl tests. Add several different types of tests to judge
the correction of the interface.

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v19:
- Incorporated interface changes and update tests

Changes in v18:
- Rebase on top of 20230613 (Resolve conflict in Makefile)
- Add temp files to .gitignore

Changes in v17:
- Rebase on top of next-20230525

Changes in v16:
- Added yet more tests which is a randomization test case to catch the
  corner cases
- Add reset by exclusive PM_SCAN_OP_WP as well

Changes in v13:
- Update tests and rebase Makefile

Changes in v12:
- Updates and add more memory type tests

Changes in v11:
- Rebase on top of next-20230216 and update tests

Chages in v7:
- Add and update all test cases

Changes in v6:
- Rename variables

Changes in v4:
- Updated all the tests to conform to new IOCTL

Changes in v3:
- Add another test to do sanity of flags

Changes in v2:
- Update the tests to use the ioctl interface instead of syscall
---
TAP version 13
1..92
ok 1 sanity_tests_sd memory size must be valid
ok 2 sanity_tests_sd output buffer must be specified
ok 3 sanity_tests_sd output buffer size must be valid
ok 4 sanity_tests_sd wrong flag specified
ok 5 sanity_tests_sd flag has extra bits specified
ok 6 sanity_tests_sd no selection mask is specified
ok 7 sanity_tests_sd no return mask is specified
ok 8 sanity_tests_sd wrong return mask specified
ok 9 sanity_tests_sd mixture of correct and wrong flag
ok 10 sanity_tests_sd PAGEMAP_BITS_ALL cannot be specified with PM_SCAN_OP_WP
ok 11 sanity_tests_sd Clear area with larger vec size
ok 12 sanity_tests_sd Repeated pattern of written and non-written pages
ok 13 sanity_tests_sd Repeated pattern of written and non-written pages in parts
ok 14 sanity_tests_sd Repeated pattern of written and non-written pages max_pages
ok 15 sanity_tests_sd only get 2 written pages and clear them as well
ok 16 sanity_tests_sd Two regions
ok 17 sanity_tests_sd Smaller max_pages
ok 18 Smaller vec 46 50
ok 19 Page testing: all new pages must not be written (dirty)
ok 20 Page testing: all pages must be written (dirty)
ok 21 Page testing: all pages dirty other than first and the last one
ok 22 Page testing: PM_SCAN_OP_WP
ok 23 Page testing: only middle page dirty
ok 24 Page testing: only two middle pages dirty
ok 25 Large Page testing: all new pages must not be written (dirty)
ok 26 Large Page testing: all pages must be written (dirty)
ok 27 Large Page testing: all pages dirty other than first and the last one
ok 28 Large Page testing: PM_SCAN_OP_WP
ok 29 Large Page testing: only middle page dirty
ok 30 Large Page testing: only two middle pages dirty
ok 31 Huge page testing: all new pages must not be written (dirty)
ok 32 Huge page testing: all pages must be written (dirty)
ok 33 Huge page testing: all pages dirty other than first and the last one
ok 34 Huge page testing: PM_SCAN_OP_WP
ok 35 Huge page testing: only middle page dirty
ok 36 Huge page testing: only two middle pages dirty
ok 37 Hugetlb shmem testing: all new pages must not be written (dirty)
ok 38 Hugetlb shmem testing: all pages must be written (dirty)
ok 39 Hugetlb shmem testing: all pages dirty other than first and the last one
ok 40 Hugetlb shmem testing: PM_SCAN_OP_WP
ok 41 Hugetlb shmem testing: only middle page dirty
ok 42 Hugetlb shmem testing: only two middle pages dirty
ok 43 Hugetlb mem testing: all new pages must not be written (dirty)
ok 44 Hugetlb mem testing: all pages must be written (dirty)
ok 45 Hugetlb mem testing: all pages dirty other than first and the last one
ok 46 Hugetlb mem testing: PM_SCAN_OP_WP
ok 47 Hugetlb mem testing: only middle page dirty
ok 48 Hugetlb mem testing: only two middle pages dirty
ok 49 File memory testing: all new pages must not be written (dirty)
ok 50 File memory testing: all pages must be written (dirty)
ok 51 File memory testing: all pages dirty other than first and the last one
ok 52 File memory testing: PM_SCAN_OP_WP
ok 53 File memory testing: only middle page dirty
ok 54 File memory testing: only two middle pages dirty
ok 55 File anonymous memory testing: all new pages must not be written (dirty)
ok 56 File anonymous memory testing: all pages must be written (dirty)
ok 57 File anonymous memory testing: all pages dirty other than first and the last one
ok 58 File anonymous memory testing: PM_SCAN_OP_WP
ok 59 File anonymous memory testing: only middle page dirty
ok 60 File anonymous memory testing: only two middle pages dirty
ok 61 hpage_unit_tests all new huge page must not be written (dirty)
ok 62 hpage_unit_tests all the huge page must not be written
ok 63 hpage_unit_tests all the huge page must be written and clear
ok 64 hpage_unit_tests only middle page written
ok 65 hpage_unit_tests clear first half of huge page
ok 66 hpage_unit_tests clear first half of huge page with limited buffer
ok 67 hpage_unit_tests clear second half huge page
ok 68 hpage_unit_tests get half huge page
ok 69 hpage_unit_tests get half huge page
ok 70 Test test_simple
ok 71 mprotect_tests Both pages written
ok 72 mprotect_tests Both pages are not written (dirty)
ok 73 mprotect_tests Both pages written after remap and mprotect
ok 74 mprotect_tests Clear and make the pages written
ok 75 transact_test count 192
ok 76 transact_test count 0
ok 77 transact_test Extra pages 0 (0.0%), extra thread faults 0.
ok 78 sanity_tests clear op can only be specified with PAGE_IS_WRITTEN
ok 79 sanity_tests required_mask specified
ok 80 sanity_tests anyof_mask specified
ok 81 sanity_tests excluded_mask specified
ok 82 sanity_tests required_mask and anyof_mask specified
ok 83 sanity_tests Get sd and present pages with anyof_mask
ok 84 sanity_tests Get all the pages with required_mask
ok 85 sanity_tests Get sd and present pages with required_mask and anyof_mask
ok 86 sanity_tests Don't get sd pages
ok 87 sanity_tests Don't get present pages
ok 88 sanity_tests Find written present pages with return mask
ok 89 sanity_tests Memory mapped file
ok 90 sanity_tests Read/write to private memory mapped file
ok 91 unmapped_region_tests Get status of pages
ok 92 userfaultfd_tests all new pages must not be written (dirty)
 # Totals: pass:92 fail:0 xfail:0 xpass:0 skip:0 error:0
---
 tools/testing/selftests/mm/.gitignore      |    2 +
 tools/testing/selftests/mm/Makefile        |    3 +-
 tools/testing/selftests/mm/config          |    1 +
 tools/testing/selftests/mm/pagemap_ioctl.c | 1464 ++++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh  |    4 +
 5 files changed, 1473 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
 mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh

diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore
index cdc9ce4426b9..cc920c79ff1c 100644
--- a/tools/testing/selftests/mm/.gitignore
+++ b/tools/testing/selftests/mm/.gitignore
@@ -18,6 +18,8 @@ mremap_dontunmap
 mremap_test
 on-fault-limit
 transhuge-stress
+pagemap_ioctl
+*.tmp*
 protection_keys
 protection_keys_32
 protection_keys_64
diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index b7fce9073279..1dac4e8477e4 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -33,7 +33,7 @@ endif
 MAKEFLAGS += --no-builtin-rules
 
 CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES)
-LDLIBS = -lrt -lpthread
+LDLIBS = -lrt -lpthread -lm
 
 TEST_GEN_PROGS = cow
 TEST_GEN_PROGS += compaction_test
@@ -60,6 +60,7 @@ TEST_GEN_PROGS += mrelease_test
 TEST_GEN_PROGS += mremap_dontunmap
 TEST_GEN_PROGS += mremap_test
 TEST_GEN_PROGS += on-fault-limit
+TEST_GEN_PROGS += pagemap_ioctl
 TEST_GEN_PROGS += thuge-gen
 TEST_GEN_PROGS += transhuge-stress
 TEST_GEN_PROGS += uffd-stress
diff --git a/tools/testing/selftests/mm/config b/tools/testing/selftests/mm/config
index be087c4bc396..4309916f629e 100644
--- a/tools/testing/selftests/mm/config
+++ b/tools/testing/selftests/mm/config
@@ -1,5 +1,6 @@
 CONFIG_SYSVIPC=y
 CONFIG_USERFAULTFD=y
+CONFIG_PTE_MARKER_UFFD_WP=y
 CONFIG_TEST_VMALLOC=m
 CONFIG_DEVICE_PRIVATE=y
 CONFIG_TEST_HMM=m
diff --git a/tools/testing/selftests/mm/pagemap_ioctl.c b/tools/testing/selftests/mm/pagemap_ioctl.c
new file mode 100644
index 000000000000..a2d2d9086b83
--- /dev/null
+++ b/tools/testing/selftests/mm/pagemap_ioctl.c
@@ -0,0 +1,1464 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <fcntl.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <malloc.h>
+#include "vm_util.h"
+#include "../kselftest.h"
+#include <linux/types.h>
+#include <linux/memfd.h>
+#include <linux/userfaultfd.h>
+#include <linux/fs.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <math.h>
+#include <asm/unistd.h>
+#include <pthread.h>
+#include <sys/resource.h>
+#include <assert.h>
+#include <sys/ipc.h>
+#include <sys/shm.h>
+
+#define PAGEMAP_BITS_ALL		(PAGE_IS_WRITTEN | PAGE_IS_FILE |	\
+					 PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PAGEMAP_NON_WRITTEN_BITS	(PAGE_IS_FILE |	PAGE_IS_PRESENT |	\
+					 PAGE_IS_SWAPPED)
+
+#define TEST_ITERATIONS 100
+#define PAGEMAP "/proc/self/pagemap"
+int pagemap_fd;
+int uffd;
+int page_size;
+int hpage_size;
+
+static long pagemap_ioctl(void *start, int len, void *vec, int vec_len, int flag,
+			  int max_pages, long required_mask, long anyof_mask, long excluded_mask,
+			  long return_mask)
+{
+	struct pm_scan_arg arg;
+
+	arg.start = (uintptr_t)start;
+	arg.end = (uintptr_t)(start + len);
+	arg.vec = (uintptr_t)vec;
+	arg.vec_len = vec_len;
+	arg.flags = flag;
+	arg.size = sizeof(struct pm_scan_arg);
+	arg.max_pages = max_pages;
+	arg.required_mask = required_mask;
+	arg.anyof_mask = anyof_mask;
+	arg.excluded_mask = excluded_mask;
+	arg.return_mask = return_mask;
+
+	return ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+}
+
+int init_uffd(void)
+{
+	struct uffdio_api uffdio_api;
+
+	uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+	if (uffd == -1)
+		ksft_exit_fail_msg("uffd syscall failed\n");
+
+	uffdio_api.api = UFFD_API;
+	uffdio_api.features = UFFD_FEATURE_WP_UNPOPULATED | UFFD_FEATURE_WP_ASYNC |
+			      UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
+	if (ioctl(uffd, UFFDIO_API, &uffdio_api))
+		ksft_exit_fail_msg("UFFDIO_API\n");
+
+	if (!(uffdio_api.api & UFFDIO_REGISTER_MODE_WP) ||
+	    !(uffdio_api.features & UFFD_FEATURE_WP_UNPOPULATED) ||
+	    !(uffdio_api.features & UFFD_FEATURE_WP_ASYNC) ||
+	    !(uffdio_api.features & UFFD_FEATURE_WP_HUGETLBFS_SHMEM))
+		ksft_exit_fail_msg("UFFDIO_API error %llu\n", uffdio_api.api);
+
+	return 0;
+}
+
+int wp_init(void *lpBaseAddress, int dwRegionSize)
+{
+	struct uffdio_register uffdio_register;
+	struct uffdio_writeprotect wp;
+
+	uffdio_register.range.start = (unsigned long)lpBaseAddress;
+	uffdio_register.range.len = dwRegionSize;
+	uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+	if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
+		ksft_exit_fail_msg("ioctl(UFFDIO_REGISTER) %d %s\n", errno, strerror(errno));
+
+	if (!(uffdio_register.ioctls & UFFDIO_WRITEPROTECT))
+		ksft_exit_fail_msg("ioctl set is incorrect\n");
+
+	wp.range.start = (unsigned long)lpBaseAddress;
+	wp.range.len = dwRegionSize;
+	wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+	if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp))
+		ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+
+	return 0;
+}
+
+int wp_free(void *lpBaseAddress, int dwRegionSize)
+{
+	struct uffdio_register uffdio_register;
+
+	uffdio_register.range.start = (unsigned long)lpBaseAddress;
+	uffdio_register.range.len = dwRegionSize;
+	uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+	if (ioctl(uffd, UFFDIO_UNREGISTER, &uffdio_register.range))
+		ksft_exit_fail_msg("ioctl unregister failure\n");
+	return 0;
+}
+
+int wp_addr_range(void *lpBaseAddress, int dwRegionSize)
+{
+	struct uffdio_writeprotect wp;
+
+	if (rand() % 2) {
+		wp.range.start = (unsigned long)lpBaseAddress;
+		wp.range.len = dwRegionSize;
+		wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+		if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp))
+			ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+	} else {
+		if (pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, PM_SCAN_OP_WP,
+				  0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", 1, errno, strerror(errno));
+	}
+
+	return 0;
+}
+
+void *gethugetlb_mem(int size, int *shmid)
+{
+	char *mem;
+
+	if (shmid) {
+		*shmid = shmget(2, size, SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W);
+		if (*shmid < 0)
+			return NULL;
+
+		mem = shmat(*shmid, 0, 0);
+		if (mem == (char *)-1) {
+			shmctl(*shmid, IPC_RMID, NULL);
+			ksft_exit_fail_msg("Shared memory attach failure\n");
+		}
+	} else {
+		mem = mmap(NULL, size, PROT_READ | PROT_WRITE,
+			   MAP_ANONYMOUS | MAP_HUGETLB | MAP_PRIVATE, -1, 0);
+		if (mem == MAP_FAILED)
+			return NULL;
+	}
+
+	return mem;
+}
+
+int userfaultfd_tests(void)
+{
+	int mem_size, vec_size, written, num_pages = 16;
+	char *mem, *vec;
+
+	mem_size = num_pages * page_size;
+	mem = mmap(NULL, mem_size, PROT_NONE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+
+	wp_init(mem, mem_size);
+
+	/* Change protection of pages differently */
+	mprotect(mem, mem_size/8, PROT_READ|PROT_WRITE);
+	mprotect(mem + 1 * mem_size/8, mem_size/8, PROT_READ);
+	mprotect(mem + 2 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE);
+	mprotect(mem + 3 * mem_size/8, mem_size/8, PROT_READ);
+	mprotect(mem + 4 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE);
+	mprotect(mem + 5 * mem_size/8, mem_size/8, PROT_NONE);
+	mprotect(mem + 6 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE);
+	mprotect(mem + 7 * mem_size/8, mem_size/8, PROT_READ);
+
+	wp_addr_range(mem + (mem_size/16), mem_size - 2 * (mem_size/8));
+	wp_addr_range(mem, mem_size);
+
+	vec_size = mem_size/page_size;
+	vec = malloc(sizeof(struct page_region) * vec_size);
+
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+				vec_size - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", __func__);
+
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+	free(vec);
+	return 0;
+}
+
+int get_reads(struct page_region *vec, int vec_size)
+{
+	int i, sum = 0;
+
+	for (i = 0; i < vec_size; i++)
+		sum += vec[i].len;
+
+	return sum;
+}
+
+int sanity_tests_sd(void)
+{
+	int mem_size, vec_size, ret, ret2, ret3, i, num_pages = 10, total_pages = 0;
+	int total_writes, total_reads, reads, count;
+	struct page_region *vec, *vec2;
+	char *mem, *m[2];
+
+	vec_size = 100;
+	mem_size = num_pages * page_size;
+
+	vec = malloc(sizeof(struct page_region) * vec_size);
+	if (!vec)
+		ksft_exit_fail_msg("error nomem\n");
+
+	vec2 = malloc(sizeof(struct page_region) * vec_size);
+	if (!vec2)
+		ksft_exit_fail_msg("error nomem\n");
+
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	/* 1. wrong operation */
+	ksft_test_result(pagemap_ioctl(mem, 0, vec, vec_size, PM_SCAN_OP_GET,
+				       0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0,
+			 "%s memory size must be valid\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, NULL, vec_size, PM_SCAN_OP_GET,
+				       0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0,
+			 "%s output buffer must be specified\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, 0, PM_SCAN_OP_GET,
+				       0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0,
+			 "%s output buffer size must be valid\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, -1,
+				       0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0,
+			 "%s wrong flag specified\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size,
+				       PM_SCAN_OP_GET | PM_SCAN_OP_WP | 0xFF,
+				       0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0,
+			 "%s flag has extra bits specified\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET,
+				       0, 0, 0, 0, PAGE_IS_WRITTEN) < 0,
+			 "%s no selection mask is specified\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET,
+				       0, PAGE_IS_WRITTEN, PAGE_IS_WRITTEN, 0, 0) < 0,
+			 "%s no return mask is specified\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET,
+				       0, PAGE_IS_WRITTEN, 0, 0, 0x1000) < 0,
+			 "%s wrong return mask specified\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+				       0, 0xFFF, PAGE_IS_WRITTEN, 0, PAGE_IS_WRITTEN) < 0,
+			 "%s mixture of correct and wrong flag\n", __func__);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+				       0, 0, 0, PAGEMAP_BITS_ALL, PAGE_IS_WRITTEN) >= 0,
+			 "%s PAGEMAP_BITS_ALL can be specified with PM_SCAN_OP_WP\n", __func__);
+
+	/* 2. Clear area with larger vec size */
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+			    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	ksft_test_result(ret >= 0, "%s Clear area with larger vec size\n", __func__);
+
+	/* 3. Repeated pattern of written and non-written pages */
+	for (i = 0; i < mem_size; i += 2 * page_size)
+		mem[i]++;
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0,
+			    0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ksft_test_result(ret == mem_size/(page_size * 2),
+			 "%s Repeated pattern of written and non-written pages\n", __func__);
+
+	/* 4. Repeated pattern of written and non-written pages in parts */
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			    num_pages/2 - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ret2 = pagemap_ioctl(mem, mem_size, vec, 2, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0,
+			     PAGE_IS_WRITTEN);
+	if (ret2 < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+	ret3 = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			     0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret3 < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret3, errno, strerror(errno));
+
+	ksft_test_result((ret + ret3) == num_pages/2 && ret2 == 2,
+			 "%s Repeated pattern of written and non-written pages in parts\n",
+			 __func__);
+
+	/* 5. Repeated pattern of written and non-written pages max_pages */
+	for (i = 0; i < mem_size; i += 2 * page_size)
+		mem[i]++;
+	mem[(mem_size/page_size - 1) * page_size]++;
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			    num_pages/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ret2 = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			     0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret2 < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+	ksft_test_result(ret == num_pages/2 && ret2 == 1,
+			 "%s Repeated pattern of written and non-written pages max_pages\n",
+			 __func__);
+
+	/* 6. only get 2 dirty pages and clear them as well */
+	vec_size = mem_size/page_size;
+	memset(mem, -1, mem_size);
+
+	/* get and clear second and third pages */
+	ret = pagemap_ioctl(mem + page_size, 2 * page_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			    2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ret2 = pagemap_ioctl(mem, mem_size, vec2, vec_size, PM_SCAN_OP_GET, 0,
+			      PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret2 < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+	ksft_test_result(ret == 1 && vec[0].len == 2 &&
+			 vec[0].start == (uintptr_t)(mem + page_size) &&
+			 ret2 == 2 && vec2[0].len == 1 && vec2[0].start == (uintptr_t)mem &&
+			 vec2[1].len == vec_size - 3 &&
+			 vec2[1].start == (uintptr_t)(mem + 3 * page_size),
+			 "%s only get 2 written pages and clear them as well\n", __func__);
+
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+
+	/* 7. Two regions */
+	m[0] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (m[0] == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+	m[1] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (m[1] == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+
+	wp_init(m[0], mem_size);
+	wp_init(m[1], mem_size);
+	wp_addr_range(m[0], mem_size);
+	wp_addr_range(m[1], mem_size);
+
+	memset(m[0], 'a', mem_size);
+	memset(m[1], 'b', mem_size);
+
+	wp_addr_range(m[0], mem_size);
+
+	ret = pagemap_ioctl(m[1], mem_size, vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0,
+			    PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ksft_test_result(ret == 1 && vec[0].len == mem_size/page_size,
+			 "%s Two regions\n", __func__);
+
+	wp_free(m[0], mem_size);
+	wp_free(m[1], mem_size);
+	munmap(m[0], mem_size);
+	munmap(m[1], mem_size);
+
+	free(vec);
+	free(vec2);
+
+	/* 8. Smaller vec */
+	mem_size = 1050 * page_size;
+	vec_size = mem_size/(page_size*2);
+
+	vec = malloc(sizeof(struct page_region) * vec_size);
+	if (!vec)
+		ksft_exit_fail_msg("error nomem\n");
+
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+			    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	for (i = 0; i < mem_size/page_size; i += 2)
+		mem[i * page_size]++;
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			    mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	total_pages += ret;
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			    mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	total_pages += ret;
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+			    mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	total_pages += ret;
+
+	ksft_test_result(total_pages == mem_size/(page_size*2), "%s Smaller max_pages\n", __func__);
+
+	free(vec);
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+	total_pages = 0;
+
+	/* 9. Smaller vec */
+	mem_size = 10000 * page_size;
+	vec_size = 50;
+
+	vec = malloc(sizeof(struct page_region) * vec_size);
+	if (!vec)
+		ksft_exit_fail_msg("error nomem\n");
+
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	for (count = 0; count < TEST_ITERATIONS; count++) {
+		total_writes = total_reads = 0;
+
+		for (i = 0; i < mem_size; i += page_size) {
+			if (rand() % 2) {
+				mem[i]++;
+				total_writes++;
+			}
+		}
+
+		while (total_reads < total_writes) {
+			ret = pagemap_ioctl(mem, mem_size, vec, vec_size,
+					    PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+					    0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+			if (ret < 0)
+				ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+			if (ret > vec_size)
+				break;
+
+			reads = get_reads(vec, ret);
+			total_reads += reads;
+		}
+
+		if (total_reads != total_writes)
+			break;
+	}
+
+	ksft_test_result(count == TEST_ITERATIONS, "Smaller vec %d %d\n", ret, vec_size);
+
+	free(vec);
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+
+	return 0;
+}
+
+int base_tests(char *prefix, char *mem, int mem_size, int skip)
+{
+	int vec_size, written;
+	struct page_region *vec, *vec2;
+
+	if (skip) {
+		ksft_test_result_skip("%s all new pages must not be written (dirty)\n", prefix);
+		ksft_test_result_skip("%s all pages must be written (dirty)\n", prefix);
+		ksft_test_result_skip("%s all pages dirty other than first and the last one\n",
+				      prefix);
+		ksft_test_result_skip("%s PM_SCAN_OP_WP\n", prefix);
+		ksft_test_result_skip("%s only middle page dirty\n", prefix);
+		ksft_test_result_skip("%s only two middle pages dirty\n", prefix);
+		return 0;
+	}
+
+	vec_size = mem_size/page_size;
+	vec = malloc(sizeof(struct page_region) * vec_size);
+	vec2 = malloc(sizeof(struct page_region) * vec_size);
+
+	/* 1. all new pages must be not be written (dirty) */
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, vec_size - 2,
+			      PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", prefix);
+
+	/* 2. all pages must be written */
+	memset(mem, -1, mem_size);
+
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0,
+			      PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written == 1 && vec[0].len == mem_size/page_size,
+			 "%s all pages must be written (dirty)\n", prefix);
+
+	/* 3. all pages dirty other than first and the last one */
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+				PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	memset(mem + page_size, 0, mem_size - (2 * page_size));
+
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+				PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written == 1 && vec[0].len >= vec_size - 2 && vec[0].len <= vec_size,
+			 "%s all pages dirty other than first and the last one\n", prefix);
+
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET, 0,
+				PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written == 0,
+			 "%s PM_SCAN_OP_WP\n", prefix);
+
+	/* 4. only middle page dirty */
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+				PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	mem[vec_size/2 * page_size]++;
+
+	written = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+				0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written == 1 && vec[0].len >= 1,
+			 "%s only middle page dirty\n", prefix);
+
+	/* 5. only two middle pages dirty and walk over only middle pages */
+	written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+				PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	mem[vec_size/2 * page_size]++;
+	mem[(vec_size/2 + 1) * page_size]++;
+
+	written = pagemap_ioctl(&mem[vec_size/2 * page_size], 2 * page_size, vec, 1, PM_SCAN_OP_GET,
+				0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written == 1 && vec[0].start == (uintptr_t)(&mem[vec_size/2 * page_size])
+			 && vec[0].len == 2,
+			 "%s only two middle pages dirty\n", prefix);
+
+	free(vec);
+	free(vec2);
+	return 0;
+}
+
+void *gethugepage(int map_size)
+{
+	int ret;
+	char *map;
+
+	map = memalign(hpage_size, map_size);
+	if (!map)
+		ksft_exit_fail_msg("memalign failed %d %s\n", errno, strerror(errno));
+
+	ret = madvise(map, map_size, MADV_HUGEPAGE);
+	if (ret)
+		return NULL;
+
+	memset(map, 0, map_size);
+
+	return map;
+}
+
+int hpage_unit_tests(void)
+{
+	char *map;
+	int ret, ret2;
+	size_t num_pages = 10;
+	int map_size = hpage_size * num_pages;
+	int vec_size = map_size/page_size;
+	struct page_region *vec, *vec2;
+
+	vec = malloc(sizeof(struct page_region) * vec_size);
+	vec2 = malloc(sizeof(struct page_region) * vec_size);
+	if (!vec || !vec2)
+		ksft_exit_fail_msg("malloc failed\n");
+
+	map = gethugepage(map_size);
+	if (map) {
+		wp_init(map, map_size);
+		wp_addr_range(map, map_size);
+
+		/* 1. all new huge page must not be written (dirty) */
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+				    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 0, "%s all new huge page must not be written (dirty)\n",
+				 __func__);
+
+		/* 2. all the huge page must not be written */
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 0, "%s all the huge page must not be written\n", __func__);
+
+		/* 3. all the huge page must be written and clear dirty as well */
+		memset(map, -1, map_size);
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+				    0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 1 && vec[0].start == (uintptr_t)map &&
+				 vec[0].len == vec_size && vec[0].flags == PAGE_IS_WRITTEN,
+				 "%s all the huge page must be written and clear\n", __func__);
+
+		/* 4. only middle page written */
+		wp_free(map, map_size);
+		free(map);
+		map = gethugepage(map_size);
+		wp_init(map, map_size);
+		wp_addr_range(map, map_size);
+		map[vec_size/2 * page_size]++;
+
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 1 && vec[0].len > 0,
+				 "%s only middle page written\n", __func__);
+
+		wp_free(map, map_size);
+		free(map);
+	} else {
+		ksft_test_result_skip("%s all new huge page must be written\n", __func__);
+		ksft_test_result_skip("%s all the huge page must not be written\n", __func__);
+		ksft_test_result_skip("%s all the huge page must be written and clear\n", __func__);
+		ksft_test_result_skip("%s only middle page written\n", __func__);
+	}
+
+	/* 5. clear first half of huge page */
+	map = gethugepage(map_size);
+	if (map) {
+		wp_init(map, map_size);
+		wp_addr_range(map, map_size);
+
+		memset(map, 0, map_size);
+
+		wp_addr_range(map, map_size/2);
+
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+				 vec[0].start == (uintptr_t)(map + map_size/2),
+				 "%s clear first half of huge page\n", __func__);
+		wp_free(map, map_size);
+		free(map);
+	} else {
+		ksft_test_result_skip("%s clear first half of huge page\n", __func__);
+	}
+
+	/* 6. clear first half of huge page with limited buffer */
+	map = gethugepage(map_size);
+	if (map) {
+		wp_init(map, map_size);
+		wp_addr_range(map, map_size);
+
+		memset(map, 0, map_size);
+
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+				    vec_size/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+				 vec[0].start == (uintptr_t)(map + map_size/2),
+				 "%s clear first half of huge page with limited buffer\n",
+				 __func__);
+		wp_free(map, map_size);
+		free(map);
+	} else {
+		ksft_test_result_skip("%s clear first half of huge page with limited buffer\n",
+				      __func__);
+	}
+
+	/* 7. clear second half of huge page */
+	map = gethugepage(map_size);
+	if (map) {
+		wp_init(map, map_size);
+		wp_addr_range(map, map_size);
+
+		memset(map, -1, map_size);
+
+		ret = pagemap_ioctl(map + map_size/2, map_size/2, vec, vec_size,
+				    PM_SCAN_OP_GET | PM_SCAN_OP_WP, vec_size/2, PAGE_IS_WRITTEN, 0,
+				    0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 1 && vec[0].len == vec_size/2,
+				 "%s clear second half huge page\n", __func__);
+		wp_free(map, map_size);
+		free(map);
+	} else {
+		ksft_test_result_skip("%s clear second half huge page\n", __func__);
+	}
+
+	/* 8. get half huge page */
+	map = gethugepage(map_size);
+	if (map) {
+		wp_init(map, map_size);
+		wp_addr_range(map, map_size);
+
+		memset(map, -1, map_size);
+		usleep(100);
+
+		ret = pagemap_ioctl(map, map_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+				    hpage_size/(2*page_size), PAGE_IS_WRITTEN, 0, 0,
+				    PAGE_IS_WRITTEN);
+		if (ret < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+		ksft_test_result(ret == 1 && vec[0].len == hpage_size/(2*page_size),
+				 "%s get half huge page\n", __func__);
+
+		ret2 = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				    PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN);
+		if (ret2 < 0)
+			ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+		ksft_test_result(ret2 == 1 && vec[0].len == (map_size - hpage_size/2)/page_size,
+				 "%s get half huge page\n", __func__);
+
+		wp_free(map, map_size);
+		free(map);
+	} else {
+		ksft_test_result_skip("%s get half huge page\n", __func__);
+		ksft_test_result_skip("%s get half huge page\n", __func__);
+	}
+
+	free(vec);
+	free(vec2);
+	return 0;
+}
+
+int unmapped_region_tests(void)
+{
+	void *start = (void *)0x10000000;
+	int written, len = 0x00040000;
+	int vec_size = len / page_size;
+	struct page_region *vec = malloc(sizeof(struct page_region) * vec_size);
+
+	/* 1. Get written pages */
+	written = pagemap_ioctl(start, len, vec, vec_size, PM_SCAN_OP_GET, 0,
+			      PAGEMAP_NON_WRITTEN_BITS, 0, 0, PAGEMAP_NON_WRITTEN_BITS);
+	if (written < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno));
+
+	ksft_test_result(written >= 0, "%s Get status of pages\n", __func__);
+
+	free(vec);
+	return 0;
+}
+
+static void test_simple(void)
+{
+	int i;
+	char *map;
+	struct page_region vec;
+
+	map = aligned_alloc(page_size, page_size);
+	if (!map)
+		ksft_exit_fail_msg("aligned_alloc failed\n");
+
+	wp_init(map, page_size);
+	wp_addr_range(map, page_size);
+
+	for (i = 0 ; i < TEST_ITERATIONS; i++) {
+		if (pagemap_ioctl(map, page_size, &vec, 1, PM_SCAN_OP_GET, 0,
+				  PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 1) {
+			ksft_print_msg("written bit was 1, but should be 0 (i=%d)\n", i);
+			break;
+		}
+
+		wp_addr_range(map, page_size);
+		/* Write something to the page to get the written bit enabled on the page */
+		map[0]++;
+
+		if (pagemap_ioctl(map, page_size, &vec, 1, PM_SCAN_OP_GET, 0,
+				  PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0) {
+			ksft_print_msg("written bit was 0, but should be 1 (i=%d)\n", i);
+			break;
+		}
+
+		wp_addr_range(map, page_size);
+	}
+	wp_free(map, page_size);
+	free(map);
+
+	ksft_test_result(i == TEST_ITERATIONS, "Test %s\n", __func__);
+}
+
+int sanity_tests(void)
+{
+	int mem_size, vec_size, ret, fd, i, buf_size;
+	struct page_region *vec;
+	char *mem, *fmem;
+	struct stat sbuf;
+	char *tmp_buf;
+
+	/* 1. wrong operation */
+	mem_size = 10 * page_size;
+	vec_size = mem_size / page_size;
+
+	vec = malloc(sizeof(struct page_region) * vec_size);
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED || vec == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP,
+				       0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) >= 0,
+			 "%s WP op can be specified with !PAGE_IS_WRITTEN\n", __func__);
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				       PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) >= 0,
+			 "%s required_mask specified\n", __func__);
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				       0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL) >= 0,
+			 "%s anyof_mask specified\n", __func__);
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				       0, 0, PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL) >= 0,
+			 "%s excluded_mask specified\n", __func__);
+	ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+				       PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL, 0,
+				       PAGEMAP_BITS_ALL) >= 0,
+			 "%s required_mask and anyof_mask specified\n", __func__);
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+
+	/* 2. Get sd and present pages with anyof_mask */
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	memset(mem, 0, mem_size);
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+			    0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL);
+	ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+			 vec[0].flags == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT),
+			 "%s Get sd and present pages with anyof_mask\n", __func__);
+
+	/* 3. Get sd and present pages with required_mask */
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+			    PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL);
+	ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+			 vec[0].flags == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT),
+			 "%s Get all the pages with required_mask\n", __func__);
+
+	/* 4. Get sd and present pages with required_mask and anyof_mask */
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+			    PAGE_IS_WRITTEN, PAGE_IS_PRESENT, 0, PAGEMAP_BITS_ALL);
+	ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+			 vec[0].flags == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT),
+			 "%s Get sd and present pages with required_mask and anyof_mask\n",
+			 __func__);
+
+	/* 5. Don't get sd pages */
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+			    0, 0, PAGE_IS_WRITTEN, PAGEMAP_BITS_ALL);
+	ksft_test_result(ret == 0, "%s Don't get sd pages\n", __func__);
+
+	/* 6. Don't get present pages */
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+			    0, 0, PAGE_IS_PRESENT, PAGEMAP_BITS_ALL);
+	ksft_test_result(ret == 0, "%s Don't get present pages\n", __func__);
+
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+
+	/* 8. Find written present pages with return mask */
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	memset(mem, 0, mem_size);
+
+	ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0,
+			    0, PAGEMAP_BITS_ALL, 0, PAGE_IS_WRITTEN);
+	ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+			 vec[0].flags == PAGE_IS_WRITTEN,
+			 "%s Find written present pages with return mask\n", __func__);
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+
+	/* 9. Memory mapped file */
+	fd = open(__FILE__, O_RDONLY);
+	if (fd < 0)
+		ksft_exit_fail_msg("%s Memory mapped file\n");
+
+	ret = stat(__FILE__, &sbuf);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	fmem = mmap(NULL, sbuf.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
+	if (fmem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+	tmp_buf = malloc(sbuf.st_size);
+	memcpy(tmp_buf, fmem, sbuf.st_size);
+
+	ret = pagemap_ioctl(fmem, sbuf.st_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+			    0, PAGEMAP_NON_WRITTEN_BITS, 0, PAGEMAP_NON_WRITTEN_BITS);
+
+	ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem &&
+			 vec[0].len == ceilf((float)sbuf.st_size/page_size) &&
+			 (vec[0].flags & PAGE_IS_FILE),
+			 "%s Memory mapped file\n", __func__);
+
+	munmap(fmem, sbuf.st_size);
+	close(fd);
+
+	/* 10. Create and read/write to a memory mapped file */
+	buf_size = page_size * 10;
+
+	fd = open(__FILE__".tmp2", O_RDWR | O_CREAT, 0666);
+	if (fd < 0)
+		ksft_exit_fail_msg("Read/write to memory: %s\n",
+				   strerror(errno));
+
+	for (i = 0; i < buf_size; i++)
+		if (write(fd, "c", 1) < 0)
+			ksft_exit_fail_msg("Create and read/write to a memory mapped file\n");
+
+	fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+	if (fmem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+	wp_init(fmem, buf_size);
+	wp_addr_range(fmem, buf_size);
+
+	for (i = 0; i < buf_size; i++)
+		fmem[i] = 'z';
+
+	msync(fmem, buf_size, MS_SYNC);
+
+	ret = pagemap_ioctl(fmem, buf_size, vec, vec_size, PM_SCAN_OP_GET, 0,
+			    PAGE_IS_WRITTEN, PAGE_IS_PRESENT | PAGE_IS_SWAPPED | PAGE_IS_FILE, 0,
+			    PAGEMAP_BITS_ALL);
+
+	ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem &&
+			 vec[0].len == (buf_size/page_size) &&
+			 (vec[0].flags & PAGE_IS_WRITTEN),
+			 "%s Read/write to memory\n", __func__);
+
+	wp_free(fmem, buf_size);
+	munmap(fmem, buf_size);
+	close(fd);
+
+	free(vec);
+	return 0;
+}
+
+int mprotect_tests(void)
+{
+	int ret;
+	char *mem, *mem2;
+	struct page_region vec;
+	int pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+
+	if (pagemap_fd < 0) {
+		fprintf(stderr, "open() failed\n");
+		exit(1);
+	}
+
+	/* 1. Map two pages */
+	mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+	wp_init(mem, 2 * page_size);
+	wp_addr_range(mem, 2 * page_size);
+
+	/* Populate both pages. */
+	memset(mem, 1, 2 * page_size);
+
+	ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+			    0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ksft_test_result(ret == 1 && vec.len == 2, "%s Both pages written\n", __func__);
+
+	/* 2. Start tracking */
+	wp_addr_range(mem, 2 * page_size);
+
+	ksft_test_result(pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0,
+				       PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0,
+			 "%s Both pages are not written (dirty)\n", __func__);
+
+	/* 3. Remap the second page */
+	mem2 = mmap(mem + page_size, page_size, PROT_READ|PROT_WRITE,
+		    MAP_PRIVATE|MAP_ANON|MAP_FIXED, -1, 0);
+	if (mem2 == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+	wp_init(mem2, page_size);
+	wp_addr_range(mem2, page_size);
+
+	/* Protect + unprotect. */
+	mprotect(mem, page_size, PROT_NONE);
+	mprotect(mem, 2 * page_size, PROT_READ);
+	mprotect(mem, 2 * page_size, PROT_READ|PROT_WRITE);
+
+	/* Modify both pages. */
+	memset(mem, 2, 2 * page_size);
+
+	/* Protect + unprotect. */
+	mprotect(mem, page_size, PROT_NONE);
+	mprotect(mem, page_size, PROT_READ);
+	mprotect(mem, page_size, PROT_READ|PROT_WRITE);
+
+	ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+			    0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ksft_test_result(ret == 1 && vec.len == 2,
+			 "%s Both pages written after remap and mprotect\n", __func__);
+
+	/* 4. Clear and make the pages written */
+	wp_addr_range(mem, 2 * page_size);
+
+	memset(mem, 'A', 2 * page_size);
+
+	ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN,
+			    0, 0, PAGE_IS_WRITTEN);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	ksft_test_result(ret == 1 && vec.len == 2,
+			 "%s Clear and make the pages written\n", __func__);
+
+	wp_free(mem, 2 * page_size);
+	munmap(mem, 2 * page_size);
+	return 0;
+}
+
+/* transact test */
+static const unsigned int nthreads = 6, pages_per_thread = 32, access_per_thread = 8;
+static pthread_barrier_t start_barrier, end_barrier;
+static unsigned int extra_thread_faults;
+static unsigned int iter_count = 1000;
+static volatile int finish;
+
+static ssize_t get_dirty_pages_reset(char *mem, unsigned int count,
+				     int reset, int page_size)
+{
+	struct pm_scan_arg arg = {0};
+	struct page_region rgns[256];
+	int i, j, cnt, ret;
+
+	arg.size = sizeof(struct pm_scan_arg);
+	arg.start = (uintptr_t)mem;
+	arg.max_pages = count;
+	arg.end = (uintptr_t)(mem + count * page_size);
+	arg.vec = (uintptr_t)rgns;
+	arg.vec_len = sizeof(rgns) / sizeof(*rgns);
+	arg.flags = PM_SCAN_OP_GET;
+	if (reset)
+		arg.flags |= PM_SCAN_OP_WP;
+	arg.required_mask = PAGE_IS_WRITTEN;
+	arg.return_mask = PAGE_IS_WRITTEN;
+
+	ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+	if (ret < 0)
+		ksft_exit_fail_msg("ioctl failed\n");
+
+	cnt = 0;
+	for (i = 0; i < ret; ++i) {
+		if (rgns[i].flags != PAGE_IS_WRITTEN)
+			ksft_exit_fail_msg("wrong flags\n");
+
+		for (j = 0; j < rgns[i].len; ++j)
+			cnt++;
+	}
+
+	return cnt;
+}
+
+void *thread_proc(void *mem)
+{
+	int *m = mem;
+	long curr_faults, faults;
+	struct rusage r;
+	unsigned int i;
+	int ret;
+
+	if (getrusage(RUSAGE_THREAD, &r))
+		ksft_exit_fail_msg("getrusage\n");
+
+	curr_faults = r.ru_minflt;
+
+	while (!finish) {
+		ret = pthread_barrier_wait(&start_barrier);
+		if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+			ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+		for (i = 0; i < access_per_thread; ++i)
+			__atomic_add_fetch(m + i * (0x1000 / sizeof(*m)), 1, __ATOMIC_SEQ_CST);
+
+		ret = pthread_barrier_wait(&end_barrier);
+		if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+			ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+		if (getrusage(RUSAGE_THREAD, &r))
+			ksft_exit_fail_msg("getrusage\n");
+
+		faults = r.ru_minflt - curr_faults;
+		if (faults < access_per_thread)
+			ksft_exit_fail_msg("faults < access_per_thread");
+
+		__atomic_add_fetch(&extra_thread_faults, faults - access_per_thread,
+				   __ATOMIC_SEQ_CST);
+		curr_faults = r.ru_minflt;
+	}
+
+	return NULL;
+}
+
+static void transact_test(int page_size)
+{
+	unsigned int i, count, extra_pages;
+	pthread_t th;
+	char *mem;
+	int ret, c;
+
+	if (pthread_barrier_init(&start_barrier, NULL, nthreads + 1))
+		ksft_exit_fail_msg("pthread_barrier_init\n");
+
+	if (pthread_barrier_init(&end_barrier, NULL, nthreads + 1))
+		ksft_exit_fail_msg("pthread_barrier_init\n");
+
+	mem = mmap(NULL, 0x1000 * nthreads * pages_per_thread, PROT_READ | PROT_WRITE,
+		   MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("Error mmap %s.\n", strerror(errno));
+
+	wp_init(mem, 0x1000 * nthreads * pages_per_thread);
+	wp_addr_range(mem, 0x1000 * nthreads * pages_per_thread);
+
+	memset(mem, 0, 0x1000 * nthreads * pages_per_thread);
+
+	count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size);
+	ksft_test_result(count > 0, "%s count %d\n", __func__, count);
+	count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size);
+	ksft_test_result(count == 0, "%s count %d\n", __func__, count);
+
+	finish = 0;
+	for (i = 0; i < nthreads; ++i)
+		pthread_create(&th, NULL, thread_proc, mem + 0x1000 * i * pages_per_thread);
+
+	extra_pages = 0;
+	for (i = 0; i < iter_count; ++i) {
+		count = 0;
+
+		ret = pthread_barrier_wait(&start_barrier);
+		if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+			ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+		count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1,
+					      page_size);
+
+		ret = pthread_barrier_wait(&end_barrier);
+		if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD)
+			ksft_exit_fail_msg("pthread_barrier_wait\n");
+
+		if (count > nthreads * access_per_thread)
+			ksft_exit_fail_msg("Too big count %d expected %d, iter %d\n",
+					   count, nthreads * access_per_thread, i);
+
+		c = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size);
+		count += c;
+
+		if (c > nthreads * access_per_thread) {
+			ksft_test_result_fail(" %s count > nthreads\n", __func__);
+			return;
+		}
+
+		if (count != nthreads * access_per_thread) {
+			/*
+			 * The purpose of the test is to make sure that no page updates are lost
+			 * when the page updates and read-resetting soft dirty flags are performed
+			 * in parallel. However, it is possible that the application will get the
+			 * soft dirty flags twice on the two consecutive read-resets. This seems
+			 * unavoidable as soft dirty flag is handled in software through page faults
+			 * in kernel. While the updating the flags is supposed to be synchronized
+			 * between page fault handling and read-reset, it is possible that
+			 * read-reset happens after page fault PTE update but before the application
+			 * re-executes write instruction. So read-reset gets the flag, clears write
+			 * access and application gets page fault again for the same write.
+			 */
+			if (count < nthreads * access_per_thread) {
+				ksft_test_result_fail("Lost update, iter %d, %d vs %d.\n", i, count,
+						      nthreads * access_per_thread);
+				return;
+			}
+
+			extra_pages += count - nthreads * access_per_thread;
+		}
+	}
+
+	pthread_barrier_wait(&start_barrier);
+	finish = 1;
+	pthread_barrier_wait(&end_barrier);
+
+	ksft_test_result_pass("%s Extra pages %u (%.1lf%%), extra thread faults %d.\n", __func__,
+			      extra_pages,
+			      100.0 * extra_pages / (iter_count * nthreads * access_per_thread),
+			      extra_thread_faults);
+}
+
+int main(void)
+{
+	int mem_size, shmid, buf_size, fd, i, ret;
+	char *mem, *map, *fmem;
+	struct stat sbuf;
+
+	ksft_print_header();
+	ksft_set_plan(92);
+
+	page_size = getpagesize();
+	hpage_size = read_pmd_pagesize();
+
+	pagemap_fd = open(PAGEMAP, O_RDWR);
+	if (pagemap_fd < 0)
+		return -EINVAL;
+
+	if (init_uffd())
+		ksft_exit_fail_msg("uffd init failed\n");
+
+	/*
+	 * Written (dirty) PTE bit tests
+	 */
+
+	/* 1. Sanity testing */
+	sanity_tests_sd();
+
+	/* 2. Normal page testing */
+	mem_size = 10 * page_size;
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	base_tests("Page testing:", mem, mem_size, 0);
+
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+
+	/* 3. Large page testing */
+	mem_size = 512 * 10 * page_size;
+	mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+	if (mem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem\n");
+	wp_init(mem, mem_size);
+	wp_addr_range(mem, mem_size);
+
+	base_tests("Large Page testing:", mem, mem_size, 0);
+
+	wp_free(mem, mem_size);
+	munmap(mem, mem_size);
+
+	/* 4. Huge page testing */
+	map = gethugepage(hpage_size);
+	if (map) {
+		wp_init(map, hpage_size);
+		wp_addr_range(map, hpage_size);
+		base_tests("Huge page testing:", map, hpage_size, 0);
+		wp_free(map, hpage_size);
+		free(map);
+	} else {
+		base_tests("Huge page testing:", NULL, 0, 1);
+	}
+
+	/* 5. Hugetlb page testing */
+	mem_size = 2*1024*1024;
+	mem = gethugetlb_mem(mem_size, &shmid);
+	if (mem) {
+		wp_init(mem, mem_size);
+		wp_addr_range(mem, mem_size);
+
+		base_tests("Hugetlb shmem testing:", mem, mem_size, 0);
+
+		wp_free(mem, mem_size);
+		shmctl(shmid, IPC_RMID, NULL);
+	} else {
+		base_tests("Hugetlb shmem testing:", NULL, 0, 1);
+	}
+
+	/* 6. Hugetlb page testing */
+	mem = gethugetlb_mem(mem_size, NULL);
+	if (mem) {
+		wp_init(mem, mem_size);
+		wp_addr_range(mem, mem_size);
+
+		base_tests("Hugetlb mem testing:", mem, mem_size, 0);
+
+		wp_free(mem, mem_size);
+	} else {
+		base_tests("Hugetlb mem testing:", NULL, 0, 1);
+	}
+
+	/* 7. file memory testing */
+	buf_size = page_size * 10;
+
+	fd = open(__FILE__".tmp0", O_RDWR | O_CREAT, 0777);
+	if (fd < 0)
+		ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n",
+				   strerror(errno));
+
+	for (i = 0; i < buf_size; i++)
+		if (write(fd, "c", 1) < 0)
+			ksft_exit_fail_msg("Create and read/write to a memory mapped file\n");
+
+	ret = stat(__FILE__".tmp0", &sbuf);
+	if (ret < 0)
+		ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+	fmem = mmap(NULL, sbuf.st_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+	if (fmem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+	wp_init(fmem, sbuf.st_size);
+	wp_addr_range(fmem, sbuf.st_size);
+
+	base_tests("File memory testing:", fmem, sbuf.st_size, 0);
+
+	wp_free(fmem, sbuf.st_size);
+	munmap(fmem, sbuf.st_size);
+	close(fd);
+
+	/* 8. file memory testing */
+	buf_size = page_size * 10;
+
+	fd = memfd_create(__FILE__".tmp00", MFD_NOEXEC_SEAL);
+	if (fd < 0)
+		ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n",
+				   strerror(errno));
+
+	if (ftruncate(fd, buf_size))
+		ksft_exit_fail_msg("Error ftruncate\n");
+
+	for (i = 0; i < buf_size; i++)
+		if (write(fd, "c", 1) < 0)
+			ksft_exit_fail_msg("Create and read/write to a memory mapped file\n");
+
+	fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+	if (fmem == MAP_FAILED)
+		ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno));
+
+	wp_init(fmem, buf_size);
+	wp_addr_range(fmem, buf_size);
+
+	base_tests("File anonymous memory testing:", fmem, buf_size, 0);
+
+	wp_free(fmem, buf_size);
+	munmap(fmem, buf_size);
+	close(fd);
+
+	/* 9. Huge page tests */
+	hpage_unit_tests();
+
+	/* 10. Iterative test */
+	test_simple();
+
+	/* 11. Mprotect test */
+	mprotect_tests();
+
+	/* 12. Transact test */
+	transact_test(page_size);
+
+	/*
+	 * Other PTE bit tests
+	 */
+
+	/* 1. Sanity testing */
+	sanity_tests();
+
+	/* 2. Unmapped address test */
+	unmapped_region_tests();
+
+	/* 3. Userfaultfd tests */
+	userfaultfd_tests();
+
+	close(pagemap_fd);
+	return ksft_exit_pass();
+}
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
old mode 100644
new mode 100755
index 9666c0c171ab..cca4dd8ee45d
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -56,6 +56,8 @@ separated by spaces:
 	memory protection key tests
 - soft_dirty
 	test soft dirty page bit semantics
+- pagemap
+	test pagemap_scan IOCTL
 - cow
 	test copy-on-write semantics
 example: ./run_vmtests.sh -t "hmm mmap ksm"
@@ -324,6 +326,8 @@ fi
 
 CATEGORY="soft_dirty" run_test ./soft-dirty
 
+CATEGORY="pagemap" run_test ./pagemap_ioctl
+
 # COW tests
 CATEGORY="cow" run_test ./cow
 
-- 
2.39.2

Re: [PATCH v25 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs

[Andrei Vagin]

On Thu, Jul 13, 2023 at 3:14 AM Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
> the info about page table entries. The following operations are supported
> in this ioctl:
> - Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
>   file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT), swapped
>   (PAGE_IS_SWAPPED) or page has pfn zero (PAGE_IS_PFNZERO).
> - Find pages which have been written-to and/or write protect the pages
>   (atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP)
>
> This IOCTL can be extended to get information about more PTE bits. The
> entire address range passed by user [start, end) is scanned until either
> the user provided buffer is full or max_pages have been found.
>

Reviewed-by: Andrei Vagin <avagin@gmail.com>

Here are a few inline comments.

> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> ---
> Changes in v25:
> - Do proper filtering on hole as well (hole got missed earlier)
>
> Changes in v24:
> - Place WP markers in case of hole as well
>
> Changes in v23:
> - Set vec_buf_index to 0 only when vec_buf_index is set
> - Return -EFAULT instead of -EINVAL if vec is NULL
> - Correctly return the walk ending address to the page granularity
>
> Changes in v22:
> - Interface change to return walk ending address to user:
>   - Replace [start start + len) with [start, end)
>   - Return the ending address of the address walk in start
>
> Changes in v21:
> - Abort walk instead of returning error if WP is to be performed on
>   partial hugetlb
> - Changed the data types of some variables in pagemap_scan_private to
>   long
>
> Changes in v20:
> - Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
>
> Changes in v19:
> - Interface changes such as renaming, return mask and WP can be used
>   with any flags specified in masks
> - Internal code changes
>
> Changes in v18:
> - Rebased on top of next-20230613
>   - ptep_get() updates
>   - remove pmd_trans_unstable() and add ACTION_AGAIN
> - Review updates (Micheal)
>
> Changes in v17:
> - Rebased on next-20230606
> - Made make_uffd_wp_*_pte() better and minor changes
>
> Changes in v16:
> - Fixed a corner case where kernel writes beyond user buffer by one
>   element
> - Bring back exclusive PM_SCAN_OP_WP
> - Cosmetic changes
>
> Changes in v15:
> - Build fix:
>   - Use generic tlb flush function in pagemap_scan_pmd_entry() instead of
>     using x86 specific flush function in do_pagemap_scan()
>   - Remove #ifdef from pagemap_scan_hugetlb_entry()
>   - Use mm instead of undefined vma->vm_mm
>
> Changes in v14:
> - Fix build error caused by #ifdef added at last minute in some configs
>
> Changes in v13:
> - Review updates
> - mmap_read_lock_killable() instead of mmap_read_lock()
> - Replace uffd_wp_range() with helpers which increases performance
>   drastically for OP_WP operations by reducing the number of tlb
>   flushing etc
> - Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range
>
> Changes in v12:
> - Add hugetlb support to cover all memory types
> - Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
> - Review updates to the code
>
> Changes in v11:
> - Find written pages in a better way
> - Fix a corner case (thanks Paul)
> - Improve the code/comments
> - remove ENGAGE_WP + ! GET operation
> - shorten the commit message in favour of moving documentation to
>   pagemap.rst
>
> Changes in v10:
> - move changes in tools/include/uapi/linux/fs.h to separate patch
> - update commit message
>
> Change in v8:
> - Correct is_pte_uffd_wp()
> - Improve readability and error checks
> - Remove some un-needed code
>
> Changes in v7:
> - Rebase on top of latest next
> - Fix some corner cases
> - Base soft-dirty on the uffd wp async
> - Update the terminologies
> - Optimize the memory usage inside the ioctl
> ---
>  fs/proc/task_mmu.c      | 591 ++++++++++++++++++++++++++++++++++++++++
>  include/linux/hugetlb.h |   1 +
>  include/uapi/linux/fs.h |  55 ++++
>  mm/hugetlb.c            |   2 +-
>  4 files changed, 648 insertions(+), 1 deletion(-)
>
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index cfab855fe7e9..596fdfdc4387 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -19,6 +19,7 @@
>  #include <linux/shmem_fs.h>
>  #include <linux/uaccess.h>
>  #include <linux/pkeys.h>
> +#include <linux/minmax.h>
>
>  #include <asm/elf.h>
>  #include <asm/tlb.h>
> @@ -1768,11 +1769,601 @@ static int pagemap_release(struct inode *inode, struct file *file)
>         return 0;
>  }
>
> +#define PM_SCAN_REQUIRE_UFFD   (1ULL << 63)
> +
> +#define PM_SCAN_FOUND_MAX_PAGES        (1)
> +#define PM_SCAN_END_WALK       (-256)

I would add a comment to explain why one is negative and another one
is positive.

> +
> +#define PM_SCAN_BITS_ALL       (PAGE_IS_WRITTEN | PAGE_IS_FILE |       \
> +                                PAGE_IS_PRESENT | PAGE_IS_SWAPPED |    \
> +                                PAGE_IS_PFNZERO)
> +#define PM_SCAN_OPS            (PM_SCAN_OP_GET | PM_SCAN_OP_WP)
> +#define IS_PM_SCAN_GET(flags)  (flags & PM_SCAN_OP_GET)
> +#define IS_PM_SCAN_WP(flags)   (flags & PM_SCAN_OP_WP)
> +
> +#define PM_SCAN_FLAGS(wt, file, present, swap, pfnzero)        \
> +       ((wt) | ((file) << 1) | ((present) << 2) |      \
> +        ((swap) << 3) | ((pfnzero) << 4))

it is better to use PAGE_IS_* constants here

> +
> +struct pagemap_scan_private {
> +       struct page_region *vec_buf, cur_buf;
> +       unsigned long vec_buf_len, vec_buf_index, max_pages, found_pages;
> +       unsigned long long flags, required_mask, anyof_mask, excluded_mask, return_mask, end_addr;

end_addr has to be unsigned long.

> +};
> +
> +static inline bool is_pte_uffd_wp(pte_t pte)
> +{
> +       return (pte_present(pte) && pte_uffd_wp(pte)) ||
> +              pte_swp_uffd_wp_any(pte);
> +}
> +
> +static inline void make_uffd_wp_pte(struct vm_area_struct *vma,
> +                                   unsigned long addr, pte_t *pte)
> +{
> +       pte_t ptent = ptep_get(pte);
> +
> +       if (pte_present(ptent)) {
> +               pte_t old_pte;
> +
> +               old_pte = ptep_modify_prot_start(vma, addr, pte);
> +               ptent = pte_mkuffd_wp(ptent);
> +               ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
> +       } else if (is_swap_pte(ptent)) {
> +               ptent = pte_swp_mkuffd_wp(ptent);
> +               set_pte_at(vma->vm_mm, addr, pte, ptent);
> +       } else {
> +               set_pte_at(vma->vm_mm, addr, pte,
> +                          make_pte_marker(PTE_MARKER_UFFD_WP));
> +       }
> +}
> +
> +static inline bool pagemap_scan_is_file(struct vm_area_struct *vma, pte_t ptent,
> +                                       unsigned long addr)
> +{
> +       struct page *page = NULL;
> +       swp_entry_t entry;
> +
> +       if (pte_present(ptent)) {
> +               page = vm_normal_page(vma, addr, ptent);
> +       } else {
> +               entry = pte_to_swp_entry(ptent);
> +               if (is_pfn_swap_entry(entry))
> +                       page = pfn_swap_entry_to_page(entry);
> +       }
> +
> +       if (page && !PageAnon(page))
> +               return true;
> +
> +       return false;
> +}
> +
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> +static inline bool is_pmd_uffd_wp(pmd_t pmd)
> +{
> +       return (pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
> +              (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd));
> +}
> +
> +static inline void make_uffd_wp_pmd(struct vm_area_struct *vma,
> +                                   unsigned long addr, pmd_t *pmdp)
> +{
> +       pmd_t old, pmd = *pmdp;
> +
> +       if (pmd_present(pmd)) {
> +               old = pmdp_invalidate_ad(vma, addr, pmdp);
> +               pmd = pmd_mkuffd_wp(old);
> +               set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
> +       } else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
> +               pmd = pmd_swp_mkuffd_wp(pmd);
> +               set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
> +       }
> +}
> +#endif
> +
> +#ifdef CONFIG_HUGETLB_PAGE
> +static inline bool is_huge_pte_uffd_wp(pte_t pte)
> +{
> +       return (pte_present(pte) && huge_pte_uffd_wp(pte)) ||
> +              pte_swp_uffd_wp_any(pte);
> +}
> +
> +static inline void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
> +                                        unsigned long addr, pte_t *ptep,
> +                                        pte_t ptent)
> +{
> +       if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
> +               return;
> +
> +       if (is_hugetlb_entry_migration(ptent))
> +               set_huge_pte_at(vma->vm_mm, addr, ptep,
> +                               pte_swp_mkuffd_wp(ptent));
> +       else if (!huge_pte_none(ptent))
> +               huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
> +                                            huge_pte_mkuffd_wp(ptent));
> +       else
> +               set_huge_pte_at(vma->vm_mm, addr, ptep,
> +                               make_pte_marker(PTE_MARKER_UFFD_WP));
> +}
> +
> +static inline bool pagemap_scan_is_huge_file(pte_t pte)
> +{
> +       if (pte_present(pte) && (!PageAnon(pte_page(pte))))
> +               return true;
> +
> +       return false;
> +}
> +#endif
> +
> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
> +                                 struct mm_walk *walk)
> +{
> +       struct pagemap_scan_private *p = walk->private;
> +       struct vm_area_struct *vma = walk->vma;
> +
> +       if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) ||
> +           !userfaultfd_wp_use_markers(vma)))
> +               return -EPERM;
> +
> +       if (vma->vm_flags & VM_PFNMAP)
> +               return 1;
> +
> +       return 0;
> +}
> +
> +static int pagemap_scan_output(unsigned long bitmap,
> +                              struct pagemap_scan_private *p,
> +                              unsigned long addr, unsigned int n_pages)
> +{
> +       struct page_region *cur_buf = &p->cur_buf;
> +
> +       bitmap &= p->return_mask;
> +
> +       if (cur_buf->flags == bitmap &&
> +           cur_buf->start + cur_buf->len * PAGE_SIZE == addr) {
> +               cur_buf->len += n_pages;
> +               p->found_pages += n_pages;
> +       } else {
> +               if (cur_buf->len) {
> +                       if (p->vec_buf_index >= p->vec_buf_len) {
> +                               p->end_addr = addr;
> +                               return PM_SCAN_END_WALK;
> +                       }
> +
> +                       memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
> +                              sizeof(*p->vec_buf));
> +                       p->vec_buf_index++;
> +               }
> +
> +               cur_buf->start = addr;
> +               cur_buf->len = n_pages;
> +               cur_buf->flags = bitmap;
> +               p->found_pages += n_pages;
> +       }
> +
> +       if (p->found_pages == p->max_pages) {
> +               p->end_addr = addr + n_pages * PAGE_SIZE;
> +               return PM_SCAN_FOUND_MAX_PAGES;
> +       }
> +
> +       return 0;
> +}
> +
> +static bool pagemap_scan_is_interesting_page(unsigned long bitmap,
> +                                            struct pagemap_scan_private *p)
> +{
> +       if ((p->required_mask & bitmap) != p->required_mask)
> +               return false;
> +       if (p->anyof_mask && !(p->anyof_mask & bitmap))
> +               return false;
> +       if (p->excluded_mask & bitmap)
> +               return false;
> +
> +       return true;
> +}
> +
> +static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
> +                                 unsigned long end, struct mm_walk *walk)
> +{
> +       bool is_written, flush = false, interesting = true;
> +       struct pagemap_scan_private *p = walk->private;
> +       struct vm_area_struct *vma = walk->vma;
> +       unsigned long bitmap, addr = end;
> +       pte_t *pte, *orig_pte, ptent;
> +       spinlock_t *ptl;
> +       int ret = 0;
> +
> +       arch_enter_lazy_mmu_mode();
> +
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> +       ptl = pmd_trans_huge_lock(pmd, vma);
> +       if (ptl) {
> +               unsigned long n_pages = (end - start)/PAGE_SIZE;
> +
> +               if (n_pages > p->max_pages - p->found_pages)
> +                       n_pages = p->max_pages - p->found_pages;
> +
> +               is_written = !is_pmd_uffd_wp(*pmd);
> +
> +               bitmap = PM_SCAN_FLAGS(is_written, false,
> +                                      pmd_present(*pmd), is_swap_pmd(*pmd),
> +                                      pmd_present(*pmd) && is_zero_pfn(pmd_pfn(*pmd)));
> +
> +               if (IS_PM_SCAN_GET(p->flags))
> +                       interesting = pagemap_scan_is_interesting_page(bitmap, p);
> +
> +               if (interesting) {
> +                       /*
> +                        * Break huge page into small pages if the WP operation
> +                        * need to be performed is on a portion of the huge page.
> +                        */
> +                       if (is_written && IS_PM_SCAN_WP(p->flags) &&
> +                           n_pages < HPAGE_SIZE/PAGE_SIZE) {
> +                               spin_unlock(ptl);
> +
> +                               split_huge_pmd(vma, pmd, start);
> +                               goto process_smaller_pages;
> +                       }
> +
> +                       if (IS_PM_SCAN_GET(p->flags))
> +                               ret = pagemap_scan_output(bitmap, p, start, n_pages);
> +
> +                       if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) {
> +                               make_uffd_wp_pmd(vma, start, pmd);
> +                               flush_tlb_range(vma, start, end);
> +                       }
> +               }
> +
> +               spin_unlock(ptl);
> +               arch_leave_lazy_mmu_mode();
> +
> +               return ret;
> +       }
> +
> +process_smaller_pages:
> +#endif
> +
> +       orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
> +       if (!pte) {
> +               walk->action = ACTION_AGAIN;
> +               return 0;
> +       }
> +
> +       for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE) {
> +               ptent = ptep_get(pte);
> +               is_written = !is_pte_uffd_wp(ptent);
> +
> +               bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_file(vma, ptent, addr),
> +                                      pte_present(ptent), is_swap_pte(ptent),
> +                                      pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
> +
> +               if (IS_PM_SCAN_GET(p->flags)) {
> +                       interesting = pagemap_scan_is_interesting_page(bitmap, p);
> +                       if (interesting)
> +                               ret = pagemap_scan_output(bitmap, p, addr, 1);
> +               }
> +
> +               if (IS_PM_SCAN_WP(p->flags) && is_written && interesting &&
> +                   ret >= 0) {
> +                       make_uffd_wp_pte(vma, addr, pte);
> +                       flush = true;
> +               }
> +       }
> +
> +       if (flush)
> +               flush_tlb_range(vma, start, addr);
> +
> +       pte_unmap_unlock(orig_pte, ptl);
> +       arch_leave_lazy_mmu_mode();
> +
> +       cond_resched();
> +       return ret;
> +}
> +
> +#ifdef CONFIG_HUGETLB_PAGE
> +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
> +                                     unsigned long start, unsigned long end,
> +                                     struct mm_walk *walk)
> +{
> +       unsigned long n_pages = (end - start)/PAGE_SIZE;
> +       struct pagemap_scan_private *p = walk->private;
> +       struct vm_area_struct *vma = walk->vma;
> +       bool is_written, interesting = true;
> +       struct hstate *h = hstate_vma(vma);
> +       unsigned long bitmap;
> +       spinlock_t *ptl;
> +       int ret = 0;
> +       pte_t ptent;
> +
> +       if (IS_PM_SCAN_WP(p->flags) && n_pages < HPAGE_SIZE/PAGE_SIZE) {
> +               p->end_addr = start;
> +               return -EINVAL;
> +       }
> +
> +       if (n_pages > p->max_pages - p->found_pages)
> +               n_pages = p->max_pages - p->found_pages;
> +
> +       if (IS_PM_SCAN_WP(p->flags)) {
> +               i_mmap_lock_write(vma->vm_file->f_mapping);
> +               ptl = huge_pte_lock(h, vma->vm_mm, ptep);
> +       }
> +
> +       ptent = huge_ptep_get(ptep);
> +       is_written = !is_huge_pte_uffd_wp(ptent);
> +
> +       bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_huge_file(ptent),
> +                              pte_present(ptent), is_swap_pte(ptent),
> +                              pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
> +
> +       if (IS_PM_SCAN_GET(p->flags))
> +               interesting = pagemap_scan_is_interesting_page(bitmap, p);
> +
> +       if (interesting) {
> +               /*
> +                * Partial hugetlb page clear isn't supported
> +                */
> +               if (is_written && IS_PM_SCAN_WP(p->flags) &&
> +                   n_pages < HPAGE_SIZE/PAGE_SIZE) {
> +                       ret = PM_SCAN_END_WALK;
> +                       p->end_addr = start;
> +                       goto unlock_and_return;
> +               }
> +
> +               if (IS_PM_SCAN_GET(p->flags))
> +                       ret = pagemap_scan_output(bitmap, p, start, n_pages);
> +
> +               if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) {
> +                       make_uffd_wp_huge_pte(vma, start, ptep, ptent);
> +                       flush_hugetlb_tlb_range(vma, start, end);
> +               }
> +       }
> +
> +unlock_and_return:
> +       if (IS_PM_SCAN_WP(p->flags)) {
> +               spin_unlock(ptl);
> +               i_mmap_unlock_write(vma->vm_file->f_mapping);
> +       }
> +
> +       return ret;
> +}
> +#else
> +#define pagemap_scan_hugetlb_entry NULL
> +#endif
> +
> +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
> +                                int depth, struct mm_walk *walk)
> +{
> +       unsigned long n_pages = (end - addr)/PAGE_SIZE;
> +       struct pagemap_scan_private *p = walk->private;
> +       struct vm_area_struct *vma = walk->vma;
> +       bool interesting = true;
> +       unsigned long bitmap;
> +       int ret = 0;
> +
> +       if (!vma)
> +               return 0;
> +
> +       bitmap = PM_SCAN_FLAGS(false, false, false, false, false);
> +
> +       if (IS_PM_SCAN_GET(p->flags))
> +               interesting = pagemap_scan_is_interesting_page(bitmap, p);
> +
> +       if (interesting) {
> +               if (IS_PM_SCAN_GET(p->flags)) {
> +                       if (n_pages > p->max_pages - p->found_pages)
> +                               n_pages = p->max_pages - p->found_pages;
> +
> +                       ret = pagemap_scan_output(bitmap, p, addr, n_pages);
> +               }
> +
> +               if (IS_PM_SCAN_WP(p->flags) && !ret &&
> +                   uffd_wp_range(vma, addr, end - addr, true) < 0)

Why do we need to call uffd_wp_range for holes? Should we call
flush_tlb_range after it?

> +                       ret = -EINVAL;
> +       }
> +
> +       return ret;
> +}
> +
> +static const struct mm_walk_ops pagemap_scan_ops = {
> +       .test_walk = pagemap_scan_test_walk,
> +       .pmd_entry = pagemap_scan_pmd_entry,
> +       .pte_hole = pagemap_scan_pte_hole,
> +       .hugetlb_entry = pagemap_scan_hugetlb_entry,
> +};
> +
> +static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start,
> +                                  unsigned long end, struct page_region __user *vec)
> +{
> +       /* Detect illegal size, flags, len and masks */
> +       if (arg->size != sizeof(struct pm_scan_arg))
> +               return -EINVAL;
> +       if (!arg->flags)
> +               return -EINVAL;
> +       if (arg->flags & ~PM_SCAN_OPS)
> +               return -EINVAL;
> +       if (!(end - start))
> +               return -EINVAL;
> +       if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask |
> +            arg->return_mask) & ~PM_SCAN_BITS_ALL)
> +               return -EINVAL;
> +       if (!arg->required_mask && !arg->anyof_mask &&
> +           !arg->excluded_mask)
> +               return -EINVAL;
> +       if (!arg->return_mask)
> +               return -EINVAL;
> +
> +       /* Validate memory range */
> +       if (!IS_ALIGNED(start, PAGE_SIZE))
> +               return -EINVAL;
> +       if (!access_ok((void __user *)start, end - start))
> +               return -EFAULT;
> +
> +       if (IS_PM_SCAN_GET(arg->flags)) {
> +               if (arg->vec_len == 0)
> +                       return -EINVAL;
> +               if (!vec)
> +                       return -EFAULT;
> +               if (!access_ok((void __user *)vec,
> +                              arg->vec_len * sizeof(struct page_region)))
> +                       return -EFAULT;
> +       }
> +
> +       if (IS_PM_SCAN_WP(arg->flags) && !IS_PM_SCAN_GET(arg->flags) &&
> +           arg->max_pages)
> +               return -EINVAL;
> +
> +       return 0;
> +}
> +
> +static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg)
> +{
> +       struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg;
> +       unsigned long long start, end, walk_start, walk_end;
> +       unsigned long empty_slots, vec_index = 0;
> +       struct mmu_notifier_range range;
> +       struct page_region __user *vec;
> +       struct pagemap_scan_private p;
> +       struct pm_scan_arg arg;
> +       int ret = 0;
> +
> +       if (copy_from_user(&arg, uarg, sizeof(arg)))
> +               return -EFAULT;
> +
> +       start = untagged_addr((unsigned long)arg.start);
> +       end = untagged_addr((unsigned long)arg.end);
> +       vec = (struct page_region __user *)untagged_addr((unsigned long)arg.vec);
> +
> +       ret = pagemap_scan_args_valid(&arg, start, end, vec);
> +       if (ret)
> +               return ret;
> +
> +       p.max_pages = (arg.max_pages) ? arg.max_pages : ULONG_MAX;
> +       p.found_pages = 0;
> +       p.required_mask = arg.required_mask;
> +       p.anyof_mask = arg.anyof_mask;
> +       p.excluded_mask = arg.excluded_mask;
> +       p.return_mask = arg.return_mask;
> +       p.flags = arg.flags;
> +       p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) &
> +                   PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0;
> +       p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0;
> +       p.vec_buf = NULL;
> +       p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
> +       p.vec_buf_index = 0;
> +       p.end_addr = 0;
> +
> +       /*
> +        * Allocate smaller buffer to get output from inside the page walk
> +        * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
> +        * we want to return output to user in compact form where no two
> +        * consecutive regions should be continuous and have the same flags.
> +        * So store the latest element in p.cur_buf between different walks and
> +        * store the p.cur_buf at the end of the walk to the user buffer.
> +        */
> +       if (IS_PM_SCAN_GET(p.flags)) {
> +               p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf),
> +                                         GFP_KERNEL);
> +               if (!p.vec_buf)
> +                       return -ENOMEM;
> +       }
> +
> +       /*
> +        * Protection change for the range is going to happen.
> +        */
> +       if (IS_PM_SCAN_WP(p.flags)) {
> +               mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
> +                                       mm, start, end);
> +               mmu_notifier_invalidate_range_start(&range);
> +       }
> +
> +       walk_start = walk_end = start;
> +       while (walk_end < end && !ret) {
> +               if (IS_PM_SCAN_GET(p.flags)) {
> +                       /*
> +                        * All data is copied to cur_buf first. When more data
> +                        * is found, we push cur_buf to vec_buf and copy new
> +                        * data to cur_buf. Subtract 1 from length as the
> +                        * index of cur_buf isn't counted in length.
> +                        */
> +                       empty_slots = arg.vec_len - vec_index;
> +                       p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1);
> +               }
> +

I still don't understand why we don't want/need to check for pending signals.

> +               ret = mmap_read_lock_killable(mm);
> +               if (ret)
> +                       goto out;
> +
> +               walk_end = min((walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK, end);
> +
> +               ret = walk_page_range(mm, walk_start, walk_end,
> +                                     &pagemap_scan_ops, &p);
> +               mmap_read_unlock(mm);
> +
> +               if (ret == PM_SCAN_FOUND_MAX_PAGES || ret == PM_SCAN_END_WALK)
> +                       arg.start = p.end_addr;

nit: this check can be moved out of the loop.

> +
> +               if (ret && ret != PM_SCAN_FOUND_MAX_PAGES &&
> +                   ret != PM_SCAN_END_WALK)
> +                       goto out;
> +
> +               if (p.vec_buf_index) {
> +                       if (copy_to_user(&vec[vec_index], p.vec_buf,
> +                                        p.vec_buf_index * sizeof(*p.vec_buf))) {
> +                               /*
> +                                * Return error even though the OP succeeded
> +                                */
> +                               ret = -EFAULT;
> +                               goto out;
> +                       }
> +                       vec_index += p.vec_buf_index;
> +                       p.vec_buf_index = 0;
> +               }
> +               walk_start = walk_end;
> +       }
> +
> +       if (p.cur_buf.len) {
> +               if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) {
> +                       ret = -EFAULT;
> +                       goto out;
> +               }
> +               vec_index++;
> +       }
> +
> +       ret = vec_index;
> +
> +out:
> +       if (!p.end_addr)
> +               arg.start = walk_start;
> +       if (copy_to_user(&uarg->start, &arg.start, sizeof(arg.start)))
> +               ret = -EFAULT;
> +
> +       if (IS_PM_SCAN_WP(p.flags))
> +               mmu_notifier_invalidate_range_end(&range);
> +
> +       kfree(p.vec_buf);
> +       return ret;
> +}
> +
> +static long do_pagemap_cmd(struct file *file, unsigned int cmd,
> +                          unsigned long arg)
> +{
> +       struct mm_struct *mm = file->private_data;
> +
> +       switch (cmd) {
> +       case PAGEMAP_SCAN:
> +               return do_pagemap_scan(mm, arg);
> +
> +       default:
> +               return -EINVAL;
> +       }
> +}
> +
>  const struct file_operations proc_pagemap_operations = {
>         .llseek         = mem_lseek, /* borrow this */
>         .read           = pagemap_read,
>         .open           = pagemap_open,
>         .release        = pagemap_release,
> +       .unlocked_ioctl = do_pagemap_cmd,
> +       .compat_ioctl   = do_pagemap_cmd,
>  };
>  #endif /* CONFIG_PROC_PAGE_MONITOR */
>
> diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
> index 623e98d62df3..c44e7bde0339 100644
> --- a/include/linux/hugetlb.h
> +++ b/include/linux/hugetlb.h
> @@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
>                 unsigned long cp_flags);
>
>  bool is_hugetlb_entry_migration(pte_t pte);
> +bool is_hugetlb_entry_hwpoisoned(pte_t pte);
>  void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
>
>  #else /* !CONFIG_HUGETLB_PAGE */
> diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> index b7b56871029c..ac684d99e68f 100644
> --- a/include/uapi/linux/fs.h
> +++ b/include/uapi/linux/fs.h
> @@ -305,4 +305,59 @@ typedef int __bitwise __kernel_rwf_t;
>  #define RWF_SUPPORTED  (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
>                          RWF_APPEND)
>
> +/* Pagemap ioctl */
> +#define PAGEMAP_SCAN   _IOWR('f', 16, struct pm_scan_arg)
> +
> +/* Bits are set in flags of the page_region and masks in pm_scan_args */
> +#define PAGE_IS_WRITTEN                (1 << 0)
> +#define PAGE_IS_FILE           (1 << 1)
> +#define PAGE_IS_PRESENT                (1 << 2)
> +#define PAGE_IS_SWAPPED                (1 << 3)
> +#define PAGE_IS_PFNZERO                (1 << 4)
> +
> +/*
> + * struct page_region - Page region with flags
> + * @start:     Start of the region
> + * @len:       Length of the region in pages
> + * @bitmap:    Bits sets for the region
> + */
> +struct page_region {
> +       __u64 start;
> +       __u64 len;
> +       __u64 flags;
> +};
> +
> +/*
> + * struct pm_scan_arg - Pagemap ioctl argument
> + * @size:              Size of the structure
> + * @flags:             Flags for the IOCTL
> + * @start:             Starting address of the region
> + *                     (Ending address of the walk is also returned in it)
> + * @end:               Ending address of the region
> + * @vec:               Address of page_region struct array for output
> + * @vec_len:           Length of the page_region struct array
> + * @max_pages:         Optional max return pages
> + * @required_mask:     Required mask - All of these bits have to be set in the PTE
> + * @anyof_mask:                Any mask - Any of these bits are set in the PTE
> + * @excluded_mask:     Exclude mask - None of these bits are set in the PTE
> + * @return_mask:       Bits that are to be reported in page_region
> + */
> +struct pm_scan_arg {
> +       __u64 size;
> +       __u64 flags;
> +       __u64 start;
> +       __u64 end;
> +       __u64 vec;
> +       __u64 vec_len;
> +       __u64 max_pages;
> +       __u64 required_mask;
> +       __u64 anyof_mask;
> +       __u64 excluded_mask;
> +       __u64 return_mask;
> +};
> +
> +/* Supported flags */
> +#define PM_SCAN_OP_GET (1 << 0)
> +#define PM_SCAN_OP_WP  (1 << 1)
> +
>  #endif /* _UAPI_LINUX_FS_H */
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index ee88a2a6649f..f7cefdcb114c 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -4981,7 +4981,7 @@ bool is_hugetlb_entry_migration(pte_t pte)
>                 return false;
>  }
>
> -static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
> +bool is_hugetlb_entry_hwpoisoned(pte_t pte)
>  {
>         swp_entry_t swp;
>
> --
> 2.39.2
>

Re: [PATCH v25 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs

[Muhammad Usama Anjum]

On 7/17/23 10:26 PM, Andrei Vagin wrote:
> On Thu, Jul 13, 2023 at 3:14 AM Muhammad Usama Anjum
> <usama.anjum@collabora.com> wrote:
>>
>> This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
>> the info about page table entries. The following operations are supported
>> in this ioctl:
>> - Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
>>   file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT), swapped
>>   (PAGE_IS_SWAPPED) or page has pfn zero (PAGE_IS_PFNZERO).
>> - Find pages which have been written-to and/or write protect the pages
>>   (atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP)
>>
>> This IOCTL can be extended to get information about more PTE bits. The
>> entire address range passed by user [start, end) is scanned until either
>> the user provided buffer is full or max_pages have been found.
>>
> 
> Reviewed-by: Andrei Vagin <avagin@gmail.com>
Thank you.

> 
> Here are a few inline comments.
> 
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>> ---
>> Changes in v25:
>> - Do proper filtering on hole as well (hole got missed earlier)
>>
>> Changes in v24:
>> - Place WP markers in case of hole as well
>>
>> Changes in v23:
>> - Set vec_buf_index to 0 only when vec_buf_index is set
>> - Return -EFAULT instead of -EINVAL if vec is NULL
>> - Correctly return the walk ending address to the page granularity
>>
>> Changes in v22:
>> - Interface change to return walk ending address to user:
>>   - Replace [start start + len) with [start, end)
>>   - Return the ending address of the address walk in start
>>
>> Changes in v21:
>> - Abort walk instead of returning error if WP is to be performed on
>>   partial hugetlb
>> - Changed the data types of some variables in pagemap_scan_private to
>>   long
>>
>> Changes in v20:
>> - Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
>>
>> Changes in v19:
>> - Interface changes such as renaming, return mask and WP can be used
>>   with any flags specified in masks
>> - Internal code changes
>>
>> Changes in v18:
>> - Rebased on top of next-20230613
>>   - ptep_get() updates
>>   - remove pmd_trans_unstable() and add ACTION_AGAIN
>> - Review updates (Micheal)
>>
>> Changes in v17:
>> - Rebased on next-20230606
>> - Made make_uffd_wp_*_pte() better and minor changes
>>
>> Changes in v16:
>> - Fixed a corner case where kernel writes beyond user buffer by one
>>   element
>> - Bring back exclusive PM_SCAN_OP_WP
>> - Cosmetic changes
>>
>> Changes in v15:
>> - Build fix:
>>   - Use generic tlb flush function in pagemap_scan_pmd_entry() instead of
>>     using x86 specific flush function in do_pagemap_scan()
>>   - Remove #ifdef from pagemap_scan_hugetlb_entry()
>>   - Use mm instead of undefined vma->vm_mm
>>
>> Changes in v14:
>> - Fix build error caused by #ifdef added at last minute in some configs
>>
>> Changes in v13:
>> - Review updates
>> - mmap_read_lock_killable() instead of mmap_read_lock()
>> - Replace uffd_wp_range() with helpers which increases performance
>>   drastically for OP_WP operations by reducing the number of tlb
>>   flushing etc
>> - Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range
>>
>> Changes in v12:
>> - Add hugetlb support to cover all memory types
>> - Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
>> - Review updates to the code
>>
>> Changes in v11:
>> - Find written pages in a better way
>> - Fix a corner case (thanks Paul)
>> - Improve the code/comments
>> - remove ENGAGE_WP + ! GET operation
>> - shorten the commit message in favour of moving documentation to
>>   pagemap.rst
>>
>> Changes in v10:
>> - move changes in tools/include/uapi/linux/fs.h to separate patch
>> - update commit message
>>
>> Change in v8:
>> - Correct is_pte_uffd_wp()
>> - Improve readability and error checks
>> - Remove some un-needed code
>>
>> Changes in v7:
>> - Rebase on top of latest next
>> - Fix some corner cases
>> - Base soft-dirty on the uffd wp async
>> - Update the terminologies
>> - Optimize the memory usage inside the ioctl
>> ---
>>  fs/proc/task_mmu.c      | 591 ++++++++++++++++++++++++++++++++++++++++
>>  include/linux/hugetlb.h |   1 +
>>  include/uapi/linux/fs.h |  55 ++++
>>  mm/hugetlb.c            |   2 +-
>>  4 files changed, 648 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>> index cfab855fe7e9..596fdfdc4387 100644
>> --- a/fs/proc/task_mmu.c
>> +++ b/fs/proc/task_mmu.c
>> @@ -19,6 +19,7 @@
>>  #include <linux/shmem_fs.h>
>>  #include <linux/uaccess.h>
>>  #include <linux/pkeys.h>
>> +#include <linux/minmax.h>
>>
>>  #include <asm/elf.h>
>>  #include <asm/tlb.h>
>> @@ -1768,11 +1769,601 @@ static int pagemap_release(struct inode *inode, struct file *file)
>>         return 0;
>>  }
>>
>> +#define PM_SCAN_REQUIRE_UFFD   (1ULL << 63)
>> +
>> +#define PM_SCAN_FOUND_MAX_PAGES        (1)
>> +#define PM_SCAN_END_WALK       (-256)
> 
> I would add a comment to explain why one is negative and another one
> is positive.
Sure. I'll add.

> 
>> +
>> +#define PM_SCAN_BITS_ALL       (PAGE_IS_WRITTEN | PAGE_IS_FILE |       \
>> +                                PAGE_IS_PRESENT | PAGE_IS_SWAPPED |    \
>> +                                PAGE_IS_PFNZERO)
>> +#define PM_SCAN_OPS            (PM_SCAN_OP_GET | PM_SCAN_OP_WP)
>> +#define IS_PM_SCAN_GET(flags)  (flags & PM_SCAN_OP_GET)
>> +#define IS_PM_SCAN_WP(flags)   (flags & PM_SCAN_OP_WP)
>> +
>> +#define PM_SCAN_FLAGS(wt, file, present, swap, pfnzero)        \
>> +       ((wt) | ((file) << 1) | ((present) << 2) |      \
>> +        ((swap) << 3) | ((pfnzero) << 4))
> 
> it is better to use PAGE_IS_* constants here
#define PAGE_IS_PFNZERO		(1 << 4)
PAGE_IS_PFNZERO macro cannot be used here.

> 
>> +
>> +struct pagemap_scan_private {
>> +       struct page_region *vec_buf, cur_buf;
>> +       unsigned long vec_buf_len, vec_buf_index, max_pages, found_pages;
>> +       unsigned long long flags, required_mask, anyof_mask, excluded_mask, return_mask, end_addr;
> 
> end_addr has to be unsigned long.
Sure.

> 
>> +};
>> +
>> +static inline bool is_pte_uffd_wp(pte_t pte)
>> +{
>> +       return (pte_present(pte) && pte_uffd_wp(pte)) ||
>> +              pte_swp_uffd_wp_any(pte);
>> +}
>> +
>> +static inline void make_uffd_wp_pte(struct vm_area_struct *vma,
>> +                                   unsigned long addr, pte_t *pte)
>> +{
>> +       pte_t ptent = ptep_get(pte);
>> +
>> +       if (pte_present(ptent)) {
>> +               pte_t old_pte;
>> +
>> +               old_pte = ptep_modify_prot_start(vma, addr, pte);
>> +               ptent = pte_mkuffd_wp(ptent);
>> +               ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
>> +       } else if (is_swap_pte(ptent)) {
>> +               ptent = pte_swp_mkuffd_wp(ptent);
>> +               set_pte_at(vma->vm_mm, addr, pte, ptent);
>> +       } else {
>> +               set_pte_at(vma->vm_mm, addr, pte,
>> +                          make_pte_marker(PTE_MARKER_UFFD_WP));
>> +       }
>> +}
>> +
>> +static inline bool pagemap_scan_is_file(struct vm_area_struct *vma, pte_t ptent,
>> +                                       unsigned long addr)
>> +{
>> +       struct page *page = NULL;
>> +       swp_entry_t entry;
>> +
>> +       if (pte_present(ptent)) {
>> +               page = vm_normal_page(vma, addr, ptent);
>> +       } else {
>> +               entry = pte_to_swp_entry(ptent);
>> +               if (is_pfn_swap_entry(entry))
>> +                       page = pfn_swap_entry_to_page(entry);
>> +       }
>> +
>> +       if (page && !PageAnon(page))
>> +               return true;
>> +
>> +       return false;
>> +}
>> +
>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
>> +static inline bool is_pmd_uffd_wp(pmd_t pmd)
>> +{
>> +       return (pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
>> +              (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd));
>> +}
>> +
>> +static inline void make_uffd_wp_pmd(struct vm_area_struct *vma,
>> +                                   unsigned long addr, pmd_t *pmdp)
>> +{
>> +       pmd_t old, pmd = *pmdp;
>> +
>> +       if (pmd_present(pmd)) {
>> +               old = pmdp_invalidate_ad(vma, addr, pmdp);
>> +               pmd = pmd_mkuffd_wp(old);
>> +               set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
>> +       } else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
>> +               pmd = pmd_swp_mkuffd_wp(pmd);
>> +               set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
>> +       }
>> +}
>> +#endif
>> +
>> +#ifdef CONFIG_HUGETLB_PAGE
>> +static inline bool is_huge_pte_uffd_wp(pte_t pte)
>> +{
>> +       return (pte_present(pte) && huge_pte_uffd_wp(pte)) ||
>> +              pte_swp_uffd_wp_any(pte);
>> +}
>> +
>> +static inline void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
>> +                                        unsigned long addr, pte_t *ptep,
>> +                                        pte_t ptent)
>> +{
>> +       if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
>> +               return;
>> +
>> +       if (is_hugetlb_entry_migration(ptent))
>> +               set_huge_pte_at(vma->vm_mm, addr, ptep,
>> +                               pte_swp_mkuffd_wp(ptent));
>> +       else if (!huge_pte_none(ptent))
>> +               huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
>> +                                            huge_pte_mkuffd_wp(ptent));
>> +       else
>> +               set_huge_pte_at(vma->vm_mm, addr, ptep,
>> +                               make_pte_marker(PTE_MARKER_UFFD_WP));
>> +}
>> +
>> +static inline bool pagemap_scan_is_huge_file(pte_t pte)
>> +{
>> +       if (pte_present(pte) && (!PageAnon(pte_page(pte))))
>> +               return true;
>> +
>> +       return false;
>> +}
>> +#endif
>> +
>> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
>> +                                 struct mm_walk *walk)
>> +{
>> +       struct pagemap_scan_private *p = walk->private;
>> +       struct vm_area_struct *vma = walk->vma;
>> +
>> +       if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) ||
>> +           !userfaultfd_wp_use_markers(vma)))
>> +               return -EPERM;
>> +
>> +       if (vma->vm_flags & VM_PFNMAP)
>> +               return 1;
>> +
>> +       return 0;
>> +}
>> +
>> +static int pagemap_scan_output(unsigned long bitmap,
>> +                              struct pagemap_scan_private *p,
>> +                              unsigned long addr, unsigned int n_pages)
>> +{
>> +       struct page_region *cur_buf = &p->cur_buf;
>> +
>> +       bitmap &= p->return_mask;
>> +
>> +       if (cur_buf->flags == bitmap &&
>> +           cur_buf->start + cur_buf->len * PAGE_SIZE == addr) {
>> +               cur_buf->len += n_pages;
>> +               p->found_pages += n_pages;
>> +       } else {
>> +               if (cur_buf->len) {
>> +                       if (p->vec_buf_index >= p->vec_buf_len) {
>> +                               p->end_addr = addr;
>> +                               return PM_SCAN_END_WALK;
>> +                       }
>> +
>> +                       memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
>> +                              sizeof(*p->vec_buf));
>> +                       p->vec_buf_index++;
>> +               }
>> +
>> +               cur_buf->start = addr;
>> +               cur_buf->len = n_pages;
>> +               cur_buf->flags = bitmap;
>> +               p->found_pages += n_pages;
>> +       }
>> +
>> +       if (p->found_pages == p->max_pages) {
>> +               p->end_addr = addr + n_pages * PAGE_SIZE;
>> +               return PM_SCAN_FOUND_MAX_PAGES;
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>> +static bool pagemap_scan_is_interesting_page(unsigned long bitmap,
>> +                                            struct pagemap_scan_private *p)
>> +{
>> +       if ((p->required_mask & bitmap) != p->required_mask)
>> +               return false;
>> +       if (p->anyof_mask && !(p->anyof_mask & bitmap))
>> +               return false;
>> +       if (p->excluded_mask & bitmap)
>> +               return false;
>> +
>> +       return true;
>> +}
>> +
>> +static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
>> +                                 unsigned long end, struct mm_walk *walk)
>> +{
>> +       bool is_written, flush = false, interesting = true;
>> +       struct pagemap_scan_private *p = walk->private;
>> +       struct vm_area_struct *vma = walk->vma;
>> +       unsigned long bitmap, addr = end;
>> +       pte_t *pte, *orig_pte, ptent;
>> +       spinlock_t *ptl;
>> +       int ret = 0;
>> +
>> +       arch_enter_lazy_mmu_mode();
>> +
>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
>> +       ptl = pmd_trans_huge_lock(pmd, vma);
>> +       if (ptl) {
>> +               unsigned long n_pages = (end - start)/PAGE_SIZE;
>> +
>> +               if (n_pages > p->max_pages - p->found_pages)
>> +                       n_pages = p->max_pages - p->found_pages;
>> +
>> +               is_written = !is_pmd_uffd_wp(*pmd);
>> +
>> +               bitmap = PM_SCAN_FLAGS(is_written, false,
>> +                                      pmd_present(*pmd), is_swap_pmd(*pmd),
>> +                                      pmd_present(*pmd) && is_zero_pfn(pmd_pfn(*pmd)));
>> +
>> +               if (IS_PM_SCAN_GET(p->flags))
>> +                       interesting = pagemap_scan_is_interesting_page(bitmap, p);
>> +
>> +               if (interesting) {
>> +                       /*
>> +                        * Break huge page into small pages if the WP operation
>> +                        * need to be performed is on a portion of the huge page.
>> +                        */
>> +                       if (is_written && IS_PM_SCAN_WP(p->flags) &&
>> +                           n_pages < HPAGE_SIZE/PAGE_SIZE) {
>> +                               spin_unlock(ptl);
>> +
>> +                               split_huge_pmd(vma, pmd, start);
>> +                               goto process_smaller_pages;
>> +                       }
>> +
>> +                       if (IS_PM_SCAN_GET(p->flags))
>> +                               ret = pagemap_scan_output(bitmap, p, start, n_pages);
>> +
>> +                       if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) {
>> +                               make_uffd_wp_pmd(vma, start, pmd);
>> +                               flush_tlb_range(vma, start, end);
>> +                       }
>> +               }
>> +
>> +               spin_unlock(ptl);
>> +               arch_leave_lazy_mmu_mode();
>> +
>> +               return ret;
>> +       }
>> +
>> +process_smaller_pages:
>> +#endif
>> +
>> +       orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
>> +       if (!pte) {
>> +               walk->action = ACTION_AGAIN;
>> +               return 0;
>> +       }
>> +
>> +       for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE) {
>> +               ptent = ptep_get(pte);
>> +               is_written = !is_pte_uffd_wp(ptent);
>> +
>> +               bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_file(vma, ptent, addr),
>> +                                      pte_present(ptent), is_swap_pte(ptent),
>> +                                      pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
>> +
>> +               if (IS_PM_SCAN_GET(p->flags)) {
>> +                       interesting = pagemap_scan_is_interesting_page(bitmap, p);
>> +                       if (interesting)
>> +                               ret = pagemap_scan_output(bitmap, p, addr, 1);
>> +               }
>> +
>> +               if (IS_PM_SCAN_WP(p->flags) && is_written && interesting &&
>> +                   ret >= 0) {
>> +                       make_uffd_wp_pte(vma, addr, pte);
>> +                       flush = true;
>> +               }
>> +       }
>> +
>> +       if (flush)
>> +               flush_tlb_range(vma, start, addr);
>> +
>> +       pte_unmap_unlock(orig_pte, ptl);
>> +       arch_leave_lazy_mmu_mode();
>> +
>> +       cond_resched();
>> +       return ret;
>> +}
>> +
>> +#ifdef CONFIG_HUGETLB_PAGE
>> +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
>> +                                     unsigned long start, unsigned long end,
>> +                                     struct mm_walk *walk)
>> +{
>> +       unsigned long n_pages = (end - start)/PAGE_SIZE;
>> +       struct pagemap_scan_private *p = walk->private;
>> +       struct vm_area_struct *vma = walk->vma;
>> +       bool is_written, interesting = true;
>> +       struct hstate *h = hstate_vma(vma);
>> +       unsigned long bitmap;
>> +       spinlock_t *ptl;
>> +       int ret = 0;
>> +       pte_t ptent;
>> +
>> +       if (IS_PM_SCAN_WP(p->flags) && n_pages < HPAGE_SIZE/PAGE_SIZE) {
>> +               p->end_addr = start;
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (n_pages > p->max_pages - p->found_pages)
>> +               n_pages = p->max_pages - p->found_pages;
>> +
>> +       if (IS_PM_SCAN_WP(p->flags)) {
>> +               i_mmap_lock_write(vma->vm_file->f_mapping);
>> +               ptl = huge_pte_lock(h, vma->vm_mm, ptep);
>> +       }
>> +
>> +       ptent = huge_ptep_get(ptep);
>> +       is_written = !is_huge_pte_uffd_wp(ptent);
>> +
>> +       bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_huge_file(ptent),
>> +                              pte_present(ptent), is_swap_pte(ptent),
>> +                              pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
>> +
>> +       if (IS_PM_SCAN_GET(p->flags))
>> +               interesting = pagemap_scan_is_interesting_page(bitmap, p);
>> +
>> +       if (interesting) {
>> +               /*
>> +                * Partial hugetlb page clear isn't supported
>> +                */
>> +               if (is_written && IS_PM_SCAN_WP(p->flags) &&
>> +                   n_pages < HPAGE_SIZE/PAGE_SIZE) {
>> +                       ret = PM_SCAN_END_WALK;
>> +                       p->end_addr = start;
>> +                       goto unlock_and_return;
>> +               }
>> +
>> +               if (IS_PM_SCAN_GET(p->flags))
>> +                       ret = pagemap_scan_output(bitmap, p, start, n_pages);
>> +
>> +               if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) {
>> +                       make_uffd_wp_huge_pte(vma, start, ptep, ptent);
>> +                       flush_hugetlb_tlb_range(vma, start, end);
>> +               }
>> +       }
>> +
>> +unlock_and_return:
>> +       if (IS_PM_SCAN_WP(p->flags)) {
>> +               spin_unlock(ptl);
>> +               i_mmap_unlock_write(vma->vm_file->f_mapping);
>> +       }
>> +
>> +       return ret;
>> +}
>> +#else
>> +#define pagemap_scan_hugetlb_entry NULL
>> +#endif
>> +
>> +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
>> +                                int depth, struct mm_walk *walk)
>> +{
>> +       unsigned long n_pages = (end - addr)/PAGE_SIZE;
>> +       struct pagemap_scan_private *p = walk->private;
>> +       struct vm_area_struct *vma = walk->vma;
>> +       bool interesting = true;
>> +       unsigned long bitmap;
>> +       int ret = 0;
>> +
>> +       if (!vma)
>> +               return 0;
>> +
>> +       bitmap = PM_SCAN_FLAGS(false, false, false, false, false);
>> +
>> +       if (IS_PM_SCAN_GET(p->flags))
>> +               interesting = pagemap_scan_is_interesting_page(bitmap, p);
>> +
>> +       if (interesting) {
>> +               if (IS_PM_SCAN_GET(p->flags)) {
>> +                       if (n_pages > p->max_pages - p->found_pages)
>> +                               n_pages = p->max_pages - p->found_pages;
>> +
>> +                       ret = pagemap_scan_output(bitmap, p, addr, n_pages);
>> +               }
>> +
>> +               if (IS_PM_SCAN_WP(p->flags) && !ret &&
>> +                   uffd_wp_range(vma, addr, end - addr, true) < 0)
> 
> Why do we need to call uffd_wp_range for holes? Should we call
> flush_tlb_range after it?
> 
>> +                       ret = -EINVAL;
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +static const struct mm_walk_ops pagemap_scan_ops = {
>> +       .test_walk = pagemap_scan_test_walk,
>> +       .pmd_entry = pagemap_scan_pmd_entry,
>> +       .pte_hole = pagemap_scan_pte_hole,
>> +       .hugetlb_entry = pagemap_scan_hugetlb_entry,
>> +};
>> +
>> +static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start,
>> +                                  unsigned long end, struct page_region __user *vec)
>> +{
>> +       /* Detect illegal size, flags, len and masks */
>> +       if (arg->size != sizeof(struct pm_scan_arg))
>> +               return -EINVAL;
>> +       if (!arg->flags)
>> +               return -EINVAL;
>> +       if (arg->flags & ~PM_SCAN_OPS)
>> +               return -EINVAL;
>> +       if (!(end - start))
>> +               return -EINVAL;
>> +       if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask |
>> +            arg->return_mask) & ~PM_SCAN_BITS_ALL)
>> +               return -EINVAL;
>> +       if (!arg->required_mask && !arg->anyof_mask &&
>> +           !arg->excluded_mask)
>> +               return -EINVAL;
>> +       if (!arg->return_mask)
>> +               return -EINVAL;
>> +
>> +       /* Validate memory range */
>> +       if (!IS_ALIGNED(start, PAGE_SIZE))
>> +               return -EINVAL;
>> +       if (!access_ok((void __user *)start, end - start))
>> +               return -EFAULT;
>> +
>> +       if (IS_PM_SCAN_GET(arg->flags)) {
>> +               if (arg->vec_len == 0)
>> +                       return -EINVAL;
>> +               if (!vec)
>> +                       return -EFAULT;
>> +               if (!access_ok((void __user *)vec,
>> +                              arg->vec_len * sizeof(struct page_region)))
>> +                       return -EFAULT;
>> +       }
>> +
>> +       if (IS_PM_SCAN_WP(arg->flags) && !IS_PM_SCAN_GET(arg->flags) &&
>> +           arg->max_pages)
>> +               return -EINVAL;
>> +
>> +       return 0;
>> +}
>> +
>> +static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg)
>> +{
>> +       struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg;
>> +       unsigned long long start, end, walk_start, walk_end;
>> +       unsigned long empty_slots, vec_index = 0;
>> +       struct mmu_notifier_range range;
>> +       struct page_region __user *vec;
>> +       struct pagemap_scan_private p;
>> +       struct pm_scan_arg arg;
>> +       int ret = 0;
>> +
>> +       if (copy_from_user(&arg, uarg, sizeof(arg)))
>> +               return -EFAULT;
>> +
>> +       start = untagged_addr((unsigned long)arg.start);
>> +       end = untagged_addr((unsigned long)arg.end);
>> +       vec = (struct page_region __user *)untagged_addr((unsigned long)arg.vec);
>> +
>> +       ret = pagemap_scan_args_valid(&arg, start, end, vec);
>> +       if (ret)
>> +               return ret;
>> +
>> +       p.max_pages = (arg.max_pages) ? arg.max_pages : ULONG_MAX;
>> +       p.found_pages = 0;
>> +       p.required_mask = arg.required_mask;
>> +       p.anyof_mask = arg.anyof_mask;
>> +       p.excluded_mask = arg.excluded_mask;
>> +       p.return_mask = arg.return_mask;
>> +       p.flags = arg.flags;
>> +       p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) &
>> +                   PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0;
>> +       p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0;
>> +       p.vec_buf = NULL;
>> +       p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
>> +       p.vec_buf_index = 0;
>> +       p.end_addr = 0;
>> +
>> +       /*
>> +        * Allocate smaller buffer to get output from inside the page walk
>> +        * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
>> +        * we want to return output to user in compact form where no two
>> +        * consecutive regions should be continuous and have the same flags.
>> +        * So store the latest element in p.cur_buf between different walks and
>> +        * store the p.cur_buf at the end of the walk to the user buffer.
>> +        */
>> +       if (IS_PM_SCAN_GET(p.flags)) {
>> +               p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf),
>> +                                         GFP_KERNEL);
>> +               if (!p.vec_buf)
>> +                       return -ENOMEM;
>> +       }
>> +
>> +       /*
>> +        * Protection change for the range is going to happen.
>> +        */
>> +       if (IS_PM_SCAN_WP(p.flags)) {
>> +               mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
>> +                                       mm, start, end);
>> +               mmu_notifier_invalidate_range_start(&range);
>> +       }
>> +
>> +       walk_start = walk_end = start;
>> +       while (walk_end < end && !ret) {
>> +               if (IS_PM_SCAN_GET(p.flags)) {
>> +                       /*
>> +                        * All data is copied to cur_buf first. When more data
>> +                        * is found, we push cur_buf to vec_buf and copy new
>> +                        * data to cur_buf. Subtract 1 from length as the
>> +                        * index of cur_buf isn't counted in length.
>> +                        */
>> +                       empty_slots = arg.vec_len - vec_index;
>> +                       p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1);
>> +               }
>> +
> 
> I still don't understand why we don't want/need to check for pending signals.
We haven't added it as other existing code such as mincore() and
pagemap_read() don't have it either. Also mmap_read_lock_killable would
return error if there is some critical single pending.

> 
>> +               ret = mmap_read_lock_killable(mm);
>> +               if (ret)
>> +                       goto out;
>> +
>> +               walk_end = min((walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK, end);
>> +
>> +               ret = walk_page_range(mm, walk_start, walk_end,
>> +                                     &pagemap_scan_ops, &p);
>> +               mmap_read_unlock(mm);
>> +
>> +               if (ret == PM_SCAN_FOUND_MAX_PAGES || ret == PM_SCAN_END_WALK)
>> +                       arg.start = p.end_addr;
> 
> nit: this check can be moved out of the loop.
No, ret could get replaced by error if copy_to_user() fails. So we have to
do this before that.

> 
>> +
>> +               if (ret && ret != PM_SCAN_FOUND_MAX_PAGES &&
>> +                   ret != PM_SCAN_END_WALK)
>> +                       goto out;
>> +
>> +               if (p.vec_buf_index) {
>> +                       if (copy_to_user(&vec[vec_index], p.vec_buf,
>> +                                        p.vec_buf_index * sizeof(*p.vec_buf))) {
>> +                               /*
>> +                                * Return error even though the OP succeeded
>> +                                */
>> +                               ret = -EFAULT;
>> +                               goto out;
>> +                       }
>> +                       vec_index += p.vec_buf_index;
>> +                       p.vec_buf_index = 0;
>> +               }
>> +               walk_start = walk_end;
>> +       }
>> +
>> +       if (p.cur_buf.len) {
>> +               if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) {
>> +                       ret = -EFAULT;
>> +                       goto out;
>> +               }
>> +               vec_index++;
>> +       }
>> +
>> +       ret = vec_index;
>> +
>> +out:
>> +       if (!p.end_addr)
>> +               arg.start = walk_start;
>> +       if (copy_to_user(&uarg->start, &arg.start, sizeof(arg.start)))
>> +               ret = -EFAULT;
>> +
>> +       if (IS_PM_SCAN_WP(p.flags))
>> +               mmu_notifier_invalidate_range_end(&range);
>> +
>> +       kfree(p.vec_buf);
>> +       return ret;
>> +}
>> +
>> +static long do_pagemap_cmd(struct file *file, unsigned int cmd,
>> +                          unsigned long arg)
>> +{
>> +       struct mm_struct *mm = file->private_data;
>> +
>> +       switch (cmd) {
>> +       case PAGEMAP_SCAN:
>> +               return do_pagemap_scan(mm, arg);
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +}
>> +
>>  const struct file_operations proc_pagemap_operations = {
>>         .llseek         = mem_lseek, /* borrow this */
>>         .read           = pagemap_read,
>>         .open           = pagemap_open,
>>         .release        = pagemap_release,
>> +       .unlocked_ioctl = do_pagemap_cmd,
>> +       .compat_ioctl   = do_pagemap_cmd,
>>  };
>>  #endif /* CONFIG_PROC_PAGE_MONITOR */
>>
>> diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
>> index 623e98d62df3..c44e7bde0339 100644
>> --- a/include/linux/hugetlb.h
>> +++ b/include/linux/hugetlb.h
>> @@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
>>                 unsigned long cp_flags);
>>
>>  bool is_hugetlb_entry_migration(pte_t pte);
>> +bool is_hugetlb_entry_hwpoisoned(pte_t pte);
>>  void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
>>
>>  #else /* !CONFIG_HUGETLB_PAGE */
>> diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
>> index b7b56871029c..ac684d99e68f 100644
>> --- a/include/uapi/linux/fs.h
>> +++ b/include/uapi/linux/fs.h
>> @@ -305,4 +305,59 @@ typedef int __bitwise __kernel_rwf_t;
>>  #define RWF_SUPPORTED  (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
>>                          RWF_APPEND)
>>
>> +/* Pagemap ioctl */
>> +#define PAGEMAP_SCAN   _IOWR('f', 16, struct pm_scan_arg)
>> +
>> +/* Bits are set in flags of the page_region and masks in pm_scan_args */
>> +#define PAGE_IS_WRITTEN                (1 << 0)
>> +#define PAGE_IS_FILE           (1 << 1)
>> +#define PAGE_IS_PRESENT                (1 << 2)
>> +#define PAGE_IS_SWAPPED                (1 << 3)
>> +#define PAGE_IS_PFNZERO                (1 << 4)
>> +
>> +/*
>> + * struct page_region - Page region with flags
>> + * @start:     Start of the region
>> + * @len:       Length of the region in pages
>> + * @bitmap:    Bits sets for the region
>> + */
>> +struct page_region {
>> +       __u64 start;
>> +       __u64 len;
>> +       __u64 flags;
>> +};
>> +
>> +/*
>> + * struct pm_scan_arg - Pagemap ioctl argument
>> + * @size:              Size of the structure
>> + * @flags:             Flags for the IOCTL
>> + * @start:             Starting address of the region
>> + *                     (Ending address of the walk is also returned in it)
>> + * @end:               Ending address of the region
>> + * @vec:               Address of page_region struct array for output
>> + * @vec_len:           Length of the page_region struct array
>> + * @max_pages:         Optional max return pages
>> + * @required_mask:     Required mask - All of these bits have to be set in the PTE
>> + * @anyof_mask:                Any mask - Any of these bits are set in the PTE
>> + * @excluded_mask:     Exclude mask - None of these bits are set in the PTE
>> + * @return_mask:       Bits that are to be reported in page_region
>> + */
>> +struct pm_scan_arg {
>> +       __u64 size;
>> +       __u64 flags;
>> +       __u64 start;
>> +       __u64 end;
>> +       __u64 vec;
>> +       __u64 vec_len;
>> +       __u64 max_pages;
>> +       __u64 required_mask;
>> +       __u64 anyof_mask;
>> +       __u64 excluded_mask;
>> +       __u64 return_mask;
>> +};
>> +
>> +/* Supported flags */
>> +#define PM_SCAN_OP_GET (1 << 0)
>> +#define PM_SCAN_OP_WP  (1 << 1)
>> +
>>  #endif /* _UAPI_LINUX_FS_H */
>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>> index ee88a2a6649f..f7cefdcb114c 100644
>> --- a/mm/hugetlb.c
>> +++ b/mm/hugetlb.c
>> @@ -4981,7 +4981,7 @@ bool is_hugetlb_entry_migration(pte_t pte)
>>                 return false;
>>  }
>>
>> -static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
>> +bool is_hugetlb_entry_hwpoisoned(pte_t pte)
>>  {
>>         swp_entry_t swp;
>>
>> --
>> 2.39.2
>>

-- 
BR,
Muhammad Usama Anjum

Re: [PATCH v25 0/5] Implement IOCTL to get and optionally clear info about PTEs

[Rogerio Alves]

Tested v25 here on linux-next branch next-20230710, did not see any 
regressions with patch applied.

TAP version 13
1..92
ok 1 sanity_tests_sd memory size must be valid
ok 2 sanity_tests_sd output buffer must be specified
ok 3 sanity_tests_sd output buffer size must be valid
ok 4 sanity_tests_sd wrong flag specified
ok 5 sanity_tests_sd flag has extra bits specified
ok 6 sanity_tests_sd no selection mask is specified
ok 7 sanity_tests_sd no return mask is specified
ok 8 sanity_tests_sd wrong return mask specified
ok 9 sanity_tests_sd mixture of correct and wrong flag
ok 10 sanity_tests_sd PAGEMAP_BITS_ALL can be specified with PM_SCAN_OP_WP
ok 11 sanity_tests_sd Clear area with larger vec size
ok 12 sanity_tests_sd Repeated pattern of written and non-written pages
ok 13 sanity_tests_sd Repeated pattern of written and non-written pages 
in parts
ok 14 sanity_tests_sd Repeated pattern of written and non-written pages 
max_pages
ok 15 sanity_tests_sd only get 2 written pages and clear them as well
ok 16 sanity_tests_sd Two regions
ok 17 sanity_tests_sd Smaller max_pages
ok 18 Smaller vec 46 50
ok 19 Page testing: all new pages must not be written (dirty)
ok 20 Page testing: all pages must be written (dirty)
ok 21 Page testing: all pages dirty other than first and the last one
ok 22 Page testing: PM_SCAN_OP_WP
ok 23 Page testing: only middle page dirty
ok 24 Page testing: only two middle pages dirty
ok 25 Large Page testing: all new pages must not be written (dirty)
ok 26 Large Page testing: all pages must be written (dirty)
ok 27 Large Page testing: all pages dirty other than first and the last one
ok 28 Large Page testing: PM_SCAN_OP_WP
ok 29 Large Page testing: only middle page dirty
ok 30 Large Page testing: only two middle pages dirty
ok 31 Huge page testing: all new pages must not be written (dirty)
ok 32 Huge page testing: all pages must be written (dirty)
ok 33 Huge page testing: all pages dirty other than first and the last one
ok 34 Huge page testing: PM_SCAN_OP_WP
ok 35 Huge page testing: only middle page dirty
ok 36 Huge page testing: only two middle pages dirty
ok 37 # SKIP Hugetlb shmem testing: all new pages must not be written 
(dirty)
ok 38 # SKIP Hugetlb shmem testing: all pages must be written (dirty)
ok 39 # SKIP Hugetlb shmem testing: all pages dirty other than first and 
the last one
ok 40 # SKIP Hugetlb shmem testing: PM_SCAN_OP_WP
ok 41 # SKIP Hugetlb shmem testing: only middle page dirty
ok 42 # SKIP Hugetlb shmem testing: only two middle pages dirty
ok 43 # SKIP Hugetlb mem testing: all new pages must not be written (dirty)
ok 44 # SKIP Hugetlb mem testing: all pages must be written (dirty)
ok 45 # SKIP Hugetlb mem testing: all pages dirty other than first and 
the last one
ok 46 # SKIP Hugetlb mem testing: PM_SCAN_OP_WP
ok 47 # SKIP Hugetlb mem testing: only middle page dirty
ok 48 # SKIP Hugetlb mem testing: only two middle pages dirty
ok 49 File memory testing: all new pages must not be written (dirty)
ok 50 File memory testing: all pages must be written (dirty)
ok 51 File memory testing: all pages dirty other than first and the last one
ok 52 File memory testing: PM_SCAN_OP_WP
ok 53 File memory testing: only middle page dirty
ok 54 File memory testing: only two middle pages dirty
ok 55 File anonymous memory testing: all new pages must not be written 
(dirty)
ok 56 File anonymous memory testing: all pages must be written (dirty)
ok 57 File anonymous memory testing: all pages dirty other than first 
and the last one
ok 58 File anonymous memory testing: PM_SCAN_OP_WP
ok 59 File anonymous memory testing: only middle page dirty
ok 60 File anonymous memory testing: only two middle pages dirty
ok 61 hpage_unit_tests all new huge page must not be written (dirty)
ok 62 hpage_unit_tests all the huge page must not be written
ok 63 hpage_unit_tests all the huge page must be written and clear
ok 64 hpage_unit_tests only middle page written
ok 65 hpage_unit_tests clear first half of huge page
ok 66 hpage_unit_tests clear first half of huge page with limited buffer
ok 67 hpage_unit_tests clear second half huge page
ok 68 hpage_unit_tests get half huge page
ok 69 hpage_unit_tests get half huge page
ok 70 Test test_simple
ok 71 mprotect_tests Both pages written
ok 72 mprotect_tests Both pages are not written (dirty)
ok 73 mprotect_tests Both pages written after remap and mprotect
ok 74 mprotect_tests Clear and make the pages written
ok 75 transact_test count 192
ok 76 transact_test count 0
ok 77 transact_test Extra pages 130 (0.3%), extra thread faults 141.
ok 78 sanity_tests WP op can be specified with !PAGE_IS_WRITTEN
ok 79 sanity_tests required_mask specified
ok 80 sanity_tests anyof_mask specified
ok 81 sanity_tests excluded_mask specified
ok 82 sanity_tests required_mask and anyof_mask specified
ok 83 sanity_tests Get sd and present pages with anyof_mask
ok 84 sanity_tests Get all the pages with required_mask
ok 85 sanity_tests Get sd and present pages with required_mask and 
anyof_mask
ok 86 sanity_tests Don't get sd pages
ok 87 sanity_tests Don't get present pages
ok 88 sanity_tests Find written present pages with return mask
ok 89 sanity_tests Memory mapped file
ok 90 sanity_tests Read/write to memory
ok 91 unmapped_region_tests Get status of pages
ok 92 userfaultfd_tests all new pages must not be written (dirty)
# Totals: pass:80 fail:0 xfail:0 xpass:0 skip:12 error:0

On 7/13/23 07:14, Muhammad Usama Anjum wrote:
> *Changes in v25*:
> - Do proper filtering on hole as well (hole got missed earlier)
>
> *Changes in v24*:
> - Rebase on top of next-20230710
> - Place WP markers in case of hole as well
>
> *Changes in v23*:
> - Set vec_buf_index in loop only when vec_buf_index is set
> - Return -EFAULT instead of -EINVAL if vec is NULL
> - Correctly return the walk ending address to the page granularity
>
> *Changes in v22*:
> - Interface change:
>    - Replace [start start + len) with [start, end)
>    - Return the ending address of the address walk in start
>
> *Changes in v21*:
> - Abort walk instead of returning error if WP is to be performed on
>    partial hugetlb
>
> *Changes in v20*
> - Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO
>
> *Changes in v19*
> - Minor changes and interface updates
>
> *Changes in v18*
> - Rebase on top of next-20230613
> - Minor updates
>
> *Changes in v17*
> - Rebase on top of next-20230606
> - Minor improvements in PAGEMAP_SCAN IOCTL patch
>
> *Changes in v16*
> - Fix a corner case
> - Add exclusive PM_SCAN_OP_WP back
>
> *Changes in v15*
> - Build fix (Add missed build fix in RESEND)
>
> *Changes in v14*
> - Fix build error caused by #ifdef added at last minute in some configs
>
> *Changes in v13*
> - Rebase on top of next-20230414
> - Give-up on using uffd_wp_range() and write new helpers, flush tlb only
>    once
>
> *Changes in v12*
> - Update and other memory types to UFFD_FEATURE_WP_ASYNC
> - Rebaase on top of next-20230406
> - Review updates
>
> *Changes in v11*
> - Rebase on top of next-20230307
> - Base patches on UFFD_FEATURE_WP_UNPOPULATED
> - Do a lot of cosmetic changes and review updates
> - Remove ENGAGE_WP + !GET operation as it can be performed with
>    UFFDIO_WRITEPROTECT
>
> *Changes in v10*
> - Add specific condition to return error if hugetlb is used with wp
>    async
> - Move changes in tools/include/uapi/linux/fs.h to separate patch
> - Add documentation
>
> *Changes in v9:*
> - Correct fault resolution for userfaultfd wp async
> - Fix build warnings and errors which were happening on some configs
> - Simplify pagemap ioctl's code
>
> *Changes in v8:*
> - Update uffd async wp implementation
> - Improve PAGEMAP_IOCTL implementation
>
> *Changes in v7:*
> - Add uffd wp async
> - Update the IOCTL to use uffd under the hood instead of soft-dirty
>    flags
>
> *Motivation*
> The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows
> GetWriteWatch() syscall [1]. The GetWriteWatch{} retrieves the addresses of
> the pages that are written to in a region of virtual memory.
>
> This syscall is used in Windows applications and games etc. This syscall is
> being emulated in pretty slow manner in userspace. Our purpose is to
> enhance the kernel such that we translate it efficiently in a better way.
> Currently some out of tree hack patches are being used to efficiently
> emulate it in some kernels. We intend to replace those with these patches.
> So the whole gaming on Linux can effectively get benefit from this. It
> means there would be tons of users of this code.
>
> CRIU use case [2] was mentioned by Andrei and Danylo:
>> Use cases for migrating sparse VMAs are binaries sanitized with ASAN,
>> MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of
>> shadow memory [4]. Being able to migrate such binaries allows to highly
>> reduce the amount of work needed to identify and fix post-migration
>> crashes, which happen constantly.
> Andrei's defines the following uses of this code:
> * it is more granular and allows us to track changed pages more
>    effectively. The current interface can clear dirty bits for the entire
>    process only. In addition, reading info about pages is a separate
>    operation. It means we must freeze the process to read information
>    about all its pages, reset dirty bits, only then we can start dumping
>    pages. The information about pages becomes more and more outdated,
>    while we are processing pages. The new interface solves both these
>    downsides. First, it allows us to read pte bits and clear the
>    soft-dirty bit atomically. It means that CRIU will not need to freeze
>    processes to pre-dump their memory. Second, it clears soft-dirty bits
>    for a specified region of memory. It means CRIU will have actual info
>    about pages to the moment of dumping them.
> * The new interface has to be much faster because basic page filtering
>    is happening in the kernel. With the old interface, we have to read
>    pagemap for each page.
>
> *Implementation Evolution (Short Summary)*
>  From the definition of GetWriteWatch(), we feel like kernel's soft-dirty
> feature can be used under the hood with some additions like:
> * reset soft-dirty flag for only a specific region of memory instead of
> clearing the flag for the entire process
> * get and clear soft-dirty flag for a specific region atomically
>
> So we decided to use ioctl on pagemap file to read or/and reset soft-dirty
> flag. But using soft-dirty flag, sometimes we get extra pages which weren't
> even written. They had become soft-dirty because of VMA merging and
> VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were
> able to by-pass this short coming by ignoring VM_SOFTDIRTY until David
> reported that mprotect etc messes up the soft-dirty flag while ignoring
> VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We
> discussed if we can revert these patches. But we could not reach to any
> conclusion. So at this point, I made couple of tries to solve this whole
> VM_SOFTDIRTY issue by correcting the soft-dirty implementation:
> * [7] Correct the bug fixed wrongly back in 2014. It had potential to cause
> regression. We left it behind.
> * [8] Keep a list of soft-dirty part of a VMA across splits and merges. I
> got the reply don't increase the size of the VMA by 8 bytes.
>
> At this point, we left soft-dirty considering it is too much delicate and
> userfaultfd [9] seemed like the only way forward. From there onward, we
> have been basing soft-dirty emulation on userfaultfd wp feature where
> kernel resolves the faults itself when WP_ASYNC feature is used. It was
> straight forward to add WP_ASYNC feature in userfautlfd. Now we get only
> those pages dirty or written-to which are really written in reality. (PS
> There is another WP_UNPOPULATED userfautfd feature is required which is
> needed to avoid pre-faulting memory before write-protecting [9].)
>
> All the different masks were added on the request of CRIU devs to create
> interface more generic and better.
>
> [1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getwritewatch
> [2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com
> [3] https://github.com/google/sanitizers
> [4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit
> [5] https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@redhat.com
> [6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@redhat.com/
> [7] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
> [8] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
> [9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com
> [10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@google.com
>
> * Original Cover letter from v8*
> Hello,
>
> Note:
> Soft-dirty pages and pages which have been written-to are synonyms. As
> kernel already has soft-dirty feature inside which we have given up to
> use, we are using written-to terminology while using UFFD async WP under
> the hood.
>
> This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
> the info about page table entries. The following operations are
> supported in this ioctl:
> - Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
>    file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
>    (PAGE_IS_SWAPPED).
> - Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
>    pages have been written-to.
> - Find pages which have been written-to and write protect the pages
>    (atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE)
>
> It is possible to find and clear soft-dirty pages entirely in userspace.
> But it isn't efficient:
> - The mprotect and SIGSEGV handler for bookkeeping
> - The userfaultfd wp (synchronous) with the handler for bookkeeping
>
> Some benchmarks can be seen here[1]. This series adds features that weren't
> present earlier:
> - There is no atomic get soft-dirty/Written-to status and clear present in
>    the kernel.
> - The pages which have been written-to can not be found in accurate way.
>    (Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty
>    pages than there actually are.)
>
> Historically, soft-dirty PTE bit tracking has been used in the CRIU
> project. The procfs interface is enough for finding the soft-dirty bit
> status and clearing the soft-dirty bit of all the pages of a process.
> We have the use case where we need to track the soft-dirty PTE bit for
> only specific pages on-demand. We need this tracking and clear mechanism
> of a region of memory while the process is running to emulate the
> getWriteWatch() syscall of Windows.
>
> *(Moved to using UFFD instead of soft-dirtyi feature to find pages which
> have been written-to from v7 patch series)*:
> Stop using the soft-dirty flags for finding which pages have been
> written to. It is too delicate and wrong as it shows more soft-dirty
> pages than the actual soft-dirty pages. There is no interest in
> correcting it [2][3] as this is how the feature was written years ago.
> It shouldn't be updated to changed behaviour. Peter Xu has suggested
> using the async version of the UFFD WP [4] as it is based inherently
> on the PTEs.
>
> So in this patch series, I've added a new mode to the UFFD which is
> asynchronous version of the write protect. When this variant of the
> UFFD WP is used, the page faults are resolved automatically by the
> kernel. The pages which have been written-to can be found by reading
> pagemap file (!PM_UFFD_WP). This feature can be used successfully to
> find which pages have been written to from the time the pages were
> write protected. This works just like the soft-dirty flag without
> showing any extra pages which aren't soft-dirty in reality.
>
> The information related to pages if the page is file mapped, present and
> swapped is required for the CRIU project [5][6]. The addition of the
> required mask, any mask, excluded mask and return masks are also required
> for the CRIU project [5].
>
> The IOCTL returns the addresses of the pages which match the specific
> masks. The page addresses are returned in struct page_region in a compact
> form. The max_pages is needed to support a use case where user only wants
> to get a specific number of pages. So there is no need to find all the
> pages of interest in the range when max_pages is specified. The IOCTL
> returns when the maximum number of the pages are found. The max_pages is
> optional. If max_pages is specified, it must be equal or greater than the
> vec_size. This restriction is needed to handle worse case when one
> page_region only contains info of one page and it cannot be compacted.
> This is needed to emulate the Windows getWriteWatch() syscall.
>
> The patch series include the detailed selftest which can be used as an
> example for the uffd async wp test and PAGEMAP_IOCTL. It shows the
> interface usages as well.
>
> [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/
> [2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
> [3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
> [4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n
> [5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/
> [6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/
>
> Regards,
> Muhammad Usama Anjum
>
> Muhammad Usama Anjum (4):
>    fs/proc/task_mmu: Implement IOCTL to get and optionally clear info
>      about PTEs
>    tools headers UAPI: Update linux/fs.h with the kernel sources
>    mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL
>    selftests: mm: add pagemap ioctl tests
>
> Peter Xu (1):
>    userfaultfd: UFFD_FEATURE_WP_ASYNC
>
>   Documentation/admin-guide/mm/pagemap.rst     |   58 +
>   Documentation/admin-guide/mm/userfaultfd.rst |   35 +
>   fs/proc/task_mmu.c                           |  591 +++++++
>   fs/userfaultfd.c                             |   26 +-
>   include/linux/hugetlb.h                      |    1 +
>   include/linux/userfaultfd_k.h                |   21 +-
>   include/uapi/linux/fs.h                      |   55 +
>   include/uapi/linux/userfaultfd.h             |    9 +-
>   mm/hugetlb.c                                 |   34 +-
>   mm/memory.c                                  |   27 +-
>   tools/include/uapi/linux/fs.h                |   55 +
>   tools/testing/selftests/mm/.gitignore        |    2 +
>   tools/testing/selftests/mm/Makefile          |    3 +-
>   tools/testing/selftests/mm/config            |    1 +
>   tools/testing/selftests/mm/pagemap_ioctl.c   | 1464 ++++++++++++++++++
>   tools/testing/selftests/mm/run_vmtests.sh    |    4 +
>   16 files changed, 2362 insertions(+), 24 deletions(-)
>   create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c
>   mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh
>
-- 
Rogerio Alves Cardoso
Consultant Senior Software Engineer

Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v25 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs

[Andrei Vagin]

On Tue, Jul 18, 2023 at 1:18 AM Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> On 7/17/23 10:26 PM, Andrei Vagin wrote:
> > On Thu, Jul 13, 2023 at 3:14 AM Muhammad Usama Anjum
> > <usama.anjum@collabora.com> wrote:
> >>
> >> This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
> >> the info about page table entries. The following operations are supported
> >> in this ioctl:
> >> - Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
> >>   file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT), swapped
> >>   (PAGE_IS_SWAPPED) or page has pfn zero (PAGE_IS_PFNZERO).
> >> - Find pages which have been written-to and/or write protect the pages
> >>   (atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP)
> >>
> >> This IOCTL can be extended to get information about more PTE bits. The
> >> entire address range passed by user [start, end) is scanned until either
> >> the user provided buffer is full or max_pages have been found.
> >>
> >
> > Reviewed-by: Andrei Vagin <avagin@gmail.com>
> Thank you.
>

<snip>

> >> +#ifdef CONFIG_HUGETLB_PAGE
> >> +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
> >> +                                     unsigned long start, unsigned long end,
> >> +                                     struct mm_walk *walk)
> >> +{
> >> +       unsigned long n_pages = (end - start)/PAGE_SIZE;
> >> +       struct pagemap_scan_private *p = walk->private;
> >> +       struct vm_area_struct *vma = walk->vma;
> >> +       bool is_written, interesting = true;
> >> +       struct hstate *h = hstate_vma(vma);
> >> +       unsigned long bitmap;
> >> +       spinlock_t *ptl;
> >> +       int ret = 0;
> >> +       pte_t ptent;
> >> +
> >> +       if (IS_PM_SCAN_WP(p->flags) && n_pages < HPAGE_SIZE/PAGE_SIZE) {
> >> +               p->end_addr = start;
> >> +               return -EINVAL;
> >> +       }
> >> +
> >> +       if (n_pages > p->max_pages - p->found_pages)
> >> +               n_pages = p->max_pages - p->found_pages;
> >> +
> >> +       if (IS_PM_SCAN_WP(p->flags)) {
> >> +               i_mmap_lock_write(vma->vm_file->f_mapping);
> >> +               ptl = huge_pte_lock(h, vma->vm_mm, ptep);
> >> +       }
> >> +
> >> +       ptent = huge_ptep_get(ptep);
> >> +       is_written = !is_huge_pte_uffd_wp(ptent);
> >> +
> >> +       bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_huge_file(ptent),
> >> +                              pte_present(ptent), is_swap_pte(ptent),
> >> +                              pte_present(ptent) && is_zero_pfn(pte_pfn(ptent)));
> >> +
> >> +       if (IS_PM_SCAN_GET(p->flags))
> >> +               interesting = pagemap_scan_is_interesting_page(bitmap, p);
> >> +
> >> +       if (interesting) {
> >> +               /*
> >> +                * Partial hugetlb page clear isn't supported
> >> +                */
> >> +               if (is_written && IS_PM_SCAN_WP(p->flags) &&
> >> +                   n_pages < HPAGE_SIZE/PAGE_SIZE) {
> >> +                       ret = PM_SCAN_END_WALK;
> >> +                       p->end_addr = start;
> >> +                       goto unlock_and_return;
> >> +               }
> >> +
> >> +               if (IS_PM_SCAN_GET(p->flags))
> >> +                       ret = pagemap_scan_output(bitmap, p, start, n_pages);
> >> +
> >> +               if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) {
> >> +                       make_uffd_wp_huge_pte(vma, start, ptep, ptent);
> >> +                       flush_hugetlb_tlb_range(vma, start, end);
> >> +               }
> >> +       }
> >> +
> >> +unlock_and_return:
> >> +       if (IS_PM_SCAN_WP(p->flags)) {
> >> +               spin_unlock(ptl);
> >> +               i_mmap_unlock_write(vma->vm_file->f_mapping);
> >> +       }
> >> +
> >> +       return ret;
> >> +}
> >> +#else
> >> +#define pagemap_scan_hugetlb_entry NULL
> >> +#endif
> >> +
> >> +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
> >> +                                int depth, struct mm_walk *walk)
> >> +{
> >> +       unsigned long n_pages = (end - addr)/PAGE_SIZE;
> >> +       struct pagemap_scan_private *p = walk->private;
> >> +       struct vm_area_struct *vma = walk->vma;
> >> +       bool interesting = true;
> >> +       unsigned long bitmap;
> >> +       int ret = 0;
> >> +
> >> +       if (!vma)
> >> +               return 0;
> >> +
> >> +       bitmap = PM_SCAN_FLAGS(false, false, false, false, false);
> >> +
> >> +       if (IS_PM_SCAN_GET(p->flags))
> >> +               interesting = pagemap_scan_is_interesting_page(bitmap, p);
> >> +
> >> +       if (interesting) {
> >> +               if (IS_PM_SCAN_GET(p->flags)) {
> >> +                       if (n_pages > p->max_pages - p->found_pages)
> >> +                               n_pages = p->max_pages - p->found_pages;
> >> +
> >> +                       ret = pagemap_scan_output(bitmap, p, addr, n_pages);
> >> +               }
> >> +
> >> +               if (IS_PM_SCAN_WP(p->flags) && !ret &&
> >> +                   uffd_wp_range(vma, addr, end - addr, true) < 0)
> >
> > Why do we need to call uffd_wp_range for holes? Should we call
> > flush_tlb_range after it?

Did you skip this question?

> >
> >> +                       ret = -EINVAL;
> >> +       }
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static const struct mm_walk_ops pagemap_scan_ops = {
> >> +       .test_walk = pagemap_scan_test_walk,
> >> +       .pmd_entry = pagemap_scan_pmd_entry,
> >> +       .pte_hole = pagemap_scan_pte_hole,
> >> +       .hugetlb_entry = pagemap_scan_hugetlb_entry,
> >> +};
> >> +
> >> +static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start,
> >> +                                  unsigned long end, struct page_region __user *vec)
> >> +{
> >> +       /* Detect illegal size, flags, len and masks */
> >> +       if (arg->size != sizeof(struct pm_scan_arg))
> >> +               return -EINVAL;
> >> +       if (!arg->flags)
> >> +               return -EINVAL;
> >> +       if (arg->flags & ~PM_SCAN_OPS)
> >> +               return -EINVAL;
> >> +       if (!(end - start))
> >> +               return -EINVAL;
> >> +       if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask |
> >> +            arg->return_mask) & ~PM_SCAN_BITS_ALL)
> >> +               return -EINVAL;
> >> +       if (!arg->required_mask && !arg->anyof_mask &&
> >> +           !arg->excluded_mask)
> >> +               return -EINVAL;
> >> +       if (!arg->return_mask)
> >> +               return -EINVAL;
> >> +
> >> +       /* Validate memory range */
> >> +       if (!IS_ALIGNED(start, PAGE_SIZE))
> >> +               return -EINVAL;
> >> +       if (!access_ok((void __user *)start, end - start))
> >> +               return -EFAULT;
> >> +
> >> +       if (IS_PM_SCAN_GET(arg->flags)) {
> >> +               if (arg->vec_len == 0)
> >> +                       return -EINVAL;
> >> +               if (!vec)
> >> +                       return -EFAULT;
> >> +               if (!access_ok((void __user *)vec,
> >> +                              arg->vec_len * sizeof(struct page_region)))
> >> +                       return -EFAULT;
> >> +       }
> >> +
> >> +       if (IS_PM_SCAN_WP(arg->flags) && !IS_PM_SCAN_GET(arg->flags) &&
> >> +           arg->max_pages)
> >> +               return -EINVAL;
> >> +
> >> +       return 0;
> >> +}
> >> +
> >> +static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg)
> >> +{
> >> +       struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg;
> >> +       unsigned long long start, end, walk_start, walk_end;
> >> +       unsigned long empty_slots, vec_index = 0;
> >> +       struct mmu_notifier_range range;
> >> +       struct page_region __user *vec;
> >> +       struct pagemap_scan_private p;
> >> +       struct pm_scan_arg arg;
> >> +       int ret = 0;
> >> +
> >> +       if (copy_from_user(&arg, uarg, sizeof(arg)))
> >> +               return -EFAULT;
> >> +
> >> +       start = untagged_addr((unsigned long)arg.start);
> >> +       end = untagged_addr((unsigned long)arg.end);
> >> +       vec = (struct page_region __user *)untagged_addr((unsigned long)arg.vec);
> >> +
> >> +       ret = pagemap_scan_args_valid(&arg, start, end, vec);
> >> +       if (ret)
> >> +               return ret;
> >> +
> >> +       p.max_pages = (arg.max_pages) ? arg.max_pages : ULONG_MAX;
> >> +       p.found_pages = 0;
> >> +       p.required_mask = arg.required_mask;
> >> +       p.anyof_mask = arg.anyof_mask;
> >> +       p.excluded_mask = arg.excluded_mask;
> >> +       p.return_mask = arg.return_mask;
> >> +       p.flags = arg.flags;
> >> +       p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) &
> >> +                   PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0;
> >> +       p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0;
> >> +       p.vec_buf = NULL;
> >> +       p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
> >> +       p.vec_buf_index = 0;
> >> +       p.end_addr = 0;
> >> +
> >> +       /*
> >> +        * Allocate smaller buffer to get output from inside the page walk
> >> +        * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
> >> +        * we want to return output to user in compact form where no two
> >> +        * consecutive regions should be continuous and have the same flags.
> >> +        * So store the latest element in p.cur_buf between different walks and
> >> +        * store the p.cur_buf at the end of the walk to the user buffer.
> >> +        */
> >> +       if (IS_PM_SCAN_GET(p.flags)) {
> >> +               p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf),
> >> +                                         GFP_KERNEL);
> >> +               if (!p.vec_buf)
> >> +                       return -ENOMEM;
> >> +       }
> >> +
> >> +       /*
> >> +        * Protection change for the range is going to happen.
> >> +        */
> >> +       if (IS_PM_SCAN_WP(p.flags)) {
> >> +               mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
> >> +                                       mm, start, end);
> >> +               mmu_notifier_invalidate_range_start(&range);
> >> +       }
> >> +
> >> +       walk_start = walk_end = start;
> >> +       while (walk_end < end && !ret) {
> >> +               if (IS_PM_SCAN_GET(p.flags)) {
> >> +                       /*
> >> +                        * All data is copied to cur_buf first. When more data
> >> +                        * is found, we push cur_buf to vec_buf and copy new
> >> +                        * data to cur_buf. Subtract 1 from length as the
> >> +                        * index of cur_buf isn't counted in length.
> >> +                        */
> >> +                       empty_slots = arg.vec_len - vec_index;
> >> +                       p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1);
> >> +               }
> >> +
> >
> > I still don't understand why we don't want/need to check for pending signals.
> We haven't added it as other existing code such as mincore() and

It doesn't convince me. There should be reasons to do or not to do
certain things.
We can't say how long this loop can be running, so it is the reason
why we can want
to check pending signals.

> pagemap_read() don't have it either.

I already explained that this case is different, because the size of
the output buffer is
limited for pagemap_read.

> Also mmap_read_lock_killable would return error if there is some critical single pending.\

It isn't completely true. It doesn't return errors in the fast path
when it takes the lock right
away. It checks signals only when it needs to wait for the lock.

>
> >
> >> +               ret = mmap_read_lock_killable(mm);
> >> +               if (ret)
> >> +                       goto out;
> >> +
> >> +               walk_end = min((walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK, end);
> >> +
> >> +               ret = walk_page_range(mm, walk_start, walk_end,
> >> +                                     &pagemap_scan_ops, &p);
> >> +               mmap_read_unlock(mm);
> >> +
> >> +               if (ret == PM_SCAN_FOUND_MAX_PAGES || ret == PM_SCAN_END_WALK)
> >> +                       arg.start = p.end_addr;
> >
> > nit: this check can be moved out of the loop.
> No, ret could get replaced by error if copy_to_user() fails. So we have to
> do this before that.

If we fail to copy a vector, it is a fatal error and it probably doesn't matter
what end address has been there. It is up to you to leave it here or not.

>
> >
> >> +
> >> +               if (ret && ret != PM_SCAN_FOUND_MAX_PAGES &&
> >> +                   ret != PM_SCAN_END_WALK)
> >> +                       goto out;
> >> +
> >> +               if (p.vec_buf_index) {
> >> +                       if (copy_to_user(&vec[vec_index], p.vec_buf,
> >> +                                        p.vec_buf_index * sizeof(*p.vec_buf))) {
> >> +                               /*
> >> +                                * Return error even though the OP succeeded
> >> +                                */
> >> +                               ret = -EFAULT;
> >> +                               goto out;
> >> +                       }
> >> +                       vec_index += p.vec_buf_index;
> >> +                       p.vec_buf_index = 0;
> >> +               }
> >> +               walk_start = walk_end;
> >> +       }
> >> +
> >> +       if (p.cur_buf.len) {
> >> +               if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) {
> >> +                       ret = -EFAULT;
> >> +                       goto out;
> >> +               }
> >> +               vec_index++;
> >> +       }
> >> +
> >> +       ret = vec_index;
> >> +
> >> +out:
> >> +       if (!p.end_addr)
> >> +               arg.start = walk_start;
> >> +       if (copy_to_user(&uarg->start, &arg.start, sizeof(arg.start)))
> >> +               ret = -EFAULT;
> >> +
> >> +       if (IS_PM_SCAN_WP(p.flags))
> >> +               mmu_notifier_invalidate_range_end(&range);
> >> +
> >> +       kfree(p.vec_buf);
> >> +       return ret;
> >> +}
> >> +

Thanks,
Andrei

Re: [PATCH v25 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs

[Muhammad Usama Anjum]

On 7/18/23 9:08 PM, Andrei Vagin wrote:
...
>>>> +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
>>>> +                                int depth, struct mm_walk *walk)
>>>> +{
>>>> +       unsigned long n_pages = (end - addr)/PAGE_SIZE;
>>>> +       struct pagemap_scan_private *p = walk->private;
>>>> +       struct vm_area_struct *vma = walk->vma;
>>>> +       bool interesting = true;
>>>> +       unsigned long bitmap;
>>>> +       int ret = 0;
>>>> +
>>>> +       if (!vma)
>>>> +               return 0;
>>>> +
>>>> +       bitmap = PM_SCAN_FLAGS(false, false, false, false, false);
>>>> +
>>>> +       if (IS_PM_SCAN_GET(p->flags))
>>>> +               interesting = pagemap_scan_is_interesting_page(bitmap, p);
>>>> +
>>>> +       if (interesting) {
>>>> +               if (IS_PM_SCAN_GET(p->flags)) {
>>>> +                       if (n_pages > p->max_pages - p->found_pages)
>>>> +                               n_pages = p->max_pages - p->found_pages;
>>>> +
>>>> +                       ret = pagemap_scan_output(bitmap, p, addr, n_pages);
>>>> +               }
>>>> +
>>>> +               if (IS_PM_SCAN_WP(p->flags) && !ret &&
>>>> +                   uffd_wp_range(vma, addr, end - addr, true) < 0)
>>>
>>> Why do we need to call uffd_wp_range for holes? Should we call
>>> flush_tlb_range after it?
> 
> Did you skip this question?
Sorry, missed it the first time. In case of holes, there isn't any pmd or
pte. But we need to place the PTE markers indicating that this memory is
WPed. So we can parse the address range from PGD ourselves and place the
markers. Or we can use the uffd_wp_range(). Using uffd_wp_range() for this
case seems optimal. We don't need to do flush as uffd_wp_range() flushes
the range by itself.

> 
>>>
>>>> +                       ret = -EINVAL;
>>>> +       }
>>>> +
>>>> +       return ret;
>>>> +}
>>>> +
>>>> +static const struct mm_walk_ops pagemap_scan_ops = {
>>>> +       .test_walk = pagemap_scan_test_walk,
>>>> +       .pmd_entry = pagemap_scan_pmd_entry,
>>>> +       .pte_hole = pagemap_scan_pte_hole,
>>>> +       .hugetlb_entry = pagemap_scan_hugetlb_entry,
>>>> +};
>>>> +
>>>> +static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start,
>>>> +                                  unsigned long end, struct page_region __user *vec)
>>>> +{
>>>> +       /* Detect illegal size, flags, len and masks */
>>>> +       if (arg->size != sizeof(struct pm_scan_arg))
>>>> +               return -EINVAL;
>>>> +       if (!arg->flags)
>>>> +               return -EINVAL;
>>>> +       if (arg->flags & ~PM_SCAN_OPS)
>>>> +               return -EINVAL;
>>>> +       if (!(end - start))
>>>> +               return -EINVAL;
>>>> +       if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask |
>>>> +            arg->return_mask) & ~PM_SCAN_BITS_ALL)
>>>> +               return -EINVAL;
>>>> +       if (!arg->required_mask && !arg->anyof_mask &&
>>>> +           !arg->excluded_mask)
>>>> +               return -EINVAL;
>>>> +       if (!arg->return_mask)
>>>> +               return -EINVAL;
>>>> +
>>>> +       /* Validate memory range */
>>>> +       if (!IS_ALIGNED(start, PAGE_SIZE))
>>>> +               return -EINVAL;
>>>> +       if (!access_ok((void __user *)start, end - start))
>>>> +               return -EFAULT;
>>>> +
>>>> +       if (IS_PM_SCAN_GET(arg->flags)) {
>>>> +               if (arg->vec_len == 0)
>>>> +                       return -EINVAL;
>>>> +               if (!vec)
>>>> +                       return -EFAULT;
>>>> +               if (!access_ok((void __user *)vec,
>>>> +                              arg->vec_len * sizeof(struct page_region)))
>>>> +                       return -EFAULT;
>>>> +       }
>>>> +
>>>> +       if (IS_PM_SCAN_WP(arg->flags) && !IS_PM_SCAN_GET(arg->flags) &&
>>>> +           arg->max_pages)
>>>> +               return -EINVAL;
>>>> +
>>>> +       return 0;
>>>> +}
>>>> +
>>>> +static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg)
>>>> +{
>>>> +       struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg;
>>>> +       unsigned long long start, end, walk_start, walk_end;
>>>> +       unsigned long empty_slots, vec_index = 0;
>>>> +       struct mmu_notifier_range range;
>>>> +       struct page_region __user *vec;
>>>> +       struct pagemap_scan_private p;
>>>> +       struct pm_scan_arg arg;
>>>> +       int ret = 0;
>>>> +
>>>> +       if (copy_from_user(&arg, uarg, sizeof(arg)))
>>>> +               return -EFAULT;
>>>> +
>>>> +       start = untagged_addr((unsigned long)arg.start);
>>>> +       end = untagged_addr((unsigned long)arg.end);
>>>> +       vec = (struct page_region __user *)untagged_addr((unsigned long)arg.vec);
>>>> +
>>>> +       ret = pagemap_scan_args_valid(&arg, start, end, vec);
>>>> +       if (ret)
>>>> +               return ret;
>>>> +
>>>> +       p.max_pages = (arg.max_pages) ? arg.max_pages : ULONG_MAX;
>>>> +       p.found_pages = 0;
>>>> +       p.required_mask = arg.required_mask;
>>>> +       p.anyof_mask = arg.anyof_mask;
>>>> +       p.excluded_mask = arg.excluded_mask;
>>>> +       p.return_mask = arg.return_mask;
>>>> +       p.flags = arg.flags;
>>>> +       p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) &
>>>> +                   PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0;
>>>> +       p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0;
>>>> +       p.vec_buf = NULL;
>>>> +       p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT;
>>>> +       p.vec_buf_index = 0;
>>>> +       p.end_addr = 0;
>>>> +
>>>> +       /*
>>>> +        * Allocate smaller buffer to get output from inside the page walk
>>>> +        * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As
>>>> +        * we want to return output to user in compact form where no two
>>>> +        * consecutive regions should be continuous and have the same flags.
>>>> +        * So store the latest element in p.cur_buf between different walks and
>>>> +        * store the p.cur_buf at the end of the walk to the user buffer.
>>>> +        */
>>>> +       if (IS_PM_SCAN_GET(p.flags)) {
>>>> +               p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf),
>>>> +                                         GFP_KERNEL);
>>>> +               if (!p.vec_buf)
>>>> +                       return -ENOMEM;
>>>> +       }
>>>> +
>>>> +       /*
>>>> +        * Protection change for the range is going to happen.
>>>> +        */
>>>> +       if (IS_PM_SCAN_WP(p.flags)) {
>>>> +               mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
>>>> +                                       mm, start, end);
>>>> +               mmu_notifier_invalidate_range_start(&range);
>>>> +       }
>>>> +
>>>> +       walk_start = walk_end = start;
>>>> +       while (walk_end < end && !ret) {
>>>> +               if (IS_PM_SCAN_GET(p.flags)) {
>>>> +                       /*
>>>> +                        * All data is copied to cur_buf first. When more data
>>>> +                        * is found, we push cur_buf to vec_buf and copy new
>>>> +                        * data to cur_buf. Subtract 1 from length as the
>>>> +                        * index of cur_buf isn't counted in length.
>>>> +                        */
>>>> +                       empty_slots = arg.vec_len - vec_index;
>>>> +                       p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1);
>>>> +               }
>>>> +
>>>
>>> I still don't understand why we don't want/need to check for pending signals.
>> We haven't added it as other existing code such as mincore() and
> 
> It doesn't convince me. There should be reasons to do or not to do
> certain things.
> We can't say how long this loop can be running, so it is the reason
> why we can want
> to check pending signals.
> 
>> pagemap_read() don't have it either.
> 
> I already explained that this case is different, because the size of
> the output buffer is
> limited for pagemap_read.
> 
>> Also mmap_read_lock_killable would return error if there is some critical single pending.\
> 
> It isn't completely true. It doesn't return errors in the fast path
> when it takes the lock right
> away. It checks signals only when it needs to wait for the lock.
> 
Okay. It seems reasonable. I'll add the following at the start of the loop:

	if (fatal_signal_pending(current))
		return -EINTR;


>>
>>>
>>>> +               ret = mmap_read_lock_killable(mm);
>>>> +               if (ret)
>>>> +                       goto out;
>>>> +
>>>> +               walk_end = min((walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK, end);
>>>> +
>>>> +               ret = walk_page_range(mm, walk_start, walk_end,
>>>> +                                     &pagemap_scan_ops, &p);
>>>> +               mmap_read_unlock(mm);
>>>> +
>>>> +               if (ret == PM_SCAN_FOUND_MAX_PAGES || ret == PM_SCAN_END_WALK)
>>>> +                       arg.start = p.end_addr;
>>>
>>> nit: this check can be moved out of the loop.
>> No, ret could get replaced by error if copy_to_user() fails. So we have to
>> do this before that.
> 
> If we fail to copy a vector, it is a fatal error and it probably doesn't matter
> what end address has been there. It is up to you to leave it here or not.
Sure.


-- 
BR,
Muhammad Usama Anjum

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Thu, Jul 20, 2023 at 09:28:52PM +0200, Michał Mirosław wrote:
> This is a massaged version of patch by Muhammad Usama Anjum [1]
> to illustrate my review comments and hopefully push the implementation
> efforts closer to conclusion. The changes are:
[...]
> +static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
> +				       unsigned long addr, unsigned long end)
> +{
> +	struct page_region *cur_buf = &p->cur_buf;
> +
> +	if (cur_buf->start != addr) {
> +		cur_buf->end = addr;
> +	} else {
> +		cur_buf->start = cur_buf->end = 0;
> +	}
> +
> +	p->end_addr = 0;

Just noticed that this is missing:

	p->found_pages -= (end - addr) / PAGE_SIZE;

> +}
[...]

Best Regards
Michał Mirosław

Re: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[kernel test robot]

Hi Michał,

kernel test robot noticed the following build warnings:

[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master v6.5-rc2 next-20230720]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Micha-Miros-aw/Re-fs-proc-task_mmu-Implement-IOCTL-for-efficient-page-table-scanning/20230721-033050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/a0b5c6776b2ed91f78a7575649f8b100e58bd3a9.1689881078.git.mirq-linux%40rere.qmqm.pl
patch subject: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning
config: mips-allyesconfig (https://download.01.org/0day-ci/archive/20230721/202307210528.2qgK1vwi-lkp@intel.com/config)
compiler: mips-linux-gcc (GCC) 12.3.0
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307210528.2qgK1vwi-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307210528.2qgK1vwi-lkp@intel.com/

All warnings (new ones prefixed by >>):

   fs/proc/task_mmu.c: In function 'pagemap_scan_test_walk':
   fs/proc/task_mmu.c:1921:13: error: implicit declaration of function 'userfaultfd_wp_async'; did you mean 'userfaultfd_wp'? [-Werror=implicit-function-declaration]
    1921 |         if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
         |             ^~~~~~~~~~~~~~~~~~~~
         |             userfaultfd_wp
   fs/proc/task_mmu.c: In function 'pagemap_scan_init_bounce_buffer':
>> fs/proc/task_mmu.c:2290:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    2290 |         p->vec_out = (void __user *)p->arg.vec;
         |                      ^
   fs/proc/task_mmu.c: At top level:
   fs/proc/task_mmu.c:1967:13: warning: 'pagemap_scan_backout_range' defined but not used [-Wunused-function]
    1967 | static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
         |             ^~~~~~~~~~~~~~~~~~~~~~~~~~
   cc1: some warnings being treated as errors


vim +2290 fs/proc/task_mmu.c

  2264	
  2265	static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p)
  2266	{
  2267		if (!p->arg.vec_len) {
  2268			/*
  2269			 * An arbitrary non-page-aligned sentinel value for
  2270			 * pagemap_scan_push_range().
  2271			 */
  2272			p->cur_buf.start = p->cur_buf.end = ULLONG_MAX;
  2273			return 0;
  2274		}
  2275	
  2276		/*
  2277		 * Allocate a smaller buffer to get output from inside the page
  2278		 * walk functions and walk the range in PAGEMAP_WALK_SIZE chunks.
  2279		 * The last range is always stored in p.cur_buf to allow coalescing
  2280		 * consecutive ranges that have the same categories returned across
  2281		 * walk_page_range() calls.
  2282		 */
  2283		p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT,
  2284				       p->arg.vec_len - 1);
  2285		p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf),
  2286					   GFP_KERNEL);
  2287		if (!p->vec_buf)
  2288			return -ENOMEM;
  2289	
> 2290		p->vec_out = (void __user *)p->arg.vec;
  2291	
  2292		return 0;
  2293	}
  2294	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[kernel test robot]

Hi Michał,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.5-rc2 next-20230720]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Micha-Miros-aw/Re-fs-proc-task_mmu-Implement-IOCTL-for-efficient-page-table-scanning/20230721-033050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/a0b5c6776b2ed91f78a7575649f8b100e58bd3a9.1689881078.git.mirq-linux%40rere.qmqm.pl
patch subject: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning
config: arc-randconfig-r035-20230720 (https://download.01.org/0day-ci/archive/20230721/202307211030.2CJH6TkM-lkp@intel.com/config)
compiler: arceb-elf-gcc (GCC) 12.3.0
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307211030.2CJH6TkM-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307211030.2CJH6TkM-lkp@intel.com/

All errors (new ones prefixed by >>):

   fs/proc/task_mmu.c: In function 'pagemap_scan_test_walk':
   fs/proc/task_mmu.c:1921:13: error: implicit declaration of function 'userfaultfd_wp_async'; did you mean 'userfaultfd_wp'? [-Werror=implicit-function-declaration]
    1921 |         if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
         |             ^~~~~~~~~~~~~~~~~~~~
         |             userfaultfd_wp
   fs/proc/task_mmu.c: In function 'pagemap_scan_pte_hole':
>> fs/proc/task_mmu.c:2200:19: error: implicit declaration of function 'uffd_wp_range' [-Werror=implicit-function-declaration]
    2200 |         int err = uffd_wp_range(vma, addr, end - addr, true);
         |                   ^~~~~~~~~~~~~
   fs/proc/task_mmu.c: In function 'pagemap_scan_init_bounce_buffer':
   fs/proc/task_mmu.c:2290:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    2290 |         p->vec_out = (void __user *)p->arg.vec;
         |                      ^
   fs/proc/task_mmu.c: At top level:
   fs/proc/task_mmu.c:1967:13: warning: 'pagemap_scan_backout_range' defined but not used [-Wunused-function]
    1967 | static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
         |             ^~~~~~~~~~~~~~~~~~~~~~~~~~
   cc1: some warnings being treated as errors


vim +/uffd_wp_range +2200 fs/proc/task_mmu.c

  2182	
  2183	static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
  2184					 int depth, struct mm_walk *walk)
  2185	{
  2186		struct pagemap_scan_private *p = walk->private;
  2187		struct vm_area_struct *vma = walk->vma;
  2188		int ret;
  2189	
  2190		if (!vma || !pagemap_scan_is_interesting_page(p->cur_vma_category, p))
  2191			return 0;
  2192	
  2193		ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end);
  2194		if (addr == end)
  2195			return ret;
  2196	
  2197		if (~p->arg.flags & PM_SCAN_WP_MATCHING)
  2198			return ret;
  2199	
> 2200		int err = uffd_wp_range(vma, addr, end - addr, true);
  2201		if (err < 0)
  2202			ret = err;
  2203	
  2204		return ret;
  2205	}
  2206	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

Thank you Michał.

On 7/21/23 12:28 AM, Michał Mirosław wrote:
>   b. rename match "flags" to 'page categories' everywhere - this makes
> 	it easier to differentiate the ioctl()s categorisation of pages
> 	from struct page flags;
>   c. change {required + excluded} to {inverted + required}. This was
> 	rejected before, but I'd like to illustrate the difference.
> 	Old interface can be translated to the new by:
> 		categories_inverted = excluded_mask
> 		categories_mask = required_mask | excluded_mask
> 		categories_anyof_mask = anyof_mask
> 	The new way allows filtering by: A & (B | !C)
> 		categories_inverted = C
> 		categories_mask = A
> 		categories_anyof_mask = B | C
Andrei and Danylo,

Are you okay with these masks? It were you two who had proposed these.


-- 
BR,
Muhammad Usama Anjum

Re: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[kernel test robot]

Hi Michał,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.5-rc2 next-20230720]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Micha-Miros-aw/Re-fs-proc-task_mmu-Implement-IOCTL-for-efficient-page-table-scanning/20230721-033050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/a0b5c6776b2ed91f78a7575649f8b100e58bd3a9.1689881078.git.mirq-linux%40rere.qmqm.pl
patch subject: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning
config: i386-randconfig-r022-20230720 (https://download.01.org/0day-ci/archive/20230721/202307211337.5dwCMeHb-lkp@intel.com/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307211337.5dwCMeHb-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307211337.5dwCMeHb-lkp@intel.com/

All errors (new ones prefixed by >>):

>> fs/proc/task_mmu.c:1921:6: error: call to undeclared function 'userfaultfd_wp_async'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
           if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
               ^
>> fs/proc/task_mmu.c:2200:12: error: call to undeclared function 'uffd_wp_range'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
           int err = uffd_wp_range(vma, addr, end - addr, true);
                     ^
   2 errors generated.


vim +/userfaultfd_wp_async +1921 fs/proc/task_mmu.c

  1913	
  1914	static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
  1915					  struct mm_walk *walk)
  1916	{
  1917		struct pagemap_scan_private *p = walk->private;
  1918		struct vm_area_struct *vma = walk->vma;
  1919		unsigned long vma_category = 0;
  1920	
> 1921		if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
  1922			vma_category |= PAGE_IS_WPASYNC;
  1923		else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
  1924			return -EPERM;
  1925	
  1926		if (vma->vm_flags & VM_PFNMAP)
  1927			return 1;
  1928	
  1929		if (!pagemap_scan_is_interesting_vma(vma_category, p))
  1930			return 1;
  1931	
  1932		p->cur_vma_category = vma_category;
  1933		return 0;
  1934	}
  1935	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[kernel test robot]

Hi Michał,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.5-rc2 next-20230721]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Micha-Miros-aw/Re-fs-proc-task_mmu-Implement-IOCTL-for-efficient-page-table-scanning/20230721-033050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/a0b5c6776b2ed91f78a7575649f8b100e58bd3a9.1689881078.git.mirq-linux%40rere.qmqm.pl
patch subject: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning
config: powerpc-randconfig-r015-20230720 (https://download.01.org/0day-ci/archive/20230721/202307211507.xOl45LiR-lkp@intel.com/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a)
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307211507.xOl45LiR-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307211507.xOl45LiR-lkp@intel.com/

All errors (new ones prefixed by >>):

>> fs/proc/task_mmu.c:1921:6: error: call to undeclared function 'userfaultfd_wp_async'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    1921 |         if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
         |             ^
>> fs/proc/task_mmu.c:2200:12: error: call to undeclared function 'uffd_wp_range'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    2200 |         int err = uffd_wp_range(vma, addr, end - addr, true);
         |                   ^
   2 errors generated.


vim +/userfaultfd_wp_async +1921 fs/proc/task_mmu.c

  1913	
  1914	static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
  1915					  struct mm_walk *walk)
  1916	{
  1917		struct pagemap_scan_private *p = walk->private;
  1918		struct vm_area_struct *vma = walk->vma;
  1919		unsigned long vma_category = 0;
  1920	
> 1921		if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
  1922			vma_category |= PAGE_IS_WPASYNC;
  1923		else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
  1924			return -EPERM;
  1925	
  1926		if (vma->vm_flags & VM_PFNMAP)
  1927			return 1;
  1928	
  1929		if (!pagemap_scan_is_interesting_vma(vma_category, p))
  1930			return 1;
  1931	
  1932		p->cur_vma_category = vma_category;
  1933		return 0;
  1934	}
  1935	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/21/23 12:28 AM, Michał Mirosław wrote:
> This is a massaged version of patch by Muhammad Usama Anjum [1]
> to illustrate my review comments and hopefully push the implementation
> efforts closer to conclusion. The changes are:
Thank you so much for this effort. I also want to reach conclusion. I'll
agree with all the changes which don't affect me. But some requirements
aren't being fulfilled with this current design.

> 
> 1. the API:
> 
>   a. return ranges as {begin, end} instead of {begin + len};
Agreed.

>   b. rename match "flags" to 'page categories' everywhere - this makes
> 	it easier to differentiate the ioctl()s categorisation of pages
> 	from struct page flags;
I've no problem with it.

#define PAGE_IS_WPASYNC		(1 << 0)
#define PAGE_IS_WRITTEN		(1 << 1)
You have another new flag PAGE_IS_WPASYNC. But there is no application of
PAGE_IS_WPASYNC. We must not add a flag which don't have any user.

>   c. change {required + excluded} to {inverted + required}. This was
> 	rejected before, but I'd like to illustrate the difference.
> 	Old interface can be translated to the new by:
> 		categories_inverted = excluded_mask
> 		categories_mask = required_mask | excluded_mask
> 		categories_anyof_mask = anyof_mask
> 	The new way allows filtering by: A & (B | !C)
> 		categories_inverted = C
> 		categories_mask = A
> 		categories_anyof_mask = B | C
I'm still unable to get the idea of inverted masks. IIRC Andei had also not
supported/accepted this masking scheme. But I'll be okay with it if he
supports this masking.

>   d. change the ioctl to be a SCAN with optional WP. Addressing the
> 	original use-case, GetWriteWatch() can be implemented as:
As I've mentioned several times previously (without the name of
ResetWriteWatch()) that we need exclusive WP without GET. This could be
implemented with UFFDIO_WRITEPROTECT. But when we use UFFDIO_WRITEPROTECT,
we hit some special case and performance is very slow. So with UFFD WP
expert, Peter Xu we have decided to put exclusive WP in this IOCTL for
implementation of ResetWriteWatch().

A lot of simplification of the patch is made possible because of not
keeping exclusive WP. (You have also written some quality code, more better.)

> 
> 		memset(&args, 0, sizeof(args));
> 		args.start = lpBaseAddress;
> 		args.end = lpBaseAddress + dwRegionSize;
> 		args.max_pages = *lpdwCount;
> 		*lpdwGranularity = PAGE_SIZE;
> 		args.flags = PM_SCAN_CHECK_WPASYNC;
> 		if (dwFlags & WRITE_WATCH_FLAG_RESET)
> 			args.flags |= PM_SCAN_WP_MATCHED;
> 		args.categories_mask = PAGE_IS_WRITTEN;
> 		args.return_mask = PAGE_IS_WRITTEN;
> 		args.vec = tmp_buffer;
> 		args.vec_len = tmp_buffer_len;
> 		do {
> 			n = ioctl(..., &args);
> 			if (n < 0)
> 				return error(n);
> 			for (page_region *p = ...) {
> 				write page addresses;
> 			}
> 			if (output_full)
> 				return ...;
> 		} while (args.start != args.end);
> 		return ...;
> 
> 	For the CRIU's usecase of live migration this would be similar,
> 	but without using max_pages limit. The other CRIU usecase is to
> 	efficiently dump sparse memory mappings - those could use the
> 	full range of page category filtering as needed in a checkpoint
> 	phase.a
>   e. allow no-op calls
I've no issue with it.

> 
> 2. the implementation:
>   a. gather the page-categorising and write-protecting code in one place;
Agreed.

>   b. optimization: add whole-vma skipping for WP usecase;
I don't know who can benefit from it. Do you have any user in mind? When
the user come of this optimization, this can be added later.

>   c. extracted output limiting code to pagemap_scan_output();
If user passes half THP, current code wouldn't split huge page and WP the
whole THP. We would loose the dirty state of other half huge page. This is
bug. consoliding the output limiting code looks optimal, but we'll need to
same limiting code to detect if full THP hasn't been passed in case of THP
and HugeTLB.

>   d. extracted range coalescing to pagemap_scan_push_range();
My old pagemap_scan_output has become pagemap_scan_push_range().

>   e. extracted THP entry handling to pagemap_scan_thp_entry();
Good. But I didn't found value in seperating it just like other historic
pagemap code.

>   f. added a shortcut for non-WP hugetlb scan; avoids conditional
> 	locking;
Yeah, some if conditions have been removed. But overall did couple of calls
to is_interesting and scan_output functions instead of just one.

>   g. extracted scan buffer handling code out of do_pagemap_scan();
>   h. rework output code to always try to write pending ranges; if EFAULT
> 	is generated it always overwrites the original error code;
> 	(the case of SIGKILL is needlessly trying to write the output
> 	now, but this should be rare case and ignoring it makes the code
> 	not needing a goto)
> 
> PTAL and comment/fix. Compile tested with allyesconfig. For the (unlikely)
> case that the patch is good as is:
> 
> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
> 
> [1] https://lore.kernel.org/lkml/20230713101415.108875-3-usama.anjum@collabora.com/
>   "[PATCH v25 2/5] fs/proc/task_mmu: Implement IOCTL to get and
>    optionally clear info about PTEs"
> 
> Best Regards
> Michał Mirosław
> 
> Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
> ---
>  fs/proc/task_mmu.c      | 649 ++++++++++++++++++++++++++++++++++++++++
>  include/linux/hugetlb.h |   1 +
>  include/uapi/linux/fs.h |  56 ++++
>  mm/hugetlb.c            |   2 +-
>  4 files changed, 707 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index c1e6531cb02a..7d624bafecf9 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -19,6 +19,8 @@
>  #include <linux/shmem_fs.h>
>  #include <linux/uaccess.h>
>  #include <linux/pkeys.h>
> +#include <linux/minmax.h>
> +#include <linux/overflow.h>
>  
>  #include <asm/elf.h>
>  #include <asm/tlb.h>
> @@ -1749,11 +1751,658 @@ static int pagemap_release(struct inode *inode, struct file *file)
>  	return 0;
>  }
>  
> +#define PM_SCAN_CATEGORIES	(PAGE_IS_WPASYNC | PAGE_IS_WRITTEN | \
> +				 PAGE_IS_FILE |	PAGE_IS_PRESENT | \
> +				 PAGE_IS_SWAPPED | PAGE_IS_PFNZERO)
> +#define PM_SCAN_FLAGS		(PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
> +
> +struct pagemap_scan_private {
> +	struct pm_scan_arg arg;
> +	unsigned long cur_vma_category;
> +	struct page_region *vec_buf, cur_buf;
> +	unsigned long vec_buf_len, vec_buf_index, found_pages, end_addr;
> +	struct page_region __user* vec_out;
> +};
> +
> +static unsigned long pagemap_page_category(struct vm_area_struct *vma, unsigned long addr, pte_t pte)
> +{
> +	unsigned long categories = 0;
> +
> +	if (pte_present(pte)) {
> +		struct page *page = vm_normal_page(vma, addr, pte);
> +
> +		categories |= PAGE_IS_PRESENT;
> +		if (!pte_uffd_wp(pte))
> +			categories |= PAGE_IS_WRITTEN;
> +		if (page && !PageAnon(page))
> +			categories |= PAGE_IS_FILE;
> +		if (is_zero_pfn(pte_pfn(pte)))
> +			categories |= PAGE_IS_PFNZERO;
> +	} else if (is_swap_pte(pte)) {
> +		swp_entry_t swp = pte_to_swp_entry(pte);
> +
> +		categories |= PAGE_IS_SWAPPED;
> +		if (!pte_swp_uffd_wp_any(pte))
> +			categories |= PAGE_IS_WRITTEN;
> +		if (is_pfn_swap_entry(swp) && !PageAnon(pfn_swap_entry_to_page(swp)))
> +			categories |= PAGE_IS_FILE;
> +	}
> +
> +	return categories;
> +}
> +
> +static void make_uffd_wp_pte(struct vm_area_struct *vma,
> +			     unsigned long addr, pte_t *pte)
> +{
> +	pte_t ptent = ptep_get(pte);
> +
> +	if (pte_present(ptent)) {
> +		pte_t old_pte;
> +
> +		old_pte = ptep_modify_prot_start(vma, addr, pte);
> +		ptent = pte_mkuffd_wp(ptent);
> +		ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
> +	} else if (is_swap_pte(ptent)) {
> +		ptent = pte_swp_mkuffd_wp(ptent);
> +		set_pte_at(vma->vm_mm, addr, pte, ptent);
> +	} else {
> +		set_pte_at(vma->vm_mm, addr, pte,
> +			   make_pte_marker(PTE_MARKER_UFFD_WP));
> +	}
> +}
> +
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> +static unsigned long pagemap_thp_category(pmd_t pmd)
> +{
> +	unsigned long categories = 0;
> +
> +	if (pmd_present(pmd)) {
> +		categories |= PAGE_IS_PRESENT;
> +		if (!pmd_uffd_wp(pmd))
> +			categories |= PAGE_IS_WRITTEN;
> +		if (is_zero_pfn(pmd_pfn(pmd)))
> +			categories |= PAGE_IS_PFNZERO;
> +	} else if (is_swap_pmd(pmd)) {
> +		categories |= PAGE_IS_SWAPPED;
> +		if (!pmd_swp_uffd_wp(pmd))
> +			categories |= PAGE_IS_WRITTEN;
> +	}
> +
> +	return categories;
> +}
> +
> +static void make_uffd_wp_pmd(struct vm_area_struct *vma,
> +			     unsigned long addr, pmd_t *pmdp)
> +{
> +	pmd_t old, pmd = *pmdp;
> +
> +	if (pmd_present(pmd)) {
> +		old = pmdp_invalidate_ad(vma, addr, pmdp);
> +		pmd = pmd_mkuffd_wp(old);
> +		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
> +	} else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
> +		pmd = pmd_swp_mkuffd_wp(pmd);
> +		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
> +	}
> +}
> +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> +
> +#ifdef CONFIG_HUGETLB_PAGE
> +static unsigned long pagemap_hugetlb_category(pte_t pte)
> +{
> +	unsigned long categories = 0;
> +
> +	if (pte_present(pte)) {
> +		categories |= PAGE_IS_PRESENT;
> +		if (!huge_pte_uffd_wp(pte))
> +			categories |= PAGE_IS_WRITTEN;
> +		if (!PageAnon(pte_page(pte)))
> +			categories |= PAGE_IS_FILE;
> +		if (is_zero_pfn(pte_pfn(pte)))
> +			categories |= PAGE_IS_PFNZERO;
> +	} else if (is_swap_pte(pte)) {
> +		categories |= PAGE_IS_SWAPPED;
> +		if (!pte_swp_uffd_wp_any(pte))
> +			categories |= PAGE_IS_WRITTEN;
> +	}
> +
> +	return categories;
> +}
> +
> +static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
> +				  unsigned long addr, pte_t *ptep,
> +				  pte_t ptent)
> +{
> +	if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
> +		return;
> +
> +	if (is_hugetlb_entry_migration(ptent))
> +		set_huge_pte_at(vma->vm_mm, addr, ptep,
> +				pte_swp_mkuffd_wp(ptent));
> +	else if (!huge_pte_none(ptent))
> +		huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
> +					     huge_pte_mkuffd_wp(ptent));
> +	else
> +		set_huge_pte_at(vma->vm_mm, addr, ptep,
> +				make_pte_marker(PTE_MARKER_UFFD_WP));
> +}
> +#endif /* CONFIG_HUGETLB_PAGE */
> +
> +static bool pagemap_scan_is_interesting_page(unsigned long categories,
> +					     const struct pagemap_scan_private *p)
> +{
> +	categories ^= p->arg.category_inverted;
> +	if ((categories & p->arg.category_mask) != p->arg.category_mask)
> +		return false;
> +	if (p->arg.category_anyof_mask && !(categories & p->arg.category_anyof_mask))
> +		return false;
> +
> +	return true;
> +}
> +
> +static bool pagemap_scan_is_interesting_vma(unsigned long categories,
> +					    const struct pagemap_scan_private *p)
> +{
> +	unsigned long required = p->arg.category_mask & PAGE_IS_WPASYNC;
> +
> +	categories ^= p->arg.category_inverted;
> +	if ((categories & required) != required)
> +		return false;
> +	return true;
> +}
> +
> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
> +				  struct mm_walk *walk)
> +{
> +	struct pagemap_scan_private *p = walk->private;
> +	struct vm_area_struct *vma = walk->vma;
> +	unsigned long vma_category = 0;
> +
> +	if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
> +		vma_category |= PAGE_IS_WPASYNC;
> +	else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
> +		return -EPERM;
> +
> +	if (vma->vm_flags & VM_PFNMAP)
> +		return 1;
> +
> +	if (!pagemap_scan_is_interesting_vma(vma_category, p))
> +		return 1;
> +
> +	p->cur_vma_category = vma_category;
> +	return 0;
> +}
> +
> +static bool pagemap_scan_push_range(unsigned long categories,
> +				    struct pagemap_scan_private *p,
> +				    unsigned long addr, unsigned long end)
> +{
> +	struct page_region *cur_buf = &p->cur_buf;
> +
> +	/*
> +	 * When there is no output buffer provided at all, the sentinel values
> +	 * won't match here. There is no other way for `cur_buf->end` to be
> +	 * non-zero other than it being non-empty.
> +	 */
> +	if (addr == cur_buf->end && categories == cur_buf->categories) {
> +		cur_buf->end = end;
> +		return true;
> +	}
> +
> +	if (cur_buf->end) {
> +		if (p->vec_buf_index >= p->vec_buf_len)
> +			return false;
> +
> +		memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
> +		       sizeof(*p->vec_buf));
> +		++p->vec_buf_index;
> +	}
> +
> +	cur_buf->start = addr;
> +	cur_buf->end = end;
> +	cur_buf->categories = categories;
> +	return true;
> +}
> +
> +static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
> +				       unsigned long addr, unsigned long end)
> +{
> +	struct page_region *cur_buf = &p->cur_buf;
> +
> +	if (cur_buf->start != addr) {
> +		cur_buf->end = addr;
> +	} else {
> +		cur_buf->start = cur_buf->end = 0;
> +	}
> +
> +	p->end_addr = 0;
> +}
> +
> +static int pagemap_scan_output(unsigned long categories,
> +			       struct pagemap_scan_private *p,
> +			       unsigned long addr, unsigned long *end)
> +{
> +	unsigned long n_pages, total_pages;
> +	int ret = 0;
> +
> +	if (!p->arg.return_mask)
> +		return 0;
> +
> +	categories &= p->arg.return_mask;
> +
> +	n_pages = (*end - addr) / PAGE_SIZE;
> +	if (check_add_overflow(p->found_pages, n_pages, &total_pages) || total_pages > p->arg.max_pages) {
> +		size_t n_too_much = total_pages - p->arg.max_pages;
> +		*end -= n_too_much * PAGE_SIZE;
> +		n_pages -= n_too_much;
> +		ret = -ENOSPC;
> +	}
> +
> +	if (!pagemap_scan_push_range(categories, p, addr, *end)) {
> +		*end = addr;
> +		n_pages = 0;
> +		ret = -ENOSPC;
> +	}
> +
> +	p->found_pages += n_pages;
> +	if (ret)
> +		p->end_addr = *end;
> +	return ret;
> +}
> +
> +static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
> +				  unsigned long end, struct mm_walk *walk)
> +{
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> +	struct pagemap_scan_private *p = walk->private;
> +	struct vm_area_struct *vma = walk->vma;
> +	unsigned long categories;
> +	spinlock_t *ptl;
> +	int ret = 0;
> +
> +	ptl = pmd_trans_huge_lock(pmd, vma);
> +	if (!ptl)
> +		return -ENOENT;
> +
> +	categories = p->cur_vma_category | pagemap_thp_category(*pmd);
> +
> +	if (!pagemap_scan_is_interesting_page(categories, p))
> +		goto out_unlock;
> +
> +	ret = pagemap_scan_output(categories, p, start, &end);
> +	if (start == end)
> +		goto out_unlock;
> +
> +	if (~p->arg.flags & PM_SCAN_WP_MATCHING)
> +		goto out_unlock;
> +	if (~categories & PAGE_IS_WRITTEN)
> +		goto out_unlock;
> +
> +	/*
> +	 * Break huge page into small pages if the WP operation
> +	 * need to be performed is on a portion of the huge page.
> +	 */
> +	if (ret == -ENOSPC) {
> +		spin_unlock(ptl);
> +		split_huge_pmd(vma, pmd, start);
> +		pagemap_scan_backout_range(p, start, end);
> +		return -ENOENT;
> +	}
> +
> +	make_uffd_wp_pmd(vma, start, pmd);
> +	flush_tlb_range(vma, start, end);
> +out_unlock:
> +	spin_unlock(ptl);
> +	return ret;
> +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
> +	return -ENOENT;
> +#endif
> +}
> +
> +static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
> +				  unsigned long end, struct mm_walk *walk)
> +{
> +	struct pagemap_scan_private *p = walk->private;
> +	struct vm_area_struct *vma = walk->vma;
> +	pte_t *pte, *start_pte;
> +	unsigned long addr;
> +	bool flush = false;
> +	spinlock_t *ptl;
> +	int ret;
> +
> +	arch_enter_lazy_mmu_mode();
> +
> +	ret = pagemap_scan_thp_entry(pmd, start, end, walk);
> +	if (ret != -ENOENT) {
> +		arch_leave_lazy_mmu_mode();
> +		return ret;
> +	}
> +
> +	start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
> +	if (!pte) {
> +		arch_leave_lazy_mmu_mode();
> +		walk->action = ACTION_AGAIN;
> +		return 0;
> +	}
> +
> +	for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
> +		unsigned long categories = p->cur_vma_category |
> +			pagemap_page_category(vma, addr, ptep_get(pte));
> +		unsigned long next = addr + PAGE_SIZE;
> +
> +		if (!pagemap_scan_is_interesting_page(categories, p))
> +			continue;
> +
> +		ret = pagemap_scan_output(categories, p, addr, &next);
> +		if (next == addr)
> +			break;
> +
> +		if (~p->arg.flags & PM_SCAN_WP_MATCHING)
> +			continue;
> +		if (~categories & PAGE_IS_WRITTEN)
> +			continue;
> +
> +		make_uffd_wp_pte(vma, addr, pte);
> +		if (!flush) {
> +			start = addr;
> +			flush = true;
> +		}
> +	}
> +
> +	if (flush)
> +		flush_tlb_range(vma, start, addr);
> +
> +	pte_unmap_unlock(start_pte, ptl);
> +	arch_leave_lazy_mmu_mode();
> +
> +	cond_resched();
> +	return ret;
> +}
> +
> +#ifdef CONFIG_HUGETLB_PAGE
> +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
> +				      unsigned long start, unsigned long end,
> +				      struct mm_walk *walk)
> +{
> +	unsigned long n_pages = (end - start)/PAGE_SIZE;
> +	struct pagemap_scan_private *p = walk->private;
> +	struct vm_area_struct *vma = walk->vma;
> +	unsigned long categories;
> +	spinlock_t *ptl;
> +	int ret = 0;
> +	pte_t pte;
> +
> +	if (~p->arg.flags & PM_SCAN_WP_MATCHING) {
> +		/* Go the short route when not write-protecting pages. */
> +
> +		pte = huge_ptep_get(ptep);
> +		categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
> +
> +		if (!pagemap_scan_is_interesting_page(categories, p))
> +			return 0;
> +
> +		return pagemap_scan_output(categories, p, start, &end);
> +	}
> +
> +	if (n_pages < HPAGE_SIZE/PAGE_SIZE) {
> +		/*
> +		 * Partial hugetlb page clear isn't supported
> +		 */
> +		p->end_addr = start;
> +		return -EINVAL;
> +	}
> +
> +	i_mmap_lock_write(vma->vm_file->f_mapping);
> +	ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
> +
> +	pte = huge_ptep_get(ptep);
> +	categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
> +
> +	if (!pagemap_scan_is_interesting_page(categories, p))
> +		goto out_unlock;
> +
> +	ret = pagemap_scan_output(categories, p, start, &end);
> +	if (start == end)
> +		goto out_unlock;
> +
> +	if (categories & PAGE_IS_WRITTEN) {
> +		make_uffd_wp_huge_pte(vma, start, ptep, pte);
> +		flush_hugetlb_tlb_range(vma, start, end);
> +	}
> +
> +out_unlock:
> +	spin_unlock(ptl);
> +	i_mmap_unlock_write(vma->vm_file->f_mapping);
> +
> +	return ret;
> +}
> +#else
> +#define pagemap_scan_hugetlb_entry NULL
> +#endif
> +
> +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
> +				 int depth, struct mm_walk *walk)
> +{
> +	struct pagemap_scan_private *p = walk->private;
> +	struct vm_area_struct *vma = walk->vma;
> +	int ret;
> +
> +	if (!vma || !pagemap_scan_is_interesting_page(p->cur_vma_category, p))
> +		return 0;
> +
> +	ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end);
> +	if (addr == end)
> +		return ret;
> +
> +	if (~p->arg.flags & PM_SCAN_WP_MATCHING)
> +		return ret;
> +
> +	int err = uffd_wp_range(vma, addr, end - addr, true);
> +	if (err < 0)
> +		ret = err;
> +
> +	return ret;
> +}
> +
> +static const struct mm_walk_ops pagemap_scan_ops = {
> +	.test_walk = pagemap_scan_test_walk,
> +	.pmd_entry = pagemap_scan_pmd_entry,
> +	.pte_hole = pagemap_scan_pte_hole,
> +	.hugetlb_entry = pagemap_scan_hugetlb_entry,
> +};
> +
> +static int pagemap_scan_get_args(struct pm_scan_arg *arg,
> +				 unsigned long uarg)
> +{
> +	unsigned long start, end, vec;
> +
> +	if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
> +		return -EFAULT;
> +
> +	if (arg->size != sizeof(struct pm_scan_arg))
> +		return -EINVAL;
> +
> +	/* Validate requested features */
> +	if (arg->flags & ~PM_SCAN_FLAGS)
> +		return -EINVAL;
> +	if ((arg->category_inverted | arg->category_mask |
> +	     arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
> +		return -EINVAL;
> +
> +	start = untagged_addr((unsigned long)arg->start);
> +	end = untagged_addr((unsigned long)arg->end);
> +	vec = untagged_addr((unsigned long)arg->vec);
> +
> +	/* Validate memory pointers */
> +	if (!IS_ALIGNED(start, PAGE_SIZE))
> +		return -EINVAL;
> +	if (!access_ok((void __user *)start, end - start))
> +		return -EFAULT;
> +	if (arg->vec_len && !vec)
> +		return -EFAULT;
> +	if (!access_ok((void __user *)vec,
> +		       arg->vec_len * sizeof(struct page_region)))
> +		return -EFAULT;
> +
> +	/* Fixup default values */
> +	if (!arg->max_pages)
> +		arg->max_pages = ULONG_MAX;
> +
> +	return 0;
> +}
> +
> +static int pagemap_scan_writeback_args(struct pm_scan_arg *arg,
> +				       unsigned long uargl)
> +{
> +	struct pm_scan_arg __user *uarg	= (void __user *)uargl;
> +
> +	if (copy_to_user(&uarg->start, &arg->start, sizeof(arg->start)))
> +		return -EFAULT;
> +
> +	return 0;
> +}
> +
> +static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p)
> +{
> +	if (!p->arg.vec_len) {
> +		/*
> +		 * An arbitrary non-page-aligned sentinel value for
> +		 * pagemap_scan_push_range().
> +		 */
> +		p->cur_buf.start = p->cur_buf.end = ULLONG_MAX;
> +		return 0;
> +	}
> +
> +	/*
> +	 * Allocate a smaller buffer to get output from inside the page
> +	 * walk functions and walk the range in PAGEMAP_WALK_SIZE chunks.
> +	 * The last range is always stored in p.cur_buf to allow coalescing
> +	 * consecutive ranges that have the same categories returned across
> +	 * walk_page_range() calls.
> +	 */
> +	p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT,
> +			       p->arg.vec_len - 1);
> +	p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf),
> +				   GFP_KERNEL);
> +	if (!p->vec_buf)
> +		return -ENOMEM;
> +
> +	p->vec_out = (void __user *)p->arg.vec;
> +
> +	return 0;
> +}
> +
> +static int pagemap_scan_flush_buffer(struct pagemap_scan_private *p)
> +{
> +	const struct page_region *buf = p->vec_buf;
> +	int n = (int)p->vec_buf_index;
> +
> +	if (!n)
> +		return 0;
> +
> +	if (copy_to_user(p->vec_out, buf, n * sizeof(*buf)))
> +		return -EFAULT;
> +
> +	p->arg.vec_len -= n;
> +	p->vec_out += n;
> +
> +	p->vec_buf_index = 0;
> +	p->vec_buf_len = min_t(size_t, p->vec_buf_len, p->arg.vec_len - 1);
> +
> +	return n;
> +}
> +
> +static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg)
> +{
> +	unsigned long walk_start, walk_end;
> +	struct mmu_notifier_range range;
> +	struct pagemap_scan_private p;
> +	size_t n_ranges_out = 0;
> +	int ret;
> +
> +	memset(&p, 0, sizeof(p));
> +	ret = pagemap_scan_get_args(&p.arg, uarg);
> +	if (ret)
> +		return ret;
> +
> +	ret = pagemap_scan_init_bounce_buffer(&p);
> +	if (ret)
> +		return ret;
> +
> +	/* Protection change for the range is going to happen. */
> +	if (p.arg.flags & PM_SCAN_WP_MATCHING) {
> +		mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
> +					mm, p.arg.start, p.arg.end);
> +		mmu_notifier_invalidate_range_start(&range);
> +	}
> +
> +	walk_start = walk_end = p.arg.start;
> +	for (; walk_end != p.arg.end; walk_start = walk_end) {
> +		int n_out;
> +		walk_end = min_t(unsigned long,
> +				 (walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK,
> +				 p.arg.end);
> +
> +		ret = mmap_read_lock_killable(mm);
> +		if (ret)
> +			break;
> +		ret = walk_page_range(mm, walk_start, walk_end,
> +				      &pagemap_scan_ops, &p);
> +		mmap_read_unlock(mm);
> +
> +		n_out = pagemap_scan_flush_buffer(&p);
> +		if (n_out < 0)
> +			ret = n_out;
> +		else
> +			n_ranges_out += n_out;
> +
> +		if (ret)
> +			break;
> +	}
> +
> +	if (p.cur_buf.start != p.cur_buf.end) {
> +		if (copy_to_user(p.vec_out, &p.cur_buf, sizeof(p.cur_buf)))
> +			ret = -EFAULT;
> +		else
> +			++n_ranges_out;
> +	}
> +
> +	/* ENOSPC signifies early stop (buffer full) from the walk. */
> +	if (!ret || ret == -ENOSPC)
> +		ret = n_ranges_out;
> +
> +	p.arg.start = p.end_addr ? p.end_addr : walk_start;
> +	if (pagemap_scan_writeback_args(&p.arg, uarg))
> +		ret = -EFAULT;
> +
> +	if (p.arg.flags & PM_SCAN_WP_MATCHING)
> +		mmu_notifier_invalidate_range_end(&range);
> +
> +	kfree(p.vec_buf);
> +	return ret;
> +}
> +
> +static long do_pagemap_cmd(struct file *file, unsigned int cmd,
> +			   unsigned long arg)
> +{
> +	struct mm_struct *mm = file->private_data;
> +
> +	switch (cmd) {
> +	case PAGEMAP_SCAN:
> +		return do_pagemap_scan(mm, arg);
> +
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
>  const struct file_operations proc_pagemap_operations = {
>  	.llseek		= mem_lseek, /* borrow this */
>  	.read		= pagemap_read,
>  	.open		= pagemap_open,
>  	.release	= pagemap_release,
> +	.unlocked_ioctl = do_pagemap_cmd,
> +	.compat_ioctl	= do_pagemap_cmd,
>  };
>  #endif /* CONFIG_PROC_PAGE_MONITOR */
>  
> diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
> index 9f4bac3df59e..4f5fed605538 100644
> --- a/include/linux/hugetlb.h
> +++ b/include/linux/hugetlb.h
> @@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
>  		unsigned long cp_flags);
>  
>  bool is_hugetlb_entry_migration(pte_t pte);
> +bool is_hugetlb_entry_hwpoisoned(pte_t pte);
>  void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
>  
>  #else /* !CONFIG_HUGETLB_PAGE */
> diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> index b7b56871029c..db39442befcb 100644
> --- a/include/uapi/linux/fs.h
> +++ b/include/uapi/linux/fs.h
> @@ -305,4 +305,60 @@ typedef int __bitwise __kernel_rwf_t;
>  #define RWF_SUPPORTED	(RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
>  			 RWF_APPEND)
>  
> +/* Pagemap ioctl */
> +#define PAGEMAP_SCAN	_IOWR('f', 16, struct pm_scan_arg)
> +
> +/* Bits are set in flags of the page_region and masks in pm_scan_args */
> +#define PAGE_IS_WPASYNC		(1 << 0)
> +#define PAGE_IS_WRITTEN		(1 << 1)
> +#define PAGE_IS_FILE		(1 << 2)
> +#define PAGE_IS_PRESENT		(1 << 3)
> +#define PAGE_IS_SWAPPED		(1 << 4)
> +#define PAGE_IS_PFNZERO		(1 << 5)
> +
> +/*
> + * struct page_region - Page region with flags
> + * @start:	Start of the region
> + * @end:	End of the region (exclusive)
> + * @categories:	PAGE_IS_* category bitmask for the region
> + */
> +struct page_region {
> +	__u64 start;
> +	__u64 end;
> +	__u64 categories;
> +};
> +
> +/* Flags for PAGEMAP_SCAN ioctl */
> +#define PM_SCAN_WP_MATCHING	(1 << 0)	/* Write protect the pages matched. */
> +#define PM_SCAN_CHECK_WPASYNC	(1 << 1)	/* Abort the scan when a non-WP-enabled page is found. */
> +
> +/*
> + * struct pm_scan_arg - Pagemap ioctl argument
> + * @size:		Size of the structure
> + * @flags:		Flags for the IOCTL
> + * @start:		Starting address of the region
> + *			(Ending address of the walk is also returned in it)
> + * @end:		Ending address of the region
> + * @vec:		Address of page_region struct array for output
> + * @vec_len:		Length of the page_region struct array
> + * @max_pages:		Optional limit for number of returned pages (0 = disabled)
> + * @category_inverted:	PAGE_IS_* categories which values match if 0 instead of 1
> + * @category_mask:	Skip pages for which any category doesn't match
> + * @category_anyof_mask: Skip pages for which no category matches
> + * @return_mask:	PAGE_IS_* categories that are to be reported in `page_region`s returned
> + */
> +struct pm_scan_arg {
> +	__u64 size;
> +	__u64 flags;
> +	__u64 start;
> +	__u64 end;
> +	__u64 vec;
> +	__u64 vec_len;
> +	__u64 max_pages;
> +	__u64 category_inverted;
> +	__u64 category_mask;
> +	__u64 category_anyof_mask;
> +	__u64 return_mask;
> +};
> +
>  #endif /* _UAPI_LINUX_FS_H */
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 751968eff92d..4bf8cc4aba2e 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -5008,7 +5008,7 @@ bool is_hugetlb_entry_migration(pte_t pte)
>  		return false;
>  }
>  
> -static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
> +bool is_hugetlb_entry_hwpoisoned(pte_t pte)
>  {
>  	swp_entry_t swp;
>  

-- 
BR,
Muhammad Usama Anjum

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Fri, Jul 21, 2023 at 03:48:22PM +0500, Muhammad Usama Anjum wrote:
> On 7/21/23 12:28 AM, Michał Mirosław wrote:
> > This is a massaged version of patch by Muhammad Usama Anjum [1]
> > to illustrate my review comments and hopefully push the implementation
> > efforts closer to conclusion. The changes are:
> Thank you so much for this effort. I also want to reach conclusion. I'll
> agree with all the changes which don't affect me. But some requirements
> aren't being fulfilled with this current design.
> 
> > 
> > 1. the API:
[...]
> >   b. rename match "flags" to 'page categories' everywhere - this makes
> > 	it easier to differentiate the ioctl()s categorisation of pages
> > 	from struct page flags;
> I've no problem with it.
> 
> #define PAGE_IS_WPASYNC		(1 << 0)
> #define PAGE_IS_WRITTEN		(1 << 1)
> You have another new flag PAGE_IS_WPASYNC. But there is no application of
> PAGE_IS_WPASYNC. We must not add a flag which don't have any user.

Please see below.

> >   c. change {required + excluded} to {inverted + required}. This was
> > 	rejected before, but I'd like to illustrate the difference.
> > 	Old interface can be translated to the new by:
> > 		categories_inverted = excluded_mask
> > 		categories_mask = required_mask | excluded_mask
> > 		categories_anyof_mask = anyof_mask
> > 	The new way allows filtering by: A & (B | !C)
> > 		categories_inverted = C
> > 		categories_mask = A
> > 		categories_anyof_mask = B | C
> I'm still unable to get the idea of inverted masks. IIRC Andei had also not
> supported/accepted this masking scheme. But I'll be okay with it if he
> supports this masking.

Please note that the masks are not inverted -- the values are. Masks
select which categories you want to filter on, and category_inverted
invert the meaning of a match (match 0 instead of 1).

> >   d. change the ioctl to be a SCAN with optional WP. Addressing the
> > 	original use-case, GetWriteWatch() can be implemented as:
> As I've mentioned several times previously (without the name of
> ResetWriteWatch()) that we need exclusive WP without GET. This could be
> implemented with UFFDIO_WRITEPROTECT. But when we use UFFDIO_WRITEPROTECT,
> we hit some special case and performance is very slow. So with UFFD WP
> expert, Peter Xu we have decided to put exclusive WP in this IOCTL for
> implementation of ResetWriteWatch().
> 
> A lot of simplification of the patch is made possible because of not
> keeping exclusive WP. (You have also written some quality code, more better.)
> > 
> > 		memset(&args, 0, sizeof(args));
> > 		args.start = lpBaseAddress;
> > 		args.end = lpBaseAddress + dwRegionSize;
> > 		args.max_pages = *lpdwCount;
> > 		*lpdwGranularity = PAGE_SIZE;
> > 		args.flags = PM_SCAN_CHECK_WPASYNC;
> > 		if (dwFlags & WRITE_WATCH_FLAG_RESET)
> > 			args.flags |= PM_SCAN_WP_MATCHED;
> > 		args.categories_mask = PAGE_IS_WRITTEN;
> > 		args.return_mask = PAGE_IS_WRITTEN;

For ResetWriteWatch() you would:

args.flags = PM_SCAN_WP_MATCHING;
args.categories_mask = PAGE_IS_WPASYNC | PAGE_IS_WRITTEN;
args.return_mask = 0;

Or (if you want to error out if the range doesn't have WP enabled):

args.flags = PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC;
args.categories_mask = PAGE_IS_WRITTEN;
args.return_mask = 0;

(PM_SCAN_CHECK_WPASYNC is effectively adding PAGE_IS_WPASYNC to the
required categories.)

[...]

> > 2. the implementation:
> >   a. gather the page-categorising and write-protecting code in one place;
> Agreed.
> 
> >   b. optimization: add whole-vma skipping for WP usecase;
> I don't know who can benefit from it. Do you have any user in mind? When
> the user come of this optimization, this can be added later.

This is for users of WP that want to ignore WP for non-registered ranges
instead of erroring out on them. (I anticipate CRIU to use this.)

> >   c. extracted output limiting code to pagemap_scan_output();
> If user passes half THP, current code wouldn't split huge page and WP the
> whole THP. We would loose the dirty state of other half huge page. This is
> bug. consoliding the output limiting code looks optimal, but we'll need to
> same limiting code to detect if full THP hasn't been passed in case of THP
> and HugeTLB.

For THP indeed - the code should check `end != start + HPAGE_SIZE`
instead of `ret == -ENOSPC`.

For HugeTLB there is a check that returns EINVAL when trying to WP
a partial page. I think I didn't change that part.

> >   d. extracted range coalescing to pagemap_scan_push_range();
> My old pagemap_scan_output has become pagemap_scan_push_range().

Indeed. I did first push the max_pages check in, hence the 'extracting'
later.

> >   e. extracted THP entry handling to pagemap_scan_thp_entry();
> Good. But I didn't found value in seperating it just like other historic
> pagemap code.

This is to avoid having to much indentation and long functions that do
many things at once.

> >   f. added a shortcut for non-WP hugetlb scan; avoids conditional
> > 	locking;
> Yeah, some if conditions have been removed. But overall did couple of calls
> to is_interesting and scan_output functions instead of just one.

Yes, there are now two pairs instead of one. I see that I haven't pushed
the is_interesting calls into scan_output. This is now trivial:
	if (!interesting...) {
		*end = start;
		return 0;
	}
and could save some typing (but would need a different name for
scan_output as it would do filter & output), but I'm not sure about
readability.

Best Regards
Michał Mirosław

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Fri, Jul 21, 2023 at 03:48:22PM +0500, Muhammad Usama Anjum wrote:
> On 7/21/23 12:28 AM, Michał Mirosław wrote:
> > This is a massaged version of patch by Muhammad Usama Anjum [1]
> > to illustrate my review comments and hopefully push the implementation
> > efforts closer to conclusion. The changes are:
> Thank you so much for this effort. I also want to reach conclusion. I'll
> agree with all the changes which don't affect me. But some requirements
> aren't being fulfilled with this current design.
[...]
> #define PAGE_IS_WPASYNC		(1 << 0)
> #define PAGE_IS_WRITTEN		(1 << 1)
> You have another new flag PAGE_IS_WPASYNC. But there is no application of
> PAGE_IS_WPASYNC. We must not add a flag which don't have any user.

BTW, I'm not sure I got the PAGE_IS_WPASYNC naming right - this is to
designate pages that can be write-protected (are UFFD-registered I believe).

Best Regards
Michał Mirosław

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Andrei Vagin]

On Thu, Jul 20, 2023 at 9:27 PM Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> Thank you Michał.
>
> On 7/21/23 12:28 AM, Michał Mirosław wrote:
> >   b. rename match "flags" to 'page categories' everywhere - this makes
> >       it easier to differentiate the ioctl()s categorisation of pages
> >       from struct page flags;
> >   c. change {required + excluded} to {inverted + required}. This was
> >       rejected before, but I'd like to illustrate the difference.
> >       Old interface can be translated to the new by:
> >               categories_inverted = excluded_mask
> >               categories_mask = required_mask | excluded_mask
> >               categories_anyof_mask = anyof_mask
> >       The new way allows filtering by: A & (B | !C)
> >               categories_inverted = C
> >               categories_mask = A
> >               categories_anyof_mask = B | C
> Andrei and Danylo,
>
> Are you okay with these masks? It were you two who had proposed these.

I am okay.

>
>
> --
> BR,
> Muhammad Usama Anjum

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/21/23 4:23 PM, Michał Mirosław wrote:
> On Fri, Jul 21, 2023 at 03:48:22PM +0500, Muhammad Usama Anjum wrote:
>> On 7/21/23 12:28 AM, Michał Mirosław wrote:
>>> This is a massaged version of patch by Muhammad Usama Anjum [1]
>>> to illustrate my review comments and hopefully push the implementation
>>> efforts closer to conclusion. The changes are:
>> Thank you so much for this effort. I also want to reach conclusion. I'll
>> agree with all the changes which don't affect me. But some requirements
>> aren't being fulfilled with this current design.
>>
>>>
>>> 1. the API:
> [...]
>>>   b. rename match "flags" to 'page categories' everywhere - this makes
>>> 	it easier to differentiate the ioctl()s categorisation of pages
>>> 	from struct page flags;
>> I've no problem with it.
>>
>> #define PAGE_IS_WPASYNC		(1 << 0)
>> #define PAGE_IS_WRITTEN		(1 << 1)
>> You have another new flag PAGE_IS_WPASYNC. But there is no application of
>> PAGE_IS_WPASYNC. We must not add a flag which don't have any user.
> 
> Please see below.
> 
>>>   c. change {required + excluded} to {inverted + required}. This was
>>> 	rejected before, but I'd like to illustrate the difference.
>>> 	Old interface can be translated to the new by:
>>> 		categories_inverted = excluded_mask
>>> 		categories_mask = required_mask | excluded_mask
>>> 		categories_anyof_mask = anyof_mask
>>> 	The new way allows filtering by: A & (B | !C)
>>> 		categories_inverted = C
>>> 		categories_mask = A
>>> 		categories_anyof_mask = B | C
>> I'm still unable to get the idea of inverted masks. IIRC Andei had also not
>> supported/accepted this masking scheme. But I'll be okay with it if he
>> supports this masking.
> 
> Please note that the masks are not inverted -- the values are. Masks
> select which categories you want to filter on, and category_inverted
> invert the meaning of a match (match 0 instead of 1).
> 
>>>   d. change the ioctl to be a SCAN with optional WP. Addressing the
>>> 	original use-case, GetWriteWatch() can be implemented as:
>> As I've mentioned several times previously (without the name of
>> ResetWriteWatch()) that we need exclusive WP without GET. This could be
>> implemented with UFFDIO_WRITEPROTECT. But when we use UFFDIO_WRITEPROTECT,
>> we hit some special case and performance is very slow. So with UFFD WP
>> expert, Peter Xu we have decided to put exclusive WP in this IOCTL for
>> implementation of ResetWriteWatch().
>>
>> A lot of simplification of the patch is made possible because of not
>> keeping exclusive WP. (You have also written some quality code, more better.)
>>>
>>> 		memset(&args, 0, sizeof(args));
>>> 		args.start = lpBaseAddress;
>>> 		args.end = lpBaseAddress + dwRegionSize;
>>> 		args.max_pages = *lpdwCount;
>>> 		*lpdwGranularity = PAGE_SIZE;
>>> 		args.flags = PM_SCAN_CHECK_WPASYNC;
>>> 		if (dwFlags & WRITE_WATCH_FLAG_RESET)
>>> 			args.flags |= PM_SCAN_WP_MATCHED;
>>> 		args.categories_mask = PAGE_IS_WRITTEN;
>>> 		args.return_mask = PAGE_IS_WRITTEN;
> 
> For ResetWriteWatch() you would:
> 
> args.flags = PM_SCAN_WP_MATCHING;
> args.categories_mask = PAGE_IS_WPASYNC | PAGE_IS_WRITTEN;
> args.return_mask = 0;
> 
> Or (if you want to error out if the range doesn't have WP enabled):
> 
> args.flags = PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC;
> args.categories_mask = PAGE_IS_WRITTEN;
> args.return_mask = 0;
> 
> (PM_SCAN_CHECK_WPASYNC is effectively adding PAGE_IS_WPASYNC to the
> required categories.)
Right. But we don't want to perform GET in case of ResetWriteWatch(). It'll
be wasted effort to perform GET as well when we don't care about the dirty
status of the pages.


> 
> [...]
> 
>>> 2. the implementation:
>>>   a. gather the page-categorising and write-protecting code in one place;
>> Agreed.
>>
>>>   b. optimization: add whole-vma skipping for WP usecase;
>> I don't know who can benefit from it. Do you have any user in mind? When
>> the user come of this optimization, this can be added later.
> 
> This is for users of WP that want to ignore WP for non-registered ranges
> instead of erroring out on them. (I anticipate CRIU to use this.)
> 
>>>   c. extracted output limiting code to pagemap_scan_output();
>> If user passes half THP, current code wouldn't split huge page and WP the
>> whole THP. We would loose the dirty state of other half huge page. This is
>> bug. consoliding the output limiting code looks optimal, but we'll need to
>> same limiting code to detect if full THP hasn't been passed in case of THP
>> and HugeTLB.
> 
> For THP indeed - the code should check `end != start + HPAGE_SIZE`
> instead of `ret == -ENOSPC`.
Yeah, this need to be fixed.

> 
> For HugeTLB there is a check that returns EINVAL when trying to WP
> a partial page. I think I didn't change that part.
> 
>>>   d. extracted range coalescing to pagemap_scan_push_range();
>> My old pagemap_scan_output has become pagemap_scan_push_range().
> 
> Indeed. I did first push the max_pages check in, hence the 'extracting'
> later.
> 
>>>   e. extracted THP entry handling to pagemap_scan_thp_entry();
>> Good. But I didn't found value in seperating it just like other historic
>> pagemap code.
> 
> This is to avoid having to much indentation and long functions that do
> many things at once.
> 
>>>   f. added a shortcut for non-WP hugetlb scan; avoids conditional
>>> 	locking;
>> Yeah, some if conditions have been removed. But overall did couple of calls
>> to is_interesting and scan_output functions instead of just one.
> 
> Yes, there are now two pairs instead of one. I see that I haven't pushed
> the is_interesting calls into scan_output. This is now trivial:
> 	if (!interesting...) {
> 		*end = start;
> 		return 0;
> 	}
This can be the 3rd thing to fix.

Is it possible for you to fix the above mentioned 3 things and send the
patch for my testing:
1 Make GET optional
2 Detect partial THP WP operation and split
3 Optimization of moving this interesting logic to output() function

Please let me know if you cannot make the above fixes. I'll mix my patch
version and your patch and fix these things up.

> and could save some typing (but would need a different name for
> scan_output as it would do filter & output), but I'm not sure about
> readability.
> 
> Best Regards
> Michał Mirosław

-- 
BR,
Muhammad Usama Anjum

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/21/23 4:29 PM, Michał Mirosław wrote:
> On Fri, Jul 21, 2023 at 03:48:22PM +0500, Muhammad Usama Anjum wrote:
>> On 7/21/23 12:28 AM, Michał Mirosław wrote:
>>> This is a massaged version of patch by Muhammad Usama Anjum [1]
>>> to illustrate my review comments and hopefully push the implementation
>>> efforts closer to conclusion. The changes are:
>> Thank you so much for this effort. I also want to reach conclusion. I'll
>> agree with all the changes which don't affect me. But some requirements
>> aren't being fulfilled with this current design.
> [...]
>> #define PAGE_IS_WPASYNC		(1 << 0)
>> #define PAGE_IS_WRITTEN		(1 << 1)
>> You have another new flag PAGE_IS_WPASYNC. But there is no application of
>> PAGE_IS_WPASYNC. We must not add a flag which don't have any user.
> 
> BTW, I'm not sure I got the PAGE_IS_WPASYNC naming right - this is to
> designate pages that can be write-protected (are UFFD-registered I believe).
PAGE_IS_UFFDWP_REGISTERED or PAGE_IS_WP_REG?

> 
> Best Regards
> Michał Mirosław

-- 
BR,
Muhammad Usama Anjum

Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Fri, Jul 21, 2023 at 10:50:19PM +0500, Muhammad Usama Anjum wrote:
> On 7/21/23 4:23 PM, Michał Mirosław wrote:
> > On Fri, Jul 21, 2023 at 03:48:22PM +0500, Muhammad Usama Anjum wrote:
> >> On 7/21/23 12:28 AM, Michał Mirosław wrote:
[...]
> >>>   d. change the ioctl to be a SCAN with optional WP. Addressing the
> >>> 	original use-case, GetWriteWatch() can be implemented as:
> >> As I've mentioned several times previously (without the name of
> >> ResetWriteWatch()) that we need exclusive WP without GET. This could be
> >> implemented with UFFDIO_WRITEPROTECT. But when we use UFFDIO_WRITEPROTECT,
> >> we hit some special case and performance is very slow. So with UFFD WP
> >> expert, Peter Xu we have decided to put exclusive WP in this IOCTL for
> >> implementation of ResetWriteWatch().
> >>
> >> A lot of simplification of the patch is made possible because of not
> >> keeping exclusive WP. (You have also written some quality code, more better.)
> >>>
> >>> 		memset(&args, 0, sizeof(args));
> >>> 		args.start = lpBaseAddress;
> >>> 		args.end = lpBaseAddress + dwRegionSize;
> >>> 		args.max_pages = *lpdwCount;
> >>> 		*lpdwGranularity = PAGE_SIZE;
> >>> 		args.flags = PM_SCAN_CHECK_WPASYNC;
> >>> 		if (dwFlags & WRITE_WATCH_FLAG_RESET)
> >>> 			args.flags |= PM_SCAN_WP_MATCHED;
> >>> 		args.categories_mask = PAGE_IS_WRITTEN;
> >>> 		args.return_mask = PAGE_IS_WRITTEN;
> > 
> > For ResetWriteWatch() you would:
> > 
> > args.flags = PM_SCAN_WP_MATCHING;
> > args.categories_mask = PAGE_IS_WPASYNC | PAGE_IS_WRITTEN;
> > args.return_mask = 0;
> > 
> > Or (if you want to error out if the range doesn't have WP enabled):
> > 
> > args.flags = PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC;
> > args.categories_mask = PAGE_IS_WRITTEN;
> > args.return_mask = 0;
> > 
> > (PM_SCAN_CHECK_WPASYNC is effectively adding PAGE_IS_WPASYNC to the
> > required categories.)
> Right. But we don't want to perform GET in case of ResetWriteWatch(). It'll
> be wasted effort to perform GET as well when we don't care about the dirty
> status of the pages.

This doesn't really do GET: return_mask == 0 means that there won't be any
ranges reported (and you can pass {NULL, 0} for arg.{vec, vec_len}). But
I've changed the no-GET criteria to vec == NULL (requires vec_len == 0).

> Is it possible for you to fix the above mentioned 3 things and send the
> patch for my testing:
> 1 Make GET optional
> 2 Detect partial THP WP operation and split
> 3 Optimization of moving this interesting logic to output() function
> 
> Please let me know if you cannot make the above fixes. I'll mix my patch
> version and your patch and fix these things up.

Sending as a reply.

Best Regards
Michał Mirosław

[v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>

Changes from v1 [1]:

API:
  1. Change no-GET operation condition from `arg.return_mask == 0` to
     `arg.vec == NULL`. This will allow issuing the ioctl with
     return_mask == 0 to gather matching ranges when the exact category
     is not interesting. (Anticipated for CRIU scanning a large sparse
     anonymous mapping).

Implementation:
  1. Fixed splitting THPs if processing a non-full range.
  2. Pushed ...is_interesting_page() call to ...output().

[1] https://lore.kernel.org/linux-mm/202307211507.xOl45LiR-lkp@intel.com/T/#mc8ad9ca249a8ec01980791539abcb8a221154e20

Best Regards
Michał Mirosław

Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
 fs/proc/task_mmu.c      | 641 ++++++++++++++++++++++++++++++++++++++++
 include/linux/hugetlb.h |   1 +
 include/uapi/linux/fs.h |  56 ++++
 mm/hugetlb.c            |   2 +-
 4 files changed, 699 insertions(+), 1 deletion(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index c1e6531cb02a..568e76f64824 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -19,6 +19,8 @@
 #include <linux/shmem_fs.h>
 #include <linux/uaccess.h>
 #include <linux/pkeys.h>
+#include <linux/minmax.h>
+#include <linux/overflow.h>
 
 #include <asm/elf.h>
 #include <asm/tlb.h>
@@ -1749,11 +1751,650 @@ static int pagemap_release(struct inode *inode, struct file *file)
 	return 0;
 }
 
+#define PM_SCAN_CATEGORIES	(PAGE_IS_WPASYNC | PAGE_IS_WRITTEN | \
+				 PAGE_IS_FILE |	PAGE_IS_PRESENT | \
+				 PAGE_IS_SWAPPED | PAGE_IS_PFNZERO)
+#define PM_SCAN_FLAGS		(PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
+
+struct pagemap_scan_private {
+	struct pm_scan_arg arg;
+	unsigned long cur_vma_category;
+	struct page_region *vec_buf, cur_buf;
+	unsigned long vec_buf_len, vec_buf_index, found_pages, end_addr;
+	struct page_region __user* vec_out;
+};
+
+static unsigned long pagemap_page_category(struct vm_area_struct *vma, unsigned long addr, pte_t pte)
+{
+	unsigned long categories = 0;
+
+	if (pte_present(pte)) {
+		struct page *page = vm_normal_page(vma, addr, pte);
+
+		categories |= PAGE_IS_PRESENT;
+		if (!pte_uffd_wp(pte))
+			categories |= PAGE_IS_WRITTEN;
+		if (page && !PageAnon(page))
+			categories |= PAGE_IS_FILE;
+		if (is_zero_pfn(pte_pfn(pte)))
+			categories |= PAGE_IS_PFNZERO;
+	} else if (is_swap_pte(pte)) {
+		swp_entry_t swp = pte_to_swp_entry(pte);
+
+		categories |= PAGE_IS_SWAPPED;
+		if (!pte_swp_uffd_wp_any(pte))
+			categories |= PAGE_IS_WRITTEN;
+		if (is_pfn_swap_entry(swp) && !PageAnon(pfn_swap_entry_to_page(swp)))
+			categories |= PAGE_IS_FILE;
+	}
+
+	return categories;
+}
+
+static void make_uffd_wp_pte(struct vm_area_struct *vma,
+			     unsigned long addr, pte_t *pte)
+{
+	pte_t ptent = ptep_get(pte);
+
+	if (pte_present(ptent)) {
+		pte_t old_pte;
+
+		old_pte = ptep_modify_prot_start(vma, addr, pte);
+		ptent = pte_mkuffd_wp(ptent);
+		ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
+	} else if (is_swap_pte(ptent)) {
+		ptent = pte_swp_mkuffd_wp(ptent);
+		set_pte_at(vma->vm_mm, addr, pte, ptent);
+	} else {
+		set_pte_at(vma->vm_mm, addr, pte,
+			   make_pte_marker(PTE_MARKER_UFFD_WP));
+	}
+}
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static unsigned long pagemap_thp_category(pmd_t pmd)
+{
+	unsigned long categories = 0;
+
+	if (pmd_present(pmd)) {
+		categories |= PAGE_IS_PRESENT;
+		if (!pmd_uffd_wp(pmd))
+			categories |= PAGE_IS_WRITTEN;
+		if (is_zero_pfn(pmd_pfn(pmd)))
+			categories |= PAGE_IS_PFNZERO;
+	} else if (is_swap_pmd(pmd)) {
+		categories |= PAGE_IS_SWAPPED;
+		if (!pmd_swp_uffd_wp(pmd))
+			categories |= PAGE_IS_WRITTEN;
+	}
+
+	return categories;
+}
+
+static void make_uffd_wp_pmd(struct vm_area_struct *vma,
+			     unsigned long addr, pmd_t *pmdp)
+{
+	pmd_t old, pmd = *pmdp;
+
+	if (pmd_present(pmd)) {
+		old = pmdp_invalidate_ad(vma, addr, pmdp);
+		pmd = pmd_mkuffd_wp(old);
+		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+	} else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
+		pmd = pmd_swp_mkuffd_wp(pmd);
+		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+	}
+}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+
+#ifdef CONFIG_HUGETLB_PAGE
+static unsigned long pagemap_hugetlb_category(pte_t pte)
+{
+	unsigned long categories = 0;
+
+	if (pte_present(pte)) {
+		categories |= PAGE_IS_PRESENT;
+		if (!huge_pte_uffd_wp(pte))
+			categories |= PAGE_IS_WRITTEN;
+		if (!PageAnon(pte_page(pte)))
+			categories |= PAGE_IS_FILE;
+		if (is_zero_pfn(pte_pfn(pte)))
+			categories |= PAGE_IS_PFNZERO;
+	} else if (is_swap_pte(pte)) {
+		categories |= PAGE_IS_SWAPPED;
+		if (!pte_swp_uffd_wp_any(pte))
+			categories |= PAGE_IS_WRITTEN;
+	}
+
+	return categories;
+}
+
+static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
+				  unsigned long addr, pte_t *ptep,
+				  pte_t ptent)
+{
+	if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
+		return;
+
+	if (is_hugetlb_entry_migration(ptent))
+		set_huge_pte_at(vma->vm_mm, addr, ptep,
+				pte_swp_mkuffd_wp(ptent));
+	else if (!huge_pte_none(ptent))
+		huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
+					     huge_pte_mkuffd_wp(ptent));
+	else
+		set_huge_pte_at(vma->vm_mm, addr, ptep,
+				make_pte_marker(PTE_MARKER_UFFD_WP));
+}
+#endif /* CONFIG_HUGETLB_PAGE */
+
+static bool pagemap_scan_is_interesting_page(unsigned long categories,
+					     const struct pagemap_scan_private *p)
+{
+	categories ^= p->arg.category_inverted;
+	if ((categories & p->arg.category_mask) != p->arg.category_mask)
+		return false;
+	if (p->arg.category_anyof_mask && !(categories & p->arg.category_anyof_mask))
+		return false;
+
+	return true;
+}
+
+static bool pagemap_scan_is_interesting_vma(unsigned long categories,
+					    const struct pagemap_scan_private *p)
+{
+	unsigned long required = p->arg.category_mask & PAGE_IS_WPASYNC;
+
+	categories ^= p->arg.category_inverted;
+	if ((categories & required) != required)
+		return false;
+	return true;
+}
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
+				  struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	unsigned long vma_category = 0;
+
+	if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
+		vma_category |= PAGE_IS_WPASYNC;
+	else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
+		return -EPERM;
+
+	if (vma->vm_flags & VM_PFNMAP)
+		return 1;
+
+	if (!pagemap_scan_is_interesting_vma(vma_category, p))
+		return 1;
+
+	p->cur_vma_category = vma_category;
+	return 0;
+}
+
+static bool pagemap_scan_push_range(unsigned long categories,
+				    struct pagemap_scan_private *p,
+				    unsigned long addr, unsigned long end)
+{
+	struct page_region *cur_buf = &p->cur_buf;
+
+	/*
+	 * When there is no output buffer provided at all, the sentinel values
+	 * won't match here. There is no other way for `cur_buf->end` to be
+	 * non-zero other than it being non-empty.
+	 */
+	if (addr == cur_buf->end && categories == cur_buf->categories) {
+		cur_buf->end = end;
+		return true;
+	}
+
+	if (cur_buf->end) {
+		if (p->vec_buf_index >= p->vec_buf_len)
+			return false;
+
+		memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
+		       sizeof(*p->vec_buf));
+		++p->vec_buf_index;
+	}
+
+	cur_buf->start = addr;
+	cur_buf->end = end;
+	cur_buf->categories = categories;
+	return true;
+}
+
+static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
+				       unsigned long addr, unsigned long end)
+{
+	struct page_region *cur_buf = &p->cur_buf;
+
+	if (cur_buf->start != addr)
+		cur_buf->end = addr;
+	else
+		cur_buf->start = cur_buf->end = 0;
+
+	p->end_addr = 0;
+	p->found_pages -= (end - addr) / PAGE_SIZE;
+}
+
+static int pagemap_scan_output(unsigned long categories,
+			       struct pagemap_scan_private *p,
+			       unsigned long addr, unsigned long *end)
+{
+	unsigned long n_pages, total_pages;
+	int ret = 0;
+
+	if (!pagemap_scan_is_interesting_page(categories, p)) {
+		*end = addr;
+		return 0;
+	}
+
+	if (!p->vec_buf)
+		return 0;
+
+	categories &= p->arg.return_mask;
+
+	n_pages = (*end - addr) / PAGE_SIZE;
+	if (check_add_overflow(p->found_pages, n_pages, &total_pages) || total_pages > p->arg.max_pages) {
+		size_t n_too_much = total_pages - p->arg.max_pages;
+		*end -= n_too_much * PAGE_SIZE;
+		n_pages -= n_too_much;
+		ret = -ENOSPC;
+	}
+
+	if (!pagemap_scan_push_range(categories, p, addr, *end)) {
+		*end = addr;
+		n_pages = 0;
+		ret = -ENOSPC;
+	}
+
+	p->found_pages += n_pages;
+	if (ret)
+		p->end_addr = *end;
+	return ret;
+}
+
+static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
+				  unsigned long end, struct mm_walk *walk)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	unsigned long categories;
+	spinlock_t *ptl;
+	int ret = 0;
+
+	ptl = pmd_trans_huge_lock(pmd, vma);
+	if (!ptl)
+		return -ENOENT;
+
+	categories = p->cur_vma_category | pagemap_thp_category(*pmd);
+
+	ret = pagemap_scan_output(categories, p, start, &end);
+	if (start == end)
+		goto out_unlock;
+
+	if (~p->arg.flags & PM_SCAN_WP_MATCHING)
+		goto out_unlock;
+	if (~categories & PAGE_IS_WRITTEN)
+		goto out_unlock;
+
+	/*
+	 * Break huge page into small pages if the WP operation
+	 * need to be performed is on a portion of the huge page.
+	 */
+	if (end != addr + HPAGE_SIZE) {
+		spin_unlock(ptl);
+		split_huge_pmd(vma, pmd, start);
+		pagemap_scan_backout_range(p, start, end);
+		return -ENOENT;
+	}
+
+	make_uffd_wp_pmd(vma, start, pmd);
+	flush_tlb_range(vma, start, end);
+out_unlock:
+	spin_unlock(ptl);
+	return ret;
+#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
+	return -ENOENT;
+#endif
+}
+
+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+				  unsigned long end, struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	pte_t *pte, *start_pte;
+	unsigned long addr;
+	bool flush = false;
+	spinlock_t *ptl;
+	int ret;
+
+	arch_enter_lazy_mmu_mode();
+
+	ret = pagemap_scan_thp_entry(pmd, start, end, walk);
+	if (ret != -ENOENT) {
+		arch_leave_lazy_mmu_mode();
+		return ret;
+	}
+
+	start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+	if (!pte) {
+		arch_leave_lazy_mmu_mode();
+		walk->action = ACTION_AGAIN;
+		return 0;
+	}
+
+	for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
+		unsigned long categories = p->cur_vma_category |
+			pagemap_page_category(vma, addr, ptep_get(pte));
+		unsigned long next = addr + PAGE_SIZE;
+
+		ret = pagemap_scan_output(categories, p, addr, &next);
+		if (next == addr)
+			break;
+
+		if (~p->arg.flags & PM_SCAN_WP_MATCHING)
+			continue;
+		if (~categories & PAGE_IS_WRITTEN)
+			continue;
+
+		make_uffd_wp_pte(vma, addr, pte);
+		if (!flush) {
+			start = addr;
+			flush = true;
+		}
+	}
+
+	if (flush)
+		flush_tlb_range(vma, start, addr);
+
+	pte_unmap_unlock(start_pte, ptl);
+	arch_leave_lazy_mmu_mode();
+
+	cond_resched();
+	return ret;
+}
+
+#ifdef CONFIG_HUGETLB_PAGE
+static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
+				      unsigned long start, unsigned long end,
+				      struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	unsigned long categories;
+	spinlock_t *ptl;
+	int ret = 0;
+	pte_t pte;
+
+	if (~p->arg.flags & PM_SCAN_WP_MATCHING) {
+		/* Go the short route when not write-protecting pages. */
+
+		pte = huge_ptep_get(ptep);
+		categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
+
+		return pagemap_scan_output(categories, p, start, &end);
+	}
+
+	if (end != start + HPAGE_SIZE) {
+		/* Partial HugeTLB page WP isn't possible. */
+		p->end_addr = start;
+		return -EINVAL;
+	}
+
+	i_mmap_lock_write(vma->vm_file->f_mapping);
+	ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
+
+	pte = huge_ptep_get(ptep);
+	categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
+
+	ret = pagemap_scan_output(categories, p, start, &end);
+	if (start == end)
+		goto out_unlock;
+
+	if (categories & PAGE_IS_WRITTEN) {
+		make_uffd_wp_huge_pte(vma, start, ptep, pte);
+		flush_hugetlb_tlb_range(vma, start, end);
+	}
+
+out_unlock:
+	spin_unlock(ptl);
+	i_mmap_unlock_write(vma->vm_file->f_mapping);
+
+	return ret;
+}
+#else
+#define pagemap_scan_hugetlb_entry NULL
+#endif
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
+				 int depth, struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	int ret;
+
+	if (!vma)
+		return 0;
+
+	ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end);
+	if (addr == end)
+		return ret;
+
+	if (~p->arg.flags & PM_SCAN_WP_MATCHING)
+		return ret;
+
+	int err = uffd_wp_range(vma, addr, end - addr, true);
+	if (err < 0)
+		ret = err;
+
+	return ret;
+}
+
+static const struct mm_walk_ops pagemap_scan_ops = {
+	.test_walk = pagemap_scan_test_walk,
+	.pmd_entry = pagemap_scan_pmd_entry,
+	.pte_hole = pagemap_scan_pte_hole,
+	.hugetlb_entry = pagemap_scan_hugetlb_entry,
+};
+
+static int pagemap_scan_get_args(struct pm_scan_arg *arg,
+				 unsigned long uarg)
+{
+	unsigned long start, end, vec;
+
+	if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
+		return -EFAULT;
+
+	if (arg->size != sizeof(struct pm_scan_arg))
+		return -EINVAL;
+
+	/* Validate requested features */
+	if (arg->flags & ~PM_SCAN_FLAGS)
+		return -EINVAL;
+	if ((arg->category_inverted | arg->category_mask |
+	     arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
+		return -EINVAL;
+
+	start = untagged_addr((unsigned long)arg->start);
+	end = untagged_addr((unsigned long)arg->end);
+	vec = untagged_addr((unsigned long)arg->vec);
+
+	/* Validate memory pointers */
+	if (!IS_ALIGNED(start, PAGE_SIZE))
+		return -EINVAL;
+	if (!access_ok((void __user *)start, end - start))
+		return -EFAULT;
+	if (!vec && arg->vec_len)
+		return -EFAULT;
+	if (vec && !access_ok((void __user *)vec,
+			      arg->vec_len * sizeof(struct page_region)))
+		return -EFAULT;
+
+	/* Fixup default values */
+	if (!arg->max_pages)
+		arg->max_pages = ULONG_MAX;
+
+	return 0;
+}
+
+static int pagemap_scan_writeback_args(struct pm_scan_arg *arg,
+				       unsigned long uargl)
+{
+	struct pm_scan_arg __user *uarg	= (void __user *)uargl;
+
+	if (copy_to_user(&uarg->start, &arg->start, sizeof(arg->start)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p)
+{
+	if (!p->arg.vec_len) {
+		/*
+		 * An arbitrary non-page-aligned sentinel value for
+		 * pagemap_scan_push_range().
+		 */
+		p->cur_buf.start = p->cur_buf.end = ULLONG_MAX;
+		if (p->arg.vec)
+			p->vec_buf = ZERO_SIZE_PTR;
+		return 0;
+	}
+
+	/*
+	 * Allocate a smaller buffer to get output from inside the page
+	 * walk functions and walk the range in PAGEMAP_WALK_SIZE chunks.
+	 * The last range is always stored in p.cur_buf to allow coalescing
+	 * consecutive ranges that have the same categories returned across
+	 * walk_page_range() calls.
+	 */
+	p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT,
+			       p->arg.vec_len - 1);
+	p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf),
+				   GFP_KERNEL);
+	if (!p->vec_buf)
+		return -ENOMEM;
+
+	p->vec_out = (void __user *)p->arg.vec;
+
+	return 0;
+}
+
+static int pagemap_scan_flush_buffer(struct pagemap_scan_private *p)
+{
+	const struct page_region *buf = p->vec_buf;
+	int n = (int)p->vec_buf_index;
+
+	if (!n)
+		return 0;
+
+	if (copy_to_user(p->vec_out, buf, n * sizeof(*buf)))
+		return -EFAULT;
+
+	p->arg.vec_len -= n;
+	p->vec_out += n;
+
+	p->vec_buf_index = 0;
+	p->vec_buf_len = min_t(size_t, p->vec_buf_len, p->arg.vec_len - 1);
+
+	return n;
+}
+
+static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg)
+{
+	unsigned long walk_start, walk_end;
+	struct mmu_notifier_range range;
+	struct pagemap_scan_private p;
+	size_t n_ranges_out = 0;
+	int ret;
+
+	memset(&p, 0, sizeof(p));
+	ret = pagemap_scan_get_args(&p.arg, uarg);
+	if (ret)
+		return ret;
+
+	ret = pagemap_scan_init_bounce_buffer(&p);
+	if (ret)
+		return ret;
+
+	/* Protection change for the range is going to happen. */
+	if (p.arg.flags & PM_SCAN_WP_MATCHING) {
+		mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
+					mm, p.arg.start, p.arg.end);
+		mmu_notifier_invalidate_range_start(&range);
+	}
+
+	walk_start = walk_end = p.arg.start;
+	for (; walk_end != p.arg.end; walk_start = walk_end) {
+		int n_out;
+		walk_end = min_t(unsigned long,
+				 (walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK,
+				 p.arg.end);
+
+		ret = mmap_read_lock_killable(mm);
+		if (ret)
+			break;
+		ret = walk_page_range(mm, walk_start, walk_end,
+				      &pagemap_scan_ops, &p);
+		mmap_read_unlock(mm);
+
+		n_out = pagemap_scan_flush_buffer(&p);
+		if (n_out < 0)
+			ret = n_out;
+		else
+			n_ranges_out += n_out;
+
+		if (ret)
+			break;
+	}
+
+	if (p.cur_buf.start != p.cur_buf.end) {
+		if (copy_to_user(p.vec_out, &p.cur_buf, sizeof(p.cur_buf)))
+			ret = -EFAULT;
+		else
+			++n_ranges_out;
+	}
+
+	/* ENOSPC signifies early stop (buffer full) from the walk. */
+	if (!ret || ret == -ENOSPC)
+		ret = n_ranges_out;
+
+	p.arg.start = p.end_addr ? p.end_addr : walk_start;
+	if (pagemap_scan_writeback_args(&p.arg, uarg))
+		ret = -EFAULT;
+
+	if (p.arg.flags & PM_SCAN_WP_MATCHING)
+		mmu_notifier_invalidate_range_end(&range);
+
+	kfree(p.vec_buf);
+	return ret;
+}
+
+static long do_pagemap_cmd(struct file *file, unsigned int cmd,
+			   unsigned long arg)
+{
+	struct mm_struct *mm = file->private_data;
+
+	switch (cmd) {
+	case PAGEMAP_SCAN:
+		return do_pagemap_scan(mm, arg);
+
+	default:
+		return -EINVAL;
+	}
+}
+
 const struct file_operations proc_pagemap_operations = {
 	.llseek		= mem_lseek, /* borrow this */
 	.read		= pagemap_read,
 	.open		= pagemap_open,
 	.release	= pagemap_release,
+	.unlocked_ioctl = do_pagemap_cmd,
+	.compat_ioctl	= do_pagemap_cmd,
 };
 #endif /* CONFIG_PROC_PAGE_MONITOR */
 
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 9f4bac3df59e..4f5fed605538 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 		unsigned long cp_flags);
 
 bool is_hugetlb_entry_migration(pte_t pte);
+bool is_hugetlb_entry_hwpoisoned(pte_t pte);
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
 
 #else /* !CONFIG_HUGETLB_PAGE */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index b7b56871029c..db39442befcb 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -305,4 +305,60 @@ typedef int __bitwise __kernel_rwf_t;
 #define RWF_SUPPORTED	(RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
 			 RWF_APPEND)
 
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN	_IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WPASYNC		(1 << 0)
+#define PAGE_IS_WRITTEN		(1 << 1)
+#define PAGE_IS_FILE		(1 << 2)
+#define PAGE_IS_PRESENT		(1 << 3)
+#define PAGE_IS_SWAPPED		(1 << 4)
+#define PAGE_IS_PFNZERO		(1 << 5)
+
+/*
+ * struct page_region - Page region with flags
+ * @start:	Start of the region
+ * @end:	End of the region (exclusive)
+ * @categories:	PAGE_IS_* category bitmask for the region
+ */
+struct page_region {
+	__u64 start;
+	__u64 end;
+	__u64 categories;
+};
+
+/* Flags for PAGEMAP_SCAN ioctl */
+#define PM_SCAN_WP_MATCHING	(1 << 0)	/* Write protect the pages matched. */
+#define PM_SCAN_CHECK_WPASYNC	(1 << 1)	/* Abort the scan when a non-WP-enabled page is found. */
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size:		Size of the structure
+ * @flags:		Flags for the IOCTL
+ * @start:		Starting address of the region
+ *			(Ending address of the walk is also returned in it)
+ * @end:		Ending address of the region
+ * @vec:		Address of page_region struct array for output
+ * @vec_len:		Length of the page_region struct array
+ * @max_pages:		Optional limit for number of returned pages (0 = disabled)
+ * @category_inverted:	PAGE_IS_* categories which values match if 0 instead of 1
+ * @category_mask:	Skip pages for which any category doesn't match
+ * @category_anyof_mask: Skip pages for which no category matches
+ * @return_mask:	PAGE_IS_* categories that are to be reported in `page_region`s returned
+ */
+struct pm_scan_arg {
+	__u64 size;
+	__u64 flags;
+	__u64 start;
+	__u64 end;
+	__u64 vec;
+	__u64 vec_len;
+	__u64 max_pages;
+	__u64 category_inverted;
+	__u64 category_mask;
+	__u64 category_anyof_mask;
+	__u64 return_mask;
+};
+
 #endif /* _UAPI_LINUX_FS_H */
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 751968eff92d..4bf8cc4aba2e 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5008,7 +5008,7 @@ bool is_hugetlb_entry_migration(pte_t pte)
 		return false;
 }
 
-static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
+bool is_hugetlb_entry_hwpoisoned(pte_t pte)
 {
 	swp_entry_t swp;
 
-- 
2.39.2

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[kernel test robot]

Hi Michał,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.5-rc2 next-20230721]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Micha-Miros-aw/fs-proc-task_mmu-Implement-IOCTL-for-efficient-page-table-scanning/20230722-082500
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/ZLshsAj5PbsEAHhP%40qmqm.qmqm.pl
patch subject: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning
config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20230722/202307222107.GdtIG4nq-lkp@intel.com/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce: (https://download.01.org/0day-ci/archive/20230722/202307222107.GdtIG4nq-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307222107.GdtIG4nq-lkp@intel.com/

All errors (new ones prefixed by >>):

   fs/proc/task_mmu.c:1921:6: error: call to undeclared function 'userfaultfd_wp_async'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
           if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
               ^
>> fs/proc/task_mmu.c:2047:13: error: use of undeclared identifier 'addr'
           if (end != addr + HPAGE_SIZE) {
                      ^
   2 errors generated.


vim +/addr +2047 fs/proc/task_mmu.c

  1913	
  1914	static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
  1915					  struct mm_walk *walk)
  1916	{
  1917		struct pagemap_scan_private *p = walk->private;
  1918		struct vm_area_struct *vma = walk->vma;
  1919		unsigned long vma_category = 0;
  1920	
> 1921		if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
  1922			vma_category |= PAGE_IS_WPASYNC;
  1923		else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
  1924			return -EPERM;
  1925	
  1926		if (vma->vm_flags & VM_PFNMAP)
  1927			return 1;
  1928	
  1929		if (!pagemap_scan_is_interesting_vma(vma_category, p))
  1930			return 1;
  1931	
  1932		p->cur_vma_category = vma_category;
  1933		return 0;
  1934	}
  1935	
  1936	static bool pagemap_scan_push_range(unsigned long categories,
  1937					    struct pagemap_scan_private *p,
  1938					    unsigned long addr, unsigned long end)
  1939	{
  1940		struct page_region *cur_buf = &p->cur_buf;
  1941	
  1942		/*
  1943		 * When there is no output buffer provided at all, the sentinel values
  1944		 * won't match here. There is no other way for `cur_buf->end` to be
  1945		 * non-zero other than it being non-empty.
  1946		 */
  1947		if (addr == cur_buf->end && categories == cur_buf->categories) {
  1948			cur_buf->end = end;
  1949			return true;
  1950		}
  1951	
  1952		if (cur_buf->end) {
  1953			if (p->vec_buf_index >= p->vec_buf_len)
  1954				return false;
  1955	
  1956			memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
  1957			       sizeof(*p->vec_buf));
  1958			++p->vec_buf_index;
  1959		}
  1960	
  1961		cur_buf->start = addr;
  1962		cur_buf->end = end;
  1963		cur_buf->categories = categories;
  1964		return true;
  1965	}
  1966	
  1967	static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
  1968					       unsigned long addr, unsigned long end)
  1969	{
  1970		struct page_region *cur_buf = &p->cur_buf;
  1971	
  1972		if (cur_buf->start != addr)
  1973			cur_buf->end = addr;
  1974		else
  1975			cur_buf->start = cur_buf->end = 0;
  1976	
  1977		p->end_addr = 0;
  1978		p->found_pages -= (end - addr) / PAGE_SIZE;
  1979	}
  1980	
  1981	static int pagemap_scan_output(unsigned long categories,
  1982				       struct pagemap_scan_private *p,
  1983				       unsigned long addr, unsigned long *end)
  1984	{
  1985		unsigned long n_pages, total_pages;
  1986		int ret = 0;
  1987	
  1988		if (!pagemap_scan_is_interesting_page(categories, p)) {
  1989			*end = addr;
  1990			return 0;
  1991		}
  1992	
  1993		if (!p->vec_buf)
  1994			return 0;
  1995	
  1996		categories &= p->arg.return_mask;
  1997	
  1998		n_pages = (*end - addr) / PAGE_SIZE;
  1999		if (check_add_overflow(p->found_pages, n_pages, &total_pages) || total_pages > p->arg.max_pages) {
  2000			size_t n_too_much = total_pages - p->arg.max_pages;
  2001			*end -= n_too_much * PAGE_SIZE;
  2002			n_pages -= n_too_much;
  2003			ret = -ENOSPC;
  2004		}
  2005	
  2006		if (!pagemap_scan_push_range(categories, p, addr, *end)) {
  2007			*end = addr;
  2008			n_pages = 0;
  2009			ret = -ENOSPC;
  2010		}
  2011	
  2012		p->found_pages += n_pages;
  2013		if (ret)
  2014			p->end_addr = *end;
  2015		return ret;
  2016	}
  2017	
  2018	static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
  2019					  unsigned long end, struct mm_walk *walk)
  2020	{
  2021	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
  2022		struct pagemap_scan_private *p = walk->private;
  2023		struct vm_area_struct *vma = walk->vma;
  2024		unsigned long categories;
  2025		spinlock_t *ptl;
  2026		int ret = 0;
  2027	
  2028		ptl = pmd_trans_huge_lock(pmd, vma);
  2029		if (!ptl)
  2030			return -ENOENT;
  2031	
  2032		categories = p->cur_vma_category | pagemap_thp_category(*pmd);
  2033	
  2034		ret = pagemap_scan_output(categories, p, start, &end);
  2035		if (start == end)
  2036			goto out_unlock;
  2037	
  2038		if (~p->arg.flags & PM_SCAN_WP_MATCHING)
  2039			goto out_unlock;
  2040		if (~categories & PAGE_IS_WRITTEN)
  2041			goto out_unlock;
  2042	
  2043		/*
  2044		 * Break huge page into small pages if the WP operation
  2045		 * need to be performed is on a portion of the huge page.
  2046		 */
> 2047		if (end != addr + HPAGE_SIZE) {
  2048			spin_unlock(ptl);
  2049			split_huge_pmd(vma, pmd, start);
  2050			pagemap_scan_backout_range(p, start, end);
  2051			return -ENOENT;
  2052		}
  2053	
  2054		make_uffd_wp_pmd(vma, start, pmd);
  2055		flush_tlb_range(vma, start, end);
  2056	out_unlock:
  2057		spin_unlock(ptl);
  2058		return ret;
  2059	#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
  2060		return -ENOENT;
  2061	#endif
  2062	}
  2063	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[kernel test robot]

Hi Michał,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.5-rc2 next-20230721]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Micha-Miros-aw/fs-proc-task_mmu-Implement-IOCTL-for-efficient-page-table-scanning/20230722-082500
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/ZLshsAj5PbsEAHhP%40qmqm.qmqm.pl
patch subject: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning
config: x86_64-rhel-8.3 (https://download.01.org/0day-ci/archive/20230722/202307222141.3RAiL0jR-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce: (https://download.01.org/0day-ci/archive/20230722/202307222141.3RAiL0jR-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307222141.3RAiL0jR-lkp@intel.com/

All errors (new ones prefixed by >>):

   fs/proc/task_mmu.c: In function 'pagemap_scan_test_walk':
   fs/proc/task_mmu.c:1921:13: error: implicit declaration of function 'userfaultfd_wp_async'; did you mean 'userfaultfd_wp'? [-Werror=implicit-function-declaration]
    1921 |         if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
         |             ^~~~~~~~~~~~~~~~~~~~
         |             userfaultfd_wp
   fs/proc/task_mmu.c: In function 'pagemap_scan_thp_entry':
>> fs/proc/task_mmu.c:2047:20: error: 'addr' undeclared (first use in this function)
    2047 |         if (end != addr + HPAGE_SIZE) {
         |                    ^~~~
   fs/proc/task_mmu.c:2047:20: note: each undeclared identifier is reported only once for each function it appears in
   cc1: some warnings being treated as errors


vim +/addr +2047 fs/proc/task_mmu.c

  2017	
  2018	static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
  2019					  unsigned long end, struct mm_walk *walk)
  2020	{
  2021	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
  2022		struct pagemap_scan_private *p = walk->private;
  2023		struct vm_area_struct *vma = walk->vma;
  2024		unsigned long categories;
  2025		spinlock_t *ptl;
  2026		int ret = 0;
  2027	
  2028		ptl = pmd_trans_huge_lock(pmd, vma);
  2029		if (!ptl)
  2030			return -ENOENT;
  2031	
  2032		categories = p->cur_vma_category | pagemap_thp_category(*pmd);
  2033	
  2034		ret = pagemap_scan_output(categories, p, start, &end);
  2035		if (start == end)
  2036			goto out_unlock;
  2037	
  2038		if (~p->arg.flags & PM_SCAN_WP_MATCHING)
  2039			goto out_unlock;
  2040		if (~categories & PAGE_IS_WRITTEN)
  2041			goto out_unlock;
  2042	
  2043		/*
  2044		 * Break huge page into small pages if the WP operation
  2045		 * need to be performed is on a portion of the huge page.
  2046		 */
> 2047		if (end != addr + HPAGE_SIZE) {
  2048			spin_unlock(ptl);
  2049			split_huge_pmd(vma, pmd, start);
  2050			pagemap_scan_backout_range(p, start, end);
  2051			return -ENOENT;
  2052		}
  2053	
  2054		make_uffd_wp_pmd(vma, start, pmd);
  2055		flush_tlb_range(vma, start, end);
  2056	out_unlock:
  2057		spin_unlock(ptl);
  2058		return ret;
  2059	#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
  2060		return -ENOENT;
  2061	#endif
  2062	}
  2063	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

Fixed found bugs. Testing it further.

- Split and backoff in case buffer full case as well
- Fix the wrong breaking of loop if page isn't interesting, skip intead
- Untag the address and save them into struct
- Round off the end address to next page

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
 fs/proc/task_mmu.c | 54 ++++++++++++++++++++++++++--------------------
 1 file changed, 31 insertions(+), 23 deletions(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index add21fdf3c9a..64b326d0ec6d 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1764,7 +1764,8 @@ struct pagemap_scan_private {
 	struct page_region __user* vec_out;
 };

-static unsigned long pagemap_page_category(struct vm_area_struct *vma,
unsigned long addr, pte_t pte)
+static unsigned long pagemap_page_category(struct vm_area_struct *vma,
+					   unsigned long addr, pte_t pte)
 {
 	unsigned long categories = 0;

@@ -1908,6 +1909,7 @@ static bool pagemap_scan_is_interesting_vma(unsigned
long categories,
 	categories ^= p->arg.category_inverted;
 	if ((categories & required) != required)
 		return false;
+
 	return true;
 }

@@ -1930,6 +1932,7 @@ static int pagemap_scan_test_walk(unsigned long
start, unsigned long end,
 		return 1;

 	p->cur_vma_category = vma_category;
+
 	return 0;
 }

@@ -1961,6 +1964,7 @@ static bool pagemap_scan_push_range(unsigned long
categories,
 	cur_buf->start = addr;
 	cur_buf->end = end;
 	cur_buf->categories = categories;
+
 	return true;
 }

@@ -1985,18 +1989,19 @@ static int pagemap_scan_output(unsigned long
categories,
 	unsigned long n_pages, total_pages;
 	int ret = 0;

+	if (!p->vec_buf)
+		return 0;
+
 	if (!pagemap_scan_is_interesting_page(categories, p)) {
 		*end = addr;
 		return 0;
 	}

-	if (!p->vec_buf)
-		return 0;
-
 	categories &= p->arg.return_mask;

 	n_pages = (*end - addr) / PAGE_SIZE;
-	if (check_add_overflow(p->found_pages, n_pages, &total_pages) ||
total_pages > p->arg.max_pages) {
+	if (check_add_overflow(p->found_pages, n_pages, &total_pages) ||
+	    total_pages > p->arg.max_pages) {
 		size_t n_too_much = total_pages - p->arg.max_pages;
 		*end -= n_too_much * PAGE_SIZE;
 		n_pages -= n_too_much;
@@ -2012,6 +2017,7 @@ static int pagemap_scan_output(unsigned long categories,
 	p->found_pages += n_pages;
 	if (ret)
 		p->end_addr = *end;
+
 	return ret;
 }

@@ -2044,7 +2050,7 @@ static int pagemap_scan_thp_entry(pmd_t *pmd,
unsigned long start,
 	 * Break huge page into small pages if the WP operation
 	 * need to be performed is on a portion of the huge page.
 	 */
-	if (end != start + HPAGE_SIZE) {
+	if (end != start + HPAGE_SIZE || ret == -ENOSPC) {
 		spin_unlock(ptl);
 		split_huge_pmd(vma, pmd, start);
 		pagemap_scan_backout_range(p, start, end);
@@ -2066,8 +2072,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
unsigned long start,
 {
 	struct pagemap_scan_private *p = walk->private;
 	struct vm_area_struct *vma = walk->vma;
+	unsigned long addr, categories, next;
 	pte_t *pte, *start_pte;
-	unsigned long addr;
 	bool flush = false;
 	spinlock_t *ptl;
 	int ret;
@@ -2088,12 +2094,14 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
unsigned long start,
 	}

 	for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
-		unsigned long categories = p->cur_vma_category |
-			pagemap_page_category(vma, addr, ptep_get(pte));
-		unsigned long next = addr + PAGE_SIZE;
+		categories = p->cur_vma_category |
+			     pagemap_page_category(vma, addr, ptep_get(pte));
+		next = addr + PAGE_SIZE;

 		ret = pagemap_scan_output(categories, p, addr, &next);
-		if (next == addr)
+		if (ret == 0 && next == addr)
+			continue;
+		else if (next == addr)
 			break;

 		if (~p->arg.flags & PM_SCAN_WP_MATCHING)
@@ -2175,7 +2183,7 @@ static int pagemap_scan_pte_hole(unsigned long addr,
unsigned long end,
 {
 	struct pagemap_scan_private *p = walk->private;
 	struct vm_area_struct *vma = walk->vma;
-	int ret;
+	int ret, err;

 	if (!vma)
 		return 0;
@@ -2187,7 +2195,7 @@ static int pagemap_scan_pte_hole(unsigned long addr,
unsigned long end,
 	if (~p->arg.flags & PM_SCAN_WP_MATCHING)
 		return ret;

-	int err = uffd_wp_range(vma, addr, end - addr, true);
+	err = uffd_wp_range(vma, addr, end - addr, true);
 	if (err < 0)
 		ret = err;

@@ -2204,8 +2212,6 @@ static const struct mm_walk_ops pagemap_scan_ops = {
 static int pagemap_scan_get_args(struct pm_scan_arg *arg,
 				 unsigned long uarg)
 {
-	unsigned long start, end, vec;
-
 	if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
 		return -EFAULT;

@@ -2219,22 +2225,24 @@ static int pagemap_scan_get_args(struct pm_scan_arg
*arg,
 	     arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
 		return -EINVAL;

-	start = untagged_addr((unsigned long)arg->start);
-	end = untagged_addr((unsigned long)arg->end);
-	vec = untagged_addr((unsigned long)arg->vec);
+	arg->start = untagged_addr((unsigned long)arg->start);
+	arg->end = untagged_addr((unsigned long)arg->end);
+	arg->vec = untagged_addr((unsigned long)arg->vec);

 	/* Validate memory pointers */
-	if (!IS_ALIGNED(start, PAGE_SIZE))
+	if (!IS_ALIGNED(arg->start, PAGE_SIZE))
 		return -EINVAL;
-	if (!access_ok((void __user *)start, end - start))
+	if (!access_ok((void __user *)arg->start, arg->end - arg->start))
 		return -EFAULT;
-	if (!vec && arg->vec_len)
+	if (!arg->vec && arg->vec_len)
 		return -EFAULT;
-	if (vec && !access_ok((void __user *)vec,
+	if (arg->vec && !access_ok((void __user *)arg->vec,
 			      arg->vec_len * sizeof(struct page_region)))
 		return -EFAULT;

 	/* Fixup default values */
+	arg->end = (arg->end & ~PAGE_MASK) ?
+		   ((arg->end & PAGE_MASK) + PAGE_SIZE) : (arg->end);
 	if (!arg->max_pages)
 		arg->max_pages = ULONG_MAX;

@@ -2279,7 +2287,7 @@ static int pagemap_scan_init_bounce_buffer(struct
pagemap_scan_private *p)
 	if (!p->vec_buf)
 		return -ENOMEM;

-	p->vec_out = (void __user *)p->arg.vec;
+	p->vec_out = (struct page_region __user *)p->arg.vec;

 	return 0;
 }
-- 
2.39.2

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Mon, 24 Jul 2023 at 16:04, Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> Fixed found bugs. Testing it further.
>
> - Split and backoff in case buffer full case as well
> - Fix the wrong breaking of loop if page isn't interesting, skip intead
> - Untag the address and save them into struct
> - Round off the end address to next page
>
> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> ---
>  fs/proc/task_mmu.c | 54 ++++++++++++++++++++++++++--------------------
>  1 file changed, 31 insertions(+), 23 deletions(-)
>
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index add21fdf3c9a..64b326d0ec6d 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -1985,18 +1989,19 @@ static int pagemap_scan_output(unsigned long
> categories,
>         unsigned long n_pages, total_pages;
>         int ret = 0;
>
> +       if (!p->vec_buf)
> +               return 0;
> +
>         if (!pagemap_scan_is_interesting_page(categories, p)) {
>                 *end = addr;
>                 return 0;
>         }
>
> -       if (!p->vec_buf)
> -               return 0;
> -
>         categories &= p->arg.return_mask;

This is wrong - is_interesting() check must happen before output as
the `*end = addr` means the range should be skipped, but return 0
requests continuing of the walk.

> @@ -2044,7 +2050,7 @@ static int pagemap_scan_thp_entry(pmd_t *pmd,
> unsigned long start,
>          * Break huge page into small pages if the WP operation
>          * need to be performed is on a portion of the huge page.
>          */
> -       if (end != start + HPAGE_SIZE) {
> +       if (end != start + HPAGE_SIZE || ret == -ENOSPC) {

Why is it needed? If `end == start + HPAGE_SIZE` then we're handling a
full hugepage anyway.

> @@ -2066,8 +2072,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
> unsigned long start,
>  {
>         struct pagemap_scan_private *p = walk->private;
>         struct vm_area_struct *vma = walk->vma;
> +       unsigned long addr, categories, next;
>         pte_t *pte, *start_pte;
> -       unsigned long addr;
>         bool flush = false;
>         spinlock_t *ptl;
>         int ret;
> @@ -2088,12 +2094,14 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
> unsigned long start,
>         }
>
>         for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
> -               unsigned long categories = p->cur_vma_category |
> -                       pagemap_page_category(vma, addr, ptep_get(pte));
> -               unsigned long next = addr + PAGE_SIZE;
> +               categories = p->cur_vma_category |
> +                            pagemap_page_category(vma, addr, ptep_get(pte));
> +               next = addr + PAGE_SIZE;

Why moving the variable declarations out of the loop?

>
>                 ret = pagemap_scan_output(categories, p, addr, &next);
> -               if (next == addr)
> +               if (ret == 0 && next == addr)
> +                       continue;
> +               else if (next == addr)
>                         break;

Ah, this indeed was a bug. Nit:

if (next == addr) { if (!ret) continue; break; }

> @@ -2204,8 +2212,6 @@ static const struct mm_walk_ops pagemap_scan_ops = {
>  static int pagemap_scan_get_args(struct pm_scan_arg *arg,
>                                  unsigned long uarg)
>  {
> -       unsigned long start, end, vec;
> -
>         if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
>                 return -EFAULT;
>
> @@ -2219,22 +2225,24 @@ static int pagemap_scan_get_args(struct pm_scan_arg
> *arg,
>              arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
>                 return -EINVAL;
>
> -       start = untagged_addr((unsigned long)arg->start);
> -       end = untagged_addr((unsigned long)arg->end);
> -       vec = untagged_addr((unsigned long)arg->vec);
> +       arg->start = untagged_addr((unsigned long)arg->start);
> +       arg->end = untagged_addr((unsigned long)arg->end);
> +       arg->vec = untagged_addr((unsigned long)arg->vec);

BTW, We should we keep the tag in args writeback().

>         /* Validate memory pointers */
> -       if (!IS_ALIGNED(start, PAGE_SIZE))
> +       if (!IS_ALIGNED(arg->start, PAGE_SIZE))
>                 return -EINVAL;
> -       if (!access_ok((void __user *)start, end - start))
> +       if (!access_ok((void __user *)arg->start, arg->end - arg->start))
>                 return -EFAULT;
> -       if (!vec && arg->vec_len)
> +       if (!arg->vec && arg->vec_len)
>                 return -EFAULT;
> -       if (vec && !access_ok((void __user *)vec,
> +       if (arg->vec && !access_ok((void __user *)arg->vec,
>                               arg->vec_len * sizeof(struct page_region)))
>                 return -EFAULT;
>
>         /* Fixup default values */
> +       arg->end = (arg->end & ~PAGE_MASK) ?
> +                  ((arg->end & PAGE_MASK) + PAGE_SIZE) : (arg->end);

arg->end = ALIGN(arg->end, PAGE_SIZE);

>         if (!arg->max_pages)
>                 arg->max_pages = ULONG_MAX;
>

Best Regards
Michał Mirosław

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/24/23 7:38 PM, Michał Mirosław wrote:
> On Mon, 24 Jul 2023 at 16:04, Muhammad Usama Anjum
> <usama.anjum@collabora.com> wrote:
>>
>> Fixed found bugs. Testing it further.
>>
>> - Split and backoff in case buffer full case as well
>> - Fix the wrong breaking of loop if page isn't interesting, skip intead
>> - Untag the address and save them into struct
>> - Round off the end address to next page
>>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>> ---
>>  fs/proc/task_mmu.c | 54 ++++++++++++++++++++++++++--------------------
>>  1 file changed, 31 insertions(+), 23 deletions(-)
>>
>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>> index add21fdf3c9a..64b326d0ec6d 100644
>> --- a/fs/proc/task_mmu.c
>> +++ b/fs/proc/task_mmu.c
>> @@ -1985,18 +1989,19 @@ static int pagemap_scan_output(unsigned long
>> categories,
>>         unsigned long n_pages, total_pages;
>>         int ret = 0;
>>
>> +       if (!p->vec_buf)
>> +               return 0;
>> +
>>         if (!pagemap_scan_is_interesting_page(categories, p)) {
>>                 *end = addr;
>>                 return 0;
>>         }
>>
>> -       if (!p->vec_buf)
>> -               return 0;
>> -
>>         categories &= p->arg.return_mask;
> 
> This is wrong - is_interesting() check must happen before output as
> the `*end = addr` means the range should be skipped, but return 0
> requests continuing of the walk.
Will revert.

> 
>> @@ -2044,7 +2050,7 @@ static int pagemap_scan_thp_entry(pmd_t *pmd,
>> unsigned long start,
>>          * Break huge page into small pages if the WP operation
>>          * need to be performed is on a portion of the huge page.
>>          */
>> -       if (end != start + HPAGE_SIZE) {
>> +       if (end != start + HPAGE_SIZE || ret == -ENOSPC) {
> 
> Why is it needed? If `end == start + HPAGE_SIZE` then we're handling a
> full hugepage anyway.
If we weren't able to add the complete thp in the output buffer and we need
to perform WP on the entire page, we should split and rollback. Otherwise
we'll WP the entire thp and we'll lose the state on the remaining THP which
wasn't added to output.

Lets say max=100
only 100 pages would be added to output
we need to split and rollback otherwise other 412 pages would get WP

> 
>> @@ -2066,8 +2072,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
>> unsigned long start,
>>  {
>>         struct pagemap_scan_private *p = walk->private;
>>         struct vm_area_struct *vma = walk->vma;
>> +       unsigned long addr, categories, next;
>>         pte_t *pte, *start_pte;
>> -       unsigned long addr;
>>         bool flush = false;
>>         spinlock_t *ptl;
>>         int ret;
>> @@ -2088,12 +2094,14 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
>> unsigned long start,
>>         }
>>
>>         for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
>> -               unsigned long categories = p->cur_vma_category |
>> -                       pagemap_page_category(vma, addr, ptep_get(pte));
>> -               unsigned long next = addr + PAGE_SIZE;
>> +               categories = p->cur_vma_category |
>> +                            pagemap_page_category(vma, addr, ptep_get(pte));
>> +               next = addr + PAGE_SIZE;
> 
> Why moving the variable declarations out of the loop?
Saving spaces inside loop. What are pros of declation of variable in loop?

> 
>>
>>                 ret = pagemap_scan_output(categories, p, addr, &next);
>> -               if (next == addr)
>> +               if (ret == 0 && next == addr)
>> +                       continue;
>> +               else if (next == addr)
>>                         break;
> 
> Ah, this indeed was a bug. Nit:
> 
> if (next == addr) { if (!ret) continue; break; }
> 
>> @@ -2204,8 +2212,6 @@ static const struct mm_walk_ops pagemap_scan_ops = {
>>  static int pagemap_scan_get_args(struct pm_scan_arg *arg,
>>                                  unsigned long uarg)
>>  {
>> -       unsigned long start, end, vec;
>> -
>>         if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
>>                 return -EFAULT;
>>
>> @@ -2219,22 +2225,24 @@ static int pagemap_scan_get_args(struct pm_scan_arg
>> *arg,
>>              arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
>>                 return -EINVAL;
>>
>> -       start = untagged_addr((unsigned long)arg->start);
>> -       end = untagged_addr((unsigned long)arg->end);
>> -       vec = untagged_addr((unsigned long)arg->vec);
>> +       arg->start = untagged_addr((unsigned long)arg->start);
>> +       arg->end = untagged_addr((unsigned long)arg->end);
>> +       arg->vec = untagged_addr((unsigned long)arg->vec);
> 
> BTW, We should we keep the tag in args writeback().
Sorry what?
After this function, the start, end and vec would be used. We need to make
sure that the address are untagged before that.

> 
>>         /* Validate memory pointers */
>> -       if (!IS_ALIGNED(start, PAGE_SIZE))
>> +       if (!IS_ALIGNED(arg->start, PAGE_SIZE))
>>                 return -EINVAL;
>> -       if (!access_ok((void __user *)start, end - start))
>> +       if (!access_ok((void __user *)arg->start, arg->end - arg->start))
>>                 return -EFAULT;
>> -       if (!vec && arg->vec_len)
>> +       if (!arg->vec && arg->vec_len)
>>                 return -EFAULT;
>> -       if (vec && !access_ok((void __user *)vec,
>> +       if (arg->vec && !access_ok((void __user *)arg->vec,
>>                               arg->vec_len * sizeof(struct page_region)))
>>                 return -EFAULT;
>>
>>         /* Fixup default values */
>> +       arg->end = (arg->end & ~PAGE_MASK) ?
>> +                  ((arg->end & PAGE_MASK) + PAGE_SIZE) : (arg->end);
> 
> arg->end = ALIGN(arg->end, PAGE_SIZE);
> 
>>         if (!arg->max_pages)
>>                 arg->max_pages = ULONG_MAX;
>>
> 
> Best Regards
> Michał Mirosław

-- 
BR,
Muhammad Usama Anjum

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Mon, 24 Jul 2023 at 17:22, Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> On 7/24/23 7:38 PM, Michał Mirosław wrote:
> > On Mon, 24 Jul 2023 at 16:04, Muhammad Usama Anjum
> > <usama.anjum@collabora.com> wrote:
> >>
> >> Fixed found bugs. Testing it further.
> >>
> >> - Split and backoff in case buffer full case as well
> >> - Fix the wrong breaking of loop if page isn't interesting, skip intead
> >> - Untag the address and save them into struct
> >> - Round off the end address to next page
> >>
> >> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> >> ---
> >>  fs/proc/task_mmu.c | 54 ++++++++++++++++++++++++++--------------------
> >>  1 file changed, 31 insertions(+), 23 deletions(-)
> >>
> >> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> >> index add21fdf3c9a..64b326d0ec6d 100644
> >> --- a/fs/proc/task_mmu.c
> >> +++ b/fs/proc/task_mmu.c
> >> @@ -2044,7 +2050,7 @@ static int pagemap_scan_thp_entry(pmd_t *pmd,
> >> unsigned long start,
> >>          * Break huge page into small pages if the WP operation
> >>          * need to be performed is on a portion of the huge page.
> >>          */
> >> -       if (end != start + HPAGE_SIZE) {
> >> +       if (end != start + HPAGE_SIZE || ret == -ENOSPC) {
> >
> > Why is it needed? If `end == start + HPAGE_SIZE` then we're handling a
> > full hugepage anyway.
> If we weren't able to add the complete thp in the output buffer and we need
> to perform WP on the entire page, we should split and rollback. Otherwise
> we'll WP the entire thp and we'll lose the state on the remaining THP which
> wasn't added to output.
>
> Lets say max=100
> only 100 pages would be added to output
> we need to split and rollback otherwise other 412 pages would get WP

In this case *end will be truncated by output() to match the number of
pages that fit.

> >> @@ -2066,8 +2072,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
> >> unsigned long start,
> >>  {
> >>         struct pagemap_scan_private *p = walk->private;
> >>         struct vm_area_struct *vma = walk->vma;
> >> +       unsigned long addr, categories, next;
> >>         pte_t *pte, *start_pte;
> >> -       unsigned long addr;
> >>         bool flush = false;
> >>         spinlock_t *ptl;
> >>         int ret;
> >> @@ -2088,12 +2094,14 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd,
> >> unsigned long start,
> >>         }
> >>
> >>         for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
> >> -               unsigned long categories = p->cur_vma_category |
> >> -                       pagemap_page_category(vma, addr, ptep_get(pte));
> >> -               unsigned long next = addr + PAGE_SIZE;
> >> +               categories = p->cur_vma_category |
> >> +                            pagemap_page_category(vma, addr, ptep_get(pte));
> >> +               next = addr + PAGE_SIZE;
> >
> > Why moving the variable declarations out of the loop?
> Saving spaces inside loop. What are pros of declation of variable in loop?

Informing the reader that the variables have scope limited to the loop body.

[...]
> >> @@ -2219,22 +2225,24 @@ static int pagemap_scan_get_args(struct pm_scan_arg
> >> *arg,
> >>              arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
> >>                 return -EINVAL;
> >>
> >> -       start = untagged_addr((unsigned long)arg->start);
> >> -       end = untagged_addr((unsigned long)arg->end);
> >> -       vec = untagged_addr((unsigned long)arg->vec);
> >> +       arg->start = untagged_addr((unsigned long)arg->start);
> >> +       arg->end = untagged_addr((unsigned long)arg->end);
> >> +       arg->vec = untagged_addr((unsigned long)arg->vec);
> >
> > BTW, We should we keep the tag in args writeback().
> Sorry what?
> After this function, the start, end and vec would be used. We need to make
> sure that the address are untagged before that.

We do write back the address the walk ended at to arg->start in
userspace. This pointer I think needs the tag reconstructed so that
retrying the ioctl() will be possible.

 Best Regards
Michał Mirosław

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/24/23 9:10 PM, Michał Mirosław wrote:
[...]>>>> @@ -2219,22 +2225,24 @@ static int pagemap_scan_get_args(struct
pm_scan_arg
>>>> *arg,
>>>>              arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
>>>>                 return -EINVAL;
>>>>
>>>> -       start = untagged_addr((unsigned long)arg->start);
>>>> -       end = untagged_addr((unsigned long)arg->end);
>>>> -       vec = untagged_addr((unsigned long)arg->vec);
>>>> +       arg->start = untagged_addr((unsigned long)arg->start);
>>>> +       arg->end = untagged_addr((unsigned long)arg->end);
>>>> +       arg->vec = untagged_addr((unsigned long)arg->vec);
>>>
>>> BTW, We should we keep the tag in args writeback().
>> Sorry what?
>> After this function, the start, end and vec would be used. We need to make
>> sure that the address are untagged before that.
> 
> We do write back the address the walk ended at to arg->start in
> userspace. This pointer I think needs the tag reconstructed so that
> retrying the ioctl() will be possible.
Even if we reconstruct the tag for end and vec, We need to update the start
address. Can we just put same tag as original start in it? I'm not sure.

-- 
BR,
Muhammad Usama Anjum

Re: [v2] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/25/23 12:23 PM, Muhammad Usama Anjum wrote:
> On 7/24/23 9:10 PM, Michał Mirosław wrote:
> [...]>>>> @@ -2219,22 +2225,24 @@ static int pagemap_scan_get_args(struct
> pm_scan_arg
>>>>> *arg,
>>>>>              arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
>>>>>                 return -EINVAL;
>>>>>
>>>>> -       start = untagged_addr((unsigned long)arg->start);
>>>>> -       end = untagged_addr((unsigned long)arg->end);
>>>>> -       vec = untagged_addr((unsigned long)arg->vec);
>>>>> +       arg->start = untagged_addr((unsigned long)arg->start);
>>>>> +       arg->end = untagged_addr((unsigned long)arg->end);
>>>>> +       arg->vec = untagged_addr((unsigned long)arg->vec);
>>>>
>>>> BTW, We should we keep the tag in args writeback().
>>> Sorry what?
>>> After this function, the start, end and vec would be used. We need to make
>>> sure that the address are untagged before that.
>>
>> We do write back the address the walk ended at to arg->start in
>> userspace. This pointer I think needs the tag reconstructed so that
>> retrying the ioctl() will be possible.
> Even if we reconstruct the tag for end and vec, We need to update the start
> address. Can we just put same tag as original start in it? I'm not sure.
The special users would use tags. If they are using it, they'll just re-add
the tag in next invocation. I think this implementation is correct.

> 

-- 
BR,
Muhammad Usama Anjum

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

----
Michal please post your thoughts before I post this as v26.
----

This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are supported
in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
  file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT), swapped
  (PAGE_IS_SWAPPED) or page has pfn zero (PAGE_IS_PFNZERO).
- Find pages which have been written-to and/or write protect the pages
  (atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP)

This IOCTL can be extended to get information about more PTE bits. The
entire address range passed by user [start, end) is scanned until either
the user provided buffer is full or max_pages have been found.

Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Didn't added Andrei Vagin's reviewed-by tag as he'll need to review
again after Michal's input.

Changes in v26:
The changes are:

1. the API:
  a. return ranges as {begin, end} instead of {begin + len};
  b. rename match "flags" to 'page categories' everywhere - this makes
	it easier to differentiate the ioctl()s categorisation of pages
	from struct page flags;
  c. change {required + excluded} to {inverted + required}. This was
	rejected before, but I'd like to illustrate the difference.
	Old interface can be translated to the new by:
		categories_inverted = excluded_mask
		categories_mask = required_mask | excluded_mask
		categories_anyof_mask = anyof_mask
	The new way allows filtering by: A & (B | !C)
		categories_inverted = C
		categories_mask = A
		categories_anyof_mask = B | C
  e. allow no-op calls

2. the implementation:
  a. gather the page-categorising and write-protecting code in one place;
  b. optimization: add whole-vma skipping for WP usecase;
  c. extracted output limiting code to pagemap_scan_output();
  d. extracted range coalescing to pagemap_scan_push_range();
  e. extracted THP entry handling to pagemap_scan_thp_entry();
  f. added a shortcut for non-WP hugetlb scan; avoids conditional
	locking;
  g. extracted scan buffer handling code out of do_pagemap_scan();
  h. rework output code to always try to write pending ranges; if EFAULT
	is generated it always overwrites the original error code;
	(the case of SIGKILL is needlessly trying to write the output
	now, but this should be rare case and ignoring it makes the code
	not needing a goto)

  1. Change no-GET operation condition from `arg.return_mask == 0` to
     `arg.vec == NULL`. This will allow issuing the ioctl with
     return_mask == 0 to gather matching ranges when the exact category
     is not interesting. (Anticipated for CRIU scanning a large sparse
     anonymous mapping).

- Fix the wrong breaking of loop if page isn't interesting, skip intsead
- Untag the address and save them into struct
- Round off the end address to next page

Changes in v25:
- Do proper filtering on hole as well (hole got missed earlier)

Changes in v24:
- Place WP markers in case of hole as well

Changes in v23:
- Set vec_buf_index to 0 only when vec_buf_index is set
- Return -EFAULT instead of -EINVAL if vec is NULL
- Correctly return the walk ending address to the page granularity

Changes in v22:
- Interface change to return walk ending address to user:
  - Replace [start start + len) with [start, end)
  - Return the ending address of the address walk in start

Changes in v21:
- Abort walk instead of returning error if WP is to be performed on
  partial hugetlb
- Changed the data types of some variables in pagemap_scan_private to
  long

Changes in v20:
- Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO

Changes in v19:
- Interface changes such as renaming, return mask and WP can be used
  with any flags specified in masks
- Internal code changes

Changes in v18:
- Rebased on top of next-20230613
  - ptep_get() updates
  - remove pmd_trans_unstable() and add ACTION_AGAIN
- Review updates (Micheal)

Changes in v17:
- Rebased on next-20230606
- Made make_uffd_wp_*_pte() better and minor changes

Changes in v16:
- Fixed a corner case where kernel writes beyond user buffer by one
  element
- Bring back exclusive PM_SCAN_OP_WP
- Cosmetic changes

Changes in v15:
- Build fix:
  - Use generic tlb flush function in pagemap_scan_pmd_entry() instead of
    using x86 specific flush function in do_pagemap_scan()
  - Remove #ifdef from pagemap_scan_hugetlb_entry()
  - Use mm instead of undefined vma->vm_mm

Changes in v14:
- Fix build error caused by #ifdef added at last minute in some configs

Changes in v13:
- Review updates
- mmap_read_lock_killable() instead of mmap_read_lock()
- Replace uffd_wp_range() with helpers which increases performance
  drastically for OP_WP operations by reducing the number of tlb
  flushing etc
- Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range

Changes in v12:
- Add hugetlb support to cover all memory types
- Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch
- Review updates to the code

Changes in v11:
- Find written pages in a better way
- Fix a corner case (thanks Paul)
- Improve the code/comments
- remove ENGAGE_WP + ! GET operation
- shorten the commit message in favour of moving documentation to
  pagemap.rst

Changes in v10:
- move changes in tools/include/uapi/linux/fs.h to separate patch
- update commit message

Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code

Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
---
 fs/proc/task_mmu.c      | 649 ++++++++++++++++++++++++++++++++++++++++
 include/linux/hugetlb.h |   1 +
 include/uapi/linux/fs.h |  56 ++++
 mm/hugetlb.c            |   2 +-
 4 files changed, 707 insertions(+), 1 deletion(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index c1e6531cb02a..1593d26605d3 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -19,6 +19,8 @@
 #include <linux/shmem_fs.h>
 #include <linux/uaccess.h>
 #include <linux/pkeys.h>
+#include <linux/minmax.h>
+#include <linux/overflow.h>

 #include <asm/elf.h>
 #include <asm/tlb.h>
@@ -1749,11 +1751,658 @@ static int pagemap_release(struct inode *inode,
struct file *file)
 	return 0;
 }

+#define PM_SCAN_CATEGORIES	(PAGE_IS_WPASYNC | PAGE_IS_WRITTEN | \
+				 PAGE_IS_FILE |	PAGE_IS_PRESENT | \
+				 PAGE_IS_SWAPPED | PAGE_IS_PFNZERO)
+#define PM_SCAN_FLAGS		(PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
+
+struct pagemap_scan_private {
+	struct pm_scan_arg arg;
+	unsigned long cur_vma_category;
+	struct page_region *vec_buf, cur_buf;
+	unsigned long vec_buf_len, vec_buf_index, found_pages, end_addr;
+	struct page_region __user* vec_out;
+};
+
+static unsigned long pagemap_page_category(struct vm_area_struct *vma,
+					   unsigned long addr, pte_t pte)
+{
+	unsigned long categories = 0;
+
+	if (pte_present(pte)) {
+		struct page *page = vm_normal_page(vma, addr, pte);
+
+		categories |= PAGE_IS_PRESENT;
+		if (!pte_uffd_wp(pte))
+			categories |= PAGE_IS_WRITTEN;
+		if (page && !PageAnon(page))
+			categories |= PAGE_IS_FILE;
+		if (is_zero_pfn(pte_pfn(pte)))
+			categories |= PAGE_IS_PFNZERO;
+	} else if (is_swap_pte(pte)) {
+		swp_entry_t swp = pte_to_swp_entry(pte);
+
+		categories |= PAGE_IS_SWAPPED;
+		if (!pte_swp_uffd_wp_any(pte))
+			categories |= PAGE_IS_WRITTEN;
+		if (is_pfn_swap_entry(swp) && !PageAnon(pfn_swap_entry_to_page(swp)))
+			categories |= PAGE_IS_FILE;
+	}
+
+	return categories;
+}
+
+static void make_uffd_wp_pte(struct vm_area_struct *vma,
+			     unsigned long addr, pte_t *pte)
+{
+	pte_t ptent = ptep_get(pte);
+
+	if (pte_present(ptent)) {
+		pte_t old_pte;
+
+		old_pte = ptep_modify_prot_start(vma, addr, pte);
+		ptent = pte_mkuffd_wp(ptent);
+		ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
+	} else if (is_swap_pte(ptent)) {
+		ptent = pte_swp_mkuffd_wp(ptent);
+		set_pte_at(vma->vm_mm, addr, pte, ptent);
+	} else {
+		set_pte_at(vma->vm_mm, addr, pte,
+			   make_pte_marker(PTE_MARKER_UFFD_WP));
+	}
+}
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static unsigned long pagemap_thp_category(pmd_t pmd)
+{
+	unsigned long categories = 0;
+
+	if (pmd_present(pmd)) {
+		categories |= PAGE_IS_PRESENT;
+		if (!pmd_uffd_wp(pmd))
+			categories |= PAGE_IS_WRITTEN;
+		if (is_zero_pfn(pmd_pfn(pmd)))
+			categories |= PAGE_IS_PFNZERO;
+	} else if (is_swap_pmd(pmd)) {
+		categories |= PAGE_IS_SWAPPED;
+		if (!pmd_swp_uffd_wp(pmd))
+			categories |= PAGE_IS_WRITTEN;
+	}
+
+	return categories;
+}
+
+static void make_uffd_wp_pmd(struct vm_area_struct *vma,
+			     unsigned long addr, pmd_t *pmdp)
+{
+	pmd_t old, pmd = *pmdp;
+
+	if (pmd_present(pmd)) {
+		old = pmdp_invalidate_ad(vma, addr, pmdp);
+		pmd = pmd_mkuffd_wp(old);
+		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+	} else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
+		pmd = pmd_swp_mkuffd_wp(pmd);
+		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
+	}
+}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+
+#ifdef CONFIG_HUGETLB_PAGE
+static unsigned long pagemap_hugetlb_category(pte_t pte)
+{
+	unsigned long categories = 0;
+
+	if (pte_present(pte)) {
+		categories |= PAGE_IS_PRESENT;
+		if (!huge_pte_uffd_wp(pte))
+			categories |= PAGE_IS_WRITTEN;
+		if (!PageAnon(pte_page(pte)))
+			categories |= PAGE_IS_FILE;
+		if (is_zero_pfn(pte_pfn(pte)))
+			categories |= PAGE_IS_PFNZERO;
+	} else if (is_swap_pte(pte)) {
+		categories |= PAGE_IS_SWAPPED;
+		if (!pte_swp_uffd_wp_any(pte))
+			categories |= PAGE_IS_WRITTEN;
+	}
+
+	return categories;
+}
+
+static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
+				  unsigned long addr, pte_t *ptep,
+				  pte_t ptent)
+{
+	if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
+		return;
+
+	if (is_hugetlb_entry_migration(ptent))
+		set_huge_pte_at(vma->vm_mm, addr, ptep,
+				pte_swp_mkuffd_wp(ptent));
+	else if (!huge_pte_none(ptent))
+		huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
+					     huge_pte_mkuffd_wp(ptent));
+	else
+		set_huge_pte_at(vma->vm_mm, addr, ptep,
+				make_pte_marker(PTE_MARKER_UFFD_WP));
+}
+#endif /* CONFIG_HUGETLB_PAGE */
+
+static bool pagemap_scan_is_interesting_page(unsigned long categories,
+					     const struct pagemap_scan_private *p)
+{
+	categories ^= p->arg.category_inverted;
+	if ((categories & p->arg.category_mask) != p->arg.category_mask)
+		return false;
+	if (p->arg.category_anyof_mask && !(categories & p->arg.category_anyof_mask))
+		return false;
+
+	return true;
+}
+
+static bool pagemap_scan_is_interesting_vma(unsigned long categories,
+					    const struct pagemap_scan_private *p)
+{
+	unsigned long required = p->arg.category_mask & PAGE_IS_WPASYNC;
+
+	categories ^= p->arg.category_inverted;
+	if ((categories & required) != required)
+		return false;
+
+	return true;
+}
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
+				  struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	unsigned long vma_category = 0;
+
+	if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
+		vma_category |= PAGE_IS_WPASYNC;
+	else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
+		return -EPERM;
+
+	if (vma->vm_flags & VM_PFNMAP)
+		return 1;
+
+	if (!pagemap_scan_is_interesting_vma(vma_category, p))
+		return 1;
+
+	p->cur_vma_category = vma_category;
+
+	return 0;
+}
+
+static bool pagemap_scan_push_range(unsigned long categories,
+				    struct pagemap_scan_private *p,
+				    unsigned long addr, unsigned long end)
+{
+	struct page_region *cur_buf = &p->cur_buf;
+
+	/*
+	 * When there is no output buffer provided at all, the sentinel values
+	 * won't match here. There is no other way for `cur_buf->end` to be
+	 * non-zero other than it being non-empty.
+	 */
+	if (addr == cur_buf->end && categories == cur_buf->categories) {
+		cur_buf->end = end;
+		return true;
+	}
+
+	if (cur_buf->end) {
+		if (p->vec_buf_index >= p->vec_buf_len)
+			return false;
+
+		memcpy(&p->vec_buf[p->vec_buf_index], cur_buf,
+		       sizeof(*p->vec_buf));
+		++p->vec_buf_index;
+	}
+
+	cur_buf->start = addr;
+	cur_buf->end = end;
+	cur_buf->categories = categories;
+
+	return true;
+}
+
+static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
+				       unsigned long addr, unsigned long end)
+{
+	struct page_region *cur_buf = &p->cur_buf;
+
+	if (cur_buf->start != addr)
+		cur_buf->end = addr;
+	else
+		cur_buf->start = cur_buf->end = 0;
+
+	p->end_addr = 0;
+	p->found_pages -= (end - addr) / PAGE_SIZE;
+}
+
+static int pagemap_scan_output(unsigned long categories,
+			       struct pagemap_scan_private *p,
+			       unsigned long addr, unsigned long *end)
+{
+	unsigned long n_pages, total_pages;
+	int ret = 0;
+
+	if (!pagemap_scan_is_interesting_page(categories, p)) {
+		*end = addr;
+		return 0;
+	}
+
+	if (!p->vec_buf)
+		return 0;
+
+	categories &= p->arg.return_mask;
+
+	n_pages = (*end - addr) / PAGE_SIZE;
+	if (check_add_overflow(p->found_pages, n_pages, &total_pages) ||
+	    total_pages > p->arg.max_pages) {
+		size_t n_too_much = total_pages - p->arg.max_pages;
+		*end -= n_too_much * PAGE_SIZE;
+		n_pages -= n_too_much;
+		ret = -ENOSPC;
+	}
+
+	if (!pagemap_scan_push_range(categories, p, addr, *end)) {
+		*end = addr;
+		n_pages = 0;
+		ret = -ENOSPC;
+	}
+
+	p->found_pages += n_pages;
+	if (ret)
+		p->end_addr = *end;
+
+	return ret;
+}
+
+static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
+				  unsigned long end, struct mm_walk *walk)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	unsigned long categories;
+	spinlock_t *ptl;
+	int ret = 0;
+
+	ptl = pmd_trans_huge_lock(pmd, vma);
+	if (!ptl)
+		return -ENOENT;
+
+	categories = p->cur_vma_category | pagemap_thp_category(*pmd);
+
+	ret = pagemap_scan_output(categories, p, start, &end);
+	if (start == end)
+		goto out_unlock;
+
+	if (~p->arg.flags & PM_SCAN_WP_MATCHING)
+		goto out_unlock;
+	if (~categories & PAGE_IS_WRITTEN)
+		goto out_unlock;
+
+	/*
+	 * Break huge page into small pages if the WP operation
+	 * need to be performed is on a portion of the huge page.
+	 */
+	if (end != start + HPAGE_SIZE) {
+		spin_unlock(ptl);
+		split_huge_pmd(vma, pmd, start);
+		pagemap_scan_backout_range(p, start, end);
+		return -ENOENT;
+	}
+
+	make_uffd_wp_pmd(vma, start, pmd);
+	flush_tlb_range(vma, start, end);
+out_unlock:
+	spin_unlock(ptl);
+	return ret;
+#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
+	return -ENOENT;
+#endif
+}
+
+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+				  unsigned long end, struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	pte_t *pte, *start_pte;
+	unsigned long addr;
+	bool flush = false;
+	spinlock_t *ptl;
+	int ret;
+
+	arch_enter_lazy_mmu_mode();
+
+	ret = pagemap_scan_thp_entry(pmd, start, end, walk);
+	if (ret != -ENOENT) {
+		arch_leave_lazy_mmu_mode();
+		return ret;
+	}
+
+	start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+	if (!pte) {
+		arch_leave_lazy_mmu_mode();
+		walk->action = ACTION_AGAIN;
+		return 0;
+	}
+
+	for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
+		unsigned long categories = p->cur_vma_category |
+					   pagemap_page_category(vma, addr, ptep_get(pte));
+		unsigned long next = addr + PAGE_SIZE;
+
+		ret = pagemap_scan_output(categories, p, addr, &next);
+		if (next == addr) {
+			if (!ret)
+				continue;
+			break;
+		}
+
+		if (~p->arg.flags & PM_SCAN_WP_MATCHING)
+			continue;
+		if (~categories & PAGE_IS_WRITTEN)
+			continue;
+
+		make_uffd_wp_pte(vma, addr, pte);
+		if (!flush) {
+			start = addr;
+			flush = true;
+		}
+	}
+
+	if (flush)
+		flush_tlb_range(vma, start, addr);
+
+	pte_unmap_unlock(start_pte, ptl);
+	arch_leave_lazy_mmu_mode();
+
+	cond_resched();
+	return ret;
+}
+
+#ifdef CONFIG_HUGETLB_PAGE
+static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
+				      unsigned long start, unsigned long end,
+				      struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	unsigned long categories;
+	spinlock_t *ptl;
+	int ret = 0;
+	pte_t pte;
+
+	if (~p->arg.flags & PM_SCAN_WP_MATCHING) {
+		/* Go the short route when not write-protecting pages. */
+
+		pte = huge_ptep_get(ptep);
+		categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
+
+		return pagemap_scan_output(categories, p, start, &end);
+	}
+
+	if (end != start + HPAGE_SIZE) {
+		/* Partial HugeTLB page WP isn't possible. */
+		p->end_addr = start;
+		return -EINVAL;
+	}
+
+	i_mmap_lock_write(vma->vm_file->f_mapping);
+	ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
+
+	pte = huge_ptep_get(ptep);
+	categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
+
+	ret = pagemap_scan_output(categories, p, start, &end);
+	if (start == end)
+		goto out_unlock;
+
+	if (categories & PAGE_IS_WRITTEN) {
+		make_uffd_wp_huge_pte(vma, start, ptep, pte);
+		flush_hugetlb_tlb_range(vma, start, end);
+	}
+
+out_unlock:
+	spin_unlock(ptl);
+	i_mmap_unlock_write(vma->vm_file->f_mapping);
+
+	return ret;
+}
+#else
+#define pagemap_scan_hugetlb_entry NULL
+#endif
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
+				 int depth, struct mm_walk *walk)
+{
+	struct pagemap_scan_private *p = walk->private;
+	struct vm_area_struct *vma = walk->vma;
+	int ret, err;
+
+	if (!vma)
+		return 0;
+
+	ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end);
+	if (addr == end)
+		return ret;
+
+	if (~p->arg.flags & PM_SCAN_WP_MATCHING)
+		return ret;
+
+	err = uffd_wp_range(vma, addr, end - addr, true);
+	if (err < 0)
+		ret = err;
+
+	return ret;
+}
+
+static const struct mm_walk_ops pagemap_scan_ops = {
+	.test_walk = pagemap_scan_test_walk,
+	.pmd_entry = pagemap_scan_pmd_entry,
+	.pte_hole = pagemap_scan_pte_hole,
+	.hugetlb_entry = pagemap_scan_hugetlb_entry,
+};
+
+static int pagemap_scan_get_args(struct pm_scan_arg *arg,
+				 unsigned long uarg)
+{
+	if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
+		return -EFAULT;
+
+	if (arg->size != sizeof(struct pm_scan_arg))
+		return -EINVAL;
+
+	/* Validate requested features */
+	if (arg->flags & ~PM_SCAN_FLAGS)
+		return -EINVAL;
+	if ((arg->category_inverted | arg->category_mask |
+	     arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
+		return -EINVAL;
+
+	arg->start = untagged_addr((unsigned long)arg->start);
+	arg->end = untagged_addr((unsigned long)arg->end);
+	arg->vec = untagged_addr((unsigned long)arg->vec);
+
+	/* Validate memory pointers */
+	if (!IS_ALIGNED(arg->start, PAGE_SIZE))
+		return -EINVAL;
+	if (!access_ok((void __user *)arg->start, arg->end - arg->start))
+		return -EFAULT;
+	if (!arg->vec && arg->vec_len)
+		return -EFAULT;
+	if (arg->vec && !access_ok((void __user *)arg->vec,
+			      arg->vec_len * sizeof(struct page_region)))
+		return -EFAULT;
+
+	/* Fixup default values */
+	arg->end = ALIGN(arg->end, PAGE_SIZE);
+	if (!arg->max_pages)
+		arg->max_pages = ULONG_MAX;
+
+	return 0;
+}
+
+static int pagemap_scan_writeback_args(struct pm_scan_arg *arg,
+				       unsigned long uargl)
+{
+	struct pm_scan_arg __user *uarg	= (void __user *)uargl;
+
+	if (copy_to_user(&uarg->start, &arg->start, sizeof(arg->start)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p)
+{
+	if (!p->arg.vec_len) {
+		/*
+		 * An arbitrary non-page-aligned sentinel value for
+		 * pagemap_scan_push_range().
+		 */
+		p->cur_buf.start = p->cur_buf.end = ULLONG_MAX;
+		if (p->arg.vec)
+			p->vec_buf = ZERO_SIZE_PTR;
+		return 0;
+	}
+
+	/*
+	 * Allocate a smaller buffer to get output from inside the page
+	 * walk functions and walk the range in PAGEMAP_WALK_SIZE chunks.
+	 * The last range is always stored in p.cur_buf to allow coalescing
+	 * consecutive ranges that have the same categories returned across
+	 * walk_page_range() calls.
+	 */
+	p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT,
+			       p->arg.vec_len - 1);
+	p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf),
+				   GFP_KERNEL);
+	if (!p->vec_buf)
+		return -ENOMEM;
+
+	p->vec_out = (struct page_region __user *)p->arg.vec;
+
+	return 0;
+}
+
+static int pagemap_scan_flush_buffer(struct pagemap_scan_private *p)
+{
+	const struct page_region *buf = p->vec_buf;
+	int n = (int)p->vec_buf_index;
+
+	if (!n)
+		return 0;
+
+	if (copy_to_user(p->vec_out, buf, n * sizeof(*buf)))
+		return -EFAULT;
+
+	p->arg.vec_len -= n;
+	p->vec_out += n;
+
+	p->vec_buf_index = 0;
+	p->vec_buf_len = min_t(size_t, p->vec_buf_len, p->arg.vec_len - 1);
+
+	return n;
+}
+
+static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg)
+{
+	unsigned long walk_start, walk_end;
+	struct mmu_notifier_range range;
+	struct pagemap_scan_private p;
+	size_t n_ranges_out = 0;
+	int ret;
+
+	memset(&p, 0, sizeof(p));
+	ret = pagemap_scan_get_args(&p.arg, uarg);
+	if (ret)
+		return ret;
+
+	ret = pagemap_scan_init_bounce_buffer(&p);
+	if (ret)
+		return ret;
+
+	/* Protection change for the range is going to happen. */
+	if (p.arg.flags & PM_SCAN_WP_MATCHING) {
+		mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
+					mm, p.arg.start, p.arg.end);
+		mmu_notifier_invalidate_range_start(&range);
+	}
+
+	walk_start = walk_end = p.arg.start;
+	for (; walk_end != p.arg.end; walk_start = walk_end) {
+		int n_out;
+		walk_end = min_t(unsigned long,
+				 (walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK,
+				 p.arg.end);
+
+		ret = mmap_read_lock_killable(mm);
+		if (ret)
+			break;
+		ret = walk_page_range(mm, walk_start, walk_end,
+				      &pagemap_scan_ops, &p);
+		mmap_read_unlock(mm);
+
+		n_out = pagemap_scan_flush_buffer(&p);
+		if (n_out < 0)
+			ret = n_out;
+		else
+			n_ranges_out += n_out;
+
+		if (ret)
+			break;
+	}
+
+	if (p.cur_buf.start != p.cur_buf.end) {
+		if (copy_to_user(p.vec_out, &p.cur_buf, sizeof(p.cur_buf)))
+			ret = -EFAULT;
+		else
+			++n_ranges_out;
+	}
+
+	/* ENOSPC signifies early stop (buffer full) from the walk. */
+	if (!ret || ret == -ENOSPC)
+		ret = n_ranges_out;
+
+	p.arg.start = p.end_addr ? p.end_addr : walk_start;
+	if (pagemap_scan_writeback_args(&p.arg, uarg))
+		ret = -EFAULT;
+
+	if (p.arg.flags & PM_SCAN_WP_MATCHING)
+		mmu_notifier_invalidate_range_end(&range);
+
+	kfree(p.vec_buf);
+	return ret;
+}
+
+static long do_pagemap_cmd(struct file *file, unsigned int cmd,
+			   unsigned long arg)
+{
+	struct mm_struct *mm = file->private_data;
+
+	switch (cmd) {
+	case PAGEMAP_SCAN:
+		return do_pagemap_scan(mm, arg);
+
+	default:
+		return -EINVAL;
+	}
+}
+
 const struct file_operations proc_pagemap_operations = {
 	.llseek		= mem_lseek, /* borrow this */
 	.read		= pagemap_read,
 	.open		= pagemap_open,
 	.release	= pagemap_release,
+	.unlocked_ioctl = do_pagemap_cmd,
+	.compat_ioctl	= do_pagemap_cmd,
 };
 #endif /* CONFIG_PROC_PAGE_MONITOR */

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 0a393bc02f25..8f8ff07453f2 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 		unsigned long cp_flags);

 bool is_hugetlb_entry_migration(pte_t pte);
+bool is_hugetlb_entry_hwpoisoned(pte_t pte);
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);

 #else /* !CONFIG_HUGETLB_PAGE */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index b7b56871029c..db39442befcb 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -305,4 +305,60 @@ typedef int __bitwise __kernel_rwf_t;
 #define RWF_SUPPORTED	(RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
 			 RWF_APPEND)

+/* Pagemap ioctl */
+#define PAGEMAP_SCAN	_IOWR('f', 16, struct pm_scan_arg)
+
+/* Bits are set in flags of the page_region and masks in pm_scan_args */
+#define PAGE_IS_WPASYNC		(1 << 0)
+#define PAGE_IS_WRITTEN		(1 << 1)
+#define PAGE_IS_FILE		(1 << 2)
+#define PAGE_IS_PRESENT		(1 << 3)
+#define PAGE_IS_SWAPPED		(1 << 4)
+#define PAGE_IS_PFNZERO		(1 << 5)
+
+/*
+ * struct page_region - Page region with flags
+ * @start:	Start of the region
+ * @end:	End of the region (exclusive)
+ * @categories:	PAGE_IS_* category bitmask for the region
+ */
+struct page_region {
+	__u64 start;
+	__u64 end;
+	__u64 categories;
+};
+
+/* Flags for PAGEMAP_SCAN ioctl */
+#define PM_SCAN_WP_MATCHING	(1 << 0)	/* Write protect the pages matched. */
+#define PM_SCAN_CHECK_WPASYNC	(1 << 1)	/* Abort the scan when a
non-WP-enabled page is found. */
+
+/*
+ * struct pm_scan_arg - Pagemap ioctl argument
+ * @size:		Size of the structure
+ * @flags:		Flags for the IOCTL
+ * @start:		Starting address of the region
+ *			(Ending address of the walk is also returned in it)
+ * @end:		Ending address of the region
+ * @vec:		Address of page_region struct array for output
+ * @vec_len:		Length of the page_region struct array
+ * @max_pages:		Optional limit for number of returned pages (0 = disabled)
+ * @category_inverted:	PAGE_IS_* categories which values match if 0
instead of 1
+ * @category_mask:	Skip pages for which any category doesn't match
+ * @category_anyof_mask: Skip pages for which no category matches
+ * @return_mask:	PAGE_IS_* categories that are to be reported in
`page_region`s returned
+ */
+struct pm_scan_arg {
+	__u64 size;
+	__u64 flags;
+	__u64 start;
+	__u64 end;
+	__u64 vec;
+	__u64 vec_len;
+	__u64 max_pages;
+	__u64 category_inverted;
+	__u64 category_mask;
+	__u64 category_anyof_mask;
+	__u64 return_mask;
+};
+
 #endif /* _UAPI_LINUX_FS_H */
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index a073e6ed8900..3b07db0a4f2d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5008,7 +5008,7 @@ bool is_hugetlb_entry_migration(pte_t pte)
 		return false;
 }

-static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
+bool is_hugetlb_entry_hwpoisoned(pte_t pte)
 {
 	swp_entry_t swp;

-- 
2.39.2

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Tue, 25 Jul 2023 at 11:11, Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> ----
> Michal please post your thoughts before I post this as v26.
> ----
[...]

Looks ok - minor things below.

1. I'd change the _WPASYNC things to something better, if this can
also work with "normal" UFFD WP.

2. For the address tagging part I'd prefer someone who knows how this
is used take a look. We're ignoring the tag (but clear it on return in
->start) - so it doesn't matter for the ioctl() itself.

3. BTW, One of the uses is the GetWriteWatch and I wonder how it
behaves on HugeTLB (MEM_LARGE_PAGES allocation)? Shouldn't it return a
list of huge pages and write *lpdwGranularity = HPAGE_SIZE?

4. The docs and commit messages need some rewording due to the changes
in the API.

Other than that:

Reviewed-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/25/23 11:05 PM, Michał Mirosław wrote:
> On Tue, 25 Jul 2023 at 11:11, Muhammad Usama Anjum
> <usama.anjum@collabora.com> wrote:
>>
>> ----
>> Michal please post your thoughts before I post this as v26.
>> ----
> [...]
> 
> Looks ok - minor things below.
> 
> 1. I'd change the _WPASYNC things to something better, if this can
> also work with "normal" UFFD WP.
Yeah, but we don't have any use case where UFFD WP is required. It can be
easily added later when user case arrives. Also UFFD WP sends messages to
userspace. User can easily do the bookkeeping in userspace as performance
isn't a concern there.

> 
> 2. For the address tagging part I'd prefer someone who knows how this
> is used take a look. We're ignoring the tag (but clear it on return in
> ->start) - so it doesn't matter for the ioctl() itself.
I've added Kirill if he can give his thoughts about tagged memory.

Right now we are removing the tags from all 3 pointers (start, end, vec)
before using the pointers on kernel side. But we are overwriting and
writing the walk ending address in start which user can read/use.

I think we shouldn't over-write the start (and its tag) and instead return
the ending walk address in new variable, walk_end.

> 
> 3. BTW, One of the uses is the GetWriteWatch and I wonder how it
> behaves on HugeTLB (MEM_LARGE_PAGES allocation)? Shouldn't it return a
> list of huge pages and write *lpdwGranularity = HPAGE_SIZE?
Wine/Proton doesn't used hugetlb by default. Hugetlb isn't enabled by
default on Debian as well. For GetWriteWatch() we don't care about the
hugetlb at all. We have added hugetlb's implementation to complete the
feature and leave out something.

Also GetWriteWatch() implementation wouldn't require THP support as well
because you start to get 2MB of memory dirty even when only 4kB of memory
shouldn't have been dirty.

> 
> 4. The docs and commit messages need some rewording due to the changes
> in the API.
Yeah, I've updated the doc. I'll update the commit message as well.

> 
> Other than that:
> 
> Reviewed-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>

-- 
BR,
Muhammad Usama Anjum

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Wed, 26 Jul 2023 at 10:34, Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
> On 7/25/23 11:05 PM, Michał Mirosław wrote:
> > On Tue, 25 Jul 2023 at 11:11, Muhammad Usama Anjum
> > <usama.anjum@collabora.com> wrote:
> >>
> >> ----
> >> Michal please post your thoughts before I post this as v26.
> >> ----
> > [...]
> >
> > Looks ok - minor things below.
> >
> > 1. I'd change the _WPASYNC things to something better, if this can
> > also work with "normal" UFFD WP.
> Yeah, but we don't have any use case where UFFD WP is required. It can be
> easily added later when user case arrives. Also UFFD WP sends messages to
> userspace. User can easily do the bookkeeping in userspace as performance
> isn't a concern there.

We shouldn't name the flags based on the use case but based on what
they actually do. So if this checks UFFD registration for WP, then
maybe PAGE_IS_WPALLOWED or something better describing the trait it
matches?

> > 2. For the address tagging part I'd prefer someone who knows how this
> > is used take a look. We're ignoring the tag (but clear it on return in
> > ->start) - so it doesn't matter for the ioctl() itself.
> I've added Kirill if he can give his thoughts about tagged memory.
>
> Right now we are removing the tags from all 3 pointers (start, end, vec)
> before using the pointers on kernel side. But we are overwriting and
> writing the walk ending address in start which user can read/use.
>
> I think we shouldn't over-write the start (and its tag) and instead return
> the ending walk address in new variable, walk_end.

The overwrite of `start` is making the ioctl restart (continuation)
easier to handle. I prefer the current way, but it's not a strong
opinion.

> > 3. BTW, One of the uses is the GetWriteWatch and I wonder how it
> > behaves on HugeTLB (MEM_LARGE_PAGES allocation)? Shouldn't it return a
> > list of huge pages and write *lpdwGranularity = HPAGE_SIZE?
> Wine/Proton doesn't used hugetlb by default. Hugetlb isn't enabled by
> default on Debian as well. For GetWriteWatch() we don't care about the
> hugetlb at all. We have added hugetlb's implementation to complete the
> feature and leave out something.

How is GetWriteWatch() working when passed a VirtualAlloc(...,
MEM_LARGE_PAGES|MEM_WRITE_WATCH...)-allocated range? Does it still
report 4K pages?
This is only a problem when using max_pages: a hugetlb range might
need counting and reporting huge pages and not 4K parts.

Best Regards
Michał Mirosław

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Paul Gofman]

Hello Michał,

     I was looking into that from the Wine point of view and did a bit 
of testing, so will try to answer the question cited below.

     Without Windows large pages I guess the only way to make this work 
correctly is to disable THP with madvise(MADV_NOHUGEPAGE) on the memory 
ranges allocated with MEM_WRITE_WATCH, as the memory changes should not 
only be reported but also tracked with 4k page granularity as Windows 
applications expect.

     Currently we don't implement MEM_LARGE_PAGES flag support in Wine 
(while of course might want to do that in the future). On Windows using 
this flag requires special permissions and implies more than just using 
huge pages under the hood but also, in particular, locking pages in 
memory. I'd expect that support to be extended in Windows though in the 
future in some way. WRT write watches, the range is watched with large 
page granularity. GetWriteWatch lpdwGranularity output parameter returns 
the value of "large page minimum" (returned by GetLargePageMinimum) and 
the returned addresses correspond to those large pages. I suppose to 
implement that on top of Linux huge pages we'd need a way to control 
huge pages allocation at the first place, i. e., a way to enforce the 
specified size for the huge pages for the memory ranged being mapped. 
Without that I am afraid the only way to correctly implement that is to 
still disable THP on the range and only adjust our API output so that 
matches expected.

     Not related to the question, but without any relation to Wine and 
Windows API current way of dealing with THP in the API design looks a 
bit not straightforward to me. In a sense that transparent huge pages 
will appear not so transparent when it comes to dirty pages tracking. If 
I understand correctly, the application which allocated a reasonably big 
memory area and didn't use madvise(MADV_NOHUGEPAGE) might end up with a 
whole range being a single page and getting dirtified as a whole, which 
may likely void app's optimization based on changed memory tracking. Not 
that I know an ideal way out of this, maybe it is a matter of having THP 
disabled by default on watched ranges or clearly warning about this 
caveat in documentation?

Regards,
     Paul.


On 7/26/23 15:10, Michał Mirosław wrote:
>
>>> 3. BTW, One of the uses is the GetWriteWatch and I wonder how it
>>> behaves on HugeTLB (MEM_LARGE_PAGES allocation)? Shouldn't it return a
>>> list of huge pages and write *lpdwGranularity = HPAGE_SIZE?
>> Wine/Proton doesn't used hugetlb by default. Hugetlb isn't enabled by
>> default on Debian as well. For GetWriteWatch() we don't care about the
>> hugetlb at all. We have added hugetlb's implementation to complete the
>> feature and leave out something.
> How is GetWriteWatch() working when passed a VirtualAlloc(...,
> MEM_LARGE_PAGES|MEM_WRITE_WATCH...)-allocated range? Does it still
> report 4K pages?
> This is only a problem when using max_pages: a hugetlb range might
> need counting and reporting huge pages and not 4K parts.
>
> Best Regards
> Michał Mirosław

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/27/23 2:10 AM, Michał Mirosław wrote:
> On Wed, 26 Jul 2023 at 10:34, Muhammad Usama Anjum
> <usama.anjum@collabora.com> wrote:
>> On 7/25/23 11:05 PM, Michał Mirosław wrote:
>>> On Tue, 25 Jul 2023 at 11:11, Muhammad Usama Anjum
>>> <usama.anjum@collabora.com> wrote:
>>>>
>>>> ----
>>>> Michal please post your thoughts before I post this as v26.
>>>> ----
>>> [...]
>>>
>>> Looks ok - minor things below.
>>>
>>> 1. I'd change the _WPASYNC things to something better, if this can
>>> also work with "normal" UFFD WP.
>> Yeah, but we don't have any use case where UFFD WP is required. It can be
>> easily added later when user case arrives. Also UFFD WP sends messages to
>> userspace. User can easily do the bookkeeping in userspace as performance
>> isn't a concern there.
> 
> We shouldn't name the flags based on the use case but based on what
> they actually do. So if this checks UFFD registration for WP, then
> maybe PAGE_IS_WPALLOWED or something better describing the trait it
> matches?
PAGE_IS_WPALLOWED seems appropriate.

> 
>>> 2. For the address tagging part I'd prefer someone who knows how this
>>> is used take a look. We're ignoring the tag (but clear it on return in
>>> ->start) - so it doesn't matter for the ioctl() itself.
>> I've added Kirill if he can give his thoughts about tagged memory.
>>
>> Right now we are removing the tags from all 3 pointers (start, end, vec)
>> before using the pointers on kernel side. But we are overwriting and
>> writing the walk ending address in start which user can read/use.
>>
>> I think we shouldn't over-write the start (and its tag) and instead return
>> the ending walk address in new variable, walk_end.
> 
> The overwrite of `start` is making the ioctl restart (continuation)
> easier to handle. I prefer the current way, but it's not a strong
> opinion.
We shouldn't overwrite the start if we aren't gonna put the correct tag. So
I've resorted to adding another variable `walk_end` to return the walk
ending address.

> 
>>> 3. BTW, One of the uses is the GetWriteWatch and I wonder how it
>>> behaves on HugeTLB (MEM_LARGE_PAGES allocation)? Shouldn't it return a
>>> list of huge pages and write *lpdwGranularity = HPAGE_SIZE?
>> Wine/Proton doesn't used hugetlb by default. Hugetlb isn't enabled by
>> default on Debian as well. For GetWriteWatch() we don't care about the
>> hugetlb at all. We have added hugetlb's implementation to complete the
>> feature and leave out something.
> 
> How is GetWriteWatch() working when passed a VirtualAlloc(...,
> MEM_LARGE_PAGES|MEM_WRITE_WATCH...)-allocated range? Does it still
> report 4K pages?
> This is only a problem when using max_pages: a hugetlb range might
> need counting and reporting huge pages and not 4K parts.
> 
> Best Regards
> Michał Mirosław

I'll send v26 in next hour.

-- 
BR,
Muhammad Usama Anjum

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Thu, 27 Jul 2023 at 01:06, Paul Gofman <pgofman@codeweavers.com> wrote:
>
> Hello Michał,
>
>      I was looking into that from the Wine point of view and did a bit
> of testing, so will try to answer the question cited below.

Thanks for the extensive explanation!

>      Without Windows large pages I guess the only way to make this work
> correctly is to disable THP with madvise(MADV_NOHUGEPAGE) on the memory
> ranges allocated with MEM_WRITE_WATCH, as the memory changes should not
> only be reported but also tracked with 4k page granularity as Windows
> applications expect.
>
>      Currently we don't implement MEM_LARGE_PAGES flag support in Wine
> (while of course might want to do that in the future). On Windows using
> this flag requires special permissions and implies more than just using
> huge pages under the hood but also, in particular, locking pages in
> memory. I'd expect that support to be extended in Windows though in the
> future in some way. WRT write watches, the range is watched with large
> page granularity. GetWriteWatch lpdwGranularity output parameter returns
> the value of "large page minimum" (returned by GetLargePageMinimum) and
> the returned addresses correspond to those large pages. I suppose to
> implement that on top of Linux huge pages we'd need a way to control
> huge pages allocation at the first place, i. e., a way to enforce the
> specified size for the huge pages for the memory ranged being mapped.
> Without that I am afraid the only way to correctly implement that is to
> still disable THP on the range and only adjust our API output so that
> matches expected.
>
>      Not related to the question, but without any relation to Wine and
> Windows API current way of dealing with THP in the API design looks a
> bit not straightforward to me. In a sense that transparent huge pages
> will appear not so transparent when it comes to dirty pages tracking. If
> I understand correctly, the application which allocated a reasonably big
> memory area and didn't use madvise(MADV_NOHUGEPAGE) might end up with a
> whole range being a single page and getting dirtified as a whole, which
> may likely void app's optimization based on changed memory tracking. Not
> that I know an ideal way out of this, maybe it is a matter of having THP
> disabled by default on watched ranges or clearly warning about this
> caveat in documentation?

So, this means that the max_pages limit should count HugeTLB pages as
1 not HPAGE_SIZE/PAGE_SIZE pages.
Also, to get this right, we might need another PAGE_IS_HUGETLB
category, so that userspace can differentiate the ranges if needed.

Is it possible (on Windows) to have MEM_LARGE_PAGES allocation near a
normal one and run GetWriteWatch() on both in one call? If so, how
does it behave / what is expected?

Best Regards
Michał Mirosław

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Thu, 27 Jul 2023 at 13:18, Michał Mirosław <emmir@google.com> wrote:
> On Thu, 27 Jul 2023 at 01:06, Paul Gofman <pgofman@codeweavers.com> wrote:
> > Hello Michał,
> >
> >      I was looking into that from the Wine point of view and did a bit
> > of testing, so will try to answer the question cited below.
>
> Thanks for the extensive explanation!
>
> >      Without Windows large pages I guess the only way to make this work
> > correctly is to disable THP with madvise(MADV_NOHUGEPAGE) on the memory
> > ranges allocated with MEM_WRITE_WATCH, as the memory changes should not
> > only be reported but also tracked with 4k page granularity as Windows
> > applications expect.
> >
> >      Currently we don't implement MEM_LARGE_PAGES flag support in Wine
> > (while of course might want to do that in the future). On Windows using
> > this flag requires special permissions and implies more than just using
> > huge pages under the hood but also, in particular, locking pages in
> > memory. I'd expect that support to be extended in Windows though in the
> > future in some way. WRT write watches, the range is watched with large
> > page granularity. GetWriteWatch lpdwGranularity output parameter returns
> > the value of "large page minimum" (returned by GetLargePageMinimum) and
> > the returned addresses correspond to those large pages. I suppose to
> > implement that on top of Linux huge pages we'd need a way to control
> > huge pages allocation at the first place, i. e., a way to enforce the
> > specified size for the huge pages for the memory ranged being mapped.
> > Without that I am afraid the only way to correctly implement that is to
> > still disable THP on the range and only adjust our API output so that
> > matches expected.
[...]

The THP case we can leave it to userspace, as it can madvise() if the
optimization lost due to dirty tracking of THP is more than THP adds.
(BTW, the WP-watch is done via uffd - the default change would need to
happen there).

Best Regards
> Michał Mirosław

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Michał Mirosław]

On Thu, 27 Jul 2023 at 10:03, Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
> On 7/27/23 2:10 AM, Michał Mirosław wrote:
> > On Wed, 26 Jul 2023 at 10:34, Muhammad Usama Anjum
> > <usama.anjum@collabora.com> wrote:
> >> On 7/25/23 11:05 PM, Michał Mirosław wrote:
> >>> On Tue, 25 Jul 2023 at 11:11, Muhammad Usama Anjum
> >>> <usama.anjum@collabora.com> wrote:
[...]
> >>> 2. For the address tagging part I'd prefer someone who knows how this
> >>> is used take a look. We're ignoring the tag (but clear it on return in
> >>> ->start) - so it doesn't matter for the ioctl() itself.
> >> I've added Kirill if he can give his thoughts about tagged memory.
> >>
> >> Right now we are removing the tags from all 3 pointers (start, end, vec)
> >> before using the pointers on kernel side. But we are overwriting and
> >> writing the walk ending address in start which user can read/use.
> >>
> >> I think we shouldn't over-write the start (and its tag) and instead return
> >> the ending walk address in new variable, walk_end.
> >
> > The overwrite of `start` is making the ioctl restart (continuation)
> > easier to handle. I prefer the current way, but it's not a strong
> > opinion.
> We shouldn't overwrite the start if we aren't gonna put the correct tag. So
> I've resorted to adding another variable `walk_end` to return the walk
> ending address.

Yes. We have two options:

1. add new field and have the userspace check it and update start
itself to continue the scan,
 or:
2. reconstruct the tag from either orignal `start` or `end` and have
the userspace re-set `start` if it wants to restart the scan instead
of continuing.

(the second one, using `end`'s tag, might be the easiest for
userspace, as it can check `start` == `end` when deciding to continue
or restart).

Best Regards
Michał Mirosław

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Muhammad Usama Anjum]

On 7/27/23 4:26 PM, Michał Mirosław wrote:
> On Thu, 27 Jul 2023 at 10:03, Muhammad Usama Anjum
> <usama.anjum@collabora.com> wrote:
>> On 7/27/23 2:10 AM, Michał Mirosław wrote:
>>> On Wed, 26 Jul 2023 at 10:34, Muhammad Usama Anjum
>>> <usama.anjum@collabora.com> wrote:
>>>> On 7/25/23 11:05 PM, Michał Mirosław wrote:
>>>>> On Tue, 25 Jul 2023 at 11:11, Muhammad Usama Anjum
>>>>> <usama.anjum@collabora.com> wrote:
> [...]
>>>>> 2. For the address tagging part I'd prefer someone who knows how this
>>>>> is used take a look. We're ignoring the tag (but clear it on return in
>>>>> ->start) - so it doesn't matter for the ioctl() itself.
>>>> I've added Kirill if he can give his thoughts about tagged memory.
>>>>
>>>> Right now we are removing the tags from all 3 pointers (start, end, vec)
>>>> before using the pointers on kernel side. But we are overwriting and
>>>> writing the walk ending address in start which user can read/use.
>>>>
>>>> I think we shouldn't over-write the start (and its tag) and instead return
>>>> the ending walk address in new variable, walk_end.
>>>
>>> The overwrite of `start` is making the ioctl restart (continuation)
>>> easier to handle. I prefer the current way, but it's not a strong
>>> opinion.
>> We shouldn't overwrite the start if we aren't gonna put the correct tag. So
>> I've resorted to adding another variable `walk_end` to return the walk
>> ending address.
> 
> Yes. We have two options:
> 
> 1. add new field and have the userspace check it and update start
> itself to continue the scan,
I've selected this option and sent v26 already:
https://lore.kernel.org/all/20230727093637.1262110-1-usama.anjum@collabora.com

>  or:
> 2. reconstruct the tag from either orignal `start` or `end` and have
> the userspace re-set `start` if it wants to restart the scan instead
> of continuing.
In some case, compiler can put integrity checking metadata in the pointer's
upper byte. So copying start or end's meta data would be wrong.

> 
> (the second one, using `end`'s tag, might be the easiest for
> userspace, as it can check `start` == `end` when deciding to continue
> or restart).
> 
> Best Regards
> Michał Mirosław

-- 
BR,
Muhammad Usama Anjum

Re: [v3] fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[Paul Gofman]

On Windows GetWriteWatch won't work if the range is not withing a single 
memory view regardless of MEM_LARGE_PAGES (that is, single VirtualAlloc 
result, somewhat analogous to VMA but not quite), such a call will 
result in an error.

For the context, Windows never auto splits or joins memory views, and 
basically nothing can be done across views boundary (merging placeholder 
views is an exception, while unrelated). VirtualProtect (which stands 
for mprotect) can be done for individual pages withing the view of 
course, but it is purely per page and doesn't affect view protection 
flags or result in any splitting of those. VirtualProtect across views 
adjacent in memory will also fail.

On 7/27/23 05:18, Michał Mirosław wrote:
>
> Is it possible (on Windows) to have MEM_LARGE_PAGES allocation near a
> normal one and run GetWriteWatch() on both in one call? If so, how
> does it behave / what is expected?
>
> Best Regards
> Michał Mirosław

Re: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning

[kernel test robot]

Hi Michał,

kernel test robot noticed the following build warnings:

[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master v6.5-rc7 next-20230825]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Micha-Miros-aw/Re-fs-proc-task_mmu-Implement-IOCTL-for-efficient-page-table-scanning/20230721-033050
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/a0b5c6776b2ed91f78a7575649f8b100e58bd3a9.1689881078.git.mirq-linux%40rere.qmqm.pl
patch subject: Re: fs/proc/task_mmu: Implement IOCTL for efficient page table scanning
config: i386-randconfig-i004-20230720 (https://download.01.org/0day-ci/archive/20230826/202308262125.VHTuZ7uV-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce: (https://download.01.org/0day-ci/archive/20230826/202308262125.VHTuZ7uV-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202308262125.VHTuZ7uV-lkp@intel.com/

All warnings (new ones prefixed by >>):

   fs/proc/task_mmu.c: In function 'pagemap_scan_test_walk':
   fs/proc/task_mmu.c:1921:13: error: implicit declaration of function 'userfaultfd_wp_async'; did you mean 'userfaultfd_wp'? [-Werror=implicit-function-declaration]
    1921 |         if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
         |             ^~~~~~~~~~~~~~~~~~~~
         |             userfaultfd_wp
   fs/proc/task_mmu.c: In function 'pagemap_scan_init_bounce_buffer':
   fs/proc/task_mmu.c:2290:22: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    2290 |         p->vec_out = (void __user *)p->arg.vec;
         |                      ^
   fs/proc/task_mmu.c: At top level:
>> fs/proc/task_mmu.c:1967:13: warning: 'pagemap_scan_backout_range' defined but not used [-Wunused-function]
    1967 | static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
         |             ^~~~~~~~~~~~~~~~~~~~~~~~~~
   cc1: some warnings being treated as errors


vim +/pagemap_scan_backout_range +1967 fs/proc/task_mmu.c

  1966	
> 1967	static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
  1968					       unsigned long addr, unsigned long end)
  1969	{
  1970		struct page_region *cur_buf = &p->cur_buf;
  1971	
  1972		if (cur_buf->start != addr) {
  1973			cur_buf->end = addr;
  1974		} else {
  1975			cur_buf->start = cur_buf->end = 0;
  1976		}
  1977	
  1978		p->end_addr = 0;
  1979	}
  1980	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki