* [PATCH v8 0/4] Implement IOCTL to get and/or the clear info about PTEs
@ 2023-01-24 8:43 Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
` (3 more replies)
0 siblings, 4 replies; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-24 8:43 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
*Changes in v8:*
- Update uffd async wp implementation
- Improve PAGEMAP_IOCTL implementation
*Changes in v7:*
- Add uffd wp async
- Update the IOCTL to use uffd under the hood instead of soft-dirty
flags
Hello,
Note:
Soft-dirty pages and pages which have been written-to are synonyms. As
kernel already has soft-dirty feature inside which we have given up to
use, we are using written-to terminology while using UFFD async WP under
the hood.
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are
supported in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WT),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
pages have been written-to.
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WT + PAGEMAP_WP_ENGAGE)
It is possible to find and clear soft-dirty pages entirely in userspace.
But it isn't efficient:
- The mprotect and SIGSEGV handler for bookkeeping
- The userfaultfd wp with the handler for bookkeeping
Some benchmarks can be seen here[1]. This series adds features that weren't
present earlier:
- There is no atomic get soft-dirty PTE bit status and clear present in
the kernel.
- The pages which have been written-to can not be found in accurate way.
(Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty
pages than there actually are.)
Historically, soft-dirty PTE bit tracking has been used in the CRIU
project. The procfs interface is enough for finding the soft-dirty bit
status and clearing the soft-dirty bit of all the pages of a process.
We have the use case where we need to track the soft-dirty PTE bit for
only specific pages on demand. We need this tracking and clear mechanism
of a region of memory while the process is running to emulate the
getWriteWatch() syscall of Windows.
*(Moved to using UFFD instead of soft-dirty to find pages which have been
written-to from v7 patch series)*:
Stop using the soft-dirty flags for finding which pages have been
written to. It is too delicate and wrong as it shows more soft-dirty
pages than the actual soft-dirty pages. There is no interest in
correcting it [2][3] as this is how the feature was written years ago.
It shouldn't be updated to changed behaviour. Peter Xu has suggested
using the async version of the UFFD WP [4] as it is based inherently
on the PTEs.
So in this patch series, I've added a new mode to the UFFD which is
asynchronous version of the write protect. When this variant of the
UFFD WP is used, the page faults are resolved automatically by the
kernel. The pages which have been written-to can be found by reading
pagemap file (!PM_UFFD_WP). This feature can be used successfully to
find which pages have been written to from the time the pages were
write protected. This works just like the soft-dirty flag without
showing any extra pages which aren't soft-dirty in reality.
The information related to pages if the page is file mapped, present and
swapped is required for the CRIU project [5][6]. The addition of the
required mask, any mask, excluded mask and return masks are also required
for the CRIU project [5].
The IOCTL returns the addresses of the pages which match the specific masks.
The page addresses are returned in struct page_region in a compact form.
The max_pages is needed to support a use case where user only wants to get
a specific number of pages. So there is no need to find all the pages of
interest in the range when max_pages is specified. The IOCTL returns when
the maximum number of the pages are found. The max_pages is optional. If
max_pages is specified, it must be equal or greater than the vec_size.
This restriction is needed to handle worse case when one page_region only
contains info of one page and it cannot be compacted. This is needed to
emulate the Windows getWriteWatch() syscall.
The patch series include the detailed selftest which can be used as an example
for the uffd async wp test and PAGEMAP_IOCTL. It shows the interface usages as
well.
[1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/
[2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
[3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
[4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n
[5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/
[6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/
Regards,
Muhammad Usama Anjum
Muhammad Usama Anjum (4):
userfaultfd: Add UFFD WP Async support
userfaultfd: split mwriteprotect_range()
fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about
PTEs
selftests: vm: add pagemap ioctl tests
fs/proc/task_mmu.c | 294 +++++++
fs/userfaultfd.c | 21 +
include/linux/userfaultfd_k.h | 16 +
include/uapi/linux/fs.h | 50 ++
include/uapi/linux/userfaultfd.h | 8 +-
mm/memory.c | 29 +-
mm/userfaultfd.c | 40 +-
tools/include/uapi/linux/fs.h | 50 ++
tools/testing/selftests/vm/.gitignore | 1 +
tools/testing/selftests/vm/Makefile | 5 +-
tools/testing/selftests/vm/pagemap_ioctl.c | 880 +++++++++++++++++++++
11 files changed, 1374 insertions(+), 20 deletions(-)
create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c
--
2.30.2
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-24 8:43 [PATCH v8 0/4] Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
@ 2023-01-24 8:43 ` Muhammad Usama Anjum
2023-01-24 10:43 ` kernel test robot
` (3 more replies)
2023-01-24 8:43 ` [PATCH v8 2/4] userfaultfd: split mwriteprotect_range() Muhammad Usama Anjum
` (2 subsequent siblings)
3 siblings, 4 replies; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-24 8:43 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
Add new WP Async mode (UFFD_FEATURE_WP_ASYNC) which resolves the page
faults on its own. It can be used to track that which pages have been
written-to from the time the pages were write-protected. It is very
efficient way to track the changes as uffd is by nature pte/pmd based.
UFFD synchronous WP sends the page faults to the userspace where the
pages which have been written-to can be tracked. But it is not efficient.
This is why this asynchronous version is being added. After setting the
WP Async, the pages which have been written to can be found in the pagemap
file or information can be obtained from the PAGEMAP_IOCTL.
Suggested-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v7:
- Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC
- Handle automatic page fault resolution in better way (thanks to Peter)
---
fs/userfaultfd.c | 11 +++++++++++
include/linux/userfaultfd_k.h | 6 ++++++
include/uapi/linux/userfaultfd.h | 8 +++++++-
mm/memory.c | 29 +++++++++++++++++++++++++++--
4 files changed, 51 insertions(+), 3 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 15a5bf765d43..b82af02092ce 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -1867,6 +1867,10 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
+ /* Write protection cannot be disabled in case of aync WP */
+ if (!mode_wp && (ctx->features & UFFD_FEATURE_WP_ASYNC))
+ return -EINVAL;
+
if (mode_wp && mode_dontwake)
return -EINVAL;
@@ -1950,6 +1954,13 @@ static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
return ret;
}
+int userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+ struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
+
+ return (ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC));
+}
+
static inline unsigned int uffd_ctx_features(__u64 user_features)
{
/*
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index 9df0b9a762cc..5db51fccae1d 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -179,6 +179,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
unsigned long end, struct list_head *uf);
extern void userfaultfd_unmap_complete(struct mm_struct *mm,
struct list_head *uf);
+extern int userfaultfd_wp_async(struct vm_area_struct *vma);
#else /* CONFIG_USERFAULTFD */
@@ -274,6 +275,11 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
return false;
}
+int userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+ return false;
+}
+
#endif /* CONFIG_USERFAULTFD */
static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
index 005e5e306266..f4252ef40071 100644
--- a/include/uapi/linux/userfaultfd.h
+++ b/include/uapi/linux/userfaultfd.h
@@ -38,7 +38,8 @@
UFFD_FEATURE_MINOR_HUGETLBFS | \
UFFD_FEATURE_MINOR_SHMEM | \
UFFD_FEATURE_EXACT_ADDRESS | \
- UFFD_FEATURE_WP_HUGETLBFS_SHMEM)
+ UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
+ UFFD_FEATURE_WP_ASYNC)
#define UFFD_API_IOCTLS \
((__u64)1 << _UFFDIO_REGISTER | \
(__u64)1 << _UFFDIO_UNREGISTER | \
@@ -203,6 +204,10 @@ struct uffdio_api {
*
* UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd
* write-protection mode is supported on both shmem and hugetlbfs.
+ *
+ * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
+ * asynchronous mode is supported in which the write fault is automatically
+ * resolved and write-protection is un-set.
*/
#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
#define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -217,6 +222,7 @@ struct uffdio_api {
#define UFFD_FEATURE_MINOR_SHMEM (1<<10)
#define UFFD_FEATURE_EXACT_ADDRESS (1<<11)
#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
+#define UFFD_FEATURE_WP_ASYNC (1<<13)
__u64 features;
__u64 ioctls;
diff --git a/mm/memory.c b/mm/memory.c
index 4000e9f017e0..8c03b133d483 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
if (likely(!unshare)) {
if (userfaultfd_pte_wp(vma, *vmf->pte)) {
+ if (userfaultfd_wp_async(vma)) {
+ /*
+ * Nothing needed (cache flush, TLB invalidations,
+ * etc.) because we're only removing the uffd-wp bit,
+ * which is completely invisible to the user. This
+ * falls through to possible CoW.
+ */
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
+ pte_clear_uffd_wp(*vmf->pte));
+ return 0;
+ }
pte_unmap_unlock(vmf->pte, vmf->ptl);
return handle_userfault(vmf, VM_UFFD_WP);
}
@@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
if (vma_is_anonymous(vmf->vma)) {
if (likely(!unshare) &&
- userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
- return handle_userfault(vmf, VM_UFFD_WP);
+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
+ if (userfaultfd_wp_async(vmf->vma)) {
+ /*
+ * Nothing needed (cache flush, TLB invalidations,
+ * etc.) because we're only removing the uffd-wp bit,
+ * which is completely invisible to the user. This
+ * falls through to possible CoW.
+ */
+ set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
+ pmd_clear_uffd_wp(*vmf->pmd));
+ return 0;
+ } else {
+ return handle_userfault(vmf, VM_UFFD_WP);
+ }
+ }
return do_huge_pmd_wp_page(vmf);
}
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v8 2/4] userfaultfd: split mwriteprotect_range()
2023-01-24 8:43 [PATCH v8 0/4] Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
@ 2023-01-24 8:43 ` Muhammad Usama Anjum
2023-01-24 10:23 ` kernel test robot
2023-01-27 17:05 ` Peter Xu
2023-01-24 8:43 ` [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 4/4] selftests: vm: add pagemap ioctl tests Muhammad Usama Anjum
3 siblings, 2 replies; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-24 8:43 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
Split mwriteprotect_range() to create a unlocked version. This
will be used in the next patch to write protect a memory area.
Add a helper function, wp_range_async() as well.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v7:
- Remove async being set in the PAGEMAP_IOCTL
---
fs/userfaultfd.c | 10 +++++++++
include/linux/userfaultfd_k.h | 10 +++++++++
mm/userfaultfd.c | 40 ++++++++++++++++++++++-------------
3 files changed, 45 insertions(+), 15 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index b82af02092ce..cd52c9ee8db9 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -1961,6 +1961,16 @@ int userfaultfd_wp_async(struct vm_area_struct *vma)
return (ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC));
}
+int wp_range_async(struct vm_area_struct *vma, unsigned long start, unsigned long len)
+{
+ struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
+
+ if (!ctx)
+ return -1;
+
+ return __mwriteprotect_range(ctx->mm, start, len, true, &ctx->mmap_changing);
+}
+
static inline unsigned int uffd_ctx_features(__u64 user_features)
{
/*
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index 5db51fccae1d..40d1d9e8f34d 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -73,6 +73,9 @@ extern ssize_t mcopy_continue(struct mm_struct *dst_mm, unsigned long dst_start,
extern int mwriteprotect_range(struct mm_struct *dst_mm,
unsigned long start, unsigned long len,
bool enable_wp, atomic_t *mmap_changing);
+extern int __mwriteprotect_range(struct mm_struct *dst_mm,
+ unsigned long start, unsigned long len,
+ bool enable_wp, atomic_t *mmap_changing);
extern void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *vma,
unsigned long start, unsigned long len, bool enable_wp);
@@ -180,6 +183,8 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
extern void userfaultfd_unmap_complete(struct mm_struct *mm,
struct list_head *uf);
extern int userfaultfd_wp_async(struct vm_area_struct *vma);
+extern int wp_range_async(struct vm_area_struct *vma, unsigned long start,
+ unsigned long len);
#else /* CONFIG_USERFAULTFD */
@@ -280,6 +285,11 @@ int userfaultfd_wp_async(struct vm_area_struct *vma)
return false;
}
+int wp_range_async(struct vm_area_struct *vma, unsigned long start, unsigned long len)
+{
+ return -1;
+}
+
#endif /* CONFIG_USERFAULTFD */
static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 65ad172add27..9d8a43faf764 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -734,25 +734,13 @@ void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *dst_vma,
tlb_finish_mmu(&tlb);
}
-int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
- unsigned long len, bool enable_wp,
- atomic_t *mmap_changing)
+int __mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
+ unsigned long len, bool enable_wp,
+ atomic_t *mmap_changing)
{
struct vm_area_struct *dst_vma;
unsigned long page_mask;
int err;
-
- /*
- * Sanitize the command parameters:
- */
- BUG_ON(start & ~PAGE_MASK);
- BUG_ON(len & ~PAGE_MASK);
-
- /* Does the address range wrap, or is the span zero-sized? */
- BUG_ON(start + len <= start);
-
- mmap_read_lock(dst_mm);
-
/*
* If memory mappings are changing because of non-cooperative
* operation (e.g. mremap) running in parallel, bail out and
@@ -783,6 +771,28 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
err = 0;
out_unlock:
+ return err;
+}
+
+int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
+ unsigned long len, bool enable_wp,
+ atomic_t *mmap_changing)
+{
+ int err;
+
+ /*
+ * Sanitize the command parameters:
+ */
+ BUG_ON(start & ~PAGE_MASK);
+ BUG_ON(len & ~PAGE_MASK);
+
+ /* Does the address range wrap, or is the span zero-sized? */
+ BUG_ON(start + len <= start);
+
+ mmap_read_lock(dst_mm);
+
+ err = __mwriteprotect_range(dst_mm, start, len, enable_wp, mmap_changing);
+
mmap_read_unlock(dst_mm);
return err;
}
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
2023-01-24 8:43 [PATCH v8 0/4] Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 2/4] userfaultfd: split mwriteprotect_range() Muhammad Usama Anjum
@ 2023-01-24 8:43 ` Muhammad Usama Anjum
2023-01-24 10:02 ` kernel test robot
2023-01-27 17:36 ` Peter Xu
2023-01-24 8:43 ` [PATCH v8 4/4] selftests: vm: add pagemap ioctl tests Muhammad Usama Anjum
3 siblings, 2 replies; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-24 8:43 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are supported
in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WT),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
pages have been written-to.
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WT + PAGEMAP_WP_ENGAGE)
The uffd should have been registered on the memory range before performing
any WP/WT (Write Protect/Writtern-To) related operations with the IOCTL.
struct pagemap_scan_args is used as the argument of the IOCTL. In this
struct:
- The range is specified through start and len.
- The output buffer of struct page_region array and size is specified as
vec and vec_len.
- The optional maximum requested pages are specified in the max_pages.
- The flags can be specified in the flags field. The PAGEMAP_WP_ENGAGE
is the only added flag at this time.
- The masks are specified in required_mask, anyof_mask, excluded_ mask
and return_mask.
This IOCTL can be extended to get information about more PTE bits. This
IOCTL doesn't support hugetlbs at the moment. No information about
hugetlb can be obtained. This patch has evolved from a basic patch from
Gabriel Krisman Bertazi.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code
Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
Changes in v6:
- Rename variables and update comments
- Make IOCTL independent of soft_dirty config
- Change masks and bitmap type to _u64
- Improve code quality
Changes in v5:
- Remove tlb flushing even for clear operation
Changes in v4:
- Update the interface and implementation
Changes in v3:
- Tighten the user-kernel interface by using explicit types and add more
error checking
Changes in v2:
- Convert the interface from syscall to ioctl
- Remove pidfd support as it doesn't make sense in ioctl
---
fs/proc/task_mmu.c | 294 ++++++++++++++++++++++++++++++++++
include/uapi/linux/fs.h | 50 ++++++
tools/include/uapi/linux/fs.h | 50 ++++++
3 files changed, 394 insertions(+)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index e35a0398db63..f9fbda00eb5c 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -19,6 +19,7 @@
#include <linux/shmem_fs.h>
#include <linux/uaccess.h>
#include <linux/pkeys.h>
+#include <linux/minmax.h>
#include <asm/elf.h>
#include <asm/tlb.h>
@@ -1135,6 +1136,22 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma,
}
#endif
+static inline bool is_pte_uffd_wp(pte_t pte)
+{
+ if ((pte_present(pte) && pte_uffd_wp(pte)) ||
+ (is_swap_pte(pte) && pte_swp_uffd_wp_any(pte)))
+ return true;
+ return false;
+}
+
+static inline bool is_pmd_uffd_wp(pmd_t pmd)
+{
+ if ((pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
+ (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)))
+ return true;
+ return false;
+}
+
#if defined(CONFIG_MEM_SOFT_DIRTY) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp)
@@ -1763,11 +1780,288 @@ static int pagemap_release(struct inode *inode, struct file *file)
return 0;
}
+#define PAGEMAP_OP_MASK (PAGE_IS_WT | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PAGEMAP_NONWT_OP_MASK (PAGE_IS_FILE | PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define IS_WP_ENGAGE_OP(a) (a->flags & PAGEMAP_WP_ENGAGE)
+#define IS_GET_OP(a) (a->vec)
+#define PAGEMAP_SCAN_BITMAP(wt, file, present, swap) \
+ (wt | file << 1 | present << 2 | swap << 3)
+#define IS_WT_REQUIRED(a) \
+ ((a->required_mask & PAGE_IS_WT) || \
+ (a->anyof_mask & PAGE_IS_WT))
+#define HAS_NO_SPACE(p) (p->max_pages && (p->found_pages == p->max_pages))
+
+struct pagemap_scan_private {
+ struct page_region *vec;
+ struct page_region prev;
+ unsigned long vec_len, vec_index;
+ unsigned int max_pages, found_pages, flags;
+ unsigned long required_mask, anyof_mask, excluded_mask, return_mask;
+};
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end, struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+
+ if (IS_WT_REQUIRED(p) && !userfaultfd_wp(vma) && !userfaultfd_wp_async(vma))
+ return -EPERM;
+ if (IS_GET_OP(p) && HAS_NO_SPACE(p))
+ return -ENOSPC;
+ if (vma->vm_flags & VM_PFNMAP)
+ return 1;
+ return 0;
+}
+
+static inline int add_to_out(bool wt, bool file, bool pres, bool swap,
+ struct pagemap_scan_private *p, unsigned long addr, unsigned int len)
+{
+ unsigned long bitmap, cur = PAGEMAP_SCAN_BITMAP(wt, file, pres, swap);
+ bool cpy = true;
+ struct page_region *prev = &p->prev;
+
+ if (HAS_NO_SPACE(p))
+ return -ENOSPC;
+
+ if (p->required_mask)
+ cpy = ((p->required_mask & cur) == p->required_mask);
+ if (cpy && p->anyof_mask)
+ cpy = (p->anyof_mask & cur);
+ if (cpy && p->excluded_mask)
+ cpy = !(p->excluded_mask & cur);
+ bitmap = cur & p->return_mask;
+ if (cpy && bitmap) {
+ if ((prev->len) && (prev->bitmap == bitmap) &&
+ (prev->start + prev->len * PAGE_SIZE == addr)) {
+ prev->len += len;
+ p->found_pages += len;
+ } else if (p->vec_index < p->vec_len) {
+ if (prev->len) {
+ memcpy(&p->vec[p->vec_index], prev, sizeof(struct page_region));
+ p->vec_index++;
+ }
+ prev->start = addr;
+ prev->len = len;
+ prev->bitmap = bitmap;
+ p->found_pages += len;
+ } else {
+ return -ENOSPC;
+ }
+ }
+ return 0;
+}
+
+static inline int export_prev_to_out(struct pagemap_scan_private *p, struct page_region __user *vec,
+ unsigned long *vec_index)
+{
+ struct page_region *prev = &p->prev;
+
+ if (prev->len) {
+ if (copy_to_user(&vec[*vec_index], prev, sizeof(struct page_region)))
+ return -EFAULT;
+ p->vec_index++;
+ (*vec_index)++;
+ prev->len = 0;
+ }
+ return 0;
+}
+
+static inline int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+ unsigned long end, struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ unsigned long addr = end;
+ unsigned int len;
+ spinlock_t *ptl;
+ int ret = 0;
+ pte_t *pte;
+ bool pmd_wt;
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ pmd_wt = !is_pmd_uffd_wp(*pmd);
+ /*
+ * Break huge page into small pages if operation needs to be performed is
+ * on a portion of the huge page or the return buffer cannot store complete
+ * data.
+ */
+ if (pmd_wt && (IS_WP_ENGAGE_OP(p) && (end - start < HPAGE_SIZE))) {
+ spin_unlock(ptl);
+ split_huge_pmd(vma, pmd, start);
+ goto process_smaller_pages;
+ }
+ if (IS_GET_OP(p)) {
+ len = (end - start)/PAGE_SIZE;
+ if (p->max_pages && p->found_pages + len > p->max_pages)
+ len = p->max_pages - p->found_pages;
+
+ ret = add_to_out(pmd_wt, vma->vm_file, pmd_present(*pmd),
+ is_swap_pmd(*pmd), p, start, len);
+ if (ret) {
+ spin_unlock(ptl);
+ return ret;
+ }
+ }
+ spin_unlock(ptl);
+ if (IS_WP_ENGAGE_OP(p) && pmd_wt)
+ ret = wp_range_async(vma, start, HPAGE_SIZE);
+ return ret;
+ }
+process_smaller_pages:
+ if (pmd_trans_unstable(pmd))
+ return 0;
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+ if (IS_GET_OP(p)) {
+ for (addr = start;
+ addr < end && !ret && (!p->max_pages || (p->found_pages < p->max_pages));
+ pte++, addr += PAGE_SIZE) {
+ ret = add_to_out(!is_pte_uffd_wp(*pte), vma->vm_file, pte_present(*pte),
+ is_swap_pte(*pte), p, addr, 1);
+ if (ret)
+ break;
+ }
+ }
+ pte_unmap_unlock(pte - 1, ptl);
+ if (!ret && IS_WP_ENGAGE_OP(p))
+ ret = wp_range_async(vma, start, addr - start);
+
+ cond_resched();
+ return ret;
+}
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, int depth,
+ struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ unsigned int len;
+ int ret = 0;
+
+ if (vma) {
+ len = (end - addr)/PAGE_SIZE;
+ if (p->max_pages && p->found_pages + len > p->max_pages)
+ len = p->max_pages - p->found_pages;
+ ret = add_to_out(false, vma->vm_file, false, false, p, addr, len);
+ }
+ return ret;
+}
+
+/* No hugetlb support is present. */
+static const struct mm_walk_ops pagemap_scan_ops = {
+ .test_walk = pagemap_scan_test_walk,
+ .pmd_entry = pagemap_scan_pmd_entry,
+ .pte_hole = pagemap_scan_pte_hole,
+};
+
+static long do_pagemap_cmd(struct mm_struct *mm, struct pagemap_scan_arg *arg)
+{
+ unsigned long empty_slots, vec_index = 0;
+ unsigned long __user start, end;
+ unsigned long __start, __end;
+ struct page_region __user *vec;
+ struct pagemap_scan_private p;
+ int ret;
+
+ start = (unsigned long)untagged_addr(arg->start);
+ vec = (struct page_region *)untagged_addr(arg->vec);
+ if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
+ return -EINVAL;
+ if (IS_GET_OP(arg) && ((arg->vec_len == 0) ||
+ (!access_ok((void __user *)vec, arg->vec_len * sizeof(struct page_region)))))
+ return -ENOMEM;
+ if ((arg->flags & ~PAGEMAP_WP_ENGAGE) || (arg->required_mask & ~PAGEMAP_OP_MASK) ||
+ (arg->anyof_mask & ~PAGEMAP_OP_MASK) || (arg->excluded_mask & ~PAGEMAP_OP_MASK) ||
+ (arg->return_mask & ~PAGEMAP_OP_MASK))
+ return -EINVAL;
+ if (IS_GET_OP(arg) && ((!arg->required_mask && !arg->anyof_mask && !arg->excluded_mask) ||
+ !arg->return_mask))
+ return -EINVAL;
+ /* The non-WT flags cannot be obtained if PAGEMAP_WP_ENGAGE is also specified. */
+ if (IS_WP_ENGAGE_OP(arg) && ((arg->required_mask & PAGEMAP_NONWT_OP_MASK) ||
+ (arg->anyof_mask & PAGEMAP_NONWT_OP_MASK)))
+ return -EINVAL;
+
+ end = start + arg->len;
+ p.max_pages = arg->max_pages;
+ p.found_pages = 0;
+ p.flags = arg->flags;
+ p.required_mask = arg->required_mask;
+ p.anyof_mask = arg->anyof_mask;
+ p.excluded_mask = arg->excluded_mask;
+ p.return_mask = arg->return_mask;
+ p.prev.len = 0;
+ p.vec_len = (PAGEMAP_WALK_SIZE >> PAGE_SHIFT);
+
+ if (IS_GET_OP(arg)) {
+ p.vec = kmalloc_array(p.vec_len, sizeof(struct page_region), GFP_KERNEL);
+ if (!p.vec)
+ return -ENOMEM;
+ } else {
+ p.vec = NULL;
+ }
+ __start = __end = start;
+ while (__end < end) {
+ p.vec_index = 0;
+ empty_slots = arg->vec_len - vec_index;
+ if (p.vec_len > empty_slots)
+ p.vec_len = empty_slots;
+
+ __end = (__start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
+ if (__end > end)
+ __end = end;
+
+ mmap_read_lock(mm);
+ ret = walk_page_range(mm, __start, __end, &pagemap_scan_ops, &p);
+ mmap_read_unlock(mm);
+ if (!(!ret || ret == -ENOSPC))
+ goto free_data;
+
+ __start = __end;
+ if (IS_GET_OP(arg) && p.vec_index) {
+ if (copy_to_user(&vec[vec_index], p.vec,
+ p.vec_index * sizeof(struct page_region))) {
+ ret = -EFAULT;
+ goto free_data;
+ }
+ vec_index += p.vec_index;
+ }
+ }
+ ret = export_prev_to_out(&p, vec, &vec_index);
+ if (!ret)
+ ret = vec_index;
+free_data:
+ if (IS_GET_OP(arg))
+ kfree(p.vec);
+
+ return ret;
+}
+
+static long pagemap_scan_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+ struct pagemap_scan_arg __user *uarg = (struct pagemap_scan_arg __user *)arg;
+ struct mm_struct *mm = file->private_data;
+ struct pagemap_scan_arg argument;
+
+ if (cmd == PAGEMAP_SCAN) {
+ if (copy_from_user(&argument, uarg, sizeof(struct pagemap_scan_arg)))
+ return -EFAULT;
+ return do_pagemap_cmd(mm, &argument);
+ }
+ return -EINVAL;
+}
+
const struct file_operations proc_pagemap_operations = {
.llseek = mem_lseek, /* borrow this */
.read = pagemap_read,
.open = pagemap_open,
.release = pagemap_release,
+ .unlocked_ioctl = pagemap_scan_ioctl,
+ .compat_ioctl = pagemap_scan_ioctl,
};
#endif /* CONFIG_PROC_PAGE_MONITOR */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index b7b56871029c..6d03a903a745 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
+#define PAGE_IS_WT (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pagemap_scan_arg - Pagemap ioctl argument
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @flags: Flags for the IOCTL
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pagemap_scan_arg {
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u32 max_pages;
+ __u32 flags;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Special flags */
+#define PAGEMAP_WP_ENGAGE (1 << 0)
+
#endif /* _UAPI_LINUX_FS_H */
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index b7b56871029c..6d03a903a745 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
+#define PAGE_IS_WT (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pagemap_scan_arg - Pagemap ioctl argument
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @flags: Flags for the IOCTL
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pagemap_scan_arg {
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u32 max_pages;
+ __u32 flags;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Special flags */
+#define PAGEMAP_WP_ENGAGE (1 << 0)
+
#endif /* _UAPI_LINUX_FS_H */
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v8 4/4] selftests: vm: add pagemap ioctl tests
2023-01-24 8:43 [PATCH v8 0/4] Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
` (2 preceding siblings ...)
2023-01-24 8:43 ` [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
@ 2023-01-24 8:43 ` Muhammad Usama Anjum
3 siblings, 0 replies; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-24 8:43 UTC (permalink / raw)
To: Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Muhammad Usama Anjum, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
Add pagemap ioctl tests. Add several different types of tests to judge
the correction of the interface.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Chages in v7:
- Add and update all test cases
Changes in v6:
- Rename variables
Changes in v4:
- Updated all the tests to conform to new IOCTL
Changes in v3:
- Add another test to do sanity of flags
Changes in v2:
- Update the tests to use the ioctl interface instead of syscall
TAP version 13
1..54
ok 1 sanity_tests_sd wrong flag specified
ok 2 sanity_tests_sd wrong mask specified
ok 3 sanity_tests_sd wrong return mask specified
ok 4 sanity_tests_sd mixture of correct and wrong flag
ok 5 sanity_tests_sd Clear area with larger vec size
ok 6 sanity_tests_sd Repeated pattern of dirty and non-dirty pages
ok 7 sanity_tests_sd Repeated pattern of dirty and non-dirty pages in parts
ok 8 sanity_tests_sd Two regions
ok 9 Page testing: all new pages must be soft dirty
ok 10 Page testing: all pages must not be soft dirty
ok 11 Page testing: all pages dirty other than first and the last one
ok 12 Page testing: only middle page dirty
ok 13 Page testing: only two middle pages dirty
ok 14 Page testing: only get 2 dirty pages and clear them as well
ok 15 Page testing: Range clear only
ok 16 Large Page testing: all new pages must be soft dirty
ok 17 Large Page testing: all pages must not be soft dirty
ok 18 Large Page testing: all pages dirty other than first and the last one
ok 19 Large Page testing: only middle page dirty
ok 20 Large Page testing: only two middle pages dirty
ok 21 Large Page testing: only get 2 dirty pages and clear them as well
ok 22 Large Page testing: Range clear only
ok 23 Huge page testing: all new pages must be soft dirty
ok 24 Huge page testing: all pages must not be soft dirty
ok 25 Huge page testing: all pages dirty other than first and the last one
ok 26 Huge page testing: only middle page dirty
ok 27 Huge page testing: only two middle pages dirty
ok 28 Huge page testing: only get 2 dirty pages and clear them as well
ok 29 Huge page testing: Range clear only
ok 30 hpage_unit_tests all new huge page must be dirty
ok 31 hpage_unit_tests all the huge page must not be dirty
ok 32 hpage_unit_tests all the huge page must be dirty and clear
ok 33 hpage_unit_tests only middle page dirty
ok 34 hpage_unit_tests clear first half of huge page
ok 35 hpage_unit_tests clear first half of huge page with limited buffer
ok 36 hpage_unit_tests clear second half huge page
ok 37 Test test_simple
ok 38 mprotect_tests Both pages dirty
ok 39 mprotect_tests Both pages are not soft dirty
ok 40 mprotect_tests Both pages dirty after remap and mprotect
ok 41 mprotect_tests Clear and make the pages dirty
ok 42 sanity_tests clear op can only be specified with PAGE_IS_WT
ok 43 sanity_tests required_mask specified
ok 44 sanity_tests anyof_mask specified
ok 45 sanity_tests excluded_mask specified
ok 46 sanity_tests required_mask and anyof_mask specified
ok 47 sanity_tests Get sd and present pages with anyof_mask
ok 48 sanity_tests Get all the pages with required_mask
ok 49 sanity_tests Get sd and present pages with required_mask and anyof_mask
ok 50 sanity_tests Don't get sd pages
ok 51 sanity_tests Don't get present pages
ok 52 sanity_tests Find dirty present pages with return mask
ok 53 sanity_tests Memory mapped file
ok 54 unmapped_region_tests Get status of pages
# Totals: pass:54 fail:0 xfail:0 xpass:0 skip:0 error:0
---
tools/testing/selftests/vm/.gitignore | 1 +
tools/testing/selftests/vm/Makefile | 5 +-
tools/testing/selftests/vm/pagemap_ioctl.c | 880 +++++++++++++++++++++
3 files changed, 884 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c
diff --git a/tools/testing/selftests/vm/.gitignore b/tools/testing/selftests/vm/.gitignore
index 1f8c36a9fa10..9e7e0ae26582 100644
--- a/tools/testing/selftests/vm/.gitignore
+++ b/tools/testing/selftests/vm/.gitignore
@@ -17,6 +17,7 @@ mremap_dontunmap
mremap_test
on-fault-limit
transhuge-stress
+pagemap_ioctl
protection_keys
protection_keys_32
protection_keys_64
diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile
index 89c14e41bd43..54c074440a1b 100644
--- a/tools/testing/selftests/vm/Makefile
+++ b/tools/testing/selftests/vm/Makefile
@@ -24,9 +24,8 @@ MACHINE ?= $(shell echo $(uname_M) | sed -e 's/aarch64.*/arm64/' -e 's/ppc64.*/p
# things despite using incorrect values such as an *occasionally* incomplete
# LDLIBS.
MAKEFLAGS += --no-builtin-rules
-
CFLAGS = -Wall -I $(top_srcdir) -I $(top_srcdir)/usr/include $(EXTRA_CFLAGS) $(KHDR_INCLUDES)
-LDLIBS = -lrt -lpthread
+LDLIBS = -lrt -lpthread -lm
TEST_GEN_FILES = cow
TEST_GEN_FILES += compaction_test
TEST_GEN_FILES += gup_test
@@ -52,6 +51,7 @@ TEST_GEN_FILES += on-fault-limit
TEST_GEN_FILES += thuge-gen
TEST_GEN_FILES += transhuge-stress
TEST_GEN_FILES += userfaultfd
+TEST_GEN_PROGS += pagemap_ioctl
TEST_GEN_PROGS += soft-dirty
TEST_GEN_PROGS += split_huge_page_test
TEST_GEN_FILES += ksm_tests
@@ -103,6 +103,7 @@ $(OUTPUT)/cow: vm_util.c
$(OUTPUT)/khugepaged: vm_util.c
$(OUTPUT)/ksm_functional_tests: vm_util.c
$(OUTPUT)/madv_populate: vm_util.c
+$(OUTPUT)/pagemap_ioctl: vm_util.c
$(OUTPUT)/soft-dirty: vm_util.c
$(OUTPUT)/split_huge_page_test: vm_util.c
$(OUTPUT)/userfaultfd: vm_util.c
diff --git a/tools/testing/selftests/vm/pagemap_ioctl.c b/tools/testing/selftests/vm/pagemap_ioctl.c
new file mode 100644
index 000000000000..d192c20deb48
--- /dev/null
+++ b/tools/testing/selftests/vm/pagemap_ioctl.c
@@ -0,0 +1,880 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <stdio.h>
+#include <fcntl.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <malloc.h>
+#include "vm_util.h"
+#include "../kselftest.h"
+#include <linux/types.h>
+#include <linux/userfaultfd.h>
+#include <linux/fs.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <math.h>
+#include <asm/unistd.h>
+
+#define PAGEMAP_OP_MASK (PAGE_IS_WT | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PAGEMAP_NON_WT_MASK (PAGE_IS_FILE | PAGE_IS_PRESENT | \
+ PAGE_IS_SWAPPED)
+
+#define TEST_ITERATIONS 10
+#define PAGEMAP "/proc/self/pagemap"
+int pagemap_fd;
+int uffd;
+int page_size;
+int hpage_size;
+
+static long pagemap_ioctl(void *start, int len, void *vec, int vec_len, int flag,
+ int max_pages, long required_mask, long anyof_mask, long excluded_mask,
+ long return_mask)
+{
+ struct pagemap_scan_arg arg;
+ int ret;
+
+ arg.start = (uintptr_t)start;
+ arg.len = len;
+ arg.vec = (uintptr_t)vec;
+ arg.vec_len = vec_len;
+ arg.flags = flag;
+ arg.max_pages = max_pages;
+ arg.required_mask = required_mask;
+ arg.anyof_mask = anyof_mask;
+ arg.excluded_mask = excluded_mask;
+ arg.return_mask = return_mask;
+
+ ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+
+ return ret;
+}
+
+int init_uffd(void)
+{
+ struct uffdio_api uffdio_api;
+
+ uffd = syscall(323, O_CLOEXEC | O_NONBLOCK);
+ if (uffd == -1)
+ ksft_exit_fail_msg("uffd syscall failed\n");
+
+ uffdio_api.api = UFFD_API;
+ uffdio_api.features = UFFD_FEATURE_WP_ASYNC;
+ if (ioctl(uffd, UFFDIO_API, &uffdio_api))
+ ksft_exit_fail_msg("UFFDIO_API\n");
+
+ if (uffdio_api.api != UFFD_API)
+ ksft_exit_fail_msg("UFFDIO_API error %llu\n", uffdio_api.api);
+
+ return 0;
+}
+
+int wp_init(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_register uffdio_register;
+ struct uffdio_writeprotect wp;
+
+ /* TODO: can it be avoided? Write protect doesn't engage on the pages if they aren't
+ * present already. The pages can be made present by writing to them.
+ */
+ memset(lpBaseAddress, -1, dwRegionSize);
+
+ uffdio_register.range.start = (unsigned long)lpBaseAddress;
+ uffdio_register.range.len = dwRegionSize;
+ uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+ if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register) == -1)
+ ksft_exit_fail_msg("ioctl(UFFDIO_REGISTER)\n");
+
+ if (!(uffdio_register.ioctls & UFFDIO_WRITEPROTECT))
+ ksft_exit_fail_msg("ioctl set is incorrect\n");
+
+ if (rand() % 2) {
+ wp.range.start = (unsigned long)lpBaseAddress;
+ wp.range.len = dwRegionSize;
+ wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+ if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp) == -1)
+ ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+ } else {
+ if (pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0) < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", 1, errno, strerror(errno));
+ }
+ return 0;
+}
+
+int wp_free(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_register uffdio_register;
+
+ uffdio_register.range.start = (unsigned long)lpBaseAddress;
+ uffdio_register.range.len = dwRegionSize;
+ uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+ if (ioctl(uffd, UFFDIO_UNREGISTER, &uffdio_register.range))
+ ksft_exit_fail_msg("ioctl unregister failure\n");
+ return 0;
+}
+
+int clear_softdirty_wp(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_writeprotect wp;
+
+ if (rand() % 2) {
+ wp.range.start = (unsigned long)lpBaseAddress;
+ wp.range.len = dwRegionSize;
+ wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+ if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp) == -1)
+ ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+ } else {
+ if (pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0) < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", 1, errno, strerror(errno));
+ }
+ return 0;
+}
+
+int sanity_tests_sd(void)
+{
+ char *mem, *m[2];
+ int mem_size, vec_size, ret, ret2, ret3, i, num_pages = 10;
+ struct page_region *vec;
+
+ vec_size = 100;
+ mem_size = num_pages * page_size;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ if (!vec)
+ ksft_exit_fail_msg("error nomem\n");
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(mem, mem_size);
+
+ /* 1. wrong operation */
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, -1,
+ 0, PAGE_IS_WT, 0, 0, PAGE_IS_WT) < 0,
+ "%s wrong flag specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 8,
+ 0, 0x1111, 0, 0, PAGE_IS_WT) < 0,
+ "%s wrong mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0,
+ 0, PAGE_IS_WT, 0, 0, 0x1000) < 0,
+ "%s wrong return mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size,
+ PAGEMAP_WP_ENGAGE | 0x32,
+ 0, PAGE_IS_WT, 0, 0, PAGE_IS_WT) < 0,
+ "%s mixture of correct and wrong flag\n", __func__);
+
+ /* 2. Clear area with larger vec size */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ ksft_test_result(ret >= 0, "%s Clear area with larger vec size\n", __func__);
+
+ /* 3. Repeated pattern of dirty and non-dirty pages */
+ for (i = 0; i < mem_size; i += 2 * page_size)
+ mem[i]++;
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == mem_size/(page_size * 2),
+ "%s Repeated pattern of dirty and non-dirty pages\n", __func__);
+
+ /* 4. Repeated pattern of dirty and non-dirty pages in parts */
+ ret = pagemap_ioctl(mem, mem_size, vec, num_pages/5, PAGEMAP_WP_ENGAGE,
+ num_pages/2 - 2, PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret2 = pagemap_ioctl(mem, mem_size, vec, 2, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+ ret3 = pagemap_ioctl(mem, mem_size, vec, num_pages/2, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret3 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret3, errno, strerror(errno));
+
+ ksft_test_result((ret + ret3) == num_pages/2 && ret2 == 2,
+ "%s Repeated pattern of dirty and non-dirty pages in parts\n", __func__);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 5. Two regions */
+ m[0] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (m[0] == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ m[1] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (m[1] == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(m[0], mem_size);
+ wp_init(m[1], mem_size);
+
+ memset(m[0], 'a', mem_size);
+ memset(m[1], 'b', mem_size);
+
+ ret = pagemap_ioctl(m[0], mem_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(m[1], mem_size, vec, 1, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == mem_size/page_size,
+ "%s Two regions\n", __func__);
+
+ wp_free(m[0], mem_size);
+ wp_free(m[1], mem_size);
+ munmap(m[0], mem_size);
+ munmap(m[1], mem_size);
+
+ free(vec);
+ return 0;
+}
+
+int base_tests(char *prefix, char *mem, int mem_size, int skip)
+{
+ int vec_size, ret, dirty, dirty2;
+ struct page_region *vec, *vec2;
+
+ if (skip) {
+ ksft_test_result_skip("%s all new pages must be soft dirty\n", prefix);
+ ksft_test_result_skip("%s all pages must not be soft dirty\n", prefix);
+ ksft_test_result_skip("%s all pages dirty other than first and the last one\n",
+ prefix);
+ ksft_test_result_skip("%s only middle page dirty\n", prefix);
+ ksft_test_result_skip("%s only two middle pages dirty\n", prefix);
+ ksft_test_result_skip("%s only get 2 dirty pages and clear them as well\n", prefix);
+ ksft_test_result_skip("%s Range clear only\n", prefix);
+ return 0;
+ }
+
+ vec_size = mem_size/page_size;
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ vec2 = malloc(sizeof(struct page_region) * vec_size);
+
+ /* 1. all new pages must be not be soft dirty */
+ dirty = pagemap_ioctl(mem, mem_size, vec, 1, PAGEMAP_WP_ENGAGE, vec_size - 2,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ dirty2 = pagemap_ioctl(mem, mem_size, vec2, 1, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty2, errno, strerror(errno));
+
+ ksft_test_result(dirty == 0 && dirty2 == 0,
+ "%s all new pages must be soft dirty\n", prefix);
+
+ /* 2. all pages must not be soft dirty */
+ dirty = pagemap_ioctl(mem, mem_size, vec, 1, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 0, "%s all pages must not be soft dirty\n", prefix);
+
+ /* 3. all pages dirty other than first and the last one */
+ memset(mem + page_size, 0, mem_size - (2 * page_size));
+
+ dirty = pagemap_ioctl(mem, mem_size, vec, 1, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 1 && vec[0].len >= vec_size - 2 && vec[0].len <= vec_size,
+ "%s all pages dirty other than first and the last one\n", prefix);
+
+ /* 4. only middle page dirty */
+ clear_softdirty_wp(mem, mem_size);
+ mem[vec_size/2 * page_size]++;
+
+ dirty = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 1 && vec[0].len >= 1,
+ "%s only middle page dirty\n", prefix);
+
+ /* 5. only two middle pages dirty and walk over only middle pages */
+ clear_softdirty_wp(mem, mem_size);
+ mem[vec_size/2 * page_size]++;
+ mem[(vec_size/2 + 1) * page_size]++;
+
+ dirty = pagemap_ioctl(&mem[vec_size/2 * page_size], 2 * page_size, vec, 1, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 1 && vec[0].start == (uintptr_t)(&mem[vec_size/2 * page_size]) &&
+ vec[0].len == 2,
+ "%s only two middle pages dirty\n", prefix);
+
+ /* 6. only get 2 dirty pages and clear them as well */
+ memset(mem, -1, mem_size);
+
+ /* get and clear second and third pages */
+ ret = pagemap_ioctl(mem + page_size, 2 * page_size, vec, 1, PAGEMAP_WP_ENGAGE,
+ 2, PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ dirty = pagemap_ioctl(mem, mem_size, vec2, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == 2 &&
+ vec[0].start == (uintptr_t)(mem + page_size) &&
+ dirty == 2 && vec2[0].len == 1 && vec2[0].start == (uintptr_t)mem &&
+ vec2[1].len == vec_size - 3 &&
+ vec2[1].start == (uintptr_t)(mem + 3 * page_size),
+ "%s only get 2 dirty pages and clear them as well\n", prefix);
+
+ /* 7. Range clear only */
+ memset(mem, -1, mem_size);
+
+ dirty = pagemap_ioctl(mem, mem_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ dirty2 = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty2, errno, strerror(errno));
+
+ ksft_test_result(dirty == 0 && dirty2 == 0, "%s Range clear only\n",
+ prefix);
+
+ free(vec);
+ free(vec2);
+ return 0;
+}
+
+void *gethugepage(int map_size)
+{
+ int ret;
+ char *map;
+
+ map = memalign(hpage_size, map_size);
+ if (!map)
+ ksft_exit_fail_msg("memalign failed %d %s\n", errno, strerror(errno));
+
+ ret = madvise(map, map_size, MADV_HUGEPAGE);
+ if (ret)
+ ksft_exit_fail_msg("madvise failed %d %d %s\n", ret, errno, strerror(errno));
+
+ wp_init(map, map_size);
+
+ if (check_huge_anon(map, map_size/hpage_size, hpage_size))
+ return map;
+
+ free(map);
+ return NULL;
+
+}
+
+int hpage_unit_tests(void)
+{
+ char *map;
+ int ret;
+ size_t num_pages = 10;
+ int map_size = hpage_size * num_pages;
+ int vec_size = map_size/page_size;
+ struct page_region *vec, *vec2;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ vec2 = malloc(sizeof(struct page_region) * vec_size);
+ if (!vec || !vec2)
+ ksft_exit_fail_msg("malloc failed\n");
+
+ map = gethugepage(map_size);
+ if (map) {
+ /* 1. all new huge page must not be dirty */
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 0, "%s all new huge page must be dirty\n", __func__);
+
+ /* 2. all the huge page must not be dirty */
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 0, "%s all the huge page must not be dirty\n", __func__);
+
+ /* 3. all the huge page must be dirty and clear dirty as well */
+ memset(map, -1, map_size);
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].start == (uintptr_t)map &&
+ vec[0].len == vec_size && vec[0].bitmap == PAGE_IS_WT,
+ "%s all the huge page must be dirty and clear\n", __func__);
+
+ /* 4. only middle page dirty */
+ wp_free(map, map_size);
+ free(map);
+ map = gethugepage(map_size);
+ wp_init(map, map_size);
+ clear_softdirty_wp(map, map_size);
+ map[vec_size/2 * page_size]++;
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len > 0,
+ "%s only middle page dirty\n", __func__);
+
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("all new huge page must be dirty\n");
+ ksft_test_result_skip("all the huge page must not be dirty\n");
+ ksft_test_result_skip("all the huge page must be dirty and clear\n");
+ ksft_test_result_skip("only middle page dirty\n");
+ }
+
+ /* 5. clear first half of huge page */
+ map = gethugepage(map_size);
+ if (map) {
+
+ memset(map, 0, map_size);
+
+ ret = pagemap_ioctl(map, map_size/2, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+ vec[0].start == (uintptr_t)(map + map_size/2),
+ "%s clear first half of huge page\n", __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("clear first half of huge page\n");
+ }
+
+ /* 6. clear first half of huge page with limited buffer */
+ map = gethugepage(map_size);
+ if (map) {
+ memset(map, 0, map_size);
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PAGEMAP_WP_ENGAGE,
+ vec_size/2, PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+ vec[0].start == (uintptr_t)(map + map_size/2),
+ "%s clear first half of huge page with limited buffer\n",
+ __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("clear first half of huge page with limited buffer\n");
+ }
+
+ /* 7. clear second half of huge page */
+ map = gethugepage(map_size);
+ if (map) {
+ memset(map, -1, map_size);
+ ret = pagemap_ioctl(map + map_size/2, map_size/2, NULL, 0, PAGEMAP_WP_ENGAGE,
+ 0, 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2,
+ "%s clear second half huge page\n", __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("clear second half huge page\n");
+ }
+
+ free(vec);
+ free(vec2);
+ return 0;
+}
+
+int unmapped_region_tests(void)
+{
+ void *start = (void *)0x10000000;
+ int dirty, len = 0x00040000;
+ int vec_size = len / page_size;
+ struct page_region *vec = malloc(sizeof(struct page_region) * vec_size);
+
+ /* 1. Get dirty pages */
+ dirty = pagemap_ioctl(start, len, vec, vec_size, 0, 0, PAGEMAP_NON_WT_MASK, 0, 0,
+ PAGEMAP_NON_WT_MASK);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty >= 0, "%s Get status of pages\n", __func__);
+
+ free(vec);
+ return 0;
+}
+
+static void test_simple(void)
+{
+ int i;
+ char *map;
+ struct page_region vec;
+
+ map = aligned_alloc(page_size, page_size);
+ if (!map)
+ ksft_exit_fail_msg("aligned_alloc failed\n");
+ wp_init(map, page_size);
+
+ clear_softdirty_wp(map, page_size);
+
+ for (i = 0 ; i < TEST_ITERATIONS; i++) {
+ if (pagemap_ioctl(map, page_size, &vec, 1, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT) == 1) {
+ ksft_print_msg("dirty bit was 1, but should be 0 (i=%d)\n", i);
+ break;
+ }
+
+ clear_softdirty_wp(map, page_size);
+ /* Write something to the page to get the dirty bit enabled on the page */
+ map[0]++;
+
+ if (pagemap_ioctl(map, page_size, &vec, 1, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT) == 0) {
+ ksft_print_msg("dirty bit was 0, but should be 1 (i=%d)\n", i);
+ break;
+ }
+
+ clear_softdirty_wp(map, page_size);
+ }
+ wp_free(map, page_size);
+ free(map);
+
+ ksft_test_result(i == TEST_ITERATIONS, "Test %s\n", __func__);
+}
+
+int sanity_tests(void)
+{
+ char *mem, *fmem;
+ int mem_size, vec_size, ret;
+ struct page_region *vec;
+
+ /* 1. wrong operation */
+ mem_size = 10 * page_size;
+ vec_size = mem_size / page_size;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED || vec == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(mem, mem_size);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGEMAP_OP_MASK, 0, 0, PAGEMAP_OP_MASK) < 0,
+ "%s clear op can only be specified with PAGE_IS_WT\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGEMAP_OP_MASK, 0, 0, PAGEMAP_OP_MASK) >= 0,
+ "%s required_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_OP_MASK, 0, PAGEMAP_OP_MASK) >= 0,
+ "%s anyof_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, 0, PAGEMAP_OP_MASK, PAGEMAP_OP_MASK) >= 0,
+ "%s excluded_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGEMAP_OP_MASK, PAGEMAP_OP_MASK, 0, PAGEMAP_OP_MASK) >= 0,
+ "%s required_mask and anyof_mask specified\n", __func__);
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 2. Get sd and present pages with anyof_mask */
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ memset(mem, 0, mem_size);
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_OP_MASK, 0, PAGEMAP_OP_MASK);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WT | PAGE_IS_PRESENT),
+ "%s Get sd and present pages with anyof_mask\n", __func__);
+
+ /* 3. Get sd and present pages with required_mask */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGEMAP_OP_MASK, 0, 0, PAGEMAP_OP_MASK);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WT | PAGE_IS_PRESENT),
+ "%s Get all the pages with required_mask\n", __func__);
+
+ /* 4. Get sd and present pages with required_mask and anyof_mask */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, PAGE_IS_PRESENT, 0, PAGEMAP_OP_MASK);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WT | PAGE_IS_PRESENT),
+ "%s Get sd and present pages with required_mask and anyof_mask\n",
+ __func__);
+
+ /* 5. Don't get sd pages */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, 0, PAGE_IS_WT, PAGEMAP_OP_MASK);
+ ksft_test_result(ret == 0, "%s Don't get sd pages\n", __func__);
+
+ /* 6. Don't get present pages */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, 0, PAGE_IS_PRESENT, PAGEMAP_OP_MASK);
+ ksft_test_result(ret == 0, "%s Don't get present pages\n", __func__);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 8. Find dirty present pages with return mask */
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ memset(mem, 0, mem_size);
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_OP_MASK, 0, PAGE_IS_WT);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == PAGE_IS_WT,
+ "%s Find dirty present pages with return mask\n", __func__);
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 9. Memory mapped file */
+ int fd;
+ struct stat sbuf;
+
+ fd = open(__FILE__, O_RDONLY);
+ if (fd < 0) {
+ ksft_test_result_skip("%s Memory mapped file\n");
+ goto free_vec_and_return;
+ }
+
+ ret = stat(__FILE__, &sbuf);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ fmem = mmap(NULL, sbuf.st_size, PROT_READ, MAP_SHARED, fd, 0);
+ if (fmem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ ret = pagemap_ioctl(fmem, sbuf.st_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_NON_WT_MASK, 0, PAGEMAP_NON_WT_MASK);
+
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem &&
+ vec[0].len == ceilf((float)sbuf.st_size/page_size) &&
+ vec[0].bitmap == PAGE_IS_FILE,
+ "%s Memory mapped file\n", __func__);
+
+ munmap(fmem, sbuf.st_size);
+ close(fd);
+
+free_vec_and_return:
+ free(vec);
+ return 0;
+}
+
+int mprotect_tests(void)
+{
+ int ret;
+ char *mem, *mem2;
+ struct page_region vec;
+ int pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+
+ if (pagemap_fd < 0) {
+ fprintf(stderr, "open() failed\n");
+ exit(1);
+ }
+
+ /* 1. Map two pages */
+ mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, 2 * page_size);
+
+ /* Populate both pages. */
+ memset(mem, 1, 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2, "%s Both pages dirty\n", __func__);
+
+ /* 2. Start softdirty tracking. Clear VM_SOFTDIRTY and clear the softdirty PTE bit. */
+ ret = pagemap_ioctl(mem, 2 * page_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT) == 0,
+ "%s Both pages are not soft dirty\n", __func__);
+
+ /* 3. Remap the second page */
+ mem2 = mmap(mem + page_size, page_size, PROT_READ|PROT_WRITE,
+ MAP_PRIVATE|MAP_ANON|MAP_FIXED, -1, 0);
+ if (mem2 == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem2, page_size);
+
+ /* Protect + unprotect. */
+ mprotect(mem, 2 * page_size, PROT_READ);
+ mprotect(mem, 2 * page_size, PROT_READ|PROT_WRITE);
+
+ /* Modify both pages. */
+ memset(mem, 2, 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2,
+ "%s Both pages dirty after remap and mprotect\n", __func__);
+
+ /* 4. Clear and make the pages dirty */
+ ret = pagemap_ioctl(mem, 2 * page_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ memset(mem, 'A', 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2,
+ "%s Clear and make the pages dirty\n", __func__);
+
+ wp_free(mem, 2 * page_size);
+ munmap(mem, 2 * page_size);
+ return 0;
+}
+
+int main(void)
+{
+ char *mem, *map;
+ int mem_size;
+
+ ksft_print_header();
+ ksft_set_plan(54);
+
+ page_size = getpagesize();
+ hpage_size = read_pmd_pagesize();
+
+ pagemap_fd = open(PAGEMAP, O_RDWR);
+ if (pagemap_fd < 0)
+ return -EINVAL;
+
+ if (init_uffd())
+ ksft_exit_fail_msg("uffd init failed\n");
+
+ /*
+ * Soft-dirty PTE bit tests
+ */
+
+ /* 1. Sanity testing */
+ sanity_tests_sd();
+
+ /* 2. Normal page testing */
+ mem_size = 10 * page_size;
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ base_tests("Page testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 3. Large page testing */
+ mem_size = 512 * 10 * page_size;
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ base_tests("Large Page testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 4. Huge page testing */
+ map = gethugepage(hpage_size);
+ if (map)
+ base_tests("Huge page testing:", map, hpage_size, 0);
+ else
+ base_tests("Huge page testing:", NULL, 0, 1);
+
+ wp_free(map, hpage_size);
+ free(map);
+
+ /* 6. Huge page tests */
+ hpage_unit_tests();
+
+ /* 7. Iterative test */
+ test_simple();
+
+ /* 8. Mprotect test */
+ mprotect_tests();
+
+ /*
+ * Other PTE bit tests
+ */
+
+ /* 1. Sanity testing */
+ sanity_tests();
+
+ /* 2. Unmapped address test */
+ unmapped_region_tests();
+
+ close(pagemap_fd);
+ return ksft_exit_pass();
+}
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
2023-01-24 8:43 ` [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
@ 2023-01-24 10:02 ` kernel test robot
2023-01-27 17:36 ` Peter Xu
1 sibling, 0 replies; 21+ messages in thread
From: kernel test robot @ 2023-01-24 10:02 UTC (permalink / raw)
To: Muhammad Usama Anjum, Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: oe-kbuild-all, Linux Memory Management List, Alexander Viro,
Shuah Khan, Christian Brauner, Yang Shi, Vlastimil Babka,
Liam R . Howlett, Yun Zhou, Suren Baghdasaryan, Alex Sierra,
Muhammad Usama Anjum, Matthew Wilcox, Pasha Tatashin,
Mike Rapoport, Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-kselftest,
Greg KH
Hi Muhammad,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on shuah-kselftest/next]
[also build test WARNING on shuah-kselftest/fixes linus/master v6.2-rc5]
[cannot apply to next-20230124]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next
patch link: https://lore.kernel.org/r/20230124084323.1363825-4-usama.anjum%40collabora.com
patch subject: [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
config: m68k-allyesconfig (https://download.01.org/0day-ci/archive/20230124/202301241708.pfzy132a-lkp@intel.com/config)
compiler: m68k-linux-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/8d3b72e0605d479fbc5c2bc6f4ba9ddfecdb9ccb
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
git checkout 8d3b72e0605d479fbc5c2bc6f4ba9ddfecdb9ccb
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=m68k olddefconfig
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=m68k SHELL=/bin/bash fs/proc/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
fs/proc/task_mmu.c: In function 'pagemap_scan_pmd_entry':
fs/proc/task_mmu.c:1880:14: warning: unused variable 'pmd_wt' [-Wunused-variable]
1880 | bool pmd_wt;
| ^~~~~~
fs/proc/task_mmu.c:1876:22: warning: unused variable 'len' [-Wunused-variable]
1876 | unsigned int len;
| ^~~
fs/proc/task_mmu.c: In function 'do_pagemap_cmd':
>> fs/proc/task_mmu.c:1971:15: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
1971 | vec = (struct page_region *)untagged_addr(arg->vec);
| ^
vim +1971 fs/proc/task_mmu.c
1960
1961 static long do_pagemap_cmd(struct mm_struct *mm, struct pagemap_scan_arg *arg)
1962 {
1963 unsigned long empty_slots, vec_index = 0;
1964 unsigned long __user start, end;
1965 unsigned long __start, __end;
1966 struct page_region __user *vec;
1967 struct pagemap_scan_private p;
1968 int ret;
1969
1970 start = (unsigned long)untagged_addr(arg->start);
> 1971 vec = (struct page_region *)untagged_addr(arg->vec);
1972 if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
1973 return -EINVAL;
1974 if (IS_GET_OP(arg) && ((arg->vec_len == 0) ||
1975 (!access_ok((void __user *)vec, arg->vec_len * sizeof(struct page_region)))))
1976 return -ENOMEM;
1977 if ((arg->flags & ~PAGEMAP_WP_ENGAGE) || (arg->required_mask & ~PAGEMAP_OP_MASK) ||
1978 (arg->anyof_mask & ~PAGEMAP_OP_MASK) || (arg->excluded_mask & ~PAGEMAP_OP_MASK) ||
1979 (arg->return_mask & ~PAGEMAP_OP_MASK))
1980 return -EINVAL;
1981 if (IS_GET_OP(arg) && ((!arg->required_mask && !arg->anyof_mask && !arg->excluded_mask) ||
1982 !arg->return_mask))
1983 return -EINVAL;
1984 /* The non-WT flags cannot be obtained if PAGEMAP_WP_ENGAGE is also specified. */
1985 if (IS_WP_ENGAGE_OP(arg) && ((arg->required_mask & PAGEMAP_NONWT_OP_MASK) ||
1986 (arg->anyof_mask & PAGEMAP_NONWT_OP_MASK)))
1987 return -EINVAL;
1988
1989 end = start + arg->len;
1990 p.max_pages = arg->max_pages;
1991 p.found_pages = 0;
1992 p.flags = arg->flags;
1993 p.required_mask = arg->required_mask;
1994 p.anyof_mask = arg->anyof_mask;
1995 p.excluded_mask = arg->excluded_mask;
1996 p.return_mask = arg->return_mask;
1997 p.prev.len = 0;
1998 p.vec_len = (PAGEMAP_WALK_SIZE >> PAGE_SHIFT);
1999
2000 if (IS_GET_OP(arg)) {
2001 p.vec = kmalloc_array(p.vec_len, sizeof(struct page_region), GFP_KERNEL);
2002 if (!p.vec)
2003 return -ENOMEM;
2004 } else {
2005 p.vec = NULL;
2006 }
2007 __start = __end = start;
2008 while (__end < end) {
2009 p.vec_index = 0;
2010 empty_slots = arg->vec_len - vec_index;
2011 if (p.vec_len > empty_slots)
2012 p.vec_len = empty_slots;
2013
2014 __end = (__start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
2015 if (__end > end)
2016 __end = end;
2017
2018 mmap_read_lock(mm);
2019 ret = walk_page_range(mm, __start, __end, &pagemap_scan_ops, &p);
2020 mmap_read_unlock(mm);
2021 if (!(!ret || ret == -ENOSPC))
2022 goto free_data;
2023
2024 __start = __end;
2025 if (IS_GET_OP(arg) && p.vec_index) {
2026 if (copy_to_user(&vec[vec_index], p.vec,
2027 p.vec_index * sizeof(struct page_region))) {
2028 ret = -EFAULT;
2029 goto free_data;
2030 }
2031 vec_index += p.vec_index;
2032 }
2033 }
2034 ret = export_prev_to_out(&p, vec, &vec_index);
2035 if (!ret)
2036 ret = vec_index;
2037 free_data:
2038 if (IS_GET_OP(arg))
2039 kfree(p.vec);
2040
2041 return ret;
2042 }
2043
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 2/4] userfaultfd: split mwriteprotect_range()
2023-01-24 8:43 ` [PATCH v8 2/4] userfaultfd: split mwriteprotect_range() Muhammad Usama Anjum
@ 2023-01-24 10:23 ` kernel test robot
2023-01-27 17:05 ` Peter Xu
1 sibling, 0 replies; 21+ messages in thread
From: kernel test robot @ 2023-01-24 10:23 UTC (permalink / raw)
To: Muhammad Usama Anjum, Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: oe-kbuild-all, Linux Memory Management List, Alexander Viro,
Shuah Khan, Christian Brauner, Yang Shi, Vlastimil Babka,
Liam R . Howlett, Yun Zhou, Suren Baghdasaryan, Alex Sierra,
Muhammad Usama Anjum, Matthew Wilcox, Pasha Tatashin,
Mike Rapoport, Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-kselftest,
Greg KH
Hi Muhammad,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on shuah-kselftest/next]
[also build test ERROR on shuah-kselftest/fixes linus/master v6.2-rc5]
[cannot apply to next-20230124]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next
patch link: https://lore.kernel.org/r/20230124084323.1363825-3-usama.anjum%40collabora.com
patch subject: [PATCH v8 2/4] userfaultfd: split mwriteprotect_range()
config: i386-tinyconfig (https://download.01.org/0day-ci/archive/20230124/202301241831.AYRPKsH5-lkp@intel.com/config)
compiler: gcc-11 (Debian 11.3.0-8) 11.3.0
reproduce (this is a W=1 build):
# https://github.com/intel-lab-lkp/linux/commit/2806717805539421b82e971890ebbaf83b3deee4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
git checkout 2806717805539421b82e971890ebbaf83b3deee4
# save the config file
mkdir build_dir && cp config build_dir/.config
make W=1 O=build_dir ARCH=i386 olddefconfig
make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
ld: arch/x86/mm/init_32.o: in function `userfaultfd_wp_async':
init_32.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: arch/x86/mm/init_32.o: in function `wp_range_async':
>> init_32.c:(.text+0x3): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: arch/x86/mm/fault.o: in function `userfaultfd_wp_async':
fault.c:(.text+0x8cd): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: arch/x86/mm/fault.o: in function `wp_range_async':
fault.c:(.text+0x8d0): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: arch/x86/mm/pgtable.o: in function `userfaultfd_wp_async':
pgtable.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: arch/x86/mm/pgtable.o: in function `wp_range_async':
pgtable.c:(.text+0x3): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: kernel/fork.o: in function `userfaultfd_wp_async':
fork.c:(.text+0x5bb): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/fork.o: in function `wp_range_async':
fork.c:(.text+0x5be): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: kernel/sysctl.o: in function `userfaultfd_wp_async':
sysctl.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/sysctl.o: in function `wp_range_async':
sysctl.c:(.text+0x3): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: kernel/sys.o: in function `userfaultfd_wp_async':
sys.c:(.text+0xb5e): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/sys.o: in function `wp_range_async':
sys.c:(.text+0xb61): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: kernel/events/core.o: in function `userfaultfd_wp_async':
core.c:(.text+0x404c): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/events/core.o: in function `wp_range_async':
core.c:(.text+0x404f): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/filemap.o: in function `userfaultfd_wp_async':
filemap.c:(.text+0x7c5): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/filemap.o: in function `wp_range_async':
filemap.c:(.text+0x7c8): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/page-writeback.o: in function `userfaultfd_wp_async':
page-writeback.c:(.text+0xb69): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/page-writeback.o: in function `wp_range_async':
page-writeback.c:(.text+0xb6c): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/folio-compat.o: in function `userfaultfd_wp_async':
folio-compat.c:(.text+0xc): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/folio-compat.o: in function `wp_range_async':
folio-compat.c:(.text+0xf): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/readahead.o: in function `userfaultfd_wp_async':
readahead.c:(.text+0x128): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/readahead.o: in function `wp_range_async':
readahead.c:(.text+0x12b): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/swap.o: in function `userfaultfd_wp_async':
swap.c:(.text+0x6e5): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/swap.o: in function `wp_range_async':
swap.c:(.text+0x6e8): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/vmscan.o: in function `userfaultfd_wp_async':
vmscan.c:(.text+0xf96): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/vmscan.o: in function `wp_range_async':
vmscan.c:(.text+0xf99): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/shmem.o: in function `userfaultfd_wp_async':
shmem.c:(.text+0x91): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/shmem.o: in function `wp_range_async':
shmem.c:(.text+0x94): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/util.o: in function `userfaultfd_wp_async':
util.c:(.text+0x2b): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/util.o: in function `wp_range_async':
util.c:(.text+0x2e): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/vmstat.o: in function `userfaultfd_wp_async':
vmstat.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/vmstat.o: in function `wp_range_async':
vmstat.c:(.text+0x3): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/compaction.o: in function `userfaultfd_wp_async':
compaction.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/compaction.o: in function `wp_range_async':
compaction.c:(.text+0x3): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/workingset.o: in function `userfaultfd_wp_async':
workingset.c:(.text+0x181): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/workingset.o: in function `wp_range_async':
workingset.c:(.text+0x184): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/debug.o: in function `userfaultfd_wp_async':
debug.c:(.text+0xb9): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/debug.o: in function `wp_range_async':
debug.c:(.text+0xbc): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/gup.o: in function `userfaultfd_wp_async':
gup.c:(.text+0x2ae): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/gup.o: in function `wp_range_async':
gup.c:(.text+0x2b1): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/memory.o: in function `userfaultfd_wp_async':
memory.c:(.text+0x737): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/memory.o: in function `wp_range_async':
memory.c:(.text+0x73a): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/mincore.o: in function `userfaultfd_wp_async':
mincore.c:(.text+0x149): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mincore.o: in function `wp_range_async':
mincore.c:(.text+0x14c): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/mlock.o: in function `userfaultfd_wp_async':
mlock.c:(.text+0x90e): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mlock.o: in function `wp_range_async':
mlock.c:(.text+0x911): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/mmap.o: in function `userfaultfd_wp_async':
mmap.c:(.text+0x4d8): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mmap.o: in function `wp_range_async':
mmap.c:(.text+0x4db): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/mmu_gather.o: in function `userfaultfd_wp_async':
mmu_gather.c:(.text+0x29): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mmu_gather.o: in function `wp_range_async':
mmu_gather.c:(.text+0x2c): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
ld: mm/mprotect.o: in function `userfaultfd_wp_async':
mprotect.c:(.text+0x45): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mprotect.o: in function `wp_range_async':
mprotect.c:(.text+0x48): multiple definition of `wp_range_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x6): first defined here
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
@ 2023-01-24 10:43 ` kernel test robot
2023-01-24 11:04 ` kernel test robot
` (2 subsequent siblings)
3 siblings, 0 replies; 21+ messages in thread
From: kernel test robot @ 2023-01-24 10:43 UTC (permalink / raw)
To: Muhammad Usama Anjum, Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: oe-kbuild-all, Linux Memory Management List, Alexander Viro,
Shuah Khan, Christian Brauner, Yang Shi, Vlastimil Babka,
Liam R . Howlett, Yun Zhou, Suren Baghdasaryan, Alex Sierra,
Muhammad Usama Anjum, Matthew Wilcox, Pasha Tatashin,
Mike Rapoport, Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-kselftest,
Greg KH
Hi Muhammad,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on shuah-kselftest/next]
[also build test ERROR on shuah-kselftest/fixes linus/master v6.2-rc5 next-20230124]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next
patch link: https://lore.kernel.org/r/20230124084323.1363825-2-usama.anjum%40collabora.com
patch subject: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
config: i386-tinyconfig (https://download.01.org/0day-ci/archive/20230124/202301241804.zHrFxA0L-lkp@intel.com/config)
compiler: gcc-11 (Debian 11.3.0-8) 11.3.0
reproduce (this is a W=1 build):
# https://github.com/intel-lab-lkp/linux/commit/59e98aec663b7ca8fd5f3b3d2a0f17f777f425c4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
git checkout 59e98aec663b7ca8fd5f3b3d2a0f17f777f425c4
# save the config file
mkdir build_dir && cp config build_dir/.config
make W=1 O=build_dir ARCH=i386 olddefconfig
make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
ld: arch/x86/mm/init_32.o: in function `userfaultfd_wp_async':
>> init_32.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: arch/x86/mm/fault.o: in function `userfaultfd_wp_async':
fault.c:(.text+0x8cd): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: arch/x86/mm/pgtable.o: in function `userfaultfd_wp_async':
pgtable.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/fork.o: in function `userfaultfd_wp_async':
fork.c:(.text+0x5bb): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/sysctl.o: in function `userfaultfd_wp_async':
sysctl.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/sys.o: in function `userfaultfd_wp_async':
sys.c:(.text+0xb5e): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: kernel/events/core.o: in function `userfaultfd_wp_async':
core.c:(.text+0x404c): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/filemap.o: in function `userfaultfd_wp_async':
filemap.c:(.text+0x7c5): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/page-writeback.o: in function `userfaultfd_wp_async':
page-writeback.c:(.text+0xb69): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/folio-compat.o: in function `userfaultfd_wp_async':
folio-compat.c:(.text+0xc): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/readahead.o: in function `userfaultfd_wp_async':
readahead.c:(.text+0x128): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/swap.o: in function `userfaultfd_wp_async':
swap.c:(.text+0x6e5): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/vmscan.o: in function `userfaultfd_wp_async':
vmscan.c:(.text+0xf96): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/shmem.o: in function `userfaultfd_wp_async':
shmem.c:(.text+0x91): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/util.o: in function `userfaultfd_wp_async':
util.c:(.text+0x2b): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/vmstat.o: in function `userfaultfd_wp_async':
vmstat.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/compaction.o: in function `userfaultfd_wp_async':
compaction.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/workingset.o: in function `userfaultfd_wp_async':
workingset.c:(.text+0x181): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/debug.o: in function `userfaultfd_wp_async':
debug.c:(.text+0xb9): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/gup.o: in function `userfaultfd_wp_async':
gup.c:(.text+0x2ae): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/memory.o: in function `userfaultfd_wp_async':
memory.c:(.text+0x737): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mincore.o: in function `userfaultfd_wp_async':
mincore.c:(.text+0x149): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mlock.o: in function `userfaultfd_wp_async':
mlock.c:(.text+0x90e): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mmap.o: in function `userfaultfd_wp_async':
mmap.c:(.text+0x4d8): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mmu_gather.o: in function `userfaultfd_wp_async':
mmu_gather.c:(.text+0x29): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mprotect.o: in function `userfaultfd_wp_async':
mprotect.c:(.text+0x45): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/mremap.o: in function `userfaultfd_wp_async':
mremap.c:(.text+0x36b): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/page_vma_mapped.o: in function `userfaultfd_wp_async':
page_vma_mapped.c:(.text+0x28): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/pagewalk.o: in function `userfaultfd_wp_async':
pagewalk.c:(.text+0x30d): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/pgtable-generic.o: in function `userfaultfd_wp_async':
pgtable-generic.c:(.text+0x0): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/rmap.o: in function `userfaultfd_wp_async':
rmap.c:(.text+0x655): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/vmalloc.o: in function `userfaultfd_wp_async':
vmalloc.c:(.text+0x1546): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: mm/page_alloc.o: in function `userfaultfd_wp_async':
page_alloc.c:(.text+0xdb1): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: fs/splice.o: in function `userfaultfd_wp_async':
splice.c:(.text+0x6f8): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
ld: security/commoncap.o: in function `userfaultfd_wp_async':
commoncap.c:(.text+0x8b): multiple definition of `userfaultfd_wp_async'; arch/x86/kernel/setup.o:setup.c:(.text+0x3): first defined here
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
2023-01-24 10:43 ` kernel test robot
@ 2023-01-24 11:04 ` kernel test robot
2023-01-24 11:15 ` kernel test robot
2023-01-26 23:05 ` Peter Xu
3 siblings, 0 replies; 21+ messages in thread
From: kernel test robot @ 2023-01-24 11:04 UTC (permalink / raw)
To: Muhammad Usama Anjum, Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: oe-kbuild-all, Linux Memory Management List, Alexander Viro,
Shuah Khan, Christian Brauner, Yang Shi, Vlastimil Babka,
Liam R . Howlett, Yun Zhou, Suren Baghdasaryan, Alex Sierra,
Muhammad Usama Anjum, Matthew Wilcox, Pasha Tatashin,
Mike Rapoport, Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-kselftest,
Greg KH
Hi Muhammad,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on shuah-kselftest/next]
[also build test WARNING on shuah-kselftest/fixes linus/master v6.2-rc5 next-20230124]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next
patch link: https://lore.kernel.org/r/20230124084323.1363825-2-usama.anjum%40collabora.com
patch subject: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
config: um-i386_defconfig (https://download.01.org/0day-ci/archive/20230124/202301241800.pdnow11d-lkp@intel.com/config)
compiler: gcc-11 (Debian 11.3.0-8) 11.3.0
reproduce (this is a W=1 build):
# https://github.com/intel-lab-lkp/linux/commit/59e98aec663b7ca8fd5f3b3d2a0f17f777f425c4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
git checkout 59e98aec663b7ca8fd5f3b3d2a0f17f777f425c4
# save the config file
mkdir build_dir && cp config build_dir/.config
make W=1 O=build_dir ARCH=um SUBARCH=i386 olddefconfig
make W=1 O=build_dir ARCH=um SUBARCH=i386 SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
In file included from include/linux/mm_inline.h:9,
from kernel/fork.c:46:
>> include/linux/userfaultfd_k.h:278:5: warning: no previous prototype for 'userfaultfd_wp_async' [-Wmissing-prototypes]
278 | int userfaultfd_wp_async(struct vm_area_struct *vma)
| ^~~~~~~~~~~~~~~~~~~~
kernel/fork.c:162:13: warning: no previous prototype for 'arch_release_task_struct' [-Wmissing-prototypes]
162 | void __weak arch_release_task_struct(struct task_struct *tsk)
| ^~~~~~~~~~~~~~~~~~~~~~~~
kernel/fork.c:862:20: warning: no previous prototype for 'arch_task_cache_init' [-Wmissing-prototypes]
862 | void __init __weak arch_task_cache_init(void) { }
| ^~~~~~~~~~~~~~~~~~~~
kernel/fork.c:957:12: warning: no previous prototype for 'arch_dup_task_struct' [-Wmissing-prototypes]
957 | int __weak arch_dup_task_struct(struct task_struct *dst,
| ^~~~~~~~~~~~~~~~~~~~
--
In file included from include/linux/hugetlb.h:14,
from kernel/sysctl.c:46:
>> include/linux/userfaultfd_k.h:278:5: warning: no previous prototype for 'userfaultfd_wp_async' [-Wmissing-prototypes]
278 | int userfaultfd_wp_async(struct vm_area_struct *vma)
| ^~~~~~~~~~~~~~~~~~~~
--
In file included from include/linux/mm_inline.h:9,
from mm/memory.c:44:
>> include/linux/userfaultfd_k.h:278:5: warning: no previous prototype for 'userfaultfd_wp_async' [-Wmissing-prototypes]
278 | int userfaultfd_wp_async(struct vm_area_struct *vma)
| ^~~~~~~~~~~~~~~~~~~~
mm/memory.c: In function 'wp_huge_pmd':
mm/memory.c:4824:33: error: implicit declaration of function 'set_pmd_at'; did you mean 'set_pte_at'? [-Werror=implicit-function-declaration]
4824 | set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
| ^~~~~~~~~~
| set_pte_at
cc1: some warnings being treated as errors
--
In file included from include/linux/hugetlb.h:14,
from fs/proc/meminfo.c:6:
>> include/linux/userfaultfd_k.h:278:5: warning: no previous prototype for 'userfaultfd_wp_async' [-Wmissing-prototypes]
278 | int userfaultfd_wp_async(struct vm_area_struct *vma)
| ^~~~~~~~~~~~~~~~~~~~
fs/proc/meminfo.c:22:28: warning: no previous prototype for 'arch_report_meminfo' [-Wmissing-prototypes]
22 | void __attribute__((weak)) arch_report_meminfo(struct seq_file *m)
| ^~~~~~~~~~~~~~~~~~~
vim +/userfaultfd_wp_async +278 include/linux/userfaultfd_k.h
277
> 278 int userfaultfd_wp_async(struct vm_area_struct *vma)
279 {
280 return false;
281 }
282
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
2023-01-24 10:43 ` kernel test robot
2023-01-24 11:04 ` kernel test robot
@ 2023-01-24 11:15 ` kernel test robot
2023-01-26 23:05 ` Peter Xu
3 siblings, 0 replies; 21+ messages in thread
From: kernel test robot @ 2023-01-24 11:15 UTC (permalink / raw)
To: Muhammad Usama Anjum, Peter Xu, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov
Cc: oe-kbuild-all, Linux Memory Management List, Alexander Viro,
Shuah Khan, Christian Brauner, Yang Shi, Vlastimil Babka,
Liam R . Howlett, Yun Zhou, Suren Baghdasaryan, Alex Sierra,
Muhammad Usama Anjum, Matthew Wilcox, Pasha Tatashin,
Mike Rapoport, Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva,
Dan Williams, linux-kernel, linux-fsdevel, linux-kselftest,
Greg KH
Hi Muhammad,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on shuah-kselftest/next]
[also build test ERROR on shuah-kselftest/fixes linus/master v6.2-rc5 next-20230124]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git next
patch link: https://lore.kernel.org/r/20230124084323.1363825-2-usama.anjum%40collabora.com
patch subject: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
config: powerpc-allnoconfig (https://download.01.org/0day-ci/archive/20230124/202301241830.KlT5Xcjy-lkp@intel.com/config)
compiler: powerpc-linux-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/59e98aec663b7ca8fd5f3b3d2a0f17f777f425c4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230124-164601
git checkout 59e98aec663b7ca8fd5f3b3d2a0f17f777f425c4
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=powerpc olddefconfig
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=powerpc SHELL=/bin/bash arch/powerpc/kernel/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
In file included from include/linux/hugetlb.h:14,
from arch/powerpc/kernel/setup-common.c:37:
>> include/linux/userfaultfd_k.h:278:5: error: no previous prototype for 'userfaultfd_wp_async' [-Werror=missing-prototypes]
278 | int userfaultfd_wp_async(struct vm_area_struct *vma)
| ^~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
vim +/userfaultfd_wp_async +278 include/linux/userfaultfd_k.h
277
> 278 int userfaultfd_wp_async(struct vm_area_struct *vma)
279 {
280 return false;
281 }
282
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
` (2 preceding siblings ...)
2023-01-24 11:15 ` kernel test robot
@ 2023-01-26 23:05 ` Peter Xu
2023-01-27 6:47 ` Muhammad Usama Anjum
3 siblings, 1 reply; 21+ messages in thread
From: Peter Xu @ 2023-01-26 23:05 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: David Hildenbrand, Andrew Morton, Michał Mirosław,
Andrei Vagin, Danylo Mocherniuk, Paul Gofman, Cyrill Gorcunov,
Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Matthew Wilcox, Pasha Tatashin, Mike Rapoport,
Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva, Dan Williams,
linux-kernel, linux-fsdevel, linux-mm, linux-kselftest, Greg KH,
kernel
On Tue, Jan 24, 2023 at 01:43:20PM +0500, Muhammad Usama Anjum wrote:
> Add new WP Async mode (UFFD_FEATURE_WP_ASYNC) which resolves the page
> faults on its own. It can be used to track that which pages have been
> written-to from the time the pages were write-protected. It is very
> efficient way to track the changes as uffd is by nature pte/pmd based.
>
> UFFD synchronous WP sends the page faults to the userspace where the
> pages which have been written-to can be tracked. But it is not efficient.
> This is why this asynchronous version is being added. After setting the
> WP Async, the pages which have been written to can be found in the pagemap
> file or information can be obtained from the PAGEMAP_IOCTL.
>
> Suggested-by: Peter Xu <peterx@redhat.com>
> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> ---
> Changes in v7:
> - Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC
> - Handle automatic page fault resolution in better way (thanks to Peter)
> ---
> fs/userfaultfd.c | 11 +++++++++++
> include/linux/userfaultfd_k.h | 6 ++++++
> include/uapi/linux/userfaultfd.h | 8 +++++++-
> mm/memory.c | 29 +++++++++++++++++++++++++++--
> 4 files changed, 51 insertions(+), 3 deletions(-)
>
> diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
> index 15a5bf765d43..b82af02092ce 100644
> --- a/fs/userfaultfd.c
> +++ b/fs/userfaultfd.c
> @@ -1867,6 +1867,10 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
> mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
> mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
>
> + /* Write protection cannot be disabled in case of aync WP */
s/aync/async/
A slight reworded version:
/* Unprotection is not supported if in async WP mode */
> + if (!mode_wp && (ctx->features & UFFD_FEATURE_WP_ASYNC))
> + return -EINVAL;
> +
> if (mode_wp && mode_dontwake)
> return -EINVAL;
>
> @@ -1950,6 +1954,13 @@ static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
> return ret;
> }
>
> +int userfaultfd_wp_async(struct vm_area_struct *vma)
> +{
> + struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
> +
> + return (ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC));
> +}
> +
> static inline unsigned int uffd_ctx_features(__u64 user_features)
> {
> /*
> diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
> index 9df0b9a762cc..5db51fccae1d 100644
> --- a/include/linux/userfaultfd_k.h
> +++ b/include/linux/userfaultfd_k.h
> @@ -179,6 +179,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
> unsigned long end, struct list_head *uf);
> extern void userfaultfd_unmap_complete(struct mm_struct *mm,
> struct list_head *uf);
> +extern int userfaultfd_wp_async(struct vm_area_struct *vma);
>
> #else /* CONFIG_USERFAULTFD */
>
> @@ -274,6 +275,11 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
> return false;
> }
>
> +int userfaultfd_wp_async(struct vm_area_struct *vma)
> +{
> + return false;
> +}
> +
> #endif /* CONFIG_USERFAULTFD */
>
> static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
> diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
> index 005e5e306266..f4252ef40071 100644
> --- a/include/uapi/linux/userfaultfd.h
> +++ b/include/uapi/linux/userfaultfd.h
> @@ -38,7 +38,8 @@
> UFFD_FEATURE_MINOR_HUGETLBFS | \
> UFFD_FEATURE_MINOR_SHMEM | \
> UFFD_FEATURE_EXACT_ADDRESS | \
> - UFFD_FEATURE_WP_HUGETLBFS_SHMEM)
> + UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
> + UFFD_FEATURE_WP_ASYNC)
> #define UFFD_API_IOCTLS \
> ((__u64)1 << _UFFDIO_REGISTER | \
> (__u64)1 << _UFFDIO_UNREGISTER | \
> @@ -203,6 +204,10 @@ struct uffdio_api {
> *
> * UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd
> * write-protection mode is supported on both shmem and hugetlbfs.
> + *
> + * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
> + * asynchronous mode is supported in which the write fault is automatically
> + * resolved and write-protection is un-set.
> */
> #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
> #define UFFD_FEATURE_EVENT_FORK (1<<1)
> @@ -217,6 +222,7 @@ struct uffdio_api {
> #define UFFD_FEATURE_MINOR_SHMEM (1<<10)
> #define UFFD_FEATURE_EXACT_ADDRESS (1<<11)
> #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
> +#define UFFD_FEATURE_WP_ASYNC (1<<13)
> __u64 features;
>
> __u64 ioctls;
> diff --git a/mm/memory.c b/mm/memory.c
> index 4000e9f017e0..8c03b133d483 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>
> if (likely(!unshare)) {
> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
> + if (userfaultfd_wp_async(vma)) {
> + /*
> + * Nothing needed (cache flush, TLB invalidations,
> + * etc.) because we're only removing the uffd-wp bit,
> + * which is completely invisible to the user. This
> + * falls through to possible CoW.
Here it says it falls through to CoW, but..
> + */
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
> + pte_clear_uffd_wp(*vmf->pte));
> + return 0;
... it's not doing so. The original lines should do:
https://lore.kernel.org/all/Y8qq0dKIJBshua+X@x1n/
Side note: you cannot modify pgtable after releasing the pgtable lock.
It's racy.
> + }
> pte_unmap_unlock(vmf->pte, vmf->ptl);
> return handle_userfault(vmf, VM_UFFD_WP);
> }
> @@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
>
> if (vma_is_anonymous(vmf->vma)) {
> if (likely(!unshare) &&
> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
> - return handle_userfault(vmf, VM_UFFD_WP);
> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
> + if (userfaultfd_wp_async(vmf->vma)) {
> + /*
> + * Nothing needed (cache flush, TLB invalidations,
> + * etc.) because we're only removing the uffd-wp bit,
> + * which is completely invisible to the user. This
> + * falls through to possible CoW.
> + */
> + set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
> + pmd_clear_uffd_wp(*vmf->pmd));
This is for THP, not hugetlb.
Clearing uffd-wp bit here for the whole pmd is wrong to me, because we
track writes in small page sizes only. We should just split.
The relevant code for hugetlb resides in hugetlb_fault().
> + return 0;
> + } else {
> + return handle_userfault(vmf, VM_UFFD_WP);
> + }
> + }
> return do_huge_pmd_wp_page(vmf);
> }
>
> --
> 2.30.2
>
--
Peter Xu
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-26 23:05 ` Peter Xu
@ 2023-01-27 6:47 ` Muhammad Usama Anjum
2023-01-27 15:32 ` Peter Xu
0 siblings, 1 reply; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-27 6:47 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Suren Baghdasaryan, Alex Sierra, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
[-- Attachment #1: Type: text/plain, Size: 9531 bytes --]
On 1/27/23 4:05 AM, Peter Xu wrote:
> On Tue, Jan 24, 2023 at 01:43:20PM +0500, Muhammad Usama Anjum wrote:
>> Add new WP Async mode (UFFD_FEATURE_WP_ASYNC) which resolves the page
>> faults on its own. It can be used to track that which pages have been
>> written-to from the time the pages were write-protected. It is very
>> efficient way to track the changes as uffd is by nature pte/pmd based.
>>
>> UFFD synchronous WP sends the page faults to the userspace where the
>> pages which have been written-to can be tracked. But it is not efficient.
>> This is why this asynchronous version is being added. After setting the
>> WP Async, the pages which have been written to can be found in the pagemap
>> file or information can be obtained from the PAGEMAP_IOCTL.
>>
>> Suggested-by: Peter Xu <peterx@redhat.com>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>> ---
>> Changes in v7:
>> - Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC
>> - Handle automatic page fault resolution in better way (thanks to Peter)
>> ---
>> fs/userfaultfd.c | 11 +++++++++++
>> include/linux/userfaultfd_k.h | 6 ++++++
>> include/uapi/linux/userfaultfd.h | 8 +++++++-
>> mm/memory.c | 29 +++++++++++++++++++++++++++--
>> 4 files changed, 51 insertions(+), 3 deletions(-)
>>
>> diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
>> index 15a5bf765d43..b82af02092ce 100644
>> --- a/fs/userfaultfd.c
>> +++ b/fs/userfaultfd.c
>> @@ -1867,6 +1867,10 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
>> mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
>> mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
>>
>> + /* Write protection cannot be disabled in case of aync WP */
>
> s/aync/async/
>
> A slight reworded version:
>
> /* Unprotection is not supported if in async WP mode */
>
Will fix in next version.
>> + if (!mode_wp && (ctx->features & UFFD_FEATURE_WP_ASYNC))
>> + return -EINVAL;
>> +
>> if (mode_wp && mode_dontwake)
>> return -EINVAL;
>>
>> @@ -1950,6 +1954,13 @@ static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
>> return ret;
>> }
>>
>> +int userfaultfd_wp_async(struct vm_area_struct *vma)
>> +{
>> + struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
>> +
>> + return (ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC));
>> +}
>> +
>> static inline unsigned int uffd_ctx_features(__u64 user_features)
>> {
>> /*
>> diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
>> index 9df0b9a762cc..5db51fccae1d 100644
>> --- a/include/linux/userfaultfd_k.h
>> +++ b/include/linux/userfaultfd_k.h
>> @@ -179,6 +179,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
>> unsigned long end, struct list_head *uf);
>> extern void userfaultfd_unmap_complete(struct mm_struct *mm,
>> struct list_head *uf);
>> +extern int userfaultfd_wp_async(struct vm_area_struct *vma);
>>
>> #else /* CONFIG_USERFAULTFD */
>>
>> @@ -274,6 +275,11 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
>> return false;
>> }
>>
>> +int userfaultfd_wp_async(struct vm_area_struct *vma)
>> +{
>> + return false;
>> +}
>> +
>> #endif /* CONFIG_USERFAULTFD */
>>
>> static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
>> diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
>> index 005e5e306266..f4252ef40071 100644
>> --- a/include/uapi/linux/userfaultfd.h
>> +++ b/include/uapi/linux/userfaultfd.h
>> @@ -38,7 +38,8 @@
>> UFFD_FEATURE_MINOR_HUGETLBFS | \
>> UFFD_FEATURE_MINOR_SHMEM | \
>> UFFD_FEATURE_EXACT_ADDRESS | \
>> - UFFD_FEATURE_WP_HUGETLBFS_SHMEM)
>> + UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
>> + UFFD_FEATURE_WP_ASYNC)
>> #define UFFD_API_IOCTLS \
>> ((__u64)1 << _UFFDIO_REGISTER | \
>> (__u64)1 << _UFFDIO_UNREGISTER | \
>> @@ -203,6 +204,10 @@ struct uffdio_api {
>> *
>> * UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd
>> * write-protection mode is supported on both shmem and hugetlbfs.
>> + *
>> + * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
>> + * asynchronous mode is supported in which the write fault is automatically
>> + * resolved and write-protection is un-set.
>> */
>> #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
>> #define UFFD_FEATURE_EVENT_FORK (1<<1)
>> @@ -217,6 +222,7 @@ struct uffdio_api {
>> #define UFFD_FEATURE_MINOR_SHMEM (1<<10)
>> #define UFFD_FEATURE_EXACT_ADDRESS (1<<11)
>> #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
>> +#define UFFD_FEATURE_WP_ASYNC (1<<13)
>> __u64 features;
>>
>> __u64 ioctls;
>> diff --git a/mm/memory.c b/mm/memory.c
>> index 4000e9f017e0..8c03b133d483 100644
>> --- a/mm/memory.c
>> +++ b/mm/memory.c
>> @@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>>
>> if (likely(!unshare)) {
>> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
>> + if (userfaultfd_wp_async(vma)) {
>> + /*
>> + * Nothing needed (cache flush, TLB invalidations,
>> + * etc.) because we're only removing the uffd-wp bit,
>> + * which is completely invisible to the user. This
>> + * falls through to possible CoW.
>
> Here it says it falls through to CoW, but..
>
>> + */
>> + pte_unmap_unlock(vmf->pte, vmf->ptl);
>> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
>> + pte_clear_uffd_wp(*vmf->pte));
>> + return 0;
>
> ... it's not doing so. The original lines should do:
>
> https://lore.kernel.org/all/Y8qq0dKIJBshua+X@x1n/
>
> Side note: you cannot modify pgtable after releasing the pgtable lock.
> It's racy.
If I don't unlock and return after removing the UFFD_WP flag in case of
async wp, the target just gets stuck. Maybe the pte lock is not unlocked in
some path.
If I unlock and don't return, the crash happens.
So I'd put unlock and return from here. Please comment on the below patch
and what do you think should be done. I've missed something.
>
>> + }
>> pte_unmap_unlock(vmf->pte, vmf->ptl);
>> return handle_userfault(vmf, VM_UFFD_WP);
>> }
>> @@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
>>
>> if (vma_is_anonymous(vmf->vma)) {
>> if (likely(!unshare) &&
>> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
>> - return handle_userfault(vmf, VM_UFFD_WP);
>> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
>> + if (userfaultfd_wp_async(vmf->vma)) {
>> + /*
>> + * Nothing needed (cache flush, TLB invalidations,
>> + * etc.) because we're only removing the uffd-wp bit,
>> + * which is completely invisible to the user. This
>> + * falls through to possible CoW.
>> + */
>> + set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
>> + pmd_clear_uffd_wp(*vmf->pmd));
>
> This is for THP, not hugetlb.
>
> Clearing uffd-wp bit here for the whole pmd is wrong to me, because we
> track writes in small page sizes only. We should just split.
By detecting if the fault is async wp, just splitting the PMD doesn't work.
The below given snippit is working right now. But definately, the fault of
the whole PMD is being resolved which if we can bypass by correctly
splitting would be highly desirable. Can you please take a look on UFFD
side and suggest the changes? It would be much appreciated. I'm attaching
WIP v9 patches for you to apply on next(next-20230105) and pagemap_ioctl
selftest can be ran to test things after making changes.
>
> The relevant code for hugetlb resides in hugetlb_fault().
>
>> + return 0;
>> + } else {
>> + return handle_userfault(vmf, VM_UFFD_WP);
>> + }
>> + }
>> return do_huge_pmd_wp_page(vmf);
>> }
>>
>> --
>> 2.30.2
>>
>
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
if (likely(!unshare)) {
if (userfaultfd_pte_wp(vma, *vmf->pte)) {
+ if (userfaultfd_wp_async(vma)) {
+ set_pte_at(vma->vm_mm, vmf->address,
+ vmf->pte,
+ pte_clear_uffd_wp(*vmf->pte));
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+ }
pte_unmap_unlock(vmf->pte, vmf->ptl);
return handle_userfault(vmf, VM_UFFD_WP);
}
@@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault
*vmf)
if (vma_is_anonymous(vmf->vma)) {
if (likely(!unshare) &&
- userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
- return handle_userfault(vmf, VM_UFFD_WP);
+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
+ if (userfaultfd_wp_async(vmf->vma)) {
+ set_pmd_at(vmf->vma->vm_mm, vmf->address,
+ vmf->pmd,
+ pmd_clear_uffd_wp(*vmf->pmd));
+ return 0;
+ } else {
+ return handle_userfault(vmf, VM_UFFD_WP);
+ }
+ }
return do_huge_pmd_wp_page(vmf);
}
--
BR,
Muhammad Usama Anjum
[-- Attachment #2: 0000-cover-letter.patch --]
[-- Type: text/x-patch, Size: 5989 bytes --]
From 774e3cd2e67f5e94f9994a75aac3a3e13ccd6deb Mon Sep 17 00:00:00 2001
From: Muhammad Usama Anjum <usama.anjum@collabora.com>
Date: Fri, 27 Jan 2023 11:42:53 +0500
Subject: [PATCH WIP v9 0/4] Implement IOCTL to get and/or the clear info about
PTEs
*Changes in v9:*
- Fix build warnings and errors which were happening on some configs
- Simplify pagemap ioctl's code
*Changes in v8:*
- Update uffd async wp implementation
- Improve PAGEMAP_IOCTL implementation
*Changes in v7:*
- Add uffd wp async
- Update the IOCTL to use uffd under the hood instead of soft-dirty
flags
Hello,
Note:
Soft-dirty pages and pages which have been written-to are synonyms. As
kernel already has soft-dirty feature inside which we have given up to
use, we are using written-to terminology while using UFFD async WP under
the hood.
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are
supported in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WT),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
pages have been written-to.
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WT + PAGEMAP_WP_ENGAGE)
It is possible to find and clear soft-dirty pages entirely in userspace.
But it isn't efficient:
- The mprotect and SIGSEGV handler for bookkeeping
- The userfaultfd wp with the handler for bookkeeping
Some benchmarks can be seen here[1]. This series adds features that weren't
present earlier:
- There is no atomic get soft-dirty PTE bit status and clear present in
the kernel.
- The pages which have been written-to can not be found in accurate way.
(Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty
pages than there actually are.)
Historically, soft-dirty PTE bit tracking has been used in the CRIU
project. The procfs interface is enough for finding the soft-dirty bit
status and clearing the soft-dirty bit of all the pages of a process.
We have the use case where we need to track the soft-dirty PTE bit for
only specific pages on demand. We need this tracking and clear mechanism
of a region of memory while the process is running to emulate the
getWriteWatch() syscall of Windows.
*(Moved to using UFFD instead of soft-dirty to find pages which have been
written-to from v7 patch series)*:
Stop using the soft-dirty flags for finding which pages have been
written to. It is too delicate and wrong as it shows more soft-dirty
pages than the actual soft-dirty pages. There is no interest in
correcting it [2][3] as this is how the feature was written years ago.
It shouldn't be updated to changed behaviour. Peter Xu has suggested
using the async version of the UFFD WP [4] as it is based inherently
on the PTEs.
So in this patch series, I've added a new mode to the UFFD which is
asynchronous version of the write protect. When this variant of the
UFFD WP is used, the page faults are resolved automatically by the
kernel. The pages which have been written-to can be found by reading
pagemap file (!PM_UFFD_WP). This feature can be used successfully to
find which pages have been written to from the time the pages were
write protected. This works just like the soft-dirty flag without
showing any extra pages which aren't soft-dirty in reality.
The information related to pages if the page is file mapped, present and
swapped is required for the CRIU project [5][6]. The addition of the
required mask, any mask, excluded mask and return masks are also required
for the CRIU project [5].
The IOCTL returns the addresses of the pages which match the specific masks.
The page addresses are returned in struct page_region in a compact form.
The max_pages is needed to support a use case where user only wants to get
a specific number of pages. So there is no need to find all the pages of
interest in the range when max_pages is specified. The IOCTL returns when
the maximum number of the pages are found. The max_pages is optional. If
max_pages is specified, it must be equal or greater than the vec_size.
This restriction is needed to handle worse case when one page_region only
contains info of one page and it cannot be compacted. This is needed to
emulate the Windows getWriteWatch() syscall.
The patch series include the detailed selftest which can be used as an example
for the uffd async wp test and PAGEMAP_IOCTL. It shows the interface usages as
well.
[1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/
[2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.com
[3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.com
[4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n
[5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/
[6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/
Regards,
Muhammad Usama Anjum
Muhammad Usama Anjum (4):
userfaultfd: Add UFFD WP Async support
userfaultfd: split mwriteprotect_range()
fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about
PTEs
selftests: vm: add pagemap ioctl tests
fs/proc/task_mmu.c | 283 +++++++
fs/userfaultfd.c | 21 +
include/linux/userfaultfd_k.h | 17 +
include/uapi/linux/fs.h | 50 ++
include/uapi/linux/userfaultfd.h | 8 +-
mm/memory.c | 29 +-
mm/userfaultfd.c | 40 +-
tools/include/uapi/linux/fs.h | 50 ++
tools/testing/selftests/vm/.gitignore | 1 +
tools/testing/selftests/vm/Makefile | 5 +-
tools/testing/selftests/vm/pagemap_ioctl.c | 880 +++++++++++++++++++++
11 files changed, 1364 insertions(+), 20 deletions(-)
create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c
--
2.30.2
[-- Attachment #3: 0001-userfaultfd-Add-UFFD-WP-Async-support.patch --]
[-- Type: text/x-patch, Size: 6038 bytes --]
From 5fb45900be6104936d88c3a2e3bd14ea937dd920 Mon Sep 17 00:00:00 2001
From: Muhammad Usama Anjum <usama.anjum@collabora.com>
Date: Tue, 3 Jan 2023 12:47:31 +0500
Subject: [PATCH WIP v9 1/4] userfaultfd: Add UFFD WP Async support
Add new WP Async mode (UFFD_FEATURE_WP_ASYNC) which resolves the page
faults on its own. It can be used to track that which pages have been
written-to from the time the pages were write-protected. It is very
efficient way to track the changes as uffd is by nature pte/pmd based.
UFFD synchronous WP sends the page faults to the userspace where the
pages which have been written-to can be tracked. But it is not efficient.
This is why this asynchronous version is being added. After setting the
WP Async, the pages which have been written to can be found in the pagemap
file or information can be obtained from the PAGEMAP_IOCTL.
Suggested-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v7:
- Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC
- Handle automatic page fault resolution in better way (thanks to Peter)
---
fs/userfaultfd.c | 11 +++++++++++
include/linux/userfaultfd_k.h | 6 ++++++
include/uapi/linux/userfaultfd.h | 8 +++++++-
mm/memory.c | 29 +++++++++++++++++++++++++++--
4 files changed, 51 insertions(+), 3 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 15a5bf765d43..c17835a0e842 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -1867,6 +1867,10 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
+ /* The unprotection is not supported if in async WP mode */
+ if (!mode_wp && (ctx->features & UFFD_FEATURE_WP_ASYNC))
+ return -EINVAL;
+
if (mode_wp && mode_dontwake)
return -EINVAL;
@@ -1950,6 +1954,13 @@ static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
return ret;
}
+int userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+ struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
+
+ return (ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC));
+}
+
static inline unsigned int uffd_ctx_features(__u64 user_features)
{
/*
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index 9df0b9a762cc..94dcb4dc1b4a 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -179,6 +179,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
unsigned long end, struct list_head *uf);
extern void userfaultfd_unmap_complete(struct mm_struct *mm,
struct list_head *uf);
+extern int userfaultfd_wp_async(struct vm_area_struct *vma);
#else /* CONFIG_USERFAULTFD */
@@ -274,6 +275,11 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
return false;
}
+static inline int userfaultfd_wp_async(struct vm_area_struct *vma)
+{
+ return false;
+}
+
#endif /* CONFIG_USERFAULTFD */
static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
index 005e5e306266..f4252ef40071 100644
--- a/include/uapi/linux/userfaultfd.h
+++ b/include/uapi/linux/userfaultfd.h
@@ -38,7 +38,8 @@
UFFD_FEATURE_MINOR_HUGETLBFS | \
UFFD_FEATURE_MINOR_SHMEM | \
UFFD_FEATURE_EXACT_ADDRESS | \
- UFFD_FEATURE_WP_HUGETLBFS_SHMEM)
+ UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
+ UFFD_FEATURE_WP_ASYNC)
#define UFFD_API_IOCTLS \
((__u64)1 << _UFFDIO_REGISTER | \
(__u64)1 << _UFFDIO_UNREGISTER | \
@@ -203,6 +204,10 @@ struct uffdio_api {
*
* UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd
* write-protection mode is supported on both shmem and hugetlbfs.
+ *
+ * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
+ * asynchronous mode is supported in which the write fault is automatically
+ * resolved and write-protection is un-set.
*/
#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
#define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -217,6 +222,7 @@ struct uffdio_api {
#define UFFD_FEATURE_MINOR_SHMEM (1<<10)
#define UFFD_FEATURE_EXACT_ADDRESS (1<<11)
#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
+#define UFFD_FEATURE_WP_ASYNC (1<<13)
__u64 features;
__u64 ioctls;
diff --git a/mm/memory.c b/mm/memory.c
index 4000e9f017e0..da53fb53ddb7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
if (likely(!unshare)) {
if (userfaultfd_pte_wp(vma, *vmf->pte)) {
+ if (userfaultfd_wp_async(vma)) {
+ /*
+ * Nothing needed (cache flush, TLB invalidations,
+ * etc.) because we're only removing the uffd-wp bit,
+ * which is completely invisible to the user. This
+ * falls through to possible CoW.
+ */
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
+ pte_clear_uffd_wp(*vmf->pte));
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+ }
pte_unmap_unlock(vmf->pte, vmf->ptl);
return handle_userfault(vmf, VM_UFFD_WP);
}
@@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
if (vma_is_anonymous(vmf->vma)) {
if (likely(!unshare) &&
- userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
- return handle_userfault(vmf, VM_UFFD_WP);
+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
+ if (userfaultfd_wp_async(vmf->vma)) {
+ /*
+ * Nothing needed (cache flush, TLB invalidations,
+ * etc.) because we're only removing the uffd-wp bit,
+ * which is completely invisible to the user. This
+ * falls through to possible CoW.
+ */
+ set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
+ pmd_clear_uffd_wp(*vmf->pmd));
+ return 0;
+ } else {
+ return handle_userfault(vmf, VM_UFFD_WP);
+ }
+ }
return do_huge_pmd_wp_page(vmf);
}
--
2.30.2
[-- Attachment #4: 0002-userfaultfd-split-mwriteprotect_range.patch --]
[-- Type: text/x-patch, Size: 4513 bytes --]
From ee03645744e3129fb93022bfeb003b89252923ac Mon Sep 17 00:00:00 2001
From: Muhammad Usama Anjum <usama.anjum@collabora.com>
Date: Tue, 3 Jan 2023 12:48:59 +0500
Subject: [PATCH WIP v9 2/4] userfaultfd: split mwriteprotect_range()
Split mwriteprotect_range() to create a unlocked version. This
will be used in the next patch to write protect a memory area.
Add a helper function, wp_range_async() as well.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Changes in v7:
- Remove async being set in the PAGEMAP_IOCTL
---
fs/userfaultfd.c | 10 +++++++++
include/linux/userfaultfd_k.h | 11 ++++++++++
mm/userfaultfd.c | 40 ++++++++++++++++++++++-------------
3 files changed, 46 insertions(+), 15 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index c17835a0e842..8791a2d371c1 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -1961,6 +1961,16 @@ int userfaultfd_wp_async(struct vm_area_struct *vma)
return (ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC));
}
+int wp_range_async(struct vm_area_struct *vma, unsigned long start, unsigned long len)
+{
+ struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
+
+ if (!ctx)
+ return -1;
+
+ return __mwriteprotect_range(ctx->mm, start, len, true, &ctx->mmap_changing);
+}
+
static inline unsigned int uffd_ctx_features(__u64 user_features)
{
/*
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index 94dcb4dc1b4a..76407d21d03b 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -73,6 +73,9 @@ extern ssize_t mcopy_continue(struct mm_struct *dst_mm, unsigned long dst_start,
extern int mwriteprotect_range(struct mm_struct *dst_mm,
unsigned long start, unsigned long len,
bool enable_wp, atomic_t *mmap_changing);
+extern int __mwriteprotect_range(struct mm_struct *dst_mm,
+ unsigned long start, unsigned long len,
+ bool enable_wp, atomic_t *mmap_changing);
extern void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *vma,
unsigned long start, unsigned long len, bool enable_wp);
@@ -180,6 +183,8 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
extern void userfaultfd_unmap_complete(struct mm_struct *mm,
struct list_head *uf);
extern int userfaultfd_wp_async(struct vm_area_struct *vma);
+extern int wp_range_async(struct vm_area_struct *vma, unsigned long start,
+ unsigned long len);
#else /* CONFIG_USERFAULTFD */
@@ -280,6 +285,12 @@ static inline int userfaultfd_wp_async(struct vm_area_struct *vma)
return false;
}
+static inline int wp_range_async(struct vm_area_struct *vma, unsigned long start,
+ unsigned long len)
+{
+ return -1;
+}
+
#endif /* CONFIG_USERFAULTFD */
static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 65ad172add27..9d8a43faf764 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -734,25 +734,13 @@ void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *dst_vma,
tlb_finish_mmu(&tlb);
}
-int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
- unsigned long len, bool enable_wp,
- atomic_t *mmap_changing)
+int __mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
+ unsigned long len, bool enable_wp,
+ atomic_t *mmap_changing)
{
struct vm_area_struct *dst_vma;
unsigned long page_mask;
int err;
-
- /*
- * Sanitize the command parameters:
- */
- BUG_ON(start & ~PAGE_MASK);
- BUG_ON(len & ~PAGE_MASK);
-
- /* Does the address range wrap, or is the span zero-sized? */
- BUG_ON(start + len <= start);
-
- mmap_read_lock(dst_mm);
-
/*
* If memory mappings are changing because of non-cooperative
* operation (e.g. mremap) running in parallel, bail out and
@@ -783,6 +771,28 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
err = 0;
out_unlock:
+ return err;
+}
+
+int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,
+ unsigned long len, bool enable_wp,
+ atomic_t *mmap_changing)
+{
+ int err;
+
+ /*
+ * Sanitize the command parameters:
+ */
+ BUG_ON(start & ~PAGE_MASK);
+ BUG_ON(len & ~PAGE_MASK);
+
+ /* Does the address range wrap, or is the span zero-sized? */
+ BUG_ON(start + len <= start);
+
+ mmap_read_lock(dst_mm);
+
+ err = __mwriteprotect_range(dst_mm, start, len, enable_wp, mmap_changing);
+
mmap_read_unlock(dst_mm);
return err;
}
--
2.30.2
[-- Attachment #5: 0003-fs-proc-task_mmu-Implement-IOCTL-to-get-and-or-the-c.patch --]
[-- Type: text/x-patch, Size: 15761 bytes --]
From ff4453af31a0504cc09647bbfffc11eaf80b07b8 Mon Sep 17 00:00:00 2001
From: Muhammad Usama Anjum <usama.anjum@collabora.com>
Date: Tue, 3 Jan 2023 12:50:21 +0500
Subject: [PATCH WIP v9 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or
the clear info about PTEs
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are supported
in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WT),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
pages have been written-to.
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WT + PAGEMAP_WP_ENGAGE)
The uffd should have been registered on the memory range before performing
any WP/WT (Write Protect/Writtern-To) related operations with the IOCTL.
struct pagemap_scan_args is used as the argument of the IOCTL. In this
struct:
- The range is specified through start and len.
- The output buffer of struct page_region array and size is specified as
vec and vec_len.
- The optional maximum requested pages are specified in the max_pages.
- The flags can be specified in the flags field. The PAGEMAP_WP_ENGAGE
is the only added flag at this time.
- The masks are specified in required_mask, anyof_mask, excluded_ mask
and return_mask.
This IOCTL can be extended to get information about more PTE bits. This
IOCTL doesn't support hugetlbs at the moment. No information about
hugetlb can be obtained. This patch has evolved from a basic patch from
Gabriel Krisman Bertazi.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code
Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
Changes in v6:
- Rename variables and update comments
- Make IOCTL independent of soft_dirty config
- Change masks and bitmap type to _u64
- Improve code quality
Changes in v5:
- Remove tlb flushing even for clear operation
Changes in v4:
- Update the interface and implementation
Changes in v3:
- Tighten the user-kernel interface by using explicit types and add more
error checking
Changes in v2:
- Convert the interface from syscall to ioctl
- Remove pidfd support as it doesn't make sense in ioctl
---
fs/proc/task_mmu.c | 283 ++++++++++++++++++++++++++++++++++
include/uapi/linux/fs.h | 50 ++++++
tools/include/uapi/linux/fs.h | 50 ++++++
3 files changed, 383 insertions(+)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index e35a0398db63..64a6597aec06 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -19,6 +19,7 @@
#include <linux/shmem_fs.h>
#include <linux/uaccess.h>
#include <linux/pkeys.h>
+#include <linux/minmax.h>
#include <asm/elf.h>
#include <asm/tlb.h>
@@ -1135,6 +1136,22 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma,
}
#endif
+static inline bool is_pte_uffd_wp(pte_t pte)
+{
+ if ((pte_present(pte) && pte_uffd_wp(pte)) ||
+ (is_swap_pte(pte) && pte_swp_uffd_wp_any(pte)))
+ return true;
+ return false;
+}
+
+static inline bool is_pmd_uffd_wp(pmd_t pmd)
+{
+ if ((pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
+ (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)))
+ return true;
+ return false;
+}
+
#if defined(CONFIG_MEM_SOFT_DIRTY) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp)
@@ -1763,11 +1780,277 @@ static int pagemap_release(struct inode *inode, struct file *file)
return 0;
}
+#define PAGEMAP_OP_MASK (PAGE_IS_WT | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PAGEMAP_NONWT_OP_MASK (PAGE_IS_FILE | PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define IS_WP_ENGAGE_OP(a) (a->flags & PAGEMAP_WP_ENGAGE)
+#define IS_GET_OP(a) (a->vec)
+#define PAGEMAP_SCAN_BITMAP(wt, file, present, swap) \
+ (wt | file << 1 | present << 2 | swap << 3)
+#define IS_WT_REQUIRED(a) \
+ ((a->required_mask & PAGE_IS_WT) || \
+ (a->anyof_mask & PAGE_IS_WT))
+#define HAS_NO_SPACE(p) (p->max_pages && (p->found_pages == p->max_pages))
+
+struct pagemap_scan_private {
+ struct page_region *vec;
+ struct page_region prev;
+ unsigned long vec_len, vec_index;
+ unsigned int max_pages, found_pages, flags;
+ unsigned long required_mask, anyof_mask, excluded_mask, return_mask;
+};
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end, struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+
+ if (IS_WT_REQUIRED(p) && !userfaultfd_wp(vma) && !userfaultfd_wp_async(vma))
+ return -EPERM;
+ if (vma->vm_flags & VM_PFNMAP)
+ return 1;
+ return 0;
+}
+
+static inline int add_to_out(bool wt, bool file, bool pres, bool swap,
+ struct pagemap_scan_private *p, unsigned long addr, unsigned int len)
+{
+ unsigned long bitmap, cur = PAGEMAP_SCAN_BITMAP(wt, file, pres, swap);
+ bool cpy = true;
+ struct page_region *prev = &p->prev;
+
+ if (HAS_NO_SPACE(p))
+ return -ENOSPC;
+
+ if (p->max_pages && p->found_pages + len >= p->max_pages)
+ len = p->max_pages - p->found_pages;
+ if (!len)
+ return -EINVAL;
+
+ if (p->required_mask)
+ cpy = ((p->required_mask & cur) == p->required_mask);
+ if (cpy && p->anyof_mask)
+ cpy = (p->anyof_mask & cur);
+ if (cpy && p->excluded_mask)
+ cpy = !(p->excluded_mask & cur);
+ bitmap = cur & p->return_mask;
+ if (cpy && bitmap) {
+ if ((prev->len) && (prev->bitmap == bitmap) &&
+ (prev->start + prev->len * PAGE_SIZE == addr)) {
+ prev->len += len;
+ p->found_pages += len;
+ } else if (p->vec_index < p->vec_len) {
+ if (prev->len) {
+ memcpy(&p->vec[p->vec_index], prev, sizeof(struct page_region));
+ p->vec_index++;
+ }
+ prev->start = addr;
+ prev->len = len;
+ prev->bitmap = bitmap;
+ p->found_pages += len;
+ } else {
+ return -ENOSPC;
+ }
+ }
+ return 0;
+}
+
+static inline int export_prev_to_out(struct pagemap_scan_private *p, struct page_region __user *vec,
+ unsigned long *vec_index)
+{
+ struct page_region *prev = &p->prev;
+
+ if (prev->len) {
+ if (copy_to_user(&vec[*vec_index], prev, sizeof(struct page_region)))
+ return -EFAULT;
+ p->vec_index++;
+ (*vec_index)++;
+ prev->len = 0;
+ }
+ return 0;
+}
+
+static inline int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+ unsigned long end, struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ unsigned long addr = end;
+ spinlock_t *ptl;
+ int ret = 0, ret2;
+ pte_t *pte;
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ bool pmd_wt;
+
+ pmd_wt = !is_pmd_uffd_wp(*pmd);
+ /*
+ * Break huge page into small pages if operation needs to be performed is
+ * on a portion of the huge page.
+ */
+ if (pmd_wt && IS_WP_ENGAGE_OP(p) && (end - start < HPAGE_SIZE)) {
+ spin_unlock(ptl);
+ split_huge_pmd(vma, pmd, start);
+ goto process_smaller_pages;
+ }
+ if (IS_GET_OP(p))
+ ret = add_to_out(pmd_wt, vma->vm_file, pmd_present(*pmd),
+ is_swap_pmd(*pmd), p, start, (end - start)/PAGE_SIZE);
+ spin_unlock(ptl);
+ if (!ret) {
+ if (pmd_wt && IS_WP_ENGAGE_OP(p))
+ ret = wp_range_async(vma, start, HPAGE_SIZE);
+ }
+ return ret;
+ }
+process_smaller_pages:
+ if (pmd_trans_unstable(pmd))
+ return 0;
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+ if (IS_GET_OP(p)) {
+ for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE)
+ ret = add_to_out(!is_pte_uffd_wp(*pte), vma->vm_file, pte_present(*pte),
+ is_swap_pte(*pte), p, addr, 1);
+ }
+ pte_unmap_unlock(pte - 1, ptl);
+ if ((!ret || ret == -ENOSPC) && IS_WP_ENGAGE_OP(p))
+ ret2 = wp_range_async(vma, start, addr - start);
+ if (ret2)
+ ret = ret2;
+
+ cond_resched();
+ return ret;
+}
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, int depth,
+ struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ int ret = 0;
+
+ if (vma)
+ ret = add_to_out(false, vma->vm_file, false, false, p, addr,
+ (end - addr)/PAGE_SIZE);
+ return ret;
+}
+
+/* No hugetlb support is present. */
+static const struct mm_walk_ops pagemap_scan_ops = {
+ .test_walk = pagemap_scan_test_walk,
+ .pmd_entry = pagemap_scan_pmd_entry,
+ .pte_hole = pagemap_scan_pte_hole,
+};
+
+static long do_pagemap_cmd(struct mm_struct *mm, struct pagemap_scan_arg *arg)
+{
+ unsigned long empty_slots, vec_index = 0;
+ unsigned long __user start, end;
+ unsigned long __start, __end;
+ struct page_region __user *vec;
+ struct pagemap_scan_private p;
+ int ret;
+
+ start = (unsigned long)untagged_addr(arg->start);
+ vec = (struct page_region *)(unsigned long)untagged_addr(arg->vec);
+ if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
+ return -EINVAL;
+ if (IS_GET_OP(arg) && ((arg->vec_len == 0) ||
+ (!access_ok((void __user *)vec, arg->vec_len * sizeof(struct page_region)))))
+ return -ENOMEM;
+ if ((arg->flags & ~PAGEMAP_WP_ENGAGE) || (arg->required_mask & ~PAGEMAP_OP_MASK) ||
+ (arg->anyof_mask & ~PAGEMAP_OP_MASK) || (arg->excluded_mask & ~PAGEMAP_OP_MASK) ||
+ (arg->return_mask & ~PAGEMAP_OP_MASK))
+ return -EINVAL;
+ if (IS_GET_OP(arg) && ((!arg->required_mask && !arg->anyof_mask && !arg->excluded_mask) ||
+ !arg->return_mask))
+ return -EINVAL;
+ /* The non-WT flags cannot be obtained if PAGEMAP_WP_ENGAGE is also specified. */
+ if (IS_WP_ENGAGE_OP(arg) && ((arg->required_mask & PAGEMAP_NONWT_OP_MASK) ||
+ (arg->anyof_mask & PAGEMAP_NONWT_OP_MASK)))
+ return -EINVAL;
+
+ end = start + arg->len;
+ p.max_pages = arg->max_pages;
+ p.found_pages = 0;
+ p.flags = arg->flags;
+ p.required_mask = arg->required_mask;
+ p.anyof_mask = arg->anyof_mask;
+ p.excluded_mask = arg->excluded_mask;
+ p.return_mask = arg->return_mask;
+ p.prev.len = 0;
+ p.vec_len = (PAGEMAP_WALK_SIZE >> PAGE_SHIFT);
+
+ if (IS_GET_OP(arg)) {
+ p.vec = kmalloc_array(p.vec_len, sizeof(struct page_region), GFP_KERNEL);
+ if (!p.vec)
+ return -ENOMEM;
+ } else {
+ p.vec = NULL;
+ }
+ __start = __end = start;
+ while (!ret && __end < end) {
+ p.vec_index = 0;
+ empty_slots = arg->vec_len - vec_index;
+ if (p.vec_len > empty_slots)
+ p.vec_len = empty_slots;
+
+ __end = (__start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
+ if (__end > end)
+ __end = end;
+
+ mmap_read_lock(mm);
+ ret = walk_page_range(mm, __start, __end, &pagemap_scan_ops, &p);
+ mmap_read_unlock(mm);
+ if (!(!ret || ret == -ENOSPC))
+ goto free_data;
+
+ __start = __end;
+ if (IS_GET_OP(arg) && p.vec_index) {
+ if (copy_to_user(&vec[vec_index], p.vec,
+ p.vec_index * sizeof(struct page_region))) {
+ ret = -EFAULT;
+ goto free_data;
+ }
+ vec_index += p.vec_index;
+ }
+ }
+ if (!ret || ret == -ENOSPC)
+ ret = export_prev_to_out(&p, vec, &vec_index);
+ if (!ret)
+ ret = vec_index;
+free_data:
+ if (IS_GET_OP(arg))
+ kfree(p.vec);
+
+ return ret;
+}
+
+static long pagemap_scan_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+ struct pagemap_scan_arg __user *uarg = (struct pagemap_scan_arg __user *)arg;
+ struct mm_struct *mm = file->private_data;
+ struct pagemap_scan_arg argument;
+
+ if (cmd == PAGEMAP_SCAN) {
+ if (copy_from_user(&argument, uarg, sizeof(struct pagemap_scan_arg)))
+ return -EFAULT;
+ return do_pagemap_cmd(mm, &argument);
+ }
+ return -EINVAL;
+}
+
const struct file_operations proc_pagemap_operations = {
.llseek = mem_lseek, /* borrow this */
.read = pagemap_read,
.open = pagemap_open,
.release = pagemap_release,
+ .unlocked_ioctl = pagemap_scan_ioctl,
+ .compat_ioctl = pagemap_scan_ioctl,
};
#endif /* CONFIG_PROC_PAGE_MONITOR */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index b7b56871029c..6d03a903a745 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
+#define PAGE_IS_WT (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pagemap_scan_arg - Pagemap ioctl argument
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @flags: Flags for the IOCTL
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pagemap_scan_arg {
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u32 max_pages;
+ __u32 flags;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Special flags */
+#define PAGEMAP_WP_ENGAGE (1 << 0)
+
#endif /* _UAPI_LINUX_FS_H */
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index b7b56871029c..6d03a903a745 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
+#define PAGE_IS_WT (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pagemap_scan_arg - Pagemap ioctl argument
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @flags: Flags for the IOCTL
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pagemap_scan_arg {
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u32 max_pages;
+ __u32 flags;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Special flags */
+#define PAGEMAP_WP_ENGAGE (1 << 0)
+
#endif /* _UAPI_LINUX_FS_H */
--
2.30.2
[-- Attachment #6: 0004-selftests-vm-add-pagemap-ioctl-tests.patch --]
[-- Type: text/x-patch, Size: 33946 bytes --]
From 774e3cd2e67f5e94f9994a75aac3a3e13ccd6deb Mon Sep 17 00:00:00 2001
From: Muhammad Usama Anjum <usama.anjum@collabora.com>
Date: Tue, 24 Jan 2023 13:36:23 +0500
Subject: [PATCH WIP v9 4/4] selftests: vm: add pagemap ioctl tests
Add pagemap ioctl tests. Add several different types of tests to judge
the correction of the interface.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Chages in v7:
- Add and update all test cases
Changes in v6:
- Rename variables
Changes in v4:
- Updated all the tests to conform to new IOCTL
Changes in v3:
- Add another test to do sanity of flags
Changes in v2:
- Update the tests to use the ioctl interface instead of syscall
TAP version 13
1..54
ok 1 sanity_tests_sd wrong flag specified
ok 2 sanity_tests_sd wrong mask specified
ok 3 sanity_tests_sd wrong return mask specified
ok 4 sanity_tests_sd mixture of correct and wrong flag
ok 5 sanity_tests_sd Clear area with larger vec size
ok 6 sanity_tests_sd Repeated pattern of dirty and non-dirty pages
ok 7 sanity_tests_sd Repeated pattern of dirty and non-dirty pages in parts
ok 8 sanity_tests_sd Two regions
ok 9 Page testing: all new pages must be soft dirty
ok 10 Page testing: all pages must not be soft dirty
ok 11 Page testing: all pages dirty other than first and the last one
ok 12 Page testing: only middle page dirty
ok 13 Page testing: only two middle pages dirty
ok 14 Page testing: only get 2 dirty pages and clear them as well
ok 15 Page testing: Range clear only
ok 16 Large Page testing: all new pages must be soft dirty
ok 17 Large Page testing: all pages must not be soft dirty
ok 18 Large Page testing: all pages dirty other than first and the last one
ok 19 Large Page testing: only middle page dirty
ok 20 Large Page testing: only two middle pages dirty
ok 21 Large Page testing: only get 2 dirty pages and clear them as well
ok 22 Large Page testing: Range clear only
ok 23 Huge page testing: all new pages must be soft dirty
ok 24 Huge page testing: all pages must not be soft dirty
ok 25 Huge page testing: all pages dirty other than first and the last one
ok 26 Huge page testing: only middle page dirty
ok 27 Huge page testing: only two middle pages dirty
ok 28 Huge page testing: only get 2 dirty pages and clear them as well
ok 29 Huge page testing: Range clear only
ok 30 hpage_unit_tests all new huge page must be dirty
ok 31 hpage_unit_tests all the huge page must not be dirty
ok 32 hpage_unit_tests all the huge page must be dirty and clear
ok 33 hpage_unit_tests only middle page dirty
ok 34 hpage_unit_tests clear first half of huge page
ok 35 hpage_unit_tests clear first half of huge page with limited buffer
ok 36 hpage_unit_tests clear second half huge page
ok 37 Test test_simple
ok 38 mprotect_tests Both pages dirty
ok 39 mprotect_tests Both pages are not soft dirty
ok 40 mprotect_tests Both pages dirty after remap and mprotect
ok 41 mprotect_tests Clear and make the pages dirty
ok 42 sanity_tests clear op can only be specified with PAGE_IS_WT
ok 43 sanity_tests required_mask specified
ok 44 sanity_tests anyof_mask specified
ok 45 sanity_tests excluded_mask specified
ok 46 sanity_tests required_mask and anyof_mask specified
ok 47 sanity_tests Get sd and present pages with anyof_mask
ok 48 sanity_tests Get all the pages with required_mask
ok 49 sanity_tests Get sd and present pages with required_mask and anyof_mask
ok 50 sanity_tests Don't get sd pages
ok 51 sanity_tests Don't get present pages
ok 52 sanity_tests Find dirty present pages with return mask
ok 53 sanity_tests Memory mapped file
ok 54 unmapped_region_tests Get status of pages
# Totals: pass:54 fail:0 xfail:0 xpass:0 skip:0 error:0
---
tools/testing/selftests/vm/.gitignore | 1 +
tools/testing/selftests/vm/Makefile | 5 +-
tools/testing/selftests/vm/pagemap_ioctl.c | 880 +++++++++++++++++++++
3 files changed, 884 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c
diff --git a/tools/testing/selftests/vm/.gitignore b/tools/testing/selftests/vm/.gitignore
index 1f8c36a9fa10..9e7e0ae26582 100644
--- a/tools/testing/selftests/vm/.gitignore
+++ b/tools/testing/selftests/vm/.gitignore
@@ -17,6 +17,7 @@ mremap_dontunmap
mremap_test
on-fault-limit
transhuge-stress
+pagemap_ioctl
protection_keys
protection_keys_32
protection_keys_64
diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile
index 89c14e41bd43..54c074440a1b 100644
--- a/tools/testing/selftests/vm/Makefile
+++ b/tools/testing/selftests/vm/Makefile
@@ -24,9 +24,8 @@ MACHINE ?= $(shell echo $(uname_M) | sed -e 's/aarch64.*/arm64/' -e 's/ppc64.*/p
# things despite using incorrect values such as an *occasionally* incomplete
# LDLIBS.
MAKEFLAGS += --no-builtin-rules
-
CFLAGS = -Wall -I $(top_srcdir) -I $(top_srcdir)/usr/include $(EXTRA_CFLAGS) $(KHDR_INCLUDES)
-LDLIBS = -lrt -lpthread
+LDLIBS = -lrt -lpthread -lm
TEST_GEN_FILES = cow
TEST_GEN_FILES += compaction_test
TEST_GEN_FILES += gup_test
@@ -52,6 +51,7 @@ TEST_GEN_FILES += on-fault-limit
TEST_GEN_FILES += thuge-gen
TEST_GEN_FILES += transhuge-stress
TEST_GEN_FILES += userfaultfd
+TEST_GEN_PROGS += pagemap_ioctl
TEST_GEN_PROGS += soft-dirty
TEST_GEN_PROGS += split_huge_page_test
TEST_GEN_FILES += ksm_tests
@@ -103,6 +103,7 @@ $(OUTPUT)/cow: vm_util.c
$(OUTPUT)/khugepaged: vm_util.c
$(OUTPUT)/ksm_functional_tests: vm_util.c
$(OUTPUT)/madv_populate: vm_util.c
+$(OUTPUT)/pagemap_ioctl: vm_util.c
$(OUTPUT)/soft-dirty: vm_util.c
$(OUTPUT)/split_huge_page_test: vm_util.c
$(OUTPUT)/userfaultfd: vm_util.c
diff --git a/tools/testing/selftests/vm/pagemap_ioctl.c b/tools/testing/selftests/vm/pagemap_ioctl.c
new file mode 100644
index 000000000000..877646f9ee8a
--- /dev/null
+++ b/tools/testing/selftests/vm/pagemap_ioctl.c
@@ -0,0 +1,880 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <stdio.h>
+#include <fcntl.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <malloc.h>
+#include "vm_util.h"
+#include "../kselftest.h"
+#include <linux/types.h>
+#include <linux/userfaultfd.h>
+#include <linux/fs.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <math.h>
+#include <asm/unistd.h>
+
+#define PAGEMAP_OP_MASK (PAGE_IS_WT | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PAGEMAP_NON_WT_MASK (PAGE_IS_FILE | PAGE_IS_PRESENT | \
+ PAGE_IS_SWAPPED)
+
+#define TEST_ITERATIONS 10
+#define PAGEMAP "/proc/self/pagemap"
+int pagemap_fd;
+int uffd;
+int page_size;
+int hpage_size;
+
+static long pagemap_ioctl(void *start, int len, void *vec, int vec_len, int flag,
+ int max_pages, long required_mask, long anyof_mask, long excluded_mask,
+ long return_mask)
+{
+ struct pagemap_scan_arg arg;
+ int ret;
+
+ arg.start = (uintptr_t)start;
+ arg.len = len;
+ arg.vec = (uintptr_t)vec;
+ arg.vec_len = vec_len;
+ arg.flags = flag;
+ arg.max_pages = max_pages;
+ arg.required_mask = required_mask;
+ arg.anyof_mask = anyof_mask;
+ arg.excluded_mask = excluded_mask;
+ arg.return_mask = return_mask;
+
+ ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+
+ return ret;
+}
+
+int init_uffd(void)
+{
+ struct uffdio_api uffdio_api;
+
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd == -1)
+ ksft_exit_fail_msg("uffd syscall failed\n");
+
+ uffdio_api.api = UFFD_API;
+ uffdio_api.features = UFFD_FEATURE_WP_ASYNC;
+ if (ioctl(uffd, UFFDIO_API, &uffdio_api))
+ ksft_exit_fail_msg("UFFDIO_API\n");
+
+ if (uffdio_api.api != UFFD_API)
+ ksft_exit_fail_msg("UFFDIO_API error %llu\n", uffdio_api.api);
+
+ return 0;
+}
+
+int wp_init(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_register uffdio_register;
+ struct uffdio_writeprotect wp;
+
+ /* TODO: can it be avoided? Write protect doesn't engage on the pages if they aren't
+ * present already. The pages can be made present by writing to them.
+ */
+ memset(lpBaseAddress, -1, dwRegionSize);
+
+ uffdio_register.range.start = (unsigned long)lpBaseAddress;
+ uffdio_register.range.len = dwRegionSize;
+ uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+ if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register) == -1)
+ ksft_exit_fail_msg("ioctl(UFFDIO_REGISTER)\n");
+
+ if (!(uffdio_register.ioctls & UFFDIO_WRITEPROTECT))
+ ksft_exit_fail_msg("ioctl set is incorrect\n");
+
+ if (rand() % 2) {
+ wp.range.start = (unsigned long)lpBaseAddress;
+ wp.range.len = dwRegionSize;
+ wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+ if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp) == -1)
+ ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+ } else {
+ if (pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0) < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", 1, errno, strerror(errno));
+ }
+ return 0;
+}
+
+int wp_free(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_register uffdio_register;
+
+ uffdio_register.range.start = (unsigned long)lpBaseAddress;
+ uffdio_register.range.len = dwRegionSize;
+ uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+ if (ioctl(uffd, UFFDIO_UNREGISTER, &uffdio_register.range))
+ ksft_exit_fail_msg("ioctl unregister failure\n");
+ return 0;
+}
+
+int clear_softdirty_wp(void *lpBaseAddress, int dwRegionSize)
+{
+ struct uffdio_writeprotect wp;
+
+ if (rand() % 2) {
+ wp.range.start = (unsigned long)lpBaseAddress;
+ wp.range.len = dwRegionSize;
+ wp.mode = UFFDIO_WRITEPROTECT_MODE_WP;
+
+ if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp) == -1)
+ ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n");
+ } else {
+ if (pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0) < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", 1, errno, strerror(errno));
+ }
+ return 0;
+}
+
+int sanity_tests_sd(void)
+{
+ char *mem, *m[2];
+ int mem_size, vec_size, ret, ret2, ret3, i, num_pages = 10;
+ struct page_region *vec;
+
+ vec_size = 100;
+ mem_size = num_pages * page_size;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ if (!vec)
+ ksft_exit_fail_msg("error nomem\n");
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(mem, mem_size);
+
+ /* 1. wrong operation */
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, -1,
+ 0, PAGE_IS_WT, 0, 0, PAGE_IS_WT) < 0,
+ "%s wrong flag specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 8,
+ 0, 0x1111, 0, 0, PAGE_IS_WT) < 0,
+ "%s wrong mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0,
+ 0, PAGE_IS_WT, 0, 0, 0x1000) < 0,
+ "%s wrong return mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size,
+ PAGEMAP_WP_ENGAGE | 0x32,
+ 0, PAGE_IS_WT, 0, 0, PAGE_IS_WT) < 0,
+ "%s mixture of correct and wrong flag\n", __func__);
+
+ /* 2. Clear area with larger vec size */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ ksft_test_result(ret >= 0, "%s Clear area with larger vec size\n", __func__);
+
+ /* 3. Repeated pattern of dirty and non-dirty pages */
+ for (i = 0; i < mem_size; i += 2 * page_size)
+ mem[i]++;
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == mem_size/(page_size * 2),
+ "%s Repeated pattern of dirty and non-dirty pages\n", __func__);
+
+ /* 4. Repeated pattern of dirty and non-dirty pages in parts */
+ ret = pagemap_ioctl(mem, mem_size, vec, num_pages/5, PAGEMAP_WP_ENGAGE,
+ num_pages/2 - 2, PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret2 = pagemap_ioctl(mem, mem_size, vec, 2, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno));
+
+ ret3 = pagemap_ioctl(mem, mem_size, vec, num_pages/2, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret3 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret3, errno, strerror(errno));
+
+ ksft_test_result((ret + ret3) == num_pages/2 && ret2 == 2,
+ "%s Repeated pattern of dirty and non-dirty pages in parts\n", __func__);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 5. Two regions */
+ m[0] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (m[0] == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ m[1] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (m[1] == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(m[0], mem_size);
+ wp_init(m[1], mem_size);
+
+ memset(m[0], 'a', mem_size);
+ memset(m[1], 'b', mem_size);
+
+ ret = pagemap_ioctl(m[0], mem_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(m[1], mem_size, vec, 1, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == mem_size/page_size,
+ "%s Two regions\n", __func__);
+
+ wp_free(m[0], mem_size);
+ wp_free(m[1], mem_size);
+ munmap(m[0], mem_size);
+ munmap(m[1], mem_size);
+
+ free(vec);
+ return 0;
+}
+
+int base_tests(char *prefix, char *mem, int mem_size, int skip)
+{
+ int vec_size, ret, dirty, dirty2;
+ struct page_region *vec, *vec2;
+
+ if (skip) {
+ ksft_test_result_skip("%s all new pages must be soft dirty\n", prefix);
+ ksft_test_result_skip("%s all pages must not be soft dirty\n", prefix);
+ ksft_test_result_skip("%s all pages dirty other than first and the last one\n",
+ prefix);
+ ksft_test_result_skip("%s only middle page dirty\n", prefix);
+ ksft_test_result_skip("%s only two middle pages dirty\n", prefix);
+ ksft_test_result_skip("%s only get 2 dirty pages and clear them as well\n", prefix);
+ ksft_test_result_skip("%s Range clear only\n", prefix);
+ return 0;
+ }
+
+ vec_size = mem_size/page_size;
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ vec2 = malloc(sizeof(struct page_region) * vec_size);
+
+ /* 1. all new pages must be not be soft dirty */
+ dirty = pagemap_ioctl(mem, mem_size, vec, 1, PAGEMAP_WP_ENGAGE, vec_size - 2,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ dirty2 = pagemap_ioctl(mem, mem_size, vec2, 1, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty2, errno, strerror(errno));
+
+ ksft_test_result(dirty == 0 && dirty2 == 0,
+ "%s all new pages must be soft dirty\n", prefix);
+
+ /* 2. all pages must not be soft dirty */
+ dirty = pagemap_ioctl(mem, mem_size, vec, 1, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 0, "%s all pages must not be soft dirty\n", prefix);
+
+ /* 3. all pages dirty other than first and the last one */
+ memset(mem + page_size, 0, mem_size - (2 * page_size));
+
+ dirty = pagemap_ioctl(mem, mem_size, vec, 1, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 1 && vec[0].len >= vec_size - 2 && vec[0].len <= vec_size,
+ "%s all pages dirty other than first and the last one\n", prefix);
+
+ /* 4. only middle page dirty */
+ clear_softdirty_wp(mem, mem_size);
+ mem[vec_size/2 * page_size]++;
+
+ dirty = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0, PAGE_IS_WT, 0, 0,
+ PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 1 && vec[0].len >= 1,
+ "%s only middle page dirty\n", prefix);
+
+ /* 5. only two middle pages dirty and walk over only middle pages */
+ clear_softdirty_wp(mem, mem_size);
+ mem[vec_size/2 * page_size]++;
+ mem[(vec_size/2 + 1) * page_size]++;
+
+ dirty = pagemap_ioctl(&mem[vec_size/2 * page_size], 2 * page_size, vec, 1, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty == 1 && vec[0].start == (uintptr_t)(&mem[vec_size/2 * page_size]) &&
+ vec[0].len == 2,
+ "%s only two middle pages dirty\n", prefix);
+
+ /* 6. only get 2 dirty pages and clear them as well */
+ memset(mem, -1, mem_size);
+
+ /* get and clear second and third pages */
+ ret = pagemap_ioctl(mem + page_size, 2 * page_size, vec, 1, PAGEMAP_WP_ENGAGE,
+ 2, PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ dirty = pagemap_ioctl(mem, mem_size, vec2, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == 2 &&
+ vec[0].start == (uintptr_t)(mem + page_size) &&
+ dirty == 2 && vec2[0].len == 1 && vec2[0].start == (uintptr_t)mem &&
+ vec2[1].len == vec_size - 3 &&
+ vec2[1].start == (uintptr_t)(mem + 3 * page_size),
+ "%s only get 2 dirty pages and clear them as well\n", prefix);
+
+ /* 7. Range clear only */
+ memset(mem, -1, mem_size);
+
+ dirty = pagemap_ioctl(mem, mem_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ dirty2 = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (dirty2 < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty2, errno, strerror(errno));
+
+ ksft_test_result(dirty == 0 && dirty2 == 0, "%s Range clear only\n",
+ prefix);
+
+ free(vec);
+ free(vec2);
+ return 0;
+}
+
+void *gethugepage(int map_size)
+{
+ int ret;
+ char *map;
+
+ map = memalign(hpage_size, map_size);
+ if (!map)
+ ksft_exit_fail_msg("memalign failed %d %s\n", errno, strerror(errno));
+
+ ret = madvise(map, map_size, MADV_HUGEPAGE);
+ if (ret)
+ ksft_exit_fail_msg("madvise failed %d %d %s\n", ret, errno, strerror(errno));
+
+ wp_init(map, map_size);
+
+ if (check_huge_anon(map, map_size/hpage_size, hpage_size))
+ return map;
+
+ free(map);
+ return NULL;
+
+}
+
+int hpage_unit_tests(void)
+{
+ char *map;
+ int ret;
+ size_t num_pages = 10;
+ int map_size = hpage_size * num_pages;
+ int vec_size = map_size/page_size;
+ struct page_region *vec, *vec2;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ vec2 = malloc(sizeof(struct page_region) * vec_size);
+ if (!vec || !vec2)
+ ksft_exit_fail_msg("malloc failed\n");
+
+ map = gethugepage(map_size);
+ if (map) {
+ /* 1. all new huge page must not be dirty */
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 0, "%s all new huge page must be dirty\n", __func__);
+
+ /* 2. all the huge page must not be dirty */
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 0, "%s all the huge page must not be dirty\n", __func__);
+
+ /* 3. all the huge page must be dirty and clear dirty as well */
+ memset(map, -1, map_size);
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].start == (uintptr_t)map &&
+ vec[0].len == vec_size && vec[0].bitmap == PAGE_IS_WT,
+ "%s all the huge page must be dirty and clear\n", __func__);
+
+ /* 4. only middle page dirty */
+ wp_free(map, map_size);
+ free(map);
+ map = gethugepage(map_size);
+ wp_init(map, map_size);
+ clear_softdirty_wp(map, map_size);
+ map[vec_size/2 * page_size]++;
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len > 0,
+ "%s only middle page dirty\n", __func__);
+
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("all new huge page must be dirty\n");
+ ksft_test_result_skip("all the huge page must not be dirty\n");
+ ksft_test_result_skip("all the huge page must be dirty and clear\n");
+ ksft_test_result_skip("only middle page dirty\n");
+ }
+
+ /* 5. clear first half of huge page */
+ map = gethugepage(map_size);
+ if (map) {
+
+ memset(map, 0, map_size);
+
+ ret = pagemap_ioctl(map, map_size/2, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+ vec[0].start == (uintptr_t)(map + map_size/2),
+ "%s clear first half of huge page\n", __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("clear first half of huge page\n");
+ }
+
+ /* 6. clear first half of huge page with limited buffer */
+ map = gethugepage(map_size);
+ if (map) {
+ memset(map, 0, map_size);
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, PAGEMAP_WP_ENGAGE,
+ vec_size/2, PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2 &&
+ vec[0].start == (uintptr_t)(map + map_size/2),
+ "%s clear first half of huge page with limited buffer\n",
+ __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("clear first half of huge page with limited buffer\n");
+ }
+
+ /* 7. clear second half of huge page */
+ map = gethugepage(map_size);
+ if (map) {
+ memset(map, -1, map_size);
+ ret = pagemap_ioctl(map + map_size/2, map_size/2, NULL, 0, PAGEMAP_WP_ENGAGE,
+ 0, 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ret = pagemap_ioctl(map, map_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec[0].len == vec_size/2,
+ "%s clear second half huge page\n", __func__);
+ wp_free(map, map_size);
+ free(map);
+ } else {
+ ksft_test_result_skip("clear second half huge page\n");
+ }
+
+ free(vec);
+ free(vec2);
+ return 0;
+}
+
+int unmapped_region_tests(void)
+{
+ void *start = (void *)0x10000000;
+ int dirty, len = 0x00040000;
+ int vec_size = len / page_size;
+ struct page_region *vec = malloc(sizeof(struct page_region) * vec_size);
+
+ /* 1. Get dirty pages */
+ dirty = pagemap_ioctl(start, len, vec, vec_size, 0, 0, PAGEMAP_NON_WT_MASK, 0, 0,
+ PAGEMAP_NON_WT_MASK);
+ if (dirty < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", dirty, errno, strerror(errno));
+
+ ksft_test_result(dirty >= 0, "%s Get status of pages\n", __func__);
+
+ free(vec);
+ return 0;
+}
+
+static void test_simple(void)
+{
+ int i;
+ char *map;
+ struct page_region vec;
+
+ map = aligned_alloc(page_size, page_size);
+ if (!map)
+ ksft_exit_fail_msg("aligned_alloc failed\n");
+ wp_init(map, page_size);
+
+ clear_softdirty_wp(map, page_size);
+
+ for (i = 0 ; i < TEST_ITERATIONS; i++) {
+ if (pagemap_ioctl(map, page_size, &vec, 1, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT) == 1) {
+ ksft_print_msg("dirty bit was 1, but should be 0 (i=%d)\n", i);
+ break;
+ }
+
+ clear_softdirty_wp(map, page_size);
+ /* Write something to the page to get the dirty bit enabled on the page */
+ map[0]++;
+
+ if (pagemap_ioctl(map, page_size, &vec, 1, 0, 0,
+ PAGE_IS_WT, 0, 0, PAGE_IS_WT) == 0) {
+ ksft_print_msg("dirty bit was 0, but should be 1 (i=%d)\n", i);
+ break;
+ }
+
+ clear_softdirty_wp(map, page_size);
+ }
+ wp_free(map, page_size);
+ free(map);
+
+ ksft_test_result(i == TEST_ITERATIONS, "Test %s\n", __func__);
+}
+
+int sanity_tests(void)
+{
+ char *mem, *fmem;
+ int mem_size, vec_size, ret;
+ struct page_region *vec;
+
+ /* 1. wrong operation */
+ mem_size = 10 * page_size;
+ vec_size = mem_size / page_size;
+
+ vec = malloc(sizeof(struct page_region) * vec_size);
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED || vec == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ wp_init(mem, mem_size);
+
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PAGEMAP_WP_ENGAGE, 0,
+ PAGEMAP_OP_MASK, 0, 0, PAGEMAP_OP_MASK) < 0,
+ "%s clear op can only be specified with PAGE_IS_WT\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGEMAP_OP_MASK, 0, 0, PAGEMAP_OP_MASK) >= 0,
+ "%s required_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_OP_MASK, 0, PAGEMAP_OP_MASK) >= 0,
+ "%s anyof_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, 0, PAGEMAP_OP_MASK, PAGEMAP_OP_MASK) >= 0,
+ "%s excluded_mask specified\n", __func__);
+ ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGEMAP_OP_MASK, PAGEMAP_OP_MASK, 0, PAGEMAP_OP_MASK) >= 0,
+ "%s required_mask and anyof_mask specified\n", __func__);
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 2. Get sd and present pages with anyof_mask */
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ memset(mem, 0, mem_size);
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_OP_MASK, 0, PAGEMAP_OP_MASK);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WT | PAGE_IS_PRESENT),
+ "%s Get sd and present pages with anyof_mask\n", __func__);
+
+ /* 3. Get sd and present pages with required_mask */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGEMAP_OP_MASK, 0, 0, PAGEMAP_OP_MASK);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WT | PAGE_IS_PRESENT),
+ "%s Get all the pages with required_mask\n", __func__);
+
+ /* 4. Get sd and present pages with required_mask and anyof_mask */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ PAGE_IS_WT, PAGE_IS_PRESENT, 0, PAGEMAP_OP_MASK);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == (PAGE_IS_WT | PAGE_IS_PRESENT),
+ "%s Get sd and present pages with required_mask and anyof_mask\n",
+ __func__);
+
+ /* 5. Don't get sd pages */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, 0, PAGE_IS_WT, PAGEMAP_OP_MASK);
+ ksft_test_result(ret == 0, "%s Don't get sd pages\n", __func__);
+
+ /* 6. Don't get present pages */
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, 0, PAGE_IS_PRESENT, PAGEMAP_OP_MASK);
+ ksft_test_result(ret == 0, "%s Don't get present pages\n", __func__);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 8. Find dirty present pages with return mask */
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ memset(mem, 0, mem_size);
+
+ ret = pagemap_ioctl(mem, mem_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_OP_MASK, 0, PAGE_IS_WT);
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size &&
+ vec[0].bitmap == PAGE_IS_WT,
+ "%s Find dirty present pages with return mask\n", __func__);
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 9. Memory mapped file */
+ int fd;
+ struct stat sbuf;
+
+ fd = open(__FILE__, O_RDONLY);
+ if (fd < 0) {
+ ksft_test_result_skip("%s Memory mapped file\n");
+ goto free_vec_and_return;
+ }
+
+ ret = stat(__FILE__, &sbuf);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ fmem = mmap(NULL, sbuf.st_size, PROT_READ, MAP_SHARED, fd, 0);
+ if (fmem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+
+ ret = pagemap_ioctl(fmem, sbuf.st_size, vec, vec_size, 0, 0,
+ 0, PAGEMAP_NON_WT_MASK, 0, PAGEMAP_NON_WT_MASK);
+
+ ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem &&
+ vec[0].len == ceilf((float)sbuf.st_size/page_size) &&
+ vec[0].bitmap == PAGE_IS_FILE,
+ "%s Memory mapped file\n", __func__);
+
+ munmap(fmem, sbuf.st_size);
+ close(fd);
+
+free_vec_and_return:
+ free(vec);
+ return 0;
+}
+
+int mprotect_tests(void)
+{
+ int ret;
+ char *mem, *mem2;
+ struct page_region vec;
+ int pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+
+ if (pagemap_fd < 0) {
+ fprintf(stderr, "open() failed\n");
+ exit(1);
+ }
+
+ /* 1. Map two pages */
+ mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, 2 * page_size);
+
+ /* Populate both pages. */
+ memset(mem, 1, 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2, "%s Both pages dirty\n", __func__);
+
+ /* 2. Start softdirty tracking. Clear VM_SOFTDIRTY and clear the softdirty PTE bit. */
+ ret = pagemap_ioctl(mem, 2 * page_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT) == 0,
+ "%s Both pages are not soft dirty\n", __func__);
+
+ /* 3. Remap the second page */
+ mem2 = mmap(mem + page_size, page_size, PROT_READ|PROT_WRITE,
+ MAP_PRIVATE|MAP_ANON|MAP_FIXED, -1, 0);
+ if (mem2 == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem2, page_size);
+
+ /* Protect + unprotect. */
+ mprotect(mem, 2 * page_size, PROT_READ);
+ mprotect(mem, 2 * page_size, PROT_READ|PROT_WRITE);
+
+ /* Modify both pages. */
+ memset(mem, 2, 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2,
+ "%s Both pages dirty after remap and mprotect\n", __func__);
+
+ /* 4. Clear and make the pages dirty */
+ ret = pagemap_ioctl(mem, 2 * page_size, NULL, 0, PAGEMAP_WP_ENGAGE, 0,
+ 0, 0, 0, 0);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ memset(mem, 'A', 2 * page_size);
+
+ ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, 0, 0, PAGE_IS_WT,
+ 0, 0, PAGE_IS_WT);
+ if (ret < 0)
+ ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno));
+
+ ksft_test_result(ret == 1 && vec.len == 2,
+ "%s Clear and make the pages dirty\n", __func__);
+
+ wp_free(mem, 2 * page_size);
+ munmap(mem, 2 * page_size);
+ return 0;
+}
+
+int main(void)
+{
+ char *mem, *map;
+ int mem_size;
+
+ ksft_print_header();
+ ksft_set_plan(54);
+
+ page_size = getpagesize();
+ hpage_size = read_pmd_pagesize();
+
+ pagemap_fd = open(PAGEMAP, O_RDWR);
+ if (pagemap_fd < 0)
+ return -EINVAL;
+
+ if (init_uffd())
+ ksft_exit_fail_msg("uffd init failed\n");
+
+ /*
+ * Soft-dirty PTE bit tests
+ */
+
+ /* 1. Sanity testing */
+ sanity_tests_sd();
+
+ /* 2. Normal page testing */
+ mem_size = 10 * page_size;
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ base_tests("Page testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 3. Large page testing */
+ mem_size = 512 * 10 * page_size;
+ mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+ if (mem == MAP_FAILED)
+ ksft_exit_fail_msg("error nomem\n");
+ wp_init(mem, mem_size);
+
+ base_tests("Large Page testing:", mem, mem_size, 0);
+
+ wp_free(mem, mem_size);
+ munmap(mem, mem_size);
+
+ /* 4. Huge page testing */
+ map = gethugepage(hpage_size);
+ if (map) {
+ base_tests("Huge page testing:", map, hpage_size, 0);
+ wp_free(map, hpage_size);
+ free(map);
+ } else {
+ base_tests("Huge page testing:", NULL, 0, 1);
+ }
+
+ /* 6. Huge page tests */
+ hpage_unit_tests();
+
+ /* 7. Iterative test */
+ test_simple();
+
+ /* 8. Mprotect test */
+ mprotect_tests();
+
+ /*
+ * Other PTE bit tests
+ */
+
+ /* 1. Sanity testing */
+ sanity_tests();
+
+ /* 2. Unmapped address test */
+ unmapped_region_tests();
+
+ close(pagemap_fd);
+ return ksft_exit_pass();
+}
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-27 6:47 ` Muhammad Usama Anjum
@ 2023-01-27 15:32 ` Peter Xu
2023-01-30 8:38 ` Muhammad Usama Anjum
0 siblings, 1 reply; 21+ messages in thread
From: Peter Xu @ 2023-01-27 15:32 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: David Hildenbrand, Andrew Morton, Michał Mirosław,
Andrei Vagin, Danylo Mocherniuk, Paul Gofman, Cyrill Gorcunov,
Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Matthew Wilcox, Pasha Tatashin, Mike Rapoport,
Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva, Dan Williams,
linux-kernel, linux-fsdevel, linux-mm, linux-kselftest, Greg KH,
kernel
On Fri, Jan 27, 2023 at 11:47:14AM +0500, Muhammad Usama Anjum wrote:
> >> diff --git a/mm/memory.c b/mm/memory.c
> >> index 4000e9f017e0..8c03b133d483 100644
> >> --- a/mm/memory.c
> >> +++ b/mm/memory.c
> >> @@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
> >>
> >> if (likely(!unshare)) {
> >> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
> >> + if (userfaultfd_wp_async(vma)) {
> >> + /*
> >> + * Nothing needed (cache flush, TLB invalidations,
> >> + * etc.) because we're only removing the uffd-wp bit,
> >> + * which is completely invisible to the user. This
> >> + * falls through to possible CoW.
> >
> > Here it says it falls through to CoW, but..
> >
> >> + */
> >> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> >> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
> >> + pte_clear_uffd_wp(*vmf->pte));
> >> + return 0;
> >
> > ... it's not doing so. The original lines should do:
> >
> > https://lore.kernel.org/all/Y8qq0dKIJBshua+X@x1n/
[1]
> >
> > Side note: you cannot modify pgtable after releasing the pgtable lock.
> > It's racy.
> If I don't unlock and return after removing the UFFD_WP flag in case of
> async wp, the target just gets stuck. Maybe the pte lock is not unlocked in
> some path.
>
> If I unlock and don't return, the crash happens.
>
> So I'd put unlock and return from here. Please comment on the below patch
> and what do you think should be done. I've missed something.
Have you tried to just use exactly what I suggested in [1]? I'll paste
again:
---8<---
diff --git a/mm/memory.c b/mm/memory.c
index 4000e9f017e0..09aab434654c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3351,8 +3351,20 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
if (likely(!unshare)) {
if (userfaultfd_pte_wp(vma, *vmf->pte)) {
- pte_unmap_unlock(vmf->pte, vmf->ptl);
- return handle_userfault(vmf, VM_UFFD_WP);
+ if (userfaultfd_uffd_wp_async(vma)) {
+ /*
+ * Nothing needed (cache flush, TLB
+ * invalidations, etc.) because we're only
+ * removing the uffd-wp bit, which is
+ * completely invisible to the user.
+ * This falls through to possible CoW.
+ */
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
+ pte_clear_uffd_wp(*vmf->pte));
+ } else {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return handle_userfault(vmf, VM_UFFD_WP);
+ }
}
---8<---
Note that there's no "return", neither the unlock. The lock is used in the
follow up write fault resolution and it's released later.
Meanwhile please fully digest how pgtable lock is used in this path before
moving forward on any of such changes.
>
> >
> >> + }
> >> pte_unmap_unlock(vmf->pte, vmf->ptl);
> >> return handle_userfault(vmf, VM_UFFD_WP);
> >> }
> >> @@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
> >>
> >> if (vma_is_anonymous(vmf->vma)) {
> >> if (likely(!unshare) &&
> >> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
> >> - return handle_userfault(vmf, VM_UFFD_WP);
> >> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
> >> + if (userfaultfd_wp_async(vmf->vma)) {
> >> + /*
> >> + * Nothing needed (cache flush, TLB invalidations,
> >> + * etc.) because we're only removing the uffd-wp bit,
> >> + * which is completely invisible to the user. This
> >> + * falls through to possible CoW.
> >> + */
> >> + set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
> >> + pmd_clear_uffd_wp(*vmf->pmd));
> >
> > This is for THP, not hugetlb.
> >
> > Clearing uffd-wp bit here for the whole pmd is wrong to me, because we
> > track writes in small page sizes only. We should just split.
> By detecting if the fault is async wp, just splitting the PMD doesn't work.
> The below given snippit is working right now. But definately, the fault of
> the whole PMD is being resolved which if we can bypass by correctly
> splitting would be highly desirable. Can you please take a look on UFFD
> side and suggest the changes? It would be much appreciated. I'm attaching
> WIP v9 patches for you to apply on next(next-20230105) and pagemap_ioctl
> selftest can be ran to test things after making changes.
Can you elaborate why thp split didn't work? Or if you want, I can look
into this and provide the patch to enable uffd async mode.
Thanks,
--
Peter Xu
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v8 2/4] userfaultfd: split mwriteprotect_range()
2023-01-24 8:43 ` [PATCH v8 2/4] userfaultfd: split mwriteprotect_range() Muhammad Usama Anjum
2023-01-24 10:23 ` kernel test robot
@ 2023-01-27 17:05 ` Peter Xu
2023-01-30 9:10 ` Muhammad Usama Anjum
1 sibling, 1 reply; 21+ messages in thread
From: Peter Xu @ 2023-01-27 17:05 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: David Hildenbrand, Andrew Morton, Michał Mirosław,
Andrei Vagin, Danylo Mocherniuk, Paul Gofman, Cyrill Gorcunov,
Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Matthew Wilcox, Pasha Tatashin, Mike Rapoport,
Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva, Dan Williams,
linux-kernel, linux-fsdevel, linux-mm, linux-kselftest, Greg KH,
kernel
On Tue, Jan 24, 2023 at 01:43:21PM +0500, Muhammad Usama Anjum wrote:
> Split mwriteprotect_range() to create a unlocked version. This
> will be used in the next patch to write protect a memory area.
> Add a helper function, wp_range_async() as well.
>
> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
IIUC this patch is not needed. You have a stable vma, so I think you can
directly use uffd_wp_range(), while most of the mwriteprotect_range() is
not needed.
There's one trivial detail of ignoring userfaultfd_ctx->mmap_changing when
it's set to true, but I don't think it applies here either because it was
used to resolve a problem in uffd non-cooperative mode on the predictable
behavior of events, here I don't think it matters a lot either.
--
Peter Xu
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
2023-01-24 8:43 ` [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
2023-01-24 10:02 ` kernel test robot
@ 2023-01-27 17:36 ` Peter Xu
2023-01-30 11:12 ` Muhammad Usama Anjum
1 sibling, 1 reply; 21+ messages in thread
From: Peter Xu @ 2023-01-27 17:36 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: David Hildenbrand, Andrew Morton, Michał Mirosław,
Andrei Vagin, Danylo Mocherniuk, Paul Gofman, Cyrill Gorcunov,
Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Matthew Wilcox, Pasha Tatashin, Mike Rapoport,
Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva, Dan Williams,
linux-kernel, linux-fsdevel, linux-mm, linux-kselftest, Greg KH,
kernel
On Tue, Jan 24, 2023 at 01:43:22PM +0500, Muhammad Usama Anjum wrote:
> This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
> the info about page table entries. The following operations are supported
> in this ioctl:
> - Get the information if the pages have been written-to (PAGE_IS_WT),
> file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
> (PAGE_IS_SWAPPED).
> - Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
> pages have been written-to.
> - Find pages which have been written-to and write protect the pages
> (atomic PAGE_IS_WT + PAGEMAP_WP_ENGAGE)
>
> The uffd should have been registered on the memory range before performing
> any WP/WT (Write Protect/Writtern-To) related operations with the IOCTL.
>
> struct pagemap_scan_args is used as the argument of the IOCTL. In this
> struct:
> - The range is specified through start and len.
> - The output buffer of struct page_region array and size is specified as
> vec and vec_len.
> - The optional maximum requested pages are specified in the max_pages.
> - The flags can be specified in the flags field. The PAGEMAP_WP_ENGAGE
> is the only added flag at this time.
> - The masks are specified in required_mask, anyof_mask, excluded_ mask
> and return_mask.
>
> This IOCTL can be extended to get information about more PTE bits. This
> IOCTL doesn't support hugetlbs at the moment. No information about
> hugetlb can be obtained. This patch has evolved from a basic patch from
> Gabriel Krisman Bertazi.
I think you mentioned you need hugetlb support, here it says it doesn't.
Which one is true?
>
> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> ---
> Change in v8:
> - Correct is_pte_uffd_wp()
> - Improve readability and error checks
> - Remove some un-needed code
>
> Changes in v7:
> - Rebase on top of latest next
> - Fix some corner cases
> - Base soft-dirty on the uffd wp async
> - Update the terminologies
> - Optimize the memory usage inside the ioctl
>
> Changes in v6:
> - Rename variables and update comments
> - Make IOCTL independent of soft_dirty config
> - Change masks and bitmap type to _u64
> - Improve code quality
>
> Changes in v5:
> - Remove tlb flushing even for clear operation
>
> Changes in v4:
> - Update the interface and implementation
>
> Changes in v3:
> - Tighten the user-kernel interface by using explicit types and add more
> error checking
>
> Changes in v2:
> - Convert the interface from syscall to ioctl
> - Remove pidfd support as it doesn't make sense in ioctl
> ---
> fs/proc/task_mmu.c | 294 ++++++++++++++++++++++++++++++++++
> include/uapi/linux/fs.h | 50 ++++++
> tools/include/uapi/linux/fs.h | 50 ++++++
> 3 files changed, 394 insertions(+)
>
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index e35a0398db63..f9fbda00eb5c 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -19,6 +19,7 @@
> #include <linux/shmem_fs.h>
> #include <linux/uaccess.h>
> #include <linux/pkeys.h>
> +#include <linux/minmax.h>
>
> #include <asm/elf.h>
> #include <asm/tlb.h>
> @@ -1135,6 +1136,22 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma,
> }
> #endif
>
> +static inline bool is_pte_uffd_wp(pte_t pte)
> +{
> + if ((pte_present(pte) && pte_uffd_wp(pte)) ||
> + (is_swap_pte(pte) && pte_swp_uffd_wp_any(pte)))
is_swap_pte() not needed, because pte_swp_uffd_wp_any() covers that.
> + return true;
> + return false;
> +}
> +
> +static inline bool is_pmd_uffd_wp(pmd_t pmd)
> +{
> + if ((pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
> + (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)))
> + return true;
> + return false;
> +}
> +
> #if defined(CONFIG_MEM_SOFT_DIRTY) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
> static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma,
> unsigned long addr, pmd_t *pmdp)
> @@ -1763,11 +1780,288 @@ static int pagemap_release(struct inode *inode, struct file *file)
> return 0;
> }
>
> +#define PAGEMAP_OP_MASK (PAGE_IS_WT | PAGE_IS_FILE | \
> + PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
The name is confusing. It's not a mask of OPs, it's a mask of bits that
are supported.
Maybe PAGEMAP_BITS_ALL?
I would also slightly prefer spelling WT out to be PAGE_IS_WRITTEN.
> +#define PAGEMAP_NONWT_OP_MASK (PAGE_IS_FILE | PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
Maybe PAGEMAP_BITS_WP_SUPPORTED?
> +#define IS_WP_ENGAGE_OP(a) (a->flags & PAGEMAP_WP_ENGAGE)
> +#define IS_GET_OP(a) (a->vec)
Having a->vec to imply the GET is fine, but IMHO not as clean as having
each OP a bit in the flags.
How about:
#define PAGEMAP_OP_GET (1UL << 0)
#define PAGEMAP_OP_WP (1UL << 1)
#define PAGEMAP_OP_MASK (PAGEMAP_OP_GET | PAGEMAP_OP_WP)
?
Then a->vec requried for PAGEMAP_OP_GET.
> +#define PAGEMAP_SCAN_BITMAP(wt, file, present, swap) \
> + (wt | file << 1 | present << 2 | swap << 3)
> +#define IS_WT_REQUIRED(a) \
> + ((a->required_mask & PAGE_IS_WT) || \
> + (a->anyof_mask & PAGE_IS_WT))
> +#define HAS_NO_SPACE(p) (p->max_pages && (p->found_pages == p->max_pages))
> +
> +struct pagemap_scan_private {
> + struct page_region *vec;
> + struct page_region prev;
> + unsigned long vec_len, vec_index;
> + unsigned int max_pages, found_pages, flags;
> + unsigned long required_mask, anyof_mask, excluded_mask, return_mask;
> +};
> +
> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end, struct mm_walk *walk)
> +{
> + struct pagemap_scan_private *p = walk->private;
> + struct vm_area_struct *vma = walk->vma;
> +
> + if (IS_WT_REQUIRED(p) && !userfaultfd_wp(vma) && !userfaultfd_wp_async(vma))
> + return -EPERM;
> + if (IS_GET_OP(p) && HAS_NO_SPACE(p))
> + return -ENOSPC;
As commented before, I think this can be dropped because it should never
trigger.
> + if (vma->vm_flags & VM_PFNMAP)
> + return 1;
> + return 0;
> +}
> +
> +static inline int add_to_out(bool wt, bool file, bool pres, bool swap,
There're too many names that are too general to me. I suggest some better
one. For this one, perhaps pagemap_scan_output()?
> + struct pagemap_scan_private *p, unsigned long addr, unsigned int len)
> +{
> + unsigned long bitmap, cur = PAGEMAP_SCAN_BITMAP(wt, file, pres, swap);
> + bool cpy = true;
> + struct page_region *prev = &p->prev;
> +
> + if (HAS_NO_SPACE(p))
> + return -ENOSPC;
This can be moved to below [1], we should stop scanning immediately if the
condition met.
> +
> + if (p->required_mask)
> + cpy = ((p->required_mask & cur) == p->required_mask);
> + if (cpy && p->anyof_mask)
> + cpy = (p->anyof_mask & cur);
> + if (cpy && p->excluded_mask)
> + cpy = !(p->excluded_mask & cur);
> + bitmap = cur & p->return_mask;
> + if (cpy && bitmap) {
> + if ((prev->len) && (prev->bitmap == bitmap) &&
> + (prev->start + prev->len * PAGE_SIZE == addr)) {
> + prev->len += len;
> + p->found_pages += len;
> + } else if (p->vec_index < p->vec_len) {
> + if (prev->len) {
> + memcpy(&p->vec[p->vec_index], prev, sizeof(struct page_region));
> + p->vec_index++;
> + }
> + prev->start = addr;
> + prev->len = len;
> + prev->bitmap = bitmap;
> + p->found_pages += len;
> + } else {
> + return -ENOSPC;
> + }
[1]
Here is the only place found_pages got boosted, so checking here only
should be enough, IIUC.
> + }
> + return 0;
> +}
> +
> +static inline int export_prev_to_out(struct pagemap_scan_private *p, struct page_region __user *vec,
> + unsigned long *vec_index)
> +{
> + struct page_region *prev = &p->prev;
> +
> + if (prev->len) {
> + if (copy_to_user(&vec[*vec_index], prev, sizeof(struct page_region)))
> + return -EFAULT;
> + p->vec_index++;
> + (*vec_index)++;
> + prev->len = 0;
> + }
> + return 0;
> +}
> +
> +static inline int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
> + unsigned long end, struct mm_walk *walk)
> +{
> + struct pagemap_scan_private *p = walk->private;
> + struct vm_area_struct *vma = walk->vma;
> + unsigned long addr = end;
> + unsigned int len;
> + spinlock_t *ptl;
> + int ret = 0;
> + pte_t *pte;
> + bool pmd_wt;
> +
> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
> + ptl = pmd_trans_huge_lock(pmd, vma);
> + if (ptl) {
> + pmd_wt = !is_pmd_uffd_wp(*pmd);
> + /*
> + * Break huge page into small pages if operation needs to be performed is
> + * on a portion of the huge page or the return buffer cannot store complete
> + * data.
> + */
> + if (pmd_wt && (IS_WP_ENGAGE_OP(p) && (end - start < HPAGE_SIZE))) {
> + spin_unlock(ptl);
> + split_huge_pmd(vma, pmd, start);
> + goto process_smaller_pages;
> + }
> + if (IS_GET_OP(p)) {
> + len = (end - start)/PAGE_SIZE;
> + if (p->max_pages && p->found_pages + len > p->max_pages)
> + len = p->max_pages - p->found_pages;
> +
> + ret = add_to_out(pmd_wt, vma->vm_file, pmd_present(*pmd),
> + is_swap_pmd(*pmd), p, start, len);
> + if (ret) {
> + spin_unlock(ptl);
> + return ret;
> + }
> + }
> + spin_unlock(ptl);
> + if (IS_WP_ENGAGE_OP(p) && pmd_wt)
> + ret = wp_range_async(vma, start, HPAGE_SIZE);
> + return ret;
> + }
> +process_smaller_pages:
> + if (pmd_trans_unstable(pmd))
> + return 0;
> +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
> +
> + pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
> + if (IS_GET_OP(p)) {
> + for (addr = start;
> + addr < end && !ret && (!p->max_pages || (p->found_pages < p->max_pages));
> + pte++, addr += PAGE_SIZE) {
> + ret = add_to_out(!is_pte_uffd_wp(*pte), vma->vm_file, pte_present(*pte),
> + is_swap_pte(*pte), p, addr, 1);
> + if (ret)
> + break;
> + }
> + }
> + pte_unmap_unlock(pte - 1, ptl);
> + if (!ret && IS_WP_ENGAGE_OP(p))
> + ret = wp_range_async(vma, start, addr - start);
IIUC you wanted to cover the case where e.g. the output buffer is full or
max_pages specified, so the partial pages still got wr-protected
atomically.
In that case I think we need to at least allow ret==-ENOSPC here.
> +
> + cond_resched();
> + return ret;
> +}
> +
> +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, int depth,
> + struct mm_walk *walk)
> +{
> + struct pagemap_scan_private *p = walk->private;
> + struct vm_area_struct *vma = walk->vma;
> + unsigned int len;
> + int ret = 0;
> +
> + if (vma) {
> + len = (end - addr)/PAGE_SIZE;
> + if (p->max_pages && p->found_pages + len > p->max_pages)
> + len = p->max_pages - p->found_pages;
> + ret = add_to_out(false, vma->vm_file, false, false, p, addr, len);
> + }
> + return ret;
> +}
> +
> +/* No hugetlb support is present. */
> +static const struct mm_walk_ops pagemap_scan_ops = {
> + .test_walk = pagemap_scan_test_walk,
> + .pmd_entry = pagemap_scan_pmd_entry,
> + .pte_hole = pagemap_scan_pte_hole,
> +};
> +
> +static long do_pagemap_cmd(struct mm_struct *mm, struct pagemap_scan_arg *arg)
> +{
> + unsigned long empty_slots, vec_index = 0;
> + unsigned long __user start, end;
> + unsigned long __start, __end;
> + struct page_region __user *vec;
> + struct pagemap_scan_private p;
> + int ret;
> +
> + start = (unsigned long)untagged_addr(arg->start);
> + vec = (struct page_region *)untagged_addr(arg->vec);
> + if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
> + return -EINVAL;
> + if (IS_GET_OP(arg) && ((arg->vec_len == 0) ||
> + (!access_ok((void __user *)vec, arg->vec_len * sizeof(struct page_region)))))
> + return -ENOMEM;
Is ENOMEM the right error code? I think -EINVAL seems more proper.
> + if ((arg->flags & ~PAGEMAP_WP_ENGAGE) || (arg->required_mask & ~PAGEMAP_OP_MASK) ||
> + (arg->anyof_mask & ~PAGEMAP_OP_MASK) || (arg->excluded_mask & ~PAGEMAP_OP_MASK) ||
> + (arg->return_mask & ~PAGEMAP_OP_MASK))
> + return -EINVAL;
> + if (IS_GET_OP(arg) && ((!arg->required_mask && !arg->anyof_mask && !arg->excluded_mask) ||
> + !arg->return_mask))
> + return -EINVAL;
> + /* The non-WT flags cannot be obtained if PAGEMAP_WP_ENGAGE is also specified. */
> + if (IS_WP_ENGAGE_OP(arg) && ((arg->required_mask & PAGEMAP_NONWT_OP_MASK) ||
> + (arg->anyof_mask & PAGEMAP_NONWT_OP_MASK)))
> + return -EINVAL;
The whole chunk is not very readable. Some emptly lines would be nice to
have here and there, and some groupings. No need to keep the condition so
long. How about something like:
start = (unsigned long)untagged_addr(arg->start);
vec = (struct page_region *)untagged_addr(arg->vec);
/* Detect illegal cmds */
if ((arg->flags & ~PAGEMAP_OP_MASK))
return -EINVAL;
/* Validate common parameters */
if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
return -EINVAL;
/* Validate parameters for each cmd */
if (PAGEMAP_OP_GET(arg)) {
...
}
if (PAGEMAP_OP_WP(arg)) {
...
}
> +
> + end = start + arg->len;
> + p.max_pages = arg->max_pages;
> + p.found_pages = 0;
> + p.flags = arg->flags;
> + p.required_mask = arg->required_mask;
> + p.anyof_mask = arg->anyof_mask;
> + p.excluded_mask = arg->excluded_mask;
> + p.return_mask = arg->return_mask;
> + p.prev.len = 0;
> + p.vec_len = (PAGEMAP_WALK_SIZE >> PAGE_SHIFT);
> +
> + if (IS_GET_OP(arg)) {
> + p.vec = kmalloc_array(p.vec_len, sizeof(struct page_region), GFP_KERNEL);
> + if (!p.vec)
> + return -ENOMEM;
> + } else {
> + p.vec = NULL;
> + }
> + __start = __end = start;
> + while (__end < end) {
> + p.vec_index = 0;
> + empty_slots = arg->vec_len - vec_index;
> + if (p.vec_len > empty_slots)
> + p.vec_len = empty_slots;
> +
> + __end = (__start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
> + if (__end > end)
> + __end = end;
> +
> + mmap_read_lock(mm);
> + ret = walk_page_range(mm, __start, __end, &pagemap_scan_ops, &p);
> + mmap_read_unlock(mm);
> + if (!(!ret || ret == -ENOSPC))
> + goto free_data;
> +
> + __start = __end;
> + if (IS_GET_OP(arg) && p.vec_index) {
> + if (copy_to_user(&vec[vec_index], p.vec,
> + p.vec_index * sizeof(struct page_region))) {
> + ret = -EFAULT;
> + goto free_data;
> + }
> + vec_index += p.vec_index;
> + }
> + }
> + ret = export_prev_to_out(&p, vec, &vec_index);
> + if (!ret)
> + ret = vec_index;
> +free_data:
> + if (IS_GET_OP(arg))
> + kfree(p.vec);
> +
> + return ret;
> +}
> +
> +static long pagemap_scan_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> +{
> + struct pagemap_scan_arg __user *uarg = (struct pagemap_scan_arg __user *)arg;
> + struct mm_struct *mm = file->private_data;
> + struct pagemap_scan_arg argument;
> +
> + if (cmd == PAGEMAP_SCAN) {
> + if (copy_from_user(&argument, uarg, sizeof(struct pagemap_scan_arg)))
> + return -EFAULT;
> + return do_pagemap_cmd(mm, &argument);
> + }
> + return -EINVAL;
> +}
> +
> const struct file_operations proc_pagemap_operations = {
> .llseek = mem_lseek, /* borrow this */
> .read = pagemap_read,
> .open = pagemap_open,
> .release = pagemap_release,
> + .unlocked_ioctl = pagemap_scan_ioctl,
> + .compat_ioctl = pagemap_scan_ioctl,
> };
> #endif /* CONFIG_PROC_PAGE_MONITOR */
>
> diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> index b7b56871029c..6d03a903a745 100644
> --- a/include/uapi/linux/fs.h
> +++ b/include/uapi/linux/fs.h
> @@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
> #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
> RWF_APPEND)
>
> +/* Pagemap ioctl */
> +#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
> +
> +/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
> +#define PAGE_IS_WT (1 << 0)
> +#define PAGE_IS_FILE (1 << 1)
> +#define PAGE_IS_PRESENT (1 << 2)
> +#define PAGE_IS_SWAPPED (1 << 3)
> +
> +/*
> + * struct page_region - Page region with bitmap flags
> + * @start: Start of the region
> + * @len: Length of the region
> + * bitmap: Bits sets for the region
> + */
> +struct page_region {
> + __u64 start;
> + __u64 len;
> + __u64 bitmap;
> +};
> +
> +/*
> + * struct pagemap_scan_arg - Pagemap ioctl argument
> + * @start: Starting address of the region
> + * @len: Length of the region (All the pages in this length are included)
> + * @vec: Address of page_region struct array for output
> + * @vec_len: Length of the page_region struct array
> + * @max_pages: Optional max return pages
> + * @flags: Flags for the IOCTL
> + * @required_mask: Required mask - All of these bits have to be set in the PTE
> + * @anyof_mask: Any mask - Any of these bits are set in the PTE
> + * @excluded_mask: Exclude mask - None of these bits are set in the PTE
> + * @return_mask: Bits that are to be reported in page_region
> + */
> +struct pagemap_scan_arg {
> + __u64 start;
> + __u64 len;
> + __u64 vec;
> + __u64 vec_len;
> + __u32 max_pages;
> + __u32 flags;
> + __u64 required_mask;
> + __u64 anyof_mask;
> + __u64 excluded_mask;
> + __u64 return_mask;
> +};
> +
> +/* Special flags */
> +#define PAGEMAP_WP_ENGAGE (1 << 0)
> +
> #endif /* _UAPI_LINUX_FS_H */
> diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
> index b7b56871029c..6d03a903a745 100644
> --- a/tools/include/uapi/linux/fs.h
> +++ b/tools/include/uapi/linux/fs.h
> @@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
> #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
> RWF_APPEND)
>
> +/* Pagemap ioctl */
> +#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
> +
> +/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
> +#define PAGE_IS_WT (1 << 0)
> +#define PAGE_IS_FILE (1 << 1)
> +#define PAGE_IS_PRESENT (1 << 2)
> +#define PAGE_IS_SWAPPED (1 << 3)
> +
> +/*
> + * struct page_region - Page region with bitmap flags
> + * @start: Start of the region
> + * @len: Length of the region
> + * bitmap: Bits sets for the region
> + */
> +struct page_region {
> + __u64 start;
> + __u64 len;
> + __u64 bitmap;
> +};
> +
> +/*
> + * struct pagemap_scan_arg - Pagemap ioctl argument
> + * @start: Starting address of the region
> + * @len: Length of the region (All the pages in this length are included)
> + * @vec: Address of page_region struct array for output
> + * @vec_len: Length of the page_region struct array
> + * @max_pages: Optional max return pages
> + * @flags: Flags for the IOCTL
> + * @required_mask: Required mask - All of these bits have to be set in the PTE
> + * @anyof_mask: Any mask - Any of these bits are set in the PTE
> + * @excluded_mask: Exclude mask - None of these bits are set in the PTE
> + * @return_mask: Bits that are to be reported in page_region
> + */
> +struct pagemap_scan_arg {
> + __u64 start;
> + __u64 len;
> + __u64 vec;
> + __u64 vec_len;
> + __u32 max_pages;
> + __u32 flags;
> + __u64 required_mask;
> + __u64 anyof_mask;
> + __u64 excluded_mask;
> + __u64 return_mask;
> +};
> +
> +/* Special flags */
> +#define PAGEMAP_WP_ENGAGE (1 << 0)
> +
> #endif /* _UAPI_LINUX_FS_H */
Better move the tools/ change out of the core changes, probably can be
squashed into the selftest patch later.
Thanks,
> --
> 2.30.2
>
--
Peter Xu
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-27 15:32 ` Peter Xu
@ 2023-01-30 8:38 ` Muhammad Usama Anjum
2023-01-30 21:27 ` Peter Xu
0 siblings, 1 reply; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-30 8:38 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Suren Baghdasaryan, Alex Sierra, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
On 1/27/23 8:32 PM, Peter Xu wrote:
> On Fri, Jan 27, 2023 at 11:47:14AM +0500, Muhammad Usama Anjum wrote:
>>>> diff --git a/mm/memory.c b/mm/memory.c
>>>> index 4000e9f017e0..8c03b133d483 100644
>>>> --- a/mm/memory.c
>>>> +++ b/mm/memory.c
>>>> @@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>>>>
>>>> if (likely(!unshare)) {
>>>> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
>>>> + if (userfaultfd_wp_async(vma)) {
>>>> + /*
>>>> + * Nothing needed (cache flush, TLB invalidations,
>>>> + * etc.) because we're only removing the uffd-wp bit,
>>>> + * which is completely invisible to the user. This
>>>> + * falls through to possible CoW.
>>>
>>> Here it says it falls through to CoW, but..
>>>
>>>> + */
>>>> + pte_unmap_unlock(vmf->pte, vmf->ptl);
>>>> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
>>>> + pte_clear_uffd_wp(*vmf->pte));
>>>> + return 0;
>>>
>>> ... it's not doing so. The original lines should do:
>>>
>>> https://lore.kernel.org/all/Y8qq0dKIJBshua+X@x1n/
>
> [1]
>
>>>
>>> Side note: you cannot modify pgtable after releasing the pgtable lock.
>>> It's racy.
>> If I don't unlock and return after removing the UFFD_WP flag in case of
>> async wp, the target just gets stuck. Maybe the pte lock is not unlocked in
>> some path.
>>
>> If I unlock and don't return, the crash happens.
>>
>> So I'd put unlock and return from here. Please comment on the below patch
>> and what do you think should be done. I've missed something.
>
> Have you tried to just use exactly what I suggested in [1]? I'll paste
> again:
>
> ---8<---
> diff --git a/mm/memory.c b/mm/memory.c
> index 4000e9f017e0..09aab434654c 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3351,8 +3351,20 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>
> if (likely(!unshare)) {
> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
> - pte_unmap_unlock(vmf->pte, vmf->ptl);
> - return handle_userfault(vmf, VM_UFFD_WP);
> + if (userfaultfd_uffd_wp_async(vma)) {
> + /*
> + * Nothing needed (cache flush, TLB
> + * invalidations, etc.) because we're only
> + * removing the uffd-wp bit, which is
> + * completely invisible to the user.
> + * This falls through to possible CoW.
> + */
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
> + pte_clear_uffd_wp(*vmf->pte));
> + } else {
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return handle_userfault(vmf, VM_UFFD_WP);
> + }
> }
> ---8<---
>
> Note that there's no "return", neither the unlock. The lock is used in the
> follow up write fault resolution and it's released later.
I've tried out the exact patch above. This doesn't work. The pages keep
their WP flag even after being resolved in do_wp_page() while is written on
the page.
So I'd added pte_unmap_unlock() and return 0 from here. This makes the
patch to work. Maybe you can try this on your end to see what I'm seeing here?
>
> Meanwhile please fully digest how pgtable lock is used in this path before
> moving forward on any of such changes.
>
>>
>>>
>>>> + }
>>>> pte_unmap_unlock(vmf->pte, vmf->ptl);
>>>> return handle_userfault(vmf, VM_UFFD_WP);
>>>> }
>>>> @@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
>>>>
>>>> if (vma_is_anonymous(vmf->vma)) {
>>>> if (likely(!unshare) &&
>>>> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
>>>> - return handle_userfault(vmf, VM_UFFD_WP);
>>>> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
>>>> + if (userfaultfd_wp_async(vmf->vma)) {
>>>> + /*
>>>> + * Nothing needed (cache flush, TLB invalidations,
>>>> + * etc.) because we're only removing the uffd-wp bit,
>>>> + * which is completely invisible to the user. This
>>>> + * falls through to possible CoW.
>>>> + */
>>>> + set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
>>>> + pmd_clear_uffd_wp(*vmf->pmd));
>>>
>>> This is for THP, not hugetlb.
>>>
>>> Clearing uffd-wp bit here for the whole pmd is wrong to me, because we
>>> track writes in small page sizes only. We should just split.
>> By detecting if the fault is async wp, just splitting the PMD doesn't work.
>> The below given snippit is working right now. But definately, the fault of
>> the whole PMD is being resolved which if we can bypass by correctly
>> splitting would be highly desirable. Can you please take a look on UFFD
>> side and suggest the changes? It would be much appreciated. I'm attaching
>> WIP v9 patches for you to apply on next(next-20230105) and pagemap_ioctl
>> selftest can be ran to test things after making changes.
>
> Can you elaborate why thp split didn't work? Or if you want, I can look
> into this and provide the patch to enable uffd async mode.
Sorry, I was doing the wrong way. Splitting the page does work. What do you
think about the following:
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3351,6 +3351,17 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
if (likely(!unshare)) {
if (userfaultfd_pte_wp(vma, *vmf->pte)) {
+ if (userfaultfd_wp_async(vma)) {
+ /*
+ * Nothing needed (cache flush, TLB invalidations,
+ * etc.) because we're only removing the uffd-wp bit,
+ * which is completely invisible to the user.
+ */
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
+ pte_clear_uffd_wp(*vmf->pte));
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+ }
pte_unmap_unlock(vmf->pte, vmf->ptl);
return handle_userfault(vmf, VM_UFFD_WP);
}
@@ -4812,8 +4823,13 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault
*vmf)
if (vma_is_anonymous(vmf->vma)) {
if (likely(!unshare) &&
- userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
+ userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
+ if (userfaultfd_wp_async(vmf->vma)) {
+ __split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
+ return 0;
+ }
return handle_userfault(vmf, VM_UFFD_WP);
+ }
return do_huge_pmd_wp_page(vmf);
}
>
> Thanks,
>
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 2/4] userfaultfd: split mwriteprotect_range()
2023-01-27 17:05 ` Peter Xu
@ 2023-01-30 9:10 ` Muhammad Usama Anjum
0 siblings, 0 replies; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-30 9:10 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Suren Baghdasaryan, Alex Sierra, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
On 1/27/23 10:05 PM, Peter Xu wrote:
> On Tue, Jan 24, 2023 at 01:43:21PM +0500, Muhammad Usama Anjum wrote:
>> Split mwriteprotect_range() to create a unlocked version. This
>> will be used in the next patch to write protect a memory area.
>> Add a helper function, wp_range_async() as well.
>>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>
> IIUC this patch is not needed. You have a stable vma, so I think you can
> directly use uffd_wp_range(), while most of the mwriteprotect_range() is
> not needed.
>
> There's one trivial detail of ignoring userfaultfd_ctx->mmap_changing when
> it's set to true, but I don't think it applies here either because it was
> used to resolve a problem in uffd non-cooperative mode on the predictable
> behavior of events, here I don't think it matters a lot either.
>
Thanks, I'll drop this patch and do direct wiring to uffd_wp_range().
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
2023-01-27 17:36 ` Peter Xu
@ 2023-01-30 11:12 ` Muhammad Usama Anjum
2023-01-30 21:34 ` Peter Xu
0 siblings, 1 reply; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-30 11:12 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Suren Baghdasaryan, Alex Sierra, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
On 1/27/23 10:36 PM, Peter Xu wrote:
> On Tue, Jan 24, 2023 at 01:43:22PM +0500, Muhammad Usama Anjum wrote:
>> This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
>> the info about page table entries. The following operations are supported
>> in this ioctl:
>> - Get the information if the pages have been written-to (PAGE_IS_WT),
>> file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
>> (PAGE_IS_SWAPPED).
>> - Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
>> pages have been written-to.
>> - Find pages which have been written-to and write protect the pages
>> (atomic PAGE_IS_WT + PAGEMAP_WP_ENGAGE)
>>
>> The uffd should have been registered on the memory range before performing
>> any WP/WT (Write Protect/Writtern-To) related operations with the IOCTL.
>>
>> struct pagemap_scan_args is used as the argument of the IOCTL. In this
>> struct:
>> - The range is specified through start and len.
>> - The output buffer of struct page_region array and size is specified as
>> vec and vec_len.
>> - The optional maximum requested pages are specified in the max_pages.
>> - The flags can be specified in the flags field. The PAGEMAP_WP_ENGAGE
>> is the only added flag at this time.
>> - The masks are specified in required_mask, anyof_mask, excluded_ mask
>> and return_mask.
>>
>> This IOCTL can be extended to get information about more PTE bits. This
>> IOCTL doesn't support hugetlbs at the moment. No information about
>> hugetlb can be obtained. This patch has evolved from a basic patch from
>> Gabriel Krisman Bertazi.
>
> I think you mentioned you need hugetlb support, here it says it doesn't.
> Which one is true?
Only THPs are supported. Hugetlbs aren't supported.
>
>>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>> ---
>> Change in v8:
>> - Correct is_pte_uffd_wp()
>> - Improve readability and error checks
>> - Remove some un-needed code
>>
>> Changes in v7:
>> - Rebase on top of latest next
>> - Fix some corner cases
>> - Base soft-dirty on the uffd wp async
>> - Update the terminologies
>> - Optimize the memory usage inside the ioctl
>>
>> Changes in v6:
>> - Rename variables and update comments
>> - Make IOCTL independent of soft_dirty config
>> - Change masks and bitmap type to _u64
>> - Improve code quality
>>
>> Changes in v5:
>> - Remove tlb flushing even for clear operation
>>
>> Changes in v4:
>> - Update the interface and implementation
>>
>> Changes in v3:
>> - Tighten the user-kernel interface by using explicit types and add more
>> error checking
>>
>> Changes in v2:
>> - Convert the interface from syscall to ioctl
>> - Remove pidfd support as it doesn't make sense in ioctl
>> ---
>> fs/proc/task_mmu.c | 294 ++++++++++++++++++++++++++++++++++
>> include/uapi/linux/fs.h | 50 ++++++
>> tools/include/uapi/linux/fs.h | 50 ++++++
>> 3 files changed, 394 insertions(+)
>>
>> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
>> index e35a0398db63..f9fbda00eb5c 100644
>> --- a/fs/proc/task_mmu.c
>> +++ b/fs/proc/task_mmu.c
>> @@ -19,6 +19,7 @@
>> #include <linux/shmem_fs.h>
>> #include <linux/uaccess.h>
>> #include <linux/pkeys.h>
>> +#include <linux/minmax.h>
>>
>> #include <asm/elf.h>
>> #include <asm/tlb.h>
>> @@ -1135,6 +1136,22 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma,
>> }
>> #endif
>>
>> +static inline bool is_pte_uffd_wp(pte_t pte)
>> +{
>> + if ((pte_present(pte) && pte_uffd_wp(pte)) ||
>> + (is_swap_pte(pte) && pte_swp_uffd_wp_any(pte)))
>
> is_swap_pte() not needed, because pte_swp_uffd_wp_any() covers that.
I'll remove is_swap_pte().
>
>> + return true;
>> + return false;
>> +}
>> +
>> +static inline bool is_pmd_uffd_wp(pmd_t pmd)
>> +{
>> + if ((pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
>> + (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)))
>> + return true;
>> + return false;
>> +}
>> +
>> #if defined(CONFIG_MEM_SOFT_DIRTY) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
>> static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma,
>> unsigned long addr, pmd_t *pmdp)
>> @@ -1763,11 +1780,288 @@ static int pagemap_release(struct inode *inode, struct file *file)
>> return 0;
>> }
>>
>> +#define PAGEMAP_OP_MASK (PAGE_IS_WT | PAGE_IS_FILE | \
>> + PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
>
> The name is confusing. It's not a mask of OPs, it's a mask of bits that
> are supported.
>
> Maybe PAGEMAP_BITS_ALL?
I'll change in next revision.
>
> I would also slightly prefer spelling WT out to be PAGE_IS_WRITTEN.
>
>> +#define PAGEMAP_NONWT_OP_MASK (PAGE_IS_FILE | PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
>
> Maybe PAGEMAP_BITS_WP_SUPPORTED?
I'll change it to PAGEMAP_NON_WRITTEN_BITS.
>
>> +#define IS_WP_ENGAGE_OP(a) (a->flags & PAGEMAP_WP_ENGAGE)
>> +#define IS_GET_OP(a) (a->vec)
>
> Having a->vec to imply the GET is fine, but IMHO not as clean as having
> each OP a bit in the flags.
>
> How about:
>
> #define PAGEMAP_OP_GET (1UL << 0)
> #define PAGEMAP_OP_WP (1UL << 1)
> #define PAGEMAP_OP_MASK (PAGEMAP_OP_GET | PAGEMAP_OP_WP)
>
> ?
>
> Then a->vec requried for PAGEMAP_OP_GET.
I had something like PAGEMAP_OP_GET and PAGEMAP_OP_WP flags once until I
was asked to only keep WP flag and vec != NULL will imply GET flag in a
previous review. I'm ready to change again if there are more notes on it?
>
>> +#define PAGEMAP_SCAN_BITMAP(wt, file, present, swap) \
>> + (wt | file << 1 | present << 2 | swap << 3)
>> +#define IS_WT_REQUIRED(a) \
>> + ((a->required_mask & PAGE_IS_WT) || \
>> + (a->anyof_mask & PAGE_IS_WT))
>> +#define HAS_NO_SPACE(p) (p->max_pages && (p->found_pages == p->max_pages))
>> +
>> +struct pagemap_scan_private {
>> + struct page_region *vec;
>> + struct page_region prev;
>> + unsigned long vec_len, vec_index;
>> + unsigned int max_pages, found_pages, flags;
>> + unsigned long required_mask, anyof_mask, excluded_mask, return_mask;
>> +};
>> +
>> +static int pagemap_scan_test_walk(unsigned long start, unsigned long end, struct mm_walk *walk)
>> +{
>> + struct pagemap_scan_private *p = walk->private;
>> + struct vm_area_struct *vma = walk->vma;
>> +
>> + if (IS_WT_REQUIRED(p) && !userfaultfd_wp(vma) && !userfaultfd_wp_async(vma))
>> + return -EPERM;
>> + if (IS_GET_OP(p) && HAS_NO_SPACE(p))
>> + return -ENOSPC;
>
> As commented before, I think this can be dropped because it should never
> trigger.
It'll get changed in next revision.
>
>> + if (vma->vm_flags & VM_PFNMAP)
>> + return 1;
>> + return 0;
>> +}
>> +
>> +static inline int add_to_out(bool wt, bool file, bool pres, bool swap,
>
> There're too many names that are too general to me. I suggest some better
> one. For this one, perhaps pagemap_scan_output()?
It'll get changed in next revision.
>
>> + struct pagemap_scan_private *p, unsigned long addr, unsigned int len)
>> +{
>> + unsigned long bitmap, cur = PAGEMAP_SCAN_BITMAP(wt, file, pres, swap);
>> + bool cpy = true;
>> + struct page_region *prev = &p->prev;
>> +
>> + if (HAS_NO_SPACE(p))
>> + return -ENOSPC;
>
> This can be moved to below [1], we should stop scanning immediately if the
> condition met.
It would be definately possible. Wouldn't it be strange to return error
when the operation was successful? A function should return error only when
the current execution is unsuccessful. So I'm returning error only when the
space is full and there is no space left in the buffer anymore.
>
>> +
>> + if (p->required_mask)
>> + cpy = ((p->required_mask & cur) == p->required_mask);
>> + if (cpy && p->anyof_mask)
>> + cpy = (p->anyof_mask & cur);
>> + if (cpy && p->excluded_mask)
>> + cpy = !(p->excluded_mask & cur);
>> + bitmap = cur & p->return_mask;
>> + if (cpy && bitmap) {
>> + if ((prev->len) && (prev->bitmap == bitmap) &&
>> + (prev->start + prev->len * PAGE_SIZE == addr)) {
>> + prev->len += len;
>> + p->found_pages += len;
>> + } else if (p->vec_index < p->vec_len) {
>> + if (prev->len) {
>> + memcpy(&p->vec[p->vec_index], prev, sizeof(struct page_region));
>> + p->vec_index++;
>> + }
>> + prev->start = addr;
>> + prev->len = len;
>> + prev->bitmap = bitmap;
>> + p->found_pages += len;
>> + } else {
>> + return -ENOSPC;
>> + }
>
> [1]
>
> Here is the only place found_pages got boosted, so checking here only
> should be enough, IIUC.
>
>> + }
>> + return 0;
>> +}
>> +
>> +static inline int export_prev_to_out(struct pagemap_scan_private *p, struct page_region __user *vec,
>> + unsigned long *vec_index)
>> +{
>> + struct page_region *prev = &p->prev;
>> +
>> + if (prev->len) {
>> + if (copy_to_user(&vec[*vec_index], prev, sizeof(struct page_region)))
>> + return -EFAULT;
>> + p->vec_index++;
>> + (*vec_index)++;
>> + prev->len = 0;
>> + }
>> + return 0;
>> +}
>> +
>> +static inline int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
>> + unsigned long end, struct mm_walk *walk)
>> +{
>> + struct pagemap_scan_private *p = walk->private;
>> + struct vm_area_struct *vma = walk->vma;
>> + unsigned long addr = end;
>> + unsigned int len;
>> + spinlock_t *ptl;
>> + int ret = 0;
>> + pte_t *pte;
>> + bool pmd_wt;
>> +
>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
>> + ptl = pmd_trans_huge_lock(pmd, vma);
>> + if (ptl) {
>> + pmd_wt = !is_pmd_uffd_wp(*pmd);
>> + /*
>> + * Break huge page into small pages if operation needs to be performed is
>> + * on a portion of the huge page or the return buffer cannot store complete
>> + * data.
>> + */
>> + if (pmd_wt && (IS_WP_ENGAGE_OP(p) && (end - start < HPAGE_SIZE))) {
>> + spin_unlock(ptl);
>> + split_huge_pmd(vma, pmd, start);
>> + goto process_smaller_pages;
>> + }
>> + if (IS_GET_OP(p)) {
>> + len = (end - start)/PAGE_SIZE;
>> + if (p->max_pages && p->found_pages + len > p->max_pages)
>> + len = p->max_pages - p->found_pages;
>> +
>> + ret = add_to_out(pmd_wt, vma->vm_file, pmd_present(*pmd),
>> + is_swap_pmd(*pmd), p, start, len);
>> + if (ret) {
>> + spin_unlock(ptl);
>> + return ret;
>> + }
>> + }
>> + spin_unlock(ptl);
>> + if (IS_WP_ENGAGE_OP(p) && pmd_wt)
>> + ret = wp_range_async(vma, start, HPAGE_SIZE);
>> + return ret;
>> + }
>> +process_smaller_pages:
>> + if (pmd_trans_unstable(pmd))
>> + return 0;
>> +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
>> +
>> + pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
>> + if (IS_GET_OP(p)) {
>> + for (addr = start;
>> + addr < end && !ret && (!p->max_pages || (p->found_pages < p->max_pages));
>> + pte++, addr += PAGE_SIZE) {
>> + ret = add_to_out(!is_pte_uffd_wp(*pte), vma->vm_file, pte_present(*pte),
>> + is_swap_pte(*pte), p, addr, 1);
>> + if (ret)
>> + break;
>> + }
>> + }
>> + pte_unmap_unlock(pte - 1, ptl);
>> + if (!ret && IS_WP_ENGAGE_OP(p))
>> + ret = wp_range_async(vma, start, addr - start);
>
> IIUC you wanted to cover the case where e.g. the output buffer is full or
> max_pages specified, so the partial pages still got wr-protected
> atomically.
>
> In that case I think we need to at least allow ret==-ENOSPC here.
The error handling will be updated in next version.
>
>> +
>> + cond_resched();
>> + return ret;
>> +}
>> +
>> +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, int depth,
>> + struct mm_walk *walk)
>> +{
>> + struct pagemap_scan_private *p = walk->private;
>> + struct vm_area_struct *vma = walk->vma;
>> + unsigned int len;
>> + int ret = 0;
>> +
>> + if (vma) {
>> + len = (end - addr)/PAGE_SIZE;
>> + if (p->max_pages && p->found_pages + len > p->max_pages)
>> + len = p->max_pages - p->found_pages;
>> + ret = add_to_out(false, vma->vm_file, false, false, p, addr, len);
>> + }
>> + return ret;
>> +}
>> +
>> +/* No hugetlb support is present. */
>> +static const struct mm_walk_ops pagemap_scan_ops = {
>> + .test_walk = pagemap_scan_test_walk,
>> + .pmd_entry = pagemap_scan_pmd_entry,
>> + .pte_hole = pagemap_scan_pte_hole,
>> +};
>> +
>> +static long do_pagemap_cmd(struct mm_struct *mm, struct pagemap_scan_arg *arg)
>> +{
>> + unsigned long empty_slots, vec_index = 0;
>> + unsigned long __user start, end;
>> + unsigned long __start, __end;
>> + struct page_region __user *vec;
>> + struct pagemap_scan_private p;
>> + int ret;
>> +
>> + start = (unsigned long)untagged_addr(arg->start);
>> + vec = (struct page_region *)untagged_addr(arg->vec);
>> + if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
>> + return -EINVAL;
>> + if (IS_GET_OP(arg) && ((arg->vec_len == 0) ||
>> + (!access_ok((void __user *)vec, arg->vec_len * sizeof(struct page_region)))))
>> + return -ENOMEM;
>
> Is ENOMEM the right error code? I think -EINVAL seems more proper.
Will be updated.
>
>> + if ((arg->flags & ~PAGEMAP_WP_ENGAGE) || (arg->required_mask & ~PAGEMAP_OP_MASK) ||
>> + (arg->anyof_mask & ~PAGEMAP_OP_MASK) || (arg->excluded_mask & ~PAGEMAP_OP_MASK) ||
>> + (arg->return_mask & ~PAGEMAP_OP_MASK))
>> + return -EINVAL;
>> + if (IS_GET_OP(arg) && ((!arg->required_mask && !arg->anyof_mask && !arg->excluded_mask) ||
>> + !arg->return_mask))
>> + return -EINVAL;
>> + /* The non-WT flags cannot be obtained if PAGEMAP_WP_ENGAGE is also specified. */
>> + if (IS_WP_ENGAGE_OP(arg) && ((arg->required_mask & PAGEMAP_NONWT_OP_MASK) ||
>> + (arg->anyof_mask & PAGEMAP_NONWT_OP_MASK)))
>> + return -EINVAL;
>
> The whole chunk is not very readable. Some emptly lines would be nice to
> have here and there, and some groupings. No need to keep the condition so
> long. How about something like:
>
> start = (unsigned long)untagged_addr(arg->start);
> vec = (struct page_region *)untagged_addr(arg->vec);
>
> /* Detect illegal cmds */
> if ((arg->flags & ~PAGEMAP_OP_MASK))
> return -EINVAL;
>
> /* Validate common parameters */
> if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
> return -EINVAL;
>
> /* Validate parameters for each cmd */
> if (PAGEMAP_OP_GET(arg)) {
> ...
> }
>
> if (PAGEMAP_OP_WP(arg)) {
> ...
> }
I'll try to make it redable some what.
>> +
>> + end = start + arg->len;
>> + p.max_pages = arg->max_pages;
>> + p.found_pages = 0;
>> + p.flags = arg->flags;
>> + p.required_mask = arg->required_mask;
>> + p.anyof_mask = arg->anyof_mask;
>> + p.excluded_mask = arg->excluded_mask;
>> + p.return_mask = arg->return_mask;
>> + p.prev.len = 0;
>> + p.vec_len = (PAGEMAP_WALK_SIZE >> PAGE_SHIFT);
>> +
>> + if (IS_GET_OP(arg)) {
>> + p.vec = kmalloc_array(p.vec_len, sizeof(struct page_region), GFP_KERNEL);
>> + if (!p.vec)
>> + return -ENOMEM;
>> + } else {
>> + p.vec = NULL;
>> + }
>> + __start = __end = start;
>> + while (__end < end) {
>> + p.vec_index = 0;
>> + empty_slots = arg->vec_len - vec_index;
>> + if (p.vec_len > empty_slots)
>> + p.vec_len = empty_slots;
>> +
>> + __end = (__start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
>> + if (__end > end)
>> + __end = end;
>> +
>> + mmap_read_lock(mm);
>> + ret = walk_page_range(mm, __start, __end, &pagemap_scan_ops, &p);
>> + mmap_read_unlock(mm);
>> + if (!(!ret || ret == -ENOSPC))
>> + goto free_data;
>> +
>> + __start = __end;
>> + if (IS_GET_OP(arg) && p.vec_index) {
>> + if (copy_to_user(&vec[vec_index], p.vec,
>> + p.vec_index * sizeof(struct page_region))) {
>> + ret = -EFAULT;
>> + goto free_data;
>> + }
>> + vec_index += p.vec_index;
>> + }
>> + }
>> + ret = export_prev_to_out(&p, vec, &vec_index);
>> + if (!ret)
>> + ret = vec_index;
>> +free_data:
>> + if (IS_GET_OP(arg))
>> + kfree(p.vec);
>> +
>> + return ret;
>> +}
>> +
>> +static long pagemap_scan_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
>> +{
>> + struct pagemap_scan_arg __user *uarg = (struct pagemap_scan_arg __user *)arg;
>> + struct mm_struct *mm = file->private_data;
>> + struct pagemap_scan_arg argument;
>> +
>> + if (cmd == PAGEMAP_SCAN) {
>> + if (copy_from_user(&argument, uarg, sizeof(struct pagemap_scan_arg)))
>> + return -EFAULT;
>> + return do_pagemap_cmd(mm, &argument);
>> + }
>> + return -EINVAL;
>> +}
>> +
>> const struct file_operations proc_pagemap_operations = {
>> .llseek = mem_lseek, /* borrow this */
>> .read = pagemap_read,
>> .open = pagemap_open,
>> .release = pagemap_release,
>> + .unlocked_ioctl = pagemap_scan_ioctl,
>> + .compat_ioctl = pagemap_scan_ioctl,
>> };
>> #endif /* CONFIG_PROC_PAGE_MONITOR */
>>
>> diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
>> index b7b56871029c..6d03a903a745 100644
>> --- a/include/uapi/linux/fs.h
>> +++ b/include/uapi/linux/fs.h
>> @@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
>> #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
>> RWF_APPEND)
>>
>> +/* Pagemap ioctl */
>> +#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
>> +
>> +/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
>> +#define PAGE_IS_WT (1 << 0)
>> +#define PAGE_IS_FILE (1 << 1)
>> +#define PAGE_IS_PRESENT (1 << 2)
>> +#define PAGE_IS_SWAPPED (1 << 3)
>> +
>> +/*
>> + * struct page_region - Page region with bitmap flags
>> + * @start: Start of the region
>> + * @len: Length of the region
>> + * bitmap: Bits sets for the region
>> + */
>> +struct page_region {
>> + __u64 start;
>> + __u64 len;
>> + __u64 bitmap;
>> +};
>> +
>> +/*
>> + * struct pagemap_scan_arg - Pagemap ioctl argument
>> + * @start: Starting address of the region
>> + * @len: Length of the region (All the pages in this length are included)
>> + * @vec: Address of page_region struct array for output
>> + * @vec_len: Length of the page_region struct array
>> + * @max_pages: Optional max return pages
>> + * @flags: Flags for the IOCTL
>> + * @required_mask: Required mask - All of these bits have to be set in the PTE
>> + * @anyof_mask: Any mask - Any of these bits are set in the PTE
>> + * @excluded_mask: Exclude mask - None of these bits are set in the PTE
>> + * @return_mask: Bits that are to be reported in page_region
>> + */
>> +struct pagemap_scan_arg {
>> + __u64 start;
>> + __u64 len;
>> + __u64 vec;
>> + __u64 vec_len;
>> + __u32 max_pages;
>> + __u32 flags;
>> + __u64 required_mask;
>> + __u64 anyof_mask;
>> + __u64 excluded_mask;
>> + __u64 return_mask;
>> +};
>> +
>> +/* Special flags */
>> +#define PAGEMAP_WP_ENGAGE (1 << 0)
>> +
>> #endif /* _UAPI_LINUX_FS_H */
>> diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
>> index b7b56871029c..6d03a903a745 100644
>> --- a/tools/include/uapi/linux/fs.h
>> +++ b/tools/include/uapi/linux/fs.h
>> @@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
>> #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
>> RWF_APPEND)
>>
>> +/* Pagemap ioctl */
>> +#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
>> +
>> +/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
>> +#define PAGE_IS_WT (1 << 0)
>> +#define PAGE_IS_FILE (1 << 1)
>> +#define PAGE_IS_PRESENT (1 << 2)
>> +#define PAGE_IS_SWAPPED (1 << 3)
>> +
>> +/*
>> + * struct page_region - Page region with bitmap flags
>> + * @start: Start of the region
>> + * @len: Length of the region
>> + * bitmap: Bits sets for the region
>> + */
>> +struct page_region {
>> + __u64 start;
>> + __u64 len;
>> + __u64 bitmap;
>> +};
>> +
>> +/*
>> + * struct pagemap_scan_arg - Pagemap ioctl argument
>> + * @start: Starting address of the region
>> + * @len: Length of the region (All the pages in this length are included)
>> + * @vec: Address of page_region struct array for output
>> + * @vec_len: Length of the page_region struct array
>> + * @max_pages: Optional max return pages
>> + * @flags: Flags for the IOCTL
>> + * @required_mask: Required mask - All of these bits have to be set in the PTE
>> + * @anyof_mask: Any mask - Any of these bits are set in the PTE
>> + * @excluded_mask: Exclude mask - None of these bits are set in the PTE
>> + * @return_mask: Bits that are to be reported in page_region
>> + */
>> +struct pagemap_scan_arg {
>> + __u64 start;
>> + __u64 len;
>> + __u64 vec;
>> + __u64 vec_len;
>> + __u32 max_pages;
>> + __u32 flags;
>> + __u64 required_mask;
>> + __u64 anyof_mask;
>> + __u64 excluded_mask;
>> + __u64 return_mask;
>> +};
>> +
>> +/* Special flags */
>> +#define PAGEMAP_WP_ENGAGE (1 << 0)
>> +
>> #endif /* _UAPI_LINUX_FS_H */
>
> Better move the tools/ change out of the core changes, probably can be
> squashed into the selftest patch later.
Will do.
>
> Thanks,
>
>> --
>> 2.30.2
>>
>
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-30 8:38 ` Muhammad Usama Anjum
@ 2023-01-30 21:27 ` Peter Xu
2023-01-31 8:40 ` Muhammad Usama Anjum
0 siblings, 1 reply; 21+ messages in thread
From: Peter Xu @ 2023-01-30 21:27 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: David Hildenbrand, Andrew Morton, Michał Mirosław,
Andrei Vagin, Danylo Mocherniuk, Paul Gofman, Cyrill Gorcunov,
Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Matthew Wilcox, Pasha Tatashin, Mike Rapoport,
Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva, Dan Williams,
linux-kernel, linux-fsdevel, linux-mm, linux-kselftest, Greg KH,
kernel
On Mon, Jan 30, 2023 at 01:38:16PM +0500, Muhammad Usama Anjum wrote:
> On 1/27/23 8:32 PM, Peter Xu wrote:
> > On Fri, Jan 27, 2023 at 11:47:14AM +0500, Muhammad Usama Anjum wrote:
> >>>> diff --git a/mm/memory.c b/mm/memory.c
> >>>> index 4000e9f017e0..8c03b133d483 100644
> >>>> --- a/mm/memory.c
> >>>> +++ b/mm/memory.c
> >>>> @@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
> >>>>
> >>>> if (likely(!unshare)) {
> >>>> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
> >>>> + if (userfaultfd_wp_async(vma)) {
> >>>> + /*
> >>>> + * Nothing needed (cache flush, TLB invalidations,
> >>>> + * etc.) because we're only removing the uffd-wp bit,
> >>>> + * which is completely invisible to the user. This
> >>>> + * falls through to possible CoW.
> >>>
> >>> Here it says it falls through to CoW, but..
> >>>
> >>>> + */
> >>>> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> >>>> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
> >>>> + pte_clear_uffd_wp(*vmf->pte));
> >>>> + return 0;
> >>>
> >>> ... it's not doing so. The original lines should do:
> >>>
> >>> https://lore.kernel.org/all/Y8qq0dKIJBshua+X@x1n/
> >
> > [1]
> >
> >>>
> >>> Side note: you cannot modify pgtable after releasing the pgtable lock.
> >>> It's racy.
> >> If I don't unlock and return after removing the UFFD_WP flag in case of
> >> async wp, the target just gets stuck. Maybe the pte lock is not unlocked in
> >> some path.
> >>
> >> If I unlock and don't return, the crash happens.
> >>
> >> So I'd put unlock and return from here. Please comment on the below patch
> >> and what do you think should be done. I've missed something.
> >
> > Have you tried to just use exactly what I suggested in [1]? I'll paste
> > again:
> >
> > ---8<---
> > diff --git a/mm/memory.c b/mm/memory.c
> > index 4000e9f017e0..09aab434654c 100644
> > --- a/mm/memory.c
> > +++ b/mm/memory.c
> > @@ -3351,8 +3351,20 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
> >
> > if (likely(!unshare)) {
> > if (userfaultfd_pte_wp(vma, *vmf->pte)) {
> > - pte_unmap_unlock(vmf->pte, vmf->ptl);
> > - return handle_userfault(vmf, VM_UFFD_WP);
> > + if (userfaultfd_uffd_wp_async(vma)) {
> > + /*
> > + * Nothing needed (cache flush, TLB
> > + * invalidations, etc.) because we're only
> > + * removing the uffd-wp bit, which is
> > + * completely invisible to the user.
> > + * This falls through to possible CoW.
> > + */
> > + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
> > + pte_clear_uffd_wp(*vmf->pte));
> > + } else {
> > + pte_unmap_unlock(vmf->pte, vmf->ptl);
> > + return handle_userfault(vmf, VM_UFFD_WP);
> > + }
> > }
> > ---8<---
> >
> > Note that there's no "return", neither the unlock. The lock is used in the
> > follow up write fault resolution and it's released later.
> I've tried out the exact patch above. This doesn't work. The pages keep
> their WP flag even after being resolved in do_wp_page() while is written on
> the page.
>
> So I'd added pte_unmap_unlock() and return 0 from here. This makes the
> patch to work. Maybe you can try this on your end to see what I'm seeing here?
Oh maybe it's because it didn't update orig_pte. If you want, you can try
again with doing so by changing:
set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
pte_clear_uffd_wp(*vmf->pte));
into:
pte_t pte = pte_clear_uffd_wp(*vmf->pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
/* Update this to be prepared for following up CoW handling */
vmf->orig_pte = pte;
>
> >
> > Meanwhile please fully digest how pgtable lock is used in this path before
> > moving forward on any of such changes.
> >
> >>
> >>>
> >>>> + }
> >>>> pte_unmap_unlock(vmf->pte, vmf->ptl);
> >>>> return handle_userfault(vmf, VM_UFFD_WP);
> >>>> }
> >>>> @@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
> >>>>
> >>>> if (vma_is_anonymous(vmf->vma)) {
> >>>> if (likely(!unshare) &&
> >>>> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
> >>>> - return handle_userfault(vmf, VM_UFFD_WP);
> >>>> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
> >>>> + if (userfaultfd_wp_async(vmf->vma)) {
> >>>> + /*
> >>>> + * Nothing needed (cache flush, TLB invalidations,
> >>>> + * etc.) because we're only removing the uffd-wp bit,
> >>>> + * which is completely invisible to the user. This
> >>>> + * falls through to possible CoW.
> >>>> + */
> >>>> + set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
> >>>> + pmd_clear_uffd_wp(*vmf->pmd));
> >>>
> >>> This is for THP, not hugetlb.
> >>>
> >>> Clearing uffd-wp bit here for the whole pmd is wrong to me, because we
> >>> track writes in small page sizes only. We should just split.
> >> By detecting if the fault is async wp, just splitting the PMD doesn't work.
> >> The below given snippit is working right now. But definately, the fault of
> >> the whole PMD is being resolved which if we can bypass by correctly
> >> splitting would be highly desirable. Can you please take a look on UFFD
> >> side and suggest the changes? It would be much appreciated. I'm attaching
> >> WIP v9 patches for you to apply on next(next-20230105) and pagemap_ioctl
> >> selftest can be ran to test things after making changes.
> >
> > Can you elaborate why thp split didn't work? Or if you want, I can look
> > into this and provide the patch to enable uffd async mode.
> Sorry, I was doing the wrong way. Splitting the page does work. What do you
> think about the following:
>
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3351,6 +3351,17 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>
> if (likely(!unshare)) {
> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
> + if (userfaultfd_wp_async(vma)) {
> + /*
> + * Nothing needed (cache flush, TLB invalidations,
> + * etc.) because we're only removing the uffd-wp bit,
> + * which is completely invisible to the user.
> + */
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
> + pte_clear_uffd_wp(*vmf->pte));
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return 0;
Please give it a shot with above to see whether we can avoid the "return 0"
here.
> + }
> pte_unmap_unlock(vmf->pte, vmf->ptl);
> return handle_userfault(vmf, VM_UFFD_WP);
> }
> @@ -4812,8 +4823,13 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault
> *vmf)
>
> if (vma_is_anonymous(vmf->vma)) {
> if (likely(!unshare) &&
> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
> + if (userfaultfd_wp_async(vmf->vma)) {
> + __split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
> + return 0;
Same here, I hope it'll work for you if you just goto __split_huge_pmd()
right below and return with VM_FAULT_FALLBACK. It avoids one more round of
fault just like the pte case above.
> + }
> return handle_userfault(vmf, VM_UFFD_WP);
> + }
> return do_huge_pmd_wp_page(vmf);
> }
--
Peter Xu
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
2023-01-30 11:12 ` Muhammad Usama Anjum
@ 2023-01-30 21:34 ` Peter Xu
0 siblings, 0 replies; 21+ messages in thread
From: Peter Xu @ 2023-01-30 21:34 UTC (permalink / raw)
To: Muhammad Usama Anjum
Cc: David Hildenbrand, Andrew Morton, Michał Mirosław,
Andrei Vagin, Danylo Mocherniuk, Paul Gofman, Cyrill Gorcunov,
Alexander Viro, Shuah Khan, Christian Brauner, Yang Shi,
Vlastimil Babka, Liam R . Howlett, Yun Zhou, Suren Baghdasaryan,
Alex Sierra, Matthew Wilcox, Pasha Tatashin, Mike Rapoport,
Nadav Amit, Axel Rasmussen, Gustavo A . R . Silva, Dan Williams,
linux-kernel, linux-fsdevel, linux-mm, linux-kselftest, Greg KH,
kernel
On Mon, Jan 30, 2023 at 04:12:25PM +0500, Muhammad Usama Anjum wrote:
[...]
> >
> >> +#define IS_WP_ENGAGE_OP(a) (a->flags & PAGEMAP_WP_ENGAGE)
> >> +#define IS_GET_OP(a) (a->vec)
> >
> > Having a->vec to imply the GET is fine, but IMHO not as clean as having
> > each OP a bit in the flags.
> >
> > How about:
> >
> > #define PAGEMAP_OP_GET (1UL << 0)
> > #define PAGEMAP_OP_WP (1UL << 1)
> > #define PAGEMAP_OP_MASK (PAGEMAP_OP_GET | PAGEMAP_OP_WP)
> >
> > ?
> >
> > Then a->vec requried for PAGEMAP_OP_GET.
> I had something like PAGEMAP_OP_GET and PAGEMAP_OP_WP flags once until I
> was asked to only keep WP flag and vec != NULL will imply GET flag in a
> previous review. I'm ready to change again if there are more notes on it?
Sorry to know that; that's somewhat frustrating when you need to go back
and forth on subjective comments like mine. So if you still think your
original way is better then at least you have two votes now. :) Your call
to choose any, I have a preference as I said but not that strong.
[...]
> >
> >> + struct pagemap_scan_private *p, unsigned long addr, unsigned int len)
> >> +{
> >> + unsigned long bitmap, cur = PAGEMAP_SCAN_BITMAP(wt, file, pres, swap);
> >> + bool cpy = true;
> >> + struct page_region *prev = &p->prev;
> >> +
> >> + if (HAS_NO_SPACE(p))
> >> + return -ENOSPC;
> >
> > This can be moved to below [1], we should stop scanning immediately if the
> > condition met.
> It would be definately possible. Wouldn't it be strange to return error
> when the operation was successful? A function should return error only when
> the current execution is unsuccessful. So I'm returning error only when the
> space is full and there is no space left in the buffer anymore.
I would expect the user to always provide some more space than they expect
because the merging of page_regions are kind of unpredictable from the
user's POV.
But yeah, maybe you're right. I'm fine to keep that as is.
Thanks,
--
Peter Xu
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support
2023-01-30 21:27 ` Peter Xu
@ 2023-01-31 8:40 ` Muhammad Usama Anjum
0 siblings, 0 replies; 21+ messages in thread
From: Muhammad Usama Anjum @ 2023-01-31 8:40 UTC (permalink / raw)
To: Peter Xu
Cc: Muhammad Usama Anjum, David Hildenbrand, Andrew Morton,
Michał Mirosław, Andrei Vagin, Danylo Mocherniuk,
Paul Gofman, Cyrill Gorcunov, Alexander Viro, Shuah Khan,
Christian Brauner, Yang Shi, Vlastimil Babka, Liam R . Howlett,
Yun Zhou, Suren Baghdasaryan, Alex Sierra, Matthew Wilcox,
Pasha Tatashin, Mike Rapoport, Nadav Amit, Axel Rasmussen,
Gustavo A . R . Silva, Dan Williams, linux-kernel, linux-fsdevel,
linux-mm, linux-kselftest, Greg KH, kernel
On 1/31/23 2:27 AM, Peter Xu wrote:
> On Mon, Jan 30, 2023 at 01:38:16PM +0500, Muhammad Usama Anjum wrote:
>> On 1/27/23 8:32 PM, Peter Xu wrote:
>>> On Fri, Jan 27, 2023 at 11:47:14AM +0500, Muhammad Usama Anjum wrote:
>>>>>> diff --git a/mm/memory.c b/mm/memory.c
>>>>>> index 4000e9f017e0..8c03b133d483 100644
>>>>>> --- a/mm/memory.c
>>>>>> +++ b/mm/memory.c
>>>>>> @@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>>>>>>
>>>>>> if (likely(!unshare)) {
>>>>>> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
>>>>>> + if (userfaultfd_wp_async(vma)) {
>>>>>> + /*
>>>>>> + * Nothing needed (cache flush, TLB invalidations,
>>>>>> + * etc.) because we're only removing the uffd-wp bit,
>>>>>> + * which is completely invisible to the user. This
>>>>>> + * falls through to possible CoW.
>>>>>
>>>>> Here it says it falls through to CoW, but..
>>>>>
>>>>>> + */
>>>>>> + pte_unmap_unlock(vmf->pte, vmf->ptl);
>>>>>> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
>>>>>> + pte_clear_uffd_wp(*vmf->pte));
>>>>>> + return 0;
>>>>>
>>>>> ... it's not doing so. The original lines should do:
>>>>>
>>>>> https://lore.kernel.org/all/Y8qq0dKIJBshua+X@x1n/
>>>
>>> [1]
>>>
>>>>>
>>>>> Side note: you cannot modify pgtable after releasing the pgtable lock.
>>>>> It's racy.
>>>> If I don't unlock and return after removing the UFFD_WP flag in case of
>>>> async wp, the target just gets stuck. Maybe the pte lock is not unlocked in
>>>> some path.
>>>>
>>>> If I unlock and don't return, the crash happens.
>>>>
>>>> So I'd put unlock and return from here. Please comment on the below patch
>>>> and what do you think should be done. I've missed something.
>>>
>>> Have you tried to just use exactly what I suggested in [1]? I'll paste
>>> again:
>>>
>>> ---8<---
>>> diff --git a/mm/memory.c b/mm/memory.c
>>> index 4000e9f017e0..09aab434654c 100644
>>> --- a/mm/memory.c
>>> +++ b/mm/memory.c
>>> @@ -3351,8 +3351,20 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>>>
>>> if (likely(!unshare)) {
>>> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
>>> - pte_unmap_unlock(vmf->pte, vmf->ptl);
>>> - return handle_userfault(vmf, VM_UFFD_WP);
>>> + if (userfaultfd_uffd_wp_async(vma)) {
>>> + /*
>>> + * Nothing needed (cache flush, TLB
>>> + * invalidations, etc.) because we're only
>>> + * removing the uffd-wp bit, which is
>>> + * completely invisible to the user.
>>> + * This falls through to possible CoW.
>>> + */
>>> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
>>> + pte_clear_uffd_wp(*vmf->pte));
>>> + } else {
>>> + pte_unmap_unlock(vmf->pte, vmf->ptl);
>>> + return handle_userfault(vmf, VM_UFFD_WP);
>>> + }
>>> }
>>> ---8<---
>>>
>>> Note that there's no "return", neither the unlock. The lock is used in the
>>> follow up write fault resolution and it's released later.
>> I've tried out the exact patch above. This doesn't work. The pages keep
>> their WP flag even after being resolved in do_wp_page() while is written on
>> the page.
>>
>> So I'd added pte_unmap_unlock() and return 0 from here. This makes the
>> patch to work. Maybe you can try this on your end to see what I'm seeing here?
>
> Oh maybe it's because it didn't update orig_pte. If you want, you can try
> again with doing so by changing:
>
> set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
> pte_clear_uffd_wp(*vmf->pte));
>
> into:
>
> pte_t pte = pte_clear_uffd_wp(*vmf->pte);
> set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
> /* Update this to be prepared for following up CoW handling */
> vmf->orig_pte = pte;
>
It works.
>>
>>>
>>> Meanwhile please fully digest how pgtable lock is used in this path before
>>> moving forward on any of such changes.
>>>
>>>>
>>>>>
>>>>>> + }
>>>>>> pte_unmap_unlock(vmf->pte, vmf->ptl);
>>>>>> return handle_userfault(vmf, VM_UFFD_WP);
>>>>>> }
>>>>>> @@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
>>>>>>
>>>>>> if (vma_is_anonymous(vmf->vma)) {
>>>>>> if (likely(!unshare) &&
>>>>>> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
>>>>>> - return handle_userfault(vmf, VM_UFFD_WP);
>>>>>> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
>>>>>> + if (userfaultfd_wp_async(vmf->vma)) {
>>>>>> + /*
>>>>>> + * Nothing needed (cache flush, TLB invalidations,
>>>>>> + * etc.) because we're only removing the uffd-wp bit,
>>>>>> + * which is completely invisible to the user. This
>>>>>> + * falls through to possible CoW.
>>>>>> + */
>>>>>> + set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd,
>>>>>> + pmd_clear_uffd_wp(*vmf->pmd));
>>>>>
>>>>> This is for THP, not hugetlb.
>>>>>
>>>>> Clearing uffd-wp bit here for the whole pmd is wrong to me, because we
>>>>> track writes in small page sizes only. We should just split.
>>>> By detecting if the fault is async wp, just splitting the PMD doesn't work.
>>>> The below given snippit is working right now. But definately, the fault of
>>>> the whole PMD is being resolved which if we can bypass by correctly
>>>> splitting would be highly desirable. Can you please take a look on UFFD
>>>> side and suggest the changes? It would be much appreciated. I'm attaching
>>>> WIP v9 patches for you to apply on next(next-20230105) and pagemap_ioctl
>>>> selftest can be ran to test things after making changes.
>>>
>>> Can you elaborate why thp split didn't work? Or if you want, I can look
>>> into this and provide the patch to enable uffd async mode.
>> Sorry, I was doing the wrong way. Splitting the page does work. What do you
>> think about the following:
>>
>> --- a/mm/memory.c
>> +++ b/mm/memory.c
>> @@ -3351,6 +3351,17 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
>>
>> if (likely(!unshare)) {
>> if (userfaultfd_pte_wp(vma, *vmf->pte)) {
>> + if (userfaultfd_wp_async(vma)) {
>> + /*
>> + * Nothing needed (cache flush, TLB invalidations,
>> + * etc.) because we're only removing the uffd-wp bit,
>> + * which is completely invisible to the user.
>> + */
>> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
>> + pte_clear_uffd_wp(*vmf->pte));
>> + pte_unmap_unlock(vmf->pte, vmf->ptl);
>> + return 0;
>
> Please give it a shot with above to see whether we can avoid the "return 0"
> here.
>
>> + }
>> pte_unmap_unlock(vmf->pte, vmf->ptl);
>> return handle_userfault(vmf, VM_UFFD_WP);
>> }
>> @@ -4812,8 +4823,13 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault
>> *vmf)
>>
>> if (vma_is_anonymous(vmf->vma)) {
>> if (likely(!unshare) &&
>> - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd))
>> + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) {
>> + if (userfaultfd_wp_async(vmf->vma)) {
>> + __split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
>> + return 0;
>
> Same here, I hope it'll work for you if you just goto __split_huge_pmd()
> right below and return with VM_FAULT_FALLBACK. It avoids one more round of
> fault just like the pte case above.
>
It works as well.
>> + }
>> return handle_userfault(vmf, VM_UFFD_WP);
>> + }
>> return do_huge_pmd_wp_page(vmf);
>> }
>
--
BR,
Muhammad Usama Anjum
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2023-01-31 8:45 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-24 8:43 [PATCH v8 0/4] Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Muhammad Usama Anjum
2023-01-24 10:43 ` kernel test robot
2023-01-24 11:04 ` kernel test robot
2023-01-24 11:15 ` kernel test robot
2023-01-26 23:05 ` Peter Xu
2023-01-27 6:47 ` Muhammad Usama Anjum
2023-01-27 15:32 ` Peter Xu
2023-01-30 8:38 ` Muhammad Usama Anjum
2023-01-30 21:27 ` Peter Xu
2023-01-31 8:40 ` Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 2/4] userfaultfd: split mwriteprotect_range() Muhammad Usama Anjum
2023-01-24 10:23 ` kernel test robot
2023-01-27 17:05 ` Peter Xu
2023-01-30 9:10 ` Muhammad Usama Anjum
2023-01-24 8:43 ` [PATCH v8 3/4] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs Muhammad Usama Anjum
2023-01-24 10:02 ` kernel test robot
2023-01-27 17:36 ` Peter Xu
2023-01-30 11:12 ` Muhammad Usama Anjum
2023-01-30 21:34 ` Peter Xu
2023-01-24 8:43 ` [PATCH v8 4/4] selftests: vm: add pagemap ioctl tests Muhammad Usama Anjum
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.