[PATCH 0/6] mm/uffd: Misc fix for uffd-wp and one more test

[PATCH 0/6] mm/uffd: Misc fix for uffd-wp and one more test

[Peter Xu]

This series is based on (mostly) v5.12-rc8-mmots-2021-04-21-23-08.  It tries to
fix some corner case bugs for uffd-wp on either thp or fork().  Then it
introduced a new test with pagemap/pageout.

Patch layout:

Patch 1:    cleanup for THP, it'll slightly simplify the follow up patches
Patch 2-4:  misc fixes for uffd-wp here and there; please refer to each patch
Patch 5:    add pagemap support for uffd-wp
Patch 6:    add pagemap/pageout test for uffd-wp

The last test introduced can also verify some of the fixes in previous patches,
as the test will fail without the fixes.  However it's not easy to verify all
the changes in patch 2-4, but hopefully they can still be properly reviewed.

Note that if considering the ongoing uffd-wp shmem & hugetlbfs work, patch 5
will be incomplete as it's missing e.g. hugetlbfs part or the special swap pte
detection.  However that's not needed in this series, and since that series is
still during review, this series does not depend on that one (the last test
only runs with anonymous memory, not file-backed).  So this series can be
merged even before that series.

Please review, thanks.

Peter Xu (6):
  mm/thp: Simplify copying of huge zero page pmd when fork
  mm/userfaultfd: Fix uffd-wp special cases for fork()
  mm/userfaultfd: Fix a few thp pmd missing uffd-wp bit
  mm/userfaultfd: Fail uffd-wp registeration if not supported
  mm/pagemap: Export uffd-wp protection information
  userfaultfd/selftests: Add pagemap uffd-wp test

 Documentation/admin-guide/mm/pagemap.rst |   2 +
 fs/proc/task_mmu.c                       |   9 ++
 fs/userfaultfd.c                         |   9 +-
 include/linux/huge_mm.h                  |   2 +-
 include/linux/swapops.h                  |   2 +
 mm/huge_memory.c                         |  36 +++---
 mm/memory.c                              |  25 ++--
 tools/testing/selftests/vm/userfaultfd.c | 154 +++++++++++++++++++++++
 8 files changed, 206 insertions(+), 33 deletions(-)

-- 
2.26.2

[PATCH 1/6] mm/thp: Simplify copying of huge zero page pmd when fork

[Peter Xu]

Huge zero page is handled in a special path in copy_huge_pmd(), however it
should share most codes with a normal thp page.  Trying to share more code with
it by removing the special path.  The only leftover so far is the huge zero
page refcounting (mm_get_huge_zero_page()), because that's separately done with
a global counter.

This prepares for a future patch to modify the huge pmd to be installed, so
that we don't need to duplicate it explicitly into huge zero page case too.

Cc: Kirill A. Shutemov <kirill@shutemov.name>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 mm/huge_memory.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 98456017744d6..22bf2d0fff79b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1076,17 +1076,13 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	 * a page table.
 	 */
 	if (is_huge_zero_pmd(pmd)) {
-		struct page *zero_page;
 		/*
 		 * get_huge_zero_page() will never allocate a new page here,
 		 * since we already have a zero page to copy. It just takes a
 		 * reference.
 		 */
-		zero_page = mm_get_huge_zero_page(dst_mm);
-		set_huge_zero_page(pgtable, dst_mm, vma, addr, dst_pmd,
-				zero_page);
-		ret = 0;
-		goto out_unlock;
+		mm_get_huge_zero_page(dst_mm);
+		goto out_zero_page;
 	}
 
 	src_page = pmd_page(pmd);
@@ -1110,6 +1106,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	get_page(src_page);
 	page_dup_rmap(src_page, true);
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+out_zero_page:
 	mm_inc_nr_ptes(dst_mm);
 	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
 
-- 
2.26.2

[PATCH 2/6] mm/userfaultfd: Fix uffd-wp special cases for fork()

[Peter Xu]

We tried to do something similar in b569a1760782 ("userfaultfd: wp: drop
_PAGE_UFFD_WP properly when fork") previously, but it's not doing it all
right..  A few fixes around the code path:

  1. We were referencing VM_UFFD_WP vm_flags on the _old_ vma rather than the
     new vma.  That's overlooked in b569a1760782, so it won't work as expected.
     Thanks to the recent rework on fork code (7a4830c380f3a8b3), we can easily
     get the new vma now, so switch the checks to that.

  2. Dropping the uffd-wp bit in copy_huge_pmd() could be wrong if the huge pmd
     is a migration huge pmd.  When it happens, instead of using pmd_uffd_wp(),
     we should use pmd_swp_uffd_wp(). The fix is simply to handle them separately.

  3. Forget to carry over uffd-wp bit for a write migration huge pmd entry.
     This also happens in copy_huge_pmd(), where we converted a write huge
     migration entry into a read one.

  4. In copy_nonpresent_pte(), drop uffd-wp if necessary for swap ptes.

  5. In copy_present_page() when COW is enforced when fork(), we also need to
     pass over the uffd-wp bit if VM_UFFD_WP is armed on the new vma, and when
     the pte to be copied has uffd-wp bit set.

Remove the comment in copy_present_pte() about this.  It won't help a huge lot
to only comment there, but comment everywhere would be an overkill.  Let's
assume the commit messages would help.

Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
Fixes: b569a1760782f3da03ff718d61f74163dea599ff
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/linux/huge_mm.h |  2 +-
 mm/huge_memory.c        | 23 ++++++++++-------------
 mm/memory.c             | 25 +++++++++++++------------
 3 files changed, 24 insertions(+), 26 deletions(-)

diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 9626fda5efcea..60dad7c88d72b 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -10,7 +10,7 @@
 vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf);
 int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		  pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
-		  struct vm_area_struct *vma);
+		  struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma);
 void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd);
 int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		  pud_t *dst_pud, pud_t *src_pud, unsigned long addr,
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 22bf2d0fff79b..20a4569895254 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1014,7 +1014,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
 
 int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		  pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
-		  struct vm_area_struct *vma)
+		  struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
 {
 	spinlock_t *dst_ptl, *src_ptl;
 	struct page *src_page;
@@ -1023,7 +1023,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	int ret = -ENOMEM;
 
 	/* Skip if can be re-fill on fault */
-	if (!vma_is_anonymous(vma))
+	if (!vma_is_anonymous(dst_vma))
 		return 0;
 
 	pgtable = pte_alloc_one(dst_mm);
@@ -1037,14 +1037,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	ret = -EAGAIN;
 	pmd = *src_pmd;
 
-	/*
-	 * Make sure the _PAGE_UFFD_WP bit is cleared if the new VMA
-	 * does not have the VM_UFFD_WP, which means that the uffd
-	 * fork event is not enabled.
-	 */
-	if (!(vma->vm_flags & VM_UFFD_WP))
-		pmd = pmd_clear_uffd_wp(pmd);
-
 #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
 	if (unlikely(is_swap_pmd(pmd))) {
 		swp_entry_t entry = pmd_to_swp_entry(pmd);
@@ -1055,11 +1047,15 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 			pmd = swp_entry_to_pmd(entry);
 			if (pmd_swp_soft_dirty(*src_pmd))
 				pmd = pmd_swp_mksoft_dirty(pmd);
+			if (pmd_swp_uffd_wp(*src_pmd))
+				pmd = pmd_swp_mkuffd_wp(pmd);
 			set_pmd_at(src_mm, addr, src_pmd, pmd);
 		}
 		add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 		mm_inc_nr_ptes(dst_mm);
 		pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+		if (!userfaultfd_wp(dst_vma))
+			pmd = pmd_swp_clear_uffd_wp(pmd);
 		set_pmd_at(dst_mm, addr, dst_pmd, pmd);
 		ret = 0;
 		goto out_unlock;
@@ -1095,11 +1091,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	 * best effort that the pinned pages won't be replaced by another
 	 * random page during the coming copy-on-write.
 	 */
-	if (unlikely(page_needs_cow_for_dma(vma, src_page))) {
+	if (unlikely(page_needs_cow_for_dma(src_vma, src_page))) {
 		pte_free(dst_mm, pgtable);
 		spin_unlock(src_ptl);
 		spin_unlock(dst_ptl);
-		__split_huge_pmd(vma, src_pmd, addr, false, NULL);
+		__split_huge_pmd(src_vma, src_pmd, addr, false, NULL);
 		return -EAGAIN;
 	}
 
@@ -1109,8 +1105,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 out_zero_page:
 	mm_inc_nr_ptes(dst_mm);
 	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
-
 	pmdp_set_wrprotect(src_mm, addr, src_pmd);
+	if (!userfaultfd_wp(dst_vma))
+		pmd = pmd_clear_uffd_wp(pmd);
 	pmd = pmd_mkold(pmd_wrprotect(pmd));
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);
 
diff --git a/mm/memory.c b/mm/memory.c
index 045daf58608f7..a17a53a7dade6 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -708,10 +708,10 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 
 static unsigned long
 copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
-		pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma,
-		unsigned long addr, int *rss)
+		pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *dst_vma,
+		struct vm_area_struct *src_vma, unsigned long addr, int *rss)
 {
-	unsigned long vm_flags = vma->vm_flags;
+	unsigned long vm_flags = dst_vma->vm_flags;
 	pte_t pte = *src_pte;
 	struct page *page;
 	swp_entry_t entry = pte_to_swp_entry(pte);
@@ -780,6 +780,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 			set_pte_at(src_mm, addr, src_pte, pte);
 		}
 	}
+	if (!userfaultfd_wp(dst_vma))
+		pte = pte_swp_clear_uffd_wp(pte);
 	set_pte_at(dst_mm, addr, dst_pte, pte);
 	return 0;
 }
@@ -845,6 +847,9 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma
 	/* All done, just insert the new page copy in the child */
 	pte = mk_pte(new_page, dst_vma->vm_page_prot);
 	pte = maybe_mkwrite(pte_mkdirty(pte), dst_vma);
+	if (userfaultfd_pte_wp(dst_vma, *src_pte))
+		/* Uffd-wp needs to be delivered to dest pte as well */
+		pte = pte_wrprotect(pte_mkuffd_wp(pte));
 	set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte);
 	return 0;
 }
@@ -894,12 +899,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 		pte = pte_mkclean(pte);
 	pte = pte_mkold(pte);
 
-	/*
-	 * Make sure the _PAGE_UFFD_WP bit is cleared if the new VMA
-	 * does not have the VM_UFFD_WP, which means that the uffd
-	 * fork event is not enabled.
-	 */
-	if (!(vm_flags & VM_UFFD_WP))
+	if (!userfaultfd_wp(dst_vma))
 		pte = pte_clear_uffd_wp(pte);
 
 	set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte);
@@ -974,7 +974,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 		if (unlikely(!pte_present(*src_pte))) {
 			entry.val = copy_nonpresent_pte(dst_mm, src_mm,
 							dst_pte, src_pte,
-							src_vma, addr, rss);
+							dst_vma, src_vma,
+							addr, rss);
 			if (entry.val)
 				break;
 			progress += 8;
@@ -1051,8 +1052,8 @@ copy_pmd_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 			|| pmd_devmap(*src_pmd)) {
 			int err;
 			VM_BUG_ON_VMA(next-addr != HPAGE_PMD_SIZE, src_vma);
-			err = copy_huge_pmd(dst_mm, src_mm,
-					    dst_pmd, src_pmd, addr, src_vma);
+			err = copy_huge_pmd(dst_mm, src_mm, dst_pmd, src_pmd,
+					    addr, dst_vma, src_vma);
 			if (err == -ENOMEM)
 				return -ENOMEM;
 			if (!err)
-- 
2.26.2

[PATCH 3/6] mm/userfaultfd: Fix a few thp pmd missing uffd-wp bit

[Peter Xu]

These include:

  1. When remove migration pmd entry, should keep the uffd-wp bit from swap
     pte.  Note that we need to do this after setting write bit just in case we
     need to remove it.

  2. When change huge pmd and convert write -> read migration entry, persist
     the same uffd-wp bit.

  3. When convert pmd to swap entry, should drop the uffd-wp bit always.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/linux/swapops.h | 2 ++
 mm/huge_memory.c        | 4 ++++
 2 files changed, 6 insertions(+)

diff --git a/include/linux/swapops.h b/include/linux/swapops.h
index d9b7c9132c2f6..7dd57303bb0c3 100644
--- a/include/linux/swapops.h
+++ b/include/linux/swapops.h
@@ -258,6 +258,8 @@ static inline swp_entry_t pmd_to_swp_entry(pmd_t pmd)
 
 	if (pmd_swp_soft_dirty(pmd))
 		pmd = pmd_swp_clear_soft_dirty(pmd);
+	if (pmd_swp_uffd_wp(pmd))
+		pmd = pmd_swp_clear_uffd_wp(pmd);
 	arch_entry = __pmd_to_swp_entry(pmd);
 	return swp_entry(__swp_type(arch_entry), __swp_offset(arch_entry));
 }
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 20a4569895254..e6423fe11a8f2 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1823,6 +1823,8 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 			newpmd = swp_entry_to_pmd(entry);
 			if (pmd_swp_soft_dirty(*pmd))
 				newpmd = pmd_swp_mksoft_dirty(newpmd);
+			if (pmd_swp_uffd_wp(*pmd))
+				newpmd = pmd_swp_mkuffd_wp(newpmd);
 			set_pmd_at(mm, addr, pmd, newpmd);
 		}
 		goto unlock;
@@ -3234,6 +3236,8 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)
 		pmde = pmd_mksoft_dirty(pmde);
 	if (is_write_migration_entry(entry))
 		pmde = maybe_pmd_mkwrite(pmde, vma);
+	if (pmd_swp_uffd_wp(*pvmw->pmd))
+		pmde = pmd_wrprotect(pmd_mkuffd_wp(pmde));
 
 	flush_cache_range(vma, mmun_start, mmun_start + HPAGE_PMD_SIZE);
 	if (PageAnon(new))
-- 
2.26.2

[PATCH 4/6] mm/userfaultfd: Fail uffd-wp registeration if not supported

[Peter Xu]

We should fail uffd-wp registration immediately if the arch does not even have
CONFIG_HAVE_ARCH_USERFAULTFD_WP defined.  That'll block also relevant ioctls on
e.g. UFFDIO_WRITEPROTECT because that'll check against VM_UFFD_WP, which can
only be applied with a success registration.

Remove the WP feature bit too for those archs when handling UFFDIO_API ioctl.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 fs/userfaultfd.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 14f92285d04f8..5dd78238cc156 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -1304,8 +1304,12 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 	vm_flags = 0;
 	if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING)
 		vm_flags |= VM_UFFD_MISSING;
-	if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP)
+	if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) {
+#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP
+		goto out;
+#endif
 		vm_flags |= VM_UFFD_WP;
+	}
 	if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) {
 #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
 		goto out;
@@ -1942,6 +1946,9 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
 	uffdio_api.features = UFFD_API_FEATURES;
 #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
 	uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS;
+#endif
+#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP
+	uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
 #endif
 	uffdio_api.ioctls = UFFD_API_IOCTLS;
 	ret = -EFAULT;
-- 
2.26.2

[PATCH 5/6] mm/pagemap: Export uffd-wp protection information

[Peter Xu]

Export the PTE/PMD status of uffd-wp to pagemap too.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 Documentation/admin-guide/mm/pagemap.rst | 2 ++
 fs/proc/task_mmu.c                       | 9 +++++++++
 2 files changed, 11 insertions(+)

diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst
index 340a5aee9b802..fb578fbbb76ca 100644
--- a/Documentation/admin-guide/mm/pagemap.rst
+++ b/Documentation/admin-guide/mm/pagemap.rst
@@ -21,6 +21,8 @@ There are four components to pagemap:
     * Bit  55    pte is soft-dirty (see
       :ref:`Documentation/admin-guide/mm/soft-dirty.rst <soft_dirty>`)
     * Bit  56    page exclusively mapped (since 4.2)
+    * Bit  57    pte is uffd-wp write-protected (since 5.13) (see
+      :ref:`Documentation/admin-guide/mm/userfaultfd.rst <userfaultfd>`)
     * Bits 57-60 zero
     * Bit  61    page is file-page or shared-anon (since 3.5)
     * Bit  62    page swapped
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index fc9784544b241..ce3b5f33b44c4 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1302,6 +1302,7 @@ struct pagemapread {
 #define PM_PFRAME_MASK		GENMASK_ULL(PM_PFRAME_BITS - 1, 0)
 #define PM_SOFT_DIRTY		BIT_ULL(55)
 #define PM_MMAP_EXCLUSIVE	BIT_ULL(56)
+#define PM_UFFD_WP		BIT_ULL(57)
 #define PM_FILE			BIT_ULL(61)
 #define PM_SWAP			BIT_ULL(62)
 #define PM_PRESENT		BIT_ULL(63)
@@ -1375,10 +1376,14 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
 		page = vm_normal_page(vma, addr, pte);
 		if (pte_soft_dirty(pte))
 			flags |= PM_SOFT_DIRTY;
+		if (pte_uffd_wp(pte))
+			flags |= PM_UFFD_WP;
 	} else if (is_swap_pte(pte)) {
 		swp_entry_t entry;
 		if (pte_swp_soft_dirty(pte))
 			flags |= PM_SOFT_DIRTY;
+		if (pte_swp_uffd_wp(pte))
+			flags |= PM_UFFD_WP;
 		entry = pte_to_swp_entry(pte);
 		if (pm->show_pfn)
 			frame = swp_type(entry) |
@@ -1426,6 +1431,8 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,
 			flags |= PM_PRESENT;
 			if (pmd_soft_dirty(pmd))
 				flags |= PM_SOFT_DIRTY;
+			if (pmd_uffd_wp(pmd))
+				flags |= PM_UFFD_WP;
 			if (pm->show_pfn)
 				frame = pmd_pfn(pmd) +
 					((addr & ~PMD_MASK) >> PAGE_SHIFT);
@@ -1444,6 +1451,8 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,
 			flags |= PM_SWAP;
 			if (pmd_swp_soft_dirty(pmd))
 				flags |= PM_SOFT_DIRTY;
+			if (pmd_swp_uffd_wp(pmd))
+				flags |= PM_UFFD_WP;
 			VM_BUG_ON(!is_pmd_migration_entry(pmd));
 			page = migration_entry_to_page(entry);
 		}
-- 
2.26.2

[PATCH 6/6] userfaultfd/selftests: Add pagemap uffd-wp test

[Peter Xu]

Add one anonymous specific test to start using pagemap.  With pagemap support,
we can directly read the uffd-wp bit from pgtable without triggering any fault,
so it's easier to do sanity checks in unit tests.

Meanwhile this test also leverages the newly introduced MADV_PAGEOUT madvise
function to test swap ptes with uffd-wp bit set, and across fork()s.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 tools/testing/selftests/vm/userfaultfd.c | 154 +++++++++++++++++++++++
 1 file changed, 154 insertions(+)

diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c
index 6339aeaeeff8b..93eae095b61e6 100644
--- a/tools/testing/selftests/vm/userfaultfd.c
+++ b/tools/testing/selftests/vm/userfaultfd.c
@@ -1170,6 +1170,144 @@ static int userfaultfd_minor_test(void)
 	return stats.missing_faults != 0 || stats.minor_faults != nr_pages;
 }
 
+#define BIT_ULL(nr)                   (1ULL << (nr))
+#define PM_SOFT_DIRTY                 BIT_ULL(55)
+#define PM_MMAP_EXCLUSIVE             BIT_ULL(56)
+#define PM_UFFD_WP                    BIT_ULL(57)
+#define PM_FILE                       BIT_ULL(61)
+#define PM_SWAP                       BIT_ULL(62)
+#define PM_PRESENT                    BIT_ULL(63)
+
+static int pagemap_open(void)
+{
+	int fd = open("/proc/self/pagemap", O_RDONLY);
+
+	if (fd < 0)
+		err("open pagemap");
+
+	return fd;
+}
+
+static uint64_t pagemap_read_vaddr(int fd, void *vaddr)
+{
+	uint64_t value;
+	int ret;
+
+	ret = pread(fd, &value, sizeof(uint64_t),
+		    ((uint64_t)vaddr >> 12) * sizeof(uint64_t));
+	if (ret != sizeof(uint64_t))
+		err("pread() on pagemap failed");
+
+	return value;
+}
+
+/* This macro let __LINE__ works in err() */
+#define  pagemap_check_wp(value, wp) do {				\
+		if (!!(value & PM_UFFD_WP) != wp)			\
+			err("pagemap uffd-wp bit error: 0x%"PRIx64, value); \
+	} while (0)
+
+static int pagemap_test_fork(bool present)
+{
+	pid_t child = fork();
+	uint64_t value;
+	int fd, result;
+
+	if (!child) {
+		/* Open the pagemap fd of the child itself */
+		fd = pagemap_open();
+		value = pagemap_read_vaddr(fd, area_dst);
+		/*
+		 * After fork() uffd-wp bit should be gone as long as we're
+		 * without UFFD_FEATURE_EVENT_FORK
+		 */
+		pagemap_check_wp(value, false);
+		/* Succeed */
+		exit(0);
+	}
+	waitpid(child, &result, 0);
+	return result;
+}
+
+static void userfaultfd_pagemap_test(unsigned int test_pgsize)
+{
+	struct uffdio_register uffdio_register;
+	int pagemap_fd;
+	uint64_t value;
+
+	/* Pagemap tests uffd-wp only */
+	if (!test_uffdio_wp)
+		return;
+
+	/* Not enough memory to test this page size */
+	if (test_pgsize > nr_pages * page_size)
+		return;
+
+	printf("testing uffd-wp with pagemap (pgsize=%u): ", test_pgsize);
+	/* Flush so it doesn't flush twice in parent/child later */
+	fflush(stdout);
+
+	uffd_test_ops->release_pages(area_dst);
+
+	if (test_pgsize > page_size) {
+		/* This is a thp test */
+		if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE))
+			err("madvise(MADV_HUGEPAGE) failed");
+	} else if (test_pgsize == page_size) {
+		/* This is normal page test; force no thp */
+		if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE))
+			err("madvise(MADV_NOHUGEPAGE) failed");
+	}
+
+	if (userfaultfd_open(0))
+		err("userfaultfd_open");
+
+	uffdio_register.range.start = (unsigned long) area_dst;
+	uffdio_register.range.len = nr_pages * page_size;
+	uffdio_register.mode = UFFDIO_REGISTER_MODE_WP;
+	if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
+		err("register failed");
+
+	pagemap_fd = pagemap_open();
+
+	/* Touch the page */
+	*area_dst = 1;
+	wp_range(uffd, (uint64_t)area_dst, test_pgsize, true);
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, true);
+	/* Make sure uffd-wp bit dropped when fork */
+	if (pagemap_test_fork(true))
+		err("Detected stall uffd-wp bit in child");
+
+	/* Exclusive required or PAGEOUT won't work */
+	if (!(value & PM_MMAP_EXCLUSIVE))
+		err("multiple mapping detected: 0x%"PRIx64, value);
+
+	if (madvise(area_dst, test_pgsize, MADV_PAGEOUT))
+		err("madvise(MADV_PAGEOUT) failed");
+
+	/* Uffd-wp should persist even swapped out */
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, true);
+	/* Make sure uffd-wp bit dropped when fork */
+	if (pagemap_test_fork(false))
+		err("Detected stall uffd-wp bit in child");
+
+	/* Unprotect; this tests swap pte modifications */
+	wp_range(uffd, (uint64_t)area_dst, page_size, false);
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, false);
+
+	/* Fault in the page from disk */
+	*area_dst = 2;
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, false);
+
+	close(pagemap_fd);
+	close(uffd);
+	printf("done\n");
+}
+
 static int userfaultfd_stress(void)
 {
 	void *area;
@@ -1341,6 +1479,22 @@ static int userfaultfd_stress(void)
 	}
 
 	close(uffd);
+
+	if (test_type == TEST_ANON) {
+		/*
+		 * shmem/hugetlb won't be able to run since they have different
+		 * behavior on fork() (file-backed memory normally drops ptes
+		 * directly when fork), meanwhile the pagemap test will verify
+		 * pgtable entry of fork()ed child.
+		 */
+		userfaultfd_pagemap_test(page_size);
+		/*
+		 * Hard-code for x86_64 for now for 2M THP, as x86_64 is
+		 * currently the only one that supports uffd-wp
+		 */
+		userfaultfd_pagemap_test(page_size * 512);
+	}
+
 	return userfaultfd_zeropage_test() || userfaultfd_sig_test()
 		|| userfaultfd_events_test() || userfaultfd_minor_test();
 }
-- 
2.26.2

Re: [PATCH 1/6] mm/thp: Simplify copying of huge zero page pmd when fork

[David Hildenbrand]

On 29.04.21 00:50, Peter Xu wrote:
> Huge zero page is handled in a special path in copy_huge_pmd(), however it
> should share most codes with a normal thp page.  Trying to share more code with
> it by removing the special path.  The only leftover so far is the huge zero
> page refcounting (mm_get_huge_zero_page()), because that's separately done with
> a global counter.
> 
> This prepares for a future patch to modify the huge pmd to be installed, so
> that we don't need to duplicate it explicitly into huge zero page case too.
> 
> Cc: Kirill A. Shutemov <kirill@shutemov.name>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   mm/huge_memory.c | 9 +++------
>   1 file changed, 3 insertions(+), 6 deletions(-)
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 98456017744d6..22bf2d0fff79b 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -1076,17 +1076,13 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
>   	 * a page table.
>   	 */
>   	if (is_huge_zero_pmd(pmd)) {
> -		struct page *zero_page;
>   		/*
>   		 * get_huge_zero_page() will never allocate a new page here,
>   		 * since we already have a zero page to copy. It just takes a
>   		 * reference.
>   		 */
> -		zero_page = mm_get_huge_zero_page(dst_mm);
> -		set_huge_zero_page(pgtable, dst_mm, vma, addr, dst_pmd,
> -				zero_page);
> -		ret = 0;
> -		goto out_unlock;
> +		mm_get_huge_zero_page(dst_mm);
> +		goto out_zero_page;
>   	}
>   
>   	src_page = pmd_page(pmd);
> @@ -1110,6 +1106,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
>   	get_page(src_page);
>   	page_dup_rmap(src_page, true);
>   	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
> +out_zero_page:
>   	mm_inc_nr_ptes(dst_mm);
>   	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
>   
> 

AFAIKs, the only change is that we're now doing an additional

pmdp_set_wrprotect(src_mm, addr, src_pmd)
pmd = pmd_mkold(pmd_wrprotect(pmd));

But as we are copying the zeropage, it should already be 
write-protected, so no effective change.

LGTM

-- 
Thanks,

David / dhildenb

Re: [PATCH 0/6] mm/uffd: Misc fix for uffd-wp and one more test

[Peter Xu]

On Wed, Apr 28, 2021 at 06:50:24PM -0400, Peter Xu wrote:
> This series is based on (mostly) v5.12-rc8-mmots-2021-04-21-23-08.  It tries to
> fix some corner case bugs for uffd-wp on either thp or fork().  Then it
> introduced a new test with pagemap/pageout.

Sorry I forgot to mention this series is also based on the other selftest
cleanup series applied upon the rc8 tag:

https://lore.kernel.org/patchwork/cover/1412450/

-- 
Peter Xu

Re: [PATCH 2/6] mm/userfaultfd: Fix uffd-wp special cases for fork()

[Andrew Morton]

On Wed, 28 Apr 2021 18:50:26 -0400 Peter Xu <peterx@redhat.com> wrote:

> We tried to do something similar in b569a1760782 ("userfaultfd: wp: drop
> _PAGE_UFFD_WP properly when fork") previously, but it's not doing it all
> right..  A few fixes around the code path:
> 
>   1. We were referencing VM_UFFD_WP vm_flags on the _old_ vma rather than the
>      new vma.  That's overlooked in b569a1760782, so it won't work as expected.
>      Thanks to the recent rework on fork code (7a4830c380f3a8b3), we can easily
>      get the new vma now, so switch the checks to that.
> 
>   2. Dropping the uffd-wp bit in copy_huge_pmd() could be wrong if the huge pmd
>      is a migration huge pmd.  When it happens, instead of using pmd_uffd_wp(),
>      we should use pmd_swp_uffd_wp(). The fix is simply to handle them separately.
> 
>   3. Forget to carry over uffd-wp bit for a write migration huge pmd entry.
>      This also happens in copy_huge_pmd(), where we converted a write huge
>      migration entry into a read one.
> 
>   4. In copy_nonpresent_pte(), drop uffd-wp if necessary for swap ptes.
> 
>   5. In copy_present_page() when COW is enforced when fork(), we also need to
>      pass over the uffd-wp bit if VM_UFFD_WP is armed on the new vma, and when
>      the pte to be copied has uffd-wp bit set.
> 
> Remove the comment in copy_present_pte() about this.  It won't help a huge lot
> to only comment there, but comment everywhere would be an overkill.  Let's
> assume the commit messages would help.
> 

This run afoul of Alistair's "mm: Device exclusive memory access",
https://lkml.kernel.org/r/20210524132725.12697-8-apopple@nvidia.com

`vma' is now undeclared.  I think this?

--- a/mm/memory.c~mm-userfaultfd-fix-uffd-wp-special-cases-for-fork-fix
+++ a/mm/memory.c
@@ -850,8 +850,8 @@ copy_nonpresent_pte(struct mm_struct *ds
 		 * exclusive entries currently only support private writable
 		 * (ie. COW) mappings.
 		 */
-		VM_BUG_ON(!is_cow_mapping(vma->vm_flags));
-		if (try_restore_exclusive_pte(src_mm, src_pte, vma, addr))
+		VM_BUG_ON(!is_cow_mapping(dst_vma->vm_flags));
+		if (try_restore_exclusive_pte(src_mm, src_pte, dst_vma, addr))
 			return -EBUSY;
 		return -ENOENT;
 	}
_

Re: [PATCH 2/6] mm/userfaultfd: Fix uffd-wp special cases for fork()

[Peter Xu]

On Tue, May 25, 2021 at 05:15:58PM -0700, Andrew Morton wrote:
> This run afoul of Alistair's "mm: Device exclusive memory access",
> https://lkml.kernel.org/r/20210524132725.12697-8-apopple@nvidia.com
> 
> `vma' is now undeclared.  I think this?
> 
> --- a/mm/memory.c~mm-userfaultfd-fix-uffd-wp-special-cases-for-fork-fix
> +++ a/mm/memory.c
> @@ -850,8 +850,8 @@ copy_nonpresent_pte(struct mm_struct *ds
>  		 * exclusive entries currently only support private writable
>  		 * (ie. COW) mappings.
>  		 */
> -		VM_BUG_ON(!is_cow_mapping(vma->vm_flags));
> -		if (try_restore_exclusive_pte(src_mm, src_pte, vma, addr))
> +		VM_BUG_ON(!is_cow_mapping(dst_vma->vm_flags));

This one looks good, as both src_vma/dst_vma should have the same flags related
to is_cow.

> +		if (try_restore_exclusive_pte(src_mm, src_pte, dst_vma, addr))

Should this be s/dst_vma/src_vma/ perhaps?  Alistairs please correct me
otherwise, as it tries to restore the pte for src mm not dst (the child).

I haven't yet got time to look at the new series, planning to do it tomorrow
maybe.. but I see that it's already queued in -mm.  Andrew, we do have chance
to go back if necessary, right?

I haven't looked at the rest, but I think try_restore_exclusive_pte() can at
least drop the *mm pointer as it's never used (even if we need, we've got
vma->vm_mm too)..

-- 
Peter Xu

Re: [PATCH 2/6] mm/userfaultfd: Fix uffd-wp special cases for fork()

[Andrew Morton]

On Tue, 25 May 2021 20:36:18 -0400 Peter Xu <peterx@redhat.com> wrote:

> On Tue, May 25, 2021 at 05:15:58PM -0700, Andrew Morton wrote:
> > This run afoul of Alistair's "mm: Device exclusive memory access",
> > https://lkml.kernel.org/r/20210524132725.12697-8-apopple@nvidia.com
> > 
> > `vma' is now undeclared.  I think this?
> > 
> > --- a/mm/memory.c~mm-userfaultfd-fix-uffd-wp-special-cases-for-fork-fix
> > +++ a/mm/memory.c
> > @@ -850,8 +850,8 @@ copy_nonpresent_pte(struct mm_struct *ds
> >  		 * exclusive entries currently only support private writable
> >  		 * (ie. COW) mappings.
> >  		 */
> > -		VM_BUG_ON(!is_cow_mapping(vma->vm_flags));
> > -		if (try_restore_exclusive_pte(src_mm, src_pte, vma, addr))
> > +		VM_BUG_ON(!is_cow_mapping(dst_vma->vm_flags));
> 
> This one looks good, as both src_vma/dst_vma should have the same flags related
> to is_cow.
> 
> > +		if (try_restore_exclusive_pte(src_mm, src_pte, dst_vma, addr))
> 
> Should this be s/dst_vma/src_vma/ perhaps?  Alistairs please correct me
> otherwise, as it tries to restore the pte for src mm not dst (the child).
> 
> I haven't yet got time to look at the new series, planning to do it tomorrow
> maybe.. but I see that it's already queued in -mm.  Andrew, we do have chance
> to go back if necessary, right?

Sure.

> I haven't looked at the rest, but I think try_restore_exclusive_pte() can at
> least drop the *mm pointer as it's never used (even if we need, we've got
> vma->vm_mm too)..

OK, thanks.  I just released a tree into linux-next (hopefully this
blooper won't cause too much damage).  Please send a suitable fixup.