From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org
Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org,
mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org,
tglx@linutronix.de, bp@alien8.de, peterz@infradead.org,
ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com,
jon.grimm@amd.com, boris.ostrovsky@oracle.com,
konrad.wilk@oracle.com, joao.martins@oracle.com,
ankur.a.arora@oracle.com
Subject: [PATCH v3 03/21] clear_page: add generic clear_user_pages()
Date: Mon, 6 Jun 2022 20:20:51 +0000 [thread overview]
Message-ID: <20220606202109.1306034-4-ankur.a.arora@oracle.com> (raw)
In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com>
Add generic clear_user_pages() which operates on contiguous
PAGE_SIZE'd chunks via an arch defined primitive.
The generic version defines:
#define ARCH_MAX_CLEAR_PAGES_ORDER 0
so clear_user_pages() would fallback to clear_user_page().
An arch can expose this by defining __HAVE_ARCH_CLEAR_USER_PAGES.
Also add clear_user_highpages() which, either funnels through
to clear_user_pages() or does the clearing page-at-a-time.
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes:
1. I'm not sure that a new header asm-generic/clear_page.h is ideal.
The logical place for this is asm-generic/page.h itself. However, only
H8300 includes that and so this (and the next few patches) would need
a stub everywhere else.
(Just rechecked and looks like arch/h8300 is no more.)
If adding a new header looks reasonable to the community, I'm happy
to move clear_user_page(), copy_user_page() stubs out to this file.
(Note that patches further on add non-caching clear_user_pages()
as well.)
Or, if asm-generic/page.h is the best place, then add stubs
everywhere else.
2. Shoehorning a multiple page operation in CONFIG_HIGHMEM seems
ugly but, seemed like the best choice of a bad set of options.
Is there a better way of doing this?
arch/alpha/include/asm/page.h | 1 +
arch/arc/include/asm/page.h | 1 +
arch/arm/include/asm/page.h | 1 +
arch/arm64/include/asm/page.h | 1 +
arch/csky/include/asm/page.h | 1 +
arch/hexagon/include/asm/page.h | 1 +
arch/ia64/include/asm/page.h | 1 +
arch/m68k/include/asm/page.h | 1 +
arch/microblaze/include/asm/page.h | 1 +
arch/mips/include/asm/page.h | 1 +
arch/nios2/include/asm/page.h | 2 ++
arch/openrisc/include/asm/page.h | 1 +
arch/parisc/include/asm/page.h | 1 +
arch/powerpc/include/asm/page.h | 1 +
arch/riscv/include/asm/page.h | 1 +
arch/s390/include/asm/page.h | 1 +
arch/sh/include/asm/page.h | 1 +
arch/sparc/include/asm/page_32.h | 1 +
arch/sparc/include/asm/page_64.h | 1 +
arch/um/include/asm/page.h | 1 +
arch/x86/include/asm/page.h | 1 +
arch/xtensa/include/asm/page.h | 1 +
include/asm-generic/clear_page.h | 44 ++++++++++++++++++++++++++++++
include/asm-generic/page.h | 1 +
include/linux/highmem.h | 23 ++++++++++++++++
25 files changed, 91 insertions(+)
create mode 100644 include/asm-generic/clear_page.h
diff --git a/arch/alpha/include/asm/page.h b/arch/alpha/include/asm/page.h
index 8f3f5eecba28..2d3b099e165c 100644
--- a/arch/alpha/include/asm/page.h
+++ b/arch/alpha/include/asm/page.h
@@ -93,5 +93,6 @@ typedef struct page *pgtable_t;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _ALPHA_PAGE_H */
diff --git a/arch/arc/include/asm/page.h b/arch/arc/include/asm/page.h
index 9a62e1d87967..abdbef6897bf 100644
--- a/arch/arc/include/asm/page.h
+++ b/arch/arc/include/asm/page.h
@@ -133,6 +133,7 @@ extern int pfn_valid(unsigned long pfn);
#include <asm-generic/memory_model.h> /* page_to_pfn, pfn_to_page */
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* !__ASSEMBLY__ */
diff --git a/arch/arm/include/asm/page.h b/arch/arm/include/asm/page.h
index 5fcc8a600e36..ba244baca1fa 100644
--- a/arch/arm/include/asm/page.h
+++ b/arch/arm/include/asm/page.h
@@ -167,5 +167,6 @@ extern int pfn_valid(unsigned long);
#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index 993a27ea6f54..8407ac2b5d68 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -50,5 +50,6 @@ int pfn_is_map_memory(unsigned long pfn);
#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED)
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif
diff --git a/arch/csky/include/asm/page.h b/arch/csky/include/asm/page.h
index ed7451478b1b..47cc27d4ede1 100644
--- a/arch/csky/include/asm/page.h
+++ b/arch/csky/include/asm/page.h
@@ -89,6 +89,7 @@ extern unsigned long va_pa_offset;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* !__ASSEMBLY__ */
#endif /* __ASM_CSKY_PAGE_H */
diff --git a/arch/hexagon/include/asm/page.h b/arch/hexagon/include/asm/page.h
index 7cbf719c578e..e7a8edd6903a 100644
--- a/arch/hexagon/include/asm/page.h
+++ b/arch/hexagon/include/asm/page.h
@@ -142,6 +142,7 @@ static inline void clear_page(void *page)
#include <asm-generic/memory_model.h>
/* XXX Todo: implement assembly-optimized version of getorder. */
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* ifdef __ASSEMBLY__ */
#endif /* ifdef __KERNEL__ */
diff --git a/arch/ia64/include/asm/page.h b/arch/ia64/include/asm/page.h
index 1b990466d540..1feae333e250 100644
--- a/arch/ia64/include/asm/page.h
+++ b/arch/ia64/include/asm/page.h
@@ -96,6 +96,7 @@ do { \
#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
#include <asm-generic/memory_model.h>
+#include <asm-generic/clear_page.h>
#ifdef CONFIG_FLATMEM
# define pfn_valid(pfn) ((pfn) < max_mapnr)
diff --git a/arch/m68k/include/asm/page.h b/arch/m68k/include/asm/page.h
index 2f1c54e4725d..1aeaae820670 100644
--- a/arch/m68k/include/asm/page.h
+++ b/arch/m68k/include/asm/page.h
@@ -68,5 +68,6 @@ extern unsigned long _ramend;
#endif
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _M68K_PAGE_H */
diff --git a/arch/microblaze/include/asm/page.h b/arch/microblaze/include/asm/page.h
index 4b8b2fa78fc5..baa03569477a 100644
--- a/arch/microblaze/include/asm/page.h
+++ b/arch/microblaze/include/asm/page.h
@@ -137,5 +137,6 @@ extern int page_is_ram(unsigned long pfn);
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _ASM_MICROBLAZE_PAGE_H */
diff --git a/arch/mips/include/asm/page.h b/arch/mips/include/asm/page.h
index 96bc798c1ec1..3dde03bf99f3 100644
--- a/arch/mips/include/asm/page.h
+++ b/arch/mips/include/asm/page.h
@@ -269,5 +269,6 @@ static inline unsigned long kaslr_offset(void)
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _ASM_PAGE_H */
diff --git a/arch/nios2/include/asm/page.h b/arch/nios2/include/asm/page.h
index 6a989819a7c1..9763048bd3ed 100644
--- a/arch/nios2/include/asm/page.h
+++ b/arch/nios2/include/asm/page.h
@@ -104,6 +104,8 @@ static inline bool pfn_valid(unsigned long pfn)
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
+
#endif /* !__ASSEMBLY__ */
#endif /* _ASM_NIOS2_PAGE_H */
diff --git a/arch/openrisc/include/asm/page.h b/arch/openrisc/include/asm/page.h
index aab6e64d6db4..879419c00cd4 100644
--- a/arch/openrisc/include/asm/page.h
+++ b/arch/openrisc/include/asm/page.h
@@ -88,5 +88,6 @@ typedef struct page *pgtable_t;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* __ASM_OPENRISC_PAGE_H */
diff --git a/arch/parisc/include/asm/page.h b/arch/parisc/include/asm/page.h
index 6faaaa3ebe9b..961f88d6ff63 100644
--- a/arch/parisc/include/asm/page.h
+++ b/arch/parisc/include/asm/page.h
@@ -184,6 +184,7 @@ extern int npmem_ranges;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#include <asm/pdc.h>
#define PAGE0 ((struct zeropage *)absolute_pointer(__PAGE_OFFSET))
diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h
index e5f75c70eda8..4742b1f99a3e 100644
--- a/arch/powerpc/include/asm/page.h
+++ b/arch/powerpc/include/asm/page.h
@@ -335,6 +335,7 @@ static inline unsigned long kaslr_offset(void)
}
#include <asm-generic/memory_model.h>
+#include <asm-generic/clear_page.h>
#endif /* __ASSEMBLY__ */
#endif /* _ASM_POWERPC_PAGE_H */
diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h
index 1526e410e802..ce9005ffccb0 100644
--- a/arch/riscv/include/asm/page.h
+++ b/arch/riscv/include/asm/page.h
@@ -188,5 +188,6 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x);
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _ASM_RISCV_PAGE_H */
diff --git a/arch/s390/include/asm/page.h b/arch/s390/include/asm/page.h
index 61dea67bb9c7..7a598f86ae39 100644
--- a/arch/s390/include/asm/page.h
+++ b/arch/s390/include/asm/page.h
@@ -207,5 +207,6 @@ int arch_make_page_accessible(struct page *page);
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _S390_PAGE_H */
diff --git a/arch/sh/include/asm/page.h b/arch/sh/include/asm/page.h
index eca5daa43b93..5e49bb342c2c 100644
--- a/arch/sh/include/asm/page.h
+++ b/arch/sh/include/asm/page.h
@@ -176,6 +176,7 @@ typedef struct page *pgtable_t;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
/*
* Some drivers need to perform DMA into kmalloc'ed buffers
diff --git a/arch/sparc/include/asm/page_32.h b/arch/sparc/include/asm/page_32.h
index fff8861df107..2f061d9a5a30 100644
--- a/arch/sparc/include/asm/page_32.h
+++ b/arch/sparc/include/asm/page_32.h
@@ -135,5 +135,6 @@ extern unsigned long pfn_base;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _SPARC_PAGE_H */
diff --git a/arch/sparc/include/asm/page_64.h b/arch/sparc/include/asm/page_64.h
index 254dffd85fb1..2026bf92e3e7 100644
--- a/arch/sparc/include/asm/page_64.h
+++ b/arch/sparc/include/asm/page_64.h
@@ -159,5 +159,6 @@ extern unsigned long PAGE_OFFSET;
#endif /* !(__ASSEMBLY__) */
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* _SPARC64_PAGE_H */
diff --git a/arch/um/include/asm/page.h b/arch/um/include/asm/page.h
index 95af12e82a32..79768ad6069c 100644
--- a/arch/um/include/asm/page.h
+++ b/arch/um/include/asm/page.h
@@ -113,6 +113,7 @@ extern unsigned long uml_physmem;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* __ASSEMBLY__ */
diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h
index 9cc82f305f4b..5a246a2a66aa 100644
--- a/arch/x86/include/asm/page.h
+++ b/arch/x86/include/asm/page.h
@@ -85,6 +85,7 @@ static __always_inline u64 __is_canonical_address(u64 vaddr, u8 vaddr_bits)
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA
diff --git a/arch/xtensa/include/asm/page.h b/arch/xtensa/include/asm/page.h
index 493eb7083b1a..2812f2bea844 100644
--- a/arch/xtensa/include/asm/page.h
+++ b/arch/xtensa/include/asm/page.h
@@ -200,4 +200,5 @@ static inline unsigned long ___pa(unsigned long va)
#endif /* __ASSEMBLY__ */
#include <asm-generic/memory_model.h>
+#include <asm-generic/clear_page.h>
#endif /* _XTENSA_PAGE_H */
diff --git a/include/asm-generic/clear_page.h b/include/asm-generic/clear_page.h
new file mode 100644
index 000000000000..f827d661519c
--- /dev/null
+++ b/include/asm-generic/clear_page.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_GENERIC_CLEAR_PAGE_H
+#define __ASM_GENERIC_CLEAR_PAGE_H
+
+/*
+ * clear_user_pages() operates on contiguous pages and does the clearing
+ * operation in a single arch defined primitive.
+ *
+ * To do this, arch code defines clear_user_pages() and the max granularity
+ * it can handle via ARCH_MAX_CLEAR_PAGES_ORDER.
+ *
+ * Note that given the need for contiguity, __HAVE_ARCH_CLEAR_USER_PAGES
+ * and CONFIG_HIGHMEM are mutually exclusive.
+ */
+
+#if defined(CONFIG_HIGHMEM) && defined(__HAVE_ARCH_CLEAR_USER_PAGES)
+#error CONFIG_HIGHMEM is incompatible with __HAVE_ARCH_CLEAR_USER_PAGES
+#endif
+
+#ifndef __HAVE_ARCH_CLEAR_USER_PAGES
+
+/*
+ * For architectures that do not expose __HAVE_ARCH_CLEAR_USER_PAGES, set
+ * the granularity to be identical to clear_user_page().
+ */
+#define ARCH_MAX_CLEAR_PAGES_ORDER 0
+
+#ifndef __ASSEMBLY__
+
+/*
+ * With ARCH_MAX_CLEAR_PAGES_ORDER == 0, all callers should be specifying
+ * npages == 1 and so we just fallback to clear_user_page().
+ */
+static inline void clear_user_pages(void *page, unsigned long vaddr,
+ struct page *start_page, unsigned int npages)
+{
+ clear_user_page(page, vaddr, start_page);
+}
+#endif /* __ASSEMBLY__ */
+#endif /* __HAVE_ARCH_CLEAR_USER_PAGES */
+
+#define ARCH_MAX_CLEAR_PAGES (1 << ARCH_MAX_CLEAR_PAGES_ORDER)
+
+#endif /* __ASM_GENERIC_CLEAR_PAGE_H */
diff --git a/include/asm-generic/page.h b/include/asm-generic/page.h
index 6fc47561814c..060094e7f964 100644
--- a/include/asm-generic/page.h
+++ b/include/asm-generic/page.h
@@ -93,5 +93,6 @@ extern unsigned long memory_end;
#include <asm-generic/memory_model.h>
#include <asm-generic/getorder.h>
+#include <asm-generic/clear_page.h>
#endif /* __ASM_GENERIC_PAGE_H */
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 3af34de54330..08781d7693e7 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -208,6 +208,29 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr)
}
#endif
+#ifdef __HAVE_ARCH_CLEAR_USER_PAGES
+static inline void clear_user_highpages(struct page *page, unsigned long vaddr,
+ unsigned int npages)
+{
+ void *addr = page_address(page);
+
+ clear_user_pages(addr, vaddr, page, npages);
+}
+#else
+static inline void clear_user_highpages(struct page *page, unsigned long vaddr,
+ unsigned int npages)
+{
+ void *addr;
+ unsigned int i;
+
+ for (i = 0; i < npages; i++, page++, vaddr += PAGE_SIZE) {
+ addr = kmap_local_page(page);
+ clear_user_page(addr, vaddr, page);
+ kunmap_local(addr);
+ }
+}
+#endif /* __HAVE_ARCH_CLEAR_USER_PAGES */
+
#ifndef __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
/**
* alloc_zeroed_user_highpage_movable - Allocate a zeroed HIGHMEM page for a VMA that the caller knows can move
--
2.31.1
next prev parent reply other threads:[~2022-06-06 20:22 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-06 20:20 [PATCH v3 00/21] huge page clearing optimizations Ankur Arora
2022-06-06 20:20 ` [PATCH v3 01/21] mm, huge-page: reorder arguments to process_huge_page() Ankur Arora
2022-06-06 20:20 ` [PATCH v3 02/21] mm, huge-page: refactor process_subpage() Ankur Arora
2022-06-06 20:20 ` Ankur Arora [this message]
2022-06-06 20:20 ` [PATCH v3 04/21] mm, clear_huge_page: support clear_user_pages() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 05/21] mm/huge_page: generalize process_huge_page() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 06/21] x86/clear_page: add clear_pages() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 07/21] x86/asm: add memset_movnti() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 08/21] perf bench: " Ankur Arora
2022-06-06 20:37 ` [PATCH v3 09/21] x86/asm: add clear_pages_movnt() Ankur Arora
2022-06-10 22:11 ` Noah Goldstein
2022-06-10 22:15 ` Noah Goldstein
2022-06-12 11:18 ` Ankur Arora
2022-06-06 20:37 ` [PATCH v3 10/21] x86/asm: add clear_pages_clzero() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 11/21] x86/cpuid: add X86_FEATURE_MOVNT_SLOW Ankur Arora
2022-06-06 20:37 ` [PATCH v3 12/21] sparse: add address_space __incoherent Ankur Arora
2022-06-06 20:37 ` [PATCH v3 13/21] clear_page: add generic clear_user_pages_incoherent() Ankur Arora
2022-06-08 0:01 ` Luc Van Oostenryck
2022-06-12 11:19 ` Ankur Arora
2022-06-06 20:37 ` [PATCH v3 14/21] x86/clear_page: add clear_pages_incoherent() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 15/21] mm/clear_page: add clear_page_non_caching_threshold() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 16/21] x86/clear_page: add arch_clear_page_non_caching_threshold() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 17/21] clear_huge_page: use non-cached clearing Ankur Arora
2022-06-06 20:37 ` [PATCH v3 18/21] gup: add FOLL_HINT_BULK, FAULT_FLAG_NON_CACHING Ankur Arora
2022-06-06 20:37 ` [PATCH v3 19/21] gup: hint non-caching if clearing large regions Ankur Arora
2022-06-06 20:37 ` [PATCH v3 20/21] vfio_iommu_type1: specify FOLL_HINT_BULK to pin_user_pages() Ankur Arora
2022-06-06 20:37 ` [PATCH v3 21/21] x86/cpu/intel: set X86_FEATURE_MOVNT_SLOW for Skylake Ankur Arora
2022-06-06 21:53 ` [PATCH v3 00/21] huge page clearing optimizations Linus Torvalds
2022-06-07 15:08 ` Ankur Arora
2022-06-07 17:56 ` Linus Torvalds
2022-06-08 19:24 ` Ankur Arora
2022-06-08 19:39 ` Linus Torvalds
2022-06-08 20:21 ` Ankur Arora
2022-06-08 19:49 ` Matthew Wilcox
2022-06-08 19:51 ` Matthew Wilcox
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220606202109.1306034-4-ankur.a.arora@oracle.com \
--to=ankur.a.arora@oracle.com \
--cc=ak@linux.intel.com \
--cc=akpm@linux-foundation.org \
--cc=arnd@arndb.de \
--cc=boris.ostrovsky@oracle.com \
--cc=bp@alien8.de \
--cc=jgg@nvidia.com \
--cc=joao.martins@oracle.com \
--cc=jon.grimm@amd.com \
--cc=konrad.wilk@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=luto@kernel.org \
--cc=mike.kravetz@oracle.com \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).