* [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-24 23:45 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-24 23:45 UTC (permalink / raw)
To: linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, peterx, Matt Turner,
Albert Ou, Michal Simek, Russell King, Ivan Kokshaysky,
linux-riscv, Alexander Gordeev, Dave Hansen, Jonas Bonn,
Will Deacon, James E . J . Bottomley, H . Peter Anvin,
Andrea Arcangeli, openrisc, linux-s390, Ingo Molnar, linux-m68k,
Palmer Dabbelt, Heiko Carstens, Chris Zankel, Peter Zijlstra,
Alistair Popple, linux-csky, linux-hexagon, Vlastimil Babka,
Thomas Gleixner, sparclinux, Christian Borntraeger,
Stafford Horne, Michael Ellerman, x86, Thomas Bogendoerfer,
Paul Mackerras, linux-arm-kernel, Sven Schnelle,
Benjamin Herrenschmidt, linux-xtensa, Nicholas Piggin, linux-sh,
Vasily Gorbik, Borislav Petkov, linux-mips, Max Filippov,
Helge Deller, Vineet Gupta, Al Viro, Paul Walmsley,
Johannes Weiner, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
I observed that for each of the shared file-backed page faults, we're very
likely to retry one more time for the 1st write fault upon no page. It's
because we'll need to release the mmap lock for dirty rate limit purpose
with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return
to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need
to be retried at all since the pgtable was well installed before the
throttling, so the next continuous fault (including taking mmap read lock,
walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add
more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at
"pgfault" value in /proc/vmstat, then we should expect 2 counts for each
shmem write simply because we retried, and vm event "pgfault" will capture
that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
show that we've completed the whole fault and released the lock. It's also
a hint that we should very possibly not need another fault immediately on
this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple
program sequentially dirtying 400MB shmem file being mmap()ed and these are
the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap
code before returning from pgfault), the rest changes in the page fault
handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new
fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
them as-is.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
arch/alpha/mm/fault.c | 4 ++++
arch/arc/mm/fault.c | 4 ++++
arch/arm/mm/fault.c | 4 ++++
arch/arm64/mm/fault.c | 4 ++++
arch/csky/mm/fault.c | 4 ++++
arch/hexagon/mm/vm_fault.c | 4 ++++
arch/ia64/mm/fault.c | 4 ++++
arch/m68k/mm/fault.c | 4 ++++
arch/microblaze/mm/fault.c | 4 ++++
arch/mips/mm/fault.c | 4 ++++
arch/nios2/mm/fault.c | 4 ++++
arch/openrisc/mm/fault.c | 4 ++++
arch/parisc/mm/fault.c | 4 ++++
arch/powerpc/mm/copro_fault.c | 5 +++++
arch/powerpc/mm/fault.c | 5 +++++
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 12 +++++++++++-
arch/sh/mm/fault.c | 4 ++++
arch/sparc/mm/fault_32.c | 4 ++++
arch/sparc/mm/fault_64.c | 5 +++++
arch/um/kernel/trap.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
arch/xtensa/mm/fault.c | 4 ++++
include/linux/mm_types.h | 2 ++
mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
mm/memory.c | 2 +-
26 files changed, 138 insertions(+), 3 deletions(-)
diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
index ec20c1004abf..ef427a6bdd1a 100644
--- a/arch/alpha/mm/fault.c
+++ b/arch/alpha/mm/fault.c
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
index dad27e4d69ff..5ca59a482632 100644
--- a/arch/arc/mm/fault.c
+++ b/arch/arc/mm/fault.c
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* Fault retry nuances, mmap_lock already relinquished by core mm
*/
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a062e07516dd..46cccd6bf705 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (!(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 77341b160aca..e401d416bbd6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (fault & VM_FAULT_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7215a46b6b8e..e15f736cca4b 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
index 4fac4b9eb316..f73c7cbfe326 100644
--- a/arch/hexagon/mm/vm_fault.c
+++ b/arch/hexagon/mm/vm_fault.c
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/* The most common case -- we are done. */
if (likely(!(fault & VM_FAULT_ERROR))) {
if (fault & VM_FAULT_RETRY) {
diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
index 07379d1a227f..ef78c2d66cdd 100644
--- a/arch/ia64/mm/fault.c
+++ b/arch/ia64/mm/fault.c
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We ran out of memory, or some other thing happened
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index 71aa9f6315dc..4d2837eb3e2a 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return 0;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
index a9626e6a68af..5c40c3ebe52f 100644
--- a/arch/microblaze/mm/fault.c
+++ b/arch/microblaze/mm/fault.c
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 44f98100e84e..6f72bac39bf2 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index a32f14cd72f2..edaca0a6c1c1 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
index 80bb66ad42f6..c18f7abd64df 100644
--- a/arch/openrisc/mm/fault.c
+++ b/arch/openrisc/mm/fault.c
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
index f114e102aaf2..fe57175a7792 100644
--- a/arch/parisc/mm/fault.c
+++ b/arch/parisc/mm/fault.c
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We hit a shared mapping outside of the file, or some
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index c1cb21a00884..7c507fb48182 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
ret = 0;
*flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (*flt & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(*flt & VM_FAULT_ERROR)) {
if (*flt & VM_FAULT_OOM) {
ret = -ENOMEM;
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index d53fed4eccbd..014005428687 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return user_mode(regs) ? 0 : SIGBUS;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto out;
+
/*
* Handle the retry right now, the mmap_lock has been released in that
* case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (unlikely(fault & VM_FAULT_ERROR))
return mm_fault_error(regs, address, fault);
+out:
/*
* Major/minor page fault accounting.
*/
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 4e9efbe46d5f..d6a87f4137ca 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_RETRY)) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..9503a7cfaf03 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
unsigned long address;
unsigned int flags;
vm_fault_t fault;
+ bool need_unlock = true;
bool is_write;
tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ need_unlock = false;
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
}
fault = 0;
out_up:
- mmap_read_unlock(mm);
+ if (need_unlock)
+ mmap_read_unlock(mm);
out:
return fault;
}
diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
index e175667b1363..acd2f5e50bfc 100644
--- a/arch/sh/mm/fault.c
+++ b/arch/sh/mm/fault.c
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
if (mm_fault_error(regs, error_code, address, fault))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
index ad569d9bd124..91259f291c54 100644
--- a/arch/sparc/mm/fault_32.c
+++ b/arch/sparc/mm/fault_32.c
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index 253e07043298..4acc12eafbf5 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
goto exit_exception;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto lock_released;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
}
mmap_read_unlock(mm);
+lock_released:
mm_rss = get_mm_rss(mm);
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
index d1d5d0be0308..d3ce21c4ca32 100644
--- a/arch/um/kernel/trap.c
+++ b/arch/um/kernel/trap.c
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
goto out_nosemaphore;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM) {
goto out_of_memory;
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index d0074c6ed31a..12ed70b432d6 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* If we need to retry the mmap_lock has already been released,
* and if there is a fatal signal pending there is no guarantee
diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
index 06d0973a0d74..5f64305ba8d7 100644
--- a/arch/xtensa/mm/fault.c
+++ b/arch/xtensa/mm/fault.c
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c09b7f0555b8..decc275db3c9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
* fsync() to complete (for synchronous page faults
* in DAX)
+ * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
* @VM_FAULT_HINDEX_MASK: mask HINDEX value
*
*/
@@ -746,6 +747,7 @@ enum vm_fault_reason {
VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
+ VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
};
diff --git a/mm/gup.c b/mm/gup.c
index 2e07cff3b31b..3347b083d70b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
}
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+ * mmap lock in the page fault handler. Sanity check this.
+ */
+ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+ if (locked)
+ *locked = 0;
+ /*
+ * We should do the same as VM_FAULT_RETRY, but let's not
+ * return -EBUSY since that's not reflecting the reality on
+ * what has happened - we've just fully completed a page
+ * fault, with the mmap lock released. Use -EAGAIN to show
+ * that we want to take the mmap lock _again_.
+ */
+ return -EAGAIN;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
case 0:
goto retry;
case -EBUSY:
+ case -EAGAIN:
ret = 0;
fallthrough;
case -EFAULT:
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
return -EINTR;
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * NOTE: it's a pity that we need to retake the lock here
+ * to pair with the unlock() in the callers. Ideally we
+ * could tell the callers so they do not need to unlock.
+ */
+ mmap_read_lock(mm);
+ *unlocked = true;
+ return 0;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
/* VM_FAULT_RETRY couldn't trigger, bypass */
return ret;
- /* VM_FAULT_RETRY cannot return errors */
+ /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
if (!*locked) {
BUG_ON(ret < 0);
BUG_ON(ret >= nr_pages);
diff --git a/mm/memory.c b/mm/memory.c
index 54d106e0c999..a8be2d7a8718 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
balance_dirty_pages_ratelimited(mapping);
if (fpin) {
fput(fpin);
- return VM_FAULT_RETRY;
+ return VM_FAULT_COMPLETED;
}
}
--
2.32.0
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-24 23:45 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-24 23:45 UTC (permalink / raw)
To: linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, peterx, Matt Turner,
Albert Ou, Michal Simek, Russell King, Ivan Kokshaysky,
linux-riscv, Alexander Gordeev, Dave Hansen, Jonas Bonn,
Will Deacon, James E . J . Bottomley, H . Peter Anvin,
Andrea Arcangeli, openrisc, linux-s390, Ingo Molnar, linux-m68k,
Palmer Dabbelt, Heiko Carstens, Chris Zankel, Peter Zijlstra,
Alistair Popple, linux-csky, linux-hexagon, Vlastimil Babka,
Thomas Gleixner, sparclinux, Christian Borntraeger,
Stafford Horne, Michael Ellerman, x86, Thomas Bogendoerfer,
Paul Mackerras, linux-arm-kernel, Sven Schnelle,
Benjamin Herrenschmidt, linux-xtensa, Nicholas Piggin, linux-sh,
Vasily Gorbik, Borislav Petkov, linux-mips, Max Filippov,
Helge Deller, Vineet Gupta, Al Viro, Paul Walmsley,
Johannes Weiner, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
I observed that for each of the shared file-backed page faults, we're very
likely to retry one more time for the 1st write fault upon no page. It's
because we'll need to release the mmap lock for dirty rate limit purpose
with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return
to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need
to be retried at all since the pgtable was well installed before the
throttling, so the next continuous fault (including taking mmap read lock,
walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add
more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at
"pgfault" value in /proc/vmstat, then we should expect 2 counts for each
shmem write simply because we retried, and vm event "pgfault" will capture
that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
show that we've completed the whole fault and released the lock. It's also
a hint that we should very possibly not need another fault immediately on
this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple
program sequentially dirtying 400MB shmem file being mmap()ed and these are
the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap
code before returning from pgfault), the rest changes in the page fault
handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new
fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
them as-is.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
arch/alpha/mm/fault.c | 4 ++++
arch/arc/mm/fault.c | 4 ++++
arch/arm/mm/fault.c | 4 ++++
arch/arm64/mm/fault.c | 4 ++++
arch/csky/mm/fault.c | 4 ++++
arch/hexagon/mm/vm_fault.c | 4 ++++
arch/ia64/mm/fault.c | 4 ++++
arch/m68k/mm/fault.c | 4 ++++
arch/microblaze/mm/fault.c | 4 ++++
arch/mips/mm/fault.c | 4 ++++
arch/nios2/mm/fault.c | 4 ++++
arch/openrisc/mm/fault.c | 4 ++++
arch/parisc/mm/fault.c | 4 ++++
arch/powerpc/mm/copro_fault.c | 5 +++++
arch/powerpc/mm/fault.c | 5 +++++
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 12 +++++++++++-
arch/sh/mm/fault.c | 4 ++++
arch/sparc/mm/fault_32.c | 4 ++++
arch/sparc/mm/fault_64.c | 5 +++++
arch/um/kernel/trap.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
arch/xtensa/mm/fault.c | 4 ++++
include/linux/mm_types.h | 2 ++
mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
mm/memory.c | 2 +-
26 files changed, 138 insertions(+), 3 deletions(-)
diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
index ec20c1004abf..ef427a6bdd1a 100644
--- a/arch/alpha/mm/fault.c
+++ b/arch/alpha/mm/fault.c
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
index dad27e4d69ff..5ca59a482632 100644
--- a/arch/arc/mm/fault.c
+++ b/arch/arc/mm/fault.c
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* Fault retry nuances, mmap_lock already relinquished by core mm
*/
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a062e07516dd..46cccd6bf705 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (!(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 77341b160aca..e401d416bbd6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (fault & VM_FAULT_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7215a46b6b8e..e15f736cca4b 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
index 4fac4b9eb316..f73c7cbfe326 100644
--- a/arch/hexagon/mm/vm_fault.c
+++ b/arch/hexagon/mm/vm_fault.c
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/* The most common case -- we are done. */
if (likely(!(fault & VM_FAULT_ERROR))) {
if (fault & VM_FAULT_RETRY) {
diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
index 07379d1a227f..ef78c2d66cdd 100644
--- a/arch/ia64/mm/fault.c
+++ b/arch/ia64/mm/fault.c
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We ran out of memory, or some other thing happened
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index 71aa9f6315dc..4d2837eb3e2a 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return 0;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
index a9626e6a68af..5c40c3ebe52f 100644
--- a/arch/microblaze/mm/fault.c
+++ b/arch/microblaze/mm/fault.c
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 44f98100e84e..6f72bac39bf2 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index a32f14cd72f2..edaca0a6c1c1 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
index 80bb66ad42f6..c18f7abd64df 100644
--- a/arch/openrisc/mm/fault.c
+++ b/arch/openrisc/mm/fault.c
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
index f114e102aaf2..fe57175a7792 100644
--- a/arch/parisc/mm/fault.c
+++ b/arch/parisc/mm/fault.c
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We hit a shared mapping outside of the file, or some
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index c1cb21a00884..7c507fb48182 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
ret = 0;
*flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (*flt & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(*flt & VM_FAULT_ERROR)) {
if (*flt & VM_FAULT_OOM) {
ret = -ENOMEM;
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index d53fed4eccbd..014005428687 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return user_mode(regs) ? 0 : SIGBUS;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto out;
+
/*
* Handle the retry right now, the mmap_lock has been released in that
* case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (unlikely(fault & VM_FAULT_ERROR))
return mm_fault_error(regs, address, fault);
+out:
/*
* Major/minor page fault accounting.
*/
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 4e9efbe46d5f..d6a87f4137ca 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_RETRY)) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..9503a7cfaf03 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
unsigned long address;
unsigned int flags;
vm_fault_t fault;
+ bool need_unlock = true;
bool is_write;
tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ need_unlock = false;
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
}
fault = 0;
out_up:
- mmap_read_unlock(mm);
+ if (need_unlock)
+ mmap_read_unlock(mm);
out:
return fault;
}
diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
index e175667b1363..acd2f5e50bfc 100644
--- a/arch/sh/mm/fault.c
+++ b/arch/sh/mm/fault.c
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
if (mm_fault_error(regs, error_code, address, fault))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
index ad569d9bd124..91259f291c54 100644
--- a/arch/sparc/mm/fault_32.c
+++ b/arch/sparc/mm/fault_32.c
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index 253e07043298..4acc12eafbf5 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
goto exit_exception;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto lock_released;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
}
mmap_read_unlock(mm);
+lock_released:
mm_rss = get_mm_rss(mm);
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
index d1d5d0be0308..d3ce21c4ca32 100644
--- a/arch/um/kernel/trap.c
+++ b/arch/um/kernel/trap.c
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
goto out_nosemaphore;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM) {
goto out_of_memory;
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index d0074c6ed31a..12ed70b432d6 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* If we need to retry the mmap_lock has already been released,
* and if there is a fatal signal pending there is no guarantee
diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
index 06d0973a0d74..5f64305ba8d7 100644
--- a/arch/xtensa/mm/fault.c
+++ b/arch/xtensa/mm/fault.c
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c09b7f0555b8..decc275db3c9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
* fsync() to complete (for synchronous page faults
* in DAX)
+ * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
* @VM_FAULT_HINDEX_MASK: mask HINDEX value
*
*/
@@ -746,6 +747,7 @@ enum vm_fault_reason {
VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
+ VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
};
diff --git a/mm/gup.c b/mm/gup.c
index 2e07cff3b31b..3347b083d70b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
}
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+ * mmap lock in the page fault handler. Sanity check this.
+ */
+ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+ if (locked)
+ *locked = 0;
+ /*
+ * We should do the same as VM_FAULT_RETRY, but let's not
+ * return -EBUSY since that's not reflecting the reality on
+ * what has happened - we've just fully completed a page
+ * fault, with the mmap lock released. Use -EAGAIN to show
+ * that we want to take the mmap lock _again_.
+ */
+ return -EAGAIN;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
case 0:
goto retry;
case -EBUSY:
+ case -EAGAIN:
ret = 0;
fallthrough;
case -EFAULT:
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
return -EINTR;
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * NOTE: it's a pity that we need to retake the lock here
+ * to pair with the unlock() in the callers. Ideally we
+ * could tell the callers so they do not need to unlock.
+ */
+ mmap_read_lock(mm);
+ *unlocked = true;
+ return 0;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
/* VM_FAULT_RETRY couldn't trigger, bypass */
return ret;
- /* VM_FAULT_RETRY cannot return errors */
+ /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
if (!*locked) {
BUG_ON(ret < 0);
BUG_ON(ret >= nr_pages);
diff --git a/mm/memory.c b/mm/memory.c
index 54d106e0c999..a8be2d7a8718 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
balance_dirty_pages_ratelimited(mapping);
if (fpin) {
fput(fpin);
- return VM_FAULT_RETRY;
+ return VM_FAULT_COMPLETED;
}
}
--
2.32.0
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-24 23:45 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-24 23:45 UTC (permalink / raw)
To: linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, peterx, Matt Turner,
Albert Ou, Michal Simek, Russell King, Ivan Kokshaysky,
linux-riscv, Alexander Gordeev, Dave Hansen, Jonas Bonn,
Will Deacon, James E . J . Bottomley, H . Peter Anvin,
Andrea Arcangeli, openrisc, linux-s390, Ingo Molnar, linux-m68k,
Palmer Dabbelt, Heiko Carstens, Chris Zankel, Peter Zijlstra,
Alistair Popple, linux-csky, linux-hexagon, Vlastimil Babka,
Thomas Gleixner, sparclinux, Christian Borntraeger,
Stafford Horne, Michael Ellerman, x86, Thomas Bogendoerfer,
Paul Mackerras, linux-arm-kernel, Sven Schnelle,
Benjamin Herrenschmidt, linux-xtensa, Nicholas Piggin, linux-sh,
Vasily Gorbik, Borislav Petkov, linux-mips, Max Filippov,
Helge Deller, Vineet Gupta, Al Viro, Paul Walmsley,
Johannes Weiner, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
I observed that for each of the shared file-backed page faults, we're very
likely to retry one more time for the 1st write fault upon no page. It's
because we'll need to release the mmap lock for dirty rate limit purpose
with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return
to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need
to be retried at all since the pgtable was well installed before the
throttling, so the next continuous fault (including taking mmap read lock,
walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add
more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at
"pgfault" value in /proc/vmstat, then we should expect 2 counts for each
shmem write simply because we retried, and vm event "pgfault" will capture
that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
show that we've completed the whole fault and released the lock. It's also
a hint that we should very possibly not need another fault immediately on
this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple
program sequentially dirtying 400MB shmem file being mmap()ed and these are
the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap
code before returning from pgfault), the rest changes in the page fault
handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new
fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
them as-is.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
arch/alpha/mm/fault.c | 4 ++++
arch/arc/mm/fault.c | 4 ++++
arch/arm/mm/fault.c | 4 ++++
arch/arm64/mm/fault.c | 4 ++++
arch/csky/mm/fault.c | 4 ++++
arch/hexagon/mm/vm_fault.c | 4 ++++
arch/ia64/mm/fault.c | 4 ++++
arch/m68k/mm/fault.c | 4 ++++
arch/microblaze/mm/fault.c | 4 ++++
arch/mips/mm/fault.c | 4 ++++
arch/nios2/mm/fault.c | 4 ++++
arch/openrisc/mm/fault.c | 4 ++++
arch/parisc/mm/fault.c | 4 ++++
arch/powerpc/mm/copro_fault.c | 5 +++++
arch/powerpc/mm/fault.c | 5 +++++
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 12 +++++++++++-
arch/sh/mm/fault.c | 4 ++++
arch/sparc/mm/fault_32.c | 4 ++++
arch/sparc/mm/fault_64.c | 5 +++++
arch/um/kernel/trap.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
arch/xtensa/mm/fault.c | 4 ++++
include/linux/mm_types.h | 2 ++
mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
mm/memory.c | 2 +-
26 files changed, 138 insertions(+), 3 deletions(-)
diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
index ec20c1004abf..ef427a6bdd1a 100644
--- a/arch/alpha/mm/fault.c
+++ b/arch/alpha/mm/fault.c
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
index dad27e4d69ff..5ca59a482632 100644
--- a/arch/arc/mm/fault.c
+++ b/arch/arc/mm/fault.c
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* Fault retry nuances, mmap_lock already relinquished by core mm
*/
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a062e07516dd..46cccd6bf705 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (!(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 77341b160aca..e401d416bbd6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (fault & VM_FAULT_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7215a46b6b8e..e15f736cca4b 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
index 4fac4b9eb316..f73c7cbfe326 100644
--- a/arch/hexagon/mm/vm_fault.c
+++ b/arch/hexagon/mm/vm_fault.c
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/* The most common case -- we are done. */
if (likely(!(fault & VM_FAULT_ERROR))) {
if (fault & VM_FAULT_RETRY) {
diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
index 07379d1a227f..ef78c2d66cdd 100644
--- a/arch/ia64/mm/fault.c
+++ b/arch/ia64/mm/fault.c
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We ran out of memory, or some other thing happened
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index 71aa9f6315dc..4d2837eb3e2a 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return 0;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
index a9626e6a68af..5c40c3ebe52f 100644
--- a/arch/microblaze/mm/fault.c
+++ b/arch/microblaze/mm/fault.c
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 44f98100e84e..6f72bac39bf2 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index a32f14cd72f2..edaca0a6c1c1 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
index 80bb66ad42f6..c18f7abd64df 100644
--- a/arch/openrisc/mm/fault.c
+++ b/arch/openrisc/mm/fault.c
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
index f114e102aaf2..fe57175a7792 100644
--- a/arch/parisc/mm/fault.c
+++ b/arch/parisc/mm/fault.c
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We hit a shared mapping outside of the file, or some
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index c1cb21a00884..7c507fb48182 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
ret = 0;
*flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (*flt & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(*flt & VM_FAULT_ERROR)) {
if (*flt & VM_FAULT_OOM) {
ret = -ENOMEM;
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index d53fed4eccbd..014005428687 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return user_mode(regs) ? 0 : SIGBUS;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto out;
+
/*
* Handle the retry right now, the mmap_lock has been released in that
* case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (unlikely(fault & VM_FAULT_ERROR))
return mm_fault_error(regs, address, fault);
+out:
/*
* Major/minor page fault accounting.
*/
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 4e9efbe46d5f..d6a87f4137ca 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_RETRY)) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..9503a7cfaf03 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
unsigned long address;
unsigned int flags;
vm_fault_t fault;
+ bool need_unlock = true;
bool is_write;
tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ need_unlock = false;
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
}
fault = 0;
out_up:
- mmap_read_unlock(mm);
+ if (need_unlock)
+ mmap_read_unlock(mm);
out:
return fault;
}
diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
index e175667b1363..acd2f5e50bfc 100644
--- a/arch/sh/mm/fault.c
+++ b/arch/sh/mm/fault.c
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
if (mm_fault_error(regs, error_code, address, fault))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
index ad569d9bd124..91259f291c54 100644
--- a/arch/sparc/mm/fault_32.c
+++ b/arch/sparc/mm/fault_32.c
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index 253e07043298..4acc12eafbf5 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
goto exit_exception;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto lock_released;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
}
mmap_read_unlock(mm);
+lock_released:
mm_rss = get_mm_rss(mm);
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
index d1d5d0be0308..d3ce21c4ca32 100644
--- a/arch/um/kernel/trap.c
+++ b/arch/um/kernel/trap.c
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
goto out_nosemaphore;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM) {
goto out_of_memory;
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index d0074c6ed31a..12ed70b432d6 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* If we need to retry the mmap_lock has already been released,
* and if there is a fatal signal pending there is no guarantee
diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
index 06d0973a0d74..5f64305ba8d7 100644
--- a/arch/xtensa/mm/fault.c
+++ b/arch/xtensa/mm/fault.c
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c09b7f0555b8..decc275db3c9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
* fsync() to complete (for synchronous page faults
* in DAX)
+ * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
* @VM_FAULT_HINDEX_MASK: mask HINDEX value
*
*/
@@ -746,6 +747,7 @@ enum vm_fault_reason {
VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
+ VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
};
diff --git a/mm/gup.c b/mm/gup.c
index 2e07cff3b31b..3347b083d70b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
}
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+ * mmap lock in the page fault handler. Sanity check this.
+ */
+ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+ if (locked)
+ *locked = 0;
+ /*
+ * We should do the same as VM_FAULT_RETRY, but let's not
+ * return -EBUSY since that's not reflecting the reality on
+ * what has happened - we've just fully completed a page
+ * fault, with the mmap lock released. Use -EAGAIN to show
+ * that we want to take the mmap lock _again_.
+ */
+ return -EAGAIN;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
case 0:
goto retry;
case -EBUSY:
+ case -EAGAIN:
ret = 0;
fallthrough;
case -EFAULT:
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
return -EINTR;
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * NOTE: it's a pity that we need to retake the lock here
+ * to pair with the unlock() in the callers. Ideally we
+ * could tell the callers so they do not need to unlock.
+ */
+ mmap_read_lock(mm);
+ *unlocked = true;
+ return 0;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
/* VM_FAULT_RETRY couldn't trigger, bypass */
return ret;
- /* VM_FAULT_RETRY cannot return errors */
+ /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
if (!*locked) {
BUG_ON(ret < 0);
BUG_ON(ret >= nr_pages);
diff --git a/mm/memory.c b/mm/memory.c
index 54d106e0c999..a8be2d7a8718 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
balance_dirty_pages_ratelimited(mapping);
if (fpin) {
fput(fpin);
- return VM_FAULT_RETRY;
+ return VM_FAULT_COMPLETED;
}
}
--
2.32.0
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-24 23:45 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-24 23:45 UTC (permalink / raw)
To: linux-kernel, linux-mm
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, peterx, James E . J . Bottomley, Max Filippov,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller,
Alistair Popple, Hugh Dickins, Russell King, linux-csky,
linux-sh, Ingo Molnar, Geert Uytterhoeven, linux-arm-kernel,
Vineet Gupta, Stafford Horne, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Stefan Kristiansson, Richard Weinberger, linux-m68k, openrisc,
Ivan Kokshaysky, Al Viro, Andy Lutomirski, Paul Walmsley,
Thomas Gleixner, linux-alpha, Andrew Morton, Vlastimil Babka,
Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, linux-parisc, linux-mips, Dinh Nguyen,
Palmer Dabbelt, Sven Schnelle, Guo Ren, Borislav Petkov,
Johannes Berg, linuxppc-dev, David S . Miller
I observed that for each of the shared file-backed page faults, we're very
likely to retry one more time for the 1st write fault upon no page. It's
because we'll need to release the mmap lock for dirty rate limit purpose
with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return
to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need
to be retried at all since the pgtable was well installed before the
throttling, so the next continuous fault (including taking mmap read lock,
walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add
more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at
"pgfault" value in /proc/vmstat, then we should expect 2 counts for each
shmem write simply because we retried, and vm event "pgfault" will capture
that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
show that we've completed the whole fault and released the lock. It's also
a hint that we should very possibly not need another fault immediately on
this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple
program sequentially dirtying 400MB shmem file being mmap()ed and these are
the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap
code before returning from pgfault), the rest changes in the page fault
handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new
fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
them as-is.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
arch/alpha/mm/fault.c | 4 ++++
arch/arc/mm/fault.c | 4 ++++
arch/arm/mm/fault.c | 4 ++++
arch/arm64/mm/fault.c | 4 ++++
arch/csky/mm/fault.c | 4 ++++
arch/hexagon/mm/vm_fault.c | 4 ++++
arch/ia64/mm/fault.c | 4 ++++
arch/m68k/mm/fault.c | 4 ++++
arch/microblaze/mm/fault.c | 4 ++++
arch/mips/mm/fault.c | 4 ++++
arch/nios2/mm/fault.c | 4 ++++
arch/openrisc/mm/fault.c | 4 ++++
arch/parisc/mm/fault.c | 4 ++++
arch/powerpc/mm/copro_fault.c | 5 +++++
arch/powerpc/mm/fault.c | 5 +++++
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 12 +++++++++++-
arch/sh/mm/fault.c | 4 ++++
arch/sparc/mm/fault_32.c | 4 ++++
arch/sparc/mm/fault_64.c | 5 +++++
arch/um/kernel/trap.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
arch/xtensa/mm/fault.c | 4 ++++
include/linux/mm_types.h | 2 ++
mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
mm/memory.c | 2 +-
26 files changed, 138 insertions(+), 3 deletions(-)
diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
index ec20c1004abf..ef427a6bdd1a 100644
--- a/arch/alpha/mm/fault.c
+++ b/arch/alpha/mm/fault.c
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
index dad27e4d69ff..5ca59a482632 100644
--- a/arch/arc/mm/fault.c
+++ b/arch/arc/mm/fault.c
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* Fault retry nuances, mmap_lock already relinquished by core mm
*/
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a062e07516dd..46cccd6bf705 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (!(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 77341b160aca..e401d416bbd6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (fault & VM_FAULT_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7215a46b6b8e..e15f736cca4b 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
index 4fac4b9eb316..f73c7cbfe326 100644
--- a/arch/hexagon/mm/vm_fault.c
+++ b/arch/hexagon/mm/vm_fault.c
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/* The most common case -- we are done. */
if (likely(!(fault & VM_FAULT_ERROR))) {
if (fault & VM_FAULT_RETRY) {
diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
index 07379d1a227f..ef78c2d66cdd 100644
--- a/arch/ia64/mm/fault.c
+++ b/arch/ia64/mm/fault.c
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We ran out of memory, or some other thing happened
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index 71aa9f6315dc..4d2837eb3e2a 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return 0;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
index a9626e6a68af..5c40c3ebe52f 100644
--- a/arch/microblaze/mm/fault.c
+++ b/arch/microblaze/mm/fault.c
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 44f98100e84e..6f72bac39bf2 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index a32f14cd72f2..edaca0a6c1c1 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
index 80bb66ad42f6..c18f7abd64df 100644
--- a/arch/openrisc/mm/fault.c
+++ b/arch/openrisc/mm/fault.c
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
index f114e102aaf2..fe57175a7792 100644
--- a/arch/parisc/mm/fault.c
+++ b/arch/parisc/mm/fault.c
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We hit a shared mapping outside of the file, or some
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index c1cb21a00884..7c507fb48182 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
ret = 0;
*flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (*flt & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(*flt & VM_FAULT_ERROR)) {
if (*flt & VM_FAULT_OOM) {
ret = -ENOMEM;
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index d53fed4eccbd..014005428687 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return user_mode(regs) ? 0 : SIGBUS;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto out;
+
/*
* Handle the retry right now, the mmap_lock has been released in that
* case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (unlikely(fault & VM_FAULT_ERROR))
return mm_fault_error(regs, address, fault);
+out:
/*
* Major/minor page fault accounting.
*/
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 4e9efbe46d5f..d6a87f4137ca 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_RETRY)) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..9503a7cfaf03 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
unsigned long address;
unsigned int flags;
vm_fault_t fault;
+ bool need_unlock = true;
bool is_write;
tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ need_unlock = false;
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
}
fault = 0;
out_up:
- mmap_read_unlock(mm);
+ if (need_unlock)
+ mmap_read_unlock(mm);
out:
return fault;
}
diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
index e175667b1363..acd2f5e50bfc 100644
--- a/arch/sh/mm/fault.c
+++ b/arch/sh/mm/fault.c
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
if (mm_fault_error(regs, error_code, address, fault))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
index ad569d9bd124..91259f291c54 100644
--- a/arch/sparc/mm/fault_32.c
+++ b/arch/sparc/mm/fault_32.c
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index 253e07043298..4acc12eafbf5 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
goto exit_exception;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto lock_released;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
}
mmap_read_unlock(mm);
+lock_released:
mm_rss = get_mm_rss(mm);
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
index d1d5d0be0308..d3ce21c4ca32 100644
--- a/arch/um/kernel/trap.c
+++ b/arch/um/kernel/trap.c
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
goto out_nosemaphore;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM) {
goto out_of_memory;
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index d0074c6ed31a..12ed70b432d6 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* If we need to retry the mmap_lock has already been released,
* and if there is a fatal signal pending there is no guarantee
diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
index 06d0973a0d74..5f64305ba8d7 100644
--- a/arch/xtensa/mm/fault.c
+++ b/arch/xtensa/mm/fault.c
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c09b7f0555b8..decc275db3c9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
* fsync() to complete (for synchronous page faults
* in DAX)
+ * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
* @VM_FAULT_HINDEX_MASK: mask HINDEX value
*
*/
@@ -746,6 +747,7 @@ enum vm_fault_reason {
VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
+ VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
};
diff --git a/mm/gup.c b/mm/gup.c
index 2e07cff3b31b..3347b083d70b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
}
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+ * mmap lock in the page fault handler. Sanity check this.
+ */
+ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+ if (locked)
+ *locked = 0;
+ /*
+ * We should do the same as VM_FAULT_RETRY, but let's not
+ * return -EBUSY since that's not reflecting the reality on
+ * what has happened - we've just fully completed a page
+ * fault, with the mmap lock released. Use -EAGAIN to show
+ * that we want to take the mmap lock _again_.
+ */
+ return -EAGAIN;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
case 0:
goto retry;
case -EBUSY:
+ case -EAGAIN:
ret = 0;
fallthrough;
case -EFAULT:
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
return -EINTR;
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * NOTE: it's a pity that we need to retake the lock here
+ * to pair with the unlock() in the callers. Ideally we
+ * could tell the callers so they do not need to unlock.
+ */
+ mmap_read_lock(mm);
+ *unlocked = true;
+ return 0;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
/* VM_FAULT_RETRY couldn't trigger, bypass */
return ret;
- /* VM_FAULT_RETRY cannot return errors */
+ /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
if (!*locked) {
BUG_ON(ret < 0);
BUG_ON(ret >= nr_pages);
diff --git a/mm/memory.c b/mm/memory.c
index 54d106e0c999..a8be2d7a8718 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
balance_dirty_pages_ratelimited(mapping);
if (fpin) {
fput(fpin);
- return VM_FAULT_RETRY;
+ return VM_FAULT_COMPLETED;
}
}
--
2.32.0
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-24 23:45 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-24 23:45 UTC (permalink / raw)
To: linux-kernel, linux-mm
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, peterx,
James E . J . Bottomley, Max Filippov, Rich Felker,
Paul Mackerras, H . Peter Anvin, sparclinux, linux-ia64,
Alexander Gordeev, Will Deacon, linux-riscv, Anton Ivanov,
Jonas Bonn, linux-s390, linux-snps-arc, Yoshinori Sato,
linux-xtensa, linux-hexagon, Helge Deller, Alistair Popple,
Hugh Dickins, Russell King, linux-csky, linux-sh, Ingo Molnar,
linux-arm-kernel, Vineet Gupta, Matt Turner,
Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, Heiko Carstens, Johannes Weiner,
linux-um, Nicholas Piggin, Richard Weinberger, linux-m68k,
openrisc, Ivan Kokshaysky, Al Viro, Andy Lutomirski,
Paul Walmsley, Thomas Gleixner, linux-alpha, Andrew Morton,
Vlastimil Babka, Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, linux-parisc, linux-mips, Dinh Nguyen,
Palmer Dabbelt, Sven Schnelle, Guo Ren, Michael Ellerman,
Borislav Petkov, Johannes Berg, linuxppc-dev, David S . Miller
I observed that for each of the shared file-backed page faults, we're very
likely to retry one more time for the 1st write fault upon no page. It's
because we'll need to release the mmap lock for dirty rate limit purpose
with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return
to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need
to be retried at all since the pgtable was well installed before the
throttling, so the next continuous fault (including taking mmap read lock,
walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add
more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at
"pgfault" value in /proc/vmstat, then we should expect 2 counts for each
shmem write simply because we retried, and vm event "pgfault" will capture
that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
show that we've completed the whole fault and released the lock. It's also
a hint that we should very possibly not need another fault immediately on
this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple
program sequentially dirtying 400MB shmem file being mmap()ed and these are
the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap
code before returning from pgfault), the rest changes in the page fault
handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new
fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
them as-is.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
arch/alpha/mm/fault.c | 4 ++++
arch/arc/mm/fault.c | 4 ++++
arch/arm/mm/fault.c | 4 ++++
arch/arm64/mm/fault.c | 4 ++++
arch/csky/mm/fault.c | 4 ++++
arch/hexagon/mm/vm_fault.c | 4 ++++
arch/ia64/mm/fault.c | 4 ++++
arch/m68k/mm/fault.c | 4 ++++
arch/microblaze/mm/fault.c | 4 ++++
arch/mips/mm/fault.c | 4 ++++
arch/nios2/mm/fault.c | 4 ++++
arch/openrisc/mm/fault.c | 4 ++++
arch/parisc/mm/fault.c | 4 ++++
arch/powerpc/mm/copro_fault.c | 5 +++++
arch/powerpc/mm/fault.c | 5 +++++
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 12 +++++++++++-
arch/sh/mm/fault.c | 4 ++++
arch/sparc/mm/fault_32.c | 4 ++++
arch/sparc/mm/fault_64.c | 5 +++++
arch/um/kernel/trap.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
arch/xtensa/mm/fault.c | 4 ++++
include/linux/mm_types.h | 2 ++
mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
mm/memory.c | 2 +-
26 files changed, 138 insertions(+), 3 deletions(-)
diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
index ec20c1004abf..ef427a6bdd1a 100644
--- a/arch/alpha/mm/fault.c
+++ b/arch/alpha/mm/fault.c
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
index dad27e4d69ff..5ca59a482632 100644
--- a/arch/arc/mm/fault.c
+++ b/arch/arc/mm/fault.c
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* Fault retry nuances, mmap_lock already relinquished by core mm
*/
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a062e07516dd..46cccd6bf705 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (!(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 77341b160aca..e401d416bbd6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (fault & VM_FAULT_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7215a46b6b8e..e15f736cca4b 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
index 4fac4b9eb316..f73c7cbfe326 100644
--- a/arch/hexagon/mm/vm_fault.c
+++ b/arch/hexagon/mm/vm_fault.c
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/* The most common case -- we are done. */
if (likely(!(fault & VM_FAULT_ERROR))) {
if (fault & VM_FAULT_RETRY) {
diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
index 07379d1a227f..ef78c2d66cdd 100644
--- a/arch/ia64/mm/fault.c
+++ b/arch/ia64/mm/fault.c
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We ran out of memory, or some other thing happened
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index 71aa9f6315dc..4d2837eb3e2a 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return 0;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
index a9626e6a68af..5c40c3ebe52f 100644
--- a/arch/microblaze/mm/fault.c
+++ b/arch/microblaze/mm/fault.c
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 44f98100e84e..6f72bac39bf2 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index a32f14cd72f2..edaca0a6c1c1 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
index 80bb66ad42f6..c18f7abd64df 100644
--- a/arch/openrisc/mm/fault.c
+++ b/arch/openrisc/mm/fault.c
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
index f114e102aaf2..fe57175a7792 100644
--- a/arch/parisc/mm/fault.c
+++ b/arch/parisc/mm/fault.c
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We hit a shared mapping outside of the file, or some
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index c1cb21a00884..7c507fb48182 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
ret = 0;
*flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (*flt & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(*flt & VM_FAULT_ERROR)) {
if (*flt & VM_FAULT_OOM) {
ret = -ENOMEM;
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index d53fed4eccbd..014005428687 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return user_mode(regs) ? 0 : SIGBUS;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto out;
+
/*
* Handle the retry right now, the mmap_lock has been released in that
* case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (unlikely(fault & VM_FAULT_ERROR))
return mm_fault_error(regs, address, fault);
+out:
/*
* Major/minor page fault accounting.
*/
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 4e9efbe46d5f..d6a87f4137ca 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_RETRY)) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..9503a7cfaf03 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
unsigned long address;
unsigned int flags;
vm_fault_t fault;
+ bool need_unlock = true;
bool is_write;
tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ need_unlock = false;
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
}
fault = 0;
out_up:
- mmap_read_unlock(mm);
+ if (need_unlock)
+ mmap_read_unlock(mm);
out:
return fault;
}
diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
index e175667b1363..acd2f5e50bfc 100644
--- a/arch/sh/mm/fault.c
+++ b/arch/sh/mm/fault.c
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
if (mm_fault_error(regs, error_code, address, fault))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
index ad569d9bd124..91259f291c54 100644
--- a/arch/sparc/mm/fault_32.c
+++ b/arch/sparc/mm/fault_32.c
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index 253e07043298..4acc12eafbf5 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
goto exit_exception;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto lock_released;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
}
mmap_read_unlock(mm);
+lock_released:
mm_rss = get_mm_rss(mm);
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
index d1d5d0be0308..d3ce21c4ca32 100644
--- a/arch/um/kernel/trap.c
+++ b/arch/um/kernel/trap.c
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
goto out_nosemaphore;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM) {
goto out_of_memory;
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index d0074c6ed31a..12ed70b432d6 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* If we need to retry the mmap_lock has already been released,
* and if there is a fatal signal pending there is no guarantee
diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
index 06d0973a0d74..5f64305ba8d7 100644
--- a/arch/xtensa/mm/fault.c
+++ b/arch/xtensa/mm/fault.c
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c09b7f0555b8..decc275db3c9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
* fsync() to complete (for synchronous page faults
* in DAX)
+ * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
* @VM_FAULT_HINDEX_MASK: mask HINDEX value
*
*/
@@ -746,6 +747,7 @@ enum vm_fault_reason {
VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
+ VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
};
diff --git a/mm/gup.c b/mm/gup.c
index 2e07cff3b31b..3347b083d70b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
}
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+ * mmap lock in the page fault handler. Sanity check this.
+ */
+ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+ if (locked)
+ *locked = 0;
+ /*
+ * We should do the same as VM_FAULT_RETRY, but let's not
+ * return -EBUSY since that's not reflecting the reality on
+ * what has happened - we've just fully completed a page
+ * fault, with the mmap lock released. Use -EAGAIN to show
+ * that we want to take the mmap lock _again_.
+ */
+ return -EAGAIN;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
case 0:
goto retry;
case -EBUSY:
+ case -EAGAIN:
ret = 0;
fallthrough;
case -EFAULT:
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
return -EINTR;
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * NOTE: it's a pity that we need to retake the lock here
+ * to pair with the unlock() in the callers. Ideally we
+ * could tell the callers so they do not need to unlock.
+ */
+ mmap_read_lock(mm);
+ *unlocked = true;
+ return 0;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
/* VM_FAULT_RETRY couldn't trigger, bypass */
return ret;
- /* VM_FAULT_RETRY cannot return errors */
+ /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
if (!*locked) {
BUG_ON(ret < 0);
BUG_ON(ret >= nr_pages);
diff --git a/mm/memory.c b/mm/memory.c
index 54d106e0c999..a8be2d7a8718 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
balance_dirty_pages_ratelimited(mapping);
if (fpin) {
fput(fpin);
- return VM_FAULT_RETRY;
+ return VM_FAULT_COMPLETED;
}
}
--
2.32.0
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-24 23:45 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-24 23:45 UTC (permalink / raw)
To: linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, peterx, Matt Turner,
Albert Ou, Michal Simek, Russell King, Ivan Kokshaysky,
linux-riscv, Alexander Gordeev, Dave Hansen, Jonas Bonn,
Will Deacon, James E . J . Bottomley, H . Peter Anvin,
Andrea Arcangeli, openrisc, linux-s390, Ingo Molnar, linux-m68k,
Palmer Dabbelt, Heiko Carstens, Chris Zankel, Peter Zijlstra,
Alistair Popple, linux-csky, linux-hexagon, Vlastimil Babka,
Thomas Gleixner, sparclinux, Christian Borntraeger,
Stafford Horne, Michael Ellerman, x86, Thomas Bogendoerfer,
Paul Mackerras, linux-arm-kernel, Sven Schnelle,
Benjamin Herrenschmidt, linux-xtensa, Nicholas Piggin, linux-sh,
Vasily Gorbik, Borislav Petkov, linux-mips, Max Filippov,
Helge Deller, Vineet Gupta, Al Viro, Paul Walmsley,
Johannes Weiner, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
I observed that for each of the shared file-backed page faults, we're very
likely to retry one more time for the 1st write fault upon no page. It's
because we'll need to release the mmap lock for dirty rate limit purpose
with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return
to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need
to be retried at all since the pgtable was well installed before the
throttling, so the next continuous fault (including taking mmap read lock,
walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add
more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at
"pgfault" value in /proc/vmstat, then we should expect 2 counts for each
shmem write simply because we retried, and vm event "pgfault" will capture
that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
show that we've completed the whole fault and released the lock. It's also
a hint that we should very possibly not need another fault immediately on
this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple
program sequentially dirtying 400MB shmem file being mmap()ed and these are
the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap
code before returning from pgfault), the rest changes in the page fault
handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new
fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
them as-is.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
arch/alpha/mm/fault.c | 4 ++++
arch/arc/mm/fault.c | 4 ++++
arch/arm/mm/fault.c | 4 ++++
arch/arm64/mm/fault.c | 4 ++++
arch/csky/mm/fault.c | 4 ++++
arch/hexagon/mm/vm_fault.c | 4 ++++
arch/ia64/mm/fault.c | 4 ++++
arch/m68k/mm/fault.c | 4 ++++
arch/microblaze/mm/fault.c | 4 ++++
arch/mips/mm/fault.c | 4 ++++
arch/nios2/mm/fault.c | 4 ++++
arch/openrisc/mm/fault.c | 4 ++++
arch/parisc/mm/fault.c | 4 ++++
arch/powerpc/mm/copro_fault.c | 5 +++++
arch/powerpc/mm/fault.c | 5 +++++
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 12 +++++++++++-
arch/sh/mm/fault.c | 4 ++++
arch/sparc/mm/fault_32.c | 4 ++++
arch/sparc/mm/fault_64.c | 5 +++++
arch/um/kernel/trap.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
arch/xtensa/mm/fault.c | 4 ++++
include/linux/mm_types.h | 2 ++
mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
mm/memory.c | 2 +-
26 files changed, 138 insertions(+), 3 deletions(-)
diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
index ec20c1004abf..ef427a6bdd1a 100644
--- a/arch/alpha/mm/fault.c
+++ b/arch/alpha/mm/fault.c
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
index dad27e4d69ff..5ca59a482632 100644
--- a/arch/arc/mm/fault.c
+++ b/arch/arc/mm/fault.c
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* Fault retry nuances, mmap_lock already relinquished by core mm
*/
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a062e07516dd..46cccd6bf705 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (!(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 77341b160aca..e401d416bbd6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (fault & VM_FAULT_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7215a46b6b8e..e15f736cca4b 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
index 4fac4b9eb316..f73c7cbfe326 100644
--- a/arch/hexagon/mm/vm_fault.c
+++ b/arch/hexagon/mm/vm_fault.c
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/* The most common case -- we are done. */
if (likely(!(fault & VM_FAULT_ERROR))) {
if (fault & VM_FAULT_RETRY) {
diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
index 07379d1a227f..ef78c2d66cdd 100644
--- a/arch/ia64/mm/fault.c
+++ b/arch/ia64/mm/fault.c
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We ran out of memory, or some other thing happened
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index 71aa9f6315dc..4d2837eb3e2a 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return 0;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
index a9626e6a68af..5c40c3ebe52f 100644
--- a/arch/microblaze/mm/fault.c
+++ b/arch/microblaze/mm/fault.c
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 44f98100e84e..6f72bac39bf2 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index a32f14cd72f2..edaca0a6c1c1 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
index 80bb66ad42f6..c18f7abd64df 100644
--- a/arch/openrisc/mm/fault.c
+++ b/arch/openrisc/mm/fault.c
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
index f114e102aaf2..fe57175a7792 100644
--- a/arch/parisc/mm/fault.c
+++ b/arch/parisc/mm/fault.c
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We hit a shared mapping outside of the file, or some
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index c1cb21a00884..7c507fb48182 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
ret = 0;
*flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (*flt & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(*flt & VM_FAULT_ERROR)) {
if (*flt & VM_FAULT_OOM) {
ret = -ENOMEM;
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index d53fed4eccbd..014005428687 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return user_mode(regs) ? 0 : SIGBUS;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto out;
+
/*
* Handle the retry right now, the mmap_lock has been released in that
* case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (unlikely(fault & VM_FAULT_ERROR))
return mm_fault_error(regs, address, fault);
+out:
/*
* Major/minor page fault accounting.
*/
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 4e9efbe46d5f..d6a87f4137ca 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_RETRY)) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..9503a7cfaf03 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
unsigned long address;
unsigned int flags;
vm_fault_t fault;
+ bool need_unlock = true;
bool is_write;
tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ need_unlock = false;
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
}
fault = 0;
out_up:
- mmap_read_unlock(mm);
+ if (need_unlock)
+ mmap_read_unlock(mm);
out:
return fault;
}
diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
index e175667b1363..acd2f5e50bfc 100644
--- a/arch/sh/mm/fault.c
+++ b/arch/sh/mm/fault.c
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
if (mm_fault_error(regs, error_code, address, fault))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
index ad569d9bd124..91259f291c54 100644
--- a/arch/sparc/mm/fault_32.c
+++ b/arch/sparc/mm/fault_32.c
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index 253e07043298..4acc12eafbf5 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
goto exit_exception;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto lock_released;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
}
mmap_read_unlock(mm);
+lock_released:
mm_rss = get_mm_rss(mm);
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
index d1d5d0be0308..d3ce21c4ca32 100644
--- a/arch/um/kernel/trap.c
+++ b/arch/um/kernel/trap.c
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
goto out_nosemaphore;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM) {
goto out_of_memory;
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index d0074c6ed31a..12ed70b432d6 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* If we need to retry the mmap_lock has already been released,
* and if there is a fatal signal pending there is no guarantee
diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
index 06d0973a0d74..5f64305ba8d7 100644
--- a/arch/xtensa/mm/fault.c
+++ b/arch/xtensa/mm/fault.c
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c09b7f0555b8..decc275db3c9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
* fsync() to complete (for synchronous page faults
* in DAX)
+ * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
* @VM_FAULT_HINDEX_MASK: mask HINDEX value
*
*/
@@ -746,6 +747,7 @@ enum vm_fault_reason {
VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
+ VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
};
diff --git a/mm/gup.c b/mm/gup.c
index 2e07cff3b31b..3347b083d70b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
}
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+ * mmap lock in the page fault handler. Sanity check this.
+ */
+ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+ if (locked)
+ *locked = 0;
+ /*
+ * We should do the same as VM_FAULT_RETRY, but let's not
+ * return -EBUSY since that's not reflecting the reality on
+ * what has happened - we've just fully completed a page
+ * fault, with the mmap lock released. Use -EAGAIN to show
+ * that we want to take the mmap lock _again_.
+ */
+ return -EAGAIN;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
case 0:
goto retry;
case -EBUSY:
+ case -EAGAIN:
ret = 0;
fallthrough;
case -EFAULT:
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
return -EINTR;
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * NOTE: it's a pity that we need to retake the lock here
+ * to pair with the unlock() in the callers. Ideally we
+ * could tell the callers so they do not need to unlock.
+ */
+ mmap_read_lock(mm);
+ *unlocked = true;
+ return 0;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
/* VM_FAULT_RETRY couldn't trigger, bypass */
return ret;
- /* VM_FAULT_RETRY cannot return errors */
+ /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
if (!*locked) {
BUG_ON(ret < 0);
BUG_ON(ret >= nr_pages);
diff --git a/mm/memory.c b/mm/memory.c
index 54d106e0c999..a8be2d7a8718 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
balance_dirty_pages_ratelimited(mapping);
if (fpin) {
fput(fpin);
- return VM_FAULT_RETRY;
+ return VM_FAULT_COMPLETED;
}
}
--
2.32.0
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-24 23:45 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-24 23:45 UTC (permalink / raw)
To: linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, peterx, Matt Turner,
Albert Ou, Michal Simek, Russell King, Ivan Kokshaysky,
linux-riscv, Alexander Gordeev, Dave Hansen, Jonas Bonn,
Will Deacon, James E . J . Bottomley, H . Peter Anvin,
Andrea Arcangeli, openrisc, linux-s390, Ingo Molnar, linux-m68k,
Palmer Dabbelt
I observed that for each of the shared file-backed page faults, we're very
likely to retry one more time for the 1st write fault upon no page. It's
because we'll need to release the mmap lock for dirty rate limit purpose
with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
Then after that throttling we return VM_FAULT_RETRY.
We did that probably because VM_FAULT_RETRY is the only way we can return
to the fault handler at that time telling it we've released the mmap lock.
However that's not ideal because it's very likely the fault does not need
to be retried at all since the pgtable was well installed before the
throttling, so the next continuous fault (including taking mmap read lock,
walk the pgtable, etc.) could be in most cases unnecessary.
It's not only slowing down page faults for shared file-backed, but also add
more mmap lock contention which is in most cases not needed at all.
To observe this, one could try to write to some shmem page and look at
"pgfault" value in /proc/vmstat, then we should expect 2 counts for each
shmem write simply because we retried, and vm event "pgfault" will capture
that.
To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
show that we've completed the whole fault and released the lock. It's also
a hint that we should very possibly not need another fault immediately on
this page because we've just completed it.
This patch provides a ~12% perf boost on my aarch64 test VM with a simple
program sequentially dirtying 400MB shmem file being mmap()ed and these are
the time it needs:
Before: 650.980 ms (+-1.94%)
After: 569.396 ms (+-1.38%)
I believe it could help more than that.
We need some special care on GUP and the s390 pgfault handler (for gmap
code before returning from pgfault), the rest changes in the page fault
handlers should be relatively straightforward.
Another thing to mention is that mm_account_fault() does take this new
fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
them as-is.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
v3:
- Rebase to akpm/mm-unstable
- Copy arch maintainers
---
arch/alpha/mm/fault.c | 4 ++++
arch/arc/mm/fault.c | 4 ++++
arch/arm/mm/fault.c | 4 ++++
arch/arm64/mm/fault.c | 4 ++++
arch/csky/mm/fault.c | 4 ++++
arch/hexagon/mm/vm_fault.c | 4 ++++
arch/ia64/mm/fault.c | 4 ++++
arch/m68k/mm/fault.c | 4 ++++
arch/microblaze/mm/fault.c | 4 ++++
arch/mips/mm/fault.c | 4 ++++
arch/nios2/mm/fault.c | 4 ++++
arch/openrisc/mm/fault.c | 4 ++++
arch/parisc/mm/fault.c | 4 ++++
arch/powerpc/mm/copro_fault.c | 5 +++++
arch/powerpc/mm/fault.c | 5 +++++
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 12 +++++++++++-
arch/sh/mm/fault.c | 4 ++++
arch/sparc/mm/fault_32.c | 4 ++++
arch/sparc/mm/fault_64.c | 5 +++++
arch/um/kernel/trap.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
arch/xtensa/mm/fault.c | 4 ++++
include/linux/mm_types.h | 2 ++
mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
mm/memory.c | 2 +-
26 files changed, 138 insertions(+), 3 deletions(-)
diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
index ec20c1004abf..ef427a6bdd1a 100644
--- a/arch/alpha/mm/fault.c
+++ b/arch/alpha/mm/fault.c
@@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
index dad27e4d69ff..5ca59a482632 100644
--- a/arch/arc/mm/fault.c
+++ b/arch/arc/mm/fault.c
@@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* Fault retry nuances, mmap_lock already relinquished by core mm
*/
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a062e07516dd..46cccd6bf705 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (!(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 77341b160aca..e401d416bbd6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
return 0;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (fault & VM_FAULT_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
index 7215a46b6b8e..e15f736cca4b 100644
--- a/arch/csky/mm/fault.c
+++ b/arch/csky/mm/fault.c
@@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
index 4fac4b9eb316..f73c7cbfe326 100644
--- a/arch/hexagon/mm/vm_fault.c
+++ b/arch/hexagon/mm/vm_fault.c
@@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/* The most common case -- we are done. */
if (likely(!(fault & VM_FAULT_ERROR))) {
if (fault & VM_FAULT_RETRY) {
diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
index 07379d1a227f..ef78c2d66cdd 100644
--- a/arch/ia64/mm/fault.c
+++ b/arch/ia64/mm/fault.c
@@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We ran out of memory, or some other thing happened
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index 71aa9f6315dc..4d2837eb3e2a 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return 0;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
index a9626e6a68af..5c40c3ebe52f 100644
--- a/arch/microblaze/mm/fault.c
+++ b/arch/microblaze/mm/fault.c
@@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
index 44f98100e84e..6f72bac39bf2 100644
--- a/arch/mips/mm/fault.c
+++ b/arch/mips/mm/fault.c
@@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
index a32f14cd72f2..edaca0a6c1c1 100644
--- a/arch/nios2/mm/fault.c
+++ b/arch/nios2/mm/fault.c
@@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
index 80bb66ad42f6..c18f7abd64df 100644
--- a/arch/openrisc/mm/fault.c
+++ b/arch/openrisc/mm/fault.c
@@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
index f114e102aaf2..fe57175a7792 100644
--- a/arch/parisc/mm/fault.c
+++ b/arch/parisc/mm/fault.c
@@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
/*
* We hit a shared mapping outside of the file, or some
diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
index c1cb21a00884..7c507fb48182 100644
--- a/arch/powerpc/mm/copro_fault.c
+++ b/arch/powerpc/mm/copro_fault.c
@@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
ret = 0;
*flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (*flt & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(*flt & VM_FAULT_ERROR)) {
if (*flt & VM_FAULT_OOM) {
ret = -ENOMEM;
diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
index d53fed4eccbd..014005428687 100644
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (fault_signal_pending(fault, regs))
return user_mode(regs) ? 0 : SIGBUS;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto out;
+
/*
* Handle the retry right now, the mmap_lock has been released in that
* case.
@@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
if (unlikely(fault & VM_FAULT_ERROR))
return mm_fault_error(regs, address, fault);
+out:
/*
* Major/minor page fault accounting.
*/
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 4e9efbe46d5f..d6a87f4137ca 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_RETRY)) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..9503a7cfaf03 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
unsigned long address;
unsigned int flags;
vm_fault_t fault;
+ bool need_unlock = true;
bool is_write;
tsk = current;
@@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ need_unlock = false;
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
@@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
}
fault = 0;
out_up:
- mmap_read_unlock(mm);
+ if (need_unlock)
+ mmap_read_unlock(mm);
out:
return fault;
}
diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
index e175667b1363..acd2f5e50bfc 100644
--- a/arch/sh/mm/fault.c
+++ b/arch/sh/mm/fault.c
@@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
if (mm_fault_error(regs, error_code, address, fault))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (fault & VM_FAULT_RETRY) {
flags |= FAULT_FLAG_TRIED;
diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
index ad569d9bd124..91259f291c54 100644
--- a/arch/sparc/mm/fault_32.c
+++ b/arch/sparc/mm/fault_32.c
@@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
if (fault_signal_pending(fault, regs))
return;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index 253e07043298..4acc12eafbf5 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
if (fault_signal_pending(fault, regs))
goto exit_exception;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ goto lock_released;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
@@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
}
mmap_read_unlock(mm);
+lock_released:
mm_rss = get_mm_rss(mm);
#if defined(CONFIG_TRANSPARENT_HUGEPAGE)
mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
index d1d5d0be0308..d3ce21c4ca32 100644
--- a/arch/um/kernel/trap.c
+++ b/arch/um/kernel/trap.c
@@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
goto out_nosemaphore;
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return 0;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM) {
goto out_of_memory;
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index d0074c6ed31a..12ed70b432d6 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
/*
* If we need to retry the mmap_lock has already been released,
* and if there is a fatal signal pending there is no guarantee
diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
index 06d0973a0d74..5f64305ba8d7 100644
--- a/arch/xtensa/mm/fault.c
+++ b/arch/xtensa/mm/fault.c
@@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
return;
}
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED)
+ return;
+
if (unlikely(fault & VM_FAULT_ERROR)) {
if (fault & VM_FAULT_OOM)
goto out_of_memory;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index c09b7f0555b8..decc275db3c9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
* @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
* fsync() to complete (for synchronous page faults
* in DAX)
+ * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
* @VM_FAULT_HINDEX_MASK: mask HINDEX value
*
*/
@@ -746,6 +747,7 @@ enum vm_fault_reason {
VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
+ VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
};
diff --git a/mm/gup.c b/mm/gup.c
index 2e07cff3b31b..3347b083d70b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
}
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+ * mmap lock in the page fault handler. Sanity check this.
+ */
+ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+ if (locked)
+ *locked = 0;
+ /*
+ * We should do the same as VM_FAULT_RETRY, but let's not
+ * return -EBUSY since that's not reflecting the reality on
+ * what has happened - we've just fully completed a page
+ * fault, with the mmap lock released. Use -EAGAIN to show
+ * that we want to take the mmap lock _again_.
+ */
+ return -EAGAIN;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, *flags);
@@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
case 0:
goto retry;
case -EBUSY:
+ case -EAGAIN:
ret = 0;
fallthrough;
case -EFAULT:
@@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
return -EINTR;
ret = handle_mm_fault(vma, address, fault_flags, NULL);
+
+ if (ret & VM_FAULT_COMPLETED) {
+ /*
+ * NOTE: it's a pity that we need to retake the lock here
+ * to pair with the unlock() in the callers. Ideally we
+ * could tell the callers so they do not need to unlock.
+ */
+ mmap_read_lock(mm);
+ *unlocked = true;
+ return 0;
+ }
+
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
@@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
/* VM_FAULT_RETRY couldn't trigger, bypass */
return ret;
- /* VM_FAULT_RETRY cannot return errors */
+ /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
if (!*locked) {
BUG_ON(ret < 0);
BUG_ON(ret >= nr_pages);
diff --git a/mm/memory.c b/mm/memory.c
index 54d106e0c999..a8be2d7a8718 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
balance_dirty_pages_ratelimited(mapping);
if (fpin) {
fput(fpin);
- return VM_FAULT_RETRY;
+ return VM_FAULT_COMPLETED;
}
}
--
2.32.0
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (5 preceding siblings ...)
(?)
@ 2022-05-25 8:03 ` Geert Uytterhoeven
-1 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 8:03 ` Geert Uytterhoeven
0 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 8:03 ` Geert Uytterhoeven
0 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 8:03 ` Geert Uytterhoeven
0 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, Linux MM, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-sh, Ingo Molnar, linux-arm-kernel,
Vineet Gupta, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
linux-alpha, Andrew Morton, Vlastimil Babka, Richard Henderson,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, linux-parisc,
Max Filippov, Linux Kernel Mailing List, Dinh Nguyen,
Palmer Dabbelt, Sven Schnelle, Guo Ren, Michael Ellerman,
Borislav Petkov, Johannes Berg, linuxppc-dev, David S . Miller
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 8:03 ` Geert Uytterhoeven
0 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mips, James E . J . Bottomley, Linux MM,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller,
Alistair Popple, Hugh Dickins, Russell King, linux-csky,
linux-sh, Ingo Molnar, linux-arm-kernel, Vineet Gupta,
Stafford Horne, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Stefan Kristiansson, Richard Weinberger, linux-m68k, openrisc,
Ivan Kokshaysky, Al Viro, Andy Lutomirski, Paul Walmsley,
Thomas Gleixner, linux-alpha, Andrew Morton, Vlastimil Babka,
Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, linux-parisc, Max Filippov,
Linux Kernel Mailing List, Dinh Nguyen, Palmer Dabbelt,
Sven Schnelle, Guo Ren, Borislav Petkov, Johannes Berg,
linuxppc-dev, David S . Miller
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 8:03 ` Geert Uytterhoeven
0 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux MM, Richard Henderson,
David Hildenbrand, Peter Xu, Matt Turner, Albert Ou,
Michal Simek, Russell King, Ivan Kokshaysky, linux-riscv,
Alexander Gordeev, Dave Hansen, Jonas Bonn, Will Deacon,
James E . J . Bottomley, H . Peter Anvin, Andrea Arcangeli,
openrisc, linux-s390, Ingo Molnar, linux-m68k, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, linux-hexagon, Vlastimil Babka, Thomas Gleixner,
sparclinux, Christian Borntraeger, Stafford Horne,
Michael Ellerman, x86, Thomas Bogendoerfer, Paul Mackerras,
linux-arm-kernel, Sven Schnelle, Benjamin Herrenschmidt,
linux-xtensa, Nicholas Piggin, linux-sh, Vasily Gorbik,
Borislav Petkov, linux-mips, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 8:03 ` Geert Uytterhoeven
0 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 8:03 ` Geert Uytterhoeven
0 siblings, 0 replies; 87+ messages in thread
From: Geert Uytterhoeven @ 2022-05-25 8:03 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar
On Wed, May 25, 2022 at 1:45 AM Peter Xu <peterx@redhat.com> wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> arch/m68k/mm/fault.c | 4 ++++
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (5 preceding siblings ...)
(?)
@ 2022-05-25 11:10 ` Peter Zijlstra
-1 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 11:10 ` Peter Zijlstra
0 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 11:10 ` Peter Zijlstra
0 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 11:10 ` Peter Zijlstra
0 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Benjamin Herrenschmidt,
Dave Hansen, linux-mips, James E . J . Bottomley, linux-mm,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller,
Alistair Popple, Hugh Dickins, Russell King, linux-csky,
linux-sh, Ingo Molnar, linux-arm-kernel, Vineet Gupta,
Matt Turner, Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, Heiko Carstens, Johannes Weiner,
linux-um, Nicholas Piggin, Richard Weinberger, linux-m68k,
openrisc, Ivan Kokshaysky, Al Viro, Andy Lutomirski,
Paul Walmsley, Thomas Gleixner, linux-alpha, Andrew Morton,
Vlastimil Babka, Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, linux-parisc, Max Filippov, linux-kernel,
Dinh Nguyen, Palmer Dabbelt, Sven Schnelle, Guo Ren,
Michael Ellerman, Borislav Petkov, Johannes Berg, linuxppc-dev,
David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 11:10 ` Peter Zijlstra
0 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Dave Hansen, linux-mips,
James E . J . Bottomley, linux-mm, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-sh, Ingo Molnar, Geert Uytterhoeven,
linux-arm-kernel, Vineet Gupta, Stafford Horne, Matt Turner,
Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, Heiko Carstens, Johannes Weiner,
linux-um, Nicholas Piggin, Stefan Kristiansson,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
linux-alpha, Andrew Morton, Vlastimil Babka, Richard Henderson,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, linux-parisc,
Max Filippov, linux-kernel, Dinh Nguyen, Palmer Dabbelt,
Sven Schnelle, Guo Ren, Borislav Petkov, Johannes Berg,
linuxppc-dev, David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 11:10 ` Peter Zijlstra
0 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 11:10 ` Peter Zijlstra
0 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 11:10 ` Peter Zijlstra
0 siblings, 0 replies; 87+ messages in thread
From: Peter Zijlstra @ 2022-05-25 11:10 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Alistair
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (4 preceding siblings ...)
(?)
@ 2022-05-25 12:44 ` Johannes Weiner
-1 siblings, 0 replies; 87+ messages in thread
From: Johannes Weiner @ 2022-05-25 12:44 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 12:44 ` Johannes Weiner
0 siblings, 0 replies; 87+ messages in thread
From: Johannes Weiner @ 2022-05-25 12:44 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 12:44 ` Johannes Weiner
0 siblings, 0 replies; 87+ messages in thread
From: Johannes Weiner @ 2022-05-25 12:44 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 12:44 ` Johannes Weiner
0 siblings, 0 replies; 87+ messages in thread
From: Johannes Weiner @ 2022-05-25 12:44 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mips, James E . J . Bottomley, linux-mm,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller,
Alistair Popple, Hugh Dickins, Russell King, linux-csky,
linux-sh
Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, linux-parisc@vger.kernel.org, Max Filippov <jcmvbkbc@gmail.com>, linux-kernel@vger.kernel.org, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 12:44 ` Johannes Weiner
0 siblings, 0 replies; 87+ messages in thread
From: Johannes Weiner @ 2022-05-25 12:44 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, linux-mm, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-sh, Ingo Molnar, linux-arm-kernel,
Vineet Gupta, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, linux-alpha, linux-um, Nicholas Piggin,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
Andrew Morton, Vlastimil Babka, Richard Henderson, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, linux-parisc, Max Filippov,
linux-kernel, Dinh Nguyen, Palmer Dabbelt, Sven Schnelle,
Guo Ren, Michael Ellerman, Borislav Petkov, Johannes Berg,
linuxppc-dev, David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 12:44 ` Johannes Weiner
0 siblings, 0 replies; 87+ messages in thread
From: Johannes Weiner @ 2022-05-25 12:44 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Anton Ivanov, Catalin Marinas, linux-um,
linux-alpha, Johannes Berg, linux-ia64, Geert Uytterhoeven,
Dinh Nguyen, Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-25 12:44 ` Johannes Weiner
0 siblings, 0 replies; 87+ messages in thread
From: Johannes Weiner @ 2022-05-25 12:44 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (4 preceding siblings ...)
(?)
@ 2022-05-26 3:40 ` Vineet Gupta
-1 siblings, 0 replies; 87+ messages in thread
From: Vineet Gupta @ 2022-05-26 3:40 UTC (permalink / raw)
To: Peter Xu, linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, Matt Turner, Albert Ou,
Michal Simek, Russell King, Ivan Kokshaysky, linux-riscv,
Alexander Gordeev, Dave Hansen, Jonas Bonn, Will Deacon,
James E . J . Bottomley, H . Peter Anvin, Andrea Arcangeli,
openrisc, linux-s390, Ingo Molnar, linux-m68k, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, linux-hexagon, Vlastimil Babka, Thomas Gleixner,
sparclinux, Christian Borntraeger, Stafford Horne,
Michael Ellerman, x86, Thomas Bogendoerfer, Paul Mackerras,
linux-arm-kernel, Sven Schnelle, Benjamin Herrenschmidt,
linux-xtensa, Nicholas Piggin, linux-sh, Vasily Gorbik,
Borislav Petkov, linux-mips, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On 5/24/22 16:45, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu<peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/arc/mm/fault.c | 4 ++++
Acked-by: Vineet Gupta <vgupta@kernel.org>
Thx,
-Vineet
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-26 3:40 ` Vineet Gupta
0 siblings, 0 replies; 87+ messages in thread
From: Vineet Gupta @ 2022-05-26 3:40 UTC (permalink / raw)
To: Peter Xu, linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, Matt Turner, Albert Ou,
Michal Simek, Russell King, Ivan Kokshaysky, linux-riscv,
Alexander Gordeev, Dave Hansen, Jonas Bonn, Will Deacon,
James E . J . Bottomley, H . Peter Anvin, Andrea Arcangeli,
openrisc, linux-s390, Ingo Molnar, linux-m68k, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, linux-hexagon, Vlastimil Babka, Thomas Gleixner,
sparclinux, Christian Borntraeger, Stafford Horne,
Michael Ellerman, x86, Thomas Bogendoerfer, Paul Mackerras,
linux-arm-kernel, Sven Schnelle, Benjamin Herrenschmidt,
linux-xtensa, Nicholas Piggin, linux-sh, Vasily Gorbik,
Borislav Petkov, linux-mips, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On 5/24/22 16:45, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu<peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/arc/mm/fault.c | 4 ++++
Acked-by: Vineet Gupta <vgupta@kernel.org>
Thx,
-Vineet
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-26 3:40 ` Vineet Gupta
0 siblings, 0 replies; 87+ messages in thread
From: Vineet Gupta @ 2022-05-26 3:40 UTC (permalink / raw)
To: Peter Xu, linux-kernel, linux-mm
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, James E . J . Bottomley,
Max Filippov, Rich Felker, Paul Mackerras, H . Peter Anvin,
sparclinux, linux-ia64, Alexander Gordeev, Will Deacon,
linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-sh, Ingo Molnar, linux-arm-kernel,
Vineet Gupta, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
linux-alpha, Andrew Morton, Vlastimil Babka, Richard Henderson,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, linux-parisc,
linux-mips, Dinh Nguyen, Palmer Dabbelt, Sven Schnelle, Guo Ren,
Michael Ellerman, Borislav Petkov, Johannes Berg, linuxppc-dev,
David S . Miller
On 5/24/22 16:45, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu<peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/arc/mm/fault.c | 4 ++++
Acked-by: Vineet Gupta <vgupta@kernel.org>
Thx,
-Vineet
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-26 3:40 ` Vineet Gupta
0 siblings, 0 replies; 87+ messages in thread
From: Vineet Gupta @ 2022-05-26 3:40 UTC (permalink / raw)
To: Peter Xu, linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, Matt Turner, Albert Ou,
Michal Simek, Russell King, Ivan Kokshaysky, linux-riscv,
Alexander Gordeev, Dave Hansen, Jonas Bonn, Will Deacon,
James E . J . Bottomley, H . Peter Anvin, Andrea Arcangeli,
openrisc, linux-s390, Ingo Molnar, linux-m68k, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, linux-hexagon, Vlastimil Babka, Thomas Gleixner,
sparclinux, Christian Borntraeger, Stafford Horne,
Michael Ellerman, x86, Thomas Bogendoerfer, Paul Mackerras,
linux-arm-kernel, Sven Schnelle, Benjamin Herrenschmidt,
linux-xtensa, Nicholas Piggin, linux-sh, Vasily Gorbik,
Borislav Petkov, linux-mips, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On 5/24/22 16:45, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu<peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/arc/mm/fault.c | 4 ++++
Acked-by: Vineet Gupta <vgupta@kernel.org>
Thx,
-Vineet
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-26 3:40 ` Vineet Gupta
0 siblings, 0 replies; 87+ messages in thread
From: Vineet Gupta @ 2022-05-26 3:40 UTC (permalink / raw)
To: Peter Xu, linux-kernel, linux-mm
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, James E . J . Bottomley, Max Filippov, Rich Felker,
Paul Mackerras, H . Peter Anvin, sparclinux, linux-ia64,
Alexander Gordeev, Will Deacon, linux-riscv, Anton Ivanov,
Jonas Bonn, linux-s390, linux-snps-arc, Yoshinori Sato,
linux-xtensa, linux-hexagon, Helge Deller, Alistair Popple,
Hugh Dickins, Russell King, linux-csky, linux-sh, Ing o Molnar,
Geert Uytterhoeven, linux-arm-kernel, Vineet Gupta,
Stafford Horne, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Stefan Kristiansson, Richard Weinberger, linux-m68k, openrisc,
Ivan Kokshaysky, Al Viro, Andy Lutomirski, Paul Walmsley,
Thomas Gleixner, linux-alpha, Andrew Morton, Vlastimil Babka
nkel <chris@zankel.net>, Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, linux-parisc@vger.kernel.org, linux-mips@vger.kernel.org, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
On 5/24/22 16:45, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu<peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/arc/mm/fault.c | 4 ++++
Acked-by: Vineet Gupta <vgupta@kernel.org>
Thx,
-Vineet
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-26 3:40 ` Vineet Gupta
0 siblings, 0 replies; 87+ messages in thread
From: Vineet Gupta @ 2022-05-26 3:40 UTC (permalink / raw)
To: Peter Xu, linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, Matt Turner, Albert Ou,
Michal Simek, Russell King, Ivan Kokshaysky, linux-riscv,
Alexander Gordeev, Dave Hansen, Jonas Bonn, Will Deacon,
James E . J . Bottomley, H . Peter Anvin, Andrea Arcangeli,
openrisc, linux-s390, Ingo Molnar, linux-m68k, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, linux-hexagon, Vlastimil Babka, Thomas Gleixner,
sparclinux, Christian Borntraeger, Stafford Horne,
Michael Ellerman, x86, Thomas Bogendoerfer, Paul Mackerras,
linux-arm-kernel, Sven Schnelle, Benjamin Herrenschmidt,
linux-xtensa, Nicholas Piggin, linux-sh, Vasily Gorbik,
Borislav Petkov, linux-mips, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
Guo Ren, linux-snps-arc, Hugh Dickins, Rich Felker,
Andy Lutomirski, Richard Weinberger, linuxppc-dev, Brian Cain,
Yoshinori Sato, Andrew Morton, Stefan Kristiansson, linux-parisc,
David S . Miller
On 5/24/22 16:45, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu<peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/arc/mm/fault.c | 4 ++++
Acked-by: Vineet Gupta <vgupta@kernel.org>
Thx,
-Vineet
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-26 3:40 ` Vineet Gupta
0 siblings, 0 replies; 87+ messages in thread
From: Vineet Gupta @ 2022-05-26 3:40 UTC (permalink / raw)
To: Peter Xu, linux-kernel, linux-mm
Cc: Richard Henderson, David Hildenbrand, Matt Turner, Albert Ou,
Michal Simek, Russell King, Ivan Kokshaysky, linux-riscv,
Alexander Gordeev, Dave Hansen, Jonas Bonn, Will Deacon,
James E . J . Bottomley, H . Peter Anvin, Andrea Arcangeli,
openrisc, linux-s390, Ingo Molnar, linux-m68k, Palmer Dabbelt,
Heiko
On 5/24/22 16:45, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu<peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/arc/mm/fault.c | 4 ++++
Acked-by: Vineet Gupta <vgupta@kernel.org>
Thx,
-Vineet
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (4 preceding siblings ...)
(?)
@ 2022-05-27 2:54 ` Guo Ren
-1 siblings, 0 replies; 87+ messages in thread
From: Guo Ren @ 2022-05-27 2:54 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux-MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, Openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
the arch/x86 maintainers, Thomas Bogendoerfer, Paul Mackerras,
Linux ARM, Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
open list:BROADCOM NVRAM DRIVER, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
linux-snps-arc, Hugh Dickins, Rich Felker, Andy Lutomirski,
Richard Weinberger, linuxppc-dev, Brian Cain, Yoshinori Sato,
Andrew Morton, Stefan Kristiansson, Parisc List,
David S . Miller
For csky part.
Acked-by: Guo Ren <guoren@kernel.org>
On Wed, May 25, 2022 at 7:45 AM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
>
> --
> 2.32.0
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 2:54 ` Guo Ren
0 siblings, 0 replies; 87+ messages in thread
From: Guo Ren @ 2022-05-27 2:54 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux-MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, Openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
the arch/x86 maintainers, Thomas Bogendoerfer, Paul Mackerras,
Linux ARM, Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
open list:BROADCOM NVRAM DRIVER, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
linux-snps-arc, Hugh Dickins, Rich Felker, Andy Lutomirski,
Richard Weinberger, linuxppc-dev, Brian Cain, Yoshinori Sato,
Andrew Morton, Stefan Kristiansson, Parisc List,
David S . Miller
For csky part.
Acked-by: Guo Ren <guoren@kernel.org>
On Wed, May 25, 2022 at 7:45 AM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
>
> --
> 2.32.0
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 2:54 ` Guo Ren
0 siblings, 0 replies; 87+ messages in thread
From: Guo Ren @ 2022-05-27 2:54 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux-MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, Openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
the arch/x86 maintainers, Thomas Bogendoerfer, Paul Mackerras,
Linux ARM, Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
open list:BROADCOM NVRAM DRIVER, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
linux-snps-arc, Hugh Dickins, Rich Felker, Andy Lutomirski,
Richard Weinberger, linuxppc-dev, Brian Cain, Yoshinori Sato,
Andrew Morton, Stefan Kristiansson, Parisc List,
David S . Miller
For csky part.
Acked-by: Guo Ren <guoren@kernel.org>
On Wed, May 25, 2022 at 7:45 AM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
>
> --
> 2.32.0
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 2:54 ` Guo Ren
0 siblings, 0 replies; 87+ messages in thread
From: Guo Ren @ 2022-05-27 2:54 UTC (permalink / raw)
To: Peter Xu
Cc: the arch/x86 maintainers, Catalin Marinas, David Hildenbrand,
Peter Zijlstra, Benjamin Herrenschmidt, Dave Hansen,
open list:BROADCOM NVRAM DRIVER, James E . J . Bottomley,
Linux-MM, Rich Felker, Paul Mackerras, H . Peter Anvin,
sparclinux, linux-ia64, Alexander Gordeev, Will Deacon,
linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato,
open list:TENSILICA XTENSA PORT (xtensa),
linux-hexagon, Helge Deller, Alistair Popple, Hugh Dickins,
Russell King, linux-csky, linux-sh, Ingo Molnar, Linux ARM,
Vineet Gupta, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Richard Weinberger, linux-m68k, Openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
linux-alpha, Andrew Morton, Vlastimil Babka, Richard Henderson,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, Parisc List,
Max Filippov, Linux Kernel Mailing List, Dinh Nguyen,
Palmer Dabbelt, Sven Schnelle, Michael Ellerman, Borislav Petkov,
Johannes Berg, linuxppc-dev, David S . Miller
For csky part.
Acked-by: Guo Ren <guoren@kernel.org>
On Wed, May 25, 2022 at 7:45 AM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
>
> --
> 2.32.0
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 2:54 ` Guo Ren
0 siblings, 0 replies; 87+ messages in thread
From: Guo Ren @ 2022-05-27 2:54 UTC (permalink / raw)
To: Peter Xu
Cc: the arch/x86 maintainers, Catalin Marinas, David Hildenbrand,
Peter Zijlstra, Dave Hansen, open list:BROADCOM NVRAM DRIVER,
James E . J . Bottomley, Linux-MM, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato,
open list:TENSILICA XTENSA PORT (xtensa),
linux-hexagon, Helge Deller
e.com>, Thomas Gleixner <tglx@linutronix.de>, linux-alpha@vger.kernel.org, Andrew Morton <akpm@linux-foundation.org>, Vlastimil Babka <vbabka@suse.cz>, Richard Henderson <rth@twiddle.net>, Chris Zankel <chris@zankel.net>, Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, Parisc List <linux-parisc@vger.kernel.org>, Max Filippov <jcmvbkbc@gmail.com>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev <linuxppc-dev@lists.ozlabs.org>, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
For csky part.
Acked-by: Guo Ren <guoren@kernel.org>
On Wed, May 25, 2022 at 7:45 AM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
>
> --
> 2.32.0
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 2:54 ` Guo Ren
0 siblings, 0 replies; 87+ messages in thread
From: Guo Ren @ 2022-05-27 2:54 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux-MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, Openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
the arch/x86 maintainers, Thomas Bogendoerfer, Paul Mackerras,
Linux ARM, Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
open list:BROADCOM NVRAM DRIVER, Max Filippov, Helge Deller,
Vineet Gupta, Al Viro, Paul Walmsley, Johannes Weiner,
Anton Ivanov, Catalin Marinas, linux-um, linux-alpha,
Johannes Berg, linux-ia64, Geert Uytterhoeven, Dinh Nguyen,
linux-snps-arc, Hugh Dickins, Rich Felker, Andy Lutomirski,
Richard Weinberger, linuxppc-dev, Brian Cain, Yoshinori Sato,
Andrew Morton, Stefan Kristiansson, Parisc List,
David S . Miller
For csky part.
Acked-by: Guo Ren <guoren@kernel.org>
On Wed, May 25, 2022 at 7:45 AM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
>
> --
> 2.32.0
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 2:54 ` Guo Ren
0 siblings, 0 replies; 87+ messages in thread
From: Guo Ren @ 2022-05-27 2:54 UTC (permalink / raw)
To: Peter Xu
Cc: Linux Kernel Mailing List, Linux-MM, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, Openrisc, linux-s390
For csky part.
Acked-by: Guo Ren <guoren@kernel.org>
On Wed, May 25, 2022 at 7:45 AM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
>
> --
> 2.32.0
>
--
Best Regards
Guo Ren
ML: https://lore.kernel.org/linux-csky/
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (4 preceding siblings ...)
(?)
@ 2022-05-27 5:39 ` Max Filippov
-1 siblings, 0 replies; 87+ messages in thread
From: Max Filippov @ 2022-05-27 5:39 UTC (permalink / raw)
To: Peter Xu
Cc: LKML, Linux Memory Management List, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, open list:M68K ARCHITECTURE, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, open list:QUALCOMM HEXAGON...,
Vlastimil Babka, Thomas Gleixner, open list:SPARC + UltraSPAR...,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
maintainer:X86 ARCHITECTURE...,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, open list:SUPERH, Vasily Gorbik,
Borislav Petkov, linux-mips, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, open list:ALPHA PORT, Johannes Berg,
open list:IA64 (Itanium) PL...,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, open list:PARISC ARCHITECTURE,
David S . Miller
On Tue, May 24, 2022 at 4:45 PM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
For xtensa:
Acked-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 5:39 ` Max Filippov
0 siblings, 0 replies; 87+ messages in thread
From: Max Filippov @ 2022-05-27 5:39 UTC (permalink / raw)
To: Peter Xu
Cc: LKML, Linux Memory Management List, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, open list:M68K ARCHITECTURE, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, open list:QUALCOMM HEXAGON...,
Vlastimil Babka, Thomas Gleixner, open list:SPARC + UltraSPAR...,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
maintainer:X86 ARCHITECTURE...,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, open list:SUPERH, Vasily Gorbik,
Borislav Petkov, linux-mips, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, open list:ALPHA PORT, Johannes Berg,
open list:IA64 (Itanium) PL...,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, open list:PARISC ARCHITECTURE,
David S . Miller
On Tue, May 24, 2022 at 4:45 PM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
For xtensa:
Acked-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 5:39 ` Max Filippov
0 siblings, 0 replies; 87+ messages in thread
From: Max Filippov @ 2022-05-27 5:39 UTC (permalink / raw)
To: Peter Xu
Cc: maintainer:X86 ARCHITECTURE...,
Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, Linux Memory Management List,
Rich Felker, Paul Mackerras, H . Peter Anvin,
open list:SPARC + UltraSPAR..., open list:IA64 (Itanium) PL...,
Alexander Gordeev, Will Deacon, linux-riscv, Anton Ivanov,
Jonas Bonn, linux-s390, linux-snps-arc, Yoshinori Sato,
open list:TENSILICA XTENSA PORT (xtensa),
open list:QUALCOMM HEXAGON...,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, open list:SUPERH, Ingo Molnar, linux-arm-kernel,
Vineet Gupta, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Richard Weinberger, open list:M68K ARCHITECTURE, openrisc,
Ivan Kokshaysky, Al Viro, Andy Lutomirski, Paul Walmsley,
Thomas Gleixner, open list:ALPHA PORT, Andrew Morton,
Vlastimil Babka, Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, open list:PARISC ARCHITECTURE, LKML,
Dinh Nguyen, Palmer Dabbelt, Sven Schnelle, Guo Ren,
Michael Ellerman, Borislav Petkov, Johannes Berg, linuxppc-dev,
David S . Miller
On Tue, May 24, 2022 at 4:45 PM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
For xtensa:
Acked-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 5:39 ` Max Filippov
0 siblings, 0 replies; 87+ messages in thread
From: Max Filippov @ 2022-05-27 5:39 UTC (permalink / raw)
To: Peter Xu
Cc: LKML, Linux Memory Management List, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, open list:M68K ARCHITECTURE, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, open list:QUALCOMM HEXAGON...,
Vlastimil Babka, Thomas Gleixner, open list:SPARC + UltraSPAR...,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
maintainer:X86 ARCHITECTURE...,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, open list:SUPERH, Vasily Gorbik,
Borislav Petkov, linux-mips, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, open list:ALPHA PORT, Johannes Berg,
open list:IA64 (Itanium) PL...,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, open list:PARISC ARCHITECTURE,
David S . Miller
On Tue, May 24, 2022 at 4:45 PM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
For xtensa:
Acked-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 5:39 ` Max Filippov
0 siblings, 0 replies; 87+ messages in thread
From: Max Filippov @ 2022-05-27 5:39 UTC (permalink / raw)
To: Peter Xu
Cc: maintainer:X86 ARCHITECTURE...,
Catalin Marinas, David Hildenbrand, Peter Zijlstra, Dave Hansen,
linux-mips, James E . J . Bottomley,
Linux Memory Management List, Rich Felker, Paul Mackerras,
H . Peter Anvin, open list:SPARC + UltraSPAR...,
open list:IA64 (Itanium) PL...,
Alexander Gordeev, Will Deacon, linux-riscv, Anton Ivanov,
Jonas Bonn, linux-s390, linux-snps-arc, Yoshinori Sato,
open list:TENSILICA XTENSA PORT (xtensa),
o pen list:QUALCOMM HEXAGON...,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, open list:SUPERH, Ingo Molnar, Geert Uytterhoeven,
linux-arm-kernel, Vineet Gupta, Stafford Horne, Matt Turner,
Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, Heiko Carstens, Johannes Weiner,
linux-um, Nicholas Piggin, Stefan Kristiansson,
Richard Weinberger, open list:M68K ARCHITECTURE, openrisc
k.msu.ru>, Al Viro <viro@zeniv.linux.org.uk>, Andy Lutomirski <luto@kernel.org>, Paul Walmsley <paul.walmsley@sifive.com>, Thomas Gleixner <tglx@linutronix.de>, "open list:ALPHA PORT" <linux-alpha@vger.kernel.org>, Andrew Morton <akpm@linux-foundation.org>, Vlastimil Babka <vbabka@suse.cz>, Richard Henderson <rth@twiddle.net>, Chris Zankel <chris@zankel.net>, Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, "open list:PARISC ARCHITECTURE" <linux-parisc@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
On Tue, May 24, 2022 at 4:45 PM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
For xtensa:
Acked-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 5:39 ` Max Filippov
0 siblings, 0 replies; 87+ messages in thread
From: Max Filippov @ 2022-05-27 5:39 UTC (permalink / raw)
To: Peter Xu
Cc: LKML, Linux Memory Management List, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, open list:M68K ARCHITECTURE, Palmer Dabbelt,
Heiko Carstens, Chris Zankel, Peter Zijlstra, Alistair Popple,
linux-csky, open list:QUALCOMM HEXAGON...,
Vlastimil Babka, Thomas Gleixner, open list:SPARC + UltraSPAR...,
Christian Borntraeger, Stafford Horne, Michael Ellerman,
maintainer:X86 ARCHITECTURE...,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt,
open list:TENSILICA XTENSA PORT (xtensa),
Nicholas Piggin, open list:SUPERH, Vasily Gorbik,
Borislav Petkov, linux-mips, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, open list:ALPHA PORT, Johannes Berg,
open list:IA64 (Itanium) PL...,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, open list:PARISC ARCHITECTURE,
David S . Miller
On Tue, May 24, 2022 at 4:45 PM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
For xtensa:
Acked-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 5:39 ` Max Filippov
0 siblings, 0 replies; 87+ messages in thread
From: Max Filippov @ 2022-05-27 5:39 UTC (permalink / raw)
To: Peter Xu
Cc: LKML, Linux Memory Management List, Richard Henderson,
David Hildenbrand, Matt Turner, Albert Ou, Michal Simek,
Russell King, Ivan Kokshaysky, linux-riscv, Alexander Gordeev,
Dave Hansen, Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390
On Tue, May 24, 2022 at 4:45 PM Peter Xu <peterx@redhat.com> wrote:
>
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
For xtensa:
Acked-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (5 preceding siblings ...)
(?)
@ 2022-05-27 8:21 ` Alistair Popple
-1 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 8:21 ` Alistair Popple
0 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 8:21 ` Alistair Popple
0 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 8:21 ` Alistair Popple
0 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: Rich Felker, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mips, James E . J . Bottomley, linux-mm,
Paul Mackerras, H . Peter Anvin, sparclinux, linux-ia64,
Alexander Gordeev, Will Deacon, linux-riscv, Anton Ivanov,
Jonas Bonn, linux-s390, linux-snps-arc, Yoshinori Sato,
linux-xtensa, linux-hexagon, Helge Deller, x86, Hugh Dickins,
Russell King, linux-csky, linux-sh, Ingo Molnar
Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, linux-parisc@vger.kernel.org, Max Filippov <jcmvbkbc@gmail.com>, linux-kernel@vger.kernel.org, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David
S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 8:21 ` Alistair Popple
0 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: Rich Felker, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, linux-mm, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, x86, Hugh Dickins, Russell King, linux-csky,
linux-sh, Ingo Molnar, linux-arm-kernel, Vineet Gupta,
Matt Turner, Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, Heiko Carstens, Johannes Weiner,
linux-um, Nicholas Piggin, Richard Weinberger, linux-m68k,
openrisc, Ivan Kokshaysky, Al Viro, Andy Lutomirski,
Paul Walmsley, Thomas Gleixner, linux-alpha, Andrew Morton,
Vlastimil Babka, Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, linux-parisc, Max Filippov, linux-kernel,
Dinh Nguyen, Palmer Dabbelt, Sven Schnelle, Guo Ren,
Michael Ellerman, Borislav Petkov, Johannes Berg, linuxppc-dev,
David S . Miller
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 8:21 ` Alistair Popple
0 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 8:21 ` Alistair Popple
0 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 8:21 ` Alistair Popple
0 siblings, 0 replies; 87+ messages in thread
From: Alistair Popple @ 2022-05-27 8:21 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k
Peter Xu <peterx@redhat.com> writes:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
I looked at the change generally and in particular for hmm_vma_fault()
and didn't see any issues there so feel free to add:
Reviewed-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>
> v3:
> - Rebase to akpm/mm-unstable
> - Copy arch maintainers
> ---
> arch/alpha/mm/fault.c | 4 ++++
> arch/arc/mm/fault.c | 4 ++++
> arch/arm/mm/fault.c | 4 ++++
> arch/arm64/mm/fault.c | 4 ++++
> arch/csky/mm/fault.c | 4 ++++
> arch/hexagon/mm/vm_fault.c | 4 ++++
> arch/ia64/mm/fault.c | 4 ++++
> arch/m68k/mm/fault.c | 4 ++++
> arch/microblaze/mm/fault.c | 4 ++++
> arch/mips/mm/fault.c | 4 ++++
> arch/nios2/mm/fault.c | 4 ++++
> arch/openrisc/mm/fault.c | 4 ++++
> arch/parisc/mm/fault.c | 4 ++++
> arch/powerpc/mm/copro_fault.c | 5 +++++
> arch/powerpc/mm/fault.c | 5 +++++
> arch/riscv/mm/fault.c | 4 ++++
> arch/s390/mm/fault.c | 12 +++++++++++-
> arch/sh/mm/fault.c | 4 ++++
> arch/sparc/mm/fault_32.c | 4 ++++
> arch/sparc/mm/fault_64.c | 5 +++++
> arch/um/kernel/trap.c | 4 ++++
> arch/x86/mm/fault.c | 4 ++++
> arch/xtensa/mm/fault.c | 4 ++++
> include/linux/mm_types.h | 2 ++
> mm/gup.c | 34 +++++++++++++++++++++++++++++++++-
> mm/memory.c | 2 +-
> 26 files changed, 138 insertions(+), 3 deletions(-)
>
> diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c
> index ec20c1004abf..ef427a6bdd1a 100644
> --- a/arch/alpha/mm/fault.c
> +++ b/arch/alpha/mm/fault.c
> @@ -155,6 +155,10 @@ do_page_fault(unsigned long address, unsigned long mmcsr,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c
> index dad27e4d69ff..5ca59a482632 100644
> --- a/arch/arc/mm/fault.c
> +++ b/arch/arc/mm/fault.c
> @@ -146,6 +146,10 @@ void do_page_fault(unsigned long address, struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * Fault retry nuances, mmap_lock already relinquished by core mm
> */
> diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
> index a062e07516dd..46cccd6bf705 100644
> --- a/arch/arm/mm/fault.c
> +++ b/arch/arm/mm/fault.c
> @@ -322,6 +322,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (!(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 77341b160aca..e401d416bbd6 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -607,6 +607,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
> return 0;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (fault & VM_FAULT_RETRY) {
> mm_flags |= FAULT_FLAG_TRIED;
> goto retry;
> diff --git a/arch/csky/mm/fault.c b/arch/csky/mm/fault.c
> index 7215a46b6b8e..e15f736cca4b 100644
> --- a/arch/csky/mm/fault.c
> +++ b/arch/csky/mm/fault.c
> @@ -285,6 +285,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely((fault & VM_FAULT_RETRY) && (flags & FAULT_FLAG_ALLOW_RETRY))) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c
> index 4fac4b9eb316..f73c7cbfe326 100644
> --- a/arch/hexagon/mm/vm_fault.c
> +++ b/arch/hexagon/mm/vm_fault.c
> @@ -96,6 +96,10 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /* The most common case -- we are done. */
> if (likely(!(fault & VM_FAULT_ERROR))) {
> if (fault & VM_FAULT_RETRY) {
> diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c
> index 07379d1a227f..ef78c2d66cdd 100644
> --- a/arch/ia64/mm/fault.c
> +++ b/arch/ia64/mm/fault.c
> @@ -139,6 +139,10 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We ran out of memory, or some other thing happened
> diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
> index 71aa9f6315dc..4d2837eb3e2a 100644
> --- a/arch/m68k/mm/fault.c
> +++ b/arch/m68k/mm/fault.c
> @@ -141,6 +141,10 @@ int do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return 0;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c
> index a9626e6a68af..5c40c3ebe52f 100644
> --- a/arch/microblaze/mm/fault.c
> +++ b/arch/microblaze/mm/fault.c
> @@ -222,6 +222,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c
> index 44f98100e84e..6f72bac39bf2 100644
> --- a/arch/mips/mm/fault.c
> +++ b/arch/mips/mm/fault.c
> @@ -162,6 +162,10 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/nios2/mm/fault.c b/arch/nios2/mm/fault.c
> index a32f14cd72f2..edaca0a6c1c1 100644
> --- a/arch/nios2/mm/fault.c
> +++ b/arch/nios2/mm/fault.c
> @@ -139,6 +139,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c
> index 80bb66ad42f6..c18f7abd64df 100644
> --- a/arch/openrisc/mm/fault.c
> +++ b/arch/openrisc/mm/fault.c
> @@ -167,6 +167,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c
> index f114e102aaf2..fe57175a7792 100644
> --- a/arch/parisc/mm/fault.c
> +++ b/arch/parisc/mm/fault.c
> @@ -309,6 +309,10 @@ void do_page_fault(struct pt_regs *regs, unsigned long code,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> /*
> * We hit a shared mapping outside of the file, or some
> diff --git a/arch/powerpc/mm/copro_fault.c b/arch/powerpc/mm/copro_fault.c
> index c1cb21a00884..7c507fb48182 100644
> --- a/arch/powerpc/mm/copro_fault.c
> +++ b/arch/powerpc/mm/copro_fault.c
> @@ -65,6 +65,11 @@ int copro_handle_mm_fault(struct mm_struct *mm, unsigned long ea,
>
> ret = 0;
> *flt = handle_mm_fault(vma, ea, is_write ? FAULT_FLAG_WRITE : 0, NULL);
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (*flt & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(*flt & VM_FAULT_ERROR)) {
> if (*flt & VM_FAULT_OOM) {
> ret = -ENOMEM;
> diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
> index d53fed4eccbd..014005428687 100644
> --- a/arch/powerpc/mm/fault.c
> +++ b/arch/powerpc/mm/fault.c
> @@ -511,6 +511,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (fault_signal_pending(fault, regs))
> return user_mode(regs) ? 0 : SIGBUS;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto out;
> +
> /*
> * Handle the retry right now, the mmap_lock has been released in that
> * case.
> @@ -525,6 +529,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address,
> if (unlikely(fault & VM_FAULT_ERROR))
> return mm_fault_error(regs, address, fault);
>
> +out:
> /*
> * Major/minor page fault accounting.
> */
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 4e9efbe46d5f..d6a87f4137ca 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -330,6 +330,10 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_RETRY)) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
> return fault;
> }
> diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c
> index e175667b1363..acd2f5e50bfc 100644
> --- a/arch/sh/mm/fault.c
> +++ b/arch/sh/mm/fault.c
> @@ -485,6 +485,10 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
> if (mm_fault_error(regs, error_code, address, fault))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (fault & VM_FAULT_RETRY) {
> flags |= FAULT_FLAG_TRIED;
>
> diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c
> index ad569d9bd124..91259f291c54 100644
> --- a/arch/sparc/mm/fault_32.c
> +++ b/arch/sparc/mm/fault_32.c
> @@ -190,6 +190,10 @@ asmlinkage void do_sparc_fault(struct pt_regs *regs, int text_fault, int write,
> if (fault_signal_pending(fault, regs))
> return;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
> index 253e07043298..4acc12eafbf5 100644
> --- a/arch/sparc/mm/fault_64.c
> +++ b/arch/sparc/mm/fault_64.c
> @@ -427,6 +427,10 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> if (fault_signal_pending(fault, regs))
> goto exit_exception;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + goto lock_released;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> @@ -449,6 +453,7 @@ asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)
> }
> mmap_read_unlock(mm);
>
> +lock_released:
> mm_rss = get_mm_rss(mm);
> #if defined(CONFIG_TRANSPARENT_HUGEPAGE)
> mm_rss -= (mm->context.thp_pte_count * (HPAGE_SIZE / PAGE_SIZE));
> diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c
> index d1d5d0be0308..d3ce21c4ca32 100644
> --- a/arch/um/kernel/trap.c
> +++ b/arch/um/kernel/trap.c
> @@ -76,6 +76,10 @@ int handle_page_fault(unsigned long address, unsigned long ip,
> if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
> goto out_nosemaphore;
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return 0;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM) {
> goto out_of_memory;
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..12ed70b432d6 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1408,6 +1408,10 @@ void do_user_addr_fault(struct pt_regs *regs,
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> /*
> * If we need to retry the mmap_lock has already been released,
> * and if there is a fatal signal pending there is no guarantee
> diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c
> index 06d0973a0d74..5f64305ba8d7 100644
> --- a/arch/xtensa/mm/fault.c
> +++ b/arch/xtensa/mm/fault.c
> @@ -118,6 +118,10 @@ void do_page_fault(struct pt_regs *regs)
> return;
> }
>
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED)
> + return;
> +
> if (unlikely(fault & VM_FAULT_ERROR)) {
> if (fault & VM_FAULT_OOM)
> goto out_of_memory;
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index c09b7f0555b8..decc275db3c9 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -729,6 +729,7 @@ typedef __bitwise unsigned int vm_fault_t;
> * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs
> * fsync() to complete (for synchronous page faults
> * in DAX)
> + * @VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
> * @VM_FAULT_HINDEX_MASK: mask HINDEX value
> *
> */
> @@ -746,6 +747,7 @@ enum vm_fault_reason {
> VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800,
> VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000,
> VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000,
> + VM_FAULT_COMPLETED = (__force vm_fault_t)0x004000,
> VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000,
> };
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 2e07cff3b31b..3347b083d70b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -951,6 +951,25 @@ static int faultin_page(struct vm_area_struct *vma,
> }
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
> + * mmap lock in the page fault handler. Sanity check this.
> + */
> + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
> + if (locked)
> + *locked = 0;
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
> + return -EAGAIN;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, *flags);
>
> @@ -1177,6 +1196,7 @@ static long __get_user_pages(struct mm_struct *mm,
> case 0:
> goto retry;
> case -EBUSY:
> + case -EAGAIN:
> ret = 0;
> fallthrough;
> case -EFAULT:
> @@ -1303,6 +1323,18 @@ int fixup_user_fault(struct mm_struct *mm,
> return -EINTR;
>
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
> + }
> +
> if (ret & VM_FAULT_ERROR) {
> int err = vm_fault_to_errno(ret, 0);
>
> @@ -1368,7 +1400,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
> /* VM_FAULT_RETRY couldn't trigger, bypass */
> return ret;
>
> - /* VM_FAULT_RETRY cannot return errors */
> + /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */
> if (!*locked) {
> BUG_ON(ret < 0);
> BUG_ON(ret >= nr_pages);
> diff --git a/mm/memory.c b/mm/memory.c
> index 54d106e0c999..a8be2d7a8718 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -3020,7 +3020,7 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
> balance_dirty_pages_ratelimited(mapping);
> if (fpin) {
> fput(fpin);
> - return VM_FAULT_RETRY;
> + return VM_FAULT_COMPLETED;
> }
> }
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (4 preceding siblings ...)
(?)
@ 2022-05-27 10:46 ` Ingo Molnar
-1 siblings, 0 replies; 87+ messages in thread
From: Ingo Molnar @ 2022-05-27 10:46 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
* Peter Xu <peterx@redhat.com> wrote:
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
Nice!
> arch/x86/mm/fault.c | 4 ++++
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Minor comment typo:
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
Indeed that's a pity - I guess more performance could be gained here,
especially in highly parallel threaded workloads?
Thanks,
Ingo
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 10:46 ` Ingo Molnar
0 siblings, 0 replies; 87+ messages in thread
From: Ingo Molnar @ 2022-05-27 10:46 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
* Peter Xu <peterx@redhat.com> wrote:
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
Nice!
> arch/x86/mm/fault.c | 4 ++++
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Minor comment typo:
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
Indeed that's a pity - I guess more performance could be gained here,
especially in highly parallel threaded workloads?
Thanks,
Ingo
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 10:46 ` Ingo Molnar
0 siblings, 0 replies; 87+ messages in thread
From: Ingo Molnar @ 2022-05-27 10:46 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
* Peter Xu <peterx@redhat.com> wrote:
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
Nice!
> arch/x86/mm/fault.c | 4 ++++
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Minor comment typo:
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
Indeed that's a pity - I guess more performance could be gained here,
especially in highly parallel threaded workloads?
Thanks,
Ingo
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 10:46 ` Ingo Molnar
0 siblings, 0 replies; 87+ messages in thread
From: Ingo Molnar @ 2022-05-27 10:46 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, linux-mm, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-sh, Ingo Molnar, linux-arm-kernel,
Vineet Gupta, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
linux-alpha, Andrew Morton, Vlastimil Babka, Richard Henderson,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, linux-parisc,
Max Filippov, linux-kernel, Dinh Nguyen, Palmer Dabbelt,
Sven Schnelle, Guo Ren, Michael Ellerman, Borislav Petkov,
Johannes Berg, linuxppc-dev, David S . Miller
* Peter Xu <peterx@redhat.com> wrote:
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
Nice!
> arch/x86/mm/fault.c | 4 ++++
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Minor comment typo:
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
Indeed that's a pity - I guess more performance could be gained here,
especially in highly parallel threaded workloads?
Thanks,
Ingo
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 10:46 ` Ingo Molnar
0 siblings, 0 replies; 87+ messages in thread
From: Ingo Molnar @ 2022-05-27 10:46 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mips, James E . J . Bottomley, linux-mm,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller,
Alistair Popple, Hugh Dickins, Russell King, linux-csky,
linux-sh
et>, Chris Zankel <chris@zankel.net>, Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, linux-parisc@vger.kernel.org, Max Filippov <jcmvbkbc@gmail.com>, linux-kernel@vger.kernel.org, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
* Peter Xu <peterx@redhat.com> wrote:
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
Nice!
> arch/x86/mm/fault.c | 4 ++++
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Minor comment typo:
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
Indeed that's a pity - I guess more performance could be gained here,
especially in highly parallel threaded workloads?
Thanks,
Ingo
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 10:46 ` Ingo Molnar
0 siblings, 0 replies; 87+ messages in thread
From: Ingo Molnar @ 2022-05-27 10:46 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
* Peter Xu <peterx@redhat.com> wrote:
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
Nice!
> arch/x86/mm/fault.c | 4 ++++
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Minor comment typo:
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
Indeed that's a pity - I guess more performance could be gained here,
especially in highly parallel threaded workloads?
Thanks,
Ingo
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 10:46 ` Ingo Molnar
0 siblings, 0 replies; 87+ messages in thread
From: Ingo Molnar @ 2022-05-27 10:46 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter
* Peter Xu <peterx@redhat.com> wrote:
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
Nice!
> arch/x86/mm/fault.c | 4 ++++
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Minor comment typo:
> + /*
> + * We should do the same as VM_FAULT_RETRY, but let's not
> + * return -EBUSY since that's not reflecting the reality on
> + * what has happened - we've just fully completed a page
> + * fault, with the mmap lock released. Use -EAGAIN to show
> + * that we want to take the mmap lock _again_.
> + */
s/reflecting the reality on what has happened
/reflecting the reality of what has happened
> ret = handle_mm_fault(vma, address, fault_flags, NULL);
> +
> + if (ret & VM_FAULT_COMPLETED) {
> + /*
> + * NOTE: it's a pity that we need to retake the lock here
> + * to pair with the unlock() in the callers. Ideally we
> + * could tell the callers so they do not need to unlock.
> + */
> + mmap_read_lock(mm);
> + *unlocked = true;
> + return 0;
Indeed that's a pity - I guess more performance could be gained here,
especially in highly parallel threaded workloads?
Thanks,
Ingo
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-24 23:45 ` Peter Xu
` (4 preceding siblings ...)
(?)
@ 2022-05-27 12:23 ` Heiko Carstens
-1 siblings, 0 replies; 87+ messages in thread
From: Heiko Carstens @ 2022-05-27 12:23 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 12:23 ` Heiko Carstens
0 siblings, 0 replies; 87+ messages in thread
From: Heiko Carstens @ 2022-05-27 12:23 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 12:23 ` Heiko Carstens
0 siblings, 0 replies; 87+ messages in thread
From: Heiko Carstens @ 2022-05-27 12:23 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 12:23 ` Heiko Carstens
0 siblings, 0 replies; 87+ messages in thread
From: Heiko Carstens @ 2022-05-27 12:23 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, linux-mm, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Janosch Frank, Yoshinori Sato, linux-sh,
linux-hexagon, Helge Deller, Alistair Popple, Hugh Dickins,
Russell King, linux-csky, linux-alpha, Ingo Molnar,
linux-arm-kernel, Vineet Gupta, Matt Turner,
Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, linux-xtensa, Johannes Weiner,
linux-um, Nicholas Piggin, Richard Weinberger, linux-m68k,
openrisc, Ivan Kokshaysky, Al Viro, Andy Lutomirski,
Paul Walmsley, Thomas Gleixner, Andrew Morton, Vlastimil Babka,
Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, linux-parisc, Max Filippov, linux-kernel,
Dinh Nguyen, Palmer Dabbelt, Sven Schnelle, Guo Ren,
Michael Ellerman, Borislav Petkov, Johannes Berg, linuxppc-dev,
David S . Miller
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 12:23 ` Heiko Carstens
0 siblings, 0 replies; 87+ messages in thread
From: Heiko Carstens @ 2022-05-27 12:23 UTC (permalink / raw)
To: Peter Xu
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mips, James E . J . Bottomley, linux-mm,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Janosch Frank, Yoshinori Sato, linux-sh, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-alpha, Ingo Molnar, Geert Uytterhoeven,
linux-arm-kernel, Vineet Gupta, Stafford Horne, Matt Turner,
Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, linux-xtensa, Johannes Weiner,
linux-um, Nicholas Piggin, Stefan Kristiansson,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
Andrew Morton, Vlastimil Babka, Richard Henderson
e.net>, Chris Zankel <chris@zankel.net>, Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, linux-parisc@vger.kernel.org, Max Filippov <jcmvbkbc@gmail.com>, linux-kernel@vger.kernel.org, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 12:23 ` Heiko Carstens
0 siblings, 0 replies; 87+ messages in thread
From: Heiko Carstens @ 2022-05-27 12:23 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 12:23 ` Heiko Carstens
0 siblings, 0 replies; 87+ messages in thread
From: Heiko Carstens @ 2022-05-27 12:23 UTC (permalink / raw)
To: Peter Xu
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt
On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page. It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
>
> Then after that throttling we return VM_FAULT_RETRY.
>
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
>
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
>
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
>
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
>
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock. It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
>
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
>
> Before: 650.980 ms (+-1.94%)
> After: 569.396 ms (+-1.38%)
>
> I believe it could help more than that.
>
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
>
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
>
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> unsigned long address;
> unsigned int flags;
> vm_fault_t fault;
> + bool need_unlock = true;
> bool is_write;
>
> tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> goto out_up;
> goto out;
> }
> +
> + /* The fault is fully completed (including releasing mmap lock) */
> + if (fault & VM_FAULT_COMPLETED) {
> + need_unlock = false;
> + goto out_gmap;
> + }
> +
> if (unlikely(fault & VM_FAULT_ERROR))
> goto out_up;
>
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> mmap_read_lock(mm);
> goto retry;
> }
> +out_gmap:
> if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> address = __gmap_link(gmap, current->thread.gmap_addr,
> address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> }
> fault = 0;
> out_up:
> - mmap_read_unlock(mm);
> + if (need_unlock)
> + mmap_read_unlock(mm);
> out:
This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-27 12:23 ` Heiko Carstens
` (4 preceding siblings ...)
(?)
@ 2022-05-27 13:49 ` Peter Xu
-1 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 13:49 UTC (permalink / raw)
To: Heiko Carstens
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
Hi, Heiko,
On Fri, May 27, 2022 at 02:23:42PM +0200, Heiko Carstens wrote:
> On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> > I observed that for each of the shared file-backed page faults, we're very
> > likely to retry one more time for the 1st write fault upon no page. It's
> > because we'll need to release the mmap lock for dirty rate limit purpose
> > with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> >
> > Then after that throttling we return VM_FAULT_RETRY.
> >
> > We did that probably because VM_FAULT_RETRY is the only way we can return
> > to the fault handler at that time telling it we've released the mmap lock.
> >
> > However that's not ideal because it's very likely the fault does not need
> > to be retried at all since the pgtable was well installed before the
> > throttling, so the next continuous fault (including taking mmap read lock,
> > walk the pgtable, etc.) could be in most cases unnecessary.
> >
> > It's not only slowing down page faults for shared file-backed, but also add
> > more mmap lock contention which is in most cases not needed at all.
> >
> > To observe this, one could try to write to some shmem page and look at
> > "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> > shmem write simply because we retried, and vm event "pgfault" will capture
> > that.
> >
> > To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> > show that we've completed the whole fault and released the lock. It's also
> > a hint that we should very possibly not need another fault immediately on
> > this page because we've just completed it.
> >
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
> >
> > I believe it could help more than that.
> >
> > We need some special care on GUP and the s390 pgfault handler (for gmap
> > code before returning from pgfault), the rest changes in the page fault
> > handlers should be relatively straightforward.
> >
> > Another thing to mention is that mm_account_fault() does take this new
> > fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> >
> > I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> > not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> > them as-is.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> ...
> > diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> > index e173b6187ad5..9503a7cfaf03 100644
> > --- a/arch/s390/mm/fault.c
> > +++ b/arch/s390/mm/fault.c
> > @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > unsigned long address;
> > unsigned int flags;
> > vm_fault_t fault;
> > + bool need_unlock = true;
> > bool is_write;
> >
> > tsk = current;
> > @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > goto out_up;
> > goto out;
> > }
> > +
> > + /* The fault is fully completed (including releasing mmap lock) */
> > + if (fault & VM_FAULT_COMPLETED) {
> > + need_unlock = false;
> > + goto out_gmap;
> > + }
> > +
> > if (unlikely(fault & VM_FAULT_ERROR))
> > goto out_up;
> >
> > @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > mmap_read_lock(mm);
> > goto retry;
> > }
> > +out_gmap:
> > if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> > address = __gmap_link(gmap, current->thread.gmap_addr,
> > address);
> > @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > }
> > fault = 0;
> > out_up:
> > - mmap_read_unlock(mm);
> > + if (need_unlock)
> > + mmap_read_unlock(mm);
> > out:
>
> This seems to be incorrect. __gmap_link() requires the mmap_lock to be
> held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the
comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming
gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at
least it'll avoid one pgtable work procedure even if the lock overhead was
kept.
With that, one more possible further optimization for s390 only will be
conditionally not taking that lock when !CONFIG_PGSTE, but I have totally
no idea whether that'll be a common use case, so I plan to leave that for
later in all cases.
Actually after doing that the whole changeset of s390 is even more
straightforward:
---8<---
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..4608cc962ecf 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ /*
+ * Gmap will need the mmap lock again, so retake it. TODO:
+ * only conditionally take the lock when CONFIG_PGSTE set.
+ */
+ mmap_read_lock(mm);
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
---8<---
Thanks,
--
Peter Xu
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 13:49 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 13:49 UTC (permalink / raw)
To: Heiko Carstens
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
Hi, Heiko,
On Fri, May 27, 2022 at 02:23:42PM +0200, Heiko Carstens wrote:
> On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> > I observed that for each of the shared file-backed page faults, we're very
> > likely to retry one more time for the 1st write fault upon no page. It's
> > because we'll need to release the mmap lock for dirty rate limit purpose
> > with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> >
> > Then after that throttling we return VM_FAULT_RETRY.
> >
> > We did that probably because VM_FAULT_RETRY is the only way we can return
> > to the fault handler at that time telling it we've released the mmap lock.
> >
> > However that's not ideal because it's very likely the fault does not need
> > to be retried at all since the pgtable was well installed before the
> > throttling, so the next continuous fault (including taking mmap read lock,
> > walk the pgtable, etc.) could be in most cases unnecessary.
> >
> > It's not only slowing down page faults for shared file-backed, but also add
> > more mmap lock contention which is in most cases not needed at all.
> >
> > To observe this, one could try to write to some shmem page and look at
> > "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> > shmem write simply because we retried, and vm event "pgfault" will capture
> > that.
> >
> > To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> > show that we've completed the whole fault and released the lock. It's also
> > a hint that we should very possibly not need another fault immediately on
> > this page because we've just completed it.
> >
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
> >
> > I believe it could help more than that.
> >
> > We need some special care on GUP and the s390 pgfault handler (for gmap
> > code before returning from pgfault), the rest changes in the page fault
> > handlers should be relatively straightforward.
> >
> > Another thing to mention is that mm_account_fault() does take this new
> > fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> >
> > I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> > not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> > them as-is.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> ...
> > diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> > index e173b6187ad5..9503a7cfaf03 100644
> > --- a/arch/s390/mm/fault.c
> > +++ b/arch/s390/mm/fault.c
> > @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > unsigned long address;
> > unsigned int flags;
> > vm_fault_t fault;
> > + bool need_unlock = true;
> > bool is_write;
> >
> > tsk = current;
> > @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > goto out_up;
> > goto out;
> > }
> > +
> > + /* The fault is fully completed (including releasing mmap lock) */
> > + if (fault & VM_FAULT_COMPLETED) {
> > + need_unlock = false;
> > + goto out_gmap;
> > + }
> > +
> > if (unlikely(fault & VM_FAULT_ERROR))
> > goto out_up;
> >
> > @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > mmap_read_lock(mm);
> > goto retry;
> > }
> > +out_gmap:
> > if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> > address = __gmap_link(gmap, current->thread.gmap_addr,
> > address);
> > @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > }
> > fault = 0;
> > out_up:
> > - mmap_read_unlock(mm);
> > + if (need_unlock)
> > + mmap_read_unlock(mm);
> > out:
>
> This seems to be incorrect. __gmap_link() requires the mmap_lock to be
> held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the
comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming
gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at
least it'll avoid one pgtable work procedure even if the lock overhead was
kept.
With that, one more possible further optimization for s390 only will be
conditionally not taking that lock when !CONFIG_PGSTE, but I have totally
no idea whether that'll be a common use case, so I plan to leave that for
later in all cases.
Actually after doing that the whole changeset of s390 is even more
straightforward:
---8<---
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..4608cc962ecf 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ /*
+ * Gmap will need the mmap lock again, so retake it. TODO:
+ * only conditionally take the lock when CONFIG_PGSTE set.
+ */
+ mmap_read_lock(mm);
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
---8<---
Thanks,
--
Peter Xu
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 13:49 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 13:49 UTC (permalink / raw)
To: Heiko Carstens
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
Hi, Heiko,
On Fri, May 27, 2022 at 02:23:42PM +0200, Heiko Carstens wrote:
> On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> > I observed that for each of the shared file-backed page faults, we're very
> > likely to retry one more time for the 1st write fault upon no page. It's
> > because we'll need to release the mmap lock for dirty rate limit purpose
> > with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> >
> > Then after that throttling we return VM_FAULT_RETRY.
> >
> > We did that probably because VM_FAULT_RETRY is the only way we can return
> > to the fault handler at that time telling it we've released the mmap lock.
> >
> > However that's not ideal because it's very likely the fault does not need
> > to be retried at all since the pgtable was well installed before the
> > throttling, so the next continuous fault (including taking mmap read lock,
> > walk the pgtable, etc.) could be in most cases unnecessary.
> >
> > It's not only slowing down page faults for shared file-backed, but also add
> > more mmap lock contention which is in most cases not needed at all.
> >
> > To observe this, one could try to write to some shmem page and look at
> > "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> > shmem write simply because we retried, and vm event "pgfault" will capture
> > that.
> >
> > To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> > show that we've completed the whole fault and released the lock. It's also
> > a hint that we should very possibly not need another fault immediately on
> > this page because we've just completed it.
> >
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
> >
> > I believe it could help more than that.
> >
> > We need some special care on GUP and the s390 pgfault handler (for gmap
> > code before returning from pgfault), the rest changes in the page fault
> > handlers should be relatively straightforward.
> >
> > Another thing to mention is that mm_account_fault() does take this new
> > fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> >
> > I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> > not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> > them as-is.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> ...
> > diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> > index e173b6187ad5..9503a7cfaf03 100644
> > --- a/arch/s390/mm/fault.c
> > +++ b/arch/s390/mm/fault.c
> > @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > unsigned long address;
> > unsigned int flags;
> > vm_fault_t fault;
> > + bool need_unlock = true;
> > bool is_write;
> >
> > tsk = current;
> > @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > goto out_up;
> > goto out;
> > }
> > +
> > + /* The fault is fully completed (including releasing mmap lock) */
> > + if (fault & VM_FAULT_COMPLETED) {
> > + need_unlock = false;
> > + goto out_gmap;
> > + }
> > +
> > if (unlikely(fault & VM_FAULT_ERROR))
> > goto out_up;
> >
> > @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > mmap_read_lock(mm);
> > goto retry;
> > }
> > +out_gmap:
> > if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> > address = __gmap_link(gmap, current->thread.gmap_addr,
> > address);
> > @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > }
> > fault = 0;
> > out_up:
> > - mmap_read_unlock(mm);
> > + if (need_unlock)
> > + mmap_read_unlock(mm);
> > out:
>
> This seems to be incorrect. __gmap_link() requires the mmap_lock to be
> held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the
comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming
gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at
least it'll avoid one pgtable work procedure even if the lock overhead was
kept.
With that, one more possible further optimization for s390 only will be
conditionally not taking that lock when !CONFIG_PGSTE, but I have totally
no idea whether that'll be a common use case, so I plan to leave that for
later in all cases.
Actually after doing that the whole changeset of s390 is even more
straightforward:
---8<---
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..4608cc962ecf 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ /*
+ * Gmap will need the mmap lock again, so retake it. TODO:
+ * only conditionally take the lock when CONFIG_PGSTE set.
+ */
+ mmap_read_lock(mm);
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
---8<---
Thanks,
--
Peter Xu
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 13:49 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 13:49 UTC (permalink / raw)
To: Heiko Carstens
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, linux-mm, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Janosch Frank, Yoshinori Sato, linux-sh,
linux-hexagon, Helge Deller, Alistair Popple, Hugh Dickins,
Russell King, linux-csky, linux-alpha, Ingo Molnar,
linux-arm-kernel, Vineet Gupta, Matt Turner,
Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, linux-xtensa, Johannes Weiner,
linux-um, Nicholas Piggin, Richard Weinberger, linux-m68k,
openrisc, Ivan Kokshaysky, Al Viro, Andy Lutomirski,
Paul Walmsley, Thomas Gleixner, Andrew Morton, Vlastimil Babka,
Richard Henderson, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, linux-parisc, Max Filippov, linux-kernel,
Dinh Nguyen, Palmer Dabbelt, Sven Schnelle, Guo Ren,
Michael Ellerman, Borislav Petkov, Johannes Berg, linuxppc-dev,
David S . Miller
Hi, Heiko,
On Fri, May 27, 2022 at 02:23:42PM +0200, Heiko Carstens wrote:
> On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> > I observed that for each of the shared file-backed page faults, we're very
> > likely to retry one more time for the 1st write fault upon no page. It's
> > because we'll need to release the mmap lock for dirty rate limit purpose
> > with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> >
> > Then after that throttling we return VM_FAULT_RETRY.
> >
> > We did that probably because VM_FAULT_RETRY is the only way we can return
> > to the fault handler at that time telling it we've released the mmap lock.
> >
> > However that's not ideal because it's very likely the fault does not need
> > to be retried at all since the pgtable was well installed before the
> > throttling, so the next continuous fault (including taking mmap read lock,
> > walk the pgtable, etc.) could be in most cases unnecessary.
> >
> > It's not only slowing down page faults for shared file-backed, but also add
> > more mmap lock contention which is in most cases not needed at all.
> >
> > To observe this, one could try to write to some shmem page and look at
> > "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> > shmem write simply because we retried, and vm event "pgfault" will capture
> > that.
> >
> > To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> > show that we've completed the whole fault and released the lock. It's also
> > a hint that we should very possibly not need another fault immediately on
> > this page because we've just completed it.
> >
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
> >
> > I believe it could help more than that.
> >
> > We need some special care on GUP and the s390 pgfault handler (for gmap
> > code before returning from pgfault), the rest changes in the page fault
> > handlers should be relatively straightforward.
> >
> > Another thing to mention is that mm_account_fault() does take this new
> > fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> >
> > I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> > not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> > them as-is.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> ...
> > diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> > index e173b6187ad5..9503a7cfaf03 100644
> > --- a/arch/s390/mm/fault.c
> > +++ b/arch/s390/mm/fault.c
> > @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > unsigned long address;
> > unsigned int flags;
> > vm_fault_t fault;
> > + bool need_unlock = true;
> > bool is_write;
> >
> > tsk = current;
> > @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > goto out_up;
> > goto out;
> > }
> > +
> > + /* The fault is fully completed (including releasing mmap lock) */
> > + if (fault & VM_FAULT_COMPLETED) {
> > + need_unlock = false;
> > + goto out_gmap;
> > + }
> > +
> > if (unlikely(fault & VM_FAULT_ERROR))
> > goto out_up;
> >
> > @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > mmap_read_lock(mm);
> > goto retry;
> > }
> > +out_gmap:
> > if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> > address = __gmap_link(gmap, current->thread.gmap_addr,
> > address);
> > @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > }
> > fault = 0;
> > out_up:
> > - mmap_read_unlock(mm);
> > + if (need_unlock)
> > + mmap_read_unlock(mm);
> > out:
>
> This seems to be incorrect. __gmap_link() requires the mmap_lock to be
> held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the
comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming
gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at
least it'll avoid one pgtable work procedure even if the lock overhead was
kept.
With that, one more possible further optimization for s390 only will be
conditionally not taking that lock when !CONFIG_PGSTE, but I have totally
no idea whether that'll be a common use case, so I plan to leave that for
later in all cases.
Actually after doing that the whole changeset of s390 is even more
straightforward:
---8<---
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..4608cc962ecf 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ /*
+ * Gmap will need the mmap lock again, so retake it. TODO:
+ * only conditionally take the lock when CONFIG_PGSTE set.
+ */
+ mmap_read_lock(mm);
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
---8<---
Thanks,
--
Peter Xu
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 13:49 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 13:49 UTC (permalink / raw)
To: Heiko Carstens
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mips, James E . J . Bottomley, linux-mm,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Janosch Frank, Yoshinori Sato, linux-sh, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-alpha, Ingo Molnar, Geert Uytterhoeven,
linux-arm-kernel, Vineet Gupta, Stafford Horne, Matt Turner,
Christian Borntraeger, Andrea Arcangeli, Albert Ou,
Vasily Gorbik, Brian Cain, linux-xtensa, Johannes Weiner,
linux-um, Nicholas Piggin, Stefan Kristiansson,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
Andrew Morton, Vlastimil Babka, Richard Henderson
e.net>, Chris Zankel <chris@zankel.net>, Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, linux-parisc@vger.kernel.org, Max Filippov <jcmvbkbc@gmail.com>, linux-kernel@vger.kernel.org, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
Hi, Heiko,
On Fri, May 27, 2022 at 02:23:42PM +0200, Heiko Carstens wrote:
> On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> > I observed that for each of the shared file-backed page faults, we're very
> > likely to retry one more time for the 1st write fault upon no page. It's
> > because we'll need to release the mmap lock for dirty rate limit purpose
> > with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> >
> > Then after that throttling we return VM_FAULT_RETRY.
> >
> > We did that probably because VM_FAULT_RETRY is the only way we can return
> > to the fault handler at that time telling it we've released the mmap lock.
> >
> > However that's not ideal because it's very likely the fault does not need
> > to be retried at all since the pgtable was well installed before the
> > throttling, so the next continuous fault (including taking mmap read lock,
> > walk the pgtable, etc.) could be in most cases unnecessary.
> >
> > It's not only slowing down page faults for shared file-backed, but also add
> > more mmap lock contention which is in most cases not needed at all.
> >
> > To observe this, one could try to write to some shmem page and look at
> > "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> > shmem write simply because we retried, and vm event "pgfault" will capture
> > that.
> >
> > To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> > show that we've completed the whole fault and released the lock. It's also
> > a hint that we should very possibly not need another fault immediately on
> > this page because we've just completed it.
> >
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
> >
> > I believe it could help more than that.
> >
> > We need some special care on GUP and the s390 pgfault handler (for gmap
> > code before returning from pgfault), the rest changes in the page fault
> > handlers should be relatively straightforward.
> >
> > Another thing to mention is that mm_account_fault() does take this new
> > fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> >
> > I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> > not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> > them as-is.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> ...
> > diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> > index e173b6187ad5..9503a7cfaf03 100644
> > --- a/arch/s390/mm/fault.c
> > +++ b/arch/s390/mm/fault.c
> > @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > unsigned long address;
> > unsigned int flags;
> > vm_fault_t fault;
> > + bool need_unlock = true;
> > bool is_write;
> >
> > tsk = current;
> > @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > goto out_up;
> > goto out;
> > }
> > +
> > + /* The fault is fully completed (including releasing mmap lock) */
> > + if (fault & VM_FAULT_COMPLETED) {
> > + need_unlock = false;
> > + goto out_gmap;
> > + }
> > +
> > if (unlikely(fault & VM_FAULT_ERROR))
> > goto out_up;
> >
> > @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > mmap_read_lock(mm);
> > goto retry;
> > }
> > +out_gmap:
> > if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> > address = __gmap_link(gmap, current->thread.gmap_addr,
> > address);
> > @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > }
> > fault = 0;
> > out_up:
> > - mmap_read_unlock(mm);
> > + if (need_unlock)
> > + mmap_read_unlock(mm);
> > out:
>
> This seems to be incorrect. __gmap_link() requires the mmap_lock to be
> held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the
comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming
gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at
least it'll avoid one pgtable work procedure even if the lock overhead was
kept.
With that, one more possible further optimization for s390 only will be
conditionally not taking that lock when !CONFIG_PGSTE, but I have totally
no idea whether that'll be a common use case, so I plan to leave that for
later in all cases.
Actually after doing that the whole changeset of s390 is even more
straightforward:
---8<---
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..4608cc962ecf 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ /*
+ * Gmap will need the mmap lock again, so retake it. TODO:
+ * only conditionally take the lock when CONFIG_PGSTE set.
+ */
+ mmap_read_lock(mm);
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
---8<---
Thanks,
--
Peter Xu
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 13:49 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 13:49 UTC (permalink / raw)
To: Heiko Carstens
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Chris Zankel,
Peter Zijlstra, Alistair Popple, linux-csky, linux-hexagon,
Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller,
Janosch Frank
Hi, Heiko,
On Fri, May 27, 2022 at 02:23:42PM +0200, Heiko Carstens wrote:
> On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> > I observed that for each of the shared file-backed page faults, we're very
> > likely to retry one more time for the 1st write fault upon no page. It's
> > because we'll need to release the mmap lock for dirty rate limit purpose
> > with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> >
> > Then after that throttling we return VM_FAULT_RETRY.
> >
> > We did that probably because VM_FAULT_RETRY is the only way we can return
> > to the fault handler at that time telling it we've released the mmap lock.
> >
> > However that's not ideal because it's very likely the fault does not need
> > to be retried at all since the pgtable was well installed before the
> > throttling, so the next continuous fault (including taking mmap read lock,
> > walk the pgtable, etc.) could be in most cases unnecessary.
> >
> > It's not only slowing down page faults for shared file-backed, but also add
> > more mmap lock contention which is in most cases not needed at all.
> >
> > To observe this, one could try to write to some shmem page and look at
> > "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> > shmem write simply because we retried, and vm event "pgfault" will capture
> > that.
> >
> > To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> > show that we've completed the whole fault and released the lock. It's also
> > a hint that we should very possibly not need another fault immediately on
> > this page because we've just completed it.
> >
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
> >
> > I believe it could help more than that.
> >
> > We need some special care on GUP and the s390 pgfault handler (for gmap
> > code before returning from pgfault), the rest changes in the page fault
> > handlers should be relatively straightforward.
> >
> > Another thing to mention is that mm_account_fault() does take this new
> > fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> >
> > I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> > not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> > them as-is.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> ...
> > diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> > index e173b6187ad5..9503a7cfaf03 100644
> > --- a/arch/s390/mm/fault.c
> > +++ b/arch/s390/mm/fault.c
> > @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > unsigned long address;
> > unsigned int flags;
> > vm_fault_t fault;
> > + bool need_unlock = true;
> > bool is_write;
> >
> > tsk = current;
> > @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > goto out_up;
> > goto out;
> > }
> > +
> > + /* The fault is fully completed (including releasing mmap lock) */
> > + if (fault & VM_FAULT_COMPLETED) {
> > + need_unlock = false;
> > + goto out_gmap;
> > + }
> > +
> > if (unlikely(fault & VM_FAULT_ERROR))
> > goto out_up;
> >
> > @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > mmap_read_lock(mm);
> > goto retry;
> > }
> > +out_gmap:
> > if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> > address = __gmap_link(gmap, current->thread.gmap_addr,
> > address);
> > @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > }
> > fault = 0;
> > out_up:
> > - mmap_read_unlock(mm);
> > + if (need_unlock)
> > + mmap_read_unlock(mm);
> > out:
>
> This seems to be incorrect. __gmap_link() requires the mmap_lock to be
> held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the
comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming
gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at
least it'll avoid one pgtable work procedure even if the lock overhead was
kept.
With that, one more possible further optimization for s390 only will be
conditionally not taking that lock when !CONFIG_PGSTE, but I have totally
no idea whether that'll be a common use case, so I plan to leave that for
later in all cases.
Actually after doing that the whole changeset of s390 is even more
straightforward:
---8<---
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..4608cc962ecf 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ /*
+ * Gmap will need the mmap lock again, so retake it. TODO:
+ * only conditionally take the lock when CONFIG_PGSTE set.
+ */
+ mmap_read_lock(mm);
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
---8<---
Thanks,
--
Peter Xu
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 13:49 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 13:49 UTC (permalink / raw)
To: Heiko Carstens
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k
Hi, Heiko,
On Fri, May 27, 2022 at 02:23:42PM +0200, Heiko Carstens wrote:
> On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> > I observed that for each of the shared file-backed page faults, we're very
> > likely to retry one more time for the 1st write fault upon no page. It's
> > because we'll need to release the mmap lock for dirty rate limit purpose
> > with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> >
> > Then after that throttling we return VM_FAULT_RETRY.
> >
> > We did that probably because VM_FAULT_RETRY is the only way we can return
> > to the fault handler at that time telling it we've released the mmap lock.
> >
> > However that's not ideal because it's very likely the fault does not need
> > to be retried at all since the pgtable was well installed before the
> > throttling, so the next continuous fault (including taking mmap read lock,
> > walk the pgtable, etc.) could be in most cases unnecessary.
> >
> > It's not only slowing down page faults for shared file-backed, but also add
> > more mmap lock contention which is in most cases not needed at all.
> >
> > To observe this, one could try to write to some shmem page and look at
> > "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> > shmem write simply because we retried, and vm event "pgfault" will capture
> > that.
> >
> > To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> > show that we've completed the whole fault and released the lock. It's also
> > a hint that we should very possibly not need another fault immediately on
> > this page because we've just completed it.
> >
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
> >
> > I believe it could help more than that.
> >
> > We need some special care on GUP and the s390 pgfault handler (for gmap
> > code before returning from pgfault), the rest changes in the page fault
> > handlers should be relatively straightforward.
> >
> > Another thing to mention is that mm_account_fault() does take this new
> > fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> >
> > I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> > not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> > them as-is.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> ...
> > diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> > index e173b6187ad5..9503a7cfaf03 100644
> > --- a/arch/s390/mm/fault.c
> > +++ b/arch/s390/mm/fault.c
> > @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > unsigned long address;
> > unsigned int flags;
> > vm_fault_t fault;
> > + bool need_unlock = true;
> > bool is_write;
> >
> > tsk = current;
> > @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > goto out_up;
> > goto out;
> > }
> > +
> > + /* The fault is fully completed (including releasing mmap lock) */
> > + if (fault & VM_FAULT_COMPLETED) {
> > + need_unlock = false;
> > + goto out_gmap;
> > + }
> > +
> > if (unlikely(fault & VM_FAULT_ERROR))
> > goto out_up;
> >
> > @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > mmap_read_lock(mm);
> > goto retry;
> > }
> > +out_gmap:
> > if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
> > address = __gmap_link(gmap, current->thread.gmap_addr,
> > address);
> > @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
> > }
> > fault = 0;
> > out_up:
> > - mmap_read_unlock(mm);
> > + if (need_unlock)
> > + mmap_read_unlock(mm);
> > out:
>
> This seems to be incorrect. __gmap_link() requires the mmap_lock to be
> held. Christian, Janosch, or David, could you please check?
Thanks for pointing that out. Indeed I see the clue right above the
comment of __gmap_link():
/*
* ...
* The mmap_lock of the mm that belongs to the address space must be held
* when this function gets called.
*/
int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
A further fact is it'll walk the pgtable right afterwards, assuming
gmap->mm will definitely be the current mm or it'll definitely go wrong.
I'll change s390 to retake the lock with the new COMPLETE retcode, so at
least it'll avoid one pgtable work procedure even if the lock overhead was
kept.
With that, one more possible further optimization for s390 only will be
conditionally not taking that lock when !CONFIG_PGSTE, but I have totally
no idea whether that'll be a common use case, so I plan to leave that for
later in all cases.
Actually after doing that the whole changeset of s390 is even more
straightforward:
---8<---
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index e173b6187ad5..4608cc962ecf 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -433,6 +433,17 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
goto out_up;
goto out;
}
+
+ /* The fault is fully completed (including releasing mmap lock) */
+ if (fault & VM_FAULT_COMPLETED) {
+ /*
+ * Gmap will need the mmap lock again, so retake it. TODO:
+ * only conditionally take the lock when CONFIG_PGSTE set.
+ */
+ mmap_read_lock(mm);
+ goto out_gmap;
+ }
+
if (unlikely(fault & VM_FAULT_ERROR))
goto out_up;
@@ -452,6 +463,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
mmap_read_lock(mm);
goto retry;
}
+out_gmap:
if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
address = __gmap_link(gmap, current->thread.gmap_addr,
address);
---8<---
Thanks,
--
Peter Xu
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
2022-05-27 10:46 ` Ingo Molnar
` (4 preceding siblings ...)
(?)
@ 2022-05-27 14:53 ` Peter Xu
-1 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 14:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Fri, May 27, 2022 at 12:46:31PM +0200, Ingo Molnar wrote:
>
> * Peter Xu <peterx@redhat.com> wrote:
>
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
>
> Nice!
>
> > arch/x86/mm/fault.c | 4 ++++
>
> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>
> Minor comment typo:
>
> > + /*
> > + * We should do the same as VM_FAULT_RETRY, but let's not
> > + * return -EBUSY since that's not reflecting the reality on
> > + * what has happened - we've just fully completed a page
> > + * fault, with the mmap lock released. Use -EAGAIN to show
> > + * that we want to take the mmap lock _again_.
> > + */
>
> s/reflecting the reality on what has happened
> /reflecting the reality of what has happened
Will fix.
>
> > ret = handle_mm_fault(vma, address, fault_flags, NULL);
> > +
> > + if (ret & VM_FAULT_COMPLETED) {
> > + /*
> > + * NOTE: it's a pity that we need to retake the lock here
> > + * to pair with the unlock() in the callers. Ideally we
> > + * could tell the callers so they do not need to unlock.
> > + */
> > + mmap_read_lock(mm);
> > + *unlocked = true;
> > + return 0;
>
> Indeed that's a pity - I guess more performance could be gained here,
> especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side.
Now if we retake the lock for fixup_user_fault() we still safe time for
pgtable walks but the lock overhead will be somehow kept, just with smaller
critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked==NULL
is passed - e.g. the futex code path (fault_in_user_writeable). After all
they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by
case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked==true. We need to teach
them to know "*unlocked==true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
--
Peter Xu
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 14:53 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 14:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Fri, May 27, 2022 at 12:46:31PM +0200, Ingo Molnar wrote:
>
> * Peter Xu <peterx@redhat.com> wrote:
>
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
>
> Nice!
>
> > arch/x86/mm/fault.c | 4 ++++
>
> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>
> Minor comment typo:
>
> > + /*
> > + * We should do the same as VM_FAULT_RETRY, but let's not
> > + * return -EBUSY since that's not reflecting the reality on
> > + * what has happened - we've just fully completed a page
> > + * fault, with the mmap lock released. Use -EAGAIN to show
> > + * that we want to take the mmap lock _again_.
> > + */
>
> s/reflecting the reality on what has happened
> /reflecting the reality of what has happened
Will fix.
>
> > ret = handle_mm_fault(vma, address, fault_flags, NULL);
> > +
> > + if (ret & VM_FAULT_COMPLETED) {
> > + /*
> > + * NOTE: it's a pity that we need to retake the lock here
> > + * to pair with the unlock() in the callers. Ideally we
> > + * could tell the callers so they do not need to unlock.
> > + */
> > + mmap_read_lock(mm);
> > + *unlocked = true;
> > + return 0;
>
> Indeed that's a pity - I guess more performance could be gained here,
> especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side.
Now if we retake the lock for fixup_user_fault() we still safe time for
pgtable walks but the lock overhead will be somehow kept, just with smaller
critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked==NULL
is passed - e.g. the futex code path (fault_in_user_writeable). After all
they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by
case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked==true. We need to teach
them to know "*unlocked==true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
--
Peter Xu
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 14:53 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 14:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Fri, May 27, 2022 at 12:46:31PM +0200, Ingo Molnar wrote:
>
> * Peter Xu <peterx@redhat.com> wrote:
>
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
>
> Nice!
>
> > arch/x86/mm/fault.c | 4 ++++
>
> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>
> Minor comment typo:
>
> > + /*
> > + * We should do the same as VM_FAULT_RETRY, but let's not
> > + * return -EBUSY since that's not reflecting the reality on
> > + * what has happened - we've just fully completed a page
> > + * fault, with the mmap lock released. Use -EAGAIN to show
> > + * that we want to take the mmap lock _again_.
> > + */
>
> s/reflecting the reality on what has happened
> /reflecting the reality of what has happened
Will fix.
>
> > ret = handle_mm_fault(vma, address, fault_flags, NULL);
> > +
> > + if (ret & VM_FAULT_COMPLETED) {
> > + /*
> > + * NOTE: it's a pity that we need to retake the lock here
> > + * to pair with the unlock() in the callers. Ideally we
> > + * could tell the callers so they do not need to unlock.
> > + */
> > + mmap_read_lock(mm);
> > + *unlocked = true;
> > + return 0;
>
> Indeed that's a pity - I guess more performance could be gained here,
> especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side.
Now if we retake the lock for fixup_user_fault() we still safe time for
pgtable walks but the lock overhead will be somehow kept, just with smaller
critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked==NULL
is passed - e.g. the futex code path (fault_in_user_writeable). After all
they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by
case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked==true. We need to teach
them to know "*unlocked==true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
--
Peter Xu
_______________________________________________
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 14:53 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 14:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Benjamin Herrenschmidt, Dave Hansen, linux-mips,
James E . J . Bottomley, linux-mm, Rich Felker, Paul Mackerras,
H . Peter Anvin, sparclinux, linux-ia64, Alexander Gordeev,
Will Deacon, linux-riscv, Anton Ivanov, Jonas Bonn, linux-s390,
linux-snps-arc, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, Alistair Popple, Hugh Dickins, Russell King,
linux-csky, linux-sh, Ingo Molnar, linux-arm-kernel,
Vineet Gupta, Matt Turner, Christian Borntraeger,
Andrea Arcangeli, Albert Ou, Vasily Gorbik, Brian Cain,
Heiko Carstens, Johannes Weiner, linux-um, Nicholas Piggin,
Richard Weinberger, linux-m68k, openrisc, Ivan Kokshaysky,
Al Viro, Andy Lutomirski, Paul Walmsley, Thomas Gleixner,
linux-alpha, Andrew Morton, Vlastimil Babka, Richard Henderson,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, linux-parisc,
Max Filippov, linux-kernel, Dinh Nguyen, Palmer Dabbelt,
Sven Schnelle, Guo Ren, Michael Ellerman, Borislav Petkov,
Johannes Berg, linuxppc-dev, David S . Miller
On Fri, May 27, 2022 at 12:46:31PM +0200, Ingo Molnar wrote:
>
> * Peter Xu <peterx@redhat.com> wrote:
>
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
>
> Nice!
>
> > arch/x86/mm/fault.c | 4 ++++
>
> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>
> Minor comment typo:
>
> > + /*
> > + * We should do the same as VM_FAULT_RETRY, but let's not
> > + * return -EBUSY since that's not reflecting the reality on
> > + * what has happened - we've just fully completed a page
> > + * fault, with the mmap lock released. Use -EAGAIN to show
> > + * that we want to take the mmap lock _again_.
> > + */
>
> s/reflecting the reality on what has happened
> /reflecting the reality of what has happened
Will fix.
>
> > ret = handle_mm_fault(vma, address, fault_flags, NULL);
> > +
> > + if (ret & VM_FAULT_COMPLETED) {
> > + /*
> > + * NOTE: it's a pity that we need to retake the lock here
> > + * to pair with the unlock() in the callers. Ideally we
> > + * could tell the callers so they do not need to unlock.
> > + */
> > + mmap_read_lock(mm);
> > + *unlocked = true;
> > + return 0;
>
> Indeed that's a pity - I guess more performance could be gained here,
> especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side.
Now if we retake the lock for fixup_user_fault() we still safe time for
pgtable walks but the lock overhead will be somehow kept, just with smaller
critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked==NULL
is passed - e.g. the futex code path (fault_in_user_writeable). After all
they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by
case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked==true. We need to teach
them to know "*unlocked==true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
--
Peter Xu
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 14:53 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 14:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: x86, Catalin Marinas, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mips, James E . J . Bottomley, linux-mm,
Rich Felker, Paul Mackerras, H . Peter Anvin, sparclinux,
linux-ia64, Alexander Gordeev, Will Deacon, linux-riscv,
Anton Ivanov, Jonas Bonn, linux-s390, linux-snps-arc,
Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller,
Alistair Popple, Hugh Dickins, Russell King, linux-csky,
linux-sh
et>, Chris Zankel <chris@zankel.net>, Michal Simek <monstr@monstr.eu>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, linux-parisc@vger.kernel.org, Max Filippov <jcmvbkbc@gmail.com>, linux-kernel@vger.kernel.org, Dinh Nguyen <dinguyen@kernel.org>, Palmer Dabbelt <palmer@dabbelt.com>, Sven Schnelle <svens@linux.ibm.com>, Guo Ren <guoren@kernel.org>, Borislav Petkov <bp@alien8.de>, Johannes Berg <johannes@sipsolutions.net>, linuxppc-dev@lists.ozlabs.org, "David S . Miller" <davem@davemloft.net>
Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org
Sender: "Linuxppc-dev" <linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org>
On Fri, May 27, 2022 at 12:46:31PM +0200, Ingo Molnar wrote:
>
> * Peter Xu <peterx@redhat.com> wrote:
>
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
>
> Nice!
>
> > arch/x86/mm/fault.c | 4 ++++
>
> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>
> Minor comment typo:
>
> > + /*
> > + * We should do the same as VM_FAULT_RETRY, but let's not
> > + * return -EBUSY since that's not reflecting the reality on
> > + * what has happened - we've just fully completed a page
> > + * fault, with the mmap lock released. Use -EAGAIN to show
> > + * that we want to take the mmap lock _again_.
> > + */
>
> s/reflecting the reality on what has happened
> /reflecting the reality of what has happened
Will fix.
>
> > ret = handle_mm_fault(vma, address, fault_flags, NULL);
> > +
> > + if (ret & VM_FAULT_COMPLETED) {
> > + /*
> > + * NOTE: it's a pity that we need to retake the lock here
> > + * to pair with the unlock() in the callers. Ideally we
> > + * could tell the callers so they do not need to unlock.
> > + */
> > + mmap_read_lock(mm);
> > + *unlocked = true;
> > + return 0;
>
> Indeed that's a pity - I guess more performance could be gained here,
> especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side.
Now if we retake the lock for fixup_user_fault() we still safe time for
pgtable walks but the lock overhead will be somehow kept, just with smaller
critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked==NULL
is passed - e.g. the futex code path (fault_in_user_writeable). After all
they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by
case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked==true. We need to teach
them to know "*unlocked==true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
--
Peter Xu
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 14:53 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 14:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k, Palmer Dabbelt, Heiko Carstens,
Chris Zankel, Peter Zijlstra, Alistair Popple, linux-csky,
linux-hexagon, Vlastimil Babka, Thomas Gleixner, sparclinux,
Christian Borntraeger, Stafford Horne, Michael Ellerman, x86,
Thomas Bogendoerfer, Paul Mackerras, linux-arm-kernel,
Sven Schnelle, Benjamin Herrenschmidt, linux-xtensa,
Nicholas Piggin, linux-sh, Vasily Gorbik, Borislav Petkov,
linux-mips, Max Filippov, Helge Deller, Vineet Gupta, Al Viro,
Paul Walmsley, Johannes Weiner, Anton Ivanov, Catalin Marinas,
linux-um, linux-alpha, Johannes Berg, linux-ia64,
Geert Uytterhoeven, Dinh Nguyen, Guo Ren, linux-snps-arc,
Hugh Dickins, Rich Felker, Andy Lutomirski, Richard Weinberger,
linuxppc-dev, Brian Cain, Yoshinori Sato, Andrew Morton,
Stefan Kristiansson, linux-parisc, David S . Miller
On Fri, May 27, 2022 at 12:46:31PM +0200, Ingo Molnar wrote:
>
> * Peter Xu <peterx@redhat.com> wrote:
>
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
>
> Nice!
>
> > arch/x86/mm/fault.c | 4 ++++
>
> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>
> Minor comment typo:
>
> > + /*
> > + * We should do the same as VM_FAULT_RETRY, but let's not
> > + * return -EBUSY since that's not reflecting the reality on
> > + * what has happened - we've just fully completed a page
> > + * fault, with the mmap lock released. Use -EAGAIN to show
> > + * that we want to take the mmap lock _again_.
> > + */
>
> s/reflecting the reality on what has happened
> /reflecting the reality of what has happened
Will fix.
>
> > ret = handle_mm_fault(vma, address, fault_flags, NULL);
> > +
> > + if (ret & VM_FAULT_COMPLETED) {
> > + /*
> > + * NOTE: it's a pity that we need to retake the lock here
> > + * to pair with the unlock() in the callers. Ideally we
> > + * could tell the callers so they do not need to unlock.
> > + */
> > + mmap_read_lock(mm);
> > + *unlocked = true;
> > + return 0;
>
> Indeed that's a pity - I guess more performance could be gained here,
> especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side.
Now if we retake the lock for fixup_user_fault() we still safe time for
pgtable walks but the lock overhead will be somehow kept, just with smaller
critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked=NULL
is passed - e.g. the futex code path (fault_in_user_writeable). After all
they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by
case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked=true. We need to teach
them to know "*unlocked=true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
--
Peter Xu
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types
@ 2022-05-27 14:53 ` Peter Xu
0 siblings, 0 replies; 87+ messages in thread
From: Peter Xu @ 2022-05-27 14:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, linux-mm, Richard Henderson, David Hildenbrand,
Matt Turner, Albert Ou, Michal Simek, Russell King,
Ivan Kokshaysky, linux-riscv, Alexander Gordeev, Dave Hansen,
Jonas Bonn, Will Deacon, James E . J . Bottomley,
H . Peter Anvin, Andrea Arcangeli, openrisc, linux-s390,
Ingo Molnar, linux-m68k
On Fri, May 27, 2022 at 12:46:31PM +0200, Ingo Molnar wrote:
>
> * Peter Xu <peterx@redhat.com> wrote:
>
> > This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> > program sequentially dirtying 400MB shmem file being mmap()ed and these are
> > the time it needs:
> >
> > Before: 650.980 ms (+-1.94%)
> > After: 569.396 ms (+-1.38%)
>
> Nice!
>
> > arch/x86/mm/fault.c | 4 ++++
>
> Reviewed-by: Ingo Molnar <mingo@kernel.org>
>
> Minor comment typo:
>
> > + /*
> > + * We should do the same as VM_FAULT_RETRY, but let's not
> > + * return -EBUSY since that's not reflecting the reality on
> > + * what has happened - we've just fully completed a page
> > + * fault, with the mmap lock released. Use -EAGAIN to show
> > + * that we want to take the mmap lock _again_.
> > + */
>
> s/reflecting the reality on what has happened
> /reflecting the reality of what has happened
Will fix.
>
> > ret = handle_mm_fault(vma, address, fault_flags, NULL);
> > +
> > + if (ret & VM_FAULT_COMPLETED) {
> > + /*
> > + * NOTE: it's a pity that we need to retake the lock here
> > + * to pair with the unlock() in the callers. Ideally we
> > + * could tell the callers so they do not need to unlock.
> > + */
> > + mmap_read_lock(mm);
> > + *unlocked = true;
> > + return 0;
>
> Indeed that's a pity - I guess more performance could be gained here,
> especially in highly parallel threaded workloads?
Yes I think so.
The patch avoids the page fault retry, including the mmap lock/unlock side.
Now if we retake the lock for fixup_user_fault() we still safe time for
pgtable walks but the lock overhead will be somehow kept, just with smaller
critical sections.
Some fixup_user_fault() callers won't be affected as long as unlocked==NULL
is passed - e.g. the futex code path (fault_in_user_writeable). After all
they never needed to retake the lock before/after this patch.
It's about the other callers, and they may need some more touch-ups case by
case. Examples are follow_fault_pfn() in vfio and hva_to_pfn_remapped() in
KVM: both of them returns -EAGAIN when *unlocked==true. We need to teach
them to know "*unlocked==true" does not necessarily mean a retry attempt.
I think I can look into them if this patch can be accepted as a follow up.
Thanks for taking a look!
--
Peter Xu
^ permalink raw reply [flat|nested] 87+ messages in thread
end of thread, other threads:[~2022-05-27 22:27 UTC | newest]
Thread overview: 87+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-24 23:45 [PATCH v3] mm: Avoid unnecessary page fault retires on shared memory types Peter Xu
2022-05-24 23:45 ` Peter Xu
2022-05-24 23:45 ` Peter Xu
2022-05-24 23:45 ` Peter Xu
2022-05-24 23:45 ` Peter Xu
2022-05-24 23:45 ` Peter Xu
2022-05-24 23:45 ` Peter Xu
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 8:03 ` Geert Uytterhoeven
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 11:10 ` Peter Zijlstra
2022-05-25 12:44 ` Johannes Weiner
2022-05-25 12:44 ` Johannes Weiner
2022-05-25 12:44 ` Johannes Weiner
2022-05-25 12:44 ` Johannes Weiner
2022-05-25 12:44 ` Johannes Weiner
2022-05-25 12:44 ` Johannes Weiner
2022-05-25 12:44 ` Johannes Weiner
2022-05-26 3:40 ` Vineet Gupta
2022-05-26 3:40 ` Vineet Gupta
2022-05-26 3:40 ` Vineet Gupta
2022-05-26 3:40 ` Vineet Gupta
2022-05-26 3:40 ` Vineet Gupta
2022-05-26 3:40 ` Vineet Gupta
2022-05-26 3:40 ` Vineet Gupta
2022-05-27 2:54 ` Guo Ren
2022-05-27 2:54 ` Guo Ren
2022-05-27 2:54 ` Guo Ren
2022-05-27 2:54 ` Guo Ren
2022-05-27 2:54 ` Guo Ren
2022-05-27 2:54 ` Guo Ren
2022-05-27 2:54 ` Guo Ren
2022-05-27 5:39 ` Max Filippov
2022-05-27 5:39 ` Max Filippov
2022-05-27 5:39 ` Max Filippov
2022-05-27 5:39 ` Max Filippov
2022-05-27 5:39 ` Max Filippov
2022-05-27 5:39 ` Max Filippov
2022-05-27 5:39 ` Max Filippov
2022-05-27 8:21 ` Alistair Popple
2022-05-27 8:21 ` Alistair Popple
2022-05-27 8:21 ` Alistair Popple
2022-05-27 8:21 ` Alistair Popple
2022-05-27 8:21 ` Alistair Popple
2022-05-27 8:21 ` Alistair Popple
2022-05-27 8:21 ` Alistair Popple
2022-05-27 8:21 ` Alistair Popple
2022-05-27 10:46 ` Ingo Molnar
2022-05-27 10:46 ` Ingo Molnar
2022-05-27 10:46 ` Ingo Molnar
2022-05-27 10:46 ` Ingo Molnar
2022-05-27 10:46 ` Ingo Molnar
2022-05-27 10:46 ` Ingo Molnar
2022-05-27 10:46 ` Ingo Molnar
2022-05-27 14:53 ` Peter Xu
2022-05-27 14:53 ` Peter Xu
2022-05-27 14:53 ` Peter Xu
2022-05-27 14:53 ` Peter Xu
2022-05-27 14:53 ` Peter Xu
2022-05-27 14:53 ` Peter Xu
2022-05-27 14:53 ` Peter Xu
2022-05-27 12:23 ` Heiko Carstens
2022-05-27 12:23 ` Heiko Carstens
2022-05-27 12:23 ` Heiko Carstens
2022-05-27 12:23 ` Heiko Carstens
2022-05-27 12:23 ` Heiko Carstens
2022-05-27 12:23 ` Heiko Carstens
2022-05-27 12:23 ` Heiko Carstens
2022-05-27 13:49 ` Peter Xu
2022-05-27 13:49 ` Peter Xu
2022-05-27 13:49 ` Peter Xu
2022-05-27 13:49 ` Peter Xu
2022-05-27 13:49 ` Peter Xu
2022-05-27 13:49 ` Peter Xu
2022-05-27 13:49 ` Peter Xu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.