From: Peter Zijlstra <peterz@infradead.org>
To: torvalds@linux-foundation.org, paulmck@linux.vnet.ibm.com,
tglx@linutronix.de, akpm@linux-foundation.org, riel@redhat.com,
mgorman@suse.de, oleg@redhat.com, mingo@redhat.com,
minchan@kernel.org, kamezawa.hiroyu@jp.fujitsu.com,
viro@zeniv.linux.org.uk, laijs@cn.fujitsu.com, dave@stgolabs.net
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
"Peter Zijlstra" <peterz@infradead.org>
Subject: [RFC][PATCH 6/6] mm,x86: Add speculative pagefault handling
Date: Mon, 20 Oct 2014 23:56:39 +0200 [thread overview]
Message-ID: <20141020222841.562595816@infradead.org> (raw)
In-Reply-To: 20141020215633.717315139@infradead.org
[-- Attachment #1: peterz-mm-x86.patch --]
[-- Type: text/plain, Size: 3071 bytes --]
Try a speculative fault before acquiring mmap_sem, if it returns with
VM_FAULT_RETRY continue with the mmap_sem acquisition and do the
traditional fault.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/x86/mm/fault.c | 35 ++++++++++++++++++++++-------------
1 file changed, 22 insertions(+), 13 deletions(-)
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -844,11 +844,8 @@ do_sigbus(struct pt_regs *regs, unsigned
unsigned int fault)
{
struct task_struct *tsk = current;
- struct mm_struct *mm = tsk->mm;
int code = BUS_ADRERR;
- up_read(&mm->mmap_sem);
-
/* Kernel mode? Handle exceptions or die: */
if (!(error_code & PF_USER)) {
no_context(regs, error_code, address, SIGBUS, BUS_ADRERR);
@@ -879,7 +876,6 @@ mm_fault_error(struct pt_regs *regs, uns
unsigned long address, unsigned int fault)
{
if (fatal_signal_pending(current) && !(error_code & PF_USER)) {
- up_read(¤t->mm->mmap_sem);
no_context(regs, error_code, address, 0, 0);
return;
}
@@ -887,14 +883,11 @@ mm_fault_error(struct pt_regs *regs, uns
if (fault & VM_FAULT_OOM) {
/* Kernel mode? Handle exceptions or die: */
if (!(error_code & PF_USER)) {
- up_read(¤t->mm->mmap_sem);
no_context(regs, error_code, address,
SIGSEGV, SEGV_MAPERR);
return;
}
- up_read(¤t->mm->mmap_sem);
-
/*
* We ran out of memory, call the OOM killer, and return the
* userspace (which will retry the fault, or kill us if we got
@@ -1141,6 +1134,16 @@ __do_page_fault(struct pt_regs *regs, un
if (error_code & PF_WRITE)
flags |= FAULT_FLAG_WRITE;
+ if (error_code & PF_USER) {
+ fault = handle_speculative_fault(mm, address,
+ flags & ~FAULT_FLAG_ALLOW_RETRY);
+
+ if (fault & VM_FAULT_RETRY)
+ goto retry;
+
+ goto done;
+ }
+
/*
* When running in the kernel we expect faults to occur only to
* addresses in user space. All other faults represent errors in
@@ -1225,9 +1228,15 @@ __do_page_fault(struct pt_regs *regs, un
* signal first. We do not need to release the mmap_sem because it
* would already be released in __lock_page_or_retry in mm/filemap.c.
*/
- if (unlikely((fault & VM_FAULT_RETRY) && fatal_signal_pending(current)))
- return;
+ if (unlikely(fault & VM_FAULT_RETRY)) {
+ if (fatal_signal_pending(current))
+ return;
+
+ goto done;
+ }
+ up_read(&mm->mmap_sem);
+done:
if (unlikely(fault & VM_FAULT_ERROR)) {
mm_fault_error(regs, error_code, address, fault);
return;
@@ -1249,8 +1258,10 @@ __do_page_fault(struct pt_regs *regs, un
regs, address);
}
if (fault & VM_FAULT_RETRY) {
- /* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
- * of starvation. */
+ /*
+ * Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk of
+ * starvation.
+ */
flags &= ~FAULT_FLAG_ALLOW_RETRY;
flags |= FAULT_FLAG_TRIED;
goto retry;
@@ -1258,8 +1269,6 @@ __do_page_fault(struct pt_regs *regs, un
}
check_v8086_mode(regs, address, tsk);
-
- up_read(&mm->mmap_sem);
}
NOKPROBE_SYMBOL(__do_page_fault);
next prev parent reply other threads:[~2014-10-20 22:41 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-10-20 21:56 [RFC][PATCH 0/6] Another go at speculative page faults Peter Zijlstra
2014-10-20 21:56 ` [RFC][PATCH 1/6] mm: Dont assume page-table invariance during faults Peter Zijlstra
2014-10-20 21:56 ` [RFC][PATCH 2/6] mm: Prepare for FAULT_FLAG_SPECULATIVE Peter Zijlstra
2014-10-20 21:56 ` [RFC][PATCH 3/6] mm: VMA sequence count Peter Zijlstra
2014-10-22 11:26 ` Kirill A. Shutemov
2014-10-22 11:39 ` Peter Zijlstra
2014-10-22 11:53 ` Kirill A. Shutemov
2014-10-22 12:15 ` Peter Zijlstra
2014-10-22 13:44 ` Peter Zijlstra
2014-10-23 12:36 ` Kirill A. Shutemov
2014-10-23 14:22 ` Peter Zijlstra
2014-10-23 15:05 ` Kirill A. Shutemov
2014-10-20 21:56 ` [RFC][PATCH 4/6] SRCU free VMAs Peter Zijlstra
2014-10-20 23:41 ` Linus Torvalds
2014-10-21 8:07 ` Peter Zijlstra
2014-10-24 15:16 ` Christoph Lameter
2014-10-24 15:51 ` Peter Zijlstra
2014-10-24 17:08 ` Christoph Lameter
2014-10-21 8:22 ` Peter Zijlstra
2014-10-23 10:14 ` Lai Jiangshan
2014-10-23 11:03 ` Peter Zijlstra
2014-10-24 3:33 ` Lai Jiangshan
2014-10-24 7:26 ` Peter Zijlstra
2014-10-20 21:56 ` [RFC][PATCH 5/6] mm: Provide speculative fault infrastructure Peter Zijlstra
2014-10-21 8:35 ` Kirill A. Shutemov
2014-10-21 10:41 ` Peter Zijlstra
2014-10-21 19:00 ` Peter Zijlstra
2014-10-20 21:56 ` Peter Zijlstra [this message]
2014-10-21 0:07 ` [RFC][PATCH 0/6] Another go at speculative page faults Andy Lutomirski
2014-10-21 8:11 ` Peter Zijlstra
2014-10-21 16:23 ` Ingo Molnar
2014-10-21 17:09 ` Kirill A. Shutemov
2014-10-21 17:56 ` Peter Zijlstra
2014-10-23 10:40 ` Lai Jiangshan
2014-10-23 11:04 ` Peter Zijlstra
2014-10-24 7:54 ` Ingo Molnar
2014-10-24 13:14 ` Peter Zijlstra
2014-10-28 5:32 ` Namhyung Kim
2014-10-21 17:25 ` Peter Zijlstra
2014-10-22 12:35 ` Ingo Molnar
2014-10-22 7:34 ` Davidlohr Bueso
2014-10-22 11:29 ` Kirill A. Shutemov
2014-10-22 11:45 ` Peter Zijlstra
2014-10-22 11:55 ` Kirill A. Shutemov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141020222841.562595816@infradead.org \
--to=peterz@infradead.org \
--cc=akpm@linux-foundation.org \
--cc=dave@stgolabs.net \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=laijs@cn.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=minchan@kernel.org \
--cc=mingo@redhat.com \
--cc=oleg@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=riel@redhat.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).