* [PATCH v3 1/3] rseq: optimize rseq_update_cpu_id()
2021-04-13 20:33 [PATCH v3 0/3] rseq: minor optimizations Eric Dumazet
@ 2021-04-13 20:33 ` Eric Dumazet
2021-04-15 8:37 ` [tip: sched/core] rseq: Optimize rseq_update_cpu_id() tip-bot2 for Eric Dumazet
2021-04-13 20:33 ` [PATCH v3 2/3] rseq: remove redundant access_ok() Eric Dumazet
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2021-04-13 20:33 UTC (permalink / raw)
To: Ingo Molnar, Mathieu Desnoyers, Peter Zijlstra
Cc: Paul E . McKenney, Boqun Feng, Arjun Roy, linux-kernel,
Eric Dumazet, Eric Dumazet
From: Eric Dumazet <edumazet@google.com>
Two put_user() in rseq_update_cpu_id() are replaced
by a pair of unsafe_put_user() with appropriate surroundings.
This removes one stac/clac pair on x86 in fast path.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Arjun Roy <arjunroy@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
kernel/rseq.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/kernel/rseq.c b/kernel/rseq.c
index a4f86a9d6937cdfa2f13d1dcc9be863c1943d06f..f020f18f512a3f6241c3c9b104ce50e4d2c6188c 100644
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -84,13 +84,20 @@
static int rseq_update_cpu_id(struct task_struct *t)
{
u32 cpu_id = raw_smp_processor_id();
+ struct rseq __user *rseq = t->rseq;
- if (put_user(cpu_id, &t->rseq->cpu_id_start))
- return -EFAULT;
- if (put_user(cpu_id, &t->rseq->cpu_id))
- return -EFAULT;
+ if (!user_write_access_begin(rseq, sizeof(*rseq)))
+ goto efault;
+ unsafe_put_user(cpu_id, &rseq->cpu_id_start, efault_end);
+ unsafe_put_user(cpu_id, &rseq->cpu_id, efault_end);
+ user_write_access_end();
trace_rseq_update(t);
return 0;
+
+efault_end:
+ user_write_access_end();
+efault:
+ return -EFAULT;
}
static int rseq_reset_rseq_cpu_id(struct task_struct *t)
--
2.31.1.295.g9ea45b61b8-goog
^ permalink raw reply related [flat|nested] 9+ messages in thread* [tip: sched/core] rseq: Optimize rseq_update_cpu_id()
2021-04-13 20:33 ` [PATCH v3 1/3] rseq: optimize rseq_update_cpu_id() Eric Dumazet
@ 2021-04-15 8:37 ` tip-bot2 for Eric Dumazet
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot2 for Eric Dumazet @ 2021-04-15 8:37 UTC (permalink / raw)
To: linux-tip-commits
Cc: Eric Dumazet, Peter Zijlstra (Intel),
Mathieu Desnoyers, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 60af388d23889636011488c42763876bcdda3eab
Gitweb: https://git.kernel.org/tip/60af388d23889636011488c42763876bcdda3eab
Author: Eric Dumazet <edumazet@google.com>
AuthorDate: Tue, 13 Apr 2021 13:33:50 -07:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Wed, 14 Apr 2021 18:04:09 +02:00
rseq: Optimize rseq_update_cpu_id()
Two put_user() in rseq_update_cpu_id() are replaced
by a pair of unsafe_put_user() with appropriate surroundings.
This removes one stac/clac pair on x86 in fast path.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://lkml.kernel.org/r/20210413203352.71350-2-eric.dumazet@gmail.com
---
kernel/rseq.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/kernel/rseq.c b/kernel/rseq.c
index a4f86a9..f020f18 100644
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -84,13 +84,20 @@
static int rseq_update_cpu_id(struct task_struct *t)
{
u32 cpu_id = raw_smp_processor_id();
+ struct rseq __user *rseq = t->rseq;
- if (put_user(cpu_id, &t->rseq->cpu_id_start))
- return -EFAULT;
- if (put_user(cpu_id, &t->rseq->cpu_id))
- return -EFAULT;
+ if (!user_write_access_begin(rseq, sizeof(*rseq)))
+ goto efault;
+ unsafe_put_user(cpu_id, &rseq->cpu_id_start, efault_end);
+ unsafe_put_user(cpu_id, &rseq->cpu_id, efault_end);
+ user_write_access_end();
trace_rseq_update(t);
return 0;
+
+efault_end:
+ user_write_access_end();
+efault:
+ return -EFAULT;
}
static int rseq_reset_rseq_cpu_id(struct task_struct *t)
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 2/3] rseq: remove redundant access_ok()
2021-04-13 20:33 [PATCH v3 0/3] rseq: minor optimizations Eric Dumazet
2021-04-13 20:33 ` [PATCH v3 1/3] rseq: optimize rseq_update_cpu_id() Eric Dumazet
@ 2021-04-13 20:33 ` Eric Dumazet
2021-04-15 8:37 ` [tip: sched/core] rseq: Remove " tip-bot2 for Eric Dumazet
2021-04-13 20:33 ` [PATCH v3 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs() Eric Dumazet
2021-04-13 20:40 ` [PATCH v3 0/3] rseq: minor optimizations Mathieu Desnoyers
3 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2021-04-13 20:33 UTC (permalink / raw)
To: Ingo Molnar, Mathieu Desnoyers, Peter Zijlstra
Cc: Paul E . McKenney, Boqun Feng, Arjun Roy, linux-kernel,
Eric Dumazet, Eric Dumazet
From: Eric Dumazet <edumazet@google.com>
After commit 8f2817701492 ("rseq: Use get_user/put_user rather
than __get_user/__put_user") we no longer need
an access_ok() call from __rseq_handle_notify_resume()
Mathieu pointed out the same cleanup can be done
in rseq_syscall().
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Arjun Roy <arjunroy@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
kernel/rseq.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/kernel/rseq.c b/kernel/rseq.c
index f020f18f512a3f6241c3c9b104ce50e4d2c6188c..cfe01ab5253c1c424c0e8b25acbb6a8e1b41a5b6 100644
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)
if (unlikely(t->flags & PF_EXITING))
return;
- if (unlikely(!access_ok(t->rseq, sizeof(*t->rseq))))
- goto error;
ret = rseq_ip_fixup(regs);
if (unlikely(ret < 0))
goto error;
@@ -301,8 +299,7 @@ void rseq_syscall(struct pt_regs *regs)
if (!t->rseq)
return;
- if (!access_ok(t->rseq, sizeof(*t->rseq)) ||
- rseq_get_rseq_cs(t, &rseq_cs) || in_rseq_cs(ip, &rseq_cs))
+ if (rseq_get_rseq_cs(t, &rseq_cs) || in_rseq_cs(ip, &rseq_cs))
force_sig(SIGSEGV);
}
--
2.31.1.295.g9ea45b61b8-goog
^ permalink raw reply related [flat|nested] 9+ messages in thread* [tip: sched/core] rseq: Remove redundant access_ok()
2021-04-13 20:33 ` [PATCH v3 2/3] rseq: remove redundant access_ok() Eric Dumazet
@ 2021-04-15 8:37 ` tip-bot2 for Eric Dumazet
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot2 for Eric Dumazet @ 2021-04-15 8:37 UTC (permalink / raw)
To: linux-tip-commits
Cc: Eric Dumazet, Peter Zijlstra (Intel),
Mathieu Desnoyers, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 0ed96051531ecc6965f6456d25b19b9b6bdb5c28
Gitweb: https://git.kernel.org/tip/0ed96051531ecc6965f6456d25b19b9b6bdb5c28
Author: Eric Dumazet <edumazet@google.com>
AuthorDate: Tue, 13 Apr 2021 13:33:51 -07:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Wed, 14 Apr 2021 18:04:09 +02:00
rseq: Remove redundant access_ok()
After commit 8f2817701492 ("rseq: Use get_user/put_user rather
than __get_user/__put_user") we no longer need
an access_ok() call from __rseq_handle_notify_resume()
Mathieu pointed out the same cleanup can be done
in rseq_syscall().
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://lkml.kernel.org/r/20210413203352.71350-3-eric.dumazet@gmail.com
---
kernel/rseq.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/kernel/rseq.c b/kernel/rseq.c
index f020f18..cfe01ab 100644
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)
if (unlikely(t->flags & PF_EXITING))
return;
- if (unlikely(!access_ok(t->rseq, sizeof(*t->rseq))))
- goto error;
ret = rseq_ip_fixup(regs);
if (unlikely(ret < 0))
goto error;
@@ -301,8 +299,7 @@ void rseq_syscall(struct pt_regs *regs)
if (!t->rseq)
return;
- if (!access_ok(t->rseq, sizeof(*t->rseq)) ||
- rseq_get_rseq_cs(t, &rseq_cs) || in_rseq_cs(ip, &rseq_cs))
+ if (rseq_get_rseq_cs(t, &rseq_cs) || in_rseq_cs(ip, &rseq_cs))
force_sig(SIGSEGV);
}
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()
2021-04-13 20:33 [PATCH v3 0/3] rseq: minor optimizations Eric Dumazet
2021-04-13 20:33 ` [PATCH v3 1/3] rseq: optimize rseq_update_cpu_id() Eric Dumazet
2021-04-13 20:33 ` [PATCH v3 2/3] rseq: remove redundant access_ok() Eric Dumazet
@ 2021-04-13 20:33 ` Eric Dumazet
2021-04-15 8:37 ` [tip: sched/core] rseq: Optimise " tip-bot2 for Eric Dumazet
2021-04-13 20:40 ` [PATCH v3 0/3] rseq: minor optimizations Mathieu Desnoyers
3 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2021-04-13 20:33 UTC (permalink / raw)
To: Ingo Molnar, Mathieu Desnoyers, Peter Zijlstra
Cc: Paul E . McKenney, Boqun Feng, Arjun Roy, linux-kernel,
Eric Dumazet, Eric Dumazet
From: Eric Dumazet <edumazet@google.com>
Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union,
update includes") added regressions for our servers.
Using copy_from_user() and clear_user() for 64bit values
is suboptimal.
We can use faster put_user() and get_user() on 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Arjun Roy <arjunroy@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
kernel/rseq.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/kernel/rseq.c b/kernel/rseq.c
index cfe01ab5253c1c424c0e8b25acbb6a8e1b41a5b6..35f7bd0fced0e2dd8aed819e054dac03f024388a 100644
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -127,8 +127,13 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)
u32 sig;
int ret;
+#ifdef CONFIG_64BIT
+ if (get_user(ptr, &t->rseq->rseq_cs.ptr64))
+ return -EFAULT;
+#else
if (copy_from_user(&ptr, &t->rseq->rseq_cs.ptr64, sizeof(ptr)))
return -EFAULT;
+#endif
if (!ptr) {
memset(rseq_cs, 0, sizeof(*rseq_cs));
return 0;
@@ -211,9 +216,13 @@ static int clear_rseq_cs(struct task_struct *t)
*
* Set rseq_cs to NULL.
*/
+#ifdef CONFIG_64BIT
+ return put_user(0UL, &t->rseq->rseq_cs.ptr64);
+#else
if (clear_user(&t->rseq->rseq_cs.ptr64, sizeof(t->rseq->rseq_cs.ptr64)))
return -EFAULT;
return 0;
+#endif
}
/*
--
2.31.1.295.g9ea45b61b8-goog
^ permalink raw reply related [flat|nested] 9+ messages in thread* [tip: sched/core] rseq: Optimise rseq_get_rseq_cs() and clear_rseq_cs()
2021-04-13 20:33 ` [PATCH v3 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs() Eric Dumazet
@ 2021-04-15 8:37 ` tip-bot2 for Eric Dumazet
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot2 for Eric Dumazet @ 2021-04-15 8:37 UTC (permalink / raw)
To: linux-tip-commits
Cc: Eric Dumazet, Peter Zijlstra (Intel),
Mathieu Desnoyers, x86, linux-kernel
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5e0ccd4a3b01c5a71732a13186ca110a138516ea
Gitweb: https://git.kernel.org/tip/5e0ccd4a3b01c5a71732a13186ca110a138516ea
Author: Eric Dumazet <edumazet@google.com>
AuthorDate: Tue, 13 Apr 2021 13:33:52 -07:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Wed, 14 Apr 2021 18:04:09 +02:00
rseq: Optimise rseq_get_rseq_cs() and clear_rseq_cs()
Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union,
update includes") added regressions for our servers.
Using copy_from_user() and clear_user() for 64bit values
is suboptimal.
We can use faster put_user() and get_user() on 64bit arches.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://lkml.kernel.org/r/20210413203352.71350-4-eric.dumazet@gmail.com
---
kernel/rseq.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/kernel/rseq.c b/kernel/rseq.c
index cfe01ab..35f7bd0 100644
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -127,8 +127,13 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)
u32 sig;
int ret;
+#ifdef CONFIG_64BIT
+ if (get_user(ptr, &t->rseq->rseq_cs.ptr64))
+ return -EFAULT;
+#else
if (copy_from_user(&ptr, &t->rseq->rseq_cs.ptr64, sizeof(ptr)))
return -EFAULT;
+#endif
if (!ptr) {
memset(rseq_cs, 0, sizeof(*rseq_cs));
return 0;
@@ -211,9 +216,13 @@ static int clear_rseq_cs(struct task_struct *t)
*
* Set rseq_cs to NULL.
*/
+#ifdef CONFIG_64BIT
+ return put_user(0UL, &t->rseq->rseq_cs.ptr64);
+#else
if (clear_user(&t->rseq->rseq_cs.ptr64, sizeof(t->rseq->rseq_cs.ptr64)))
return -EFAULT;
return 0;
+#endif
}
/*
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v3 0/3] rseq: minor optimizations
2021-04-13 20:33 [PATCH v3 0/3] rseq: minor optimizations Eric Dumazet
` (2 preceding siblings ...)
2021-04-13 20:33 ` [PATCH v3 3/3] rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs() Eric Dumazet
@ 2021-04-13 20:40 ` Mathieu Desnoyers
2021-04-14 7:51 ` Peter Zijlstra
3 siblings, 1 reply; 9+ messages in thread
From: Mathieu Desnoyers @ 2021-04-13 20:40 UTC (permalink / raw)
To: Eric Dumazet
Cc: Ingo Molnar, Peter Zijlstra, paulmck, Boqun Feng, Arjun Roy,
linux-kernel, Eric Dumazet
----- On Apr 13, 2021, at 4:33 PM, Eric Dumazet eric.dumazet@gmail.com wrote:
> From: Eric Dumazet <edumazet@google.com>
>
> rseq is a heavy user of copy to/from user data in fast paths.
> This series tries to reduce the cost.
>
> v3: Third patch going back to v1 (only deal with 64bit arches)
> v2: Addressed Peter and Mathieu feedbacks, thanks !
For the whole series:
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Thanks Eric!
Mathieu
>
> Eric Dumazet (3):
> rseq: optimize rseq_update_cpu_id()
> rseq: remove redundant access_ok()
> rseq: optimise rseq_get_rseq_cs() and clear_rseq_cs()
>
> kernel/rseq.c | 29 +++++++++++++++++++++--------
> 1 file changed, 21 insertions(+), 8 deletions(-)
>
> --
> 2.31.1.295.g9ea45b61b8-goog
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v3 0/3] rseq: minor optimizations
2021-04-13 20:40 ` [PATCH v3 0/3] rseq: minor optimizations Mathieu Desnoyers
@ 2021-04-14 7:51 ` Peter Zijlstra
0 siblings, 0 replies; 9+ messages in thread
From: Peter Zijlstra @ 2021-04-14 7:51 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Eric Dumazet, Ingo Molnar, paulmck, Boqun Feng, Arjun Roy,
linux-kernel, Eric Dumazet
On Tue, Apr 13, 2021 at 04:40:33PM -0400, Mathieu Desnoyers wrote:
> ----- On Apr 13, 2021, at 4:33 PM, Eric Dumazet eric.dumazet@gmail.com wrote:
>
> > From: Eric Dumazet <edumazet@google.com>
> >
> > rseq is a heavy user of copy to/from user data in fast paths.
> > This series tries to reduce the cost.
> >
> > v3: Third patch going back to v1 (only deal with 64bit arches)
> > v2: Addressed Peter and Mathieu feedbacks, thanks !
>
> For the whole series:
>
> Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Thanks!
^ permalink raw reply [flat|nested] 9+ messages in thread