From: Thomas Gleixner <tglx@linutronix.de>
To: LKML <linux-kernel@vger.kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>,
Masami Hiramatsu <mhiramat@kernel.org>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Subject: [patch 5/5] sched: Move mmdrop to RCU on RT
Date: Tue, 28 Sep 2021 14:24:32 +0200 (CEST) [thread overview]
Message-ID: <20210928122411.648582026@linutronix.de> (raw)
In-Reply-To: 20210928122339.502270600@linutronix.de
mmdrop() is invoked from finish_task_switch() by the incoming task to drop
the mm which was handed over by the previous task. mmdrop() can be quite
expensive which prevents an incoming real-time task from getting useful
work done.
Provide mmdrop_sched() which maps to mmdrop() on !RT kernels. On RT kernels
it delagates the eventually required invocation of __mmdrop() to RCU.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
include/linux/mm_types.h | 4 ++++
include/linux/sched/mm.h | 20 ++++++++++++++++++++
kernel/fork.c | 13 +++++++++++++
kernel/sched/core.c | 2 +-
4 files changed, 38 insertions(+), 1 deletion(-)
---
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -12,6 +12,7 @@
#include <linux/completion.h>
#include <linux/cpumask.h>
#include <linux/uprobes.h>
+#include <linux/rcupdate.h>
#include <linux/page-flags-layout.h>
#include <linux/workqueue.h>
#include <linux/seqlock.h>
@@ -572,6 +573,9 @@ struct mm_struct {
bool tlb_flush_batched;
#endif
struct uprobes_state uprobes_state;
+#ifdef CONFIG_PREEMPT_RT
+ struct rcu_head delayed_drop;
+#endif
#ifdef CONFIG_HUGETLB_PAGE
atomic_long_t hugetlb_usage;
#endif
--- a/include/linux/sched/mm.h
+++ b/include/linux/sched/mm.h
@@ -49,6 +49,26 @@ static inline void mmdrop(struct mm_stru
__mmdrop(mm);
}
+#ifdef CONFIG_PREEMPT_RT
+extern void __mmdrop_delayed(struct rcu_head *rhp);
+
+/*
+ * Invoked from finish_task_switch(). Delegates the heavy lifting on RT
+ * kernels via RCU.
+ */
+static inline void mmdrop_sched(struct mm_struct *mm)
+{
+ /* Provides a full memory barrier. See mmdrop() */
+ if (atomic_dec_and_test(&mm->mm_count))
+ call_rcu(&mm->delayed_drop, __mmdrop_delayed);
+}
+#else
+static inline void mmdrop_sched(struct mm_struct *mm)
+{
+ mmdrop(mm);
+}
+#endif
+
/**
* mmget() - Pin the address space associated with a &struct mm_struct.
* @mm: The address space to pin.
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -708,6 +708,19 @@ void __mmdrop(struct mm_struct *mm)
}
EXPORT_SYMBOL_GPL(__mmdrop);
+#ifdef CONFIG_PREEMPT_RT
+/*
+ * RCU callback for delayed mm drop. Not strictly RCU, but call_rcu() is
+ * by far the least expensive way to do that.
+ */
+void __mmdrop_delayed(struct rcu_head *rhp)
+{
+ struct mm_struct *mm = container_of(rhp, struct mm_struct, delayed_drop);
+
+ __mmdrop(mm);
+}
+#endif
+
static void mmdrop_async_fn(struct work_struct *work)
{
struct mm_struct *mm;
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4840,7 +4840,7 @@ static struct rq *finish_task_switch(str
*/
if (mm) {
membarrier_mm_sync_core_before_usermode(mm);
- mmdrop(mm);
+ mmdrop_sched(mm);
}
if (unlikely(prev_state == TASK_DEAD)) {
if (prev->sched_class->task_dead)
next prev parent reply other threads:[~2021-09-28 12:24 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-28 12:24 [patch 0/5] sched: Miscellaneous RT related tweaks Thomas Gleixner
2021-09-28 12:24 ` [patch 1/5] sched: Limit the number of task migrations per batch on RT Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
2021-09-28 12:24 ` [patch 2/5] sched: Disable TTWU_QUEUE " Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
2021-09-28 12:24 ` [patch 3/5] sched: Move kprobes cleanup out of finish_task_switch() Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
2021-09-28 12:24 ` [patch 4/5] sched: Delay task stack freeing on RT Thomas Gleixner
2021-09-29 11:54 ` Peter Zijlstra
2021-10-01 16:12 ` Andy Lutomirski
2021-10-01 17:24 ` Thomas Gleixner
2021-10-01 18:48 ` Andy Lutomirski
2021-10-01 19:02 ` Andy Lutomirski
2021-10-01 20:54 ` Thomas Gleixner
2021-09-28 12:24 ` Thomas Gleixner [this message]
2021-09-29 12:02 ` [patch 5/5] sched: Move mmdrop to RCU " Peter Zijlstra
2021-09-29 13:05 ` Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210928122411.648582026@linutronix.de \
--to=tglx@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.