From: Thomas Gleixner <tglx@linutronix.de>
To: LKML <linux-kernel@vger.kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>,
Masami Hiramatsu <mhiramat@kernel.org>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Subject: [patch 5/5] sched: Move mmdrop to RCU on RT
Date: Tue, 28 Sep 2021 14:24:32 +0200 (CEST) [thread overview]
Message-ID: <20210928122411.648582026@linutronix.de> (raw)
In-Reply-To: 20210928122339.502270600@linutronix.de
mmdrop() is invoked from finish_task_switch() by the incoming task to drop
the mm which was handed over by the previous task. mmdrop() can be quite
expensive which prevents an incoming real-time task from getting useful
work done.
Provide mmdrop_sched() which maps to mmdrop() on !RT kernels. On RT kernels
it delagates the eventually required invocation of __mmdrop() to RCU.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
include/linux/mm_types.h | 4 ++++
include/linux/sched/mm.h | 20 ++++++++++++++++++++
kernel/fork.c | 13 +++++++++++++
kernel/sched/core.c | 2 +-
4 files changed, 38 insertions(+), 1 deletion(-)
---
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -12,6 +12,7 @@
#include <linux/completion.h>
#include <linux/cpumask.h>
#include <linux/uprobes.h>
+#include <linux/rcupdate.h>
#include <linux/page-flags-layout.h>
#include <linux/workqueue.h>
#include <linux/seqlock.h>
@@ -572,6 +573,9 @@ struct mm_struct {
bool tlb_flush_batched;
#endif
struct uprobes_state uprobes_state;
+#ifdef CONFIG_PREEMPT_RT
+ struct rcu_head delayed_drop;
+#endif
#ifdef CONFIG_HUGETLB_PAGE
atomic_long_t hugetlb_usage;
#endif
--- a/include/linux/sched/mm.h
+++ b/include/linux/sched/mm.h
@@ -49,6 +49,26 @@ static inline void mmdrop(struct mm_stru
__mmdrop(mm);
}
+#ifdef CONFIG_PREEMPT_RT
+extern void __mmdrop_delayed(struct rcu_head *rhp);
+
+/*
+ * Invoked from finish_task_switch(). Delegates the heavy lifting on RT
+ * kernels via RCU.
+ */
+static inline void mmdrop_sched(struct mm_struct *mm)
+{
+ /* Provides a full memory barrier. See mmdrop() */
+ if (atomic_dec_and_test(&mm->mm_count))
+ call_rcu(&mm->delayed_drop, __mmdrop_delayed);
+}
+#else
+static inline void mmdrop_sched(struct mm_struct *mm)
+{
+ mmdrop(mm);
+}
+#endif
+
/**
* mmget() - Pin the address space associated with a &struct mm_struct.
* @mm: The address space to pin.
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -708,6 +708,19 @@ void __mmdrop(struct mm_struct *mm)
}
EXPORT_SYMBOL_GPL(__mmdrop);
+#ifdef CONFIG_PREEMPT_RT
+/*
+ * RCU callback for delayed mm drop. Not strictly RCU, but call_rcu() is
+ * by far the least expensive way to do that.
+ */
+void __mmdrop_delayed(struct rcu_head *rhp)
+{
+ struct mm_struct *mm = container_of(rhp, struct mm_struct, delayed_drop);
+
+ __mmdrop(mm);
+}
+#endif
+
static void mmdrop_async_fn(struct work_struct *work)
{
struct mm_struct *mm;
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4840,7 +4840,7 @@ static struct rq *finish_task_switch(str
*/
if (mm) {
membarrier_mm_sync_core_before_usermode(mm);
- mmdrop(mm);
+ mmdrop_sched(mm);
}
if (unlikely(prev_state == TASK_DEAD)) {
if (prev->sched_class->task_dead)
next prev parent reply other threads:[~2021-09-28 12:24 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-28 12:24 [patch 0/5] sched: Miscellaneous RT related tweaks Thomas Gleixner
2021-09-28 12:24 ` [patch 1/5] sched: Limit the number of task migrations per batch on RT Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
2021-09-28 12:24 ` [patch 2/5] sched: Disable TTWU_QUEUE " Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
2021-09-28 12:24 ` [patch 3/5] sched: Move kprobes cleanup out of finish_task_switch() Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
2021-09-28 12:24 ` [patch 4/5] sched: Delay task stack freeing on RT Thomas Gleixner
2021-09-29 11:54 ` Peter Zijlstra
2021-10-01 16:12 ` Andy Lutomirski
2021-10-01 17:24 ` Thomas Gleixner
2021-10-01 18:48 ` Andy Lutomirski
2021-10-01 19:02 ` Andy Lutomirski
2021-10-01 20:54 ` Thomas Gleixner
2021-09-28 12:24 ` Thomas Gleixner [this message]
2021-09-29 12:02 ` [patch 5/5] sched: Move mmdrop to RCU " Peter Zijlstra
2021-09-29 13:05 ` Thomas Gleixner
2021-10-01 15:05 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2021-10-05 14:11 ` tip-bot2 for Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210928122411.648582026@linutronix.de \
--to=tglx@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).