* [patch V4 0/2] smp: Make softirq handling RT friendly
@ 2022-04-12 20:51 Thomas Gleixner
2022-04-12 20:51 ` [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle() Thomas Gleixner
2022-04-12 20:51 ` [patch V4 2/2] smp: Make softirq handling RT safe in flush_smp_call_function_queue() Thomas Gleixner
0 siblings, 2 replies; 7+ messages in thread
From: Thomas Gleixner @ 2022-04-12 20:51 UTC (permalink / raw)
To: LKML; +Cc: Christoph Hellwig, Peter Zijlstra, Sebastian Andrzej Siewior
The invocation of do_softirq() in flush_smp_call_function_from_idle()
breaks on RT because RT does not provide do_softirq().
Aside of that flush_smp_call_function_from_idle() is a misnomer since the
function is invoked not only from idle, but also from the scheduler
migration thread.
The following series addresses this. Changes vs. V3, which can be found
here: https://lore.kernel.org/lkml/YgKgL6aPj8aBES6G@linutronix.de
- Rename flush_smp_call_function_from_idle() and add comments
- Confine the RT specifics in the softirq code
Thanks,
tglx
---
include/linux/interrupt.h | 9 +++++++++
kernel/sched/core.c | 2 +-
kernel/sched/idle.c | 2 +-
kernel/sched/sched.h | 4 ++--
kernel/smp.c | 32 ++++++++++++++++++++++++--------
kernel/softirq.c | 13 +++++++++++++
6 files changed, 50 insertions(+), 12 deletions(-)
^ permalink raw reply [flat|nested] 7+ messages in thread
* [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle()
2022-04-12 20:51 [patch V4 0/2] smp: Make softirq handling RT friendly Thomas Gleixner
@ 2022-04-12 20:51 ` Thomas Gleixner
2022-04-13 2:52 ` kernel test robot
2022-04-13 2:53 ` kernel test robot
2022-04-12 20:51 ` [patch V4 2/2] smp: Make softirq handling RT safe in flush_smp_call_function_queue() Thomas Gleixner
1 sibling, 2 replies; 7+ messages in thread
From: Thomas Gleixner @ 2022-04-12 20:51 UTC (permalink / raw)
To: LKML; +Cc: Christoph Hellwig, Peter Zijlstra, Sebastian Andrzej Siewior
This is invoked from the stopper thread too, which is definitely not idle.
Rename it to flush_smp_call_function_queue() and fixup the callers.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V4: New patch
---
kernel/sched/core.c | 2 +-
kernel/sched/idle.c | 2 +-
kernel/sched/sched.h | 4 ++--
kernel/smp.c | 27 ++++++++++++++++++++-------
4 files changed, 24 insertions(+), 11 deletions(-)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2408,7 +2408,7 @@ static int migration_cpu_stop(void *data
* __migrate_task() such that we will not miss enforcing cpus_ptr
* during wakeups, see set_cpus_allowed_ptr()'s TASK_WAKING test.
*/
- flush_smp_call_function_from_idle();
+ flush_smp_call_function_queue();
raw_spin_lock(&p->pi_lock);
rq_lock(rq, &rf);
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -327,7 +327,7 @@ static void do_idle(void)
* RCU relies on this call to be done outside of an RCU read-side
* critical section.
*/
- flush_smp_call_function_from_idle();
+ flush_smp_call_function_queue();
schedule_idle();
if (unlikely(klp_patch_pending(current)))
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1834,10 +1834,10 @@ static inline void dirty_sched_domain_sy
extern int sched_update_scaling(void);
-extern void flush_smp_call_function_from_idle(void);
+extern void flush_smp_call_function_queue(void);
#else /* !CONFIG_SMP: */
-static inline void flush_smp_call_function_from_idle(void) { }
+static inline void flush_smp_call_function_queue(void) { }
#endif
#include "stats.h"
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -96,7 +96,7 @@ static DEFINE_PER_CPU_ALIGNED(struct cal
static DEFINE_PER_CPU_SHARED_ALIGNED(struct llist_head, call_single_queue);
-static void flush_smp_call_function_queue(bool warn_cpu_offline);
+static void __flush_smp_call_function_queue(bool warn_cpu_offline);
int smpcfd_prepare_cpu(unsigned int cpu)
{
@@ -141,7 +141,7 @@ int smpcfd_dying_cpu(unsigned int cpu)
* ensure that the outgoing CPU doesn't go offline with work
* still pending.
*/
- flush_smp_call_function_queue(false);
+ __flush_smp_call_function_queue(false);
irq_work_run();
return 0;
}
@@ -541,11 +541,11 @@ void generic_smp_call_function_single_in
{
cfd_seq_store(this_cpu_ptr(&cfd_seq_local)->gotipi, CFD_SEQ_NOCPU,
smp_processor_id(), CFD_SEQ_GOTIPI);
- flush_smp_call_function_queue(true);
+ __flush_smp_call_function_queue(true);
}
/**
- * flush_smp_call_function_queue - Flush pending smp-call-function callbacks
+ * __flush_smp_call_function_queue - Flush pending smp-call-function callbacks
*
* @warn_cpu_offline: If set to 'true', warn if callbacks were queued on an
* offline CPU. Skip this check if set to 'false'.
@@ -558,7 +558,7 @@ void generic_smp_call_function_single_in
* Loop through the call_single_queue and run all the queued callbacks.
* Must be called with interrupts disabled.
*/
-static void flush_smp_call_function_queue(bool warn_cpu_offline)
+static void __flush_smp_call_function_queue(bool warn_cpu_offline)
{
call_single_data_t *csd, *csd_next;
struct llist_node *entry, *prev;
@@ -681,7 +681,20 @@ static void flush_smp_call_function_queu
smp_processor_id(), CFD_SEQ_HDLEND);
}
-void flush_smp_call_function_from_idle(void)
+
+/**
+ * flush_smp_call_function_queue - Flush pending smp-call-function callbacks
+ * from task context (idle, migration thread)
+ *
+ * When TIF_POLLING_NRFLAG is supported and a CPU is in idle and has it
+ * set, then remote CPUs can avoid sending IPIs and wake the idle CPU by
+ * setting TIF_NEED_RESCHED. The idle task on the woken up CPU has to
+ * handle queued SMP function calls before scheduling.
+ *
+ * The migration thread has to ensure that an eventually pending wakeup has
+ * been handled before it migrates a task.
+ */
+void flush_smp_call_function_queue(void)
{
unsigned long flags;
@@ -691,7 +704,7 @@ void flush_smp_call_function_from_idle(v
cfd_seq_store(this_cpu_ptr(&cfd_seq_local)->idle, CFD_SEQ_NOCPU,
smp_processor_id(), CFD_SEQ_IDLE);
local_irq_save(flags);
- flush_smp_call_function_queue(true);
+ __flush_smp_call_function_queue(true);
if (local_softirq_pending())
do_softirq();
^ permalink raw reply [flat|nested] 7+ messages in thread
* [patch V4 2/2] smp: Make softirq handling RT safe in flush_smp_call_function_queue()
2022-04-12 20:51 [patch V4 0/2] smp: Make softirq handling RT friendly Thomas Gleixner
2022-04-12 20:51 ` [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle() Thomas Gleixner
@ 2022-04-12 20:51 ` Thomas Gleixner
1 sibling, 0 replies; 7+ messages in thread
From: Thomas Gleixner @ 2022-04-12 20:51 UTC (permalink / raw)
To: LKML; +Cc: Christoph Hellwig, Peter Zijlstra, Sebastian Andrzej Siewior
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
flush_smp_call_function_queue() invokes do_softirq() which is not available
on PREEMPT_RT. flush_smp_call_function_queue() is invoked from the idle
task and the migration task with preemption or interrupts disabled.
So RT kernels cannot process soft interrupts in that context as that has to
acquire 'sleeping spinlocks' which is not possible with preemption or
interrupts disabled and forbidden from the idle task anyway.
The currently known SMP function call which raises a soft interrupt is in
the block layer, but this functionality is not enabled on RT kernels due to
latency and performance reasons.
RT could wake up ksoftirqd unconditionally, but this wants to be avoided if
there were soft interrupts pending already when this is invoked in the
context of the migration task. The migration task might have preempted a
threaded interrupt handler which raised a soft interrupt, but did not reach
the local_bh_enable() to process it. The "running" ksoftirqd might prevent
the handling in the interrupt thread context which is causing latency
issues.
Add a new function which handles this case explicitely for RT and falls
back to do_softirq() on !RT kernels. In the RT case this warns when one of
the flushed SMP function calls raised a soft interrupt so this can be
investigated.
[ tglx: Moved the RT part out of SMP code ]
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/YgKgL6aPj8aBES6G@linutronix.de
---
v4:
- Move the RT logic into softirq.c which also avoids the wakeup
when softinterrupts are disabled. The enable will handle
them anyway.
v3:
- Only wake ksoftirqd if the softirqs were raised wthin
flush_smp_call_function_queue().
- Add a warning in the wake case.
v2: Drop an empty line.
include/linux/interrupt.h | 9 +++++++++
kernel/smp.c | 5 ++++-
kernel/softirq.c | 13 +++++++++++++
3 files changed, 26 insertions(+), 1 deletion(-)
---
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -607,6 +607,15 @@ struct softirq_action
asmlinkage void do_softirq(void);
asmlinkage void __do_softirq(void);
+#ifdef CONFIG_PREEMPT_RT
+extern void do_softirq_post_smp_call_flush(unsigned int was_pending);
+#else
+static inline void do_softirq_post_smp_call_flush(unsigned int unused)
+{
+ do_softirq();
+}
+#endif
+
extern void open_softirq(int nr, void (*action)(struct softirq_action *));
extern void softirq_init(void);
extern void __raise_softirq_irqoff(unsigned int nr);
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -696,6 +696,7 @@ static void __flush_smp_call_function_qu
*/
void flush_smp_call_function_queue(void)
{
+ unsigned int was_pending;
unsigned long flags;
if (llist_empty(this_cpu_ptr(&call_single_queue)))
@@ -704,9 +705,11 @@ void flush_smp_call_function_queue(void)
cfd_seq_store(this_cpu_ptr(&cfd_seq_local)->idle, CFD_SEQ_NOCPU,
smp_processor_id(), CFD_SEQ_IDLE);
local_irq_save(flags);
+ /* Get the already pending soft interrupts for RT enabled kernels */
+ was_pending = local_softirq_pending();
__flush_smp_call_function_queue(true);
if (local_softirq_pending())
- do_softirq();
+ do_softirq_post_smp_call_flush(was_pending);
local_irq_restore(flags);
}
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -294,6 +294,19 @@ static inline void invoke_softirq(void)
wakeup_softirqd();
}
+/*
+ * flush_smp_call_function_queue() can raise a soft interrupt in a function
+ * call. On RT kernels this is undesired and the only known functionality
+ * in the block layer which does this is disabled on RT. If soft interrupts
+ * get raised which haven't been raised before the flush, warn so it can be
+ * investigated.
+ */
+void softirq_post_smp_call_flush(unsigned int was_pending)
+{
+ if (WARN_ON_ONCE(was_pending != local_softirq_pending()))
+ invoke_softirq();
+}
+
#else /* CONFIG_PREEMPT_RT */
/*
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle()
2022-04-12 20:51 ` [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle() Thomas Gleixner
@ 2022-04-13 2:52 ` kernel test robot
2022-04-13 13:00 ` Thomas Gleixner
2022-04-13 2:53 ` kernel test robot
1 sibling, 1 reply; 7+ messages in thread
From: kernel test robot @ 2022-04-13 2:52 UTC (permalink / raw)
To: Thomas Gleixner, LKML
Cc: kbuild-all, Christoph Hellwig, Sebastian Andrzej Siewior
Hi Thomas,
I love your patch! Perhaps something to improve:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on hch-configfs/for-next linus/master linux/master v5.18-rc2 next-20220412]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Thomas-Gleixner/smp-Make-softirq-handling-RT-friendly/20220413-045944
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 089c02ae2771a14af2928c59c56abfb9b885a8d7
config: arc-defconfig (https://download.01.org/0day-ci/archive/20220413/202204131011.m9M80Uip-lkp@intel.com/config)
compiler: arc-elf-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/7c7c3b99f3599eb409f237b32f330fa06bffcd5d
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Thomas-Gleixner/smp-Make-softirq-handling-RT-friendly/20220413-045944
git checkout 7c7c3b99f3599eb409f237b32f330fa06bffcd5d
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=arc SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
>> kernel/smp.c:697:6: warning: no previous prototype for 'flush_smp_call_function_queue' [-Wmissing-prototypes]
697 | void flush_smp_call_function_queue(void)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vim +/flush_smp_call_function_queue +697 kernel/smp.c
683
684
685 /**
686 * flush_smp_call_function_queue - Flush pending smp-call-function callbacks
687 * from task context (idle, migration thread)
688 *
689 * When TIF_POLLING_NRFLAG is supported and a CPU is in idle and has it
690 * set, then remote CPUs can avoid sending IPIs and wake the idle CPU by
691 * setting TIF_NEED_RESCHED. The idle task on the woken up CPU has to
692 * handle queued SMP function calls before scheduling.
693 *
694 * The migration thread has to ensure that an eventually pending wakeup has
695 * been handled before it migrates a task.
696 */
> 697 void flush_smp_call_function_queue(void)
698 {
699 unsigned long flags;
700
701 if (llist_empty(this_cpu_ptr(&call_single_queue)))
702 return;
703
704 cfd_seq_store(this_cpu_ptr(&cfd_seq_local)->idle, CFD_SEQ_NOCPU,
705 smp_processor_id(), CFD_SEQ_IDLE);
706 local_irq_save(flags);
707 __flush_smp_call_function_queue(true);
708 if (local_softirq_pending())
709 do_softirq();
710
711 local_irq_restore(flags);
712 }
713
--
0-DAY CI Kernel Test Service
https://01.org/lkp
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle()
2022-04-12 20:51 ` [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle() Thomas Gleixner
2022-04-13 2:52 ` kernel test robot
@ 2022-04-13 2:53 ` kernel test robot
1 sibling, 0 replies; 7+ messages in thread
From: kernel test robot @ 2022-04-13 2:53 UTC (permalink / raw)
To: Thomas Gleixner, LKML
Cc: llvm, kbuild-all, Christoph Hellwig, Sebastian Andrzej Siewior
Hi Thomas,
I love your patch! Perhaps something to improve:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on hch-configfs/for-next linus/master linux/master v5.18-rc2 next-20220412]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Thomas-Gleixner/smp-Make-softirq-handling-RT-friendly/20220413-045944
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 089c02ae2771a14af2928c59c56abfb9b885a8d7
config: hexagon-defconfig (https://download.01.org/0day-ci/archive/20220413/202204131056.b582vtMv-lkp@intel.com/config)
compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project fe2478d44e4f7f191c43fef629ac7a23d0251e72)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/7c7c3b99f3599eb409f237b32f330fa06bffcd5d
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Thomas-Gleixner/smp-Make-softirq-handling-RT-friendly/20220413-045944
git checkout 7c7c3b99f3599eb409f237b32f330fa06bffcd5d
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=hexagon SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
>> kernel/smp.c:697:6: warning: no previous prototype for function 'flush_smp_call_function_queue' [-Wmissing-prototypes]
void flush_smp_call_function_queue(void)
^
kernel/smp.c:697:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
void flush_smp_call_function_queue(void)
^
static
1 warning generated.
vim +/flush_smp_call_function_queue +697 kernel/smp.c
683
684
685 /**
686 * flush_smp_call_function_queue - Flush pending smp-call-function callbacks
687 * from task context (idle, migration thread)
688 *
689 * When TIF_POLLING_NRFLAG is supported and a CPU is in idle and has it
690 * set, then remote CPUs can avoid sending IPIs and wake the idle CPU by
691 * setting TIF_NEED_RESCHED. The idle task on the woken up CPU has to
692 * handle queued SMP function calls before scheduling.
693 *
694 * The migration thread has to ensure that an eventually pending wakeup has
695 * been handled before it migrates a task.
696 */
> 697 void flush_smp_call_function_queue(void)
698 {
699 unsigned long flags;
700
701 if (llist_empty(this_cpu_ptr(&call_single_queue)))
702 return;
703
704 cfd_seq_store(this_cpu_ptr(&cfd_seq_local)->idle, CFD_SEQ_NOCPU,
705 smp_processor_id(), CFD_SEQ_IDLE);
706 local_irq_save(flags);
707 __flush_smp_call_function_queue(true);
708 if (local_softirq_pending())
709 do_softirq();
710
711 local_irq_restore(flags);
712 }
713
--
0-DAY CI Kernel Test Service
https://01.org/lkp
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle()
2022-04-13 2:52 ` kernel test robot
@ 2022-04-13 13:00 ` Thomas Gleixner
0 siblings, 0 replies; 7+ messages in thread
From: Thomas Gleixner @ 2022-04-13 13:00 UTC (permalink / raw)
To: kernel test robot, LKML
Cc: kbuild-all, Christoph Hellwig, Sebastian Andrzej Siewior
On Wed, Apr 13 2022 at 10:52, kernel test robot wrote:
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
>
> All warnings (new ones prefixed by >>):
>
>>> kernel/smp.c:697:6: warning: no previous prototype for 'flush_smp_call_function_queue' [-Wmissing-prototypes]
> 697 | void flush_smp_call_function_queue(void)
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is not really new. It's "new" because the function was renamed.
I'll fix the whole pile of preexisting warnings which are related to
kernel/sched/*.
Thanks,
tglx
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle()
@ 2022-04-13 13:00 ` Thomas Gleixner
0 siblings, 0 replies; 7+ messages in thread
From: Thomas Gleixner @ 2022-04-13 13:00 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 625 bytes --]
On Wed, Apr 13 2022 at 10:52, kernel test robot wrote:
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
>
> All warnings (new ones prefixed by >>):
>
>>> kernel/smp.c:697:6: warning: no previous prototype for 'flush_smp_call_function_queue' [-Wmissing-prototypes]
> 697 | void flush_smp_call_function_queue(void)
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is not really new. It's "new" because the function was renamed.
I'll fix the whole pile of preexisting warnings which are related to
kernel/sched/*.
Thanks,
tglx
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-04-13 13:00 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-12 20:51 [patch V4 0/2] smp: Make softirq handling RT friendly Thomas Gleixner
2022-04-12 20:51 ` [patch V4 1/2] smp: Rename flush_smp_call_function_from_idle() Thomas Gleixner
2022-04-13 2:52 ` kernel test robot
2022-04-13 13:00 ` Thomas Gleixner
2022-04-13 13:00 ` Thomas Gleixner
2022-04-13 2:53 ` kernel test robot
2022-04-12 20:51 ` [patch V4 2/2] smp: Make softirq handling RT safe in flush_smp_call_function_queue() Thomas Gleixner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.