* [PATCH 1/6] preempt: Merge preempt_mask.h into preempt.h
2015-05-12 14:41 [PATCH 0/6] preempt: A few headers cleanups and preempt_schedule*() optimizations v2 Frederic Weisbecker
@ 2015-05-12 14:41 ` Frederic Weisbecker
2015-05-19 7:16 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 2/6] preempt: Rearrange a few symbols after headers merge Frederic Weisbecker
` (4 subsequent siblings)
5 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2015-05-12 14:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: LKML, Frederic Weisbecker, Ingo Molnar, Thomas Gleixner, Linus Torvalds
preempt_mask.h defines all the preempt_count semantics and related
symbols: preempt, softirq, hardirq, nmi, preempt active, need resched,
etc...
preempt.h defines the accessors and mutators of preempt_count.
But there is a messy dependency game around those two header files:
* preempt_mask.h includes preempt.h in order to access preempt_count()
* preempt_mask.h defines all preempt_count semantic and symbols
except PREEMPT_NEED_RESCHED that is needed by asm/preempt.h
Thus we need to define it from preempt.h, right before including
asm/preempt.h, instead of defining it to preempt_mask.h with the
other preempt_count symbols. Therefore the preempt_count semantics
happen to be spread out.
* We plan to introduce preempt_active_[enter,exit]() to consolidate
preempt_schedule*() code. But we'll need to access both preempt_count
mutators (preempt_count_add()) and preempt_count symbols
(PREEMPT_ACTIVE, PREEMPT_OFFSET). The usual place to define preempt
operations is in preempt.h but then we'll need symbols in
preempt_mask.h which already includes preempt.h. So we end up with
a ressource circle dependency.
Lets merge preempt_mask.h into preempt.h to solve these dependency issues.
This way we gather semantic symbols and operation definition of
preempt_count in a single file.
This is a dumb copy-paste merge. Further merge re-arrangments are
performed in a subsequent patch to ease review.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
arch/m68k/include/asm/irqflags.h | 3 -
include/linux/bottom_half.h | 1 -
include/linux/hardirq.h | 2 +-
include/linux/preempt.h | 111 +++++++++++++++++++++++++++++++++++++
include/linux/preempt_mask.h | 117 ---------------------------------------
include/linux/sched.h | 2 +-
lib/radix-tree.c | 2 +-
7 files changed, 114 insertions(+), 124 deletions(-)
delete mode 100644 include/linux/preempt_mask.h
diff --git a/arch/m68k/include/asm/irqflags.h b/arch/m68k/include/asm/irqflags.h
index a823cd7..b594181 100644
--- a/arch/m68k/include/asm/irqflags.h
+++ b/arch/m68k/include/asm/irqflags.h
@@ -2,9 +2,6 @@
#define _M68K_IRQFLAGS_H
#include <linux/types.h>
-#ifdef CONFIG_MMU
-#include <linux/preempt_mask.h>
-#endif
#include <linux/preempt.h>
#include <asm/thread_info.h>
#include <asm/entry.h>
diff --git a/include/linux/bottom_half.h b/include/linux/bottom_half.h
index 86c12c9..8fdcb78 100644
--- a/include/linux/bottom_half.h
+++ b/include/linux/bottom_half.h
@@ -2,7 +2,6 @@
#define _LINUX_BH_H
#include <linux/preempt.h>
-#include <linux/preempt_mask.h>
#ifdef CONFIG_TRACE_IRQFLAGS
extern void __local_bh_disable_ip(unsigned long ip, unsigned int cnt);
diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h
index cba442e..4e971fa 100644
--- a/include/linux/hardirq.h
+++ b/include/linux/hardirq.h
@@ -1,7 +1,7 @@
#ifndef LINUX_HARDIRQ_H
#define LINUX_HARDIRQ_H
-#include <linux/preempt_mask.h>
+#include <linux/preempt.h>
#include <linux/lockdep.h>
#include <linux/ftrace_irq.h>
#include <linux/vtime.h>
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index de83b4e..8cc0338 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -17,6 +17,117 @@
#include <asm/preempt.h>
+/*
+ * We put the hardirq and softirq counter into the preemption
+ * counter. The bitmask has the following meaning:
+ *
+ * - bits 0-7 are the preemption count (max preemption depth: 256)
+ * - bits 8-15 are the softirq count (max # of softirqs: 256)
+ *
+ * The hardirq count could in theory be the same as the number of
+ * interrupts in the system, but we run all interrupt handlers with
+ * interrupts disabled, so we cannot have nesting interrupts. Though
+ * there are a few palaeontologic drivers which reenable interrupts in
+ * the handler, so we need more than one bit here.
+ *
+ * PREEMPT_MASK: 0x000000ff
+ * SOFTIRQ_MASK: 0x0000ff00
+ * HARDIRQ_MASK: 0x000f0000
+ * NMI_MASK: 0x00100000
+ * PREEMPT_ACTIVE: 0x00200000
+ */
+#define PREEMPT_BITS 8
+#define SOFTIRQ_BITS 8
+#define HARDIRQ_BITS 4
+#define NMI_BITS 1
+
+#define PREEMPT_SHIFT 0
+#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
+#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS)
+
+#define __IRQ_MASK(x) ((1UL << (x))-1)
+
+#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
+#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
+#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT)
+
+#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT)
+#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT)
+#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT)
+#define NMI_OFFSET (1UL << NMI_SHIFT)
+
+#define SOFTIRQ_DISABLE_OFFSET (2 * SOFTIRQ_OFFSET)
+
+#define PREEMPT_ACTIVE_BITS 1
+#define PREEMPT_ACTIVE_SHIFT (NMI_SHIFT + NMI_BITS)
+#define PREEMPT_ACTIVE (__IRQ_MASK(PREEMPT_ACTIVE_BITS) << PREEMPT_ACTIVE_SHIFT)
+
+#define hardirq_count() (preempt_count() & HARDIRQ_MASK)
+#define softirq_count() (preempt_count() & SOFTIRQ_MASK)
+#define irq_count() (preempt_count() & (HARDIRQ_MASK | SOFTIRQ_MASK \
+ | NMI_MASK))
+
+/*
+ * Are we doing bottom half or hardware interrupt processing?
+ * Are we in a softirq context? Interrupt context?
+ * in_softirq - Are we currently processing softirq or have bh disabled?
+ * in_serving_softirq - Are we currently processing softirq?
+ */
+#define in_irq() (hardirq_count())
+#define in_softirq() (softirq_count())
+#define in_interrupt() (irq_count())
+#define in_serving_softirq() (softirq_count() & SOFTIRQ_OFFSET)
+
+/*
+ * Are we in NMI context?
+ */
+#define in_nmi() (preempt_count() & NMI_MASK)
+
+#if defined(CONFIG_PREEMPT_COUNT)
+# define PREEMPT_CHECK_OFFSET 1
+#else
+# define PREEMPT_CHECK_OFFSET 0
+#endif
+
+/*
+ * The preempt_count offset needed for things like:
+ *
+ * spin_lock_bh()
+ *
+ * Which need to disable both preemption (CONFIG_PREEMPT_COUNT) and
+ * softirqs, such that unlock sequences of:
+ *
+ * spin_unlock();
+ * local_bh_enable();
+ *
+ * Work as expected.
+ */
+#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_CHECK_OFFSET)
+
+/*
+ * Are we running in atomic context? WARNING: this macro cannot
+ * always detect atomic context; in particular, it cannot know about
+ * held spinlocks in non-preemptible kernels. Thus it should not be
+ * used in the general case to determine whether sleeping is possible.
+ * Do not use in_atomic() in driver code.
+ */
+#define in_atomic() ((preempt_count() & ~PREEMPT_ACTIVE) != 0)
+
+/*
+ * Check whether we were atomic before we did preempt_disable():
+ * (used by the scheduler, *after* releasing the kernel lock)
+ */
+#define in_atomic_preempt_off() \
+ ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
+
+#ifdef CONFIG_PREEMPT_COUNT
+# define preemptible() (preempt_count() == 0 && !irqs_disabled())
+#else
+# define preemptible() 0
+#endif
+
#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
extern void preempt_count_add(int val);
extern void preempt_count_sub(int val);
diff --git a/include/linux/preempt_mask.h b/include/linux/preempt_mask.h
deleted file mode 100644
index dbeec4d..0000000
--- a/include/linux/preempt_mask.h
+++ /dev/null
@@ -1,117 +0,0 @@
-#ifndef LINUX_PREEMPT_MASK_H
-#define LINUX_PREEMPT_MASK_H
-
-#include <linux/preempt.h>
-
-/*
- * We put the hardirq and softirq counter into the preemption
- * counter. The bitmask has the following meaning:
- *
- * - bits 0-7 are the preemption count (max preemption depth: 256)
- * - bits 8-15 are the softirq count (max # of softirqs: 256)
- *
- * The hardirq count could in theory be the same as the number of
- * interrupts in the system, but we run all interrupt handlers with
- * interrupts disabled, so we cannot have nesting interrupts. Though
- * there are a few palaeontologic drivers which reenable interrupts in
- * the handler, so we need more than one bit here.
- *
- * PREEMPT_MASK: 0x000000ff
- * SOFTIRQ_MASK: 0x0000ff00
- * HARDIRQ_MASK: 0x000f0000
- * NMI_MASK: 0x00100000
- * PREEMPT_ACTIVE: 0x00200000
- */
-#define PREEMPT_BITS 8
-#define SOFTIRQ_BITS 8
-#define HARDIRQ_BITS 4
-#define NMI_BITS 1
-
-#define PREEMPT_SHIFT 0
-#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)
-#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS)
-
-#define __IRQ_MASK(x) ((1UL << (x))-1)
-
-#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
-#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
-#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
-#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT)
-
-#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT)
-#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT)
-#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT)
-#define NMI_OFFSET (1UL << NMI_SHIFT)
-
-#define SOFTIRQ_DISABLE_OFFSET (2 * SOFTIRQ_OFFSET)
-
-#define PREEMPT_ACTIVE_BITS 1
-#define PREEMPT_ACTIVE_SHIFT (NMI_SHIFT + NMI_BITS)
-#define PREEMPT_ACTIVE (__IRQ_MASK(PREEMPT_ACTIVE_BITS) << PREEMPT_ACTIVE_SHIFT)
-
-#define hardirq_count() (preempt_count() & HARDIRQ_MASK)
-#define softirq_count() (preempt_count() & SOFTIRQ_MASK)
-#define irq_count() (preempt_count() & (HARDIRQ_MASK | SOFTIRQ_MASK \
- | NMI_MASK))
-
-/*
- * Are we doing bottom half or hardware interrupt processing?
- * Are we in a softirq context? Interrupt context?
- * in_softirq - Are we currently processing softirq or have bh disabled?
- * in_serving_softirq - Are we currently processing softirq?
- */
-#define in_irq() (hardirq_count())
-#define in_softirq() (softirq_count())
-#define in_interrupt() (irq_count())
-#define in_serving_softirq() (softirq_count() & SOFTIRQ_OFFSET)
-
-/*
- * Are we in NMI context?
- */
-#define in_nmi() (preempt_count() & NMI_MASK)
-
-#if defined(CONFIG_PREEMPT_COUNT)
-# define PREEMPT_CHECK_OFFSET 1
-#else
-# define PREEMPT_CHECK_OFFSET 0
-#endif
-
-/*
- * The preempt_count offset needed for things like:
- *
- * spin_lock_bh()
- *
- * Which need to disable both preemption (CONFIG_PREEMPT_COUNT) and
- * softirqs, such that unlock sequences of:
- *
- * spin_unlock();
- * local_bh_enable();
- *
- * Work as expected.
- */
-#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_CHECK_OFFSET)
-
-/*
- * Are we running in atomic context? WARNING: this macro cannot
- * always detect atomic context; in particular, it cannot know about
- * held spinlocks in non-preemptible kernels. Thus it should not be
- * used in the general case to determine whether sleeping is possible.
- * Do not use in_atomic() in driver code.
- */
-#define in_atomic() ((preempt_count() & ~PREEMPT_ACTIVE) != 0)
-
-/*
- * Check whether we were atomic before we did preempt_disable():
- * (used by the scheduler, *after* releasing the kernel lock)
- */
-#define in_atomic_preempt_off() \
- ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
-
-#ifdef CONFIG_PREEMPT_COUNT
-# define preemptible() (preempt_count() == 0 && !irqs_disabled())
-#else
-# define preemptible() 0
-#endif
-
-#endif /* LINUX_PREEMPT_MASK_H */
diff --git a/include/linux/sched.h b/include/linux/sched.h
index fdca05c..5c13be9 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -25,7 +25,7 @@ struct sched_param {
#include <linux/errno.h>
#include <linux/nodemask.h>
#include <linux/mm_types.h>
-#include <linux/preempt_mask.h>
+#include <linux/preempt.h>
#include <asm/page.h>
#include <asm/ptrace.h>
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 3d2aa27..061550d 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -33,7 +33,7 @@
#include <linux/string.h>
#include <linux/bitops.h>
#include <linux/rcupdate.h>
-#include <linux/preempt_mask.h> /* in_interrupt() */
+#include <linux/preempt.h> /* in_interrupt() */
/*
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [tip:sched/core] sched/preempt: Merge preempt_mask.h into preempt.h
2015-05-12 14:41 ` [PATCH 1/6] preempt: Merge preempt_mask.h into preempt.h Frederic Weisbecker
@ 2015-05-19 7:16 ` tip-bot for Frederic Weisbecker
0 siblings, 0 replies; 17+ messages in thread
From: tip-bot for Frederic Weisbecker @ 2015-05-19 7:16 UTC (permalink / raw)
To: linux-tip-commits
Cc: fweisbec, torvalds, hpa, peterz, mingo, linux-kernel, tglx
Commit-ID: 92cf211874e954027b8e91cc9a15485a50b58d6b
Gitweb: http://git.kernel.org/tip/92cf211874e954027b8e91cc9a15485a50b58d6b
Author: Frederic Weisbecker <fweisbec@gmail.com>
AuthorDate: Tue, 12 May 2015 16:41:46 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 19 May 2015 08:39:11 +0200
sched/preempt: Merge preempt_mask.h into preempt.h
preempt_mask.h defines all the preempt_count semantics and related
symbols: preempt, softirq, hardirq, nmi, preempt active, need resched,
etc...
preempt.h defines the accessors and mutators of preempt_count.
But there is a messy dependency game around those two header files:
* preempt_mask.h includes preempt.h in order to access preempt_count()
* preempt_mask.h defines all preempt_count semantic and symbols
except PREEMPT_NEED_RESCHED that is needed by asm/preempt.h
Thus we need to define it from preempt.h, right before including
asm/preempt.h, instead of defining it to preempt_mask.h with the
other preempt_count symbols. Therefore the preempt_count semantics
happen to be spread out.
* We plan to introduce preempt_active_[enter,exit]() to consolidate
preempt_schedule*() code. But we'll need to access both preempt_count
mutators (preempt_count_add()) and preempt_count symbols
(PREEMPT_ACTIVE, PREEMPT_OFFSET). The usual place to define preempt
operations is in preempt.h but then we'll need symbols in
preempt_mask.h which already includes preempt.h. So we end up with
a ressource circle dependency.
Lets merge preempt_mask.h into preempt.h to solve these dependency issues.
This way we gather semantic symbols and operation definition of
preempt_count in a single file.
This is a dumb copy-paste merge. Further merge re-arrangments are
performed in a subsequent patch to ease review.
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1431441711-29753-2-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/m68k/include/asm/irqflags.h | 3 -
include/linux/bottom_half.h | 1 -
include/linux/hardirq.h | 2 +-
include/linux/preempt.h | 111 +++++++++++++++++++++++++++++++++++++
include/linux/preempt_mask.h | 117 ---------------------------------------
include/linux/sched.h | 2 +-
lib/radix-tree.c | 2 +-
7 files changed, 114 insertions(+), 124 deletions(-)
diff --git a/arch/m68k/include/asm/irqflags.h b/arch/m68k/include/asm/irqflags.h
index a823cd7..b594181 100644
--- a/arch/m68k/include/asm/irqflags.h
+++ b/arch/m68k/include/asm/irqflags.h
@@ -2,9 +2,6 @@
#define _M68K_IRQFLAGS_H
#include <linux/types.h>
-#ifdef CONFIG_MMU
-#include <linux/preempt_mask.h>
-#endif
#include <linux/preempt.h>
#include <asm/thread_info.h>
#include <asm/entry.h>
diff --git a/include/linux/bottom_half.h b/include/linux/bottom_half.h
index 86c12c9..8fdcb78 100644
--- a/include/linux/bottom_half.h
+++ b/include/linux/bottom_half.h
@@ -2,7 +2,6 @@
#define _LINUX_BH_H
#include <linux/preempt.h>
-#include <linux/preempt_mask.h>
#ifdef CONFIG_TRACE_IRQFLAGS
extern void __local_bh_disable_ip(unsigned long ip, unsigned int cnt);
diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h
index f4af034..dfd59d6 100644
--- a/include/linux/hardirq.h
+++ b/include/linux/hardirq.h
@@ -1,7 +1,7 @@
#ifndef LINUX_HARDIRQ_H
#define LINUX_HARDIRQ_H
-#include <linux/preempt_mask.h>
+#include <linux/preempt.h>
#include <linux/lockdep.h>
#include <linux/ftrace_irq.h>
#include <linux/vtime.h>
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index de83b4e..8cc0338 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -17,6 +17,117 @@
#include <asm/preempt.h>
+/*
+ * We put the hardirq and softirq counter into the preemption
+ * counter. The bitmask has the following meaning:
+ *
+ * - bits 0-7 are the preemption count (max preemption depth: 256)
+ * - bits 8-15 are the softirq count (max # of softirqs: 256)
+ *
+ * The hardirq count could in theory be the same as the number of
+ * interrupts in the system, but we run all interrupt handlers with
+ * interrupts disabled, so we cannot have nesting interrupts. Though
+ * there are a few palaeontologic drivers which reenable interrupts in
+ * the handler, so we need more than one bit here.
+ *
+ * PREEMPT_MASK: 0x000000ff
+ * SOFTIRQ_MASK: 0x0000ff00
+ * HARDIRQ_MASK: 0x000f0000
+ * NMI_MASK: 0x00100000
+ * PREEMPT_ACTIVE: 0x00200000
+ */
+#define PREEMPT_BITS 8
+#define SOFTIRQ_BITS 8
+#define HARDIRQ_BITS 4
+#define NMI_BITS 1
+
+#define PREEMPT_SHIFT 0
+#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
+#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS)
+
+#define __IRQ_MASK(x) ((1UL << (x))-1)
+
+#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
+#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
+#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT)
+
+#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT)
+#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT)
+#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT)
+#define NMI_OFFSET (1UL << NMI_SHIFT)
+
+#define SOFTIRQ_DISABLE_OFFSET (2 * SOFTIRQ_OFFSET)
+
+#define PREEMPT_ACTIVE_BITS 1
+#define PREEMPT_ACTIVE_SHIFT (NMI_SHIFT + NMI_BITS)
+#define PREEMPT_ACTIVE (__IRQ_MASK(PREEMPT_ACTIVE_BITS) << PREEMPT_ACTIVE_SHIFT)
+
+#define hardirq_count() (preempt_count() & HARDIRQ_MASK)
+#define softirq_count() (preempt_count() & SOFTIRQ_MASK)
+#define irq_count() (preempt_count() & (HARDIRQ_MASK | SOFTIRQ_MASK \
+ | NMI_MASK))
+
+/*
+ * Are we doing bottom half or hardware interrupt processing?
+ * Are we in a softirq context? Interrupt context?
+ * in_softirq - Are we currently processing softirq or have bh disabled?
+ * in_serving_softirq - Are we currently processing softirq?
+ */
+#define in_irq() (hardirq_count())
+#define in_softirq() (softirq_count())
+#define in_interrupt() (irq_count())
+#define in_serving_softirq() (softirq_count() & SOFTIRQ_OFFSET)
+
+/*
+ * Are we in NMI context?
+ */
+#define in_nmi() (preempt_count() & NMI_MASK)
+
+#if defined(CONFIG_PREEMPT_COUNT)
+# define PREEMPT_CHECK_OFFSET 1
+#else
+# define PREEMPT_CHECK_OFFSET 0
+#endif
+
+/*
+ * The preempt_count offset needed for things like:
+ *
+ * spin_lock_bh()
+ *
+ * Which need to disable both preemption (CONFIG_PREEMPT_COUNT) and
+ * softirqs, such that unlock sequences of:
+ *
+ * spin_unlock();
+ * local_bh_enable();
+ *
+ * Work as expected.
+ */
+#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_CHECK_OFFSET)
+
+/*
+ * Are we running in atomic context? WARNING: this macro cannot
+ * always detect atomic context; in particular, it cannot know about
+ * held spinlocks in non-preemptible kernels. Thus it should not be
+ * used in the general case to determine whether sleeping is possible.
+ * Do not use in_atomic() in driver code.
+ */
+#define in_atomic() ((preempt_count() & ~PREEMPT_ACTIVE) != 0)
+
+/*
+ * Check whether we were atomic before we did preempt_disable():
+ * (used by the scheduler, *after* releasing the kernel lock)
+ */
+#define in_atomic_preempt_off() \
+ ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
+
+#ifdef CONFIG_PREEMPT_COUNT
+# define preemptible() (preempt_count() == 0 && !irqs_disabled())
+#else
+# define preemptible() 0
+#endif
+
#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
extern void preempt_count_add(int val);
extern void preempt_count_sub(int val);
diff --git a/include/linux/preempt_mask.h b/include/linux/preempt_mask.h
deleted file mode 100644
index dbeec4d..0000000
--- a/include/linux/preempt_mask.h
+++ /dev/null
@@ -1,117 +0,0 @@
-#ifndef LINUX_PREEMPT_MASK_H
-#define LINUX_PREEMPT_MASK_H
-
-#include <linux/preempt.h>
-
-/*
- * We put the hardirq and softirq counter into the preemption
- * counter. The bitmask has the following meaning:
- *
- * - bits 0-7 are the preemption count (max preemption depth: 256)
- * - bits 8-15 are the softirq count (max # of softirqs: 256)
- *
- * The hardirq count could in theory be the same as the number of
- * interrupts in the system, but we run all interrupt handlers with
- * interrupts disabled, so we cannot have nesting interrupts. Though
- * there are a few palaeontologic drivers which reenable interrupts in
- * the handler, so we need more than one bit here.
- *
- * PREEMPT_MASK: 0x000000ff
- * SOFTIRQ_MASK: 0x0000ff00
- * HARDIRQ_MASK: 0x000f0000
- * NMI_MASK: 0x00100000
- * PREEMPT_ACTIVE: 0x00200000
- */
-#define PREEMPT_BITS 8
-#define SOFTIRQ_BITS 8
-#define HARDIRQ_BITS 4
-#define NMI_BITS 1
-
-#define PREEMPT_SHIFT 0
-#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)
-#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS)
-
-#define __IRQ_MASK(x) ((1UL << (x))-1)
-
-#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
-#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
-#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
-#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT)
-
-#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT)
-#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT)
-#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT)
-#define NMI_OFFSET (1UL << NMI_SHIFT)
-
-#define SOFTIRQ_DISABLE_OFFSET (2 * SOFTIRQ_OFFSET)
-
-#define PREEMPT_ACTIVE_BITS 1
-#define PREEMPT_ACTIVE_SHIFT (NMI_SHIFT + NMI_BITS)
-#define PREEMPT_ACTIVE (__IRQ_MASK(PREEMPT_ACTIVE_BITS) << PREEMPT_ACTIVE_SHIFT)
-
-#define hardirq_count() (preempt_count() & HARDIRQ_MASK)
-#define softirq_count() (preempt_count() & SOFTIRQ_MASK)
-#define irq_count() (preempt_count() & (HARDIRQ_MASK | SOFTIRQ_MASK \
- | NMI_MASK))
-
-/*
- * Are we doing bottom half or hardware interrupt processing?
- * Are we in a softirq context? Interrupt context?
- * in_softirq - Are we currently processing softirq or have bh disabled?
- * in_serving_softirq - Are we currently processing softirq?
- */
-#define in_irq() (hardirq_count())
-#define in_softirq() (softirq_count())
-#define in_interrupt() (irq_count())
-#define in_serving_softirq() (softirq_count() & SOFTIRQ_OFFSET)
-
-/*
- * Are we in NMI context?
- */
-#define in_nmi() (preempt_count() & NMI_MASK)
-
-#if defined(CONFIG_PREEMPT_COUNT)
-# define PREEMPT_CHECK_OFFSET 1
-#else
-# define PREEMPT_CHECK_OFFSET 0
-#endif
-
-/*
- * The preempt_count offset needed for things like:
- *
- * spin_lock_bh()
- *
- * Which need to disable both preemption (CONFIG_PREEMPT_COUNT) and
- * softirqs, such that unlock sequences of:
- *
- * spin_unlock();
- * local_bh_enable();
- *
- * Work as expected.
- */
-#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_CHECK_OFFSET)
-
-/*
- * Are we running in atomic context? WARNING: this macro cannot
- * always detect atomic context; in particular, it cannot know about
- * held spinlocks in non-preemptible kernels. Thus it should not be
- * used in the general case to determine whether sleeping is possible.
- * Do not use in_atomic() in driver code.
- */
-#define in_atomic() ((preempt_count() & ~PREEMPT_ACTIVE) != 0)
-
-/*
- * Check whether we were atomic before we did preempt_disable():
- * (used by the scheduler, *after* releasing the kernel lock)
- */
-#define in_atomic_preempt_off() \
- ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
-
-#ifdef CONFIG_PREEMPT_COUNT
-# define preemptible() (preempt_count() == 0 && !irqs_disabled())
-#else
-# define preemptible() 0
-#endif
-
-#endif /* LINUX_PREEMPT_MASK_H */
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 5f8defa..c53a178 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -25,7 +25,7 @@ struct sched_param {
#include <linux/errno.h>
#include <linux/nodemask.h>
#include <linux/mm_types.h>
-#include <linux/preempt_mask.h>
+#include <linux/preempt.h>
#include <asm/page.h>
#include <asm/ptrace.h>
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 3d2aa27..061550d 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -33,7 +33,7 @@
#include <linux/string.h>
#include <linux/bitops.h>
#include <linux/rcupdate.h>
-#include <linux/preempt_mask.h> /* in_interrupt() */
+#include <linux/preempt.h> /* in_interrupt() */
/*
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 2/6] preempt: Rearrange a few symbols after headers merge
2015-05-12 14:41 [PATCH 0/6] preempt: A few headers cleanups and preempt_schedule*() optimizations v2 Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 1/6] preempt: Merge preempt_mask.h into preempt.h Frederic Weisbecker
@ 2015-05-12 14:41 ` Frederic Weisbecker
2015-05-19 7:16 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 3/6] preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET Frederic Weisbecker
` (3 subsequent siblings)
5 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2015-05-12 14:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: LKML, Frederic Weisbecker, Ingo Molnar, Thomas Gleixner, Linus Torvalds
Adjust a few comments, and further integrate a few definitions after
the dumb headers copy.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
include/linux/preempt.h | 34 +++++++++++++++-------------------
1 file changed, 15 insertions(+), 19 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 8cc0338..37974cd 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -10,14 +10,6 @@
#include <linux/list.h>
/*
- * We use the MSB mostly because its available; see <linux/preempt_mask.h> for
- * the other bits -- can't include that header due to inclusion hell.
- */
-#define PREEMPT_NEED_RESCHED 0x80000000
-
-#include <asm/preempt.h>
-
-/*
* We put the hardirq and softirq counter into the preemption
* counter. The bitmask has the following meaning:
*
@@ -30,11 +22,12 @@
* there are a few palaeontologic drivers which reenable interrupts in
* the handler, so we need more than one bit here.
*
- * PREEMPT_MASK: 0x000000ff
- * SOFTIRQ_MASK: 0x0000ff00
- * HARDIRQ_MASK: 0x000f0000
- * NMI_MASK: 0x00100000
- * PREEMPT_ACTIVE: 0x00200000
+ * PREEMPT_MASK: 0x000000ff
+ * SOFTIRQ_MASK: 0x0000ff00
+ * HARDIRQ_MASK: 0x000f0000
+ * NMI_MASK: 0x00100000
+ * PREEMPT_ACTIVE: 0x00200000
+ * PREEMPT_NEED_RESCHED: 0x80000000
*/
#define PREEMPT_BITS 8
#define SOFTIRQ_BITS 8
@@ -64,6 +57,12 @@
#define PREEMPT_ACTIVE_SHIFT (NMI_SHIFT + NMI_BITS)
#define PREEMPT_ACTIVE (__IRQ_MASK(PREEMPT_ACTIVE_BITS) << PREEMPT_ACTIVE_SHIFT)
+/* We use the MSB mostly because its available */
+#define PREEMPT_NEED_RESCHED 0x80000000
+
+/* preempt_count() and related functions, depends on PREEMPT_NEED_RESCHED */
+#include <asm/preempt.h>
+
#define hardirq_count() (preempt_count() & HARDIRQ_MASK)
#define softirq_count() (preempt_count() & SOFTIRQ_MASK)
#define irq_count() (preempt_count() & (HARDIRQ_MASK | SOFTIRQ_MASK \
@@ -122,12 +121,6 @@
#define in_atomic_preempt_off() \
((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
-#ifdef CONFIG_PREEMPT_COUNT
-# define preemptible() (preempt_count() == 0 && !irqs_disabled())
-#else
-# define preemptible() 0
-#endif
-
#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
extern void preempt_count_add(int val);
extern void preempt_count_sub(int val);
@@ -160,6 +153,8 @@ do { \
#define preempt_enable_no_resched() sched_preempt_enable_no_resched()
+#define preemptible() (preempt_count() == 0 && !irqs_disabled())
+
#ifdef CONFIG_PREEMPT
#define preempt_enable() \
do { \
@@ -232,6 +227,7 @@ do { \
#define preempt_disable_notrace() barrier()
#define preempt_enable_no_resched_notrace() barrier()
#define preempt_enable_notrace() barrier()
+#define preemptible() 0
#endif /* CONFIG_PREEMPT_COUNT */
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [tip:sched/core] sched/preempt: Rearrange a few symbols after headers merge
2015-05-12 14:41 ` [PATCH 2/6] preempt: Rearrange a few symbols after headers merge Frederic Weisbecker
@ 2015-05-19 7:16 ` tip-bot for Frederic Weisbecker
0 siblings, 0 replies; 17+ messages in thread
From: tip-bot for Frederic Weisbecker @ 2015-05-19 7:16 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, torvalds, peterz, linux-kernel, fweisbec, mingo, tglx
Commit-ID: 2e10e71ce88e3eaccfd09a045ae6ecebe657ba09
Gitweb: http://git.kernel.org/tip/2e10e71ce88e3eaccfd09a045ae6ecebe657ba09
Author: Frederic Weisbecker <fweisbec@gmail.com>
AuthorDate: Tue, 12 May 2015 16:41:47 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 19 May 2015 08:39:12 +0200
sched/preempt: Rearrange a few symbols after headers merge
Adjust a few comments, and further integrate a few definitions after
the dumb headers copy.
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1431441711-29753-3-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/linux/preempt.h | 34 +++++++++++++++-------------------
1 file changed, 15 insertions(+), 19 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 8cc0338..37974cd 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -10,14 +10,6 @@
#include <linux/list.h>
/*
- * We use the MSB mostly because its available; see <linux/preempt_mask.h> for
- * the other bits -- can't include that header due to inclusion hell.
- */
-#define PREEMPT_NEED_RESCHED 0x80000000
-
-#include <asm/preempt.h>
-
-/*
* We put the hardirq and softirq counter into the preemption
* counter. The bitmask has the following meaning:
*
@@ -30,11 +22,12 @@
* there are a few palaeontologic drivers which reenable interrupts in
* the handler, so we need more than one bit here.
*
- * PREEMPT_MASK: 0x000000ff
- * SOFTIRQ_MASK: 0x0000ff00
- * HARDIRQ_MASK: 0x000f0000
- * NMI_MASK: 0x00100000
- * PREEMPT_ACTIVE: 0x00200000
+ * PREEMPT_MASK: 0x000000ff
+ * SOFTIRQ_MASK: 0x0000ff00
+ * HARDIRQ_MASK: 0x000f0000
+ * NMI_MASK: 0x00100000
+ * PREEMPT_ACTIVE: 0x00200000
+ * PREEMPT_NEED_RESCHED: 0x80000000
*/
#define PREEMPT_BITS 8
#define SOFTIRQ_BITS 8
@@ -64,6 +57,12 @@
#define PREEMPT_ACTIVE_SHIFT (NMI_SHIFT + NMI_BITS)
#define PREEMPT_ACTIVE (__IRQ_MASK(PREEMPT_ACTIVE_BITS) << PREEMPT_ACTIVE_SHIFT)
+/* We use the MSB mostly because its available */
+#define PREEMPT_NEED_RESCHED 0x80000000
+
+/* preempt_count() and related functions, depends on PREEMPT_NEED_RESCHED */
+#include <asm/preempt.h>
+
#define hardirq_count() (preempt_count() & HARDIRQ_MASK)
#define softirq_count() (preempt_count() & SOFTIRQ_MASK)
#define irq_count() (preempt_count() & (HARDIRQ_MASK | SOFTIRQ_MASK \
@@ -122,12 +121,6 @@
#define in_atomic_preempt_off() \
((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
-#ifdef CONFIG_PREEMPT_COUNT
-# define preemptible() (preempt_count() == 0 && !irqs_disabled())
-#else
-# define preemptible() 0
-#endif
-
#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
extern void preempt_count_add(int val);
extern void preempt_count_sub(int val);
@@ -160,6 +153,8 @@ do { \
#define preempt_enable_no_resched() sched_preempt_enable_no_resched()
+#define preemptible() (preempt_count() == 0 && !irqs_disabled())
+
#ifdef CONFIG_PREEMPT
#define preempt_enable() \
do { \
@@ -232,6 +227,7 @@ do { \
#define preempt_disable_notrace() barrier()
#define preempt_enable_no_resched_notrace() barrier()
#define preempt_enable_notrace() barrier()
+#define preemptible() 0
#endif /* CONFIG_PREEMPT_COUNT */
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 3/6] preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET
2015-05-12 14:41 [PATCH 0/6] preempt: A few headers cleanups and preempt_schedule*() optimizations v2 Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 1/6] preempt: Merge preempt_mask.h into preempt.h Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 2/6] preempt: Rearrange a few symbols after headers merge Frederic Weisbecker
@ 2015-05-12 14:41 ` Frederic Weisbecker
2015-05-19 7:16 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 4/6] sched: Optimize preemption operations on __schedule() callers Frederic Weisbecker
` (2 subsequent siblings)
5 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2015-05-12 14:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: LKML, Frederic Weisbecker, Ingo Molnar, Thomas Gleixner, Linus Torvalds
"CHECK" suggests it's only used as a comparison mask. But now it's used
further as a config-conditional preempt disabler offset. Lets
disambiguate this name.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
include/linux/preempt.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 37974cd..4689ef2 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -85,9 +85,9 @@
#define in_nmi() (preempt_count() & NMI_MASK)
#if defined(CONFIG_PREEMPT_COUNT)
-# define PREEMPT_CHECK_OFFSET 1
+# define PREEMPT_DISABLE_OFFSET 1
#else
-# define PREEMPT_CHECK_OFFSET 0
+# define PREEMPT_DISABLE_OFFSET 0
#endif
/*
@@ -103,7 +103,7 @@
*
* Work as expected.
*/
-#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_CHECK_OFFSET)
+#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_DISABLE_OFFSET)
/*
* Are we running in atomic context? WARNING: this macro cannot
@@ -119,7 +119,7 @@
* (used by the scheduler, *after* releasing the kernel lock)
*/
#define in_atomic_preempt_off() \
- ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
+ ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_DISABLE_OFFSET)
#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
extern void preempt_count_add(int val);
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [tip:sched/core] sched/preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET
2015-05-12 14:41 ` [PATCH 3/6] preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET Frederic Weisbecker
@ 2015-05-19 7:16 ` tip-bot for Frederic Weisbecker
0 siblings, 0 replies; 17+ messages in thread
From: tip-bot for Frederic Weisbecker @ 2015-05-19 7:16 UTC (permalink / raw)
To: linux-tip-commits
Cc: fweisbec, torvalds, peterz, tglx, hpa, mingo, linux-kernel
Commit-ID: 90b62b5129d5cb50f62f40e684de7a1961e57197
Gitweb: http://git.kernel.org/tip/90b62b5129d5cb50f62f40e684de7a1961e57197
Author: Frederic Weisbecker <fweisbec@gmail.com>
AuthorDate: Tue, 12 May 2015 16:41:48 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 19 May 2015 08:39:12 +0200
sched/preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET
"CHECK" suggests it's only used as a comparison mask. But now it's used
further as a config-conditional preempt disabler offset. Lets
disambiguate this name.
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1431441711-29753-4-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/linux/preempt.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 37974cd..4689ef2 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -85,9 +85,9 @@
#define in_nmi() (preempt_count() & NMI_MASK)
#if defined(CONFIG_PREEMPT_COUNT)
-# define PREEMPT_CHECK_OFFSET 1
+# define PREEMPT_DISABLE_OFFSET 1
#else
-# define PREEMPT_CHECK_OFFSET 0
+# define PREEMPT_DISABLE_OFFSET 0
#endif
/*
@@ -103,7 +103,7 @@
*
* Work as expected.
*/
-#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_CHECK_OFFSET)
+#define SOFTIRQ_LOCK_OFFSET (SOFTIRQ_DISABLE_OFFSET + PREEMPT_DISABLE_OFFSET)
/*
* Are we running in atomic context? WARNING: this macro cannot
@@ -119,7 +119,7 @@
* (used by the scheduler, *after* releasing the kernel lock)
*/
#define in_atomic_preempt_off() \
- ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_CHECK_OFFSET)
+ ((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_DISABLE_OFFSET)
#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
extern void preempt_count_add(int val);
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 4/6] sched: Optimize preemption operations on __schedule() callers
2015-05-12 14:41 [PATCH 0/6] preempt: A few headers cleanups and preempt_schedule*() optimizations v2 Frederic Weisbecker
` (2 preceding siblings ...)
2015-05-12 14:41 ` [PATCH 3/6] preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET Frederic Weisbecker
@ 2015-05-12 14:41 ` Frederic Weisbecker
2015-05-19 7:17 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 5/6] preempt: Fix out of date comment Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 6/6] preempt: Remove PREEMPT_ACTIVE unmasking off in_atomic() Frederic Weisbecker
5 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2015-05-12 14:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: LKML, Frederic Weisbecker, Ingo Molnar, Thomas Gleixner, Linus Torvalds
__schedule() disables preemption and some of its callers
(the preempt_schedule*() family) also set PREEMPT_ACTIVE.
So we have two preempt_count() modifications that could be performed
at once.
Lets remove the preemption disablement from __schedule() and pull
this responsibility to its callers in order to optimize preempt_count()
operations in a single place.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
include/linux/preempt.h | 12 ++++++++++++
kernel/sched/core.c | 29 +++++++++--------------------
2 files changed, 21 insertions(+), 20 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 4689ef2..45da394 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -137,6 +137,18 @@ extern void preempt_count_sub(int val);
#define preempt_count_inc() preempt_count_add(1)
#define preempt_count_dec() preempt_count_sub(1)
+#define preempt_active_enter() \
+do { \
+ preempt_count_add(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
+ barrier(); \
+} while (0)
+
+#define preempt_active_exit() \
+do { \
+ barrier(); \
+ preempt_count_sub(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
+} while (0)
+
#ifdef CONFIG_PREEMPT_COUNT
#define preempt_disable() \
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8027cfd..b23def2 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2723,9 +2723,7 @@ again:
* - return from syscall or exception to user-space
* - return from interrupt-handler to user-space
*
- * WARNING: all callers must re-check need_resched() afterward and reschedule
- * accordingly in case an event triggered the need for rescheduling (such as
- * an interrupt waking up a task) while preemption was disabled in __schedule().
+ * WARNING: must be called with preemption disabled!
*/
static void __sched __schedule(void)
{
@@ -2734,7 +2732,6 @@ static void __sched __schedule(void)
struct rq *rq;
int cpu;
- preempt_disable();
cpu = smp_processor_id();
rq = cpu_rq(cpu);
rcu_note_context_switch();
@@ -2798,8 +2795,6 @@ static void __sched __schedule(void)
raw_spin_unlock_irq(&rq->lock);
post_schedule(rq);
-
- sched_preempt_enable_no_resched();
}
static inline void sched_submit_work(struct task_struct *tsk)
@@ -2820,7 +2815,9 @@ asmlinkage __visible void __sched schedule(void)
sched_submit_work(tsk);
do {
+ preempt_disable();
__schedule();
+ sched_preempt_enable_no_resched();
} while (need_resched());
}
EXPORT_SYMBOL(schedule);
@@ -2859,15 +2856,14 @@ void __sched schedule_preempt_disabled(void)
static void __sched notrace preempt_schedule_common(void)
{
do {
- __preempt_count_add(PREEMPT_ACTIVE);
+ preempt_active_enter();
__schedule();
- __preempt_count_sub(PREEMPT_ACTIVE);
+ preempt_active_exit();
/*
* Check again in case we missed a preemption opportunity
* between schedule and now.
*/
- barrier();
} while (need_resched());
}
@@ -2914,7 +2910,7 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)
return;
do {
- __preempt_count_add(PREEMPT_ACTIVE);
+ preempt_active_enter();
/*
* Needs preempt disabled in case user_exit() is traced
* and the tracer calls preempt_enable_notrace() causing
@@ -2924,8 +2920,7 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)
__schedule();
exception_exit(prev_ctx);
- __preempt_count_sub(PREEMPT_ACTIVE);
- barrier();
+ preempt_active_exit();
} while (need_resched());
}
EXPORT_SYMBOL_GPL(preempt_schedule_context);
@@ -2949,17 +2944,11 @@ asmlinkage __visible void __sched preempt_schedule_irq(void)
prev_state = exception_enter();
do {
- __preempt_count_add(PREEMPT_ACTIVE);
+ preempt_active_enter();
local_irq_enable();
__schedule();
local_irq_disable();
- __preempt_count_sub(PREEMPT_ACTIVE);
-
- /*
- * Check again in case we missed a preemption opportunity
- * between schedule and now.
- */
- barrier();
+ preempt_active_exit();
} while (need_resched());
exception_exit(prev_state);
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [tip:sched/core] sched/preempt: Optimize preemption operations on __schedule() callers
2015-05-12 14:41 ` [PATCH 4/6] sched: Optimize preemption operations on __schedule() callers Frederic Weisbecker
@ 2015-05-19 7:17 ` tip-bot for Frederic Weisbecker
2015-07-20 22:14 ` Sasha Levin
0 siblings, 1 reply; 17+ messages in thread
From: tip-bot for Frederic Weisbecker @ 2015-05-19 7:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux-kernel, fweisbec, mingo, tglx, hpa, torvalds, peterz
Commit-ID: b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
Gitweb: http://git.kernel.org/tip/b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
Author: Frederic Weisbecker <fweisbec@gmail.com>
AuthorDate: Tue, 12 May 2015 16:41:49 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 19 May 2015 08:39:12 +0200
sched/preempt: Optimize preemption operations on __schedule() callers
__schedule() disables preemption and some of its callers
(the preempt_schedule*() family) also set PREEMPT_ACTIVE.
So we have two preempt_count() modifications that could be performed
at once.
Lets remove the preemption disablement from __schedule() and pull
this responsibility to its callers in order to optimize preempt_count()
operations in a single place.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1431441711-29753-5-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/linux/preempt.h | 12 ++++++++++++
kernel/sched/core.c | 29 +++++++++--------------------
2 files changed, 21 insertions(+), 20 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 4689ef2..45da394 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -137,6 +137,18 @@ extern void preempt_count_sub(int val);
#define preempt_count_inc() preempt_count_add(1)
#define preempt_count_dec() preempt_count_sub(1)
+#define preempt_active_enter() \
+do { \
+ preempt_count_add(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
+ barrier(); \
+} while (0)
+
+#define preempt_active_exit() \
+do { \
+ barrier(); \
+ preempt_count_sub(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
+} while (0)
+
#ifdef CONFIG_PREEMPT_COUNT
#define preempt_disable() \
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 355f953..5140db6 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2773,9 +2773,7 @@ again:
* - return from syscall or exception to user-space
* - return from interrupt-handler to user-space
*
- * WARNING: all callers must re-check need_resched() afterward and reschedule
- * accordingly in case an event triggered the need for rescheduling (such as
- * an interrupt waking up a task) while preemption was disabled in __schedule().
+ * WARNING: must be called with preemption disabled!
*/
static void __sched __schedule(void)
{
@@ -2784,7 +2782,6 @@ static void __sched __schedule(void)
struct rq *rq;
int cpu;
- preempt_disable();
cpu = smp_processor_id();
rq = cpu_rq(cpu);
rcu_note_context_switch();
@@ -2848,8 +2845,6 @@ static void __sched __schedule(void)
raw_spin_unlock_irq(&rq->lock);
post_schedule(rq);
-
- sched_preempt_enable_no_resched();
}
static inline void sched_submit_work(struct task_struct *tsk)
@@ -2870,7 +2865,9 @@ asmlinkage __visible void __sched schedule(void)
sched_submit_work(tsk);
do {
+ preempt_disable();
__schedule();
+ sched_preempt_enable_no_resched();
} while (need_resched());
}
EXPORT_SYMBOL(schedule);
@@ -2909,15 +2906,14 @@ void __sched schedule_preempt_disabled(void)
static void __sched notrace preempt_schedule_common(void)
{
do {
- __preempt_count_add(PREEMPT_ACTIVE);
+ preempt_active_enter();
__schedule();
- __preempt_count_sub(PREEMPT_ACTIVE);
+ preempt_active_exit();
/*
* Check again in case we missed a preemption opportunity
* between schedule and now.
*/
- barrier();
} while (need_resched());
}
@@ -2964,7 +2960,7 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)
return;
do {
- __preempt_count_add(PREEMPT_ACTIVE);
+ preempt_active_enter();
/*
* Needs preempt disabled in case user_exit() is traced
* and the tracer calls preempt_enable_notrace() causing
@@ -2974,8 +2970,7 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)
__schedule();
exception_exit(prev_ctx);
- __preempt_count_sub(PREEMPT_ACTIVE);
- barrier();
+ preempt_active_exit();
} while (need_resched());
}
EXPORT_SYMBOL_GPL(preempt_schedule_context);
@@ -2999,17 +2994,11 @@ asmlinkage __visible void __sched preempt_schedule_irq(void)
prev_state = exception_enter();
do {
- __preempt_count_add(PREEMPT_ACTIVE);
+ preempt_active_enter();
local_irq_enable();
__schedule();
local_irq_disable();
- __preempt_count_sub(PREEMPT_ACTIVE);
-
- /*
- * Check again in case we missed a preemption opportunity
- * between schedule and now.
- */
- barrier();
+ preempt_active_exit();
} while (need_resched());
exception_exit(prev_state);
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [tip:sched/core] sched/preempt: Optimize preemption operations on __schedule() callers
2015-05-19 7:17 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
@ 2015-07-20 22:14 ` Sasha Levin
2015-07-20 22:48 ` Frederic Weisbecker
2015-07-21 11:07 ` Peter Zijlstra
0 siblings, 2 replies; 17+ messages in thread
From: Sasha Levin @ 2015-07-20 22:14 UTC (permalink / raw)
To: linux-kernel, mingo, fweisbec, tglx, hpa, torvalds, peterz,
Paul E. McKenney
On 05/19/2015 03:17 AM, tip-bot for Frederic Weisbecker wrote:
> Commit-ID: b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
> Gitweb: http://git.kernel.org/tip/b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
> Author: Frederic Weisbecker <fweisbec@gmail.com>
> AuthorDate: Tue, 12 May 2015 16:41:49 +0200
> Committer: Ingo Molnar <mingo@kernel.org>
> CommitDate: Tue, 19 May 2015 08:39:12 +0200
>
> sched/preempt: Optimize preemption operations on __schedule() callers
>
> __schedule() disables preemption and some of its callers
> (the preempt_schedule*() family) also set PREEMPT_ACTIVE.
>
> So we have two preempt_count() modifications that could be performed
> at once.
>
> Lets remove the preemption disablement from __schedule() and pull
> this responsibility to its callers in order to optimize preempt_count()
> operations in a single place.
>
> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Link: http://lkml.kernel.org/r/1431441711-29753-5-git-send-email-fweisbec@gmail.com
> Signed-off-by: Ingo Molnar <mingo@kernel.org>
Hi all,
I was seeing RCU stall warnings that appeared to have different backtrace
each time, but would reliably trigger when fuzzing and looked something like this:
[4394886.414687] INFO: rcu_preempt detected stalls on CPUs/tasks:
[4394886.416717] Tasks blocked on level-0 rcu_node (CPUs 0-23): P13356
[4394886.418699] (detected by 2, t=20502 jiffies, g=3548, c=3547, q=120)
[4394886.420712] trinity-c42 R running task 26936 13356 9574 0x10000000
[4394886.422466] ffff8807d1c7fbe8 ffff8807d1c7fb88 0000000000000022 ffff8801081e1bb0
[4394886.423711] ffff8801081e1b88 ffff8807d1eb8780 ffff8801081e11d8 ffff8807e2564000
[4394886.424956] ffff8807d1eb8000 ffff8807d1c7fbd8 ffff8807d1c78000 0000000000000000
[4394886.426196] Call Trace:
[4394886.426627] preempt_schedule_irq (./arch/x86/include/asm/paravirt.h:807 kernel/sched/core.c:3218)
[4394886.427582] ? lockdep_reset_lock (kernel/locking/lockdep.c:3105)
[4394886.428548] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
[4394886.429438] retint_kernel (arch/x86/entry/entry_64.S:578)
[4394886.430286] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
[4394886.431176] ? native_restore_fl (./arch/x86/include/asm/irqflags.h:35)
[4394886.432376] lock_is_held (kernel/locking/lockdep.c:3661)
[4394886.433241] ? kill_pid_info (include/linux/rcupdate.h:914 kernel/signal.c:1344)
[4394886.434135] rcu_read_lock_held (kernel/rcu/update.c:275)
[4394886.435012] pid_task (kernel/pid.c:440 (discriminator 5))
[4394886.435656] kill_pid_info (kernel/signal.c:1341)
[4394886.436351] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
[4394886.437065] SYSC_kill (kernel/signal.c:1426 kernel/signal.c:2903)
[4394886.437726] ? SYSC_kill (include/linux/rcupdate.h:857 kernel/signal.c:1425 kernel/signal.c:2903)
[4394886.438419] ? find_get_pid (include/linux/rcupdate.h:914 kernel/pid.c:494)
[4394886.439134] ? kill_pid (kernel/signal.c:2894)
[4394886.439782] ? find_get_pid (kernel/pid.c:497)
[4394886.440492] ? find_get_pid (kernel/pid.c:489)
[4394886.441183] ? lock_is_held (kernel/locking/lockdep.c:3661)
[4394886.441929] ? rcu_read_lock_sched_held (kernel/rcu/update.c:109)
[4394886.442772] ? syscall_trace_enter_phase2 (arch/x86/entry/common.c:196)
[4394886.443632] SyS_kill (kernel/signal.c:2893)
[4394886.444255] tracesys_phase2 (arch/x86/entry/entry_64.S:270)
I worked with Paul to rule out RCU as the cause.
I've noticed that all traces had one thing in common: being stuck in preempt_schedule_irq(),
so I've looked at recent changes there and noticed this commit.
I've tried testing the commit before that, and the problem went away. Checking out this
commit the problem reappeared.
Thanks,
Sasha
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [tip:sched/core] sched/preempt: Optimize preemption operations on __schedule() callers
2015-07-20 22:14 ` Sasha Levin
@ 2015-07-20 22:48 ` Frederic Weisbecker
2015-07-20 22:52 ` Sasha Levin
2015-07-21 11:07 ` Peter Zijlstra
1 sibling, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2015-07-20 22:48 UTC (permalink / raw)
To: Sasha Levin
Cc: linux-kernel, mingo, tglx, hpa, torvalds, peterz, Paul E. McKenney
On Mon, Jul 20, 2015 at 06:14:27PM -0400, Sasha Levin wrote:
> On 05/19/2015 03:17 AM, tip-bot for Frederic Weisbecker wrote:
> > Commit-ID: b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
> > Gitweb: http://git.kernel.org/tip/b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
> > Author: Frederic Weisbecker <fweisbec@gmail.com>
> > AuthorDate: Tue, 12 May 2015 16:41:49 +0200
> > Committer: Ingo Molnar <mingo@kernel.org>
> > CommitDate: Tue, 19 May 2015 08:39:12 +0200
> >
> > sched/preempt: Optimize preemption operations on __schedule() callers
> >
> > __schedule() disables preemption and some of its callers
> > (the preempt_schedule*() family) also set PREEMPT_ACTIVE.
> >
> > So we have two preempt_count() modifications that could be performed
> > at once.
> >
> > Lets remove the preemption disablement from __schedule() and pull
> > this responsibility to its callers in order to optimize preempt_count()
> > operations in a single place.
> >
> > Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> > Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Link: http://lkml.kernel.org/r/1431441711-29753-5-git-send-email-fweisbec@gmail.com
> > Signed-off-by: Ingo Molnar <mingo@kernel.org>
>
> Hi all,
>
> I was seeing RCU stall warnings that appeared to have different backtrace
> each time, but would reliably trigger when fuzzing and looked something like this:
>
> [4394886.414687] INFO: rcu_preempt detected stalls on CPUs/tasks:
> [4394886.416717] Tasks blocked on level-0 rcu_node (CPUs 0-23): P13356
> [4394886.418699] (detected by 2, t=20502 jiffies, g=3548, c=3547, q=120)
> [4394886.420712] trinity-c42 R running task 26936 13356 9574 0x10000000
> [4394886.422466] ffff8807d1c7fbe8 ffff8807d1c7fb88 0000000000000022 ffff8801081e1bb0
> [4394886.423711] ffff8801081e1b88 ffff8807d1eb8780 ffff8801081e11d8 ffff8807e2564000
> [4394886.424956] ffff8807d1eb8000 ffff8807d1c7fbd8 ffff8807d1c78000 0000000000000000
> [4394886.426196] Call Trace:
> [4394886.426627] preempt_schedule_irq (./arch/x86/include/asm/paravirt.h:807 kernel/sched/core.c:3218)
> [4394886.427582] ? lockdep_reset_lock (kernel/locking/lockdep.c:3105)
> [4394886.428548] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
> [4394886.429438] retint_kernel (arch/x86/entry/entry_64.S:578)
> [4394886.430286] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
> [4394886.431176] ? native_restore_fl (./arch/x86/include/asm/irqflags.h:35)
> [4394886.432376] lock_is_held (kernel/locking/lockdep.c:3661)
> [4394886.433241] ? kill_pid_info (include/linux/rcupdate.h:914 kernel/signal.c:1344)
> [4394886.434135] rcu_read_lock_held (kernel/rcu/update.c:275)
> [4394886.435012] pid_task (kernel/pid.c:440 (discriminator 5))
> [4394886.435656] kill_pid_info (kernel/signal.c:1341)
> [4394886.436351] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
> [4394886.437065] SYSC_kill (kernel/signal.c:1426 kernel/signal.c:2903)
> [4394886.437726] ? SYSC_kill (include/linux/rcupdate.h:857 kernel/signal.c:1425 kernel/signal.c:2903)
> [4394886.438419] ? find_get_pid (include/linux/rcupdate.h:914 kernel/pid.c:494)
> [4394886.439134] ? kill_pid (kernel/signal.c:2894)
> [4394886.439782] ? find_get_pid (kernel/pid.c:497)
> [4394886.440492] ? find_get_pid (kernel/pid.c:489)
> [4394886.441183] ? lock_is_held (kernel/locking/lockdep.c:3661)
> [4394886.441929] ? rcu_read_lock_sched_held (kernel/rcu/update.c:109)
> [4394886.442772] ? syscall_trace_enter_phase2 (arch/x86/entry/common.c:196)
> [4394886.443632] SyS_kill (kernel/signal.c:2893)
> [4394886.444255] tracesys_phase2 (arch/x86/entry/entry_64.S:270)
>
> I worked with Paul to rule out RCU as the cause.
>
> I've noticed that all traces had one thing in common: being stuck in preempt_schedule_irq(),
> so I've looked at recent changes there and noticed this commit.
>
> I've tried testing the commit before that, and the problem went away. Checking out this
> commit the problem reappeared.
Do you have any idea how to reproduce that?
Thanks.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [tip:sched/core] sched/preempt: Optimize preemption operations on __schedule() callers
2015-07-20 22:48 ` Frederic Weisbecker
@ 2015-07-20 22:52 ` Sasha Levin
0 siblings, 0 replies; 17+ messages in thread
From: Sasha Levin @ 2015-07-20 22:52 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: linux-kernel, mingo, tglx, hpa, torvalds, peterz, Paul E. McKenney
On 07/20/2015 06:48 PM, Frederic Weisbecker wrote:
> On Mon, Jul 20, 2015 at 06:14:27PM -0400, Sasha Levin wrote:
>> > On 05/19/2015 03:17 AM, tip-bot for Frederic Weisbecker wrote:
>>> > > Commit-ID: b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
>>> > > Gitweb: http://git.kernel.org/tip/b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
>>> > > Author: Frederic Weisbecker <fweisbec@gmail.com>
>>> > > AuthorDate: Tue, 12 May 2015 16:41:49 +0200
>>> > > Committer: Ingo Molnar <mingo@kernel.org>
>>> > > CommitDate: Tue, 19 May 2015 08:39:12 +0200
>>> > >
>>> > > sched/preempt: Optimize preemption operations on __schedule() callers
>>> > >
>>> > > __schedule() disables preemption and some of its callers
>>> > > (the preempt_schedule*() family) also set PREEMPT_ACTIVE.
>>> > >
>>> > > So we have two preempt_count() modifications that could be performed
>>> > > at once.
>>> > >
>>> > > Lets remove the preemption disablement from __schedule() and pull
>>> > > this responsibility to its callers in order to optimize preempt_count()
>>> > > operations in a single place.
>>> > >
>>> > > Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
>>> > > Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
>>> > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>>> > > Cc: Peter Zijlstra <peterz@infradead.org>
>>> > > Cc: Thomas Gleixner <tglx@linutronix.de>
>>> > > Link: http://lkml.kernel.org/r/1431441711-29753-5-git-send-email-fweisbec@gmail.com
>>> > > Signed-off-by: Ingo Molnar <mingo@kernel.org>
>> >
>> > Hi all,
>> >
>> > I was seeing RCU stall warnings that appeared to have different backtrace
>> > each time, but would reliably trigger when fuzzing and looked something like this:
>> >
>> > [4394886.414687] INFO: rcu_preempt detected stalls on CPUs/tasks:
>> > [4394886.416717] Tasks blocked on level-0 rcu_node (CPUs 0-23): P13356
>> > [4394886.418699] (detected by 2, t=20502 jiffies, g=3548, c=3547, q=120)
>> > [4394886.420712] trinity-c42 R running task 26936 13356 9574 0x10000000
>> > [4394886.422466] ffff8807d1c7fbe8 ffff8807d1c7fb88 0000000000000022 ffff8801081e1bb0
>> > [4394886.423711] ffff8801081e1b88 ffff8807d1eb8780 ffff8801081e11d8 ffff8807e2564000
>> > [4394886.424956] ffff8807d1eb8000 ffff8807d1c7fbd8 ffff8807d1c78000 0000000000000000
>> > [4394886.426196] Call Trace:
>> > [4394886.426627] preempt_schedule_irq (./arch/x86/include/asm/paravirt.h:807 kernel/sched/core.c:3218)
>> > [4394886.427582] ? lockdep_reset_lock (kernel/locking/lockdep.c:3105)
>> > [4394886.428548] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
>> > [4394886.429438] retint_kernel (arch/x86/entry/entry_64.S:578)
>> > [4394886.430286] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
>> > [4394886.431176] ? native_restore_fl (./arch/x86/include/asm/irqflags.h:35)
>> > [4394886.432376] lock_is_held (kernel/locking/lockdep.c:3661)
>> > [4394886.433241] ? kill_pid_info (include/linux/rcupdate.h:914 kernel/signal.c:1344)
>> > [4394886.434135] rcu_read_lock_held (kernel/rcu/update.c:275)
>> > [4394886.435012] pid_task (kernel/pid.c:440 (discriminator 5))
>> > [4394886.435656] kill_pid_info (kernel/signal.c:1341)
>> > [4394886.436351] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
>> > [4394886.437065] SYSC_kill (kernel/signal.c:1426 kernel/signal.c:2903)
>> > [4394886.437726] ? SYSC_kill (include/linux/rcupdate.h:857 kernel/signal.c:1425 kernel/signal.c:2903)
>> > [4394886.438419] ? find_get_pid (include/linux/rcupdate.h:914 kernel/pid.c:494)
>> > [4394886.439134] ? kill_pid (kernel/signal.c:2894)
>> > [4394886.439782] ? find_get_pid (kernel/pid.c:497)
>> > [4394886.440492] ? find_get_pid (kernel/pid.c:489)
>> > [4394886.441183] ? lock_is_held (kernel/locking/lockdep.c:3661)
>> > [4394886.441929] ? rcu_read_lock_sched_held (kernel/rcu/update.c:109)
>> > [4394886.442772] ? syscall_trace_enter_phase2 (arch/x86/entry/common.c:196)
>> > [4394886.443632] SyS_kill (kernel/signal.c:2893)
>> > [4394886.444255] tracesys_phase2 (arch/x86/entry/entry_64.S:270)
>> >
>> > I worked with Paul to rule out RCU as the cause.
>> >
>> > I've noticed that all traces had one thing in common: being stuck in preempt_schedule_irq(),
>> > so I've looked at recent changes there and noticed this commit.
>> >
>> > I've tried testing the commit before that, and the problem went away. Checking out this
>> > commit the problem reappeared.
> Do you have any idea how to reproduce that?
Beyond running trinity, no. That's why I couldn't bisect it to begin with.
Thanks,
Sasha
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [tip:sched/core] sched/preempt: Optimize preemption operations on __schedule() callers
2015-07-20 22:14 ` Sasha Levin
2015-07-20 22:48 ` Frederic Weisbecker
@ 2015-07-21 11:07 ` Peter Zijlstra
1 sibling, 0 replies; 17+ messages in thread
From: Peter Zijlstra @ 2015-07-21 11:07 UTC (permalink / raw)
To: Sasha Levin
Cc: linux-kernel, mingo, fweisbec, tglx, hpa, torvalds, Paul E. McKenney
On Mon, Jul 20, 2015 at 06:14:27PM -0400, Sasha Levin wrote:
> On 05/19/2015 03:17 AM, tip-bot for Frederic Weisbecker wrote:
> > Commit-ID: b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
> > sched/preempt: Optimize preemption operations on __schedule() callers
> >
> I was seeing RCU stall warnings that appeared to have different backtrace
> each time, but would reliably trigger when fuzzing and looked something like this:
> [4394886.426627] preempt_schedule_irq (./arch/x86/include/asm/paravirt.h:807 kernel/sched/core.c:3218)
The only 'obvious' difference there seems to be something like the
below. Prior to the patch preempt_schedule_irq() used
__preempt_count_{add,sub}() which are the !tracing versions.
Does this make it go again? If not, can you provide a .config?
---
include/linux/preempt.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 84991f185173..235a9f2d76fe 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -139,14 +139,14 @@ extern void preempt_count_sub(int val);
#define preempt_active_enter() \
do { \
- preempt_count_add(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
+ __preempt_count_add(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
barrier(); \
} while (0)
#define preempt_active_exit() \
do { \
barrier(); \
- preempt_count_sub(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
+ __preempt_count_sub(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
} while (0)
#ifdef CONFIG_PREEMPT_COUNT
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 5/6] preempt: Fix out of date comment
2015-05-12 14:41 [PATCH 0/6] preempt: A few headers cleanups and preempt_schedule*() optimizations v2 Frederic Weisbecker
` (3 preceding siblings ...)
2015-05-12 14:41 ` [PATCH 4/6] sched: Optimize preemption operations on __schedule() callers Frederic Weisbecker
@ 2015-05-12 14:41 ` Frederic Weisbecker
2015-05-19 7:17 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 6/6] preempt: Remove PREEMPT_ACTIVE unmasking off in_atomic() Frederic Weisbecker
5 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2015-05-12 14:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: LKML, Frederic Weisbecker, Ingo Molnar, Thomas Gleixner, Linus Torvalds
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
include/linux/preempt.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 45da394..4057696 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -116,7 +116,7 @@
/*
* Check whether we were atomic before we did preempt_disable():
- * (used by the scheduler, *after* releasing the kernel lock)
+ * (used by the scheduler)
*/
#define in_atomic_preempt_off() \
((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_DISABLE_OFFSET)
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [tip:sched/core] sched/preempt: Fix out of date comment
2015-05-12 14:41 ` [PATCH 5/6] preempt: Fix out of date comment Frederic Weisbecker
@ 2015-05-19 7:17 ` tip-bot for Frederic Weisbecker
0 siblings, 0 replies; 17+ messages in thread
From: tip-bot for Frederic Weisbecker @ 2015-05-19 7:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux-kernel, torvalds, mingo, peterz, fweisbec, hpa, tglx
Commit-ID: e017cf21ae82e0b36f026b22083a8ae67926f465
Gitweb: http://git.kernel.org/tip/e017cf21ae82e0b36f026b22083a8ae67926f465
Author: Frederic Weisbecker <fweisbec@gmail.com>
AuthorDate: Tue, 12 May 2015 16:41:50 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 19 May 2015 08:39:13 +0200
sched/preempt: Fix out of date comment
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1431441711-29753-6-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/linux/preempt.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 45da394..4057696 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -116,7 +116,7 @@
/*
* Check whether we were atomic before we did preempt_disable():
- * (used by the scheduler, *after* releasing the kernel lock)
+ * (used by the scheduler)
*/
#define in_atomic_preempt_off() \
((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_DISABLE_OFFSET)
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 6/6] preempt: Remove PREEMPT_ACTIVE unmasking off in_atomic()
2015-05-12 14:41 [PATCH 0/6] preempt: A few headers cleanups and preempt_schedule*() optimizations v2 Frederic Weisbecker
` (4 preceding siblings ...)
2015-05-12 14:41 ` [PATCH 5/6] preempt: Fix out of date comment Frederic Weisbecker
@ 2015-05-12 14:41 ` Frederic Weisbecker
2015-05-19 7:17 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
5 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2015-05-12 14:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: LKML, Frederic Weisbecker, Ingo Molnar, Thomas Gleixner, Linus Torvalds
Now that PREEMPT_ACTIVE implies PREEMPT_DISABLE_OFFSET, ignoring
PREEMPT_ACTIVE from in_atomic() check isn't useful anymore.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
include/linux/preempt.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 4057696..a1a00e1 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -112,7 +112,7 @@
* used in the general case to determine whether sleeping is possible.
* Do not use in_atomic() in driver code.
*/
-#define in_atomic() ((preempt_count() & ~PREEMPT_ACTIVE) != 0)
+#define in_atomic() (preempt_count() != 0)
/*
* Check whether we were atomic before we did preempt_disable():
--
2.1.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [tip:sched/core] sched/preempt: Remove PREEMPT_ACTIVE unmasking off in_atomic()
2015-05-12 14:41 ` [PATCH 6/6] preempt: Remove PREEMPT_ACTIVE unmasking off in_atomic() Frederic Weisbecker
@ 2015-05-19 7:17 ` tip-bot for Frederic Weisbecker
0 siblings, 0 replies; 17+ messages in thread
From: tip-bot for Frederic Weisbecker @ 2015-05-19 7:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: torvalds, hpa, peterz, linux-kernel, mingo, tglx, fweisbec
Commit-ID: 3e51f3c4004c9b01f66da03214a3e206f5ed627b
Gitweb: http://git.kernel.org/tip/3e51f3c4004c9b01f66da03214a3e206f5ed627b
Author: Frederic Weisbecker <fweisbec@gmail.com>
AuthorDate: Tue, 12 May 2015 16:41:51 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 19 May 2015 08:39:13 +0200
sched/preempt: Remove PREEMPT_ACTIVE unmasking off in_atomic()
Now that PREEMPT_ACTIVE implies PREEMPT_DISABLE_OFFSET, ignoring
PREEMPT_ACTIVE from in_atomic() check isn't useful anymore.
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1431441711-29753-7-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/linux/preempt.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 4057696..a1a00e1 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -112,7 +112,7 @@
* used in the general case to determine whether sleeping is possible.
* Do not use in_atomic() in driver code.
*/
-#define in_atomic() ((preempt_count() & ~PREEMPT_ACTIVE) != 0)
+#define in_atomic() (preempt_count() != 0)
/*
* Check whether we were atomic before we did preempt_disable():
^ permalink raw reply related [flat|nested] 17+ messages in thread