[RFC PATCH 0/9] printk: Make it usable on nohz CPUs v3

[RFC PATCH 0/9] printk: Make it usable on nohz CPUs v3

[Frederic Weisbecker]

Hi,

Changes since v2:

* Use an irq_work flag instead of a queue parameter to define whether
we want an IPI or not. This way it becomes a work property that can't
change during its lifecyle. It more simple that way.

* Fix some build errors reported by Wu Fengguang.

* Send an IPI even on non-empty queues, in case it's only filled with lazy
works.

* Fix a race when claiming is concurrent with another CPU.

Only boot-tested on x86 for now.

Testing can be made by pulling:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
 	nohz/printk-v3

Thanks.

Frederic Weisbecker (9):
  irq_work: Fix racy check on work pending flag
  irq_work: Remove CONFIG_HAVE_IRQ_WORK
  irq_work: Move irq_work_raise() declaration/default definition to
    arch headers
  irq_work: Let the arch tell us about self-IPI support
  x86: Implement arch_irq_work_has_ipi()
  nohz: Add API to check tick state
  irq_work: Make self-IPIs optable
  irq_work: Handle queuing without IPI support in dyntick idle mode
  printk: Wake up klogd using irq_work

 arch/alpha/Kconfig                     |    1 -
 arch/alpha/include/asm/irq_work.h      |    9 +++
 arch/alpha/kernel/time.c               |    2 +-
 arch/arm/Kconfig                       |    1 -
 arch/arm/include/asm/irq_work.h        |    1 +
 arch/arm64/Kconfig                     |    1 -
 arch/arm64/include/asm/irq_work.h      |    1 +
 arch/avr32/include/asm/irq_work.h      |    1 +
 arch/blackfin/Kconfig                  |    1 -
 arch/blackfin/include/asm/irq_work.h   |    1 +
 arch/c6x/include/asm/irq_work.h        |    1 +
 arch/cris/include/asm/irq_work.h       |    1 +
 arch/frv/Kconfig                       |    1 -
 arch/frv/include/asm/irq_work.h        |    1 +
 arch/h8300/include/asm/irq_work.h      |    1 +
 arch/hexagon/Kconfig                   |    1 -
 arch/hexagon/include/asm/irq_work.h    |    1 +
 arch/ia64/include/asm/irq_work.h       |    1 +
 arch/m32r/include/asm/irq_work.h       |    1 +
 arch/m68k/include/asm/irq_work.h       |    1 +
 arch/microblaze/include/asm/irq_work.h |    1 +
 arch/mips/Kconfig                      |    1 -
 arch/mips/include/asm/irq_work.h       |    1 +
 arch/mn10300/include/asm/irq_work.h    |    1 +
 arch/openrisc/include/asm/irq_work.h   |    1 +
 arch/parisc/Kconfig                    |    1 -
 arch/parisc/include/asm/irq_work.h     |    1 +
 arch/powerpc/Kconfig                   |    1 -
 arch/powerpc/include/asm/irq_work.h    |    8 +++
 arch/powerpc/kernel/time.c             |    2 +-
 arch/s390/Kconfig                      |    1 -
 arch/s390/include/asm/irq_work.h       |    1 +
 arch/score/include/asm/irq_work.h      |    1 +
 arch/sh/Kconfig                        |    1 -
 arch/sh/include/asm/irq_work.h         |    1 +
 arch/sparc/Kconfig                     |    1 -
 arch/sparc/include/asm/irq_work.h      |    8 +++
 arch/sparc/kernel/pcr.c                |    2 +-
 arch/tile/include/asm/irq_work.h       |    1 +
 arch/um/include/asm/irq_work.h         |    1 +
 arch/unicore32/include/asm/irq_work.h  |    1 +
 arch/x86/Kconfig                       |    1 -
 arch/x86/include/asm/irq_work.h        |   15 +++++
 arch/x86/kernel/irq_work.c             |    6 +-
 arch/xtensa/include/asm/irq_work.h     |    1 +
 drivers/staging/iio/trigger/Kconfig    |    1 -
 include/asm-generic/irq_work.h         |   23 ++++++++
 include/linux/irq_work.h               |   32 ++++++++++++
 include/linux/printk.h                 |    3 -
 include/linux/tick.h                   |   17 ++++++-
 init/Kconfig                           |    5 +--
 kernel/irq_work.c                      |   88 ++++++++++++++++++++++----------
 kernel/printk.c                        |   12 ++---
 kernel/time/tick-sched.c               |    7 ++-
 kernel/timer.c                         |    1 -
 55 files changed, 211 insertions(+), 67 deletions(-)
 create mode 100644 arch/alpha/include/asm/irq_work.h
 create mode 100644 arch/arm/include/asm/irq_work.h
 create mode 100644 arch/arm64/include/asm/irq_work.h
 create mode 100644 arch/avr32/include/asm/irq_work.h
 create mode 100644 arch/blackfin/include/asm/irq_work.h
 create mode 100644 arch/c6x/include/asm/irq_work.h
 create mode 100644 arch/cris/include/asm/irq_work.h
 create mode 100644 arch/frv/include/asm/irq_work.h
 create mode 100644 arch/h8300/include/asm/irq_work.h
 create mode 100644 arch/hexagon/include/asm/irq_work.h
 create mode 100644 arch/ia64/include/asm/irq_work.h
 create mode 100644 arch/m32r/include/asm/irq_work.h
 create mode 100644 arch/m68k/include/asm/irq_work.h
 create mode 100644 arch/microblaze/include/asm/irq_work.h
 create mode 100644 arch/mips/include/asm/irq_work.h
 create mode 100644 arch/mn10300/include/asm/irq_work.h
 create mode 100644 arch/openrisc/include/asm/irq_work.h
 create mode 100644 arch/parisc/include/asm/irq_work.h
 create mode 100644 arch/powerpc/include/asm/irq_work.h
 create mode 100644 arch/s390/include/asm/irq_work.h
 create mode 100644 arch/score/include/asm/irq_work.h
 create mode 100644 arch/sh/include/asm/irq_work.h
 create mode 100644 arch/sparc/include/asm/irq_work.h
 create mode 100644 arch/tile/include/asm/irq_work.h
 create mode 100644 arch/um/include/asm/irq_work.h
 create mode 100644 arch/unicore32/include/asm/irq_work.h
 create mode 100644 arch/x86/include/asm/irq_work.h
 create mode 100644 arch/xtensa/include/asm/irq_work.h
 create mode 100644 include/asm-generic/irq_work.h

-- 
1.7.5.4

[RFC PATCH 1/9] irq_work: Fix racy check on work pending flag

[Frederic Weisbecker]

Context requirements on irq work claim are not entirely
clear. But it appears that we can try to claim a work that
may be already claimed by another CPU.

If so then the early check on IRQ_WORK_PENDING in
irq_work_claim() is racy because another CPU may be
changing the flags concurrently and we have nothing
to synchronize against that. So the value we deal with
may be stale for a while already.

To fix this, start with our best wish as the initial
value for the work flags and feed cmpxchg with it. But
only do the check against IRQ_WORK_PENDING flag with the
cmpxchg result.

Nonetheless, if the work is not pending but our best wish
was wrong, restart with the old value returned by cmpxchg.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 kernel/irq_work.c |   17 ++++++++++++-----
 1 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 1588e3b..679c13e 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -34,15 +34,22 @@ static DEFINE_PER_CPU(struct llist_head, irq_work_list);
  */
 static bool irq_work_claim(struct irq_work *work)
 {
-	unsigned long flags, nflags;
+	unsigned long flags, oflags, nflags;
 
+	/*
+	 * Can't check IRQ_WORK_PENDING bit right now because the work
+	 * can be running on another CPU and we are not sync with its
+	 * changes to work flags. Only cmpxchg can reliably check for us.
+	 */
+	flags = work->flags & ~IRQ_WORK_PENDING;
 	for (;;) {
-		flags = work->flags;
-		if (flags & IRQ_WORK_PENDING)
-			return false;
 		nflags = flags | IRQ_WORK_FLAGS;
-		if (cmpxchg(&work->flags, flags, nflags) == flags)
+		oflags = cmpxchg(&work->flags, flags, nflags);
+		if (oflags == flags)
 			break;
+		if (oflags & IRQ_WORK_PENDING)
+			return false;
+		flags = oflags;
 		cpu_relax();
 	}
 
-- 
1.7.5.4

[RFC PATCH 2/9] irq_work: Remove CONFIG_HAVE_IRQ_WORK

[Frederic Weisbecker]

irq work is supposed to work everywhere because of the irq work
hook in the generic timer tick function.

I might be missing something though...

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 arch/alpha/Kconfig                  |    1 -
 arch/arm/Kconfig                    |    1 -
 arch/arm64/Kconfig                  |    1 -
 arch/blackfin/Kconfig               |    1 -
 arch/frv/Kconfig                    |    1 -
 arch/hexagon/Kconfig                |    1 -
 arch/mips/Kconfig                   |    1 -
 arch/parisc/Kconfig                 |    1 -
 arch/powerpc/Kconfig                |    1 -
 arch/s390/Kconfig                   |    1 -
 arch/sh/Kconfig                     |    1 -
 arch/sparc/Kconfig                  |    1 -
 arch/x86/Kconfig                    |    1 -
 drivers/staging/iio/trigger/Kconfig |    1 -
 init/Kconfig                        |    4 ----
 15 files changed, 0 insertions(+), 18 deletions(-)

diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig
index 7da9124..ea86275 100644
--- a/arch/alpha/Kconfig
+++ b/arch/alpha/Kconfig
@@ -5,7 +5,6 @@ config ALPHA
 	select HAVE_IDE
 	select HAVE_OPROFILE
 	select HAVE_SYSCALL_WRAPPERS
-	select HAVE_IRQ_WORK
 	select HAVE_PCSPKR_PLATFORM
 	select HAVE_PERF_EVENTS
 	select HAVE_DMA_ATTRS
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 767aae8..44bdbfe 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -30,7 +30,6 @@ config ARM
 	select HAVE_KERNEL_LZO
 	select HAVE_KERNEL_LZMA
 	select HAVE_KERNEL_XZ
-	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select PERF_USE_VMALLOC
 	select HAVE_REGS_AND_STACK_ACCESS_API
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 7ff68c9..efa0627 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -17,7 +17,6 @@ config ARM64
 	select HAVE_GENERIC_DMA_COHERENT
 	select HAVE_GENERIC_HARDIRQS
 	select HAVE_HW_BREAKPOINT if PERF_EVENTS
-	select HAVE_IRQ_WORK
 	select HAVE_MEMBLOCK
 	select HAVE_PERF_EVENTS
 	select HAVE_SPARSE_IRQ
diff --git a/arch/blackfin/Kconfig b/arch/blackfin/Kconfig
index ccd9193..9c588f7 100644
--- a/arch/blackfin/Kconfig
+++ b/arch/blackfin/Kconfig
@@ -24,7 +24,6 @@ config BLACKFIN
 	select HAVE_FUNCTION_TRACER
 	select HAVE_FUNCTION_TRACE_MCOUNT_TEST
 	select HAVE_IDE
-	select HAVE_IRQ_WORK
 	select HAVE_KERNEL_GZIP if RAMKERNEL
 	select HAVE_KERNEL_BZIP2 if RAMKERNEL
 	select HAVE_KERNEL_LZMA if RAMKERNEL
diff --git a/arch/frv/Kconfig b/arch/frv/Kconfig
index 9d26264..17df48f 100644
--- a/arch/frv/Kconfig
+++ b/arch/frv/Kconfig
@@ -3,7 +3,6 @@ config FRV
 	default y
 	select HAVE_IDE
 	select HAVE_ARCH_TRACEHOOK
-	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select HAVE_UID16
 	select HAVE_GENERIC_HARDIRQS
diff --git a/arch/hexagon/Kconfig b/arch/hexagon/Kconfig
index b2fdfb7..8a902a7 100644
--- a/arch/hexagon/Kconfig
+++ b/arch/hexagon/Kconfig
@@ -14,7 +14,6 @@ config HEXAGON
 	# select HAVE_CLK
 	# select IRQ_PER_CPU
 	# select GENERIC_PENDING_IRQ if SMP
-	select HAVE_IRQ_WORK
 	select GENERIC_ATOMIC64
 	select HAVE_PERF_EVENTS
 	select HAVE_GENERIC_HARDIRQS
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 35453ea..045bf4d 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -4,7 +4,6 @@ config MIPS
 	select HAVE_GENERIC_DMA_COHERENT
 	select HAVE_IDE
 	select HAVE_OPROFILE
-	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select PERF_USE_VMALLOC
 	select HAVE_ARCH_KGDB
diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index b87438b..abbdb7a 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -9,7 +9,6 @@ config PARISC
 	select RTC_DRV_GENERIC
 	select INIT_ALL_POSSIBLE
 	select BUG
-	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select GENERIC_ATOMIC64 if !64BIT
 	select HAVE_GENERIC_HARDIRQS
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index df7edb8..c071bcb 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -118,7 +118,6 @@ config PPC
 	select HAVE_SYSCALL_WRAPPERS if PPC64
 	select GENERIC_ATOMIC64 if PPC32
 	select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE
-	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_HW_BREAKPOINT if PERF_EVENTS && PPC_BOOK3S_64
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 99d2d79..d3e251b 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -78,7 +78,6 @@ config S390
 	select HAVE_KVM if 64BIT
 	select HAVE_ARCH_TRACEHOOK
 	select INIT_ALL_POSSIBLE
-	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select ARCH_HAVE_NMI_SAFE_CMPXCHG
 	select HAVE_DEBUG_KMEMLEAK
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index 3b3e27a..8dc67fd 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -11,7 +11,6 @@ config SUPERH
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_DMA_API_DEBUG
 	select HAVE_DMA_ATTRS
-	select HAVE_IRQ_WORK
 	select HAVE_PERF_EVENTS
 	select HAVE_DEBUG_BUGVERBOSE
 	select ARCH_HAVE_CUSTOM_GPIO_H
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 91c780c..17b3c9f 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -22,7 +22,6 @@ config SPARC
 	select ARCH_WANT_OPTIONAL_GPIOLIB
 	select RTC_CLASS
 	select RTC_DRV_M48T59
-	select HAVE_IRQ_WORK
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
 	select HAVE_ARCH_JUMP_LABEL
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 42d2c35..bf1821b 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -26,7 +26,6 @@ config X86
 	select HAVE_OPROFILE
 	select HAVE_PCSPKR_PLATFORM
 	select HAVE_PERF_EVENTS
-	select HAVE_IRQ_WORK
 	select HAVE_IOREMAP_PROT
 	select HAVE_KPROBES
 	select HAVE_MEMBLOCK
diff --git a/drivers/staging/iio/trigger/Kconfig b/drivers/staging/iio/trigger/Kconfig
index 7d32075..d44d3ad 100644
--- a/drivers/staging/iio/trigger/Kconfig
+++ b/drivers/staging/iio/trigger/Kconfig
@@ -21,7 +21,6 @@ config IIO_GPIO_TRIGGER
 config IIO_SYSFS_TRIGGER
 	tristate "SYSFS trigger"
 	depends on SYSFS
-	depends on HAVE_IRQ_WORK
 	select IRQ_WORK
 	help
 	  Provides support for using SYSFS entry as IIO triggers.
diff --git a/init/Kconfig b/init/Kconfig
index 4c93533..8ba9fc3 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -20,12 +20,8 @@ config CONSTRUCTORS
 	bool
 	depends on !UML
 
-config HAVE_IRQ_WORK
-	bool
-
 config IRQ_WORK
 	bool
-	depends on HAVE_IRQ_WORK
 
 config BUILDTIME_EXTABLE_SORT
 	bool
-- 
1.7.5.4

[RFC PATCH 3/9] irq_work: Move irq_work_raise() declaration/default definition to arch headers

[Frederic Weisbecker]

This optimization doesn't matter much. But this prepares the
arch headers that we need to add a new API in order to detect
when the arch can trigger self IPIs to implement the irq work.

This is necessary later to make printk working in nohz CPUs.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 arch/alpha/include/asm/irq_work.h      |    6 ++++++
 arch/arm/include/asm/irq_work.h        |    1 +
 arch/arm64/include/asm/irq_work.h      |    1 +
 arch/avr32/include/asm/irq_work.h      |    1 +
 arch/blackfin/include/asm/irq_work.h   |    1 +
 arch/c6x/include/asm/irq_work.h        |    1 +
 arch/cris/include/asm/irq_work.h       |    1 +
 arch/frv/include/asm/irq_work.h        |    1 +
 arch/h8300/include/asm/irq_work.h      |    1 +
 arch/hexagon/include/asm/irq_work.h    |    1 +
 arch/ia64/include/asm/irq_work.h       |    1 +
 arch/m32r/include/asm/irq_work.h       |    1 +
 arch/m68k/include/asm/irq_work.h       |    1 +
 arch/microblaze/include/asm/irq_work.h |    1 +
 arch/mips/include/asm/irq_work.h       |    1 +
 arch/mn10300/include/asm/irq_work.h    |    1 +
 arch/openrisc/include/asm/irq_work.h   |    1 +
 arch/parisc/include/asm/irq_work.h     |    1 +
 arch/powerpc/include/asm/irq_work.h    |    6 ++++++
 arch/s390/include/asm/irq_work.h       |    1 +
 arch/score/include/asm/irq_work.h      |    1 +
 arch/sh/include/asm/irq_work.h         |    1 +
 arch/sparc/include/asm/irq_work.h      |    6 ++++++
 arch/tile/include/asm/irq_work.h       |    1 +
 arch/um/include/asm/irq_work.h         |    1 +
 arch/unicore32/include/asm/irq_work.h  |    1 +
 arch/x86/include/asm/irq_work.h        |   10 ++++++++++
 arch/x86/kernel/irq_work.c             |    4 ++--
 arch/xtensa/include/asm/irq_work.h     |    1 +
 include/asm-generic/irq_work.h         |    9 +++++++++
 include/linux/irq_work.h               |    1 +
 kernel/irq_work.c                      |    7 -------
 32 files changed, 64 insertions(+), 9 deletions(-)
 create mode 100644 arch/alpha/include/asm/irq_work.h
 create mode 100644 arch/arm/include/asm/irq_work.h
 create mode 100644 arch/arm64/include/asm/irq_work.h
 create mode 100644 arch/avr32/include/asm/irq_work.h
 create mode 100644 arch/blackfin/include/asm/irq_work.h
 create mode 100644 arch/c6x/include/asm/irq_work.h
 create mode 100644 arch/cris/include/asm/irq_work.h
 create mode 100644 arch/frv/include/asm/irq_work.h
 create mode 100644 arch/h8300/include/asm/irq_work.h
 create mode 100644 arch/hexagon/include/asm/irq_work.h
 create mode 100644 arch/ia64/include/asm/irq_work.h
 create mode 100644 arch/m32r/include/asm/irq_work.h
 create mode 100644 arch/m68k/include/asm/irq_work.h
 create mode 100644 arch/microblaze/include/asm/irq_work.h
 create mode 100644 arch/mips/include/asm/irq_work.h
 create mode 100644 arch/mn10300/include/asm/irq_work.h
 create mode 100644 arch/openrisc/include/asm/irq_work.h
 create mode 100644 arch/parisc/include/asm/irq_work.h
 create mode 100644 arch/powerpc/include/asm/irq_work.h
 create mode 100644 arch/s390/include/asm/irq_work.h
 create mode 100644 arch/score/include/asm/irq_work.h
 create mode 100644 arch/sh/include/asm/irq_work.h
 create mode 100644 arch/sparc/include/asm/irq_work.h
 create mode 100644 arch/tile/include/asm/irq_work.h
 create mode 100644 arch/um/include/asm/irq_work.h
 create mode 100644 arch/unicore32/include/asm/irq_work.h
 create mode 100644 arch/x86/include/asm/irq_work.h
 create mode 100644 arch/xtensa/include/asm/irq_work.h
 create mode 100644 include/asm-generic/irq_work.h

diff --git a/arch/alpha/include/asm/irq_work.h b/arch/alpha/include/asm/irq_work.h
new file mode 100644
index 0000000..814ff3d
--- /dev/null
+++ b/arch/alpha/include/asm/irq_work.h
@@ -0,0 +1,6 @@
+#ifndef _ALPHA_IRQ_WORK_H
+#define _ALPHA_IRQ_WORK_H
+
+extern void arch_irq_work_raise(void);
+
+#endif
diff --git a/arch/arm/include/asm/irq_work.h b/arch/arm/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/arm/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/arm64/include/asm/irq_work.h b/arch/arm64/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/arm64/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/avr32/include/asm/irq_work.h b/arch/avr32/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/avr32/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/blackfin/include/asm/irq_work.h b/arch/blackfin/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/blackfin/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/c6x/include/asm/irq_work.h b/arch/c6x/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/c6x/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/cris/include/asm/irq_work.h b/arch/cris/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/cris/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/frv/include/asm/irq_work.h b/arch/frv/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/frv/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/h8300/include/asm/irq_work.h b/arch/h8300/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/h8300/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/hexagon/include/asm/irq_work.h b/arch/hexagon/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/hexagon/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/ia64/include/asm/irq_work.h b/arch/ia64/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/ia64/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/m32r/include/asm/irq_work.h b/arch/m32r/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/m32r/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/m68k/include/asm/irq_work.h b/arch/m68k/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/m68k/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/microblaze/include/asm/irq_work.h b/arch/microblaze/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/microblaze/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/mips/include/asm/irq_work.h b/arch/mips/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/mips/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/mn10300/include/asm/irq_work.h b/arch/mn10300/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/mn10300/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/openrisc/include/asm/irq_work.h b/arch/openrisc/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/openrisc/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/parisc/include/asm/irq_work.h b/arch/parisc/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/parisc/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/powerpc/include/asm/irq_work.h b/arch/powerpc/include/asm/irq_work.h
new file mode 100644
index 0000000..8b9927f
--- /dev/null
+++ b/arch/powerpc/include/asm/irq_work.h
@@ -0,0 +1,6 @@
+#ifndef _ASM_POWERPC_IRQ_WORK_H
+#define _ASM_POWERPC_IRQ_WORK_H
+
+extern void arch_irq_work_raise(void);
+
+#endif
diff --git a/arch/s390/include/asm/irq_work.h b/arch/s390/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/s390/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/score/include/asm/irq_work.h b/arch/score/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/score/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/sh/include/asm/irq_work.h b/arch/sh/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/sh/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/sparc/include/asm/irq_work.h b/arch/sparc/include/asm/irq_work.h
new file mode 100644
index 0000000..1d062a6
--- /dev/null
+++ b/arch/sparc/include/asm/irq_work.h
@@ -0,0 +1,6 @@
+#ifndef ___ASM_SPARC_IRQ_H
+#define ___ASM_SPARC_IRQ_H
+
+extern void arch_irq_work_raise(void);
+
+#endif
diff --git a/arch/tile/include/asm/irq_work.h b/arch/tile/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/tile/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/um/include/asm/irq_work.h b/arch/um/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/um/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/unicore32/include/asm/irq_work.h b/arch/unicore32/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/unicore32/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/arch/x86/include/asm/irq_work.h b/arch/x86/include/asm/irq_work.h
new file mode 100644
index 0000000..38eed96
--- /dev/null
+++ b/arch/x86/include/asm/irq_work.h
@@ -0,0 +1,10 @@
+#ifndef _ASM_X86_IRQ_WORK_H
+#define _ASM_X86_IRQ_WORK_H
+
+#ifndef CONFIG_X86_LOCAL_APIC
+#include <asm-generic/irq_work.h>
+# else
+extern void arch_irq_work_raise(void);
+#endif
+
+#endif
diff --git a/arch/x86/kernel/irq_work.c b/arch/x86/kernel/irq_work.c
index ca8f703..95f5d4e 100644
--- a/arch/x86/kernel/irq_work.c
+++ b/arch/x86/kernel/irq_work.c
@@ -18,13 +18,13 @@ void smp_irq_work_interrupt(struct pt_regs *regs)
 	irq_exit();
 }
 
+#ifdef CONFIG_X86_LOCAL_APIC
 void arch_irq_work_raise(void)
 {
-#ifdef CONFIG_X86_LOCAL_APIC
 	if (!cpu_has_apic)
 		return;
 
 	apic->send_IPI_self(IRQ_WORK_VECTOR);
 	apic_wait_icr_idle();
-#endif
 }
+#endif
diff --git a/arch/xtensa/include/asm/irq_work.h b/arch/xtensa/include/asm/irq_work.h
new file mode 100644
index 0000000..f1bffa2
--- /dev/null
+++ b/arch/xtensa/include/asm/irq_work.h
@@ -0,0 +1 @@
+#include <asm-generic/irq_work.h>
diff --git a/include/asm-generic/irq_work.h b/include/asm-generic/irq_work.h
new file mode 100644
index 0000000..a2d4108
--- /dev/null
+++ b/include/asm-generic/irq_work.h
@@ -0,0 +1,9 @@
+#ifndef __ASM_GENERIC_ATOMIC_H
+#define __ASM_GENERIC_ATOMIC_H
+
+/*
+ * Lame architectures will get the timer tick callback
+ */
+static inline void arch_irq_work_raise(void) { }
+
+#endif
diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
index 6a9e8f5..b39ea0b 100644
--- a/include/linux/irq_work.h
+++ b/include/linux/irq_work.h
@@ -2,6 +2,7 @@
 #define _LINUX_IRQ_WORK_H
 
 #include <linux/llist.h>
+#include <asm/irq_work.h>
 
 struct irq_work {
 	unsigned long flags;
diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 679c13e..d00011c 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -56,13 +56,6 @@ static bool irq_work_claim(struct irq_work *work)
 	return true;
 }
 
-void __weak arch_irq_work_raise(void)
-{
-	/*
-	 * Lame architectures will get the timer tick callback
-	 */
-}
-
 /*
  * Queue the entry and raise the IPI if needed.
  */
-- 
1.7.5.4

[RFC PATCH 4/9] irq_work: Let the arch tell us about self-IPI support

[Frederic Weisbecker]

This prepares us to make printk working on nohz CPUs
using irq work.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 arch/alpha/include/asm/irq_work.h   |    5 ++++-
 arch/alpha/kernel/time.c            |    2 +-
 arch/powerpc/include/asm/irq_work.h |    4 +++-
 arch/powerpc/kernel/time.c          |    2 +-
 arch/sparc/include/asm/irq_work.h   |    4 +++-
 arch/sparc/kernel/pcr.c             |    2 +-
 arch/x86/include/asm/irq_work.h     |    9 +++++----
 arch/x86/kernel/irq_work.c          |    2 +-
 include/asm-generic/irq_work.h      |   14 ++++++++++++++
 9 files changed, 33 insertions(+), 11 deletions(-)

diff --git a/arch/alpha/include/asm/irq_work.h b/arch/alpha/include/asm/irq_work.h
index 814ff3d..3d32132 100644
--- a/arch/alpha/include/asm/irq_work.h
+++ b/arch/alpha/include/asm/irq_work.h
@@ -1,6 +1,9 @@
 #ifndef _ALPHA_IRQ_WORK_H
 #define _ALPHA_IRQ_WORK_H
 
-extern void arch_irq_work_raise(void);
+extern void __arch_irq_work_raise(void);
+#define arch_irq_work_raise __arch_irq_work_raise
+
+#include <asm-generic/irq_work.h>
 
 #endif
diff --git a/arch/alpha/kernel/time.c b/arch/alpha/kernel/time.c
index e336694..91c5eec 100644
--- a/arch/alpha/kernel/time.c
+++ b/arch/alpha/kernel/time.c
@@ -90,7 +90,7 @@ DEFINE_PER_CPU(u8, irq_work_pending);
 #define test_irq_work_pending()      __get_cpu_var(irq_work_pending)
 #define clear_irq_work_pending()     __get_cpu_var(irq_work_pending) = 0
 
-void arch_irq_work_raise(void)
+void __arch_irq_work_raise(void)
 {
 	set_irq_work_pending_flag();
 }
diff --git a/arch/powerpc/include/asm/irq_work.h b/arch/powerpc/include/asm/irq_work.h
index 8b9927f..8aa36aa 100644
--- a/arch/powerpc/include/asm/irq_work.h
+++ b/arch/powerpc/include/asm/irq_work.h
@@ -1,6 +1,8 @@
 #ifndef _ASM_POWERPC_IRQ_WORK_H
 #define _ASM_POWERPC_IRQ_WORK_H
 
-extern void arch_irq_work_raise(void);
+extern void __arch_irq_work_raise(void);
+#define arch_irq_work_raise __arch_irq_work_raise
 
+#include <asm-generic/irq_work.h>
 #endif
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index c9986fd..31565ac 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -466,7 +466,7 @@ DEFINE_PER_CPU(u8, irq_work_pending);
 
 #endif /* 32 vs 64 bit */
 
-void arch_irq_work_raise(void)
+void __arch_irq_work_raise(void)
 {
 	preempt_disable();
 	set_irq_work_pending_flag();
diff --git a/arch/sparc/include/asm/irq_work.h b/arch/sparc/include/asm/irq_work.h
index 1d062a6..383772d 100644
--- a/arch/sparc/include/asm/irq_work.h
+++ b/arch/sparc/include/asm/irq_work.h
@@ -1,6 +1,8 @@
 #ifndef ___ASM_SPARC_IRQ_H
 #define ___ASM_SPARC_IRQ_H
 
-extern void arch_irq_work_raise(void);
+extern void __arch_irq_work_raise(void);
+#define arch_irq_work_raise __arch_irq_work_raise
 
+#include <asm-generic/irq_work.h>
 #endif
diff --git a/arch/sparc/kernel/pcr.c b/arch/sparc/kernel/pcr.c
index 269af58..d1e1ecf 100644
--- a/arch/sparc/kernel/pcr.c
+++ b/arch/sparc/kernel/pcr.c
@@ -43,7 +43,7 @@ void __irq_entry deferred_pcr_work_irq(int irq, struct pt_regs *regs)
 	set_irq_regs(old_regs);
 }
 
-void arch_irq_work_raise(void)
+void __arch_irq_work_raise(void)
 {
 	set_softint(1 << PIL_DEFERRED_PCR_WORK);
 }
diff --git a/arch/x86/include/asm/irq_work.h b/arch/x86/include/asm/irq_work.h
index 38eed96..dad8266 100644
--- a/arch/x86/include/asm/irq_work.h
+++ b/arch/x86/include/asm/irq_work.h
@@ -1,10 +1,11 @@
 #ifndef _ASM_X86_IRQ_WORK_H
 #define _ASM_X86_IRQ_WORK_H
 
-#ifndef CONFIG_X86_LOCAL_APIC
-#include <asm-generic/irq_work.h>
-# else
-extern void arch_irq_work_raise(void);
+#ifdef CONFIG_X86_LOCAL_APIC
+extern void __arch_irq_work_raise(void);
+#define arch_irq_work_raise __arch_irq_work_raise
 #endif
 
+#include <asm-generic/irq_work.h>
+
 #endif
diff --git a/arch/x86/kernel/irq_work.c b/arch/x86/kernel/irq_work.c
index 95f5d4e..7389d5e 100644
--- a/arch/x86/kernel/irq_work.c
+++ b/arch/x86/kernel/irq_work.c
@@ -19,7 +19,7 @@ void smp_irq_work_interrupt(struct pt_regs *regs)
 }
 
 #ifdef CONFIG_X86_LOCAL_APIC
-void arch_irq_work_raise(void)
+void __arch_irq_work_raise(void)
 {
 	if (!cpu_has_apic)
 		return;
diff --git a/include/asm-generic/irq_work.h b/include/asm-generic/irq_work.h
index a2d4108..e3e4f02 100644
--- a/include/asm-generic/irq_work.h
+++ b/include/asm-generic/irq_work.h
@@ -4,6 +4,20 @@
 /*
  * Lame architectures will get the timer tick callback
  */
+#ifndef arch_irq_work_raise
 static inline void arch_irq_work_raise(void) { }
+#endif
+
+/*
+ * Unless told otherwise, consider the arch doesn't implement irq work
+ * using self IPIs but through another way like defaulting to the hook
+ * on the sched tick.
+ */
+#ifndef arch_irq_work_has_ipi
+static inline bool arch_irq_work_has_ipi(void)
+{
+	return false;
+}
+#endif
 
 #endif
-- 
1.7.5.4

[RFC PATCH 5/9] x86: Implement arch_irq_work_has_ipi()

[Frederic Weisbecker]

Most of the time, x86 can trigger self-IPIs. Tell
irq work subsystem about it.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 arch/x86/include/asm/irq_work.h |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/irq_work.h b/arch/x86/include/asm/irq_work.h
index dad8266..c7489cd 100644
--- a/arch/x86/include/asm/irq_work.h
+++ b/arch/x86/include/asm/irq_work.h
@@ -2,8 +2,12 @@
 #define _ASM_X86_IRQ_WORK_H
 
 #ifdef CONFIG_X86_LOCAL_APIC
+#include <asm/cpufeature.h>
+
 extern void __arch_irq_work_raise(void);
 #define arch_irq_work_raise __arch_irq_work_raise
+
+#define arch_irq_work_has_ipi() (cpu_has_apic)
 #endif
 
 #include <asm-generic/irq_work.h>
-- 
1.7.5.4

[RFC PATCH 6/9] nohz: Add API to check tick state

[Frederic Weisbecker]

We need some quick way to check if the CPU has stopped
its tick. This will be useful to implement the printk tick
using the irq work subsystem.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 include/linux/tick.h     |   17 ++++++++++++++++-
 kernel/time/tick-sched.c |    2 +-
 2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/include/linux/tick.h b/include/linux/tick.h
index f37fceb..2307dd3 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -8,6 +8,8 @@
 
 #include <linux/clockchips.h>
 #include <linux/irqflags.h>
+#include <linux/percpu.h>
+#include <linux/hrtimer.h>
 
 #ifdef CONFIG_GENERIC_CLOCKEVENTS
 
@@ -122,13 +124,26 @@ static inline int tick_oneshot_mode_active(void) { return 0; }
 #endif /* !CONFIG_GENERIC_CLOCKEVENTS */
 
 # ifdef CONFIG_NO_HZ
+DECLARE_PER_CPU(struct tick_sched, tick_cpu_sched);
+
+static inline int tick_nohz_tick_stopped(void)
+{
+	return __this_cpu_read(tick_cpu_sched.tick_stopped);
+}
+
 extern void tick_nohz_idle_enter(void);
 extern void tick_nohz_idle_exit(void);
 extern void tick_nohz_irq_exit(void);
 extern ktime_t tick_nohz_get_sleep_length(void);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
-# else
+
+# else /* !CONFIG_NO_HZ */
+static inline int tick_nohz_tick_stopped(void)
+{
+	return 0;
+}
+
 static inline void tick_nohz_idle_enter(void) { }
 static inline void tick_nohz_idle_exit(void) { }
 
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index f423bdd..ccc1971 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -28,7 +28,7 @@
 /*
  * Per cpu nohz control structure
  */
-static DEFINE_PER_CPU(struct tick_sched, tick_cpu_sched);
+DEFINE_PER_CPU(struct tick_sched, tick_cpu_sched);
 
 /*
  * The time, when the last jiffy update happened. Protected by xtime_lock.
-- 
1.7.5.4

[RFC PATCH 7/9] irq_work: Make self-IPIs optable

[Frederic Weisbecker]

On irq work initialization, let the user choose to define it
as "lazy" or not. "Lazy" means that we don't want to send
an IPI (provided the arch can anyway) when we enqueue this
work but we rather prefer to wait for the next timer tick
to execute our work if possible.

This is going to be a benefit for non-urgent enqueuers
(like printk in the future) that may prefer not to raise
an IPI storm in case of frequent enqueuing on short periods
of time.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 include/linux/irq_work.h |   31 ++++++++++++++++++++++++++
 kernel/irq_work.c        |   53 ++++++++++++++++++++++++++++++++-------------
 kernel/time/tick-sched.c |    3 +-
 3 files changed, 70 insertions(+), 17 deletions(-)

diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
index b39ea0b..7b60c87 100644
--- a/include/linux/irq_work.h
+++ b/include/linux/irq_work.h
@@ -4,6 +4,20 @@
 #include <linux/llist.h>
 #include <asm/irq_work.h>
 
+/*
+ * An entry can be in one of four states:
+ *
+ * free	     NULL, 0 -> {claimed}       : free to be used
+ * claimed   NULL, 3 -> {pending}       : claimed to be enqueued
+ * pending   next, 3 -> {busy}          : queued, pending callback
+ * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
+ */
+
+#define IRQ_WORK_PENDING	1UL
+#define IRQ_WORK_BUSY		2UL
+#define IRQ_WORK_FLAGS		3UL
+#define IRQ_WORK_LAZY		4UL /* Doesn't want IPI, wait for tick */
+
 struct irq_work {
 	unsigned long flags;
 	struct llist_node llnode;
@@ -17,8 +31,25 @@ void init_irq_work(struct irq_work *work, void (*func)(struct irq_work *))
 	work->func = func;
 }
 
+#define DEFINE_IRQ_WORK(w, f)	\
+	struct irq_work w = {	\
+		.func = f,	\
+	}
+
+#define DEFINE_IRQ_WORK_LAZY(w, f)	\
+	struct irq_work w = {		\
+		.flags = IRQ_WORK_LAZY, \
+		.func = f,		\
+	}
+
 bool irq_work_queue(struct irq_work *work);
 void irq_work_run(void);
 void irq_work_sync(struct irq_work *work);
 
+#ifdef CONFIG_IRQ_WORK
+bool irq_work_needs_cpu(void);
+#else
+static bool irq_work_needs_cpu(void) { return false; }
+#endif
+
 #endif /* _LINUX_IRQ_WORK_H */
diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index d00011c..ce72b20 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -12,20 +12,10 @@
 #include <linux/percpu.h>
 #include <linux/hardirq.h>
 #include <linux/irqflags.h>
+#include <linux/tick.h>
+#include <linux/sched.h>
 #include <asm/processor.h>
 
-/*
- * An entry can be in one of four states:
- *
- * free	     NULL, 0 -> {claimed}       : free to be used
- * claimed   NULL, 3 -> {pending}       : claimed to be enqueued
- * pending   next, 3 -> {busy}          : queued, pending callback
- * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
- */
-
-#define IRQ_WORK_PENDING	1UL
-#define IRQ_WORK_BUSY		2UL
-#define IRQ_WORK_FLAGS		3UL
 
 static DEFINE_PER_CPU(struct llist_head, irq_work_list);
 
@@ -66,10 +56,28 @@ static void __irq_work_queue(struct irq_work *work)
 	preempt_disable();
 
 	empty = llist_add(&work->llnode, &__get_cpu_var(irq_work_list));
-	/* The list was empty, raise self-interrupt to start processing. */
-	if (empty)
+
+	/*
+	 * In any case, raise an IPI if requested and possible in case
+	 * the queue is empty or it's filled with lazy works.
+	 */
+	if (!(work->flags & IRQ_WORK_LAZY) && arch_irq_work_has_ipi()) {
 		arch_irq_work_raise();
+		goto out;
+	}
 
+	if (empty) {
+		/*
+		 * If the arch uses some other obscure way than IPI to raise
+		 * an irq work, just raise and don't think further.
+		 * Now if it can send an IPI, although not requested, and the tick
+		 * is stopped, send it nonetheless otherwise we may wait a long while
+		 * before that lazy work can run.
+		 */
+		if (!arch_irq_work_has_ipi() || tick_nohz_tick_stopped())
+			arch_irq_work_raise();
+	}
+out:
 	preempt_enable();
 }
 
@@ -93,12 +101,24 @@ bool irq_work_queue(struct irq_work *work)
 }
 EXPORT_SYMBOL_GPL(irq_work_queue);
 
+bool irq_work_needs_cpu(void)
+{
+	struct llist_head *this_list;
+
+	this_list = &__get_cpu_var(irq_work_list);
+	if (llist_empty(this_list))
+		return false;
+
+	return true;
+}
+
 /*
  * Run the irq_work entries on this cpu. Requires to be ran from hardirq
  * context with local IRQs disabled.
  */
 void irq_work_run(void)
 {
+	unsigned long flags;
 	struct irq_work *work;
 	struct llist_head *this_list;
 	struct llist_node *llnode;
@@ -120,13 +140,14 @@ void irq_work_run(void)
 		 * Clear the PENDING bit, after this point the @work
 		 * can be re-used.
 		 */
-		work->flags = IRQ_WORK_BUSY;
+		flags = work->flags & ~IRQ_WORK_PENDING;
+		work->flags = flags;
 		work->func(work);
 		/*
 		 * Clear the BUSY bit and return to the free state if
 		 * no-one else claimed it meanwhile.
 		 */
-		(void)cmpxchg(&work->flags, IRQ_WORK_BUSY, 0);
+		(void)cmpxchg(&work->flags, flags, flags & ~IRQ_WORK_BUSY);
 	}
 }
 EXPORT_SYMBOL_GPL(irq_work_run);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index ccc1971..7ac2fd8 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -20,6 +20,7 @@
 #include <linux/profile.h>
 #include <linux/sched.h>
 #include <linux/module.h>
+#include <linux/irq_work.h>
 
 #include <asm/irq_regs.h>
 
@@ -289,7 +290,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu, &rcu_delta_jiffies) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+	    arch_needs_cpu(cpu) || irq_work_needs_cpu()) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {
-- 
1.7.5.4

[RFC PATCH 8/9] irq_work: Handle queuing without IPI support in dyntick idle mode

[Frederic Weisbecker]

If we enqueue a work while in dyntick idle mode and the arch doesn't
have self-IPI support, we may not find an opportunity to run the work
before a while.

In this case, exit the idle loop to re-evaluate irq_work_needs_cpu()
and restart the tick.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 kernel/irq_work.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index ce72b20..b500271 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -76,6 +76,17 @@ static void __irq_work_queue(struct irq_work *work)
 		 */
 		if (!arch_irq_work_has_ipi() || tick_nohz_tick_stopped())
 			arch_irq_work_raise();
+
+		/*
+		 * If we rely on the timer tick or some obscure way to run the work
+		 * while the CPU is in dyntick idle mode, we may not have an opportunity
+		 * to do so before a while. Let's just exit the idle loop and hope we
+		 * haven't yet reached the last need_resched() check before the CPU goes
+		 * to low power mode.
+		 */
+		if (!arch_irq_work_has_ipi() && tick_nohz_tick_stopped()
+		    && is_idle_task(current))
+			set_need_resched();
 	}
 out:
 	preempt_enable();
-- 
1.7.5.4

[RFC PATCH 9/9] printk: Wake up klogd using irq_work

[Frederic Weisbecker]

klogd is woken up asynchronously from the tick in order
to do it safely.

However if printk is called when the tick is stopped, the reader
won't be woken up until the next interrupt, which might not fire
before a while. As a result, the user may miss some message.

To fix this, lets implement the printk tick using a lazy irq work.
This subsystem takes care of the timer tick state and can
fix up accordingly.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 include/linux/printk.h   |    3 ---
 init/Kconfig             |    1 +
 kernel/printk.c          |   12 +++++-------
 kernel/time/tick-sched.c |    2 +-
 kernel/timer.c           |    1 -
 5 files changed, 7 insertions(+), 12 deletions(-)

diff --git a/include/linux/printk.h b/include/linux/printk.h
index 9afc01e..86c4b62 100644
--- a/include/linux/printk.h
+++ b/include/linux/printk.h
@@ -98,9 +98,6 @@ int no_printk(const char *fmt, ...)
 extern asmlinkage __printf(1, 2)
 void early_printk(const char *fmt, ...);
 
-extern int printk_needs_cpu(int cpu);
-extern void printk_tick(void);
-
 #ifdef CONFIG_PRINTK
 asmlinkage __printf(5, 0)
 int vprintk_emit(int facility, int level,
diff --git a/init/Kconfig b/init/Kconfig
index 8ba9fc3..9c42e6a 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1184,6 +1184,7 @@ config HOTPLUG
 config PRINTK
 	default y
 	bool "Enable support for printk" if EXPERT
+	select IRQ_WORK
 	help
 	  This option enables normal printk support. Removing it
 	  eliminates most of the message strings from the kernel image
diff --git a/kernel/printk.c b/kernel/printk.c
index 66a2ea3..bb1b0d3 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -42,6 +42,7 @@
 #include <linux/notifier.h>
 #include <linux/rculist.h>
 #include <linux/poll.h>
+#include <linux/irq_work.h>
 
 #include <asm/uaccess.h>
 
@@ -1956,7 +1957,7 @@ int is_console_locked(void)
 static DEFINE_PER_CPU(int, printk_pending);
 static DEFINE_PER_CPU(char [PRINTK_BUF_SIZE], printk_sched_buf);
 
-void printk_tick(void)
+static void wake_up_klogd_work_func(struct irq_work *irq_work)
 {
 	if (__this_cpu_read(printk_pending)) {
 		int pending = __this_cpu_xchg(printk_pending, 0);
@@ -1969,17 +1970,14 @@ void printk_tick(void)
 	}
 }
 
-int printk_needs_cpu(int cpu)
-{
-	if (cpu_is_offline(cpu))
-		printk_tick();
-	return __this_cpu_read(printk_pending);
-}
+static DEFINE_IRQ_WORK_LAZY(wake_up_klogd_work, wake_up_klogd_work_func);
 
 void wake_up_klogd(void)
 {
 	if (waitqueue_active(&log_wait))
 		this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
+
+	irq_work_queue(&wake_up_klogd_work);
 }
 
 static void console_cont_flush(char *text, size_t size)
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 7ac2fd8..0efed16 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -289,7 +289,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
 		time_delta = timekeeping_max_deferment();
 	} while (read_seqretry(&xtime_lock, seq));
 
-	if (rcu_needs_cpu(cpu, &rcu_delta_jiffies) || printk_needs_cpu(cpu) ||
+	if (rcu_needs_cpu(cpu, &rcu_delta_jiffies) ||
 	    arch_needs_cpu(cpu) || irq_work_needs_cpu()) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
diff --git a/kernel/timer.c b/kernel/timer.c
index d5de1b2..5d6d0f1 100644
--- a/kernel/timer.c
+++ b/kernel/timer.c
@@ -1349,7 +1349,6 @@ void update_process_times(int user_tick)
 	account_process_tick(p, user_tick);
 	run_local_timers();
 	rcu_check_callbacks(cpu, user_tick);
-	printk_tick();
 #ifdef CONFIG_IRQ_WORK
 	if (in_irq())
 		irq_work_run();
-- 
1.7.5.4

Re: [RFC PATCH 1/9] irq_work: Fix racy check on work pending flag

[Steven Rostedt]

On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
> Context requirements on irq work claim are not entirely
> clear. But it appears that we can try to claim a work that
> may be already claimed by another CPU.
> 
> If so then the early check on IRQ_WORK_PENDING in
> irq_work_claim() is racy because another CPU may be
> changing the flags concurrently and we have nothing
> to synchronize against that. So the value we deal with
> may be stale for a while already.
> 
> To fix this, start with our best wish as the initial
> value for the work flags and feed cmpxchg with it. But
> only do the check against IRQ_WORK_PENDING flag with the
> cmpxchg result.
> 
> Nonetheless, if the work is not pending but our best wish
> was wrong, restart with the old value returned by cmpxchg.
> 
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
>  kernel/irq_work.c |   17 ++++++++++++-----
>  1 files changed, 12 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> index 1588e3b..679c13e 100644
> --- a/kernel/irq_work.c
> +++ b/kernel/irq_work.c
> @@ -34,15 +34,22 @@ static DEFINE_PER_CPU(struct llist_head, irq_work_list);
>   */
>  static bool irq_work_claim(struct irq_work *work)
>  {
> -	unsigned long flags, nflags;
> +	unsigned long flags, oflags, nflags;
>  
> +	/*
> +	 * Can't check IRQ_WORK_PENDING bit right now because the work
> +	 * can be running on another CPU and we are not sync with its
> +	 * changes to work flags. Only cmpxchg can reliably check for us.
> +	 */
> +	flags = work->flags & ~IRQ_WORK_PENDING;
>  	for (;;) {
> -		flags = work->flags;

I wonder if the bug is just a memory barrier missing here? But that also
suggests that the other CPU used a memory barrier too (or cmpxchg()
which implies one).

But this change looks fine too.

-- Steve

> -		if (flags & IRQ_WORK_PENDING)
> -			return false;
>  		nflags = flags | IRQ_WORK_FLAGS;
> -		if (cmpxchg(&work->flags, flags, nflags) == flags)
> +		oflags = cmpxchg(&work->flags, flags, nflags);
> +		if (oflags == flags)
>  			break;
> +		if (oflags & IRQ_WORK_PENDING)
> +			return false;
> +		flags = oflags;
>  		cpu_relax();
>  	}
>  

Re: [RFC PATCH 3/9] irq_work: Move irq_work_raise() declaration/default definition to arch headers

[Steven Rostedt]

On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
> This optimization doesn't matter much. But this prepares the
> arch headers that we need to add a new API in order to detect
> when the arch can trigger self IPIs to implement the irq work.
> 
> This is necessary later to make printk working in nohz CPUs.
> 
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
>  arch/alpha/include/asm/irq_work.h      |    6 ++++++
>  arch/arm/include/asm/irq_work.h        |    1 +
>  arch/arm64/include/asm/irq_work.h      |    1 +
>  arch/avr32/include/asm/irq_work.h      |    1 +
>  arch/blackfin/include/asm/irq_work.h   |    1 +
>  arch/c6x/include/asm/irq_work.h        |    1 +
>  arch/cris/include/asm/irq_work.h       |    1 +
>  arch/frv/include/asm/irq_work.h        |    1 +
>  arch/h8300/include/asm/irq_work.h      |    1 +
>  arch/hexagon/include/asm/irq_work.h    |    1 +
>  arch/ia64/include/asm/irq_work.h       |    1 +
>  arch/m32r/include/asm/irq_work.h       |    1 +
>  arch/m68k/include/asm/irq_work.h       |    1 +
>  arch/microblaze/include/asm/irq_work.h |    1 +
>  arch/mips/include/asm/irq_work.h       |    1 +
>  arch/mn10300/include/asm/irq_work.h    |    1 +
>  arch/openrisc/include/asm/irq_work.h   |    1 +
>  arch/parisc/include/asm/irq_work.h     |    1 +
>  arch/powerpc/include/asm/irq_work.h    |    6 ++++++
>  arch/s390/include/asm/irq_work.h       |    1 +
>  arch/score/include/asm/irq_work.h      |    1 +
>  arch/sh/include/asm/irq_work.h         |    1 +
>  arch/sparc/include/asm/irq_work.h      |    6 ++++++
>  arch/tile/include/asm/irq_work.h       |    1 +
>  arch/um/include/asm/irq_work.h         |    1 +
>  arch/unicore32/include/asm/irq_work.h  |    1 +
>  arch/x86/include/asm/irq_work.h        |   10 ++++++++++
>  arch/x86/kernel/irq_work.c             |    4 ++--
>  arch/xtensa/include/asm/irq_work.h     |    1 +
>  include/asm-generic/irq_work.h         |    9 +++++++++
>  include/linux/irq_work.h               |    1 +
>  kernel/irq_work.c                      |    7 -------
>  32 files changed, 64 insertions(+), 9 deletions(-)
>  create mode 100644 arch/alpha/include/asm/irq_work.h
>  create mode 100644 arch/arm/include/asm/irq_work.h
>  create mode 100644 arch/arm64/include/asm/irq_work.h
>  create mode 100644 arch/avr32/include/asm/irq_work.h
>  create mode 100644 arch/blackfin/include/asm/irq_work.h
>  create mode 100644 arch/c6x/include/asm/irq_work.h
>  create mode 100644 arch/cris/include/asm/irq_work.h
>  create mode 100644 arch/frv/include/asm/irq_work.h
>  create mode 100644 arch/h8300/include/asm/irq_work.h
>  create mode 100644 arch/hexagon/include/asm/irq_work.h
>  create mode 100644 arch/ia64/include/asm/irq_work.h
>  create mode 100644 arch/m32r/include/asm/irq_work.h
>  create mode 100644 arch/m68k/include/asm/irq_work.h
>  create mode 100644 arch/microblaze/include/asm/irq_work.h
>  create mode 100644 arch/mips/include/asm/irq_work.h
>  create mode 100644 arch/mn10300/include/asm/irq_work.h
>  create mode 100644 arch/openrisc/include/asm/irq_work.h
>  create mode 100644 arch/parisc/include/asm/irq_work.h
>  create mode 100644 arch/powerpc/include/asm/irq_work.h
>  create mode 100644 arch/s390/include/asm/irq_work.h
>  create mode 100644 arch/score/include/asm/irq_work.h
>  create mode 100644 arch/sh/include/asm/irq_work.h
>  create mode 100644 arch/sparc/include/asm/irq_work.h
>  create mode 100644 arch/tile/include/asm/irq_work.h
>  create mode 100644 arch/um/include/asm/irq_work.h
>  create mode 100644 arch/unicore32/include/asm/irq_work.h
>  create mode 100644 arch/x86/include/asm/irq_work.h
>  create mode 100644 arch/xtensa/include/asm/irq_work.h
>  create mode 100644 include/asm-generic/irq_work.h
> 
> diff --git a/arch/alpha/include/asm/irq_work.h b/arch/alpha/include/asm/irq_work.h
> new file mode 100644
> index 0000000..814ff3d
> --- /dev/null
> +++ b/arch/alpha/include/asm/irq_work.h
> @@ -0,0 +1,6 @@
> +#ifndef _ALPHA_IRQ_WORK_H
> +#define _ALPHA_IRQ_WORK_H
> +
> +extern void arch_irq_work_raise(void);
> +
> +#endif
> diff --git a/arch/arm/include/asm/irq_work.h b/arch/arm/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/arm/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>

Don't add stub files. We don't do that anymore.

The correct way of doing this is to modify:

 arch/*/include/asm/Kbuild

and add:

 generic-y += irq_work.h

-- Steve

> diff --git a/arch/arm64/include/asm/irq_work.h b/arch/arm64/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/arm64/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/avr32/include/asm/irq_work.h b/arch/avr32/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/avr32/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/blackfin/include/asm/irq_work.h b/arch/blackfin/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/blackfin/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/c6x/include/asm/irq_work.h b/arch/c6x/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/c6x/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/cris/include/asm/irq_work.h b/arch/cris/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/cris/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/frv/include/asm/irq_work.h b/arch/frv/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/frv/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/h8300/include/asm/irq_work.h b/arch/h8300/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/h8300/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/hexagon/include/asm/irq_work.h b/arch/hexagon/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/hexagon/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/ia64/include/asm/irq_work.h b/arch/ia64/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/ia64/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/m32r/include/asm/irq_work.h b/arch/m32r/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/m32r/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/m68k/include/asm/irq_work.h b/arch/m68k/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/m68k/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/microblaze/include/asm/irq_work.h b/arch/microblaze/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/microblaze/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/mips/include/asm/irq_work.h b/arch/mips/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/mips/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/mn10300/include/asm/irq_work.h b/arch/mn10300/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/mn10300/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/openrisc/include/asm/irq_work.h b/arch/openrisc/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/openrisc/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/parisc/include/asm/irq_work.h b/arch/parisc/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/parisc/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/powerpc/include/asm/irq_work.h b/arch/powerpc/include/asm/irq_work.h
> new file mode 100644
> index 0000000..8b9927f
> --- /dev/null
> +++ b/arch/powerpc/include/asm/irq_work.h
> @@ -0,0 +1,6 @@
> +#ifndef _ASM_POWERPC_IRQ_WORK_H
> +#define _ASM_POWERPC_IRQ_WORK_H
> +
> +extern void arch_irq_work_raise(void);
> +
> +#endif
> diff --git a/arch/s390/include/asm/irq_work.h b/arch/s390/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/s390/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/score/include/asm/irq_work.h b/arch/score/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/score/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/sh/include/asm/irq_work.h b/arch/sh/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/sh/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/sparc/include/asm/irq_work.h b/arch/sparc/include/asm/irq_work.h
> new file mode 100644
> index 0000000..1d062a6
> --- /dev/null
> +++ b/arch/sparc/include/asm/irq_work.h
> @@ -0,0 +1,6 @@
> +#ifndef ___ASM_SPARC_IRQ_H
> +#define ___ASM_SPARC_IRQ_H
> +
> +extern void arch_irq_work_raise(void);
> +
> +#endif
> diff --git a/arch/tile/include/asm/irq_work.h b/arch/tile/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/tile/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/um/include/asm/irq_work.h b/arch/um/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/um/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/unicore32/include/asm/irq_work.h b/arch/unicore32/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/unicore32/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/arch/x86/include/asm/irq_work.h b/arch/x86/include/asm/irq_work.h
> new file mode 100644
> index 0000000..38eed96
> --- /dev/null
> +++ b/arch/x86/include/asm/irq_work.h
> @@ -0,0 +1,10 @@
> +#ifndef _ASM_X86_IRQ_WORK_H
> +#define _ASM_X86_IRQ_WORK_H
> +
> +#ifndef CONFIG_X86_LOCAL_APIC
> +#include <asm-generic/irq_work.h>
> +# else
> +extern void arch_irq_work_raise(void);
> +#endif
> +
> +#endif
> diff --git a/arch/x86/kernel/irq_work.c b/arch/x86/kernel/irq_work.c
> index ca8f703..95f5d4e 100644
> --- a/arch/x86/kernel/irq_work.c
> +++ b/arch/x86/kernel/irq_work.c
> @@ -18,13 +18,13 @@ void smp_irq_work_interrupt(struct pt_regs *regs)
>  	irq_exit();
>  }
>  
> +#ifdef CONFIG_X86_LOCAL_APIC
>  void arch_irq_work_raise(void)
>  {
> -#ifdef CONFIG_X86_LOCAL_APIC
>  	if (!cpu_has_apic)
>  		return;
>  
>  	apic->send_IPI_self(IRQ_WORK_VECTOR);
>  	apic_wait_icr_idle();
> -#endif
>  }
> +#endif
> diff --git a/arch/xtensa/include/asm/irq_work.h b/arch/xtensa/include/asm/irq_work.h
> new file mode 100644
> index 0000000..f1bffa2
> --- /dev/null
> +++ b/arch/xtensa/include/asm/irq_work.h
> @@ -0,0 +1 @@
> +#include <asm-generic/irq_work.h>
> diff --git a/include/asm-generic/irq_work.h b/include/asm-generic/irq_work.h
> new file mode 100644
> index 0000000..a2d4108
> --- /dev/null
> +++ b/include/asm-generic/irq_work.h
> @@ -0,0 +1,9 @@
> +#ifndef __ASM_GENERIC_ATOMIC_H
> +#define __ASM_GENERIC_ATOMIC_H
> +
> +/*
> + * Lame architectures will get the timer tick callback
> + */
> +static inline void arch_irq_work_raise(void) { }
> +
> +#endif
> diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
> index 6a9e8f5..b39ea0b 100644
> --- a/include/linux/irq_work.h
> +++ b/include/linux/irq_work.h
> @@ -2,6 +2,7 @@
>  #define _LINUX_IRQ_WORK_H
>  
>  #include <linux/llist.h>
> +#include <asm/irq_work.h>
>  
>  struct irq_work {
>  	unsigned long flags;
> diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> index 679c13e..d00011c 100644
> --- a/kernel/irq_work.c
> +++ b/kernel/irq_work.c
> @@ -56,13 +56,6 @@ static bool irq_work_claim(struct irq_work *work)
>  	return true;
>  }
>  
> -void __weak arch_irq_work_raise(void)
> -{
> -	/*
> -	 * Lame architectures will get the timer tick callback
> -	 */
> -}
> -
>  /*
>   * Queue the entry and raise the IPI if needed.
>   */

Re: [RFC PATCH 4/9] irq_work: Let the arch tell us about self-IPI support

[Steven Rostedt]

On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
> This prepares us to make printk working on nohz CPUs
> using irq work.

-ENOTENOUGHINFO

Please state how this prepares printk for nohz CPUS using irq_work.

Thanks,

-- Steve

> 
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---

Re: [RFC PATCH 7/9] irq_work: Make self-IPIs optable

[Steven Rostedt]

On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
> On irq work initialization, let the user choose to define it
> as "lazy" or not. "Lazy" means that we don't want to send
> an IPI (provided the arch can anyway) when we enqueue this
> work but we rather prefer to wait for the next timer tick
> to execute our work if possible.
> 
> This is going to be a benefit for non-urgent enqueuers
> (like printk in the future) that may prefer not to raise
> an IPI storm in case of frequent enqueuing on short periods
> of time.
> 
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
>  include/linux/irq_work.h |   31 ++++++++++++++++++++++++++
>  kernel/irq_work.c        |   53 ++++++++++++++++++++++++++++++++-------------
>  kernel/time/tick-sched.c |    3 +-
>  3 files changed, 70 insertions(+), 17 deletions(-)
> 
> diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
> index b39ea0b..7b60c87 100644
> --- a/include/linux/irq_work.h
> +++ b/include/linux/irq_work.h
> @@ -4,6 +4,20 @@
>  #include <linux/llist.h>
>  #include <asm/irq_work.h>
>  
> +/*
> + * An entry can be in one of four states:
> + *

Can you add a comment to what the pointer value is. I know you just
moved it to the header, but it's still confusing.


> + * free	     NULL, 0 -> {claimed}       : free to be used
> + * claimed   NULL, 3 -> {pending}       : claimed to be enqueued
> + * pending   next, 3 -> {busy}          : queued, pending callback
> + * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
> + */
> +
> +#define IRQ_WORK_PENDING	1UL
> +#define IRQ_WORK_BUSY		2UL
> +#define IRQ_WORK_FLAGS		3UL
> +#define IRQ_WORK_LAZY		4UL /* Doesn't want IPI, wait for tick */
> +
>  struct irq_work {
>  	unsigned long flags;
>  	struct llist_node llnode;
> @@ -17,8 +31,25 @@ void init_irq_work(struct irq_work *work, void (*func)(struct irq_work *))
>  	work->func = func;
>  }
>  
> +#define DEFINE_IRQ_WORK(w, f)	\
> +	struct irq_work w = {	\
> +		.func = f,	\
> +	}
> +
> +#define DEFINE_IRQ_WORK_LAZY(w, f)	\
> +	struct irq_work w = {		\
> +		.flags = IRQ_WORK_LAZY, \
> +		.func = f,		\
> +	}
> +
>  bool irq_work_queue(struct irq_work *work);
>  void irq_work_run(void);
>  void irq_work_sync(struct irq_work *work);
>  
> +#ifdef CONFIG_IRQ_WORK
> +bool irq_work_needs_cpu(void);
> +#else
> +static bool irq_work_needs_cpu(void) { return false; }
> +#endif
> +
>  #endif /* _LINUX_IRQ_WORK_H */
> diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> index d00011c..ce72b20 100644
> --- a/kernel/irq_work.c
> +++ b/kernel/irq_work.c
> @@ -12,20 +12,10 @@
>  #include <linux/percpu.h>
>  #include <linux/hardirq.h>
>  #include <linux/irqflags.h>
> +#include <linux/tick.h>
> +#include <linux/sched.h>
>  #include <asm/processor.h>
>  
> -/*
> - * An entry can be in one of four states:
> - *
> - * free	     NULL, 0 -> {claimed}       : free to be used
> - * claimed   NULL, 3 -> {pending}       : claimed to be enqueued
> - * pending   next, 3 -> {busy}          : queued, pending callback
> - * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
> - */
> -
> -#define IRQ_WORK_PENDING	1UL
> -#define IRQ_WORK_BUSY		2UL
> -#define IRQ_WORK_FLAGS		3UL
>  
>  static DEFINE_PER_CPU(struct llist_head, irq_work_list);
>  
> @@ -66,10 +56,28 @@ static void __irq_work_queue(struct irq_work *work)
>  	preempt_disable();
>  
>  	empty = llist_add(&work->llnode, &__get_cpu_var(irq_work_list));
> -	/* The list was empty, raise self-interrupt to start processing. */
> -	if (empty)
> +
> +	/*
> +	 * In any case, raise an IPI if requested and possible in case
> +	 * the queue is empty or it's filled with lazy works.
> +	 */
> +	if (!(work->flags & IRQ_WORK_LAZY) && arch_irq_work_has_ipi()) {
>  		arch_irq_work_raise();

Doesn't this mean that now if we queue up a bunch of work (say in
tracing), that we will send out an IPI for each queue? We only want to
send out an IPI if the list isn't empty. Perhaps we should make two
lists. One for lazy work and one for immediate work. Then, when adding a
non-lazy work item, we can check the empty variable for that. No need to
check the result for the lazy queue. That will be done during tick.

Perhaps then do:

	empty = false;

	if (work->flags & IRQ_WORK_LAZY)
		llist_add(&work->llnode, &__get_cpu_var(irq_work_lazy_list));
	else
		empty = llist_add(&work->llnode, &__get_cpu_var(irq_work_list));

	if (empty) {

	}

You can add your arch_irq_work_has_ipi() stuff as well.

-- Steve


> +		goto out;
> +	}llist_add(&work->llnode, &__get_cpu_var(irq_work_list));
>  
> +	if (empty) {
> +		/*
> +		 * If the arch uses some other obscure way than IPI to raise
> +		 * an irq work, just raise and don't think further.
> +		 * Now if it can send an IPI, although not requested, and the tick
> +		 * is stopped, send it nonetheless otherwise we may wait a long while
> +		 * before that lazy work can run.
> +		 */
> +		if (!arch_irq_work_has_ipi() || tick_nohz_tick_stopped())
> +			arch_irq_work_raise();
> +	}
> +out:
>  	preempt_enable();
>  }
>  

Re: [RFC PATCH 9/9] printk: Wake up klogd using irq_work

[Steven Rostedt]

On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
> klogd is woken up asynchronously from the tick in order
> to do it safely.
> 
> However if printk is called when the tick is stopped, the reader
> won't be woken up until the next interrupt, which might not fire
> before a while. As a result, the user may miss some message.

Just a grammar nit (and goes for your previous patch as well). We say
"might not fire for a while." or you could say ".. for some time", but
not "before a while" ;-)

I wounder what the French translation of that is.

-- Steve


> 
> To fix this, lets implement the printk tick using a lazy irq work.
> This subsystem takes care of the timer tick state and can
> fix up accordingly.
> 
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>

Re: [RFC PATCH 3/9] irq_work: Move irq_work_raise() declaration/default definition to arch headers

[Frederic Weisbecker]

2012/10/29 Steven Rostedt <rostedt@goodmis.org>:
> Don't add stub files. We don't do that anymore.
>
> The correct way of doing this is to modify:
>
>  arch/*/include/asm/Kbuild
>
> and add:
>
>  generic-y += irq_work.h

Ah, ok!

Re: [RFC PATCH 1/9] irq_work: Fix racy check on work pending flag

[Frederic Weisbecker]

2012/10/29 Steven Rostedt <rostedt@goodmis.org>:
> I wonder if the bug is just a memory barrier missing here? But that also
> suggests that the other CPU used a memory barrier too (or cmpxchg()
> which implies one).
>
> But this change looks fine too.

I fear a memory barrier wouldn't be much helpful here.

Thing is I'm not sure exactly what strict semantics is expected there:
do we allow for concurrent claiming only locally or also across CPUs?
Given that we only disable preemption on work queueing, I assumed we
allow for concurrent claiming across CPUs. Also we have irq work users
that may claim on random CPUs already and thus this can result in
cross-CPUs concurrent claiming. Task wide (not per cpu) perf event are
such examples. May be drivers/acpi/apei/ghes.c and perhaps others.

Also if we allow for cross-CPUs concurrent claiming, we should set
IRQ_WORK_BUSY with cmpxchg() (or xchg() if it's atomic, because nobody
can modify it concurrently at this stage as it's claimed). Otherwise a
CPU can fail its claim whereas the other CPU that claimed it may
actually have executed the work already, or is in the middle of doing
so. The race window stops when we clear the IRQ_WORK_BUSY flag with
cmpxchg so it's very tight. But still it's there.

Also irq_work_sync() should poll with ACCESS_ONCE(). Same problem with
foggy semantics here: do we allow remote sync?

Re: [RFC PATCH 4/9] irq_work: Let the arch tell us about self-IPI support

[Frederic Weisbecker]

2012/10/29 Steven Rostedt <rostedt@goodmis.org>:
> On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
>> This prepares us to make printk working on nohz CPUs
>> using irq work.
>
> -ENOTENOUGHINFO
>
> Please state how this prepares printk for nohz CPUS using irq_work.

RIght, I'll add the details in the next version.

Re: [RFC PATCH 9/9] printk: Wake up klogd using irq_work

[Frederic Weisbecker]

2012/10/29 Steven Rostedt <rostedt@goodmis.org>:
> On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
>> klogd is woken up asynchronously from the tick in order
>> to do it safely.
>>
>> However if printk is called when the tick is stopped, the reader
>> won't be woken up until the next interrupt, which might not fire
>> before a while. As a result, the user may miss some message.
>
> Just a grammar nit (and goes for your previous patch as well). We say
> "might not fire for a while." or you could say ".. for some time", but
> not "before a while" ;-)

Right!

>
> I wounder what the French translation of that is.

You don't really want to know ;)
But as you guess, this is quite close to something like "before a while" ;)

Re: [RFC PATCH 7/9] irq_work: Make self-IPIs optable

[Frederic Weisbecker]

2012/10/29 Steven Rostedt <rostedt@goodmis.org>:
> On Mon, 2012-10-29 at 14:28 +0100, Frederic Weisbecker wrote:
>> On irq work initialization, let the user choose to define it
>> as "lazy" or not. "Lazy" means that we don't want to send
>> an IPI (provided the arch can anyway) when we enqueue this
>> work but we rather prefer to wait for the next timer tick
>> to execute our work if possible.
>>
>> This is going to be a benefit for non-urgent enqueuers
>> (like printk in the future) that may prefer not to raise
>> an IPI storm in case of frequent enqueuing on short periods
>> of time.
>>
>> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Ingo Molnar <mingo@kernel.org>
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: Steven Rostedt <rostedt@goodmis.org>
>> Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
>> ---
>>  include/linux/irq_work.h |   31 ++++++++++++++++++++++++++
>>  kernel/irq_work.c        |   53 ++++++++++++++++++++++++++++++++-------------
>>  kernel/time/tick-sched.c |    3 +-
>>  3 files changed, 70 insertions(+), 17 deletions(-)
>>
>> diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
>> index b39ea0b..7b60c87 100644
>> --- a/include/linux/irq_work.h
>> +++ b/include/linux/irq_work.h
>> @@ -4,6 +4,20 @@
>>  #include <linux/llist.h>
>>  #include <asm/irq_work.h>
>>
>> +/*
>> + * An entry can be in one of four states:
>> + *
>
> Can you add a comment to what the pointer value is. I know you just
> moved it to the header, but it's still confusing.

Which pointer value?  You mean the flag? You mean the below need more
details or?

>
>> + * free           NULL, 0 -> {claimed}       : free to be used
>> + * claimed   NULL, 3 -> {pending}       : claimed to be enqueued
>> + * pending   next, 3 -> {busy}          : queued, pending callback
>> + * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
>> + */
>> +
>> +#define IRQ_WORK_PENDING     1UL
>> +#define IRQ_WORK_BUSY                2UL
>> +#define IRQ_WORK_FLAGS               3UL
>> +#define IRQ_WORK_LAZY                4UL /* Doesn't want IPI, wait for tick */
[...]
>> @@ -66,10 +56,28 @@ static void __irq_work_queue(struct irq_work *work)
>>       preempt_disable();
>>
>>       empty = llist_add(&work->llnode, &__get_cpu_var(irq_work_list));
>> -     /* The list was empty, raise self-interrupt to start processing. */
>> -     if (empty)
>> +
>> +     /*
>> +      * In any case, raise an IPI if requested and possible in case
>> +      * the queue is empty or it's filled with lazy works.
>> +      */
>> +     if (!(work->flags & IRQ_WORK_LAZY) && arch_irq_work_has_ipi()) {
>>               arch_irq_work_raise();
>
> Doesn't this mean that now if we queue up a bunch of work (say in
> tracing), that we will send out an IPI for each queue? We only want to
> send out an IPI if the list isn't empty. Perhaps we should make two
> lists. One for lazy work and one for immediate work. Then, when adding a
> non-lazy work item, we can check the empty variable for that. No need to
> check the result for the lazy queue. That will be done during tick.

Indeed, if an IPI is pending while we queue another work, we'll raise
another one. I would prefer to avoid the complication of adding
another queue though. Perhaps a per cpu "ipi_pending" flag would be
enough. I'll try something.