All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC][PATCHv2 0/8] printk: introduce printing kernel thread
@ 2017-03-29  9:25 Sergey Senozhatsky
  2017-03-29  9:25 ` [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu Sergey Senozhatsky
                   ` (7 more replies)
  0 siblings, 8 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

Hello

	RFC

	This patch set adds a printk() kernel thread which lets us to
print kernel messages to the console from a non-atomic/schedule-able
context, avoiding different sort of lockups, stalls, etc.

	The biggest difference compared to v1 is a further expansion of
"switch to old printk() mode", which we now call printk_emergency mode,
annotations (pm, kexec, sysrq).

	We also have a bunch of other ideas to try out/consider (not in
this patch set). E.g. automatic printk_emergency switch on printk_kthread
stalls (when we can't run printk_kthread), moving klogd irq_work out of
per-CPU memory, and so on.

v1->v2:
-- introduce printk_emergency mode and API to switch it on/off
-- move printk_pending out of per-CPU memory
-- add printk emergency_mode sysfs node
-- switch sysrq handlers (some of them) to printk_emergency
-- cleanus/etc.

Sergey Senozhatsky (8):
  printk: move printk_pending out of per-cpu
  printk: introduce printing kernel thread
  printk: offload printing from wake_up_klogd_work_func()
  pm: switch to printk.emergency mode in unsafe places
  sysrq: switch to printk.emergency mode in unsafe places
  kexec: switch to printk.emergency mode in unsafe places
  printk: add printk emergency_mode parameter
  printk: enable printk offloading

 drivers/tty/sysrq.c      |   7 ++
 include/linux/console.h  |   3 +
 kernel/kexec_core.c      |   4 ++
 kernel/power/hibernate.c |   8 +++
 kernel/power/suspend.c   |   4 ++
 kernel/printk/printk.c   | 178 +++++++++++++++++++++++++++++++++++++++++------
 6 files changed, 182 insertions(+), 22 deletions(-)

-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-03-31 13:09   ` Petr Mladek
  2017-03-29  9:25 ` [RFC][PATCHv2 2/8] printk: introduce printing kernel thread Sergey Senozhatsky
                   ` (6 subsequent siblings)
  7 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

Do not keep `printk_pending' in per-CPU area. We set the following bits
of printk_pending:
a) PRINTK_PENDING_WAKEUP
	when we need to wakeup klogd
b) PRINTK_PENDING_OUTPUT
	when there is a pending output from deferred printk and we need
	to call console_unlock().

So none of the bits control/represent a state of a particular CPU and,
basically, they should be global instead.

Besides we will use `printk_pending' to control printk kthread, so this
patch is also a preparation work.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Suggested-by: Petr Mladek <pmladek@suse.com>
---
 kernel/printk/printk.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 2984fb0f0257..2d07678e9ff9 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -401,6 +401,14 @@ DEFINE_RAW_SPINLOCK(logbuf_lock);
 		printk_safe_exit_irqrestore(flags);	\
 	} while (0)
 
+/*
+ * Delayed printk version, for scheduler-internal messages:
+ */
+#define PRINTK_PENDING_WAKEUP	0x01
+#define PRINTK_PENDING_OUTPUT	0x02
+
+static unsigned long printk_pending;
+
 #ifdef CONFIG_PRINTK
 DECLARE_WAIT_QUEUE_HEAD(log_wait);
 /* the next printk record to read by syslog(READ) or /proc/kmsg */
@@ -2654,25 +2662,15 @@ static int __init printk_late_init(void)
 late_initcall(printk_late_init);
 
 #if defined CONFIG_PRINTK
-/*
- * Delayed printk version, for scheduler-internal messages:
- */
-#define PRINTK_PENDING_WAKEUP	0x01
-#define PRINTK_PENDING_OUTPUT	0x02
-
-static DEFINE_PER_CPU(int, printk_pending);
-
 static void wake_up_klogd_work_func(struct irq_work *irq_work)
 {
-	int pending = __this_cpu_xchg(printk_pending, 0);
-
-	if (pending & PRINTK_PENDING_OUTPUT) {
+	if (test_and_clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending)) {
 		/* If trylock fails, someone else is doing the printing */
 		if (console_trylock())
 			console_unlock();
 	}
 
-	if (pending & PRINTK_PENDING_WAKEUP)
+	if (test_and_clear_bit(PRINTK_PENDING_WAKEUP, &printk_pending))
 		wake_up_interruptible(&log_wait);
 }
 
@@ -2685,7 +2683,7 @@ void wake_up_klogd(void)
 {
 	preempt_disable();
 	if (waitqueue_active(&log_wait)) {
-		this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
+		set_bit(PRINTK_PENDING_WAKEUP, &printk_pending);
 		irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
 	}
 	preempt_enable();
@@ -2701,7 +2699,7 @@ int printk_deferred(const char *fmt, ...)
 	r = vprintk_emit(0, LOGLEVEL_SCHED, NULL, 0, fmt, args);
 	va_end(args);
 
-	__this_cpu_or(printk_pending, PRINTK_PENDING_OUTPUT);
+	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 	irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
 	preempt_enable();
 
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
  2017-03-29  9:25 ` [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-04-04  9:01   ` Petr Mladek
  2017-04-06 17:14   ` Pavel Machek
  2017-03-29  9:25 ` [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func() Sergey Senozhatsky
                   ` (5 subsequent siblings)
  7 siblings, 2 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

printk() is quite complex internally and, basically, it does two
slightly independent things:
 a) adds a new message to a kernel log buffer (log_store())
 b) prints kernel log messages to serial consoles (console_unlock())

while (a) is guaranteed to be executed by printk(), (b) is not, for a
variety of reasons, and, unlike log_store(), it comes at a price:

 1) console_unlock() attempts to flush all pending kernel log messages
to the console. Thus, it can loop indefinitely.

 2) while console_unlock() is executed on one particular CPU, printing
pending kernel log messages, other CPUs can simultaneously append new
messages to the kernel log buffer.

 3) the time it takes console_unlock() to print kernel messages also
depends on the speed of the console -- which may not be fast at all.

 4) console_unlock() is executed in the same context as printk(), so
it may be non-preemptible/atomic, which makes 1)-3) dangerous.

As a result, nobody knows how long a printk() call will take, so
it's not really safe to call printk() in a number of situations,
including atomic context, RCU critical sections, interrupt context,
and more.

This patch introduces a dedicated printing kernel thread - printk_kthread.
The main purpose of this kthread is to offload printing to a non-atomic
and always scheduleable context, which eliminates 4) and makes 1)-3) less
critical. printk() now just appends log messages to the kernel log buffer
and wake_up()s printk_kthread instead of locking console_sem and calling
into potentially unsafe console_unlock().

This, however, is not always safe on its own. For example, we can't call
into the scheduler from panic(), because this may cause deadlock. That's
why we introduce a concept of printk_emergency() mode, when printk()
switches back to the old behaviour (console_unlock() from vprintk_emit())
in those cases. In general, this switch happens automatically once a EMERG
log level message appears in the log buffer. Another cases when wake_up()
might not work as expected are suspend, hibernate, etc. For those situations
we provide two new functions:
 -- printk_emergency_begin()
    Disables printk offloading. All printk() calls will attempt
    to lock the console_sem and, if successful, flush kernel log
    messages.

 -- printk_emergency_end()
    Enables printk offloading.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
 include/linux/console.h |   3 ++
 kernel/printk/printk.c  | 107 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 102 insertions(+), 8 deletions(-)

diff --git a/include/linux/console.h b/include/linux/console.h
index 5949d1855589..f1a86944072e 100644
--- a/include/linux/console.h
+++ b/include/linux/console.h
@@ -187,6 +187,9 @@ extern bool console_suspend_enabled;
 extern void suspend_console(void);
 extern void resume_console(void);
 
+extern void printk_emergency_begin(void);
+extern void printk_emergency_end(void);
+
 int mda_console_init(void);
 void prom_con_init(void);
 
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 2d07678e9ff9..ab6b3b2a68c6 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -48,6 +48,7 @@
 #include <linux/sched/clock.h>
 #include <linux/sched/debug.h>
 #include <linux/sched/task_stack.h>
+#include <linux/kthread.h>
 
 #include <linux/uaccess.h>
 #include <asm/sections.h>
@@ -402,7 +403,8 @@ DEFINE_RAW_SPINLOCK(logbuf_lock);
 	} while (0)
 
 /*
- * Delayed printk version, for scheduler-internal messages:
+ * Used both for deferred printk version (scheduler-internal messages)
+ * and printk_kthread control.
  */
 #define PRINTK_PENDING_WAKEUP	0x01
 #define PRINTK_PENDING_OUTPUT	0x02
@@ -445,6 +447,42 @@ static char __log_buf[__LOG_BUF_LEN] __aligned(LOG_ALIGN);
 static char *log_buf = __log_buf;
 static u32 log_buf_len = __LOG_BUF_LEN;
 
+static struct task_struct *printk_kthread __read_mostly;
+/*
+ * We can't call into the scheduler (wake_up() printk kthread) during
+ * suspend/kexec/etc. This temporarily switches printk to old behaviour.
+ */
+static atomic_t printk_emergency __read_mostly;
+/*
+ * Disable printk_kthread permanently. Unlike `oops_in_progress'
+ * it doesn't go back to 0.
+ */
+static bool printk_kthread_disabled __read_mostly;
+
+static inline bool printk_kthread_enabled(void)
+{
+	return !printk_kthread_disabled &&
+		printk_kthread && atomic_read(&printk_emergency) == 0;
+}
+
+/*
+ * This disables printing offloading and instead attempts
+ * to do the usual console_trylock()->console_unlock().
+ *
+ * Note, this does not stop the printk_kthread if it's already
+ * printing logbuf messages.
+ */
+void printk_emergency_begin(void)
+{
+	atomic_inc(&printk_emergency);
+}
+
+/* This re-enables printk_kthread offloading. */
+void printk_emergency_end(void)
+{
+	atomic_dec(&printk_emergency);
+}
+
 /* Return log buffer address */
 char *log_buf_addr_get(void)
 {
@@ -1765,17 +1803,40 @@ asmlinkage int vprintk_emit(int facility, int level,
 
 	printed_len += log_output(facility, level, lflags, dict, dictlen, text, text_len);
 
+	/*
+	 * Emergency level indicates that the system is unstable and, thus,
+	 * we better stop relying on wake_up(printk_kthread) and try to do
+	 * a direct printing.
+	 */
+	if (level == LOGLEVEL_EMERG)
+		printk_kthread_disabled = true;
+
+	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 	logbuf_unlock_irqrestore(flags);
 
 	/* If called from the scheduler, we can not call up(). */
 	if (!in_sched) {
 		/*
-		 * Try to acquire and then immediately release the console
-		 * semaphore.  The release will print out buffers and wake up
-		 * /dev/kmsg and syslog() users.
+		 * Under heavy printing load/slow serial console/etc
+		 * console_unlock() can stall CPUs, which can result in
+		 * soft/hard-lockups, lost interrupts, RCU stalls, etc.
+		 * Therefore we attempt to print the messages to console
+		 * from a dedicated printk_kthread, which always runs in
+		 * schedulable context.
 		 */
-		if (console_trylock())
-			console_unlock();
+		if (printk_kthread_enabled()) {
+			printk_safe_enter_irqsave(flags);
+			wake_up_process(printk_kthread);
+			printk_safe_exit_irqrestore(flags);
+		} else {
+			/*
+			 * Try to acquire and then immediately release the
+			 * console semaphore. The release will print out
+			 * buffers and wake up /dev/kmsg and syslog() users.
+			 */
+			if (console_trylock())
+				console_unlock();
+		}
 	}
 
 	return printed_len;
@@ -1882,6 +1943,9 @@ static size_t msg_print_text(const struct printk_log *msg,
 			     bool syslog, char *buf, size_t size) { return 0; }
 static bool suppress_message_printing(int level) { return false; }
 
+void printk_emergency_begin(void) {}
+void printk_emergency_end(void) {}
+
 #endif /* CONFIG_PRINTK */
 
 #ifdef CONFIG_EARLY_PRINTK
@@ -2164,6 +2228,13 @@ void console_unlock(void)
 	bool do_cond_resched, retry;
 
 	if (console_suspended) {
+		/*
+		 * Avoid an infinite loop in printk_kthread function
+		 * when console_unlock() cannot flush messages because
+		 * we suspended consoles. Someone else will print the
+		 * messages from resume_console().
+		 */
+		clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 		up_console_sem();
 		return;
 	}
@@ -2182,6 +2253,7 @@ void console_unlock(void)
 	console_may_schedule = 0;
 
 again:
+	clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 	/*
 	 * We released the console_sem lock, so we need to recheck if
 	 * cpu is online and (if not) is there at least one CON_ANYTIME
@@ -2664,8 +2736,11 @@ late_initcall(printk_late_init);
 #if defined CONFIG_PRINTK
 static void wake_up_klogd_work_func(struct irq_work *irq_work)
 {
-	if (test_and_clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending)) {
-		/* If trylock fails, someone else is doing the printing */
+	if (test_bit(PRINTK_PENDING_OUTPUT, &printk_pending)) {
+		/*
+		 * If trylock fails, someone else is doing the printing.
+		 * PRINTK_PENDING_OUTPUT bit is cleared by console_unlock().
+		 */
 		if (console_trylock())
 			console_unlock();
 	}
@@ -2679,6 +2754,22 @@ static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) = {
 	.flags = IRQ_WORK_LAZY,
 };
 
+static int printk_kthread_func(void *data)
+{
+	while (1) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		if (!test_bit(PRINTK_PENDING_OUTPUT, &printk_pending))
+			schedule();
+
+		__set_current_state(TASK_RUNNING);
+
+		console_lock();
+		console_unlock();
+	}
+
+	return 0;
+}
+
 void wake_up_klogd(void)
 {
 	preempt_disable();
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func()
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
  2017-03-29  9:25 ` [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu Sergey Senozhatsky
  2017-03-29  9:25 ` [RFC][PATCHv2 2/8] printk: introduce printing kernel thread Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-03-31 14:56   ` Petr Mladek
  2017-03-29  9:25 ` [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places Sergey Senozhatsky
                   ` (4 subsequent siblings)
  7 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

Offload printing of printk_deferred() messages from IRQ context
to a schedulable printing kthread, when possible (the same way
we do it in vprintk_emit()). Otherwise, console_unlock() can
force the printing CPU to spend unbound amount of time flushing
kernel messages from IRQ context.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 kernel/printk/printk.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index ab6b3b2a68c6..1927b5cb5cbe 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -2741,8 +2741,16 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work)
 		 * If trylock fails, someone else is doing the printing.
 		 * PRINTK_PENDING_OUTPUT bit is cleared by console_unlock().
 		 */
-		if (console_trylock())
-			console_unlock();
+		if (printk_kthread_enabled()) {
+			wake_up_process(printk_kthread);
+		} else {
+			/*
+			 * If trylock fails, someone else is doing the
+			 * printing
+			 */
+			if (console_trylock())
+				console_unlock();
+		}
 	}
 
 	if (test_and_clear_bit(PRINTK_PENDING_WAKEUP, &printk_pending))
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
                   ` (2 preceding siblings ...)
  2017-03-29  9:25 ` [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func() Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-03-31 15:06   ` Petr Mladek
  2017-04-06 17:20   ` Pavel Machek
  2017-03-29  9:25 ` [RFC][PATCHv2 5/8] sysrq: " Sergey Senozhatsky
                   ` (3 subsequent siblings)
  7 siblings, 2 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

It's not always possible/safe to wake_up() printk kernel
thread. For example, late suspend/early resume may printk()
while timekeeping is not initialized yet, so calling into the
scheduler may result in recursive warnings.

Another thing to notice is the fact PM at some point
freezes user space and kernel threads: freeze_processes()
and freeze_kernel_threads(), correspondingly. Thus we need
printk() to operate in old mode there and attempt to
immediately flush pending kernel message to the console.

This patch adds printk_emergency_begin/on sections.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 kernel/power/hibernate.c | 8 ++++++++
 kernel/power/suspend.c   | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/kernel/power/hibernate.c b/kernel/power/hibernate.c
index a8b978c35a6a..a8c6bb2dbcef 100644
--- a/kernel/power/hibernate.c
+++ b/kernel/power/hibernate.c
@@ -502,6 +502,7 @@ int hibernation_restore(int platform_mode)
 {
 	int error;
 
+	printk_emergency_begin();
 	pm_prepare_console();
 	suspend_console();
 	pm_restrict_gfp_mask();
@@ -519,6 +520,7 @@ int hibernation_restore(int platform_mode)
 	pm_restore_gfp_mask();
 	resume_console();
 	pm_restore_console();
+	printk_emergency_end();
 	return error;
 }
 
@@ -542,6 +544,7 @@ int hibernation_platform_enter(void)
 		goto Close;
 
 	entering_platform_hibernation = true;
+	printk_emergency_begin();
 	suspend_console();
 	error = dpm_suspend_start(PMSG_HIBERNATE);
 	if (error) {
@@ -589,6 +592,7 @@ int hibernation_platform_enter(void)
 	entering_platform_hibernation = false;
 	dpm_resume_end(PMSG_RESTORE);
 	resume_console();
+	printk_emergency_end();
 
  Close:
 	hibernation_ops->end();
@@ -692,6 +696,7 @@ int hibernate(void)
 		goto Unlock;
 	}
 
+	printk_emergency_begin();
 	pm_prepare_console();
 	error = __pm_notifier_call_chain(PM_HIBERNATION_PREPARE, -1, &nr_calls);
 	if (error) {
@@ -759,6 +764,7 @@ int hibernate(void)
  Exit:
 	__pm_notifier_call_chain(PM_POST_HIBERNATION, nr_calls, NULL);
 	pm_restore_console();
+	printk_emergency_end();
 	atomic_inc(&snapshot_device_available);
  Unlock:
 	unlock_system_sleep();
@@ -868,6 +874,7 @@ static int software_resume(void)
 		goto Unlock;
 	}
 
+	printk_emergency_begin();
 	pm_prepare_console();
 	error = __pm_notifier_call_chain(PM_RESTORE_PREPARE, -1, &nr_calls);
 	if (error) {
@@ -884,6 +891,7 @@ static int software_resume(void)
  Finish:
 	__pm_notifier_call_chain(PM_POST_RESTORE, nr_calls, NULL);
 	pm_restore_console();
+	printk_emergency_end();
 	atomic_inc(&snapshot_device_available);
 	/* For success case, the suspend path will release the lock */
  Unlock:
diff --git a/kernel/power/suspend.c b/kernel/power/suspend.c
index 15e6baef5c73..1f897b149fc0 100644
--- a/kernel/power/suspend.c
+++ b/kernel/power/suspend.c
@@ -433,6 +433,7 @@ int suspend_devices_and_enter(suspend_state_t state)
 	if (!sleep_state_supported(state))
 		return -ENOSYS;
 
+	printk_emergency_begin();
 	error = platform_suspend_begin(state);
 	if (error)
 		goto Close;
@@ -462,6 +463,7 @@ int suspend_devices_and_enter(suspend_state_t state)
 
  Close:
 	platform_resume_end(state);
+	printk_emergency_end();
 	return error;
 
  Recover_platform:
@@ -520,6 +522,7 @@ static int enter_state(suspend_state_t state)
 #endif
 
 	pr_debug("PM: Preparing system for sleep (%s)\n", pm_states[state]);
+	printk_emergency_begin();
 	pm_suspend_clear_flags();
 	error = suspend_prepare(state);
 	if (error)
@@ -537,6 +540,7 @@ static int enter_state(suspend_state_t state)
  Finish:
 	pr_debug("PM: Finishing wakeup.\n");
 	suspend_finish();
+	printk_emergency_end();
  Unlock:
 	mutex_unlock(&pm_mutex);
 	return error;
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 5/8] sysrq: switch to printk.emergency mode in unsafe places
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
                   ` (3 preceding siblings ...)
  2017-03-29  9:25 ` [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-03-31 15:37   ` Petr Mladek
  2017-03-29  9:25 ` [RFC][PATCHv2 6/8] kexec: " Sergey Senozhatsky
                   ` (2 subsequent siblings)
  7 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

It's not always possible/safe to wake_up() printk kernel
thread from sysrq (theoretically). Thus we better switch
printk() to emergency mode in some of the sysrq handlers,
which allows us to immediately flush pending kernel message
to the console.

This patch adds printk_emergency_begin/on sections.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Suggested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
---
 drivers/tty/sysrq.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index c6fc7141d7b2..817dfb69914d 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -49,6 +49,7 @@
 #include <linux/syscalls.h>
 #include <linux/of.h>
 #include <linux/rcupdate.h>
+#include <linux/console.h>
 
 #include <asm/ptrace.h>
 #include <asm/irq_regs.h>
@@ -239,6 +240,7 @@ static DECLARE_WORK(sysrq_showallcpus, sysrq_showregs_othercpus);
 
 static void sysrq_handle_showallcpus(int key)
 {
+	printk_emergency_begin();
 	/*
 	 * Fall back to the workqueue based printing if the
 	 * backtrace printing did not succeed or the
@@ -253,6 +255,7 @@ static void sysrq_handle_showallcpus(int key)
 		}
 		schedule_work(&sysrq_showallcpus);
 	}
+	printk_emergency_end();
 }
 
 static struct sysrq_key_op sysrq_showallcpus_op = {
@@ -279,8 +282,10 @@ static struct sysrq_key_op sysrq_showregs_op = {
 
 static void sysrq_handle_showstate(int key)
 {
+	printk_emergency_begin();
 	show_state();
 	show_workqueue_state();
+	printk_emergency_end();
 }
 static struct sysrq_key_op sysrq_showstate_op = {
 	.handler	= sysrq_handle_showstate,
@@ -291,7 +296,9 @@ static struct sysrq_key_op sysrq_showstate_op = {
 
 static void sysrq_handle_showstate_blocked(int key)
 {
+	printk_emergency_begin();
 	show_state_filter(TASK_UNINTERRUPTIBLE);
+	printk_emergency_end();
 }
 static struct sysrq_key_op sysrq_showstate_blocked_op = {
 	.handler	= sysrq_handle_showstate_blocked,
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 6/8] kexec: switch to printk.emergency mode in unsafe places
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
                   ` (4 preceding siblings ...)
  2017-03-29  9:25 ` [RFC][PATCHv2 5/8] sysrq: " Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-03-31 15:39   ` Petr Mladek
  2017-03-29  9:25 ` [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter Sergey Senozhatsky
  2017-03-29  9:25 ` [RFC][PATCHv2 8/8] printk: enable printk offloading Sergey Senozhatsky
  7 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

Switch to printk emergency mode in kernel_kexec(), this
lets us to immediately flush pending kernel message to
the console and to avoid potentially unsafe calls into
the scheduler.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 kernel/kexec_core.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index bfe62d5b3872..5684ba36ec15 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -1496,6 +1496,8 @@ int kernel_kexec(void)
 		goto Unlock;
 	}
 
+	printk_emergency_begin();
+
 #ifdef CONFIG_KEXEC_JUMP
 	if (kexec_image->preserve_context) {
 		lock_system_sleep();
@@ -1565,6 +1567,8 @@ int kernel_kexec(void)
 	}
 #endif
 
+	printk_emergency_end();
+
  Unlock:
 	mutex_unlock(&kexec_mutex);
 	return error;
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
                   ` (5 preceding siblings ...)
  2017-03-29  9:25 ` [RFC][PATCHv2 6/8] kexec: " Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-04-03 15:29   ` Petr Mladek
  2017-03-29  9:25 ` [RFC][PATCHv2 8/8] printk: enable printk offloading Sergey Senozhatsky
  7 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

This param permits user-space to forcibly on/off printk emergency
mode via /sys/module/printk/parameters/emergency_mode node.

We have annotated sections in the kernel that switch printk to
emergency, but there might be places/cases when user space would
want to have printk operate in emergency mode all the time.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 kernel/printk/printk.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 1927b5cb5cbe..0d96839bb450 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -455,7 +455,7 @@ static struct task_struct *printk_kthread __read_mostly;
 static atomic_t printk_emergency __read_mostly;
 /*
  * Disable printk_kthread permanently. Unlike `oops_in_progress'
- * it doesn't go back to 0.
+ * it doesn't go back to 0 (unless enforced by user-space).
  */
 static bool printk_kthread_disabled __read_mostly;
 
@@ -483,6 +483,24 @@ void printk_emergency_end(void)
 	atomic_dec(&printk_emergency);
 }
 
+static int printk_kthread_disabled_set(const char *val,
+					const struct kernel_param *kp)
+{
+	return param_set_bool(val, kp);
+}
+
+static const struct kernel_param_ops printk_kthread_disabled_ops = {
+	.set = printk_kthread_disabled_set,
+	.get = param_get_bool,
+};
+
+module_param_cb(emergency_mode,
+		&printk_kthread_disabled_ops,
+		&printk_kthread_disabled,
+		0644);
+MODULE_PARM_DESC(emergency_mode,
+		"don't offload message printing to printk kthread");
+
 /* Return log buffer address */
 char *log_buf_addr_get(void)
 {
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [RFC][PATCHv2 8/8] printk: enable printk offloading
  2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
                   ` (6 preceding siblings ...)
  2017-03-29  9:25 ` [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter Sergey Senozhatsky
@ 2017-03-29  9:25 ` Sergey Senozhatsky
  2017-03-30 21:38   ` [printk] fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage kernel test robot
  2017-04-03 15:42   ` [RFC][PATCHv2 8/8] printk: enable printk offloading Petr Mladek
  7 siblings, 2 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-29  9:25 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek
  Cc: Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, Sergey Senozhatsky

Initialize the kernel printing thread and enable printk()
offloading.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 kernel/printk/printk.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 0d96839bb450..acfdc50580db 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -2796,6 +2796,25 @@ static int printk_kthread_func(void *data)
 	return 0;
 }
 
+/*
+ * Init printk kthread at late_initcall stage, after core/arch/device/etc.
+ * initialization.
+ */
+static int __init init_printk_kthread(void)
+{
+	struct task_struct *thread;
+
+	thread = kthread_run(printk_kthread_func, NULL, "printk");
+	if (IS_ERR(thread)) {
+		pr_err("printk: unable to create printing thread\n");
+		return PTR_ERR(thread);
+	}
+
+	printk_kthread = thread;
+	return 0;
+}
+late_initcall(init_printk_kthread);
+
 void wake_up_klogd(void)
 {
 	preempt_disable();
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-29  9:25 ` [RFC][PATCHv2 8/8] printk: enable printk offloading Sergey Senozhatsky
@ 2017-03-30 21:38   ` kernel test robot
  2017-03-31  2:35     ` Sergey Senozhatsky
  2017-04-03 15:42   ` [RFC][PATCHv2 8/8] printk: enable printk offloading Petr Mladek
  1 sibling, 1 reply; 75+ messages in thread
From: kernel test robot @ 2017-03-30 21:38 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Petr Mladek, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, Sergey Senozhatsky, Sergey Senozhatsky,
	lkp

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

FYI, we noticed the following commit:

commit: fbc14616f483788afabe77d05bfb99883dc66c73 ("printk: enable printk offloading")
url: https://github.com/0day-ci/linux/commits/Sergey-Senozhatsky/printk-introduce-printing-kernel-thread/20170330-185752


in testcase: trinity
with following parameters:

	runtime: 300s

test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/


on test machine: qemu-system-x86_64 -enable-kvm -cpu Westmere -m 512M

caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):


+-------------------------------------------------+------------+------------+
|                                                 | fd8b6b120c | fbc14616f4 |
+-------------------------------------------------+------------+------------+
| boot_successes                                  | 8          | 8          |
| boot_failures                                   | 0          | 6          |
| BUG:kernel_reboot-without-warning_in_test_stage | 0          | 4          |
| BUG:kernel_hang_in_test_stage                   | 0          | 2          |
+-------------------------------------------------+------------+------------+



[   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
[   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
[   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 

Elapsed time: 310
BUG: kernel reboot-without-warning in test stage

initrds=(
	/osimage/yocto/yocto-tiny-i386-2016-04-22.cgz
	/lkp/scheduled/vm-kbuild-yocto-ia32-9/trinity-300s-yocto-tiny-i386-2016-04-22.cgz-fbc14616f483788afabe77d05bfb99883dc66c73-20170331-102676-cxuzks-0.cgz
	/lkp/lkp/lkp-i386.cgz


To reproduce:

        git clone https://github.com/01org/lkp-tests.git
        cd lkp-tests
        bin/lkp qemu -k <bzImage> job-script  # job-script is attached in this email



Thanks,
Kernel Test Robot

[-- Attachment #2: config-4.11.0-rc4-00072-gfbc1461 --]
[-- Type: text/plain, Size: 121612 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 4.11.0-rc4 Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_MMU=y
CONFIG_ARCH_MMAP_RND_BITS_MIN=28
CONFIG_ARCH_MMAP_RND_BITS_MAX=32
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=8
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=16
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_PGTABLE_LEVELS=4
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_KERNEL_XZ=y
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
# CONFIG_SYSVIPC is not set
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_FHANDLE=y
CONFIG_USELIB=y
CONFIG_AUDIT=y
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_GENERIC_IRQ_CHIP=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
# CONFIG_IRQ_DOMAIN_DEBUG is not set
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y

#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set

#
# RCU Subsystem
#
CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
# CONFIG_TASKS_RCU is not set
CONFIG_RCU_STALL_COMMON=y
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
CONFIG_ARCH_SUPPORTS_INT128=y
CONFIG_CGROUPS=y
# CONFIG_MEMCG is not set
# CONFIG_BLK_CGROUP is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_CFS_BANDWIDTH is not set
CONFIG_RT_GROUP_SCHED=y
# CONFIG_CGROUP_PIDS is not set
# CONFIG_CGROUP_RDMA is not set
CONFIG_CGROUP_FREEZER=y
# CONFIG_CGROUP_HUGETLB is not set
CONFIG_CPUSETS=y
# CONFIG_PROC_PID_CPUSET is not set
# CONFIG_CGROUP_DEVICE is not set
# CONFIG_CGROUP_CPUACCT is not set
# CONFIG_CGROUP_PERF is not set
CONFIG_CGROUP_DEBUG=y
# CONFIG_SOCK_CGROUP_DATA is not set
# CONFIG_CHECKPOINT_RESTORE is not set
# CONFIG_NAMESPACES is not set
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
# CONFIG_RD_LZO is not set
CONFIG_RD_LZ4=y
CONFIG_INITRAMFS_COMPRESSION=".gz"
# CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE is not set
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BPF=y
CONFIG_EXPERT=y
# CONFIG_UID16 is not set
CONFIG_MULTIUSER=y
CONFIG_SGETMASK_SYSCALL=y
CONFIG_SYSFS_SYSCALL=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_POSIX_TIMERS=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PRINTK=y
CONFIG_PRINTK_NMI=y
CONFIG_BUG=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
# CONFIG_BPF_SYSCALL is not set
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_ADVISE_SYSCALLS=y
# CONFIG_USERFAULTFD is not set
CONFIG_PCI_QUIRKS=y
CONFIG_MEMBARRIER=y
CONFIG_EMBEDDED=y
CONFIG_HAVE_PERF_EVENTS=y
# CONFIG_PC104 is not set

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
CONFIG_COMPAT_BRK=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_SLAB_FREELIST_RANDOM is not set
CONFIG_SLUB_CPU_PARTIAL=y
# CONFIG_SYSTEM_DATA_VERIFICATION is not set
# CONFIG_PROFILING is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
# CONFIG_JUMP_LABEL is not set
# CONFIG_UPROBES is not set
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_NMI=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_CLK=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_GCC_PLUGINS=y
# CONFIG_GCC_PLUGINS is not set
CONFIG_HAVE_CC_STACKPROTECTOR=y
# CONFIG_CC_STACKPROTECTOR is not set
CONFIG_CC_STACKPROTECTOR_NONE=y
# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_HAVE_EXIT_THREAD=y
CONFIG_ARCH_MMAP_RND_BITS=28
CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
CONFIG_HAVE_COPY_THREAD_TLS=y
CONFIG_HAVE_STACK_VALIDATION=y
# CONFIG_HAVE_ARCH_HASH is not set
# CONFIG_ISA_BUS_API is not set
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y
# CONFIG_CPU_NO_EFFICIENT_FFS is not set
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
# CONFIG_ARCH_OPTIONAL_KERNEL_RWX is not set
# CONFIG_ARCH_OPTIONAL_KERNEL_RWX_DEFAULT is not set
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
# CONFIG_MODULES is not set
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
CONFIG_BLK_SCSI_REQUEST=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
# CONFIG_BLK_DEV_ZONED is not set
# CONFIG_BLK_CMDLINE_PARSER is not set
# CONFIG_BLK_WBT is not set
CONFIG_BLK_DEBUG_FS=y
# CONFIG_BLK_SED_OPAL is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_BLOCK_COMPAT=y
CONFIG_BLK_MQ_PCI=y
CONFIG_BLK_MQ_VIRTIO=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_MQ_IOSCHED_DEADLINE=y
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
CONFIG_QUEUED_SPINLOCKS=y
CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
CONFIG_QUEUED_RWLOCKS=y
CONFIG_FREEZER=y

#
# Processor type and features
#
# CONFIG_ZONE_DMA is not set
CONFIG_SMP=y
CONFIG_X86_FEATURE_NAMES=y
CONFIG_X86_FAST_FEATURE_TESTS=y
# CONFIG_X86_X2APIC is not set
CONFIG_X86_MPPARSE=y
# CONFIG_GOLDFISH is not set
# CONFIG_INTEL_RDT_A is not set
# CONFIG_X86_EXTENDED_PLATFORM is not set
# CONFIG_X86_INTEL_LPSS is not set
# CONFIG_X86_AMD_PLATFORM_DEVICE is not set
# CONFIG_IOSF_MBI is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_DEBUG is not set
# CONFIG_PARAVIRT_SPINLOCKS is not set
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_512GB=y
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_DEBUG_FS=y
# CONFIG_XEN_PVH is not set
CONFIG_KVM_GUEST=y
# CONFIG_KVM_DEBUG_FS is not set
CONFIG_PARAVIRT_TIME_ACCOUNTING=y
CONFIG_PARAVIRT_CLOCK=y
CONFIG_NO_BOOTMEM=y
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
# CONFIG_PROCESSOR_SELECT is not set
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=8
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
CONFIG_SCHED_MC_PRIO=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_INTEL is not set
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set

#
# Performance monitoring
#
CONFIG_PERF_EVENTS_INTEL_UNCORE=y
CONFIG_PERF_EVENTS_INTEL_RAPL=y
CONFIG_PERF_EVENTS_INTEL_CSTATE=y
# CONFIG_PERF_EVENTS_AMD_POWER is not set
# CONFIG_VM86 is not set
CONFIG_X86_16BIT=y
CONFIG_X86_ESPFIX64=y
CONFIG_X86_VSYSCALL_EMULATION=y
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_X86_DIRECT_GBPAGES=y
# CONFIG_NUMA is not set
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
# CONFIG_SPARSEMEM_VMEMMAP is not set
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_MEMORY_ISOLATION=y
# CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_MEMORY_BALLOON=y
# CONFIG_BALLOON_COMPACTION is not set
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_VIRT_TO_BUS=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
# CONFIG_HWPOISON_INJECT is not set
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_TRANSPARENT_HUGE_PAGECACHE=y
# CONFIG_CLEANCACHE is not set
# CONFIG_FRONTSWAP is not set
# CONFIG_CMA is not set
# CONFIG_ZPOOL is not set
# CONFIG_ZBUD is not set
CONFIG_ZSMALLOC=y
# CONFIG_PGTABLE_MAPPING is not set
# CONFIG_ZSMALLOC_STAT is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
CONFIG_ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT=y
# CONFIG_IDLE_PAGE_TRACKING is not set
CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
CONFIG_ARCH_HAS_PKEYS=y
# CONFIG_X86_PMEM_LEGACY is not set
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK is not set
CONFIG_X86_RESERVE_LOW=64
# CONFIG_MTRR is not set
CONFIG_ARCH_RANDOM=y
# CONFIG_X86_SMAP is not set
# CONFIG_X86_INTEL_MPX is not set
CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y
CONFIG_EFI=y
CONFIG_EFI_STUB=y
# CONFIG_EFI_MIXED is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_KEXEC_FILE is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
# CONFIG_RANDOMIZE_BASE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_BOOTPARAM_HOTPLUG_CPU0=y
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
CONFIG_COMPAT_VDSO=y
# CONFIG_LEGACY_VSYSCALL_NATIVE is not set
CONFIG_LEGACY_VSYSCALL_EMULATE=y
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_MODIFY_LDT_SYSCALL=y
CONFIG_HAVE_LIVEPATCH=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
# CONFIG_SUSPEND is not set
CONFIG_HIBERNATE_CALLBACKS=y
# CONFIG_HIBERNATION is not set
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_ADVANCED_DEBUG is not set
CONFIG_PM_SLEEP_DEBUG=y
# CONFIG_PM_TRACE_RTC is not set
CONFIG_PM_CLK=y
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
CONFIG_ACPI=y
CONFIG_ACPI_LEGACY_TABLES_LOOKUP=y
CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=y
CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=y
# CONFIG_ACPI_DEBUGGER is not set
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_REV_OVERRIDE_POSSIBLE=y
CONFIG_ACPI_EC_DEBUGFS=y
# CONFIG_ACPI_AC is not set
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_CPU_FREQ_PSS=y
CONFIG_ACPI_PROCESSOR_CSTATE=y
CONFIG_ACPI_PROCESSOR_IDLE=y
CONFIG_ACPI_CPPC_LIB=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_TABLE_UPGRADE=y
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_IOAPIC=y
CONFIG_ACPI_SBS=y
CONFIG_ACPI_HED=y
# CONFIG_ACPI_CUSTOM_METHOD is not set
# CONFIG_ACPI_BGRT is not set
# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
# CONFIG_ACPI_NFIT is not set
CONFIG_HAVE_ACPI_APEI=y
CONFIG_HAVE_ACPI_APEI_NMI=y
# CONFIG_ACPI_APEI is not set
# CONFIG_DPTF_POWER is not set
# CONFIG_ACPI_EXTLOG is not set
# CONFIG_PMIC_OPREGION is not set
# CONFIG_ACPI_CONFIGFS is not set
CONFIG_SFI=y

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
# CONFIG_CPU_FREQ_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_GOV_SCHEDUTIL is not set

#
# CPU frequency scaling drivers
#
CONFIG_X86_INTEL_PSTATE=y
CONFIG_X86_PCC_CPUFREQ=y
# CONFIG_X86_ACPI_CPUFREQ is not set
CONFIG_X86_SPEEDSTEP_CENTRINO=y
CONFIG_X86_P4_CLOCKMOD=y

#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=y

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
# CONFIG_CPU_IDLE_GOV_MENU is not set
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
CONFIG_INTEL_IDLE=y

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MMCONFIG is not set
CONFIG_PCI_XEN=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
# CONFIG_PCIEPORTBUS is not set
CONFIG_PCI_BUS_ADDR_T_64BIT=y
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
# CONFIG_PCI_DEBUG is not set
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
# CONFIG_PCI_STUB is not set
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
# CONFIG_PCI_IOV is not set
CONFIG_PCI_PRI=y
# CONFIG_PCI_PASID is not set
CONFIG_PCI_LABEL=y
CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_ACPI is not set
# CONFIG_HOTPLUG_PCI_CPCI is not set
# CONFIG_HOTPLUG_PCI_SHPC is not set

#
# DesignWare PCI Core Support
#
# CONFIG_PCIE_DW_PLAT is not set

#
# PCI host controller drivers
#
# CONFIG_VMD is not set
# CONFIG_ISA_BUS is not set
# CONFIG_ISA_DMA_API is not set
CONFIG_AMD_NB=y
# CONFIG_PCCARD is not set
# CONFIG_RAPIDIO is not set
# CONFIG_X86_SYSFB is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ELFCORE=y
CONFIG_BINFMT_SCRIPT=y
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
# CONFIG_COREDUMP is not set
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=y
CONFIG_X86_X32=y
CONFIG_COMPAT_32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_KEYS_COMPAT=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
CONFIG_NET_INGRESS=y

#
# Networking options
#
# CONFIG_PACKET is not set
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=y
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
# CONFIG_IP_PNP_RARP is not set
CONFIG_NET_IPIP=y
# CONFIG_NET_IPGRE_DEMUX is not set
CONFIG_NET_IP_TUNNEL=y
CONFIG_SYN_COOKIES=y
# CONFIG_NET_UDP_TUNNEL is not set
# CONFIG_NET_FOU is not set
# CONFIG_NET_FOU_IP_TUNNELS is not set
# CONFIG_INET_AH is not set
CONFIG_INET_ESP=y
# CONFIG_INET_ESP_OFFLOAD is not set
CONFIG_INET_IPCOMP=y
CONFIG_INET_XFRM_TUNNEL=y
CONFIG_INET_TUNNEL=y
CONFIG_INET_XFRM_MODE_TRANSPORT=y
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
CONFIG_INET_XFRM_MODE_BEET=y
# CONFIG_INET_DIAG is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=y
CONFIG_TCP_CONG_HTCP=y
# CONFIG_TCP_CONG_HSTCP is not set
CONFIG_TCP_CONG_HYBLA=y
CONFIG_TCP_CONG_VEGAS=y
# CONFIG_TCP_CONG_NV is not set
CONFIG_TCP_CONG_SCALABLE=y
# CONFIG_TCP_CONG_LP is not set
CONFIG_TCP_CONG_VENO=y
CONFIG_TCP_CONG_YEAH=y
# CONFIG_TCP_CONG_ILLINOIS is not set
# CONFIG_TCP_CONG_DCTCP is not set
# CONFIG_TCP_CONG_CDG is not set
# CONFIG_TCP_CONG_BBR is not set
# CONFIG_DEFAULT_BIC is not set
# CONFIG_DEFAULT_CUBIC is not set
# CONFIG_DEFAULT_HTCP is not set
# CONFIG_DEFAULT_HYBLA is not set
# CONFIG_DEFAULT_VEGAS is not set
CONFIG_DEFAULT_VENO=y
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="veno"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NET_PTP_CLASSIFY=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_INGRESS=y
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_ACCT=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
# CONFIG_NF_CONNTRACK is not set
CONFIG_NF_LOG_COMMON=y
# CONFIG_NF_LOG_NETDEV is not set
# CONFIG_NF_TABLES is not set
CONFIG_NETFILTER_XTABLES=y

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=y
CONFIG_NETFILTER_XT_SET=y

#
# Xtables targets
#
# CONFIG_NETFILTER_XT_TARGET_AUDIT is not set
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
CONFIG_NETFILTER_XT_TARGET_HMARK=y
# CONFIG_NETFILTER_XT_TARGET_IDLETIMER is not set
CONFIG_NETFILTER_XT_TARGET_LED=y
CONFIG_NETFILTER_XT_TARGET_LOG=y
# CONFIG_NETFILTER_XT_TARGET_MARK is not set
# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
# CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
# CONFIG_NETFILTER_XT_TARGET_TEE is not set
# CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
# CONFIG_NETFILTER_XT_MATCH_CGROUP is not set
# CONFIG_NETFILTER_XT_MATCH_COMMENT is not set
CONFIG_NETFILTER_XT_MATCH_CPU=y
CONFIG_NETFILTER_XT_MATCH_DCCP=y
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=y
# CONFIG_NETFILTER_XT_MATCH_DSCP is not set
# CONFIG_NETFILTER_XT_MATCH_ECN is not set
CONFIG_NETFILTER_XT_MATCH_ESP=y
# CONFIG_NETFILTER_XT_MATCH_HASHLIMIT is not set
# CONFIG_NETFILTER_XT_MATCH_HL is not set
# CONFIG_NETFILTER_XT_MATCH_IPCOMP is not set
# CONFIG_NETFILTER_XT_MATCH_IPRANGE is not set
# CONFIG_NETFILTER_XT_MATCH_L2TP is not set
# CONFIG_NETFILTER_XT_MATCH_LENGTH is not set
# CONFIG_NETFILTER_XT_MATCH_LIMIT is not set
# CONFIG_NETFILTER_XT_MATCH_MAC is not set
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_NFACCT=y
CONFIG_NETFILTER_XT_MATCH_OSF=y
CONFIG_NETFILTER_XT_MATCH_OWNER=y
# CONFIG_NETFILTER_XT_MATCH_POLICY is not set
# CONFIG_NETFILTER_XT_MATCH_PKTTYPE is not set
CONFIG_NETFILTER_XT_MATCH_QUOTA=y
# CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
CONFIG_NETFILTER_XT_MATCH_REALM=y
# CONFIG_NETFILTER_XT_MATCH_RECENT is not set
CONFIG_NETFILTER_XT_MATCH_SCTP=y
# CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
CONFIG_NETFILTER_XT_MATCH_STRING=y
# CONFIG_NETFILTER_XT_MATCH_TCPMSS is not set
CONFIG_NETFILTER_XT_MATCH_TIME=y
# CONFIG_NETFILTER_XT_MATCH_U32 is not set
CONFIG_IP_SET=y
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=y
# CONFIG_IP_SET_BITMAP_IPMAC is not set
CONFIG_IP_SET_BITMAP_PORT=y
CONFIG_IP_SET_HASH_IP=y
# CONFIG_IP_SET_HASH_IPMARK is not set
CONFIG_IP_SET_HASH_IPPORT=y
CONFIG_IP_SET_HASH_IPPORTIP=y
# CONFIG_IP_SET_HASH_IPPORTNET is not set
# CONFIG_IP_SET_HASH_IPMAC is not set
# CONFIG_IP_SET_HASH_MAC is not set
# CONFIG_IP_SET_HASH_NETPORTNET is not set
# CONFIG_IP_SET_HASH_NET is not set
# CONFIG_IP_SET_HASH_NETNET is not set
# CONFIG_IP_SET_HASH_NETPORT is not set
CONFIG_IP_SET_HASH_NETIFACE=y
CONFIG_IP_SET_LIST_SET=y
CONFIG_IP_VS=y
CONFIG_IP_VS_DEBUG=y
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
# CONFIG_IP_VS_PROTO_TCP is not set
# CONFIG_IP_VS_PROTO_UDP is not set
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
# CONFIG_IP_VS_PROTO_AH is not set
# CONFIG_IP_VS_PROTO_SCTP is not set

#
# IPVS scheduler
#
# CONFIG_IP_VS_RR is not set
# CONFIG_IP_VS_WRR is not set
CONFIG_IP_VS_LC=y
CONFIG_IP_VS_WLC=y
# CONFIG_IP_VS_FO is not set
# CONFIG_IP_VS_OVF is not set
CONFIG_IP_VS_LBLC=y
# CONFIG_IP_VS_LBLCR is not set
CONFIG_IP_VS_DH=y
# CONFIG_IP_VS_SH is not set
CONFIG_IP_VS_SED=y
# CONFIG_IP_VS_NQ is not set

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS application helper
#

#
# IP: Netfilter Configuration
#
# CONFIG_NF_DEFRAG_IPV4 is not set
# CONFIG_NF_SOCKET_IPV4 is not set
# CONFIG_NF_DUP_IPV4 is not set
# CONFIG_NF_LOG_ARP is not set
CONFIG_NF_LOG_IPV4=y
# CONFIG_NF_REJECT_IPV4 is not set
# CONFIG_IP_NF_IPTABLES is not set
# CONFIG_IP_NF_ARPTABLES is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
# CONFIG_BRIDGE is not set
CONFIG_HAVE_NET_DSA=y
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
# CONFIG_VSOCKETS is not set
# CONFIG_NETLINK_DIAG is not set
# CONFIG_MPLS is not set
# CONFIG_HSR is not set
# CONFIG_NET_SWITCHDEV is not set
# CONFIG_NET_L3_MASTER_DEV is not set
# CONFIG_NET_NCSI is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
# CONFIG_CGROUP_NET_PRIO is not set
# CONFIG_CGROUP_NET_CLASSID is not set
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
# CONFIG_STREAM_PARSER is not set
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set
# CONFIG_LIB80211 is not set

#
# CFG80211 needs to be enabled for MAC80211
#
CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_DEBUG=y
CONFIG_CAIF=y
# CONFIG_CAIF_DEBUG is not set
CONFIG_CAIF_NETDEV=y
# CONFIG_CAIF_USB is not set
CONFIG_CEPH_LIB=y
CONFIG_CEPH_LIB_PRETTYDEBUG=y
# CONFIG_CEPH_LIB_USE_DNS_RESOLVER is not set
# CONFIG_NFC is not set
# CONFIG_PSAMPLE is not set
# CONFIG_NET_IFE is not set
# CONFIG_LWTUNNEL is not set
CONFIG_DST_CACHE=y
CONFIG_GRO_CELLS=y
# CONFIG_NET_DEVLINK is not set
CONFIG_MAY_USE_DEVLINK=y
CONFIG_HAVE_EBPF_JIT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER=y
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set
CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_SPI=y
CONFIG_REGMAP_IRQ=y
CONFIG_DMA_SHARED_BUFFER=y
# CONFIG_DMA_FENCE_TRACE is not set

#
# Bus devices
#
CONFIG_CONNECTOR=y
# CONFIG_PROC_EVENTS is not set
CONFIG_MTD=y
# CONFIG_MTD_REDBOOT_PARTS is not set
# CONFIG_MTD_CMDLINE_PARTS is not set
# CONFIG_MTD_AR7_PARTS is not set

#
# User Modules And Translation Layers
#
CONFIG_MTD_BLKDEVS=y
# CONFIG_MTD_BLOCK is not set
# CONFIG_MTD_BLOCK_RO is not set
CONFIG_FTL=y
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set
# CONFIG_RFD_FTL is not set
CONFIG_SSFDC=y
# CONFIG_SM_FTL is not set
# CONFIG_MTD_OOPS is not set
CONFIG_MTD_SWAP=y
# CONFIG_MTD_PARTITIONED_MASTER is not set

#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=y
CONFIG_MTD_JEDECPROBE=y
CONFIG_MTD_GEN_PROBE=y
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_CFI_AMDSTD=y
# CONFIG_MTD_CFI_STAA is not set
CONFIG_MTD_CFI_UTIL=y
CONFIG_MTD_RAM=y
# CONFIG_MTD_ROM is not set
CONFIG_MTD_ABSENT=y

#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
# CONFIG_MTD_PHYSMAP is not set
# CONFIG_MTD_SBC_GXX is not set
# CONFIG_MTD_AMD76XROM is not set
CONFIG_MTD_ICHXROM=y
CONFIG_MTD_ESB2ROM=y
# CONFIG_MTD_CK804XROM is not set
# CONFIG_MTD_SCB2_FLASH is not set
# CONFIG_MTD_NETtel is not set
CONFIG_MTD_L440GX=y
CONFIG_MTD_PCI=y
CONFIG_MTD_GPIO_ADDR=y
# CONFIG_MTD_INTEL_VR_NOR is not set
CONFIG_MTD_PLATRAM=y
# CONFIG_MTD_LATCH_ADDR is not set

#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
CONFIG_MTD_DATAFLASH=y
CONFIG_MTD_DATAFLASH_WRITE_VERIFY=y
# CONFIG_MTD_DATAFLASH_OTP is not set
CONFIG_MTD_SST25L=y
CONFIG_MTD_SLRAM=y
CONFIG_MTD_PHRAM=y
CONFIG_MTD_MTDRAM=y
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=y

#
# Disk-On-Chip Device Drivers
#
CONFIG_MTD_DOCG3=y
CONFIG_BCH_CONST_M=14
CONFIG_BCH_CONST_T=4
# CONFIG_MTD_NAND is not set
# CONFIG_MTD_ONENAND is not set

#
# LPDDR & LPDDR2 PCM memory drivers
#
# CONFIG_MTD_LPDDR is not set
# CONFIG_MTD_SPI_NOR is not set
CONFIG_MTD_UBI=y
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
CONFIG_MTD_UBI_FASTMAP=y
# CONFIG_MTD_UBI_GLUEBI is not set
# CONFIG_MTD_UBI_BLOCK is not set
# CONFIG_OF is not set
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
# CONFIG_PARPORT is not set
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_NULL_BLK is not set
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=y
# CONFIG_ZRAM is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SKD is not set
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=y
CONFIG_XEN_BLKDEV_FRONTEND=y
# CONFIG_XEN_BLKDEV_BACKEND is not set
CONFIG_VIRTIO_BLK=y
# CONFIG_VIRTIO_BLK_SCSI is not set
# CONFIG_BLK_DEV_HD is not set
# CONFIG_BLK_DEV_RBD is not set
# CONFIG_BLK_DEV_RSXX is not set
# CONFIG_BLK_DEV_NVME is not set
# CONFIG_NVME_FC is not set
# CONFIG_NVME_TARGET is not set

#
# Misc devices
#
CONFIG_SENSORS_LIS3LV02D=y
CONFIG_AD525X_DPOT=y
# CONFIG_AD525X_DPOT_I2C is not set
CONFIG_AD525X_DPOT_SPI=y
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBM_ASM is not set
CONFIG_PHANTOM=y
CONFIG_SGI_IOC4=y
CONFIG_TIFM_CORE=y
CONFIG_TIFM_7XX1=y
# CONFIG_ICS932S401 is not set
CONFIG_ENCLOSURE_SERVICES=y
# CONFIG_HP_ILO is not set
# CONFIG_APDS9802ALS is not set
# CONFIG_ISL29003 is not set
CONFIG_ISL29020=y
CONFIG_SENSORS_TSL2550=y
CONFIG_SENSORS_BH1770=y
# CONFIG_SENSORS_APDS990X is not set
# CONFIG_HMC6352 is not set
CONFIG_DS1682=y
CONFIG_TI_DAC7512=y
CONFIG_USB_SWITCH_FSA9480=y
CONFIG_LATTICE_ECP3_CONFIG=y
# CONFIG_SRAM is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_AT25 is not set
# CONFIG_EEPROM_LEGACY is not set
CONFIG_EEPROM_MAX6875=y
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_EEPROM_93XX46 is not set
# CONFIG_EEPROM_IDT_89HPESX is not set
# CONFIG_CB710_CORE is not set

#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
CONFIG_SENSORS_LIS3_I2C=y

#
# Altera FPGA firmware download module
#
# CONFIG_ALTERA_STAPL is not set
# CONFIG_INTEL_MEI is not set
# CONFIG_INTEL_MEI_ME is not set
# CONFIG_INTEL_MEI_TXE is not set
# CONFIG_VMWARE_VMCI is not set

#
# Intel MIC Bus Driver
#
# CONFIG_INTEL_MIC_BUS is not set

#
# SCIF Bus Driver
#
# CONFIG_SCIF_BUS is not set

#
# VOP Bus Driver
#
# CONFIG_VOP_BUS is not set

#
# Intel MIC Host Driver
#

#
# Intel MIC Card Driver
#

#
# SCIF Driver
#

#
# Intel MIC Coprocessor State Management (COSM) Drivers
#

#
# VOP Driver
#
# CONFIG_GENWQE is not set
# CONFIG_ECHO is not set
# CONFIG_CXL_BASE is not set
# CONFIG_CXL_AFU_DRIVER_OPS is not set
CONFIG_HAVE_IDE=y
CONFIG_IDE=y

#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
CONFIG_IDE_XFER_MODE=y
CONFIG_IDE_TIMINGS=y
CONFIG_IDE_ATAPI=y
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_IDE_GD=y
# CONFIG_IDE_GD_ATA is not set
# CONFIG_IDE_GD_ATAPI is not set
# CONFIG_BLK_DEV_IDECD is not set
CONFIG_BLK_DEV_IDETAPE=y
CONFIG_BLK_DEV_IDEACPI=y
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y

#
# IDE chipset support/bugfixes
#
# CONFIG_IDE_GENERIC is not set
CONFIG_BLK_DEV_PLATFORM=y
CONFIG_BLK_DEV_CMD640=y
CONFIG_BLK_DEV_CMD640_ENHANCED=y
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEDMA_SFF=y

#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_IDEPCI_PCIBUS_ORDER is not set
CONFIG_BLK_DEV_OFFBOARD=y
# CONFIG_BLK_DEV_GENERIC is not set
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_BLK_DEV_AEC62XX=y
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
CONFIG_BLK_DEV_CMD64X=y
# CONFIG_BLK_DEV_TRIFLEX is not set
CONFIG_BLK_DEV_HPT366=y
CONFIG_BLK_DEV_JMICRON=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_IT8172=y
# CONFIG_BLK_DEV_IT8213 is not set
CONFIG_BLK_DEV_IT821X=y
CONFIG_BLK_DEV_NS87415=y
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_BLK_DEV_SVWKS=y
CONFIG_BLK_DEV_SIIMAGE=y
CONFIG_BLK_DEV_SIS5513=y
CONFIG_BLK_DEV_SLC90E66=y
CONFIG_BLK_DEV_TRM290=y
CONFIG_BLK_DEV_VIA82CXXX=y
# CONFIG_BLK_DEV_TC86C001 is not set
CONFIG_BLK_DEV_IDEDMA=y

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_NETLINK=y
# CONFIG_SCSI_MQ_DEFAULT is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set
CONFIG_SCSI_ENCLOSURE=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=y
CONFIG_SCSI_SAS_ATTRS=y
CONFIG_SCSI_SAS_LIBSAS=y
# CONFIG_SCSI_SAS_ATA is not set
# CONFIG_SCSI_SAS_HOST_SMP is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
CONFIG_ISCSI_BOOT_SYSFS=y
CONFIG_SCSI_CXGB3_ISCSI=y
CONFIG_SCSI_CXGB4_ISCSI=y
CONFIG_SCSI_BNX2_ISCSI=y
# CONFIG_SCSI_BNX2X_FCOE is not set
CONFIG_BE2ISCSI=y
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
CONFIG_SCSI_HPSA=y
CONFIG_SCSI_3W_9XXX=y
# CONFIG_SCSI_3W_SAS is not set
# CONFIG_SCSI_ACARD is not set
CONFIG_SCSI_AACRAID=y
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
CONFIG_SCSI_MVSAS=y
CONFIG_SCSI_MVSAS_DEBUG=y
CONFIG_SCSI_MVSAS_TASKLET=y
CONFIG_SCSI_MVUMI=y
CONFIG_SCSI_DPT_I2O=y
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_SCSI_ESAS2R is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_MEGARAID_SAS=y
CONFIG_SCSI_MPT3SAS=y
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT3SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS=y
# CONFIG_SCSI_SMARTPQI is not set
CONFIG_SCSI_UFSHCD=y
# CONFIG_SCSI_UFSHCD_PCI is not set
# CONFIG_SCSI_UFSHCD_PLATFORM is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_VMWARE_PVSCSI is not set
# CONFIG_XEN_SCSI_FRONTEND is not set
CONFIG_LIBFC=y
CONFIG_LIBFCOE=y
CONFIG_FCOE=y
CONFIG_FCOE_FNIC=y
# CONFIG_SCSI_SNIC is not set
CONFIG_SCSI_DMX3191D=y
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_ISCI is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_STEX is not set
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
CONFIG_SCSI_IPR=y
# CONFIG_SCSI_IPR_TRACE is not set
CONFIG_SCSI_IPR_DUMP=y
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA_FC=y
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_WD719X is not set
# CONFIG_SCSI_DEBUG is not set
CONFIG_SCSI_PMCRAID=y
CONFIG_SCSI_PM8001=y
# CONFIG_SCSI_BFA_FC is not set
CONFIG_SCSI_VIRTIO=y
CONFIG_SCSI_CHELSIO_FCOE=y
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
# CONFIG_ATA_ACPI is not set
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
# CONFIG_SATA_AHCI is not set
CONFIG_SATA_AHCI_PLATFORM=y
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
CONFIG_SATA_SIL24=y
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_ATA_BMDMA is not set

#
# PIO-only SFF controllers
#
CONFIG_PATA_CMD640_PCI=y
CONFIG_PATA_MPIIX=y
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
CONFIG_PATA_PLATFORM=y
# CONFIG_PATA_RZ1000 is not set

#
# Generic fallback / legacy drivers
#
# CONFIG_PATA_LEGACY is not set
# CONFIG_MD is not set
# CONFIG_TARGET_CORE is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_FIREWIRE_NOSY is not set
CONFIG_MACINTOSH_DRIVERS=y
# CONFIG_MAC_EMUMOUSEBTN is not set
CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y
CONFIG_BONDING=y
CONFIG_DUMMY=y
CONFIG_EQUALIZER=y
# CONFIG_NET_FC is not set
# CONFIG_NET_TEAM is not set
CONFIG_MACVLAN=y
# CONFIG_MACVTAP is not set
# CONFIG_VXLAN is not set
# CONFIG_MACSEC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_TUN is not set
# CONFIG_TUN_VNET_CROSS_LE is not set
# CONFIG_VETH is not set
CONFIG_VIRTIO_NET=y
# CONFIG_NLMON is not set
CONFIG_SUNGEM_PHY=y
# CONFIG_ARCNET is not set

#
# CAIF transport drivers
#
CONFIG_CAIF_TTY=y
CONFIG_CAIF_SPI_SLAVE=y
# CONFIG_CAIF_SPI_SYNC is not set
# CONFIG_CAIF_HSI is not set
# CONFIG_CAIF_VIRTIO is not set

#
# Distributed Switch Architecture drivers
#
CONFIG_ETHERNET=y
CONFIG_MDIO=y
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
CONFIG_NET_VENDOR_AGERE=y
CONFIG_ET131X=y
CONFIG_NET_VENDOR_ALACRITECH=y
CONFIG_SLICOSS=y
CONFIG_NET_VENDOR_ALTEON=y
# CONFIG_ACENIC is not set
# CONFIG_ALTERA_TSE is not set
CONFIG_NET_VENDOR_AMAZON=y
# CONFIG_ENA_ETHERNET is not set
CONFIG_NET_VENDOR_AMD=y
# CONFIG_AMD8111_ETH is not set
# CONFIG_PCNET32 is not set
# CONFIG_AMD_XGBE is not set
# CONFIG_AMD_XGBE_HAVE_ECC is not set
CONFIG_NET_VENDOR_AQUANTIA=y
# CONFIG_AQTION is not set
CONFIG_NET_VENDOR_ARC=y
CONFIG_NET_VENDOR_ATHEROS=y
CONFIG_ATL2=y
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
CONFIG_ATL1C=y
# CONFIG_ALX is not set
# CONFIG_NET_VENDOR_AURORA is not set
CONFIG_NET_CADENCE=y
CONFIG_MACB=y
# CONFIG_MACB_PCI is not set
CONFIG_NET_VENDOR_BROADCOM=y
CONFIG_B44=y
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
# CONFIG_BCMGENET is not set
CONFIG_BNX2=y
CONFIG_CNIC=y
# CONFIG_TIGON3 is not set
# CONFIG_BNX2X is not set
# CONFIG_BNXT is not set
# CONFIG_NET_VENDOR_BROCADE is not set
CONFIG_NET_VENDOR_CAVIUM=y
# CONFIG_THUNDER_NIC_PF is not set
# CONFIG_THUNDER_NIC_VF is not set
# CONFIG_THUNDER_NIC_BGX is not set
# CONFIG_THUNDER_NIC_RGX is not set
# CONFIG_LIQUIDIO is not set
# CONFIG_LIQUIDIO_VF is not set
CONFIG_NET_VENDOR_CHELSIO=y
# CONFIG_CHELSIO_T1 is not set
CONFIG_CHELSIO_T3=y
CONFIG_CHELSIO_T4=y
CONFIG_CHELSIO_T4VF=y
CONFIG_CHELSIO_LIB=y
CONFIG_NET_VENDOR_CISCO=y
# CONFIG_ENIC is not set
# CONFIG_CX_ECAT is not set
# CONFIG_DNET is not set
# CONFIG_NET_VENDOR_DEC is not set
CONFIG_NET_VENDOR_DLINK=y
# CONFIG_DL2K is not set
CONFIG_SUNDANCE=y
CONFIG_SUNDANCE_MMIO=y
# CONFIG_NET_VENDOR_EMULEX is not set
CONFIG_NET_VENDOR_EZCHIP=y
CONFIG_NET_VENDOR_EXAR=y
# CONFIG_S2IO is not set
# CONFIG_VXGE is not set
# CONFIG_NET_VENDOR_HP is not set
CONFIG_NET_VENDOR_INTEL=y
# CONFIG_E100 is not set
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_E1000E_HWTS=y
CONFIG_IGB=y
CONFIG_IGB_HWMON=y
# CONFIG_IGBVF is not set
# CONFIG_IXGB is not set
CONFIG_IXGBE=y
CONFIG_IXGBE_HWMON=y
# CONFIG_IXGBEVF is not set
# CONFIG_I40E is not set
# CONFIG_I40EVF is not set
# CONFIG_FM10K is not set
CONFIG_NET_VENDOR_I825XX=y
# CONFIG_JME is not set
# CONFIG_NET_VENDOR_MARVELL is not set
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=y
CONFIG_MLX4_CORE=y
CONFIG_MLX4_DEBUG=y
# CONFIG_MLX5_CORE is not set
# CONFIG_MLXSW_CORE is not set
CONFIG_NET_VENDOR_MICREL=y
# CONFIG_KS8851 is not set
# CONFIG_KS8851_MLL is not set
# CONFIG_KSZ884X_PCI is not set
# CONFIG_NET_VENDOR_MICROCHIP is not set
CONFIG_NET_VENDOR_MYRI=y
# CONFIG_MYRI10GE is not set
# CONFIG_FEALNX is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
CONFIG_NET_VENDOR_NETRONOME=y
# CONFIG_NFP is not set
CONFIG_NET_VENDOR_NVIDIA=y
CONFIG_FORCEDETH=y
CONFIG_NET_VENDOR_OKI=y
# CONFIG_ETHOC is not set
# CONFIG_NET_PACKET_ENGINE is not set
CONFIG_NET_VENDOR_QLOGIC=y
# CONFIG_QLA3XXX is not set
CONFIG_QLCNIC=y
CONFIG_QLCNIC_HWMON=y
CONFIG_QLGE=y
# CONFIG_NETXEN_NIC is not set
# CONFIG_QED is not set
CONFIG_NET_VENDOR_QUALCOMM=y
# CONFIG_QCOM_EMAC is not set
# CONFIG_NET_VENDOR_REALTEK is not set
CONFIG_NET_VENDOR_RENESAS=y
CONFIG_NET_VENDOR_RDC=y
CONFIG_R6040=y
CONFIG_NET_VENDOR_ROCKER=y
CONFIG_NET_VENDOR_SAMSUNG=y
# CONFIG_SXGBE_ETH is not set
CONFIG_NET_VENDOR_SEEQ=y
# CONFIG_NET_VENDOR_SILAN is not set
CONFIG_NET_VENDOR_SIS=y
# CONFIG_SIS900 is not set
# CONFIG_SIS190 is not set
CONFIG_NET_VENDOR_SOLARFLARE=y
# CONFIG_SFC is not set
# CONFIG_SFC_FALCON is not set
CONFIG_NET_VENDOR_SMSC=y
CONFIG_EPIC100=y
# CONFIG_SMSC911X is not set
# CONFIG_SMSC9420 is not set
# CONFIG_NET_VENDOR_STMICRO is not set
CONFIG_NET_VENDOR_SUN=y
# CONFIG_HAPPYMEAL is not set
CONFIG_SUNGEM=y
# CONFIG_CASSINI is not set
CONFIG_NIU=y
CONFIG_NET_VENDOR_TEHUTI=y
CONFIG_TEHUTI=y
CONFIG_NET_VENDOR_TI=y
# CONFIG_TI_CPSW_ALE is not set
# CONFIG_TLAN is not set
# CONFIG_NET_VENDOR_VIA is not set
CONFIG_NET_VENDOR_WIZNET=y
CONFIG_WIZNET_W5100=y
CONFIG_WIZNET_W5300=y
# CONFIG_WIZNET_BUS_DIRECT is not set
# CONFIG_WIZNET_BUS_INDIRECT is not set
CONFIG_WIZNET_BUS_ANY=y
# CONFIG_WIZNET_W5100_SPI is not set
# CONFIG_FDDI is not set
CONFIG_HIPPI=y
# CONFIG_ROADRUNNER is not set
CONFIG_NET_SB1000=y
CONFIG_PHYLIB=y
CONFIG_SWPHY=y
# CONFIG_LED_TRIGGER_PHY is not set

#
# MDIO bus device drivers
#
# CONFIG_MDIO_BCM_UNIMAC is not set
CONFIG_MDIO_BITBANG=y
# CONFIG_MDIO_GPIO is not set
# CONFIG_MDIO_OCTEON is not set
# CONFIG_MDIO_THUNDER is not set

#
# MII PHY device drivers
#
CONFIG_AMD_PHY=y
# CONFIG_AQUANTIA_PHY is not set
CONFIG_AT803X_PHY=y
# CONFIG_BCM7XXX_PHY is not set
# CONFIG_BCM87XX_PHY is not set
CONFIG_BCM_NET_PHYLIB=y
CONFIG_BROADCOM_PHY=y
# CONFIG_CICADA_PHY is not set
CONFIG_DAVICOM_PHY=y
# CONFIG_DP83848_PHY is not set
# CONFIG_DP83867_PHY is not set
CONFIG_FIXED_PHY=y
CONFIG_ICPLUS_PHY=y
# CONFIG_INTEL_XWAY_PHY is not set
# CONFIG_LSI_ET1011C_PHY is not set
CONFIG_LXT_PHY=y
# CONFIG_MARVELL_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_MICROCHIP_PHY is not set
# CONFIG_MICROSEMI_PHY is not set
CONFIG_NATIONAL_PHY=y
# CONFIG_QSEMI_PHY is not set
CONFIG_REALTEK_PHY=y
# CONFIG_SMSC_PHY is not set
CONFIG_STE10XP=y
# CONFIG_TERANETICS_PHY is not set
CONFIG_VITESSE_PHY=y
# CONFIG_XILINX_GMII2RGMII is not set
CONFIG_MICREL_KS8995MA=y
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
CONFIG_USB_NET_DRIVERS=y
CONFIG_USB_CATC=y
# CONFIG_USB_KAWETH is not set
CONFIG_USB_PEGASUS=y
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_RTL8152 is not set
# CONFIG_USB_LAN78XX is not set
# CONFIG_USB_USBNET is not set
CONFIG_USB_IPHETH=y
# CONFIG_WLAN is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
CONFIG_XEN_NETDEV_FRONTEND=y
# CONFIG_XEN_NETDEV_BACKEND is not set
# CONFIG_VMXNET3 is not set
# CONFIG_FUJITSU_ES is not set
CONFIG_ISDN=y
# CONFIG_ISDN_I4L is not set
# CONFIG_ISDN_CAPI is not set
# CONFIG_ISDN_DRV_GIGASET is not set
# CONFIG_MISDN is not set
# CONFIG_NVM is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_LEDS=y
# CONFIG_INPUT_FF_MEMLESS is not set
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=y
CONFIG_INPUT_MATRIXKMAP=y

#
# Userland interfaces
#
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ADP5588=y
CONFIG_KEYBOARD_ADP5589=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_QT1070=y
# CONFIG_KEYBOARD_QT2160 is not set
CONFIG_KEYBOARD_LKKBD=y
CONFIG_KEYBOARD_GPIO=y
CONFIG_KEYBOARD_GPIO_POLLED=y
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
CONFIG_KEYBOARD_MATRIX=y
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
CONFIG_KEYBOARD_MAX7359=y
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_SAMSUNG is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_TM2_TOUCHKEY is not set
CONFIG_KEYBOARD_XTKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_PS2_ALPS is not set
CONFIG_MOUSE_PS2_BYD=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
# CONFIG_MOUSE_PS2_CYPRESS is not set
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
CONFIG_MOUSE_PS2_SENTELIC=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_PS2_FOCALTECH=y
# CONFIG_MOUSE_PS2_VMMOUSE is not set
CONFIG_MOUSE_SERIAL=y
# CONFIG_MOUSE_APPLETOUCH is not set
CONFIG_MOUSE_BCM5974=y
CONFIG_MOUSE_CYAPA=y
# CONFIG_MOUSE_ELAN_I2C is not set
CONFIG_MOUSE_VSXXXAA=y
# CONFIG_MOUSE_GPIO is not set
CONFIG_MOUSE_SYNAPTICS_I2C=y
CONFIG_MOUSE_SYNAPTICS_USB=y
# CONFIG_INPUT_JOYSTICK is not set
CONFIG_INPUT_TABLET=y
# CONFIG_TABLET_USB_ACECAD is not set
# CONFIG_TABLET_USB_AIPTEK is not set
CONFIG_TABLET_USB_GTCO=y
CONFIG_TABLET_USB_HANWANG=y
# CONFIG_TABLET_USB_KBTAB is not set
# CONFIG_TABLET_USB_PEGASUS is not set
# CONFIG_TABLET_SERIAL_WACOM4 is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_88PM860X_ONKEY=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_ARIZONA_HAPTICS is not set
# CONFIG_INPUT_BMA150 is not set
# CONFIG_INPUT_E3X0_BUTTON is not set
# CONFIG_INPUT_PCSPKR is not set
# CONFIG_INPUT_MAX77693_HAPTIC is not set
# CONFIG_INPUT_MMA8450 is not set
CONFIG_INPUT_APANEL=y
CONFIG_INPUT_GP2A=y
# CONFIG_INPUT_GPIO_BEEPER is not set
# CONFIG_INPUT_GPIO_TILT_POLLED is not set
# CONFIG_INPUT_GPIO_DECODER is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
CONFIG_INPUT_ATI_REMOTE2=y
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
CONFIG_INPUT_KXTJ9=y
CONFIG_INPUT_KXTJ9_POLLED_MODE=y
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
CONFIG_INPUT_CM109=y
# CONFIG_INPUT_REGULATOR_HAPTIC is not set
CONFIG_INPUT_RETU_PWRBUTTON=y
# CONFIG_INPUT_TWL6040_VIBRA is not set
# CONFIG_INPUT_UINPUT is not set
# CONFIG_INPUT_PALMAS_PWRBUTTON is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_PWM_BEEPER is not set
# CONFIG_INPUT_GPIO_ROTARY_ENCODER is not set
CONFIG_INPUT_DA9052_ONKEY=y
# CONFIG_INPUT_PCAP is not set
CONFIG_INPUT_ADXL34X=y
CONFIG_INPUT_ADXL34X_I2C=y
CONFIG_INPUT_ADXL34X_SPI=y
# CONFIG_INPUT_IMS_PCU is not set
# CONFIG_INPUT_CMA3000 is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
# CONFIG_INPUT_SOC_BUTTON_ARRAY is not set
# CONFIG_INPUT_DRV260X_HAPTICS is not set
# CONFIG_INPUT_DRV2665_HAPTICS is not set
# CONFIG_INPUT_DRV2667_HAPTICS is not set
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
CONFIG_SERIO_PCIPS2=y
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=y
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_SERIO_ARC_PS2 is not set
# CONFIG_USERIO is not set
CONFIG_GAMEPORT=y
# CONFIG_GAMEPORT_NS558 is not set
CONFIG_GAMEPORT_L4=y
# CONFIG_GAMEPORT_EMU10K1 is not set
CONFIG_GAMEPORT_FM801=y

#
# Character devices
#
CONFIG_TTY=y
# CONFIG_VT is not set
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_NOZOMI=y
CONFIG_N_GSM=y
CONFIG_TRACE_ROUTER=y
CONFIG_TRACE_SINK=y
CONFIG_DEVMEM=y
CONFIG_DEVKMEM=y

#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
CONFIG_SERIAL_8250_PNP=y
# CONFIG_SERIAL_8250_FINTEK is not set
CONFIG_SERIAL_8250_CONSOLE=y
# CONFIG_SERIAL_8250_PCI is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
# CONFIG_SERIAL_8250_FSL is not set
CONFIG_SERIAL_8250_DW=y
# CONFIG_SERIAL_8250_RT288X is not set
CONFIG_SERIAL_8250_LPSS=y
CONFIG_SERIAL_8250_MID=y
# CONFIG_SERIAL_8250_MOXA is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_MAX3100=y
# CONFIG_SERIAL_MAX310X is not set
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_SERIAL_SCCNXP=y
# CONFIG_SERIAL_SCCNXP_CONSOLE is not set
# CONFIG_SERIAL_SC16IS7XX is not set
CONFIG_SERIAL_ALTERA_JTAGUART=y
# CONFIG_SERIAL_ALTERA_JTAGUART_CONSOLE is not set
# CONFIG_SERIAL_ALTERA_UART is not set
CONFIG_SERIAL_IFX6X60=y
CONFIG_SERIAL_ARC=y
CONFIG_SERIAL_ARC_CONSOLE=y
CONFIG_SERIAL_ARC_NR_PORTS=1
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_DEV_BUS is not set
# CONFIG_TTY_PRINTK is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
# CONFIG_HVC_XEN_FRONTEND is not set
CONFIG_VIRTIO_CONSOLE=y
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=y
CONFIG_HW_RANDOM_INTEL=y
CONFIG_HW_RANDOM_AMD=y
# CONFIG_HW_RANDOM_VIA is not set
CONFIG_HW_RANDOM_VIRTIO=y
# CONFIG_HW_RANDOM_TPM is not set
CONFIG_NVRAM=y
CONFIG_R3964=y
CONFIG_APPLICOM=y
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HPET_MMAP_DEFAULT=y
CONFIG_HANGCHECK_TIMER=y
CONFIG_TCG_TPM=y
# CONFIG_TCG_TIS is not set
# CONFIG_TCG_TIS_SPI is not set
# CONFIG_TCG_TIS_I2C_ATMEL is not set
# CONFIG_TCG_TIS_I2C_INFINEON is not set
# CONFIG_TCG_TIS_I2C_NUVOTON is not set
# CONFIG_TCG_NSC is not set
# CONFIG_TCG_ATMEL is not set
# CONFIG_TCG_INFINEON is not set
# CONFIG_TCG_XEN is not set
# CONFIG_TCG_CRB is not set
# CONFIG_TCG_VTPM_PROXY is not set
# CONFIG_TCG_TIS_ST33ZP24_I2C is not set
# CONFIG_TCG_TIS_ST33ZP24_SPI is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_XILLYBUS is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_ACPI_I2C_OPREGION=y
CONFIG_I2C_BOARDINFO=y
# CONFIG_I2C_COMPAT is not set
# CONFIG_I2C_CHARDEV is not set
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=y
CONFIG_I2C_ALGOBIT=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=y
CONFIG_I2C_AMD756_S4882=y
CONFIG_I2C_AMD8111=y
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_ISCH is not set
# CONFIG_I2C_ISMT is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
CONFIG_I2C_SIS5595=y
CONFIG_I2C_SIS630=y
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
CONFIG_I2C_VIAPRO=y

#
# ACPI drivers
#
CONFIG_I2C_SCMI=y

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_CBUS_GPIO is not set
# CONFIG_I2C_DESIGNWARE_PLATFORM is not set
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_EMEV2 is not set
# CONFIG_I2C_GPIO is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_PXA_PCI is not set
# CONFIG_I2C_SIMTEC is not set
CONFIG_I2C_XILINX=y

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
CONFIG_I2C_PARPORT_LIGHT=y
# CONFIG_I2C_ROBOTFUZZ_OSIF is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set
# CONFIG_I2C_VIPERBOARD is not set

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_MLXCPLD is not set
# CONFIG_I2C_SLAVE is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y

#
# SPI Master Controller Drivers
#
CONFIG_SPI_ALTERA=y
# CONFIG_SPI_AXI_SPI_ENGINE is not set
CONFIG_SPI_BITBANG=y
# CONFIG_SPI_CADENCE is not set
CONFIG_SPI_DESIGNWARE=y
# CONFIG_SPI_DW_PCI is not set
# CONFIG_SPI_DW_MMIO is not set
# CONFIG_SPI_GPIO is not set
CONFIG_SPI_OC_TINY=y
# CONFIG_SPI_PXA2XX is not set
# CONFIG_SPI_PXA2XX_PCI is not set
# CONFIG_SPI_ROCKCHIP is not set
CONFIG_SPI_SC18IS602=y
# CONFIG_SPI_XCOMM is not set
CONFIG_SPI_XILINX=y
# CONFIG_SPI_ZYNQMP_GQSPI is not set

#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
CONFIG_SPI_TLE62X0=y
# CONFIG_SPMI is not set
# CONFIG_HSI is not set

#
# PPS support
#
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set
# CONFIG_NTP_PPS is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=y
CONFIG_PPS_CLIENT_GPIO=y

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=y

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
CONFIG_PTP_1588_CLOCK_KVM=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_ACPI=y
CONFIG_DEBUG_GPIO=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_GENERIC=y
CONFIG_GPIO_MAX730X=y

#
# Memory mapped GPIO drivers
#
# CONFIG_GPIO_AMDPT is not set
# CONFIG_GPIO_DWAPB is not set
CONFIG_GPIO_GENERIC_PLATFORM=y
CONFIG_GPIO_ICH=y
# CONFIG_GPIO_LYNXPOINT is not set
# CONFIG_GPIO_MOCKUP is not set
# CONFIG_GPIO_VX855 is not set

#
# Port-mapped I/O GPIO drivers
#
# CONFIG_GPIO_F7188X is not set
# CONFIG_GPIO_IT87 is not set
CONFIG_GPIO_SCH=y
# CONFIG_GPIO_SCH311X is not set

#
# I2C GPIO expanders
#
# CONFIG_GPIO_ADP5588 is not set
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
# CONFIG_GPIO_PCF857X is not set
# CONFIG_GPIO_TPIC2810 is not set

#
# MFD GPIO expanders
#
# CONFIG_GPIO_ARIZONA is not set
CONFIG_GPIO_DA9052=y
# CONFIG_GPIO_JANZ_TTL is not set
# CONFIG_GPIO_PALMAS is not set
CONFIG_GPIO_RC5T583=y
CONFIG_GPIO_TPS6586X=y
# CONFIG_GPIO_TPS65910 is not set
CONFIG_GPIO_TWL6040=y
CONFIG_GPIO_UCB1400=y
# CONFIG_GPIO_WM8350 is not set

#
# PCI GPIO expanders
#
# CONFIG_GPIO_AMD8111 is not set
CONFIG_GPIO_BT8XX=y
CONFIG_GPIO_ML_IOH=y
# CONFIG_GPIO_PCI_IDIO_16 is not set
# CONFIG_GPIO_RDC321X is not set

#
# SPI GPIO expanders
#
CONFIG_GPIO_MAX7301=y
CONFIG_GPIO_MC33880=y
# CONFIG_GPIO_PISOSR is not set

#
# SPI or I2C GPIO expanders
#

#
# USB GPIO expanders
#
# CONFIG_GPIO_VIPERBOARD is not set
CONFIG_W1=y
# CONFIG_W1_CON is not set

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=y
# CONFIG_W1_MASTER_DS2490 is not set
CONFIG_W1_MASTER_DS2482=y
CONFIG_W1_MASTER_DS1WM=y
# CONFIG_W1_MASTER_GPIO is not set

#
# 1-wire Slaves
#
# CONFIG_W1_SLAVE_THERM is not set
CONFIG_W1_SLAVE_SMEM=y
# CONFIG_W1_SLAVE_DS2405 is not set
# CONFIG_W1_SLAVE_DS2408 is not set
CONFIG_W1_SLAVE_DS2413=y
# CONFIG_W1_SLAVE_DS2406 is not set
# CONFIG_W1_SLAVE_DS2423 is not set
CONFIG_W1_SLAVE_DS2431=y
# CONFIG_W1_SLAVE_DS2433 is not set
CONFIG_W1_SLAVE_DS2760=y
CONFIG_W1_SLAVE_DS2780=y
CONFIG_W1_SLAVE_DS2781=y
# CONFIG_W1_SLAVE_DS28E04 is not set
CONFIG_W1_SLAVE_BQ27000=y
# CONFIG_POWER_AVS is not set
CONFIG_POWER_RESET=y
# CONFIG_POWER_RESET_RESTART is not set
CONFIG_POWER_SUPPLY=y
CONFIG_POWER_SUPPLY_DEBUG=y
CONFIG_PDA_POWER=y
# CONFIG_WM8350_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_88PM860X is not set
CONFIG_BATTERY_DS2760=y
CONFIG_BATTERY_DS2780=y
CONFIG_BATTERY_DS2781=y
CONFIG_BATTERY_DS2782=y
# CONFIG_BATTERY_SBS is not set
# CONFIG_CHARGER_SBS is not set
# CONFIG_BATTERY_BQ27XXX is not set
# CONFIG_BATTERY_DA9030 is not set
CONFIG_BATTERY_DA9052=y
CONFIG_BATTERY_MAX17040=y
CONFIG_BATTERY_MAX17042=y
CONFIG_CHARGER_ISP1704=y
# CONFIG_CHARGER_MAX8903 is not set
CONFIG_CHARGER_LP8727=y
# CONFIG_CHARGER_GPIO is not set
# CONFIG_CHARGER_MANAGER is not set
# CONFIG_CHARGER_MAX77693 is not set
CONFIG_CHARGER_BQ2415X=y
# CONFIG_CHARGER_BQ24190 is not set
# CONFIG_CHARGER_BQ24257 is not set
# CONFIG_CHARGER_BQ24735 is not set
# CONFIG_CHARGER_BQ25890 is not set
CONFIG_CHARGER_SMB347=y
# CONFIG_CHARGER_TPS65090 is not set
# CONFIG_BATTERY_GAUGE_LTC2941 is not set
# CONFIG_CHARGER_RT9455 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=y
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7314 is not set
CONFIG_SENSORS_AD7414=y
CONFIG_SENSORS_AD7418=y
CONFIG_SENSORS_ADM1021=y
CONFIG_SENSORS_ADM1025=y
# CONFIG_SENSORS_ADM1026 is not set
CONFIG_SENSORS_ADM1029=y
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7310 is not set
# CONFIG_SENSORS_ADT7410 is not set
CONFIG_SENSORS_ADT7411=y
# CONFIG_SENSORS_ADT7462 is not set
CONFIG_SENSORS_ADT7470=y
CONFIG_SENSORS_ADT7475=y
CONFIG_SENSORS_ASC7621=y
# CONFIG_SENSORS_K8TEMP is not set
CONFIG_SENSORS_K10TEMP=y
# CONFIG_SENSORS_FAM15H_POWER is not set
CONFIG_SENSORS_APPLESMC=y
# CONFIG_SENSORS_ASB100 is not set
CONFIG_SENSORS_ATXP1=y
CONFIG_SENSORS_DS620=y
CONFIG_SENSORS_DS1621=y
# CONFIG_SENSORS_DELL_SMM is not set
CONFIG_SENSORS_DA9052_ADC=y
# CONFIG_SENSORS_I5K_AMB is not set
CONFIG_SENSORS_F71805F=y
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
CONFIG_SENSORS_FSCHMD=y
# CONFIG_SENSORS_FTSTEUTATES is not set
CONFIG_SENSORS_GL518SM=y
# CONFIG_SENSORS_GL520SM is not set
CONFIG_SENSORS_G760A=y
# CONFIG_SENSORS_G762 is not set
# CONFIG_SENSORS_GPIO_FAN is not set
# CONFIG_SENSORS_HIH6130 is not set
# CONFIG_SENSORS_I5500 is not set
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_POWR1220 is not set
CONFIG_SENSORS_LINEAGE=y
# CONFIG_SENSORS_LTC2945 is not set
# CONFIG_SENSORS_LTC2990 is not set
# CONFIG_SENSORS_LTC4151 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4222 is not set
CONFIG_SENSORS_LTC4245=y
# CONFIG_SENSORS_LTC4260 is not set
CONFIG_SENSORS_LTC4261=y
CONFIG_SENSORS_MAX1111=y
CONFIG_SENSORS_MAX16065=y
# CONFIG_SENSORS_MAX1619 is not set
CONFIG_SENSORS_MAX1668=y
# CONFIG_SENSORS_MAX197 is not set
# CONFIG_SENSORS_MAX31722 is not set
CONFIG_SENSORS_MAX6639=y
CONFIG_SENSORS_MAX6642=y
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_MAX6697 is not set
# CONFIG_SENSORS_MAX31790 is not set
CONFIG_SENSORS_MCP3021=y
# CONFIG_SENSORS_TC654 is not set
CONFIG_SENSORS_ADCXX=y
CONFIG_SENSORS_LM63=y
CONFIG_SENSORS_LM70=y
# CONFIG_SENSORS_LM73 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
CONFIG_SENSORS_LM80=y
CONFIG_SENSORS_LM83=y
CONFIG_SENSORS_LM85=y
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
CONFIG_SENSORS_LM92=y
CONFIG_SENSORS_LM93=y
# CONFIG_SENSORS_LM95234 is not set
CONFIG_SENSORS_LM95241=y
# CONFIG_SENSORS_LM95245 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_NTC_THERMISTOR is not set
# CONFIG_SENSORS_NCT6683 is not set
# CONFIG_SENSORS_NCT6775 is not set
# CONFIG_SENSORS_NCT7802 is not set
# CONFIG_SENSORS_NCT7904 is not set
CONFIG_SENSORS_PCF8591=y
CONFIG_PMBUS=y
# CONFIG_SENSORS_PMBUS is not set
# CONFIG_SENSORS_ADM1275 is not set
CONFIG_SENSORS_LM25066=y
CONFIG_SENSORS_LTC2978=y
# CONFIG_SENSORS_LTC2978_REGULATOR is not set
# CONFIG_SENSORS_LTC3815 is not set
# CONFIG_SENSORS_MAX16064 is not set
# CONFIG_SENSORS_MAX20751 is not set
CONFIG_SENSORS_MAX34440=y
CONFIG_SENSORS_MAX8688=y
# CONFIG_SENSORS_TPS40422 is not set
# CONFIG_SENSORS_UCD9000 is not set
CONFIG_SENSORS_UCD9200=y
CONFIG_SENSORS_ZL6100=y
# CONFIG_SENSORS_SHT15 is not set
# CONFIG_SENSORS_SHT21 is not set
# CONFIG_SENSORS_SHT3x is not set
# CONFIG_SENSORS_SHTC1 is not set
CONFIG_SENSORS_SIS5595=y
# CONFIG_SENSORS_DME1737 is not set
CONFIG_SENSORS_EMC1403=y
CONFIG_SENSORS_EMC2103=y
# CONFIG_SENSORS_EMC6W201 is not set
CONFIG_SENSORS_SMSC47M1=y
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
CONFIG_SENSORS_SCH56XX_COMMON=y
# CONFIG_SENSORS_SCH5627 is not set
CONFIG_SENSORS_SCH5636=y
# CONFIG_SENSORS_STTS751 is not set
CONFIG_SENSORS_SMM665=y
# CONFIG_SENSORS_ADC128D818 is not set
CONFIG_SENSORS_ADS1015=y
CONFIG_SENSORS_ADS7828=y
# CONFIG_SENSORS_ADS7871 is not set
CONFIG_SENSORS_AMC6821=y
# CONFIG_SENSORS_INA209 is not set
CONFIG_SENSORS_INA2XX=y
# CONFIG_SENSORS_INA3221 is not set
# CONFIG_SENSORS_TC74 is not set
CONFIG_SENSORS_THMC50=y
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP103 is not set
# CONFIG_SENSORS_TMP108 is not set
CONFIG_SENSORS_TMP401=y
CONFIG_SENSORS_TMP421=y
CONFIG_SENSORS_VIA_CPUTEMP=y
CONFIG_SENSORS_VIA686A=y
CONFIG_SENSORS_VT1211=y
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
CONFIG_SENSORS_W83795=y
# CONFIG_SENSORS_W83795_FANCTRL is not set
# CONFIG_SENSORS_W83L785TS is not set
CONFIG_SENSORS_W83L786NG=y
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_WM8350 is not set
# CONFIG_SENSORS_XGENE is not set

#
# ACPI drivers
#
CONFIG_SENSORS_ACPI_POWER=y
CONFIG_SENSORS_ATK0110=y
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
# CONFIG_THERMAL_WRITABLE_TRIPS is not set
# CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE is not set
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE=y
# CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR is not set
# CONFIG_THERMAL_GOV_FAIR_SHARE is not set
# CONFIG_THERMAL_GOV_STEP_WISE is not set
# CONFIG_THERMAL_GOV_BANG_BANG is not set
CONFIG_THERMAL_GOV_USER_SPACE=y
# CONFIG_THERMAL_GOV_POWER_ALLOCATOR is not set
# CONFIG_THERMAL_EMULATION is not set
CONFIG_INTEL_POWERCLAMP=y
# CONFIG_INTEL_SOC_DTS_THERMAL is not set

#
# ACPI INT340X thermal drivers
#
# CONFIG_INT340X_THERMAL is not set
# CONFIG_INTEL_PCH_THERMAL is not set
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
# CONFIG_WATCHDOG_SYSFS is not set

#
# Watchdog Device Drivers
#
# CONFIG_SOFT_WATCHDOG is not set
# CONFIG_DA9052_WATCHDOG is not set
# CONFIG_WDAT_WDT is not set
# CONFIG_WM8350_WATCHDOG is not set
# CONFIG_XILINX_WATCHDOG is not set
# CONFIG_ZIIRAVE_WATCHDOG is not set
# CONFIG_CADENCE_WATCHDOG is not set
# CONFIG_DW_WATCHDOG is not set
# CONFIG_MAX63XX_WATCHDOG is not set
# CONFIG_RETU_WATCHDOG is not set
CONFIG_ACQUIRE_WDT=y
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
CONFIG_F71808E_WDT=y
# CONFIG_SP5100_TCO is not set
CONFIG_SBC_FITPC2_WATCHDOG=y
# CONFIG_EUROTECH_WDT is not set
CONFIG_IB700_WDT=y
CONFIG_IBMASR=y
CONFIG_WAFER_WDT=y
# CONFIG_I6300ESB_WDT is not set
# CONFIG_IE6XX_WDT is not set
CONFIG_ITCO_WDT=y
# CONFIG_ITCO_VENDOR_SUPPORT is not set
# CONFIG_IT8712F_WDT is not set
# CONFIG_IT87_WDT is not set
# CONFIG_HP_WATCHDOG is not set
CONFIG_SC1200_WDT=y
# CONFIG_PC87413_WDT is not set
# CONFIG_NV_TCO is not set
# CONFIG_60XX_WDT is not set
CONFIG_CPU5_WDT=y
CONFIG_SMSC_SCH311X_WDT=y
CONFIG_SMSC37B787_WDT=y
# CONFIG_VIA_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
CONFIG_MACHZ_WDT=y
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
# CONFIG_NI903X_WDT is not set
# CONFIG_NIC7018_WDT is not set
# CONFIG_MEN_A21_WDT is not set
CONFIG_XEN_WDT=y

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=y
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set

#
# Watchdog Pretimeout Governors
#
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
CONFIG_SSB=y
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
# CONFIG_SSB_SILENT is not set
CONFIG_SSB_DEBUG=y
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_SSB_DRIVER_GPIO=y
CONFIG_BCMA_POSSIBLE=y

#
# Broadcom specific AMBA
#
CONFIG_BCMA=y
CONFIG_BCMA_HOST_PCI_POSSIBLE=y
# CONFIG_BCMA_HOST_PCI is not set
# CONFIG_BCMA_HOST_SOC is not set
CONFIG_BCMA_DRIVER_PCI=y
# CONFIG_BCMA_DRIVER_GMAC_CMN is not set
CONFIG_BCMA_DRIVER_GPIO=y
# CONFIG_BCMA_DEBUG is not set

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
# CONFIG_MFD_AS3711 is not set
# CONFIG_PMIC_ADP5520 is not set
CONFIG_MFD_AAT2870_CORE=y
# CONFIG_MFD_BCM590XX is not set
# CONFIG_MFD_AXP20X_I2C is not set
# CONFIG_MFD_CROS_EC is not set
CONFIG_PMIC_DA903X=y
CONFIG_PMIC_DA9052=y
# CONFIG_MFD_DA9052_SPI is not set
CONFIG_MFD_DA9052_I2C=y
# CONFIG_MFD_DA9055 is not set
# CONFIG_MFD_DA9062 is not set
# CONFIG_MFD_DA9063 is not set
# CONFIG_MFD_DA9150 is not set
# CONFIG_MFD_DLN2 is not set
# CONFIG_MFD_MC13XXX_SPI is not set
# CONFIG_MFD_MC13XXX_I2C is not set
CONFIG_HTC_PASIC3=y
CONFIG_HTC_I2CPLD=y
# CONFIG_MFD_INTEL_QUARK_I2C_GPIO is not set
CONFIG_LPC_ICH=y
CONFIG_LPC_SCH=y
# CONFIG_INTEL_SOC_PMIC is not set
# CONFIG_MFD_INTEL_LPSS_ACPI is not set
# CONFIG_MFD_INTEL_LPSS_PCI is not set
CONFIG_MFD_JANZ_CMODIO=y
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_88PM800 is not set
CONFIG_MFD_88PM805=y
CONFIG_MFD_88PM860X=y
# CONFIG_MFD_MAX14577 is not set
CONFIG_MFD_MAX77693=y
# CONFIG_MFD_MAX77843 is not set
# CONFIG_MFD_MAX8907 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_MENF21BMC is not set
CONFIG_EZX_PCAP=y
CONFIG_MFD_VIPERBOARD=y
CONFIG_MFD_RETU=y
# CONFIG_MFD_PCF50633 is not set
CONFIG_UCB1400_CORE=y
# CONFIG_MFD_RDC321X is not set
CONFIG_MFD_RTSX_PCI=y
# CONFIG_MFD_RT5033 is not set
# CONFIG_MFD_RTSX_USB is not set
CONFIG_MFD_RC5T583=y
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_MFD_SKY81452 is not set
CONFIG_MFD_SMSC=y
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_LP3943 is not set
CONFIG_MFD_LP8788=y
CONFIG_MFD_PALMAS=y
CONFIG_TPS6105X=y
# CONFIG_TPS65010 is not set
CONFIG_TPS6507X=y
# CONFIG_MFD_TPS65086 is not set
CONFIG_MFD_TPS65090=y
# CONFIG_MFD_TPS65217 is not set
# CONFIG_MFD_TI_LP873X is not set
# CONFIG_MFD_TPS65218 is not set
CONFIG_MFD_TPS6586X=y
CONFIG_MFD_TPS65910=y
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_MFD_TPS65912_SPI is not set
CONFIG_MFD_TPS80031=y
# CONFIG_TWL4030_CORE is not set
CONFIG_TWL6040_CORE=y
CONFIG_MFD_WL1273_CORE=y
CONFIG_MFD_LM3533=y
# CONFIG_MFD_TMIO is not set
CONFIG_MFD_VX855=y
CONFIG_MFD_ARIZONA=y
# CONFIG_MFD_ARIZONA_I2C is not set
CONFIG_MFD_ARIZONA_SPI=y
# CONFIG_MFD_CS47L24 is not set
# CONFIG_MFD_WM5102 is not set
CONFIG_MFD_WM5110=y
# CONFIG_MFD_WM8997 is not set
# CONFIG_MFD_WM8998 is not set
CONFIG_MFD_WM8400=y
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM831X_SPI is not set
CONFIG_MFD_WM8350=y
CONFIG_MFD_WM8350_I2C=y
# CONFIG_MFD_WM8994 is not set
CONFIG_REGULATOR=y
# CONFIG_REGULATOR_DEBUG is not set
CONFIG_REGULATOR_FIXED_VOLTAGE=y
# CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set
# CONFIG_REGULATOR_USERSPACE_CONSUMER is not set
CONFIG_REGULATOR_88PM8607=y
# CONFIG_REGULATOR_ACT8865 is not set
# CONFIG_REGULATOR_AD5398 is not set
# CONFIG_REGULATOR_AAT2870 is not set
# CONFIG_REGULATOR_ARIZONA is not set
# CONFIG_REGULATOR_DA903X is not set
# CONFIG_REGULATOR_DA9052 is not set
# CONFIG_REGULATOR_DA9210 is not set
# CONFIG_REGULATOR_DA9211 is not set
# CONFIG_REGULATOR_FAN53555 is not set
# CONFIG_REGULATOR_GPIO is not set
# CONFIG_REGULATOR_ISL9305 is not set
CONFIG_REGULATOR_ISL6271A=y
# CONFIG_REGULATOR_LP3971 is not set
CONFIG_REGULATOR_LP3972=y
# CONFIG_REGULATOR_LP872X is not set
# CONFIG_REGULATOR_LP8755 is not set
# CONFIG_REGULATOR_LP8788 is not set
# CONFIG_REGULATOR_LTC3589 is not set
# CONFIG_REGULATOR_LTC3676 is not set
# CONFIG_REGULATOR_MAX1586 is not set
CONFIG_REGULATOR_MAX8649=y
CONFIG_REGULATOR_MAX8660=y
# CONFIG_REGULATOR_MAX8952 is not set
# CONFIG_REGULATOR_MAX77693 is not set
# CONFIG_REGULATOR_MT6311 is not set
CONFIG_REGULATOR_PALMAS=y
CONFIG_REGULATOR_PCAP=y
# CONFIG_REGULATOR_PFUZE100 is not set
# CONFIG_REGULATOR_PV88060 is not set
# CONFIG_REGULATOR_PV88080 is not set
# CONFIG_REGULATOR_PV88090 is not set
# CONFIG_REGULATOR_PWM is not set
# CONFIG_REGULATOR_RC5T583 is not set
CONFIG_REGULATOR_TPS51632=y
# CONFIG_REGULATOR_TPS6105X is not set
# CONFIG_REGULATOR_TPS62360 is not set
CONFIG_REGULATOR_TPS65023=y
CONFIG_REGULATOR_TPS6507X=y
# CONFIG_REGULATOR_TPS65090 is not set
CONFIG_REGULATOR_TPS6524X=y
# CONFIG_REGULATOR_TPS6586X is not set
CONFIG_REGULATOR_TPS65910=y
# CONFIG_REGULATOR_TPS80031 is not set
CONFIG_REGULATOR_WM8350=y
# CONFIG_REGULATOR_WM8400 is not set
CONFIG_MEDIA_SUPPORT=y

#
# Multimedia core support
#
# CONFIG_MEDIA_CAMERA_SUPPORT is not set
CONFIG_MEDIA_ANALOG_TV_SUPPORT=y
CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y
# CONFIG_MEDIA_RADIO_SUPPORT is not set
# CONFIG_MEDIA_SDR_SUPPORT is not set
CONFIG_MEDIA_RC_SUPPORT=y
# CONFIG_MEDIA_CEC_SUPPORT is not set
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=y
CONFIG_VIDEO_V4L2=y
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_DVB_CORE=y
CONFIG_DVB_NET=y
# CONFIG_TTPCI_EEPROM is not set
CONFIG_DVB_MAX_ADAPTERS=8
# CONFIG_DVB_DYNAMIC_MINORS is not set
# CONFIG_DVB_DEMUX_SECTION_LOSS_LOG is not set

#
# Media drivers
#
CONFIG_RC_CORE=y
# CONFIG_RC_MAP is not set
CONFIG_RC_DECODERS=y
# CONFIG_LIRC is not set
CONFIG_IR_NEC_DECODER=y
# CONFIG_IR_RC5_DECODER is not set
# CONFIG_IR_RC6_DECODER is not set
# CONFIG_IR_JVC_DECODER is not set
# CONFIG_IR_SONY_DECODER is not set
CONFIG_IR_SANYO_DECODER=y
CONFIG_IR_SHARP_DECODER=y
# CONFIG_IR_MCE_KBD_DECODER is not set
CONFIG_IR_XMP_DECODER=y
CONFIG_RC_DEVICES=y
# CONFIG_RC_ATI_REMOTE is not set
CONFIG_IR_ENE=y
# CONFIG_IR_HIX5HD2 is not set
CONFIG_IR_IMON=y
CONFIG_IR_MCEUSB=y
CONFIG_IR_ITE_CIR=y
CONFIG_IR_FINTEK=y
CONFIG_IR_NUVOTON=y
# CONFIG_IR_REDRAT3 is not set
# CONFIG_IR_STREAMZAP is not set
# CONFIG_IR_WINBOND_CIR is not set
# CONFIG_IR_IGORPLUGUSB is not set
# CONFIG_IR_IGUANA is not set
# CONFIG_IR_TTUSBIR is not set
CONFIG_RC_LOOPBACK=y
CONFIG_IR_GPIO_CIR=y
# CONFIG_IR_SERIAL is not set
# CONFIG_MEDIA_USB_SUPPORT is not set
# CONFIG_MEDIA_PCI_SUPPORT is not set
# CONFIG_DVB_PLATFORM_DRIVERS is not set

#
# Supported MMC/SDIO adapters
#
# CONFIG_CYPRESS_FIRMWARE is not set

#
# Media ancillary drivers (tuners, sensors, i2c, spi, frontends)
#
# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
CONFIG_VIDEO_IR_I2C=y

#
# I2C Encoders, decoders, sensors and other helper chips
#

#
# Audio decoders, processors and mixers
#
# CONFIG_VIDEO_TVAUDIO is not set
CONFIG_VIDEO_TDA7432=y
CONFIG_VIDEO_TDA9840=y
# CONFIG_VIDEO_TEA6415C is not set
CONFIG_VIDEO_TEA6420=y
# CONFIG_VIDEO_MSP3400 is not set
# CONFIG_VIDEO_CS3308 is not set
CONFIG_VIDEO_CS5345=y
CONFIG_VIDEO_CS53L32A=y
# CONFIG_VIDEO_TLV320AIC23B is not set
# CONFIG_VIDEO_UDA1342 is not set
# CONFIG_VIDEO_WM8775 is not set
CONFIG_VIDEO_WM8739=y
# CONFIG_VIDEO_VP27SMPX is not set
# CONFIG_VIDEO_SONY_BTF_MPX is not set

#
# RDS decoders
#
CONFIG_VIDEO_SAA6588=y

#
# Video decoders
#
CONFIG_VIDEO_ADV7183=y
CONFIG_VIDEO_BT819=y
CONFIG_VIDEO_BT856=y
# CONFIG_VIDEO_BT866 is not set
CONFIG_VIDEO_KS0127=y
# CONFIG_VIDEO_ML86V7667 is not set
CONFIG_VIDEO_SAA7110=y
CONFIG_VIDEO_SAA711X=y
CONFIG_VIDEO_TVP514X=y
# CONFIG_VIDEO_TVP5150 is not set
# CONFIG_VIDEO_TVP7002 is not set
# CONFIG_VIDEO_TW2804 is not set
# CONFIG_VIDEO_TW9903 is not set
# CONFIG_VIDEO_TW9906 is not set
CONFIG_VIDEO_VPX3220=y

#
# Video and audio decoders
#
# CONFIG_VIDEO_SAA717X is not set
# CONFIG_VIDEO_CX25840 is not set

#
# Video encoders
#
# CONFIG_VIDEO_SAA7127 is not set
CONFIG_VIDEO_SAA7185=y
# CONFIG_VIDEO_ADV7170 is not set
# CONFIG_VIDEO_ADV7175 is not set
# CONFIG_VIDEO_ADV7343 is not set
# CONFIG_VIDEO_ADV7393 is not set
CONFIG_VIDEO_AK881X=y
# CONFIG_VIDEO_THS8200 is not set

#
# Camera sensor devices
#
# CONFIG_VIDEO_MT9M111 is not set

#
# Flash devices
#

#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=y
CONFIG_VIDEO_UPD64083=y

#
# Audio/Video compression chips
#
# CONFIG_VIDEO_SAA6752HS is not set

#
# Miscellaneous helper chips
#
CONFIG_VIDEO_THS7303=y
CONFIG_VIDEO_M52790=y

#
# Sensors used on soc_camera driver
#

#
# SPI helper chips
#
CONFIG_MEDIA_TUNER=y

#
# Customize TV tuners
#
# CONFIG_MEDIA_TUNER_SIMPLE is not set
# CONFIG_MEDIA_TUNER_TDA8290 is not set
CONFIG_MEDIA_TUNER_TDA827X=y
# CONFIG_MEDIA_TUNER_TDA18271 is not set
CONFIG_MEDIA_TUNER_TDA9887=y
# CONFIG_MEDIA_TUNER_TEA5761 is not set
# CONFIG_MEDIA_TUNER_TEA5767 is not set
CONFIG_MEDIA_TUNER_MSI001=y
# CONFIG_MEDIA_TUNER_MT20XX is not set
# CONFIG_MEDIA_TUNER_MT2060 is not set
CONFIG_MEDIA_TUNER_MT2063=y
CONFIG_MEDIA_TUNER_MT2266=y
CONFIG_MEDIA_TUNER_MT2131=y
CONFIG_MEDIA_TUNER_QT1010=y
# CONFIG_MEDIA_TUNER_XC2028 is not set
CONFIG_MEDIA_TUNER_XC5000=y
# CONFIG_MEDIA_TUNER_XC4000 is not set
# CONFIG_MEDIA_TUNER_MXL5005S is not set
CONFIG_MEDIA_TUNER_MXL5007T=y
# CONFIG_MEDIA_TUNER_MC44S803 is not set
# CONFIG_MEDIA_TUNER_MAX2165 is not set
# CONFIG_MEDIA_TUNER_TDA18218 is not set
CONFIG_MEDIA_TUNER_FC0011=y
# CONFIG_MEDIA_TUNER_FC0012 is not set
# CONFIG_MEDIA_TUNER_FC0013 is not set
CONFIG_MEDIA_TUNER_TDA18212=y
# CONFIG_MEDIA_TUNER_E4000 is not set
# CONFIG_MEDIA_TUNER_FC2580 is not set
CONFIG_MEDIA_TUNER_M88RS6000T=y
# CONFIG_MEDIA_TUNER_TUA9001 is not set
CONFIG_MEDIA_TUNER_SI2157=y
CONFIG_MEDIA_TUNER_IT913X=y
CONFIG_MEDIA_TUNER_R820T=y
CONFIG_MEDIA_TUNER_MXL301RF=y
CONFIG_MEDIA_TUNER_QM1D1C0042=y

#
# Customise DVB Frontends
#

#
# Multistandard (satellite) frontends
#
# CONFIG_DVB_STB0899 is not set
CONFIG_DVB_STB6100=y
CONFIG_DVB_STV090x=y
CONFIG_DVB_STV6110x=y

#
# Multistandard (cable + terrestrial) frontends
#
# CONFIG_DVB_DRXK is not set
CONFIG_DVB_TDA18271C2DD=y
CONFIG_DVB_SI2165=y
CONFIG_DVB_MN88472=y
CONFIG_DVB_MN88473=y

#
# DVB-S (satellite) frontends
#
CONFIG_DVB_CX24110=y
CONFIG_DVB_CX24123=y
CONFIG_DVB_MT312=y
# CONFIG_DVB_ZL10036 is not set
# CONFIG_DVB_ZL10039 is not set
# CONFIG_DVB_S5H1420 is not set
# CONFIG_DVB_STV0288 is not set
# CONFIG_DVB_STB6000 is not set
CONFIG_DVB_STV0299=y
CONFIG_DVB_STV6110=y
# CONFIG_DVB_STV0900 is not set
CONFIG_DVB_TDA8083=y
CONFIG_DVB_TDA10086=y
# CONFIG_DVB_TDA8261 is not set
CONFIG_DVB_VES1X93=y
# CONFIG_DVB_TUNER_ITD1000 is not set
CONFIG_DVB_TUNER_CX24113=y
# CONFIG_DVB_TDA826X is not set
CONFIG_DVB_TUA6100=y
# CONFIG_DVB_CX24116 is not set
CONFIG_DVB_CX24117=y
CONFIG_DVB_CX24120=y
CONFIG_DVB_SI21XX=y
CONFIG_DVB_TS2020=y
# CONFIG_DVB_DS3000 is not set
CONFIG_DVB_MB86A16=y
# CONFIG_DVB_TDA10071 is not set

#
# DVB-T (terrestrial) frontends
#
# CONFIG_DVB_SP8870 is not set
CONFIG_DVB_SP887X=y
CONFIG_DVB_CX22700=y
CONFIG_DVB_CX22702=y
CONFIG_DVB_S5H1432=y
CONFIG_DVB_DRXD=y
# CONFIG_DVB_L64781 is not set
# CONFIG_DVB_TDA1004X is not set
CONFIG_DVB_NXT6000=y
CONFIG_DVB_MT352=y
# CONFIG_DVB_ZL10353 is not set
# CONFIG_DVB_DIB3000MB is not set
# CONFIG_DVB_DIB3000MC is not set
CONFIG_DVB_DIB7000M=y
CONFIG_DVB_DIB7000P=y
# CONFIG_DVB_DIB9000 is not set
CONFIG_DVB_TDA10048=y
CONFIG_DVB_AF9013=y
# CONFIG_DVB_EC100 is not set
CONFIG_DVB_STV0367=y
# CONFIG_DVB_CXD2820R is not set
CONFIG_DVB_CXD2841ER=y
# CONFIG_DVB_AS102_FE is not set
CONFIG_DVB_ZD1301_DEMOD=y
# CONFIG_DVB_GP8PSK_FE is not set

#
# DVB-C (cable) frontends
#
# CONFIG_DVB_VES1820 is not set
# CONFIG_DVB_TDA10021 is not set
# CONFIG_DVB_TDA10023 is not set
# CONFIG_DVB_STV0297 is not set

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
# CONFIG_DVB_NXT200X is not set
# CONFIG_DVB_OR51211 is not set
# CONFIG_DVB_OR51132 is not set
CONFIG_DVB_BCM3510=y
CONFIG_DVB_LGDT330X=y
# CONFIG_DVB_LGDT3305 is not set
# CONFIG_DVB_LG2160 is not set
CONFIG_DVB_S5H1409=y
CONFIG_DVB_AU8522=y
CONFIG_DVB_AU8522_DTV=y
# CONFIG_DVB_AU8522_V4L is not set
# CONFIG_DVB_S5H1411 is not set

#
# ISDB-T (terrestrial) frontends
#
# CONFIG_DVB_S921 is not set
CONFIG_DVB_DIB8000=y
# CONFIG_DVB_MB86A20S is not set

#
# ISDB-S (satellite) & ISDB-T (terrestrial) frontends
#
CONFIG_DVB_TC90522=y

#
# Digital terrestrial only tuners/PLL
#
# CONFIG_DVB_PLL is not set
# CONFIG_DVB_TUNER_DIB0070 is not set
# CONFIG_DVB_TUNER_DIB0090 is not set

#
# SEC control devices for DVB-S
#
CONFIG_DVB_DRX39XYJ=y
CONFIG_DVB_LNBH25=y
CONFIG_DVB_LNBP21=y
CONFIG_DVB_LNBP22=y
# CONFIG_DVB_ISL6405 is not set
# CONFIG_DVB_ISL6421 is not set
CONFIG_DVB_ISL6423=y
# CONFIG_DVB_A8293 is not set
CONFIG_DVB_SP2=y
# CONFIG_DVB_LGS8GL5 is not set
# CONFIG_DVB_LGS8GXX is not set
# CONFIG_DVB_ATBM8830 is not set
# CONFIG_DVB_TDA665x is not set
# CONFIG_DVB_IX2505V is not set
# CONFIG_DVB_M88RS2000 is not set
# CONFIG_DVB_AF9033 is not set
CONFIG_DVB_HORUS3A=y
CONFIG_DVB_ASCOT2E=y
CONFIG_DVB_HELENE=y

#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
# CONFIG_VGA_ARB is not set
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=y
# CONFIG_DRM_DP_AUX_CHARDEV is not set
# CONFIG_DRM_DEBUG_MM is not set
# CONFIG_DRM_DEBUG_MM_SELFTEST is not set
CONFIG_DRM_KMS_HELPER=y
CONFIG_DRM_KMS_FB_HELPER=y
CONFIG_DRM_FBDEV_EMULATION=y
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
CONFIG_DRM_TTM=y

#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=y
# CONFIG_DRM_I2C_SIL164 is not set
# CONFIG_DRM_I2C_NXP_TDA998X is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_AMDGPU is not set

#
# ACP (Audio CoProcessor) Configuration
#
# CONFIG_DRM_NOUVEAU is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_VGEM is not set
# CONFIG_DRM_VMWGFX is not set
CONFIG_DRM_GMA500=y
CONFIG_DRM_GMA600=y
CONFIG_DRM_GMA3600=y
# CONFIG_DRM_UDL is not set
CONFIG_DRM_AST=y
CONFIG_DRM_MGAG200=y
# CONFIG_DRM_CIRRUS_QEMU is not set
# CONFIG_DRM_QXL is not set
# CONFIG_DRM_BOCHS is not set
# CONFIG_DRM_VIRTIO_GPU is not set
CONFIG_DRM_BRIDGE=y

#
# Display Interface Bridges
#
# CONFIG_DRM_ANALOGIX_ANX78XX is not set
# CONFIG_DRM_HISI_HIBMC is not set
# CONFIG_DRM_TINYDRM is not set
# CONFIG_DRM_LEGACY is not set
# CONFIG_DRM_LIB_RANDOM is not set

#
# Frame buffer Devices
#
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
CONFIG_FB_DDC=y
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_IMAGEBLIT=y
# CONFIG_FB_PROVIDE_GET_FB_UNMAPPED_AREA is not set
CONFIG_FB_FOREIGN_ENDIAN=y
CONFIG_FB_BOTH_ENDIAN=y
# CONFIG_FB_BIG_ENDIAN is not set
# CONFIG_FB_LITTLE_ENDIAN is not set
CONFIG_FB_SYS_FOPS=y
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_SVGALIB=y
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
CONFIG_FB_PM2=y
# CONFIG_FB_PM2_FIFO_DISCONNECT is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
CONFIG_FB_ASILIANT=y
CONFIG_FB_IMSTT=y
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_EFI is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_OPENCORES is not set
CONFIG_FB_S1D13XXX=y
CONFIG_FB_NVIDIA=y
# CONFIG_FB_NVIDIA_I2C is not set
# CONFIG_FB_NVIDIA_DEBUG is not set
# CONFIG_FB_NVIDIA_BACKLIGHT is not set
CONFIG_FB_RIVA=y
CONFIG_FB_RIVA_I2C=y
CONFIG_FB_RIVA_DEBUG=y
CONFIG_FB_RIVA_BACKLIGHT=y
# CONFIG_FB_I740 is not set
CONFIG_FB_LE80578=y
# CONFIG_FB_CARILLO_RANCH is not set
CONFIG_FB_MATROX=y
CONFIG_FB_MATROX_MILLENIUM=y
# CONFIG_FB_MATROX_MYSTIQUE is not set
# CONFIG_FB_MATROX_G is not set
# CONFIG_FB_MATROX_I2C is not set
# CONFIG_FB_RADEON is not set
CONFIG_FB_ATY128=y
# CONFIG_FB_ATY128_BACKLIGHT is not set
CONFIG_FB_ATY=y
# CONFIG_FB_ATY_CT is not set
# CONFIG_FB_ATY_GX is not set
# CONFIG_FB_ATY_BACKLIGHT is not set
CONFIG_FB_S3=y
CONFIG_FB_S3_DDC=y
# CONFIG_FB_SAVAGE is not set
CONFIG_FB_SIS=y
CONFIG_FB_SIS_300=y
# CONFIG_FB_SIS_315 is not set
# CONFIG_FB_VIA is not set
CONFIG_FB_NEOMAGIC=y
CONFIG_FB_KYRO=y
# CONFIG_FB_3DFX is not set
CONFIG_FB_VOODOO1=y
# CONFIG_FB_VT8623 is not set
CONFIG_FB_TRIDENT=y
CONFIG_FB_ARK=y
CONFIG_FB_PM3=y
CONFIG_FB_CARMINE=y
# CONFIG_FB_CARMINE_DRAM_EVAL is not set
CONFIG_CARMINE_DRAM_CUSTOM=y
CONFIG_FB_SMSCUFX=y
CONFIG_FB_UDL=y
# CONFIG_FB_IBM_GXT4500 is not set
CONFIG_FB_VIRTUAL=y
# CONFIG_XEN_FBDEV_FRONTEND is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
CONFIG_FB_BROADSHEET=y
CONFIG_FB_AUO_K190X=y
# CONFIG_FB_AUO_K1900 is not set
# CONFIG_FB_AUO_K1901 is not set
# CONFIG_FB_SIMPLE is not set
# CONFIG_FB_SM712 is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=y
# CONFIG_LCD_L4F00242T03 is not set
CONFIG_LCD_LMS283GF05=y
# CONFIG_LCD_LTV350QV is not set
# CONFIG_LCD_ILI922X is not set
# CONFIG_LCD_ILI9320 is not set
CONFIG_LCD_TDO24M=y
# CONFIG_LCD_VGG2432A4 is not set
# CONFIG_LCD_PLATFORM is not set
CONFIG_LCD_S6E63M0=y
CONFIG_LCD_LD9040=y
CONFIG_LCD_AMS369FG06=y
# CONFIG_LCD_LMS501KF03 is not set
# CONFIG_LCD_HX8357 is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=y
# CONFIG_BACKLIGHT_LM3533 is not set
# CONFIG_BACKLIGHT_CARILLO_RANCH is not set
CONFIG_BACKLIGHT_PWM=y
CONFIG_BACKLIGHT_DA903X=y
CONFIG_BACKLIGHT_DA9052=y
CONFIG_BACKLIGHT_APPLE=y
# CONFIG_BACKLIGHT_PM8941_WLED is not set
CONFIG_BACKLIGHT_SAHARA=y
CONFIG_BACKLIGHT_ADP8860=y
CONFIG_BACKLIGHT_ADP8870=y
CONFIG_BACKLIGHT_88PM860X=y
CONFIG_BACKLIGHT_AAT2870=y
# CONFIG_BACKLIGHT_LM3630A is not set
CONFIG_BACKLIGHT_LM3639=y
# CONFIG_BACKLIGHT_LP855X is not set
# CONFIG_BACKLIGHT_LP8788 is not set
# CONFIG_BACKLIGHT_GPIO is not set
# CONFIG_BACKLIGHT_LV5207LP is not set
# CONFIG_BACKLIGHT_BD6107 is not set
CONFIG_VGASTATE=y
CONFIG_HDMI=y
# CONFIG_LOGO is not set
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_JACK=y
CONFIG_SND_JACK_INPUT_DEV=y
CONFIG_SND_SEQUENCER=y
# CONFIG_SND_SEQ_DUMMY is not set
CONFIG_SND_OSSEMUL=y
# CONFIG_SND_MIXER_OSS is not set
# CONFIG_SND_PCM_OSS is not set
CONFIG_SND_PCM_TIMER=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=y
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_MAX_CARDS=32
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_PROC_FS=y
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VERBOSE_PRINTK=y
CONFIG_SND_DEBUG=y
CONFIG_SND_DEBUG_VERBOSE=y
# CONFIG_SND_PCM_XRUN_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=y
CONFIG_SND_OPL3_LIB_SEQ=y
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
# CONFIG_SND_EMU10K1_SEQ is not set
CONFIG_SND_MPU401_UART=y
CONFIG_SND_OPL3_LIB=y
CONFIG_SND_VX_LIB=y
CONFIG_SND_AC97_CODEC=y
# CONFIG_SND_DRIVERS is not set
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=y
# CONFIG_SND_ASIHPI is not set
CONFIG_SND_ATIIXP=y
CONFIG_SND_ATIIXP_MODEM=y
CONFIG_SND_AU8810=y
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
CONFIG_SND_BT87X=y
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=y
CONFIG_SND_CMIPCI=y
CONFIG_SND_OXYGEN_LIB=y
CONFIG_SND_OXYGEN=y
# CONFIG_SND_CS4281 is not set
CONFIG_SND_CS46XX=y
CONFIG_SND_CS46XX_NEW_DSP=y
# CONFIG_SND_CTXFI is not set
# CONFIG_SND_DARLA20 is not set
CONFIG_SND_GINA20=y
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
CONFIG_SND_LAYLA24=y
CONFIG_SND_MONA=y
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
CONFIG_SND_INDIGODJ=y
CONFIG_SND_INDIGOIOX=y
# CONFIG_SND_INDIGODJX is not set
CONFIG_SND_ENS1370=y
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDSP is not set
CONFIG_SND_HDSPM=y
CONFIG_SND_ICE1724=y
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
CONFIG_SND_KORG1212=y
CONFIG_SND_LOLA=y
CONFIG_SND_LX6464ES=y
CONFIG_SND_MIXART=y
CONFIG_SND_NM256=y
CONFIG_SND_PCXHR=y
# CONFIG_SND_RIPTIDE is not set
CONFIG_SND_RME32=y
CONFIG_SND_RME96=y
# CONFIG_SND_RME9652 is not set
CONFIG_SND_VIA82XX=y
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
CONFIG_SND_VX222=y
# CONFIG_SND_YMFPCI is not set

#
# HD-Audio
#
# CONFIG_SND_HDA_INTEL is not set
CONFIG_SND_HDA_PREALLOC_SIZE=64
# CONFIG_SND_SPI is not set
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=y
CONFIG_SND_USB_UA101=y
# CONFIG_SND_USB_USX2Y is not set
CONFIG_SND_USB_CAIAQ=y
# CONFIG_SND_USB_CAIAQ_INPUT is not set
CONFIG_SND_USB_US122L=y
# CONFIG_SND_USB_6FIRE is not set
# CONFIG_SND_USB_HIFACE is not set
# CONFIG_SND_BCD2000 is not set
# CONFIG_SND_USB_POD is not set
# CONFIG_SND_USB_PODHD is not set
# CONFIG_SND_USB_TONEPORT is not set
# CONFIG_SND_USB_VARIAX is not set
CONFIG_SND_SOC=y
# CONFIG_SND_SOC_AMD_ACP is not set
# CONFIG_SND_ATMEL_SOC is not set
# CONFIG_SND_DESIGNWARE_I2S is not set

#
# SoC Audio for Freescale CPUs
#

#
# Common SoC Audio options for Freescale CPUs:
#
# CONFIG_SND_SOC_FSL_ASRC is not set
# CONFIG_SND_SOC_FSL_SAI is not set
# CONFIG_SND_SOC_FSL_SSI is not set
# CONFIG_SND_SOC_FSL_SPDIF is not set
# CONFIG_SND_SOC_FSL_ESAI is not set
# CONFIG_SND_SOC_IMX_AUDMUX is not set
# CONFIG_SND_SOC_IMG is not set
# CONFIG_SND_SOC_INTEL_BXT_DA7219_MAX98357A_MACH is not set
# CONFIG_SND_SOC_INTEL_BXT_RT298_MACH is not set
# CONFIG_SND_SOC_INTEL_BYTCR_RT5640_MACH is not set
# CONFIG_SND_SOC_INTEL_BYTCR_RT5651_MACH is not set
# CONFIG_SND_SOC_INTEL_SKL_RT286_MACH is not set
# CONFIG_SND_SOC_XTFPGA_I2S is not set
CONFIG_SND_SOC_I2C_AND_SPI=y

#
# CODEC drivers
#
# CONFIG_SND_SOC_AC97_CODEC is not set
# CONFIG_SND_SOC_ADAU1701 is not set
# CONFIG_SND_SOC_ADAU7002 is not set
# CONFIG_SND_SOC_AK4104 is not set
# CONFIG_SND_SOC_AK4554 is not set
# CONFIG_SND_SOC_AK4613 is not set
# CONFIG_SND_SOC_AK4642 is not set
# CONFIG_SND_SOC_AK5386 is not set
# CONFIG_SND_SOC_ALC5623 is not set
# CONFIG_SND_SOC_BT_SCO is not set
# CONFIG_SND_SOC_CS35L32 is not set
# CONFIG_SND_SOC_CS35L33 is not set
# CONFIG_SND_SOC_CS35L34 is not set
# CONFIG_SND_SOC_CS42L42 is not set
# CONFIG_SND_SOC_CS42L51_I2C is not set
# CONFIG_SND_SOC_CS42L52 is not set
# CONFIG_SND_SOC_CS42L56 is not set
# CONFIG_SND_SOC_CS42L73 is not set
# CONFIG_SND_SOC_CS4265 is not set
# CONFIG_SND_SOC_CS4270 is not set
# CONFIG_SND_SOC_CS4271_I2C is not set
# CONFIG_SND_SOC_CS4271_SPI is not set
# CONFIG_SND_SOC_CS42XX8_I2C is not set
# CONFIG_SND_SOC_CS4349 is not set
# CONFIG_SND_SOC_CS53L30 is not set
# CONFIG_SND_SOC_ES8328_I2C is not set
# CONFIG_SND_SOC_ES8328_SPI is not set
# CONFIG_SND_SOC_GTM601 is not set
# CONFIG_SND_SOC_INNO_RK3036 is not set
# CONFIG_SND_SOC_MAX98504 is not set
# CONFIG_SND_SOC_MAX9860 is not set
# CONFIG_SND_SOC_MSM8916_WCD_DIGITAL is not set
# CONFIG_SND_SOC_PCM1681 is not set
# CONFIG_SND_SOC_PCM179X_I2C is not set
# CONFIG_SND_SOC_PCM179X_SPI is not set
# CONFIG_SND_SOC_PCM3168A_I2C is not set
# CONFIG_SND_SOC_PCM3168A_SPI is not set
# CONFIG_SND_SOC_PCM512x_I2C is not set
# CONFIG_SND_SOC_PCM512x_SPI is not set
# CONFIG_SND_SOC_RT5616 is not set
# CONFIG_SND_SOC_RT5631 is not set
# CONFIG_SND_SOC_RT5677_SPI is not set
# CONFIG_SND_SOC_SGTL5000 is not set
# CONFIG_SND_SOC_SIRF_AUDIO_CODEC is not set
# CONFIG_SND_SOC_SPDIF is not set
# CONFIG_SND_SOC_SSM2602_SPI is not set
# CONFIG_SND_SOC_SSM2602_I2C is not set
# CONFIG_SND_SOC_SSM4567 is not set
# CONFIG_SND_SOC_STA32X is not set
# CONFIG_SND_SOC_STA350 is not set
# CONFIG_SND_SOC_STI_SAS is not set
# CONFIG_SND_SOC_TAS2552 is not set
# CONFIG_SND_SOC_TAS5086 is not set
# CONFIG_SND_SOC_TAS571X is not set
# CONFIG_SND_SOC_TAS5720 is not set
# CONFIG_SND_SOC_TFA9879 is not set
# CONFIG_SND_SOC_TLV320AIC23_I2C is not set
# CONFIG_SND_SOC_TLV320AIC23_SPI is not set
# CONFIG_SND_SOC_TLV320AIC31XX is not set
# CONFIG_SND_SOC_TLV320AIC3X is not set
# CONFIG_SND_SOC_TS3A227E is not set
# CONFIG_SND_SOC_WM8510 is not set
# CONFIG_SND_SOC_WM8523 is not set
# CONFIG_SND_SOC_WM8580 is not set
# CONFIG_SND_SOC_WM8711 is not set
# CONFIG_SND_SOC_WM8728 is not set
# CONFIG_SND_SOC_WM8731 is not set
# CONFIG_SND_SOC_WM8737 is not set
# CONFIG_SND_SOC_WM8741 is not set
# CONFIG_SND_SOC_WM8750 is not set
# CONFIG_SND_SOC_WM8753 is not set
# CONFIG_SND_SOC_WM8770 is not set
# CONFIG_SND_SOC_WM8776 is not set
# CONFIG_SND_SOC_WM8804_I2C is not set
# CONFIG_SND_SOC_WM8804_SPI is not set
# CONFIG_SND_SOC_WM8903 is not set
# CONFIG_SND_SOC_WM8960 is not set
# CONFIG_SND_SOC_WM8962 is not set
# CONFIG_SND_SOC_WM8974 is not set
# CONFIG_SND_SOC_WM8978 is not set
# CONFIG_SND_SOC_WM8985 is not set
# CONFIG_SND_SOC_NAU8540 is not set
# CONFIG_SND_SOC_NAU8810 is not set
# CONFIG_SND_SOC_TPA6130A2 is not set
CONFIG_SND_SIMPLE_CARD_UTILS=y
CONFIG_SND_SIMPLE_CARD=y
CONFIG_SND_X86=y
CONFIG_SOUND_PRIME=y
CONFIG_AC97_BUS=y

#
# HID support
#
# CONFIG_HID is not set

#
# USB HID support
#
# CONFIG_USB_HID is not set
# CONFIG_HID_PID is not set

#
# USB HID Boot Protocol drivers
#
CONFIG_USB_KBD=y
# CONFIG_USB_MOUSE is not set

#
# I2C HID support
#
# CONFIG_I2C_HID is not set

#
# Intel ISH HID support
#
# CONFIG_INTEL_ISH_HID is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
CONFIG_USB_DEFAULT_PERSIST=y
CONFIG_USB_DYNAMIC_MINORS=y
# CONFIG_USB_OTG is not set
# CONFIG_USB_OTG_WHITELIST is not set
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
# CONFIG_USB_LEDS_TRIGGER_USBPORT is not set
# CONFIG_USB_MON is not set
CONFIG_USB_WUSB_CBAF=y
CONFIG_USB_WUSB_CBAF_DEBUG=y

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PCI=y
# CONFIG_USB_XHCI_PLATFORM is not set
# CONFIG_USB_EHCI_HCD is not set
CONFIG_USB_OXU210HP_HCD=y
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
# CONFIG_USB_FOTG210_HCD is not set
# CONFIG_USB_MAX3421_HCD is not set
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PCI=y
# CONFIG_USB_OHCI_HCD_SSB is not set
CONFIG_USB_OHCI_HCD_PLATFORM=y
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
CONFIG_USB_R8A66597_HCD=y
CONFIG_USB_HCD_BCMA=y
# CONFIG_USB_HCD_SSB is not set
# CONFIG_USB_HCD_TEST_MODE is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=y
CONFIG_USB_PRINTER=y
CONFIG_USB_WDM=y
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=y
CONFIG_USB_STORAGE_DEBUG=y
CONFIG_USB_STORAGE_REALTEK=y
CONFIG_REALTEK_AUTOPM=y
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_STORAGE_ALAUDA=y
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_STORAGE_ENE_UB6250 is not set
# CONFIG_USB_UAS is not set

#
# USB Imaging devices
#
CONFIG_USB_MDC800=y
# CONFIG_USB_MICROTEK is not set
# CONFIG_USBIP_CORE is not set
# CONFIG_USB_MUSB_HDRC is not set
# CONFIG_USB_DWC3 is not set
# CONFIG_USB_DWC2 is not set
CONFIG_USB_CHIPIDEA=y
CONFIG_USB_CHIPIDEA_PCI=y
# CONFIG_USB_CHIPIDEA_UDC is not set
# CONFIG_USB_ISP1760 is not set

#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=y
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
CONFIG_USB_SEVSEG=y
CONFIG_USB_RIO500=y
# CONFIG_USB_LEGOTOWER is not set
CONFIG_USB_LCD=y
CONFIG_USB_CYPRESS_CY7C63=y
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
CONFIG_USB_IOWARRIOR=y
CONFIG_USB_TEST=y
# CONFIG_USB_EHSET_TEST_FIXTURE is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_YUREX is not set
# CONFIG_USB_EZUSB_FX2 is not set
# CONFIG_USB_HUB_USB251XB is not set
# CONFIG_USB_HSIC_USB3503 is not set
# CONFIG_USB_HSIC_USB4604 is not set
# CONFIG_USB_LINK_LAYER_TEST is not set
# CONFIG_USB_CHAOSKEY is not set
# CONFIG_UCSI is not set

#
# USB Physical Layer drivers
#
CONFIG_USB_PHY=y
CONFIG_NOP_USB_XCEIV=y
CONFIG_USB_GPIO_VBUS=y
# CONFIG_TAHVO_USB is not set
# CONFIG_USB_ISP1301 is not set
CONFIG_USB_GADGET=y
# CONFIG_USB_GADGET_DEBUG is not set
# CONFIG_USB_GADGET_DEBUG_FILES is not set
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_VBUS_DRAW=2
CONFIG_USB_GADGET_STORAGE_NUM_BUFFERS=2

#
# USB Peripheral Controller
#
# CONFIG_USB_FOTG210_UDC is not set
# CONFIG_USB_GR_UDC is not set
# CONFIG_USB_R8A66597 is not set
# CONFIG_USB_PXA27X is not set
# CONFIG_USB_MV_UDC is not set
# CONFIG_USB_MV_U3D is not set
CONFIG_USB_M66592=y
# CONFIG_USB_BDC_UDC is not set
# CONFIG_USB_AMD5536UDC is not set
# CONFIG_USB_NET2272 is not set
# CONFIG_USB_NET2280 is not set
# CONFIG_USB_GOKU is not set
# CONFIG_USB_EG20T is not set
CONFIG_USB_DUMMY_HCD=y
CONFIG_USB_LIBCOMPOSITE=y
CONFIG_USB_F_ACM=y
CONFIG_USB_U_SERIAL=y
CONFIG_USB_F_MASS_STORAGE=y
# CONFIG_USB_CONFIGFS is not set
# CONFIG_USB_ZERO is not set
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_ETH is not set
# CONFIG_USB_G_NCM is not set
# CONFIG_USB_GADGETFS is not set
# CONFIG_USB_FUNCTIONFS is not set
# CONFIG_USB_MASS_STORAGE is not set
# CONFIG_USB_G_SERIAL is not set
# CONFIG_USB_MIDI_GADGET is not set
# CONFIG_USB_G_PRINTER is not set
# CONFIG_USB_CDC_COMPOSITE is not set
CONFIG_USB_G_ACM_MS=y
# CONFIG_USB_G_MULTI is not set
# CONFIG_USB_G_HID is not set
# CONFIG_USB_G_DBGP is not set
# CONFIG_USB_G_WEBCAM is not set
# CONFIG_USB_LED_TRIG is not set
# CONFIG_USB_ULPI_BUS is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
# CONFIG_LEDS_CLASS_FLASH is not set
# CONFIG_LEDS_BRIGHTNESS_HW_CHANGED is not set

#
# LED drivers
#
# CONFIG_LEDS_88PM860X is not set
# CONFIG_LEDS_LM3530 is not set
# CONFIG_LEDS_LM3533 is not set
CONFIG_LEDS_LM3642=y
CONFIG_LEDS_PCA9532=y
# CONFIG_LEDS_PCA9532_GPIO is not set
CONFIG_LEDS_GPIO=y
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_LP3952 is not set
# CONFIG_LEDS_LP5521 is not set
# CONFIG_LEDS_LP5523 is not set
# CONFIG_LEDS_LP5562 is not set
# CONFIG_LEDS_LP8501 is not set
CONFIG_LEDS_LP8788=y
# CONFIG_LEDS_LP8860 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_PCA963X is not set
CONFIG_LEDS_WM8350=y
# CONFIG_LEDS_DA903X is not set
# CONFIG_LEDS_DA9052 is not set
CONFIG_LEDS_DAC124S085=y
# CONFIG_LEDS_PWM is not set
CONFIG_LEDS_REGULATOR=y
CONFIG_LEDS_BD2802=y
# CONFIG_LEDS_INTEL_SS4200 is not set
CONFIG_LEDS_LT3593=y
# CONFIG_LEDS_TCA6507 is not set
# CONFIG_LEDS_TLC591XX is not set
# CONFIG_LEDS_LM355x is not set

#
# LED driver for blink(1) USB RGB LED is under Special HID drivers (HID_THINGM)
#
CONFIG_LEDS_BLINKM=y
# CONFIG_LEDS_MLXCPLD is not set
# CONFIG_LEDS_USER is not set
# CONFIG_LEDS_NIC78BX is not set

#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=y
# CONFIG_LEDS_TRIGGER_ONESHOT is not set
# CONFIG_LEDS_TRIGGER_DISK is not set
# CONFIG_LEDS_TRIGGER_MTD is not set
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
# CONFIG_LEDS_TRIGGER_CPU is not set
# CONFIG_LEDS_TRIGGER_GPIO is not set
# CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set

#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_LEDS_TRIGGER_TRANSIENT is not set
# CONFIG_LEDS_TRIGGER_CAMERA is not set
# CONFIG_LEDS_TRIGGER_PANIC is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_EDAC=y
# CONFIG_EDAC_LEGACY_SYSFS is not set
CONFIG_EDAC_DEBUG=y
# CONFIG_EDAC_DECODE_MCE is not set
# CONFIG_EDAC_MM_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_MC146818_LIB=y
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set

#
# DMABUF options
#
CONFIG_SYNC_FILE=y
# CONFIG_SW_SYNC is not set
CONFIG_AUXDISPLAY=y
# CONFIG_IMG_ASCII_LCD is not set
CONFIG_UIO=y
# CONFIG_UIO_CIF is not set
# CONFIG_UIO_PDRV_GENIRQ is not set
CONFIG_UIO_DMEM_GENIRQ=y
# CONFIG_UIO_AEC is not set
CONFIG_UIO_SERCOS3=y
# CONFIG_UIO_PCI_GENERIC is not set
CONFIG_UIO_NETX=y
# CONFIG_UIO_PRUSS is not set
# CONFIG_UIO_MF624 is not set
CONFIG_VIRT_DRIVERS=y
CONFIG_VIRTIO=y

#
# Virtio drivers
#
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=y
# CONFIG_VIRTIO_INPUT is not set
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
# CONFIG_HYPERV is not set

#
# Xen driver support
#
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
# CONFIG_XENFS is not set
# CONFIG_XEN_SYS_HYPERVISOR is not set
CONFIG_XEN_XENBUS_FRONTEND=y
# CONFIG_XEN_GNTDEV is not set
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
# CONFIG_XEN_PCIDEV_BACKEND is not set
CONFIG_XEN_PRIVCMD=y
# CONFIG_XEN_ACPI_PROCESSOR is not set
# CONFIG_XEN_MCE_LOG is not set
CONFIG_XEN_HAVE_PVMMU=y
CONFIG_XEN_EFI=y
CONFIG_XEN_AUTO_XLATE=y
CONFIG_XEN_ACPI=y
CONFIG_XEN_HAVE_VPMU=y
CONFIG_STAGING=y
# CONFIG_RTS5208 is not set
# CONFIG_FB_SM750 is not set
# CONFIG_FB_XGI is not set

#
# Speakup console speech
#
# CONFIG_STAGING_MEDIA is not set

#
# Android
#
CONFIG_ASHMEM=y
# CONFIG_ANDROID_LOW_MEMORY_KILLER is not set
# CONFIG_ION is not set
# CONFIG_DGNC is not set
# CONFIG_GS_FPGABOOT is not set
# CONFIG_CRYPTO_SKEIN is not set
# CONFIG_UNISYSSPAR is not set
# CONFIG_FB_TFT is not set
# CONFIG_MOST is not set
# CONFIG_GREYBUS is not set
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_ACER_WMI=y
# CONFIG_ACERHDF is not set
# CONFIG_ALIENWARE_WMI is not set
# CONFIG_ASUS_LAPTOP is not set
# CONFIG_DELL_LAPTOP is not set
# CONFIG_DELL_WMI is not set
# CONFIG_DELL_WMI_AIO is not set
# CONFIG_DELL_SMO8800 is not set
# CONFIG_FUJITSU_LAPTOP is not set
CONFIG_FUJITSU_TABLET=y
CONFIG_HP_ACCEL=y
# CONFIG_HP_WIRELESS is not set
CONFIG_HP_WMI=y
CONFIG_PANASONIC_LAPTOP=y
# CONFIG_SURFACE3_WMI is not set
CONFIG_THINKPAD_ACPI=y
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
# CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set
# CONFIG_THINKPAD_ACPI_DEBUG is not set
CONFIG_THINKPAD_ACPI_UNSAFE_LEDS=y
# CONFIG_THINKPAD_ACPI_VIDEO is not set
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
# CONFIG_SENSORS_HDAPS is not set
CONFIG_INTEL_MENLOW=y
# CONFIG_EEEPC_LAPTOP is not set
# CONFIG_ASUS_WMI is not set
# CONFIG_ASUS_WIRELESS is not set
CONFIG_ACPI_WMI=y
CONFIG_MSI_WMI=y
CONFIG_TOPSTAR_LAPTOP=y
# CONFIG_TOSHIBA_BT_RFKILL is not set
# CONFIG_TOSHIBA_HAPS is not set
# CONFIG_TOSHIBA_WMI is not set
CONFIG_ACPI_CMPC=y
# CONFIG_INTEL_HID_EVENT is not set
# CONFIG_INTEL_VBTN is not set
# CONFIG_INTEL_IPS is not set
# CONFIG_INTEL_PMC_CORE is not set
# CONFIG_IBM_RTL is not set
# CONFIG_SAMSUNG_LAPTOP is not set
CONFIG_MXM_WMI=y
CONFIG_SAMSUNG_Q10=y
CONFIG_APPLE_GMUX=y
# CONFIG_INTEL_RST is not set
# CONFIG_INTEL_SMARTCONNECT is not set
# CONFIG_PVPANIC is not set
# CONFIG_INTEL_PMC_IPC is not set
# CONFIG_SURFACE_PRO3_BUTTON is not set
# CONFIG_SURFACE_3_BUTTON is not set
# CONFIG_INTEL_PUNIT_IPC is not set
# CONFIG_MLX_PLATFORM is not set
# CONFIG_MLX_CPLD_PLATFORM is not set
# CONFIG_INTEL_TURBO_MAX_3 is not set
# CONFIG_SILEAD_DMI is not set
CONFIG_PMC_ATOM=y
# CONFIG_CHROME_PLATFORMS is not set
CONFIG_CLKDEV_LOOKUP=y
CONFIG_HAVE_CLK_PREPARE=y
CONFIG_COMMON_CLK=y

#
# Common Clock Framework
#
# CONFIG_COMMON_CLK_SI5351 is not set
# CONFIG_COMMON_CLK_CDCE706 is not set
# CONFIG_COMMON_CLK_CS2000_CP is not set
# CONFIG_CLK_TWL6040 is not set
# CONFIG_COMMON_CLK_NXP is not set
# CONFIG_COMMON_CLK_PALMAS is not set
# CONFIG_COMMON_CLK_PWM is not set
# CONFIG_COMMON_CLK_PXA is not set
# CONFIG_COMMON_CLK_PIC32 is not set

#
# Hardware Spinlock drivers
#

#
# Clock Source drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# CONFIG_ATMEL_PIT is not set
# CONFIG_SH_TIMER_CMT is not set
# CONFIG_SH_TIMER_MTU2 is not set
# CONFIG_SH_TIMER_TMU is not set
# CONFIG_EM_TIMER_STI is not set
CONFIG_MAILBOX=y
CONFIG_PCC=y
# CONFIG_ALTERA_MBOX is not set
# CONFIG_IOMMU_SUPPORT is not set

#
# Remoteproc drivers
#
CONFIG_REMOTEPROC=y

#
# Rpmsg drivers
#

#
# SOC (System On Chip) specific Drivers
#

#
# Broadcom SoC drivers
#
# CONFIG_SUNXI_SRAM is not set
# CONFIG_SOC_TI is not set
# CONFIG_SOC_ZTE is not set
CONFIG_PM_DEVFREQ=y

#
# DEVFREQ Governors
#
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
# CONFIG_DEVFREQ_GOV_PERFORMANCE is not set
# CONFIG_DEVFREQ_GOV_POWERSAVE is not set
# CONFIG_DEVFREQ_GOV_USERSPACE is not set
# CONFIG_DEVFREQ_GOV_PASSIVE is not set

#
# DEVFREQ Drivers
#
# CONFIG_PM_DEVFREQ_EVENT is not set
CONFIG_EXTCON=y

#
# Extcon Device Drivers
#
# CONFIG_EXTCON_ARIZONA is not set
# CONFIG_EXTCON_GPIO is not set
# CONFIG_EXTCON_INTEL_INT3496 is not set
# CONFIG_EXTCON_MAX3355 is not set
# CONFIG_EXTCON_MAX77693 is not set
# CONFIG_EXTCON_PALMAS is not set
# CONFIG_EXTCON_QCOM_SPMI_MISC is not set
# CONFIG_EXTCON_RT8973A is not set
# CONFIG_EXTCON_SM5502 is not set
# CONFIG_EXTCON_USB_GPIO is not set
CONFIG_MEMORY=y
# CONFIG_IIO is not set
# CONFIG_NTB is not set
# CONFIG_VME_BUS is not set
CONFIG_PWM=y
CONFIG_PWM_SYSFS=y
# CONFIG_PWM_LPSS_PCI is not set
# CONFIG_PWM_LPSS_PLATFORM is not set
# CONFIG_PWM_PCA9685 is not set
CONFIG_ARM_GIC_MAX_NR=1
# CONFIG_IPACK_BUS is not set
CONFIG_RESET_CONTROLLER=y
# CONFIG_RESET_ATH79 is not set
# CONFIG_RESET_BERLIN is not set
# CONFIG_RESET_LPC18XX is not set
# CONFIG_RESET_MESON is not set
# CONFIG_RESET_PISTACHIO is not set
# CONFIG_RESET_SOCFPGA is not set
# CONFIG_RESET_STM32 is not set
# CONFIG_RESET_SUNXI is not set
# CONFIG_TI_SYSCON_RESET is not set
# CONFIG_RESET_ZYNQ is not set
# CONFIG_RESET_TEGRA_BPMP is not set
# CONFIG_FMC is not set

#
# PHY Subsystem
#
# CONFIG_GENERIC_PHY is not set
# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
# CONFIG_BCM_KONA_USB2_PHY is not set
# CONFIG_POWERCAP is not set
# CONFIG_MCB is not set

#
# Performance monitor support
#
CONFIG_RAS=y
# CONFIG_MCE_AMD_INJ is not set
# CONFIG_THUNDERBOLT is not set

#
# Android
#
CONFIG_ANDROID=y
# CONFIG_ANDROID_BINDER_IPC is not set
# CONFIG_LIBNVDIMM is not set
# CONFIG_DEV_DAX is not set
# CONFIG_NVMEM is not set
# CONFIG_STM is not set
# CONFIG_INTEL_TH is not set

#
# FPGA Configuration Support
#
# CONFIG_FPGA is not set

#
# FSI support
#
# CONFIG_FSI is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DELL_RBU=y
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_DMI_SYSFS is not set
CONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=y
CONFIG_ISCSI_IBFT_FIND=y
# CONFIG_ISCSI_IBFT is not set
# CONFIG_FW_CFG_SYSFS is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# EFI (Extensible Firmware Interface) Support
#
# CONFIG_EFI_VARS is not set
CONFIG_EFI_ESRT=y
# CONFIG_EFI_FAKE_MEMMAP is not set
CONFIG_EFI_RUNTIME_WRAPPERS=y
# CONFIG_EFI_CAPSULE_LOADER is not set
# CONFIG_EFI_TEST is not set
# CONFIG_APPLE_PROPERTIES is not set
# CONFIG_EFI_DEV_PATH_PARSER is not set

#
# Tegra firmware driver
#

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_FS_IOMAP=y
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_EXT4_FS=y
# CONFIG_EXT4_FS_POSIX_ACL is not set
# CONFIG_EXT4_FS_SECURITY is not set
# CONFIG_EXT4_ENCRYPTION is not set
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
# CONFIG_REISERFS_FS_SECURITY is not set
# CONFIG_JFS_FS is not set
CONFIG_XFS_FS=y
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_XFS_WARN is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_BTRFS_FS=y
# CONFIG_BTRFS_FS_POSIX_ACL is not set
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
# CONFIG_BTRFS_ASSERT is not set
CONFIG_NILFS2_FS=y
# CONFIG_F2FS_FS is not set
# CONFIG_FS_DAX is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
# CONFIG_EXPORTFS_BLOCK_OPS is not set
CONFIG_FILE_LOCKING=y
CONFIG_MANDATORY_FILE_LOCKING=y
# CONFIG_FS_ENCRYPTION is not set
CONFIG_FSNOTIFY=y
# CONFIG_DNOTIFY is not set
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
# CONFIG_QUOTA is not set
# CONFIG_QUOTACTL is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
# CONFIG_OVERLAY_FS is not set

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
# CONFIG_ZISOFS is not set
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
# CONFIG_MSDOS_FS is not set
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_FAT_DEFAULT_UTF8 is not set
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
# CONFIG_PROC_CHILDREN is not set
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
CONFIG_CONFIGFS_FS=y
CONFIG_EFIVAR_FS=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_ROOT_NFS is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
# CONFIG_NFSD is not set
CONFIG_GRACE_PERIOD=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
# CONFIG_SUNRPC_DEBUG is not set
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=y
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
# CONFIG_CIFS_UPCALL is not set
# CONFIG_CIFS_XATTR is not set
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_SMB2 is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_9P_FS=y
# CONFIG_9P_FS_POSIX_ACL is not set
# CONFIG_9P_FS_SECURITY is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
# CONFIG_NLS_CODEPAGE_437 is not set
CONFIG_NLS_CODEPAGE_737=y
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
CONFIG_NLS_CODEPAGE_862=y
CONFIG_NLS_CODEPAGE_863=y
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
CONFIG_NLS_CODEPAGE_936=y
CONFIG_NLS_CODEPAGE_950=y
# CONFIG_NLS_CODEPAGE_932 is not set
CONFIG_NLS_CODEPAGE_949=y
# CONFIG_NLS_CODEPAGE_874 is not set
CONFIG_NLS_ISO8859_8=y
# CONFIG_NLS_CODEPAGE_1250 is not set
CONFIG_NLS_CODEPAGE_1251=y
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=y
# CONFIG_NLS_ISO8859_3 is not set
CONFIG_NLS_ISO8859_4=y
# CONFIG_NLS_ISO8859_5 is not set
CONFIG_NLS_ISO8859_6=y
CONFIG_NLS_ISO8859_7=y
# CONFIG_NLS_ISO8859_9 is not set
CONFIG_NLS_ISO8859_13=y
CONFIG_NLS_ISO8859_14=y
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_KOI8_R=y
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_MAC_ROMAN is not set
CONFIG_NLS_MAC_CELTIC=y
# CONFIG_NLS_MAC_CENTEURO is not set
# CONFIG_NLS_MAC_CROATIAN is not set
CONFIG_NLS_MAC_CYRILLIC=y
CONFIG_NLS_MAC_GAELIC=y
# CONFIG_NLS_MAC_GREEK is not set
CONFIG_NLS_MAC_ICELAND=y
# CONFIG_NLS_MAC_INUIT is not set
# CONFIG_NLS_MAC_ROMANIAN is not set
# CONFIG_NLS_MAC_TURKISH is not set
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
CONFIG_BOOT_PRINTK_DELAY=y
# CONFIG_DYNAMIC_DEBUG is not set

#
# Compile-time checks and compiler options
#
# CONFIG_DEBUG_INFO is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_PAGE_OWNER is not set
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_STACK_VALIDATION is not set
CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_MAGIC_SYSRQ_SERIAL=y
CONFIG_DEBUG_KERNEL=y

#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_RODATA_TEST is not set
CONFIG_DEBUG_OBJECTS=y
CONFIG_DEBUG_OBJECTS_SELFTEST=y
CONFIG_DEBUG_OBJECTS_FREE=y
# CONFIG_DEBUG_OBJECTS_TIMERS is not set
CONFIG_DEBUG_OBJECTS_WORK=y
# CONFIG_DEBUG_OBJECTS_RCU_HEAD is not set
# CONFIG_DEBUG_OBJECTS_PERCPU_COUNTER is not set
CONFIG_DEBUG_OBJECTS_ENABLE_DEFAULT=1
# CONFIG_SLUB_DEBUG_ON is not set
CONFIG_SLUB_STATS=y
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_VM=y
# CONFIG_DEBUG_VM_VMACACHE is not set
CONFIG_DEBUG_VM_RB=y
# CONFIG_DEBUG_VM_PGFLAGS is not set
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_MEMORY_INIT is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_ARCH_HAS_KCOV=y
# CONFIG_KCOV is not set
CONFIG_DEBUG_SHIRQ=y

#
# Debug Lockups and Hangs
#
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_HARDLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=1
# CONFIG_DETECT_HUNG_TASK is not set
# CONFIG_WQ_WATCHDOG is not set
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_PANIC_TIMEOUT=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHED_INFO=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_SCHED_STACK_END_CHECK is not set
# CONFIG_DEBUG_TIMEKEEPING is not set
# CONFIG_DEBUG_PREEMPT is not set

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
CONFIG_DEBUG_LOCK_ALLOC=y
# CONFIG_PROVE_LOCKING is not set
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_WW_MUTEX_SELFTEST is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_LIST is not set
CONFIG_DEBUG_PI_LIST=y
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
CONFIG_DEBUG_CREDENTIALS=y

#
# RCU Debugging
#
# CONFIG_PROVE_RCU is not set
CONFIG_SPARSE_RCU_POINTER=y
# CONFIG_TORTURE_TEST is not set
# CONFIG_RCU_PERF_TEST is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
# CONFIG_RCU_TRACE is not set
# CONFIG_RCU_EQS_DEBUG is not set
# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_CPU_HOTPLUG_STATE_CONTROL is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set

#
# Runtime Testing
#
# CONFIG_LKDTM is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_TEST_SORT is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_RBTREE_TEST is not set
CONFIG_ATOMIC64_SELFTEST=y
# CONFIG_TEST_HEXDUMP is not set
# CONFIG_TEST_STRING_HELPERS is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_TEST_PRINTF is not set
# CONFIG_TEST_BITMAP is not set
# CONFIG_TEST_UUID is not set
# CONFIG_TEST_RHASHTABLE is not set
# CONFIG_TEST_HASH is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_TEST_FIRMWARE is not set
# CONFIG_TEST_UDELAY is not set
# CONFIG_MEMTEST is not set
# CONFIG_BUG_ON_DATA_CORRUPTION is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
# CONFIG_ARCH_WANTS_UBSAN_NO_NULL is not set
# CONFIG_UBSAN is not set
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
# CONFIG_EARLY_PRINTK is not set
# CONFIG_X86_PTDUMP_CORE is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_EFI_PGT_DUMP is not set
# CONFIG_DEBUG_WX is not set
CONFIG_DOUBLEFAULT=y
CONFIG_DEBUG_TLBFLUSH=y
# CONFIG_IOMMU_DEBUG is not set
CONFIG_IOMMU_STRESS=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
# CONFIG_IO_DELAY_0X80 is not set
CONFIG_IO_DELAY_0XED=y
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=1
CONFIG_DEBUG_BOOT_PARAMS=y
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set
# CONFIG_DEBUG_ENTRY is not set
# CONFIG_DEBUG_NMI_SELFTEST is not set
CONFIG_X86_DEBUG_FPU=y
# CONFIG_PUNIT_ATOM_DEBUG is not set

#
# Security options
#
CONFIG_KEYS=y
# CONFIG_PERSISTENT_KEYRINGS is not set
# CONFIG_BIG_KEYS is not set
CONFIG_TRUSTED_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
# CONFIG_KEY_DH_OPERATIONS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
CONFIG_SECURITYFS=y
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
CONFIG_HAVE_ARCH_HARDENED_USERCOPY=y
# CONFIG_HARDENED_USERCOPY is not set
# CONFIG_STATIC_USERMODEHELPER is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_XOR_BLOCKS=y
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=y
CONFIG_CRYPTO_AKCIPHER2=y
CONFIG_CRYPTO_KPP2=y
CONFIG_CRYPTO_ACOMP2=y
# CONFIG_CRYPTO_RSA is not set
# CONFIG_CRYPTO_DH is not set
# CONFIG_CRYPTO_ECDH is not set
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=y
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=y
CONFIG_CRYPTO_NULL2=y
# CONFIG_CRYPTO_PCRYPT is not set
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=y
# CONFIG_CRYPTO_MCRYPTD is not set
CONFIG_CRYPTO_AUTHENC=y
CONFIG_CRYPTO_ABLK_HELPER=y
CONFIG_CRYPTO_GLUE_HELPER_X86=y

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
CONFIG_CRYPTO_GCM=y
# CONFIG_CRYPTO_CHACHA20POLY1305 is not set
CONFIG_CRYPTO_SEQIV=y
CONFIG_CRYPTO_ECHAINIV=y

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=y
CONFIG_CRYPTO_PCBC=y
CONFIG_CRYPTO_XTS=y
# CONFIG_CRYPTO_KEYWRAP is not set

#
# Hash modes
#
# CONFIG_CRYPTO_CMAC is not set
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
CONFIG_CRYPTO_VMAC=y

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_CRC32 is not set
# CONFIG_CRYPTO_CRC32_PCLMUL is not set
CONFIG_CRYPTO_CRCT10DIF=y
# CONFIG_CRYPTO_CRCT10DIF_PCLMUL is not set
CONFIG_CRYPTO_GHASH=y
# CONFIG_CRYPTO_POLY1305 is not set
# CONFIG_CRYPTO_POLY1305_X86_64 is not set
CONFIG_CRYPTO_MD4=y
CONFIG_CRYPTO_MD5=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
CONFIG_CRYPTO_RMD160=y
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
# CONFIG_CRYPTO_SHA1_SSSE3 is not set
# CONFIG_CRYPTO_SHA256_SSSE3 is not set
# CONFIG_CRYPTO_SHA512_SSSE3 is not set
# CONFIG_CRYPTO_SHA1_MB is not set
# CONFIG_CRYPTO_SHA256_MB is not set
# CONFIG_CRYPTO_SHA512_MB is not set
CONFIG_CRYPTO_SHA256=y
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_SHA3 is not set
CONFIG_CRYPTO_TGR192=y
# CONFIG_CRYPTO_WP512 is not set
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=y

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_TI is not set
# CONFIG_CRYPTO_AES_X86_64 is not set
# CONFIG_CRYPTO_AES_NI_INTEL is not set
CONFIG_CRYPTO_ANUBIS=y
CONFIG_CRYPTO_ARC4=y
CONFIG_CRYPTO_BLOWFISH=y
CONFIG_CRYPTO_BLOWFISH_COMMON=y
CONFIG_CRYPTO_BLOWFISH_X86_64=y
# CONFIG_CRYPTO_CAMELLIA is not set
CONFIG_CRYPTO_CAMELLIA_X86_64=y
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=y
# CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64 is not set
CONFIG_CRYPTO_CAST_COMMON=y
CONFIG_CRYPTO_CAST5=y
# CONFIG_CRYPTO_CAST5_AVX_X86_64 is not set
CONFIG_CRYPTO_CAST6=y
CONFIG_CRYPTO_CAST6_AVX_X86_64=y
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_DES3_EDE_X86_64 is not set
CONFIG_CRYPTO_FCRYPT=y
CONFIG_CRYPTO_KHAZAD=y
CONFIG_CRYPTO_SALSA20=y
CONFIG_CRYPTO_SALSA20_X86_64=y
# CONFIG_CRYPTO_CHACHA20 is not set
# CONFIG_CRYPTO_CHACHA20_X86_64 is not set
CONFIG_CRYPTO_SEED=y
CONFIG_CRYPTO_SERPENT=y
# CONFIG_CRYPTO_SERPENT_SSE2_X86_64 is not set
CONFIG_CRYPTO_SERPENT_AVX_X86_64=y
# CONFIG_CRYPTO_SERPENT_AVX2_X86_64 is not set
CONFIG_CRYPTO_TEA=y
# CONFIG_CRYPTO_TWOFISH is not set
CONFIG_CRYPTO_TWOFISH_COMMON=y
CONFIG_CRYPTO_TWOFISH_X86_64=y
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=y
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=y

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
# CONFIG_CRYPTO_LZO is not set
# CONFIG_CRYPTO_842 is not set
# CONFIG_CRYPTO_LZ4 is not set
# CONFIG_CRYPTO_LZ4HC is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_DRBG_MENU=y
CONFIG_CRYPTO_DRBG_HMAC=y
# CONFIG_CRYPTO_DRBG_HASH is not set
# CONFIG_CRYPTO_DRBG_CTR is not set
CONFIG_CRYPTO_DRBG=y
CONFIG_CRYPTO_JITTERENTROPY=y
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
# CONFIG_CRYPTO_USER_API_RNG is not set
# CONFIG_CRYPTO_USER_API_AEAD is not set
CONFIG_CRYPTO_HASH_INFO=y
# CONFIG_CRYPTO_HW is not set
CONFIG_ASYMMETRIC_KEY_TYPE=y
# CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE is not set

#
# Certificates for signature checking
#
# CONFIG_SYSTEM_TRUSTED_KEYRING is not set
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set
CONFIG_VHOST_NET=y
CONFIG_VHOST=y
# CONFIG_VHOST_CROSS_ENDIAN_LEGACY is not set
# CONFIG_BINARY_PRINTF is not set

#
# Library routines
#
CONFIG_RAID6_PQ=y
CONFIG_BITREVERSE=y
# CONFIG_HAVE_ARCH_BITREVERSE is not set
CONFIG_RATIONAL=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_HAS_FAST_MULTIPLIER=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_CRC32_SELFTEST=y
# CONFIG_CRC32_SLICEBY8 is not set
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
CONFIG_CRC32_BIT=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=y
# CONFIG_CRC8 is not set
# CONFIG_AUDIT_ARCH_COMPAT_GENERIC is not set
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_DECOMPRESS=y
CONFIG_XZ_DEC=y
# CONFIG_XZ_DEC_X86 is not set
# CONFIG_XZ_DEC_POWERPC is not set
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
# CONFIG_XZ_DEC_SPARC is not set
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZ4=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_BCH=y
CONFIG_BCH_CONST_PARAMS=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=y
CONFIG_TEXTSEARCH_BM=y
CONFIG_TEXTSEARCH_FSM=y
CONFIG_BTREE=y
CONFIG_RADIX_TREE_MULTIORDER=y
CONFIG_ASSOCIATIVE_ARRAY=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT_MAP=y
CONFIG_HAS_DMA=y
# CONFIG_DMA_NOOP_OPS is not set
# CONFIG_DMA_VIRT_OPS is not set
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_GLOB=y
# CONFIG_GLOB_SELFTEST is not set
CONFIG_NLATTR=y
CONFIG_CORDIC=y
# CONFIG_DDR is not set
CONFIG_IRQ_POLL=y
CONFIG_OID_REGISTRY=y
CONFIG_UCS2_STRING=y
# CONFIG_SG_SPLIT is not set
CONFIG_SG_POOL=y
CONFIG_ARCH_HAS_SG_CHAIN=y
CONFIG_ARCH_HAS_PMEM_API=y
CONFIG_ARCH_HAS_MMIO_FLUSH=y
CONFIG_SBITMAP=y

[-- Attachment #3: job-script --]
[-- Type: text/plain, Size: 3597 bytes --]

#!/bin/sh

export_top_env()
{
	export suite='trinity'
	export testcase='trinity'
	export runtime=300
	export rootfs='yocto-tiny-i386-2016-04-22.cgz'
	export job_origin='/lkp/lkp/src/allot/rand/vm-kbuild-yocto-ia32/trinity.yaml'
	export testbox='vm-kbuild-yocto-ia32-9'
	export tbox_group='vm-kbuild-yocto-ia32'
	export kconfig='x86_64-acpi-redef'
	export compiler='gcc-6'
	export queue='bisect'
	export branch='linux-devel/devel-catchup-201703302119'
	export commit='fbc14616f483788afabe77d05bfb99883dc66c73'
	export submit_id='58dd63dc0b9a93911421cba8'
	export job_file='/lkp/scheduled/vm-kbuild-yocto-ia32-9/trinity-300s-yocto-tiny-i386-2016-04-22.cgz-fbc14616f483788afabe77d05bfb99883dc66c73-20170331-102676-cxuzks-0.yaml'
	export id='c0f94ae140c2a27eeebdfa62e0b44310d1a06d02'
	export model='qemu-system-x86_64 -enable-kvm -cpu Westmere'
	export nr_vm=32
	export nr_cpu=1
	export memory='512M'
	export swap_partitions='/dev/vda'
	export need_kconfig='CONFIG_KVM_GUEST=y'
	export enqueue_time='2017-03-31 04:00:28 +0800'
	export _id='58dd63dc0b9a93911421cba8'
	export user='lkp'
	export result_root='/result/trinity/300s/vm-kbuild-yocto-ia32/yocto-tiny-i386-2016-04-22.cgz/x86_64-acpi-redef/gcc-6/fbc14616f483788afabe77d05bfb99883dc66c73/0'
	export LKP_SERVER='inn'
	export max_uptime=1500
	export initrd='/osimage/yocto/yocto-tiny-i386-2016-04-22.cgz'
	export bootloader_append='root=/dev/ram0
user=lkp
job=/lkp/scheduled/vm-kbuild-yocto-ia32-9/trinity-300s-yocto-tiny-i386-2016-04-22.cgz-fbc14616f483788afabe77d05bfb99883dc66c73-20170331-102676-cxuzks-0.yaml
ARCH=x86_64
kconfig=x86_64-acpi-redef
branch=linux-devel/devel-catchup-201703302119
commit=fbc14616f483788afabe77d05bfb99883dc66c73
BOOT_IMAGE=/pkg/linux/x86_64-acpi-redef/gcc-6/fbc14616f483788afabe77d05bfb99883dc66c73/vmlinuz-4.11.0-rc4-00072-gfbc1461
max_uptime=1500
RESULT_ROOT=/result/trinity/300s/vm-kbuild-yocto-ia32/yocto-tiny-i386-2016-04-22.cgz/x86_64-acpi-redef/gcc-6/fbc14616f483788afabe77d05bfb99883dc66c73/0
LKP_SERVER=inn
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
net.ifnames=0
printk.devkmsg=on
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
drbd.minor_count=8
systemd.log_level=err
ignore_loglevel
earlyprintk=ttyS0,115200
console=ttyS0,115200
console=tty0
vga=normal
rw'
	export lkp_initrd='/lkp/lkp/lkp-i386.cgz'
	export bm_initrd='/osimage/deps/debian-x86_64-2016-08-31.cgz/run-ipconfig.i386_2016-09-03.cgz,/osimage/pkg/static/trinity-i386.cgz'
	export site='inn'
	export LKP_CGI_PORT=80
	export LKP_CIFS_PORT=139
	export kernel='/pkg/linux/x86_64-acpi-redef/gcc-6/fbc14616f483788afabe77d05bfb99883dc66c73/vmlinuz-4.11.0-rc4-00072-gfbc1461'
	export dequeue_time='2017-03-31 04:01:01 +0800'
	export job_initrd='/lkp/scheduled/vm-kbuild-yocto-ia32-9/trinity-300s-yocto-tiny-i386-2016-04-22.cgz-fbc14616f483788afabe77d05bfb99883dc66c73-20170331-102676-cxuzks-0.cgz'

	[ -n "$LKP_SRC" ] ||
	export LKP_SRC=/lkp/${user:-lkp}/src
}

run_job()
{
	echo $$ > $TMP/run-job.pid

	. $LKP_SRC/lib/http.sh
	. $LKP_SRC/lib/job.sh
	. $LKP_SRC/lib/env.sh

	export_top_env

	run_monitor $LKP_SRC/monitors/wrapper kmsg
	run_monitor $LKP_SRC/monitors/wrapper oom-killer
	run_monitor $LKP_SRC/monitors/plain/watchdog
	run_monitor $LKP_SRC/monitors/wrapper nfs-hang

	run_test $LKP_SRC/tests/wrapper trinity
}

extract_stats()
{
	$LKP_SRC/stats/wrapper kmsg

	$LKP_SRC/stats/wrapper time trinity.time
	$LKP_SRC/stats/wrapper time
	$LKP_SRC/stats/wrapper dmesg
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper stderr
	$LKP_SRC/stats/wrapper last_state
}

"$@"

[-- Attachment #4: dmesg.xz --]
[-- Type: application/octet-stream, Size: 12884 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-30 21:38   ` [printk] fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage kernel test robot
@ 2017-03-31  2:35     ` Sergey Senozhatsky
  2017-03-31  4:04       ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-31  2:35 UTC (permalink / raw)
  To: kernel test robot
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel,
	Sergey Senozhatsky, lkp

On (03/31/17 05:38), kernel test robot wrote:
> FYI, we noticed the following commit:
> 
> commit: fbc14616f483788afabe77d05bfb99883dc66c73 ("printk: enable printk offloading")
> url: https://github.com/0day-ci/linux/commits/Sergey-Senozhatsky/printk-introduce-printing-kernel-thread/20170330-185752

thanks for the report!

[..]
> +-------------------------------------------------+------------+------------+
> |                                                 | fd8b6b120c | fbc14616f4 |
> +-------------------------------------------------+------------+------------+
> | boot_successes                                  | 8          | 8          |
> | boot_failures                                   | 0          | 6          |
> | BUG:kernel_reboot-without-warning_in_test_stage | 0          | 4          |
> | BUG:kernel_hang_in_test_stage                   | 0          | 2          |
> +-------------------------------------------------+------------+------------+
> 
> 
> 
> [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
> [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
> [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
> 
> Elapsed time: 310
> BUG: kernel reboot-without-warning in test stage

so as far as I understand, this is the "missing kernel messages"
type of bug report. a worst case scenario.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-31  2:35     ` Sergey Senozhatsky
@ 2017-03-31  4:04       ` Sergey Senozhatsky
  2017-03-31  6:39         ` Ye Xiaolong
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-31  4:04 UTC (permalink / raw)
  To: kernel test robot
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel, lkp,
	Sergey Senozhatsky

On (03/31/17 11:35), Sergey Senozhatsky wrote:
[..]
> > [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
> > [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
> > [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
> > 
> > Elapsed time: 310
> > BUG: kernel reboot-without-warning in test stage
> 
> so as far as I understand, this is the "missing kernel messages"
> type of bug report. a worst case scenario.

panic() should have called console_flush_on_panic(), which sould have
flushed the messages regardless the printk_kthread state. so it probably
was not panic() that rebooted the kernel. (probably).

kernel_restart() and kernel_halt() have pr_emerg() messages, printk switches
to printk_emergency mode the first time it sees EMERG level message. (may be
we switch to late).

on the other hand, there is a emergency_restart(), where we don't switch
to printk_emergency mode and don't flush the existing kernel messages.
there is a bunch of places that call emergency_restart(), including sysrq.

may I ask you, how do you usually restart the vm after the test?
`echo X > /proc/sysrq-trigger'?

does this patch make it any better?

---
 drivers/tty/sysrq.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index 817dfb69914d..069f5540be36 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -240,7 +240,6 @@ static DECLARE_WORK(sysrq_showallcpus, sysrq_showregs_othercpus);
 
 static void sysrq_handle_showallcpus(int key)
 {
-	printk_emergency_begin();
 	/*
 	 * Fall back to the workqueue based printing if the
 	 * backtrace printing did not succeed or the
@@ -255,7 +254,6 @@ static void sysrq_handle_showallcpus(int key)
 		}
 		schedule_work(&sysrq_showallcpus);
 	}
-	printk_emergency_end();
 }
 
 static struct sysrq_key_op sysrq_showallcpus_op = {
@@ -282,10 +280,8 @@ static struct sysrq_key_op sysrq_showregs_op = {
 
 static void sysrq_handle_showstate(int key)
 {
-	printk_emergency_begin();
 	show_state();
 	show_workqueue_state();
-	printk_emergency_end();
 }
 static struct sysrq_key_op sysrq_showstate_op = {
 	.handler	= sysrq_handle_showstate,
@@ -296,9 +292,7 @@ static struct sysrq_key_op sysrq_showstate_op = {
 
 static void sysrq_handle_showstate_blocked(int key)
 {
-	printk_emergency_begin();
 	show_state_filter(TASK_UNINTERRUPTIBLE);
-	printk_emergency_end();
 }
 static struct sysrq_key_op sysrq_showstate_blocked_op = {
 	.handler	= sysrq_handle_showstate_blocked,
@@ -537,6 +531,7 @@ void __handle_sysrq(int key, bool check_mask)
 	int orig_log_level;
 	int i;
 
+	printk_emergency_begin();
 	rcu_sysrq_start();
 	rcu_read_lock();
 	/*
@@ -582,6 +577,7 @@ void __handle_sysrq(int key, bool check_mask)
 	}
 	rcu_read_unlock();
 	rcu_sysrq_end();
+	printk_emergency_end();
 }
 
 void handle_sysrq(int key)
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-31  4:04       ` Sergey Senozhatsky
@ 2017-03-31  6:39         ` Ye Xiaolong
  2017-03-31 14:47           ` Sergey Senozhatsky
  2017-04-05  7:29           ` Ye Xiaolong
  0 siblings, 2 replies; 75+ messages in thread
From: Ye Xiaolong @ 2017-03-31  6:39 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel, lkp

On 03/31, Sergey Senozhatsky wrote:
>On (03/31/17 11:35), Sergey Senozhatsky wrote:
>[..]
>> > [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
>> > [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
>> > [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
>> > 
>> > Elapsed time: 310
>> > BUG: kernel reboot-without-warning in test stage
>> 
>> so as far as I understand, this is the "missing kernel messages"
>> type of bug report. a worst case scenario.
>
>panic() should have called console_flush_on_panic(), which sould have
>flushed the messages regardless the printk_kthread state. so it probably
>was not panic() that rebooted the kernel. (probably).
>
>kernel_restart() and kernel_halt() have pr_emerg() messages, printk switches
>to printk_emergency mode the first time it sees EMERG level message. (may be
>we switch to late).
>
>on the other hand, there is a emergency_restart(), where we don't switch
>to printk_emergency mode and don't flush the existing kernel messages.
>there is a bunch of places that call emergency_restart(), including sysrq.
>
>may I ask you, how do you usually restart the vm after the test?
>`echo X > /proc/sysrq-trigger'?

Yes.

>
>does this patch make it any better?

I am trying it and will post the result once I get it.

Thanks,
Xiaolong
>
>---
> drivers/tty/sysrq.c | 8 ++------
> 1 file changed, 2 insertions(+), 6 deletions(-)
>
>diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
>index 817dfb69914d..069f5540be36 100644
>--- a/drivers/tty/sysrq.c
>+++ b/drivers/tty/sysrq.c
>@@ -240,7 +240,6 @@ static DECLARE_WORK(sysrq_showallcpus, sysrq_showregs_othercpus);
> 
> static void sysrq_handle_showallcpus(int key)
> {
>-	printk_emergency_begin();
> 	/*
> 	 * Fall back to the workqueue based printing if the
> 	 * backtrace printing did not succeed or the
>@@ -255,7 +254,6 @@ static void sysrq_handle_showallcpus(int key)
> 		}
> 		schedule_work(&sysrq_showallcpus);
> 	}
>-	printk_emergency_end();
> }
> 
> static struct sysrq_key_op sysrq_showallcpus_op = {
>@@ -282,10 +280,8 @@ static struct sysrq_key_op sysrq_showregs_op = {
> 
> static void sysrq_handle_showstate(int key)
> {
>-	printk_emergency_begin();
> 	show_state();
> 	show_workqueue_state();
>-	printk_emergency_end();
> }
> static struct sysrq_key_op sysrq_showstate_op = {
> 	.handler	= sysrq_handle_showstate,
>@@ -296,9 +292,7 @@ static struct sysrq_key_op sysrq_showstate_op = {
> 
> static void sysrq_handle_showstate_blocked(int key)
> {
>-	printk_emergency_begin();
> 	show_state_filter(TASK_UNINTERRUPTIBLE);
>-	printk_emergency_end();
> }
> static struct sysrq_key_op sysrq_showstate_blocked_op = {
> 	.handler	= sysrq_handle_showstate_blocked,
>@@ -537,6 +531,7 @@ void __handle_sysrq(int key, bool check_mask)
> 	int orig_log_level;
> 	int i;
> 
>+	printk_emergency_begin();
> 	rcu_sysrq_start();
> 	rcu_read_lock();
> 	/*
>@@ -582,6 +577,7 @@ void __handle_sysrq(int key, bool check_mask)
> 	}
> 	rcu_read_unlock();
> 	rcu_sysrq_end();
>+	printk_emergency_end();
> }
> 
> void handle_sysrq(int key)
>-- 
>2.12.2
>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu
  2017-03-29  9:25 ` [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu Sergey Senozhatsky
@ 2017-03-31 13:09   ` Petr Mladek
  2017-03-31 13:33     ` Peter Zijlstra
  0 siblings, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-03-31 13:09 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:04, Sergey Senozhatsky wrote:
> Do not keep `printk_pending' in per-CPU area. We set the following bits
> of printk_pending:
> a) PRINTK_PENDING_WAKEUP
> 	when we need to wakeup klogd
> b) PRINTK_PENDING_OUTPUT
> 	when there is a pending output from deferred printk and we need
> 	to call console_unlock().
> 
> So none of the bits control/represent a state of a particular CPU and,
> basically, they should be global instead.
> 
> Besides we will use `printk_pending' to control printk kthread, so this
> patch is also a preparation work.
> 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> Suggested-by: Petr Mladek <pmladek@suse.com>
> ---
>  kernel/printk/printk.c | 26 ++++++++++++--------------
>  1 file changed, 12 insertions(+), 14 deletions(-)
> 
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 2984fb0f0257..2d07678e9ff9 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -401,6 +401,14 @@ DEFINE_RAW_SPINLOCK(logbuf_lock);
>  		printk_safe_exit_irqrestore(flags);	\
>  	} while (0)
>  
> +/*
> + * Delayed printk version, for scheduler-internal messages:
> + */
> +#define PRINTK_PENDING_WAKEUP	0x01
> +#define PRINTK_PENDING_OUTPUT	0x02
> +
> +static unsigned long printk_pending;
> +
>  #ifdef CONFIG_PRINTK
>  DECLARE_WAIT_QUEUE_HEAD(log_wait);
>  /* the next printk record to read by syslog(READ) or /proc/kmsg */
> @@ -2654,25 +2662,15 @@ static int __init printk_late_init(void)
>  late_initcall(printk_late_init);
>  
>  #if defined CONFIG_PRINTK
> -/*
> - * Delayed printk version, for scheduler-internal messages:
> - */
> -#define PRINTK_PENDING_WAKEUP	0x01
> -#define PRINTK_PENDING_OUTPUT	0x02
> -
> -static DEFINE_PER_CPU(int, printk_pending);
> -
>  static void wake_up_klogd_work_func(struct irq_work *irq_work)
>  {
> -	int pending = __this_cpu_xchg(printk_pending, 0);
> -
> -	if (pending & PRINTK_PENDING_OUTPUT) {
> +	if (test_and_clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending)) {
>  		/* If trylock fails, someone else is doing the printing */
>  		if (console_trylock())
>  			console_unlock();
>  	}
>  
> -	if (pending & PRINTK_PENDING_WAKEUP)
> +	if (test_and_clear_bit(PRINTK_PENDING_WAKEUP, &printk_pending))
>  		wake_up_interruptible(&log_wait);
>  }
>  
> @@ -2685,7 +2683,7 @@ void wake_up_klogd(void)
>  {
>  	preempt_disable();
>  	if (waitqueue_active(&log_wait)) {
> -		this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
> +		set_bit(PRINTK_PENDING_WAKEUP, &printk_pending);

We should add here a write barrier:

	/*
	 * irq_work_queue() uses cmpxchg() and implies the memory
	 * barrier only when the work is queued. An explicit barrier
	 * is needed here to make sure that wake_up_klogd_work_func()
	 * sees printk_pending set even when the work was already queued
	 * because of an other pending event.
	 */
	 smp_wmb();

>  		irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
>  	}
>  	preempt_enable();
> @@ -2701,7 +2699,7 @@ int printk_deferred(const char *fmt, ...)
>  	r = vprintk_emit(0, LOGLEVEL_SCHED, NULL, 0, fmt, args);
>  	va_end(args);
>  
> -	__this_cpu_or(printk_pending, PRINTK_PENDING_OUTPUT);
> +	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);

Same here.

We are going to use printk_pending even more in the other patches.
I still have to check the final state. It might be useful
to define a helper if the barrier is needed on more locations.

Otherwise the change looks fine to me. It should reduce redundant
wakeups.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu
  2017-03-31 13:09   ` Petr Mladek
@ 2017-03-31 13:33     ` Peter Zijlstra
  2017-04-03 11:23       ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Peter Zijlstra @ 2017-03-31 13:33 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Sergey Senozhatsky, Steven Rostedt, Jan Kara, Andrew Morton,
	Linus Torvalds, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Fri, Mar 31, 2017 at 03:09:50PM +0200, Petr Mladek wrote:
> On Wed 2017-03-29 18:25:04, Sergey Senozhatsky wrote:

> >  	if (waitqueue_active(&log_wait)) {
> > -		this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
> > +		set_bit(PRINTK_PENDING_WAKEUP, &printk_pending);
> 
> We should add here a write barrier:
> 
> 	/*
> 	 * irq_work_queue() uses cmpxchg() and implies the memory
> 	 * barrier only when the work is queued. An explicit barrier
> 	 * is needed here to make sure that wake_up_klogd_work_func()
> 	 * sees printk_pending set even when the work was already queued
> 	 * because of an other pending event.
> 	 */
> 	 smp_wmb();
> 
> >  		irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
> >  	}
> >  	preempt_enable();

smp_mb__after_atomic() is probably better, because if you're not
ordering with the cmpxchg, you're ordering against a load done by
cmpxchg to see it doesn't need to do anything.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-31  6:39         ` Ye Xiaolong
@ 2017-03-31 14:47           ` Sergey Senozhatsky
  2017-03-31 15:28             ` Eric W. Biederman
  2017-04-05  7:29           ` Ye Xiaolong
  1 sibling, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-03-31 14:47 UTC (permalink / raw)
  To: Ye Xiaolong
  Cc: Sergey Senozhatsky, Sergey Senozhatsky, Steven Rostedt,
	Petr Mladek, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, lkp

On (03/31/17 14:39), Ye Xiaolong wrote:
> On 03/31, Sergey Senozhatsky wrote:
> >On (03/31/17 11:35), Sergey Senozhatsky wrote:
> >[..]
> >> > [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
> >> > [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
> >> > [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
> >> > 
> >> > Elapsed time: 310
> >> > BUG: kernel reboot-without-warning in test stage
> >> 
> >> so as far as I understand, this is the "missing kernel messages"
> >> type of bug report. a worst case scenario.
> >
> >panic() should have called console_flush_on_panic(), which sould have
> >flushed the messages regardless the printk_kthread state. so it probably
> >was not panic() that rebooted the kernel. (probably).
> >
> >kernel_restart() and kernel_halt() have pr_emerg() messages, printk switches
> >to printk_emergency mode the first time it sees EMERG level message. (may be
> >we switch to late).
> >
> >on the other hand, there is a emergency_restart(), where we don't switch
> >to printk_emergency mode and don't flush the existing kernel messages.
> >there is a bunch of places that call emergency_restart(), including sysrq.
> >
> >may I ask you, how do you usually restart the vm after the test?
> >`echo X > /proc/sysrq-trigger'?
> 
> Yes.
> 
> >
> >does this patch make it any better?
> 
> I am trying it and will post the result once I get it.


... I'd also probably add pr_emerg() print-out to emergency_restart(),
the same way kernel_restart()/kernel_halt()/kernel_power_off() do.

for those cases when emergency_restart() is called with printk in
kthreaded mode, not in emergency mode.

---

diff --git a/kernel/reboot.c b/kernel/reboot.c
index e4ced883d8de..5bce29da913b 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -60,6 +60,7 @@ void (*pm_power_off_prepare)(void);
  */
 void emergency_restart(void)
 {
+	pr_emerg("Emergency restart\n");
 	kmsg_dump(KMSG_DUMP_EMERG);
 	machine_emergency_restart();
 }

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func()
  2017-03-29  9:25 ` [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func() Sergey Senozhatsky
@ 2017-03-31 14:56   ` Petr Mladek
  2017-04-04 16:15     ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-03-31 14:56 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:06, Sergey Senozhatsky wrote:
> Offload printing of printk_deferred() messages from IRQ context
> to a schedulable printing kthread, when possible (the same way
> we do it in vprintk_emit()). Otherwise, console_unlock() can
> force the printing CPU to spend unbound amount of time flushing
> kernel messages from IRQ context.
> 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> ---
>  kernel/printk/printk.c | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index ab6b3b2a68c6..1927b5cb5cbe 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -2741,8 +2741,16 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work)
>  		 * If trylock fails, someone else is doing the printing.
>  		 * PRINTK_PENDING_OUTPUT bit is cleared by console_unlock().
>  		 */
> -		if (console_trylock())
> -			console_unlock();
> +		if (printk_kthread_enabled()) {
> +			wake_up_process(printk_kthread);

Note that the relation between printk_kthread_enabled()
and wake_up_process() is racy. The conditions might change
between these two calls. It looks fine here, well almost.

The critical point is in vprintk_emit(). It must use the emergency
mode (call the consoles directly) when it is called from a process
that started the emergency mode.

We could be more relaxed here. IMHO, the only sensitive situation
is if printk_deferred() is used in the emergency context.
We might want to use the emergency mode here as well but
it is not guaranteed. printk_emergency might get cleared
in the meantime.

A solution might be to add one more bit, e.g.
PRINTK_PENDING_EMERGENCY_OUTPUT. We should force the emergency mode
here when it is set. It should be cleared together with the normal
PRINTK_PENDING_OUTPUT.

Or do you think that this is a corner case that we could
ignore for now?

Otherwise, it looks fine to me.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places
  2017-03-29  9:25 ` [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places Sergey Senozhatsky
@ 2017-03-31 15:06   ` Petr Mladek
  2017-04-06 17:20   ` Pavel Machek
  1 sibling, 0 replies; 75+ messages in thread
From: Petr Mladek @ 2017-03-31 15:06 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:07, Sergey Senozhatsky wrote:
> It's not always possible/safe to wake_up() printk kernel
> thread. For example, late suspend/early resume may printk()
> while timekeeping is not initialized yet, so calling into the
> scheduler may result in recursive warnings.
> 
> Another thing to notice is the fact PM at some point
> freezes user space and kernel threads: freeze_processes()
> and freeze_kernel_threads(), correspondingly. Thus we need
> printk() to operate in old mode there and attempt to
> immediately flush pending kernel message to the console.
> 
> This patch adds printk_emergency_begin/on sections.
>
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

It looks reasonable to me. Feel free to use:

Reviewed-by: Petr Mladek <pmladek@suse.com>

Well, it still would be great if people more familiar
with this code look at it.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-31 14:47           ` Sergey Senozhatsky
@ 2017-03-31 15:28             ` Eric W. Biederman
  2017-04-03  9:31               ` Jan Kara
  2017-04-03 10:51               ` Sergey Senozhatsky
  0 siblings, 2 replies; 75+ messages in thread
From: Eric W. Biederman @ 2017-03-31 15:28 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Ye Xiaolong, Sergey Senozhatsky, Steven Rostedt, Petr Mladek,
	Jan Kara, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, lkp

Sergey Senozhatsky <sergey.senozhatsky@gmail.com> writes:

> On (03/31/17 14:39), Ye Xiaolong wrote:
>> On 03/31, Sergey Senozhatsky wrote:
>> >On (03/31/17 11:35), Sergey Senozhatsky wrote:
>> >[..]
>> >> > [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
>> >> > [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
>> >> > [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
>> >> > 
>> >> > Elapsed time: 310
>> >> > BUG: kernel reboot-without-warning in test stage
>> >> 
>> >> so as far as I understand, this is the "missing kernel messages"
>> >> type of bug report. a worst case scenario.
>> >
>> >panic() should have called console_flush_on_panic(), which sould have
>> >flushed the messages regardless the printk_kthread state. so it probably
>> >was not panic() that rebooted the kernel. (probably).
>> >
>> >kernel_restart() and kernel_halt() have pr_emerg() messages, printk switches
>> >to printk_emergency mode the first time it sees EMERG level message. (may be
>> >we switch to late).
>> >
>> >on the other hand, there is a emergency_restart(), where we don't switch
>> >to printk_emergency mode and don't flush the existing kernel messages.
>> >there is a bunch of places that call emergency_restart(), including sysrq.
>> >
>> >may I ask you, how do you usually restart the vm after the test?
>> >`echo X > /proc/sysrq-trigger'?
>> 
>> Yes.
>> 
>> >
>> >does this patch make it any better?
>> 
>> I am trying it and will post the result once I get it.
>
>
> ... I'd also probably add pr_emerg() print-out to emergency_restart(),
> the same way kernel_restart()/kernel_halt()/kernel_power_off() do.
>
> for those cases when emergency_restart() is called with printk in
> kthreaded mode, not in emergency mode.

No. No. No.

emergency_restart should be the equivalent of a watchdog going off.
AKA it is long past the point where you want to be coordinating
with other parts of the kernel.  Rebooting is the priority.
A print statement absolutely does not belong in emergency_restart.

The fact that nothing managed to get printed out without magic flushing
code is highly disturbing.

Looking from the outside this patchset appears to be broken by design.

If you don't want kernel functions suffering from the overhead of
printing to a slow output device, don't do that then.

The point of printk is to give debugging output.  You have fundamentally
incapacitated printk from serving it's primary purpose.

NAK to the entire concept.

Eric

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 5/8] sysrq: switch to printk.emergency mode in unsafe places
  2017-03-29  9:25 ` [RFC][PATCHv2 5/8] sysrq: " Sergey Senozhatsky
@ 2017-03-31 15:37   ` Petr Mladek
  2017-04-01  0:04     ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-03-31 15:37 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:08, Sergey Senozhatsky wrote:
> It's not always possible/safe to wake_up() printk kernel
> thread from sysrq (theoretically). Thus we better switch
> printk() to emergency mode in some of the sysrq handlers,
> which allows us to immediately flush pending kernel message
> to the console.
>
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> Suggested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>

It looks like a decent selection. It is pity that some of
them might produce rather long output and theoretically
cause a softlookup. But I do not know about a better solution
at the moment. In each case, it will not be worse than before
this patchset.

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 6/8] kexec: switch to printk.emergency mode in unsafe places
  2017-03-29  9:25 ` [RFC][PATCHv2 6/8] kexec: " Sergey Senozhatsky
@ 2017-03-31 15:39   ` Petr Mladek
  0 siblings, 0 replies; 75+ messages in thread
From: Petr Mladek @ 2017-03-31 15:39 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:09, Sergey Senozhatsky wrote:
> Switch to printk emergency mode in kernel_kexec(), this
> lets us to immediately flush pending kernel message to
> the console and to avoid potentially unsafe calls into
> the scheduler.
> 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

Looks fine and works as expected.

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 5/8] sysrq: switch to printk.emergency mode in unsafe places
  2017-03-31 15:37   ` Petr Mladek
@ 2017-04-01  0:04     ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-01  0:04 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Sergey Senozhatsky, Steven Rostedt, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, Sergey Senozhatsky

Hello,

On (03/31/17 17:37), Petr Mladek wrote:
> On Wed 2017-03-29 18:25:08, Sergey Senozhatsky wrote:
> > It's not always possible/safe to wake_up() printk kernel
> > thread from sysrq (theoretically). Thus we better switch
> > printk() to emergency mode in some of the sysrq handlers,
> > which allows us to immediately flush pending kernel message
> > to the console.
> >
> > Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> > Suggested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> 
> It looks like a decent selection. It is pity that some of
> them might produce rather long output and theoretically
> cause a softlookup. But I do not know about a better solution
> at the moment. In each case, it will not be worse than before
> this patchset.

thanks.

this patch will be replaced with a more 'generic' one, which
I posted in reply yo Ye Xiaolong's report.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-31 15:28             ` Eric W. Biederman
@ 2017-04-03  9:31               ` Jan Kara
  2017-04-03 10:06                 ` Petr Mladek
  2017-04-06 17:33                 ` Pavel Machek
  2017-04-03 10:51               ` Sergey Senozhatsky
  1 sibling, 2 replies; 75+ messages in thread
From: Jan Kara @ 2017-04-03  9:31 UTC (permalink / raw)
  To: Eric W. Biederman
  Cc: Sergey Senozhatsky, Ye Xiaolong, Sergey Senozhatsky,
	Steven Rostedt, Petr Mladek, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, lkp

On Fri 31-03-17 10:28:15, Eric W. Biederman wrote:
> Sergey Senozhatsky <sergey.senozhatsky@gmail.com> writes:
> 
> > On (03/31/17 14:39), Ye Xiaolong wrote:
> >> On 03/31, Sergey Senozhatsky wrote:
> >> >On (03/31/17 11:35), Sergey Senozhatsky wrote:
> >> >[..]
> >> >> > [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
> >> >> > [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
> >> >> > [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
> >> >> > 
> >> >> > Elapsed time: 310
> >> >> > BUG: kernel reboot-without-warning in test stage
> >> >> 
> >> >> so as far as I understand, this is the "missing kernel messages"
> >> >> type of bug report. a worst case scenario.
> >> >
> >> >panic() should have called console_flush_on_panic(), which sould have
> >> >flushed the messages regardless the printk_kthread state. so it probably
> >> >was not panic() that rebooted the kernel. (probably).
> >> >
> >> >kernel_restart() and kernel_halt() have pr_emerg() messages, printk switches
> >> >to printk_emergency mode the first time it sees EMERG level message. (may be
> >> >we switch to late).
> >> >
> >> >on the other hand, there is a emergency_restart(), where we don't switch
> >> >to printk_emergency mode and don't flush the existing kernel messages.
> >> >there is a bunch of places that call emergency_restart(), including sysrq.
> >> >
> >> >may I ask you, how do you usually restart the vm after the test?
> >> >`echo X > /proc/sysrq-trigger'?
> >> 
> >> Yes.
> >> 
> >> >
> >> >does this patch make it any better?
> >> 
> >> I am trying it and will post the result once I get it.
> >
> >
> > ... I'd also probably add pr_emerg() print-out to emergency_restart(),
> > the same way kernel_restart()/kernel_halt()/kernel_power_off() do.
> >
> > for those cases when emergency_restart() is called with printk in
> > kthreaded mode, not in emergency mode.
> 
> No. No. No.
> 
> emergency_restart should be the equivalent of a watchdog going off.
> AKA it is long past the point where you want to be coordinating
> with other parts of the kernel.  Rebooting is the priority.
> A print statement absolutely does not belong in emergency_restart.
>
> The fact that nothing managed to get printed out without magic flushing
> code is highly disturbing.
> 
> Looking from the outside this patchset appears to be broken by design.
> 
> If you don't want kernel functions suffering from the overhead of
> printing to a slow output device, don't do that then.

Sorry, but the above is just contradictory. On one hand you say that
missing messages is disturbing and on the other hand you say we should have
no messages to avoid the overhead of printing. The fact is kernel has tons
of messages because people want to see what happens to possibly debug stuff.
And I don't see as viable to reduce amount of messages as it is neverending
fight and always someone will be unhappy. As a result currently some machines
are not able to boot due to printk traffic and there are other nasty
effects from CPUs getting stuck printing messages to serial console (and
this really bothers people as is proved by the fact that about every 6
months someone comes with a hack to printk to fix the particular lockup he
is hitting).

This patch set gives up part of the printk() reliability for bounded
latency (at least unless we detect we are really in trouble) which is IMHO
a good trade-off for lots of users (and others can just turn this feature
off).

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-03  9:31               ` Jan Kara
@ 2017-04-03 10:06                 ` Petr Mladek
  2017-04-06 17:33                 ` Pavel Machek
  1 sibling, 0 replies; 75+ messages in thread
From: Petr Mladek @ 2017-04-03 10:06 UTC (permalink / raw)
  To: Jan Kara, Eric W. Biederman
  Cc: Sergey Senozhatsky, Ye Xiaolong, Sergey Senozhatsky,
	Steven Rostedt, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, lkp

On Mon 2017-04-03 11:31:52, Jan Kara wrote:
> On Fri 31-03-17 10:28:15, Eric W. Biederman wrote:
> > Sergey Senozhatsky <sergey.senozhatsky@gmail.com> writes:
> > 
> > > On (03/31/17 14:39), Ye Xiaolong wrote:
> > >> On 03/31, Sergey Senozhatsky wrote:
> > >> >On (03/31/17 11:35), Sergey Senozhatsky wrote:
> > >> >[..]
> > >> >> > [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
> > >> >> > [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
> > >> >> > [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
> > >> >> > 
> > >> >> > Elapsed time: 310
> > >> >> > BUG: kernel reboot-without-warning in test stage
> > >> >> 
> > >> >> so as far as I understand, this is the "missing kernel messages"
> > >> >> type of bug report. a worst case scenario.
> > >> >
> > >> >panic() should have called console_flush_on_panic(), which sould have
> > >> >flushed the messages regardless the printk_kthread state. so it probably
> > >> >was not panic() that rebooted the kernel. (probably).
> > >> >
> > >> >kernel_restart() and kernel_halt() have pr_emerg() messages, printk switches
> > >> >to printk_emergency mode the first time it sees EMERG level message. (may be
> > >> >we switch to late).
> > >> >
> > >> >on the other hand, there is a emergency_restart(), where we don't switch
> > >> >to printk_emergency mode and don't flush the existing kernel messages.
> > >> >there is a bunch of places that call emergency_restart(), including sysrq.
> > >> >
> > >> >may I ask you, how do you usually restart the vm after the test?
> > >> >`echo X > /proc/sysrq-trigger'?
> > >> 
> > >> Yes.
> > >> 
> > >> >
> > >> >does this patch make it any better?
> > >> 
> > >> I am trying it and will post the result once I get it.
> > >
> > >
> > > ... I'd also probably add pr_emerg() print-out to emergency_restart(),
> > > the same way kernel_restart()/kernel_halt()/kernel_power_off() do.
> > >
> > > for those cases when emergency_restart() is called with printk in
> > > kthreaded mode, not in emergency mode.
> > 
> > No. No. No.
> > 
> > emergency_restart should be the equivalent of a watchdog going off.
> > AKA it is long past the point where you want to be coordinating
> > with other parts of the kernel.  Rebooting is the priority.
> > A print statement absolutely does not belong in emergency_restart.

Sergey suggested to add pr_emerg() because it would signalize
emergency situation, printk kthread would be disabled and all
messages would be printed to the console directly (the old way).
It will _not_ be necessary to wakeup the kthread.

Note that the we could do it also _without_ pr_emerg().
Instead we could call simple/fast printk_emergency_begin()
as we do in other similar situations, for example during
kexec, suspend, see 4th and 6th patch of this patchset.


> > The fact that nothing managed to get printed out without magic flushing
> > code is highly disturbing.
> > 
> > Looking from the outside this patchset appears to be broken by design.
> > 
> > If you don't want kernel functions suffering from the overhead of
> > printing to a slow output device, don't do that then.
> 
> Sorry, but the above is just contradictory. On one hand you say that
> missing messages is disturbing and on the other hand you say we should have
> no messages to avoid the overhead of printing. The fact is kernel has tons
> of messages because people want to see what happens to possibly debug stuff.
> And I don't see as viable to reduce amount of messages as it is neverending
> fight and always someone will be unhappy. As a result currently some machines
> are not able to boot due to printk traffic and there are other nasty
> effects from CPUs getting stuck printing messages to serial console (and
> this really bothers people as is proved by the fact that about every 6
> months someone comes with a hack to printk to fix the particular lockup he
> is hitting).

Yup, the fact is that there are situations when printk() itself brings
the system into problems, for example when too many messages are
flushed to a slow console in interrupt context.


> This patch set gives up part of the printk() reliability for bounded
> latency (at least unless we detect we are really in trouble) which is IMHO
> a good trade-off for lots of users (and others can just turn this feature
> off).

My view of this patchset is the following:

Deferred console output is perfectly fine when the system is in
reasonable state. The deferring is needed to keep the system safe
in some situations.

Of course, the view is different when the deferring is not longer
reliable (panic, kexec, suspend, restart). We try to detect these
situations, disable deferring, and push the messages the old way.
We call this emergency mode.

I am sure that we miss some situations. Also they might be hard
to detect. For this case, there is the kernel parameter, sysfs
knob that will allow to keep the old mode all the time, see
7th patch.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-31 15:28             ` Eric W. Biederman
  2017-04-03  9:31               ` Jan Kara
@ 2017-04-03 10:51               ` Sergey Senozhatsky
  1 sibling, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-03 10:51 UTC (permalink / raw)
  To: Eric W. Biederman
  Cc: Sergey Senozhatsky, Ye Xiaolong, Sergey Senozhatsky,
	Steven Rostedt, Petr Mladek, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, lkp

On (03/31/17 10:28), Eric W. Biederman wrote:
[..]
> > ... I'd also probably add pr_emerg() print-out to emergency_restart(),
> > the same way kernel_restart()/kernel_halt()/kernel_power_off() do.
> >
> > for those cases when emergency_restart() is called with printk in
> > kthreaded mode, not in emergency mode.
> 
> No. No. No.
> 
> emergency_restart should be the equivalent of a watchdog going off.
> AKA it is long past the point where you want to be coordinating
> with other parts of the kernel.  Rebooting is the priority.
> A print statement absolutely does not belong in emergency_restart.
> 
> The fact that nothing managed to get printed out without magic flushing
> code is highly disturbing.

Eric, have you checked what is usually going on right before the
emergency_restart() call?

a quick grep.

kernel/panic.c

	pr_emerg("Kernel panic - not syncing: %s\n", buf);
...
	console_flush_on_panic();
...
	emergency_restart();


kernel/debug/kdb/kdb_main.c

	kdb_printf("forcing reboot\n");
	kdb_reboot(0, NULL);
		emergency_restart();


kernel/debug/gdbstub.c

	gdb_cmd_reboot()
		printk(KERN_CRIT "Executing emergency reboot\n");
		machine_emergency_restart();


drivers/tty/sysrq.c

	__handle_sysrq()
		pr_info("SysRq : ");
		pr_cont("%s\n", op_p->action_msg);
		op_p->handler(key);
			sysrq_handle_reboot()
				emergency_restart()

and so on...


all those printk()-s, that are happening right before emergency_restart(),
in fact flush (!) all the pending logbuf messages to the serial console.
and seems it doesn't cause any troubles on you side. but having printk()
not one line _before_the emergency_restart(), but _in_ emergency_restart()
is all of a sudden very disturbing. how come?


> Looking from the outside this patchset appears to be broken by design.
> 
> If you don't want kernel functions suffering from the overhead of
> printing to a slow output device, don't do that then.

sorry, this is not productive. "don't use printk()" is not a solution.


> The point of printk is to give debugging output.  You have fundamentally
> incapacitated printk from serving it's primary purpose.

the point of the patch set is that printk has a fundamental issue -- it
can easily soft/hard lockup the system; it can stall RCU; it can cause
OOM; and so on and on and on.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu
  2017-03-31 13:33     ` Peter Zijlstra
@ 2017-04-03 11:23       ` Sergey Senozhatsky
  2017-04-03 12:43         ` Petr Mladek
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-03 11:23 UTC (permalink / raw)
  To: Petr Mladek, Peter Zijlstra
  Cc: Sergey Senozhatsky, Steven Rostedt, Jan Kara, Andrew Morton,
	Linus Torvalds, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On (03/31/17 15:33), Peter Zijlstra wrote:
> On Fri, Mar 31, 2017 at 03:09:50PM +0200, Petr Mladek wrote:
> > On Wed 2017-03-29 18:25:04, Sergey Senozhatsky wrote:
> 
> > >  	if (waitqueue_active(&log_wait)) {
> > > -		this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
> > > +		set_bit(PRINTK_PENDING_WAKEUP, &printk_pending);
> > 
> > We should add here a write barrier:
> > 
> > 	/*
> > 	 * irq_work_queue() uses cmpxchg() and implies the memory
> > 	 * barrier only when the work is queued. An explicit barrier
> > 	 * is needed here to make sure that wake_up_klogd_work_func()
> > 	 * sees printk_pending set even when the work was already queued
> > 	 * because of an other pending event.
> > 	 */
> > 	 smp_wmb();
> > 
> > >  		irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
> > >  	}
> > >  	preempt_enable();
> 
> smp_mb__after_atomic() is probably better, because if you're not
> ordering with the cmpxchg, you're ordering against a load done by
> cmpxchg to see it doesn't need to do anything.

Petr and Peter, thanks for the review.

can you educate me, what exactly is broken there?

when called from console_unlock(), we have something as follows

	console_unlock()
	{
		for (;;) {
			spin_lock_irqsave();
			...
			spin_unlock_irqrestore();
			...
		}

		spin_unlock_irqrestore();

<<IRQs enabled>>

		if (wake_klogd)
			wake_up_klogd()
			{
				set_bit(PRINTK_PENDING_WAKEUP, &printk_pending);
				irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
			}
	}


we queue a per-CPU irq_work. given that by the time we execute wake_up_klogd()
we have local IRQs enabled on that CPU. is it possible that we will have that
CPU's irq_work still being queued?


when called from printk_deferred().

I'm still trying to understand what scenario can cause the problem. so
basically on that CPU we have a call into the scheduler/timer which ends
up in printk_deferred()... and then we have console_unlock()->wake_up_klogd()
//* local IRQs enabled but the irq_work is still queued *// and atop of it
we have IRQ that executes that CPU's run_list and fails to see updated
PRINTK_PENDING_WAKEUP bit, because wake_up_klogd() was called on already
queued wake_up_klogd_work. is this the case? if so, can this race happen on
the CPU?

I don't object the barrier, I'm just trying to have a better understanding
what's broken. sorry if I'm missing something very obvious.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu
  2017-04-03 11:23       ` Sergey Senozhatsky
@ 2017-04-03 12:43         ` Petr Mladek
  0 siblings, 0 replies; 75+ messages in thread
From: Petr Mladek @ 2017-04-03 12:43 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Peter Zijlstra, Sergey Senozhatsky, Steven Rostedt, Jan Kara,
	Andrew Morton, Linus Torvalds, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel

On Mon 2017-04-03 20:23:01, Sergey Senozhatsky wrote:
> On (03/31/17 15:33), Peter Zijlstra wrote:
> > On Fri, Mar 31, 2017 at 03:09:50PM +0200, Petr Mladek wrote:
> > > On Wed 2017-03-29 18:25:04, Sergey Senozhatsky wrote:
> > 
> > > >  	if (waitqueue_active(&log_wait)) {
> > > > -		this_cpu_or(printk_pending, PRINTK_PENDING_WAKEUP);
> > > > +		set_bit(PRINTK_PENDING_WAKEUP, &printk_pending);
> > > 
> > > We should add here a write barrier:
> > > 
> > > 	/*
> > > 	 * irq_work_queue() uses cmpxchg() and implies the memory
> > > 	 * barrier only when the work is queued. An explicit barrier
> > > 	 * is needed here to make sure that wake_up_klogd_work_func()
> > > 	 * sees printk_pending set even when the work was already queued
> > > 	 * because of an other pending event.
> > > 	 */
> > > 	 smp_wmb();
> > > 
> > > >  		irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
> > > >  	}
> > > >  	preempt_enable();
> > 
> > smp_mb__after_atomic() is probably better, because if you're not
> > ordering with the cmpxchg, you're ordering against a load done by
> > cmpxchg to see it doesn't need to do anything.
> 
> Petr and Peter, thanks for the review.
> 
> can you educate me, what exactly is broken there?

Good point!

> when called from console_unlock(), we have something as follows
> 
> 	console_unlock()
> 	{
> 		for (;;) {
> 			spin_lock_irqsave();
> 			...
> 			spin_unlock_irqrestore();
> 			...
> 		}
> 
> 		spin_unlock_irqrestore();
> 
> <<IRQs enabled>>
> 
> 		if (wake_klogd)
> 			wake_up_klogd()
> 			{
> 				set_bit(PRINTK_PENDING_WAKEUP, &printk_pending);
> 				irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
> 			}
> 	}
> 
> 
> we queue a per-CPU irq_work.

Ah, I forgot that irq_work is still per-CPU. In this case, everything
seems to be safe even without the barrier. The important thing is that
there always will be queued an irq_work that will see and handle the
bit. I believe that the barrier would be needed if the irq_work was
global.

I am sorry for the noise.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter
  2017-03-29  9:25 ` [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter Sergey Senozhatsky
@ 2017-04-03 15:29   ` Petr Mladek
  2017-04-04  8:29     ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-04-03 15:29 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:10, Sergey Senozhatsky wrote:
> This param permits user-space to forcibly on/off printk emergency
> mode via /sys/module/printk/parameters/emergency_mode node.
> 
> We have annotated sections in the kernel that switch printk to
> emergency, but there might be places/cases when user space would
> want to have printk operate in emergency mode all the time.

Thanks a lot for the parameter.
 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> ---
>  kernel/printk/printk.c | 20 +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 1927b5cb5cbe..0d96839bb450 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -455,7 +455,7 @@ static struct task_struct *printk_kthread __read_mostly;
>  static atomic_t printk_emergency __read_mostly;
>  /*
>   * Disable printk_kthread permanently. Unlike `oops_in_progress'
> - * it doesn't go back to 0.
> + * it doesn't go back to 0 (unless enforced by user-space).
>   */
>  static bool printk_kthread_disabled __read_mostly;
>  
> @@ -483,6 +483,24 @@ void printk_emergency_end(void)
>  	atomic_dec(&printk_emergency);
>  }
>  
> +static int printk_kthread_disabled_set(const char *val,
> +					const struct kernel_param *kp)
> +{
> +	return param_set_bool(val, kp);
> +}
> +
> +static const struct kernel_param_ops printk_kthread_disabled_ops = {
> +	.set = printk_kthread_disabled_set,
> +	.get = param_get_bool,
> +};
> +
> +module_param_cb(emergency_mode,
> +		&printk_kthread_disabled_ops,
> +		&printk_kthread_disabled,
> +		0644);
> +MODULE_PARM_DESC(emergency_mode,
> +		"don't offload message printing to printk kthread");

I wonder if we could make this easier. Something like:

static bool printk_force_emergency;
module_param_named(force_emergency, printk_force_emergency,
		   bool, S_IRUGO | S_IWUSR);

and use it instead of printk_kthread_disabled variable. It was
confusing anyway. You already mentioned that it did not
stop the kthread.

Also the relation between the sysfs entry and printk code
will be cleaner. People might thing that emergency_mode
shows the immediate value of printk_emergency variable.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 8/8] printk: enable printk offloading
  2017-03-29  9:25 ` [RFC][PATCHv2 8/8] printk: enable printk offloading Sergey Senozhatsky
  2017-03-30 21:38   ` [printk] fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage kernel test robot
@ 2017-04-03 15:42   ` Petr Mladek
  2017-04-04 13:20     ` Sergey Senozhatsky
  1 sibling, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-04-03 15:42 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:11, Sergey Senozhatsky wrote:
> Initialize the kernel printing thread and enable printk()
> offloading.
> 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> ---
>  kernel/printk/printk.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 0d96839bb450..acfdc50580db 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -2796,6 +2796,25 @@ static int printk_kthread_func(void *data)
>  	return 0;
>  }
>  
> +/*
> + * Init printk kthread at late_initcall stage, after core/arch/device/etc.
> + * initialization.
> + */
> +static int __init init_printk_kthread(void)
> +{
> +	struct task_struct *thread;
> +
> +	thread = kthread_run(printk_kthread_func, NULL, "printk");
> +	if (IS_ERR(thread)) {
> +		pr_err("printk: unable to create printing thread\n");
> +		return PTR_ERR(thread);
> +	}
> +
> +	printk_kthread = thread;
> +	return 0;
> +}
> +late_initcall(init_printk_kthread);

I like the simplicity. I just wonder if people on tiny devices might
want to disable it. In each case, it does not make sense on non-SMP
machines or when people force the emergency mode all the time.

I am not sure what is the practice here. I wonder if we should be
proactive or keep it as is and wait until anyone complains. IMHO,
it is not that big deal but...

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter
  2017-04-03 15:29   ` Petr Mladek
@ 2017-04-04  8:29     ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-04  8:29 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Sergey Senozhatsky, Steven Rostedt, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, Sergey Senozhatsky

On (04/03/17 17:29), Petr Mladek wrote:
[..]
> > +module_param_cb(emergency_mode,
> > +		&printk_kthread_disabled_ops,
> > +		&printk_kthread_disabled,
> > +		0644);
> > +MODULE_PARM_DESC(emergency_mode,
> > +		"don't offload message printing to printk kthread");
> 
> I wonder if we could make this easier. Something like:
> 
> static bool printk_force_emergency;
> module_param_named(force_emergency, printk_force_emergency,
> 		   bool, S_IRUGO | S_IWUSR);

yes, can do. thanks.

> and use it instead of printk_kthread_disabled variable. It was
> confusing anyway. You already mentioned that it did not
> stop the kthread.

yeah, I didn't like the `printk_kthread_disabled' naming, but
at the same time didn't feel like having `printk_emergency' and
`printk_forced_emergency'. will take a look.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-03-29  9:25 ` [RFC][PATCHv2 2/8] printk: introduce printing kernel thread Sergey Senozhatsky
@ 2017-04-04  9:01   ` Petr Mladek
  2017-04-04  9:36     ` Sergey Senozhatsky
  2017-04-06 17:14   ` Pavel Machek
  1 sibling, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-04-04  9:01 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-03-29 18:25:05, Sergey Senozhatsky wrote:
> This patch introduces a dedicated printing kernel thread - printk_kthread.
> The main purpose of this kthread is to offload printing to a non-atomic
> and always scheduleable context, which eliminates 4) and makes 1)-3) less
> critical. printk() now just appends log messages to the kernel log buffer
> and wake_up()s printk_kthread instead of locking console_sem and calling
> into potentially unsafe console_unlock().
> 
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 2d07678e9ff9..ab6b3b2a68c6 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -445,6 +447,42 @@ static char __log_buf[__LOG_BUF_LEN] __aligned(LOG_ALIGN);
>  static char *log_buf = __log_buf;
>  static u32 log_buf_len = __LOG_BUF_LEN;
>  
> +static struct task_struct *printk_kthread __read_mostly;
> +/*
> + * We can't call into the scheduler (wake_up() printk kthread) during
> + * suspend/kexec/etc. This temporarily switches printk to old behaviour.
> + */
> +static atomic_t printk_emergency __read_mostly;
> +/*
> + * Disable printk_kthread permanently. Unlike `oops_in_progress'
> + * it doesn't go back to 0.
> + */

The comment is not valid once we allow to modify the variable using
the sysfs knob.

> @@ -1765,17 +1803,40 @@ asmlinkage int vprintk_emit(int facility, int level,
>  
>  	printed_len += log_output(facility, level, lflags, dict, dictlen, text, text_len);
>  
> +	/*
> +	 * Emergency level indicates that the system is unstable and, thus,
> +	 * we better stop relying on wake_up(printk_kthread) and try to do
> +	 * a direct printing.
> +	 */
> +	if (level == LOGLEVEL_EMERG)
> +		printk_kthread_disabled = true;
> +
> +	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
>  	logbuf_unlock_irqrestore(flags);
>  
>  	/* If called from the scheduler, we can not call up(). */
>  	if (!in_sched) {
>  		/*
> -		 * Try to acquire and then immediately release the console
> -		 * semaphore.  The release will print out buffers and wake up
> -		 * /dev/kmsg and syslog() users.
> +		 * Under heavy printing load/slow serial console/etc
> +		 * console_unlock() can stall CPUs, which can result in
> +		 * soft/hard-lockups, lost interrupts, RCU stalls, etc.
> +		 * Therefore we attempt to print the messages to console
> +		 * from a dedicated printk_kthread, which always runs in
> +		 * schedulable context.
>  		 */
> -		if (console_trylock())
> -			console_unlock();
> +		if (printk_kthread_enabled()) {
> +			printk_safe_enter_irqsave(flags);
> +			wake_up_process(printk_kthread);
> +			printk_safe_exit_irqrestore(flags);

I am really happy that we have the printk_safe stuff available!

> +		} else {
> +			/*
> +			 * Try to acquire and then immediately release the
> +			 * console semaphore. The release will print out
> +			 * buffers and wake up /dev/kmsg and syslog() users.
> +			 */
> +			if (console_trylock())
> +				console_unlock();
> +		}
>  	}
>  
>  	return printed_len;
> @@ -1882,6 +1943,9 @@ static size_t msg_print_text(const struct printk_log *msg,
>  			     bool syslog, char *buf, size_t size) { return 0; }
>  static bool suppress_message_printing(int level) { return false; }
>  
> +void printk_emergency_begin(void) {}
> +void printk_emergency_end(void) {}
> +
>  #endif /* CONFIG_PRINTK */
>  
>  #ifdef CONFIG_EARLY_PRINTK
> @@ -2164,6 +2228,13 @@ void console_unlock(void)
>  	bool do_cond_resched, retry;
>  
>  	if (console_suspended) {
> +		/*
> +		 * Avoid an infinite loop in printk_kthread function
> +		 * when console_unlock() cannot flush messages because
> +		 * we suspended consoles. Someone else will print the
> +		 * messages from resume_console().
> +		 */
> +		clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);

Great catch!

>  		up_console_sem();
>  		return;
>  	}
> @@ -2182,6 +2253,7 @@ void console_unlock(void)
>  	console_may_schedule = 0;
>  
>  again:
> +	clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);

This will not help if new messages appear during
call_console_drivers().

I would move this line after the for(;;) cycle. It will be
cleared when all messages are really handled.

Otherwise, it looks fine to me.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-04-04  9:01   ` Petr Mladek
@ 2017-04-04  9:36     ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-04  9:36 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Sergey Senozhatsky, Steven Rostedt, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, Sergey Senozhatsky

On (04/04/17 11:01), Petr Mladek wrote:
[..]
> > +static atomic_t printk_emergency __read_mostly;
> > +/*
> > + * Disable printk_kthread permanently. Unlike `oops_in_progress'
> > + * it doesn't go back to 0.
> > + */
> 
> The comment is not valid once we allow to modify the variable using
> the sysfs knob.

it's updated in that patch (sysfs knob introduction).


[..]
> > @@ -2182,6 +2253,7 @@ void console_unlock(void)
> >  	console_may_schedule = 0;
> >  
> >  again:
> > +	clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
> 
> This will not help if new messages appear during
> call_console_drivers().

you are right. wouldn't do much harm (an extra console_unlock() from
printk_kthread in the worst case), but agree.

I added it there because of that "!can_use_console()" branch. not that
I expect printk_kthread being executed on !online CPU, but we might have
no callable consoles.

probably should have that clear_bit() before and after the loop.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 8/8] printk: enable printk offloading
  2017-04-03 15:42   ` [RFC][PATCHv2 8/8] printk: enable printk offloading Petr Mladek
@ 2017-04-04 13:20     ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-04 13:20 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Sergey Senozhatsky, Steven Rostedt, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, Sergey Senozhatsky

On (04/03/17 17:42), Petr Mladek wrote:
> > +/*
> > + * Init printk kthread at late_initcall stage, after core/arch/device/etc.
> > + * initialization.
> > + */
> > +static int __init init_printk_kthread(void)
> > +{
> > +	struct task_struct *thread;
> > +
> > +	thread = kthread_run(printk_kthread_func, NULL, "printk");
> > +	if (IS_ERR(thread)) {
> > +		pr_err("printk: unable to create printing thread\n");
> > +		return PTR_ERR(thread);
> > +	}
> > +
> > +	printk_kthread = thread;
> > +	return 0;
> > +}
> > +late_initcall(init_printk_kthread);
> 
> I like the simplicity. I just wonder if people on tiny devices might
> want to disable it. In each case, it does not make sense on non-SMP
> machines or when people force the emergency mode all the time.
> 
> I am not sure what is the practice here. I wonder if we should be
> proactive or keep it as is and wait until anyone complains. IMHO,
> it is not that big deal but...

I tend to agree that this is not a big deal, as of now.
I've bigger concerns.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func()
  2017-03-31 14:56   ` Petr Mladek
@ 2017-04-04 16:15     ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-04 16:15 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Sergey Senozhatsky, Steven Rostedt, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Pavel Machek,
	Len Brown, linux-kernel, Sergey Senozhatsky

Hi Petr,

sorry for the delay.

On (03/31/17 16:56), Petr Mladek wrote:
[..]
> > diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> > index ab6b3b2a68c6..1927b5cb5cbe 100644
> > --- a/kernel/printk/printk.c
> > +++ b/kernel/printk/printk.c
> > @@ -2741,8 +2741,16 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work)
> >  		 * If trylock fails, someone else is doing the printing.
> >  		 * PRINTK_PENDING_OUTPUT bit is cleared by console_unlock().
> >  		 */
> > -		if (console_trylock())
> > -			console_unlock();
> > +		if (printk_kthread_enabled()) {
> > +			wake_up_process(printk_kthread);
> 
> Note that the relation between printk_kthread_enabled()
> and wake_up_process() is racy. The conditions might change
> between these two calls. It looks fine here, well almost.
> 
> The critical point is in vprintk_emit(). It must use the emergency
> mode (call the consoles directly) when it is called from a process
> that started the emergency mode.

hm, we don't guarantee this. printk(), both in threaded and in
emergency modes, can fail to acquire console_sem.

> We could be more relaxed here. IMHO, the only sensitive situation
> is if printk_deferred() is used in the emergency context.
> We might want to use the emergency mode here as well but
> it is not guaranteed.

hm, I don't think any path does

	printk_emergency_begin()
	printk_deferred()
	printk_emergency_end()

and expects logbuf output to be flushed by the time it does
printk_emergency_end(). it's most likely something like this

	printk_emergency_begin()
	printk()
	printk_emergency_end()

the expectations here are more reasonable, but still, no
guarantees are provided (even in non-kthreaded printk mode).

> A solution might be to add one more bit, e.g.
> PRINTK_PENDING_EMERGENCY_OUTPUT. We should force the emergency mode
> here when it is set. It should be cleared together with the normal
> PRINTK_PENDING_OUTPUT.
> 
> Or do you think that this is a corner case that we could
> ignore for now?

hm, I guess we don't really count on irq_work in emergency
situations. but I need more time to think. good questions, Petr.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-03-31  6:39         ` Ye Xiaolong
  2017-03-31 14:47           ` Sergey Senozhatsky
@ 2017-04-05  7:29           ` Ye Xiaolong
  2017-04-05  8:40             ` Sergey Senozhatsky
  1 sibling, 1 reply; 75+ messages in thread
From: Ye Xiaolong @ 2017-04-05  7:29 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Pavel Machek, Len Brown, linux-kernel, lkp

On 03/31, Ye Xiaolong wrote:
>On 03/31, Sergey Senozhatsky wrote:
>>On (03/31/17 11:35), Sergey Senozhatsky wrote:
>>[..]
>>> > [   21.009531] VFS: Warning: trinity-c2 using old stat() call. Recompile your binary.
>>> > [   21.148898] VFS: Warning: trinity-c0 using old stat() call. Recompile your binary.
>>> > [   22.298208] warning: process `trinity-c2' used the deprecated sysctl system call with 
>>> > 
>>> > Elapsed time: 310
>>> > BUG: kernel reboot-without-warning in test stage
>>> 
>>> so as far as I understand, this is the "missing kernel messages"
>>> type of bug report. a worst case scenario.
>>
>>panic() should have called console_flush_on_panic(), which sould have
>>flushed the messages regardless the printk_kthread state. so it probably
>>was not panic() that rebooted the kernel. (probably).
>>
>>kernel_restart() and kernel_halt() have pr_emerg() messages, printk switches
>>to printk_emergency mode the first time it sees EMERG level message. (may be
>>we switch to late).
>>
>>on the other hand, there is a emergency_restart(), where we don't switch
>>to printk_emergency mode and don't flush the existing kernel messages.
>>there is a bunch of places that call emergency_restart(), including sysrq.
>>
>>may I ask you, how do you usually restart the vm after the test?
>>`echo X > /proc/sysrq-trigger'?
>
>Yes.
>
>>
>>does this patch make it any better?
>
>I am trying it and will post the result once I get it.

Sorry for the late. I applied the patch of on top of the fbc14616f4 ("printk: enable printk offloading") 
and the "reboot-without-waring" issue is gone for 6 times of tests.

testcase/path_params/tbox_group/run: trinity/300s/vm-kbuild-yocto-ia32

fbc14616f483788a  a75384abd8885080d6923d7036  
----------------  --------------------------  
          4:4         -100%            0:6     dmesg.BUG:kernel_reboot-without-warning_in_test_stage

Thanks,
Xiaolong

>
>Thanks,
>Xiaolong
>>
>>---
>> drivers/tty/sysrq.c | 8 ++------
>> 1 file changed, 2 insertions(+), 6 deletions(-)
>>
>>diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
>>index 817dfb69914d..069f5540be36 100644
>>--- a/drivers/tty/sysrq.c
>>+++ b/drivers/tty/sysrq.c
>>@@ -240,7 +240,6 @@ static DECLARE_WORK(sysrq_showallcpus, sysrq_showregs_othercpus);
>> 
>> static void sysrq_handle_showallcpus(int key)
>> {
>>-	printk_emergency_begin();
>> 	/*
>> 	 * Fall back to the workqueue based printing if the
>> 	 * backtrace printing did not succeed or the
>>@@ -255,7 +254,6 @@ static void sysrq_handle_showallcpus(int key)
>> 		}
>> 		schedule_work(&sysrq_showallcpus);
>> 	}
>>-	printk_emergency_end();
>> }
>> 
>> static struct sysrq_key_op sysrq_showallcpus_op = {
>>@@ -282,10 +280,8 @@ static struct sysrq_key_op sysrq_showregs_op = {
>> 
>> static void sysrq_handle_showstate(int key)
>> {
>>-	printk_emergency_begin();
>> 	show_state();
>> 	show_workqueue_state();
>>-	printk_emergency_end();
>> }
>> static struct sysrq_key_op sysrq_showstate_op = {
>> 	.handler	= sysrq_handle_showstate,
>>@@ -296,9 +292,7 @@ static struct sysrq_key_op sysrq_showstate_op = {
>> 
>> static void sysrq_handle_showstate_blocked(int key)
>> {
>>-	printk_emergency_begin();
>> 	show_state_filter(TASK_UNINTERRUPTIBLE);
>>-	printk_emergency_end();
>> }
>> static struct sysrq_key_op sysrq_showstate_blocked_op = {
>> 	.handler	= sysrq_handle_showstate_blocked,
>>@@ -537,6 +531,7 @@ void __handle_sysrq(int key, bool check_mask)
>> 	int orig_log_level;
>> 	int i;
>> 
>>+	printk_emergency_begin();
>> 	rcu_sysrq_start();
>> 	rcu_read_lock();
>> 	/*
>>@@ -582,6 +577,7 @@ void __handle_sysrq(int key, bool check_mask)
>> 	}
>> 	rcu_read_unlock();
>> 	rcu_sysrq_end();
>>+	printk_emergency_end();
>> }
>> 
>> void handle_sysrq(int key)
>>-- 
>>2.12.2
>>

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-05  7:29           ` Ye Xiaolong
@ 2017-04-05  8:40             ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-05  8:40 UTC (permalink / raw)
  To: Ye Xiaolong
  Cc: Sergey Senozhatsky, Sergey Senozhatsky, Steven Rostedt,
	Petr Mladek, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Pavel Machek, Len Brown,
	linux-kernel, lkp

On (04/05/17 15:29), Ye Xiaolong wrote:
[..]
> >>does this patch make it any better?
> >
> >I am trying it and will post the result once I get it.
> 
> Sorry for the late. I applied the patch of on top of the fbc14616f4 ("printk: enable printk offloading") 
> and the "reboot-without-waring" issue is gone for 6 times of tests.
> 
> testcase/path_params/tbox_group/run: trinity/300s/vm-kbuild-yocto-ia32
> 
> fbc14616f483788a  a75384abd8885080d6923d7036  
> ----------------  --------------------------  
>           4:4         -100%            0:6     dmesg.BUG:kernel_reboot-without-warning_in_test_stage

thanks!

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-03-29  9:25 ` [RFC][PATCHv2 2/8] printk: introduce printing kernel thread Sergey Senozhatsky
  2017-04-04  9:01   ` Petr Mladek
@ 2017-04-06 17:14   ` Pavel Machek
  2017-04-07  5:12     ` Sergey Senozhatsky
  1 sibling, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-06 17:14 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Petr Mladek, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, Sergey Senozhatsky

[-- Attachment #1: Type: text/plain, Size: 3594 bytes --]

Hi!

> printk() is quite complex internally and, basically, it does two
> slightly independent things:
>  a) adds a new message to a kernel log buffer (log_store())
>  b) prints kernel log messages to serial consoles (console_unlock())
> 
> while (a) is guaranteed to be executed by printk(), (b) is not, for a
> variety of reasons, and, unlike log_store(), it comes at a price:
> 
>  1) console_unlock() attempts to flush all pending kernel log messages
> to the console. Thus, it can loop indefinitely.
> 
>  2) while console_unlock() is executed on one particular CPU, printing
> pending kernel log messages, other CPUs can simultaneously append new
> messages to the kernel log buffer.
> 
>  3) the time it takes console_unlock() to print kernel messages also
> depends on the speed of the console -- which may not be fast at all.
> 
>  4) console_unlock() is executed in the same context as printk(), so
> it may be non-preemptible/atomic, which makes 1)-3) dangerous.
> 
> As a result, nobody knows how long a printk() call will take, so
> it's not really safe to call printk() in a number of situations,
> including atomic context, RCU critical sections, interrupt context,
> and more.

You have made good argumentation for not flushing unlimited ammount of
messages from printk() -- okay. But I don't think this is good idea:

> @@ -1765,17 +1803,40 @@ asmlinkage int vprintk_emit(int facility, int level,
>  
>  	printed_len += log_output(facility, level, lflags, dict, dictlen, text, text_len);
>  
> +	/*
> +	 * Emergency level indicates that the system is unstable and, thus,
> +	 * we better stop relying on wake_up(printk_kthread) and try to do
> +	 * a direct printing.
> +	 */
> +	if (level == LOGLEVEL_EMERG)
> +		printk_kthread_disabled = true;
> +
> +	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);

Messages lower then _EMERG may be important, too.. and usually are,
for debugging.

And you keep both code paths, anyway, so they have to work. So you did
not really "fix" issues you are pointing out -- they still remain
there for _EMERG and above.

I agree that printing too much is a problem. Could you just print
"(messages delayed)" in that case, then wake a kernel thread to [rint
the rest?

								Pavel


>  	logbuf_unlock_irqrestore(flags);
>  
>  	/* If called from the scheduler, we can not call up(). */
>  	if (!in_sched) {
>  		/*
> -		 * Try to acquire and then immediately release the console
> -		 * semaphore.  The release will print out buffers and wake up
> -		 * /dev/kmsg and syslog() users.
> +		 * Under heavy printing load/slow serial console/etc
> +		 * console_unlock() can stall CPUs, which can result in
> +		 * soft/hard-lockups, lost interrupts, RCU stalls, etc.
> +		 * Therefore we attempt to print the messages to console
> +		 * from a dedicated printk_kthread, which always runs in
> +		 * schedulable context.
>  		 */
> -		if (console_trylock())
> -			console_unlock();
> +		if (printk_kthread_enabled()) {
> +			printk_safe_enter_irqsave(flags);
> +			wake_up_process(printk_kthread);
> +			printk_safe_exit_irqrestore(flags);
> +		} else {
> +			/*
> +			 * Try to acquire and then immediately release the
> +			 * console semaphore. The release will print out
> +			 * buffers and wake up /dev/kmsg and syslog() users.
> +			 */
> +			if (console_trylock())
> +				console_unlock();
> +		}
>  	}
>


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places
  2017-03-29  9:25 ` [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places Sergey Senozhatsky
  2017-03-31 15:06   ` Petr Mladek
@ 2017-04-06 17:20   ` Pavel Machek
  2017-04-09 10:59     ` Andreas Mohr
  1 sibling, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-06 17:20 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Petr Mladek, Jan Kara, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Eric Biederman, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, Sergey Senozhatsky

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Wed 2017-03-29 18:25:07, Sergey Senozhatsky wrote:
> It's not always possible/safe to wake_up() printk kernel
> thread. For example, late suspend/early resume may printk()
> while timekeeping is not initialized yet, so calling into the
> scheduler may result in recursive warnings.
> 
> Another thing to notice is the fact PM at some point
> freezes user space and kernel threads: freeze_processes()
> and freeze_kernel_threads(), correspondingly. Thus we need
> printk() to operate in old mode there and attempt to
> immediately flush pending kernel message to the console.
> 
> This patch adds printk_emergency_begin/on sections.
> 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

I don't like this. It is symptom of printk getting much more fragile
now.

								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-03  9:31               ` Jan Kara
  2017-04-03 10:06                 ` Petr Mladek
@ 2017-04-06 17:33                 ` Pavel Machek
  2017-04-07  4:44                   ` Sergey Senozhatsky
  1 sibling, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-06 17:33 UTC (permalink / raw)
  To: Jan Kara
  Cc: Eric W. Biederman, Sergey Senozhatsky, Ye Xiaolong,
	Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

Hi!

> This patch set gives up part of the printk() reliability for bounded
> latency (at least unless we detect we are really in trouble) which is IMHO
> a good trade-off for lots of users (and others can just turn this feature
> off).

If they can ever realize they were bitten by this feature.

Can we go for different tradeoff?

In console_unlock(), if you detect too much work, print "Too many
messages to print, %d bytes delayed" and wake up kernel thread.

You still get the latency, and people bitten by this feature will at
least get fair warning.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-06 17:33                 ` Pavel Machek
@ 2017-04-07  4:44                   ` Sergey Senozhatsky
  2017-04-07  7:15                     ` Pavel Machek
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-07  4:44 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Jan Kara, Eric W. Biederman, Sergey Senozhatsky, Ye Xiaolong,
	Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

Hello,

On (04/06/17 19:33), Pavel Machek wrote:
> > This patch set gives up part of the printk() reliability for bounded
> > latency (at least unless we detect we are really in trouble) which is IMHO
> > a good trade-off for lots of users (and others can just turn this feature
> > off).
> 
> If they can ever realize they were bitten by this feature.
> 
> Can we go for different tradeoff?
> 
> In console_unlock(), if you detect too much work, print "Too many
> messages to print, %d bytes delayed" and wake up kernel thread.

"too many messages" is undefined. console_unlock() can be called from
IRQ handler or with preemtion disabled, or under spin_lock, or under
RCU read lock, etc. etc. By the time we decide to wake up printk_kthread
from console_unlock() it may be already too late.

besides, this does not really address any of the concerns you have
pointed out in other emails. we might be unable to wake_up printk_kthread
(because there is a misbehaving higher prio process, or because the
scheduler is misbehaving, etc. etc.) so the "emergency mode" is still
here and still requires special handling.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-04-06 17:14   ` Pavel Machek
@ 2017-04-07  5:12     ` Sergey Senozhatsky
  2017-04-07  7:21       ` Pavel Machek
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-07  5:12 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, Sergey Senozhatsky

Hello,

On (04/06/17 19:14), Pavel Machek wrote:
[..]
> > @@ -1765,17 +1803,40 @@ asmlinkage int vprintk_emit(int facility, int level,
> >  
> >  	printed_len += log_output(facility, level, lflags, dict, dictlen, text, text_len);
> >  
> > +	/*
> > +	 * Emergency level indicates that the system is unstable and, thus,
> > +	 * we better stop relying on wake_up(printk_kthread) and try to do
> > +	 * a direct printing.
> > +	 */
> > +	if (level == LOGLEVEL_EMERG)
> > +		printk_kthread_disabled = true;
> > +
> > +	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
> 
> Messages lower then _EMERG may be important, too.. and usually are,
> for debugging.
> 
> And you keep both code paths, anyway, so they have to work. So you did
> not really "fix" issues you are pointing out -- they still remain
> there for _EMERG and above.

we don't drop messages of lower levels. we just print then from a
schedulable context. once the things go off the rails, and EMERG
is a good hint, I think, we stop being optimismitcs and switch to
a "best effort" mode. that is sort of reasonable. if there is a
flood of EMERG messages that are not actually important and,
basically, are result of a coding error, then, I think, the we
must fix that coding error. I mean, there should be some common
sense, and doing
		while (1)
			printk(KERN_EMERG "hello\n");
is probably not.


> I agree that printing too much is a problem. Could you just print
> "(messages delayed)" in that case, then wake a kernel thread to [rint
> the rest?

sorry, but what difference would it make?

it's really unclear at what point we should offload printing if we begin
that "we will offload sometimes". for example, I've seen many spin-lock
lockups where printk was involved.

	CPU0					CPU1		CPU2					CPU3

	spin_lock(&lock)					spin_lock(&lock)			spin_lock(&lock)
	printk("foo") // grabs the console_sem
	printk("bar")				printk("a")
						printk("b")
						printk("c")
						...
						printk("z")
								spin_dump()				spin_dump()
	  call_console_drivers()				 printk()				 printk()
	   serial_driver_write()				 printk()				 printk()
	    spin_lock_irqsave(port->lock)			 ...					 ...
	     uart_console_write(...)				 trigger_all_cpu_backtrace()		 trigger_all_cpu_backtrace()
	      serial_driver_putchar()
	       while (!txrdy(...))
	         cpu_relax()


spin_dump() and trigger_all_cpu_backtrace() result in a bunch of
additional printk()-s so CPU0 has even more job to do in console_unlock(),
while it still holds the contended spin_lock. and so on; there are
many other examples.

so should we declare a "we can spend only 2 seconds in direct printk()
and then must offload printing" rule? I don't think it's much better
than a simpler "we always offload, as long as we think it's safe".

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07  4:44                   ` Sergey Senozhatsky
@ 2017-04-07  7:15                     ` Pavel Machek
  2017-04-07  7:46                       ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-07  7:15 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Jan Kara, Eric W. Biederman, Sergey Senozhatsky, Ye Xiaolong,
	Steven Rostedt, Petr Mladek, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

[-- Attachment #1: Type: text/plain, Size: 1788 bytes --]

On Fri 2017-04-07 13:44:40, Sergey Senozhatsky wrote:
> Hello,
> 
> On (04/06/17 19:33), Pavel Machek wrote:
> > > This patch set gives up part of the printk() reliability for bounded
> > > latency (at least unless we detect we are really in trouble) which is IMHO
> > > a good trade-off for lots of users (and others can just turn this feature
> > > off).
> > 
> > If they can ever realize they were bitten by this feature.
> > 
> > Can we go for different tradeoff?
> > 
> > In console_unlock(), if you detect too much work, print "Too many
> > messages to print, %d bytes delayed" and wake up kernel thread.
> 
> "too many messages" is undefined. console_unlock() can be called from
> IRQ handler or with preemtion disabled, or under spin_lock, or under
> RCU read lock, etc. etc. By the time we decide to wake up printk_kthread
> from console_unlock() it may be already too late.

So lets define "too many messages" as 240 characters. We know printk
worked rather well for us for more than 20 years. Kernel code is used
to printk taking few miliseconds. 

> besides, this does not really address any of the concerns you have
> pointed out in other emails. we might be unable to wake_up printk_kthread
> (because there is a misbehaving higher prio process, or because the
> scheduler is misbehaving, etc. etc.) so the "emergency mode" is still
> here and still requires special handling.

Yeah? So you know modified printk() does not work, that's why
"emergency mode" exists. Unfortunately, you can't rely on fact that
you can detect half-crashed machines by printk levels. You usually
can't.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-04-07  5:12     ` Sergey Senozhatsky
@ 2017-04-07  7:21       ` Pavel Machek
  2017-04-07  8:15         ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-07  7:21 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

Hi!

> spin_dump() and trigger_all_cpu_backtrace() result in a bunch of
> additional printk()-s so CPU0 has even more job to do in console_unlock(),
> while it still holds the contended spin_lock. and so on; there are
> many other examples.
> 
> so should we declare a "we can spend only 2 seconds in direct printk()
> and then must offload printing" rule? I don't think it's much better
> than a simpler "we always offload, as long as we think it's safe".

I believe we should do the 2 seconds rule. It allows us to print "some
messages delayed" message, so at least whoever is trying to debug the
crash will have the hints that he needs to look at the printk system.

"we always offload, as long as we think it's safe" rule does not
really work, as printk() can not detect if it is safe or not.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07  7:15                     ` Pavel Machek
@ 2017-04-07  7:46                       ` Sergey Senozhatsky
  2017-04-07  8:14                         ` Pavel Machek
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-07  7:46 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Sergey Senozhatsky, Ye Xiaolong, Steven Rostedt, Petr Mladek,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, lkp

On (04/07/17 09:15), Pavel Machek wrote:
> On Fri 2017-04-07 13:44:40, Sergey Senozhatsky wrote:
> > Hello,
> > 
> > On (04/06/17 19:33), Pavel Machek wrote:
> > > > This patch set gives up part of the printk() reliability for bounded
> > > > latency (at least unless we detect we are really in trouble) which is IMHO
> > > > a good trade-off for lots of users (and others can just turn this feature
> > > > off).
> > > 
> > > If they can ever realize they were bitten by this feature.
> > > 
> > > Can we go for different tradeoff?
> > > 
> > > In console_unlock(), if you detect too much work, print "Too many
> > > messages to print, %d bytes delayed" and wake up kernel thread.
> > 
> > "too many messages" is undefined. console_unlock() can be called from
> > IRQ handler or with preemtion disabled, or under spin_lock, or under
> > RCU read lock, etc. etc. By the time we decide to wake up printk_kthread
> > from console_unlock() it may be already too late.
> 
> So lets define "too many messages" as 240 characters. We know printk
> worked rather well for us for more than 20 years. Kernel code is used
> to printk taking few miliseconds.

serial console can be quite slow. and port->lock, that is acquired by
console_unlock()->call_console_drivers()->write(), is also accessible
by serial driver's IRQ handler, and this lock may be busy long
enough -- as long as that IRQ handler transmits/receives chars. but
that's not the point.

[..]
> Yeah? So you know modified printk() does not work, that's why
> "emergency mode" exists. Unfortunately, you can't rely on fact that
> you can detect half-crashed machines by printk levels. You usually
> can't.

I'm not happy with those printk_emergency_begin()/end(), sure. but that's
the reality -- every single solution that would offload printing duty implies
that there will be cases when offloading would not be possible. either
PENDING_PRINTK_IPI to other CPUs, or irq_work(PENDING_OUTPUT) on a local CPU,
or anything else (um... what it is?... softirq? tasklet? print one logbuf
entry from every IRQ handler? dunno, anything else?). There will be cases
when we won't be able to expect that something will take over and finish
printing for us. Well, may be I'm missing some other solution that would
offload printing, eliminating lockup conditions, and at the same time work
in 100% of the cases.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07  7:46                       ` Sergey Senozhatsky
@ 2017-04-07  8:14                         ` Pavel Machek
  2017-04-07 12:10                           ` Sergey Senozhatsky
  2017-04-07 14:29                           ` Steven Rostedt
  0 siblings, 2 replies; 75+ messages in thread
From: Pavel Machek @ 2017-04-07  8:14 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Jan Kara, Eric W. Biederman, Sergey Senozhatsky, Ye Xiaolong,
	Steven Rostedt, Petr Mladek, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

[-- Attachment #1: Type: text/plain, Size: 3102 bytes --]

On Fri 2017-04-07 16:46:34, Sergey Senozhatsky wrote:
> On (04/07/17 09:15), Pavel Machek wrote:
> > On Fri 2017-04-07 13:44:40, Sergey Senozhatsky wrote:
> > > Hello,
> > > 
> > > On (04/06/17 19:33), Pavel Machek wrote:
> > > > > This patch set gives up part of the printk() reliability for bounded
> > > > > latency (at least unless we detect we are really in trouble) which is IMHO
> > > > > a good trade-off for lots of users (and others can just turn this feature
> > > > > off).
> > > > 
> > > > If they can ever realize they were bitten by this feature.
> > > > 
> > > > Can we go for different tradeoff?
> > > > 
> > > > In console_unlock(), if you detect too much work, print "Too many
> > > > messages to print, %d bytes delayed" and wake up kernel thread.
> > > 
> > > "too many messages" is undefined. console_unlock() can be called from
> > > IRQ handler or with preemtion disabled, or under spin_lock, or under
> > > RCU read lock, etc. etc. By the time we decide to wake up printk_kthread
> > > from console_unlock() it may be already too late.
> > 
> > So lets define "too many messages" as 240 characters. We know printk
> > worked rather well for us for more than 20 years. Kernel code is used
> > to printk taking few miliseconds.
> 
> serial console can be quite slow. and port->lock, that is acquired by
> console_unlock()->call_console_drivers()->write(), is also accessible
> by serial driver's IRQ handler, and this lock may be busy long
> enough -- as long as that IRQ handler transmits/receives chars. but
> that's not the point.

Well. This is what we had for 20 years.

> [..]
> > Yeah? So you know modified printk() does not work, that's why
> > "emergency mode" exists. Unfortunately, you can't rely on fact that
> > you can detect half-crashed machines by printk levels. You usually
> > can't.
> 
> I'm not happy with those printk_emergency_begin()/end(), sure. but that's
> the reality -- every single solution that would offload printing duty implies
> that there will be cases when offloading would not be possible. either
> PENDING_PRINTK_IPI to other CPUs, or irq_work(PENDING_OUTPUT) on a local CPU,
> or anything else (um... what it is?... softirq? tasklet? print one logbuf
> entry from every IRQ handler? dunno, anything else?). There will be cases
> when we won't be able to expect that something will take over and finish
> printing for us. Well, may be I'm missing some other solution that would
> offload printing, eliminating lockup conditions, and at the same time work
> in 100% of the cases.

I don't have magic solution in my sleeve. You made a good case that
spending 30 seconds in printk() is a bad idea. I agree with that. Your
solution is to introduce printk_emergency_begin()/end(). I don't agree
there.

I believe "spend at most 2 seconds in printk(), then print a warning
and offload" is a solution closer to what we had before.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-04-07  7:21       ` Pavel Machek
@ 2017-04-07  8:15         ` Sergey Senozhatsky
  2017-04-07 12:06           ` Pavel Machek
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-07  8:15 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Sergey Senozhatsky, Steven Rostedt,
	Petr Mladek, Jan Kara, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Eric Biederman,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel

Hello,

On (04/07/17 09:21), Pavel Machek wrote:
> > spin_dump() and trigger_all_cpu_backtrace() result in a bunch of
> > additional printk()-s so CPU0 has even more job to do in console_unlock(),
> > while it still holds the contended spin_lock. and so on; there are
> > many other examples.
> > 
> > so should we declare a "we can spend only 2 seconds in direct printk()
> > and then must offload printing" rule? I don't think it's much better
> > than a simpler "we always offload, as long as we think it's safe".
> 
> I believe we should do the 2 seconds rule. It allows us to print "some
> messages delayed" message, so at least whoever is trying to debug the
> crash will have the hints that he needs to look at the printk system.

do you mean panic()? in panic() we call console_flush_on_panic(),
which immediately outputs all pending logbuf messages. printk()
offloading does not happen there.


> "we always offload, as long as we think it's safe" rule does not
> really work, as printk() can not detect if it is safe or not.

but "2 seconds" rule has that "as long as we think it's safe" string
attached as well. just because we do offloading. which is sometimes
un-safe. so regardless the timeout value (0 seconds or 2 seconds) we
still need some sort of a hint from the path that issues printk()
because that path (panic, kexec, sysrq, etc.) knows for sure when
things are abnormal. printk() is pretty clueless in this regard.
/* well, I still think that EMERG loglevel thing is not completely
 broken. */

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 2/8] printk: introduce printing kernel thread
  2017-04-07  8:15         ` Sergey Senozhatsky
@ 2017-04-07 12:06           ` Pavel Machek
  0 siblings, 0 replies; 75+ messages in thread
From: Pavel Machek @ 2017-04-07 12:06 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

Hi!

> On (04/07/17 09:21), Pavel Machek wrote:
> > > spin_dump() and trigger_all_cpu_backtrace() result in a bunch of
> > > additional printk()-s so CPU0 has even more job to do in console_unlock(),
> > > while it still holds the contended spin_lock. and so on; there are
> > > many other examples.
> > > 
> > > so should we declare a "we can spend only 2 seconds in direct printk()
> > > and then must offload printing" rule? I don't think it's much better
> > > than a simpler "we always offload, as long as we think it's safe".
> > 
> > I believe we should do the 2 seconds rule. It allows us to print "some
> > messages delayed" message, so at least whoever is trying to debug the
> > crash will have the hints that he needs to look at the printk system.
> 
> do you mean panic()? in panic() we call console_flush_on_panic(),
> which immediately outputs all pending logbuf messages. printk()
> offloading does not happen there.

Not panic(). I have seen many crashes where we had printk(KERN_ERR)
and then hard hang. And the printk() was really important for debugging.

> > "we always offload, as long as we think it's safe" rule does not
> > really work, as printk() can not detect if it is safe or not.
> 
> but "2 seconds" rule has that "as long as we think it's safe" string
> attached as well. just because we do offloading. which is sometimes
> un-safe. so regardless the timeout value (0 seconds or 2 seconds) we
> still need some sort of a hint from the path that issues printk()
> because that path (panic, kexec, sysrq, etc.) knows for sure when
> things are abnormal. printk() is pretty clueless in this regard.
> /* well, I still think that EMERG loglevel thing is not completely
>  broken. */

Well, at least with my solution you know there are messages that were
not printed.

Yes, you'd still want to switch printk_now() for stuff like
panic(). But if you get it wrong (and you will), at least you will see
the "something is missing here" message in the log. 

								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07  8:14                         ` Pavel Machek
@ 2017-04-07 12:10                           ` Sergey Senozhatsky
  2017-04-07 12:44                             ` Pavel Machek
  2017-04-07 14:29                           ` Steven Rostedt
  1 sibling, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-07 12:10 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Sergey Senozhatsky, Ye Xiaolong, Steven Rostedt, Petr Mladek,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, lkp

On (04/07/17 10:14), Pavel Machek wrote:
[..]
> Well. This is what we had for 20 years.

I guess it's not just me who is a bit unhappy with printk. ask
Peter Zijlstra what's the first word that comes into his mind
when we reads "printk"  :)


[..]
> I believe "spend at most 2 seconds in printk(), then print a warning
> and offload" is a solution closer to what we had before.

a warning here can be very noisy.

it's quite common that serial console (`console_seq') is a bit behind
the logbuf head (`log_next_seq'). because log_store() can be much faster
that call into console drivers.

another case is that  printk() != console_unlock().  console_sem can be
locked by VT, TTY, fbdev, (not to mention that some other CPU might be
doing printing), etc. etc. all printk()-s in the meantime will just
log_store() messages, so we can have a bunch on pending messsges in
logbuf, it's normal. the CPU that owns the console_sem will print all
those pending messages from console_unlock() path. the distance between
`log_next_seq' and `console_seq' can be much bigger than 2 seconds or
240/320/etc chars. so wrong offloading can leave with nothing valuable
in the serial output, even if we would defer it.

well, I'm not arguing. just saying that it's not so easy to do everything
right here.

what we have been thinking about is something like printk-stall detection.
we probably (there are some if-s) can detect in printk() that offloading
does not work and we must automatically switch to printk_emergency mode.
that, in theory, can relax our dependency on printk_emergency_begin/end
being in the right place at the right time. need to think more about it.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 12:10                           ` Sergey Senozhatsky
@ 2017-04-07 12:44                             ` Pavel Machek
  2017-04-07 14:40                               ` Steven Rostedt
  2017-04-07 15:13                               ` Sergey Senozhatsky
  0 siblings, 2 replies; 75+ messages in thread
From: Pavel Machek @ 2017-04-07 12:44 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Jan Kara, Eric W. Biederman, Sergey Senozhatsky, Ye Xiaolong,
	Steven Rostedt, Petr Mladek, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

On Fri 2017-04-07 21:10:21, Sergey Senozhatsky wrote:
> On (04/07/17 10:14), Pavel Machek wrote:
> [..]
> > Well. This is what we had for 20 years.
> 
> I guess it's not just me who is a bit unhappy with printk. ask
> Peter Zijlstra what's the first word that comes into his mind
> when we reads "printk"  :)

Well, still we should make sure we are improving.

> [..]
> > I believe "spend at most 2 seconds in printk(), then print a warning
> > and offload" is a solution closer to what we had before.
> 
> a warning here can be very noisy.

Well, on normally-configured it should be ok. We don't commonly see
printk problems... If it is too noisy, perhaps we should increase from
2 seconds, but I don't think it will be problem.

> it's quite common that serial console (`console_seq') is a bit behind
> the logbuf head (`log_next_seq'). because log_store() can be much faster
> that call into console drivers.
> 
> another case is that  printk() != console_unlock().  console_sem can be
> locked by VT, TTY, fbdev, (not to mention that some other CPU might be
> doing printing), etc. etc. all printk()-s in the meantime will just
> log_store() messages, so we can have a bunch on pending messsges in
> logbuf, it's normal. the CPU that owns the console_sem will print all
> those pending messages from console_unlock() path. the distance between
> `log_next_seq' and `console_seq' can be much bigger than 2 seconds or
> 240/320/etc chars. so wrong offloading can leave with nothing valuable
> in the serial output, even if we would defer it.
> 
> well, I'm not arguing. just saying that it's not so easy to do everything
> right here.
>

Well, I have to agree here. This is 20 years worth of mess :-(.

> what we have been thinking about is something like printk-stall detection.
> we probably (there are some if-s) can detect in printk() that offloading
> does not work and we must automatically switch to printk_emergency mode.
> that, in theory, can relax our dependency on printk_emergency_begin/end
> being in the right place at the right time. need to think more about it.

So... I don't really like the begin/end interface. I would rather have
printk_emergency(KERN_ ...).

Second... I don't think "stuck detector" is that helpful. What I
usually seen was some rather innocent kernel message followed by
hard-lock. That's where "message delayed" is useful..
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07  8:14                         ` Pavel Machek
  2017-04-07 12:10                           ` Sergey Senozhatsky
@ 2017-04-07 14:29                           ` Steven Rostedt
  2017-04-09  9:57                             ` Pavel Machek
  1 sibling, 1 reply; 75+ messages in thread
From: Steven Rostedt @ 2017-04-07 14:29 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Sergey Senozhatsky, Ye Xiaolong, Petr Mladek, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

On Fri, 7 Apr 2017 10:14:49 +0200
Pavel Machek <pavel@ucw.cz> wrote:

> > serial console can be quite slow. and port->lock, that is acquired by
> > console_unlock()->call_console_drivers()->write(), is also accessible
> > by serial driver's IRQ handler, and this lock may be busy long
> > enough -- as long as that IRQ handler transmits/receives chars. but
> > that's not the point.  
> 
> Well. This is what we had for 20 years.

But for the last 20 years we were not booting on machines with over 200
CPUs. Well, we were, but those had custom kernels (which probably
dismantled printk).

-- Steve

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 12:44                             ` Pavel Machek
@ 2017-04-07 14:40                               ` Steven Rostedt
  2017-05-08  6:37                                 ` Sergey Senozhatsky
  2017-04-07 15:13                               ` Sergey Senozhatsky
  1 sibling, 1 reply; 75+ messages in thread
From: Steven Rostedt @ 2017-04-07 14:40 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Sergey Senozhatsky, Ye Xiaolong, Petr Mladek, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

On Fri, 7 Apr 2017 14:44:55 +0200
Pavel Machek <pavel@ucw.cz> wrote:

> Well, I have to agree here. This is 20 years worth of mess :-(.

Maybe someone should propose a micro-conf at Linux Plumbers where we
can brain storm a way to re-invent printk()? Seems it can do with a
completely new rewrite. ;-)

-- Steve

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 12:44                             ` Pavel Machek
  2017-04-07 14:40                               ` Steven Rostedt
@ 2017-04-07 15:13                               ` Sergey Senozhatsky
  2017-04-07 15:23                                 ` Peter Zijlstra
  2017-04-09 10:12                                 ` Pavel Machek
  1 sibling, 2 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-07 15:13 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Sergey Senozhatsky, Ye Xiaolong, Steven Rostedt, Petr Mladek,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, lkp

On (04/07/17 14:44), Pavel Machek wrote:
[..]
> > [..]
> > > I believe "spend at most 2 seconds in printk(), then print a warning
> > > and offload" is a solution closer to what we had before.
> > 
> > a warning here can be very noisy.
> 
> Well, on normally-configured it should be ok. We don't commonly see
> printk problems... If it is too noisy, perhaps we should increase from
> 2 seconds, but I don't think it will be problem.

we are looking at different typical setups :) serial console being 45
seconds behind logbuf does not surprise me anymore.

[..]
> > what we have been thinking about is something like printk-stall detection.
> > we probably (there are some if-s) can detect in printk() that offloading
> > does not work and we must automatically switch to printk_emergency mode.
> > that, in theory, can relax our dependency on printk_emergency_begin/end
> > being in the right place at the right time. need to think more about it.
> 
> So... I don't really like the begin/end interface. I would rather have
> printk_emergency(KERN_ ...).

you mean a single printk_emergency() switches printk to emergency mode
or printk_emergency(KERN_ ... ) is a single message that must be printed
in emergency mode?

printk() depends on console_trylock(). we can't expect printk_emergency(KERN_ ...)
to always do more than just log_store().

the idea behind begin/end interface is that you can do

	emergency_begin
	printk
	pr_cont
	pr_cont
	pr_cont
	printk
	dump_stack
	emergency_end

with out the need of rewriting dump_stack() or anything else to use
printk_emergency(). we, for example, do this in sysrq patch from this
series.

> Second... I don't think "stuck detector" is that helpful. What I
> usually seen was some rather innocent kernel message followed by
> hard-lock. That's where "message delayed" is useful..

a side note,
that's rather unclear to me how would "message delayed" really help.
if your system hard-lockup so badly and there are no printk messages
even from NMI watchdog, then we won't be able to print that message.
we had sort of similar type of issue years ago. cpu could receive
STOP_IPI while holding console_sem and we couldn't print anything
(that was before we learned the console_trylock();console_unlock()
trick). if you, on the other hand, can access vmcore, then you know
where to look for the messages anyway.

but let's keep it for later. this nuance is not really important now.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 15:13                               ` Sergey Senozhatsky
@ 2017-04-07 15:23                                 ` Peter Zijlstra
  2017-04-07 15:40                                   ` Sergey Senozhatsky
  2017-04-09 10:12                                 ` Pavel Machek
  1 sibling, 1 reply; 75+ messages in thread
From: Peter Zijlstra @ 2017-04-07 15:23 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Pavel Machek, Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Ye Xiaolong, Steven Rostedt, Petr Mladek, Andrew Morton,
	Linus Torvalds, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

On Sat, Apr 08, 2017 at 12:13:06AM +0900, Sergey Senozhatsky wrote:
> On (04/07/17 14:44), Pavel Machek wrote:
> [..]
> > > [..]
> > > > I believe "spend at most 2 seconds in printk(), then print a warning
> > > > and offload" is a solution closer to what we had before.
> > > 
> > > a warning here can be very noisy.
> > 
> > Well, on normally-configured it should be ok. We don't commonly see
> > printk problems... If it is too noisy, perhaps we should increase from
> > 2 seconds, but I don't think it will be problem.
> 
> we are looking at different typical setups :) serial console being 45
> seconds behind logbuf does not surprise me anymore.

That does sound like you're doing something wrong and should look at
reducing printk() more than anything else.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 15:23                                 ` Peter Zijlstra
@ 2017-04-07 15:40                                   ` Sergey Senozhatsky
  2017-04-09 18:21                                     ` Eric W. Biederman
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-07 15:40 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Sergey Senozhatsky, Pavel Machek, Sergey Senozhatsky, Jan Kara,
	Eric W. Biederman, Ye Xiaolong, Steven Rostedt, Petr Mladek,
	Andrew Morton, Linus Torvalds, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

On (04/07/17 17:23), Peter Zijlstra wrote:
[..]
> > we are looking at different typical setups :) serial console being 45
> > seconds behind logbuf does not surprise me anymore.
> 
> That does sound like you're doing something wrong and should look at
> reducing printk() more than anything else.

yeah, 45sec is an extreme case that simply doesn't surprise me anymore ;)
that's not a normal/usual delay, of course, we are not this mad. on average
it's much better and may be not so far 2 seconds after all. a massive OOM
report, of course, appends logbuf messages at a much higher rate than UART
serial console can swallow, so the delay is getting larger, expectedly.
and, no, I don't add any printk-s, I'm looking at the lockup reports

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 14:29                           ` Steven Rostedt
@ 2017-04-09  9:57                             ` Pavel Machek
  0 siblings, 0 replies; 75+ messages in thread
From: Pavel Machek @ 2017-04-09  9:57 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Sergey Senozhatsky, Ye Xiaolong, Petr Mladek, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

On Fri 2017-04-07 10:29:17, Steven Rostedt wrote:
> On Fri, 7 Apr 2017 10:14:49 +0200
> Pavel Machek <pavel@ucw.cz> wrote:
> 
> > > serial console can be quite slow. and port->lock, that is acquired by
> > > console_unlock()->call_console_drivers()->write(), is also accessible
> > > by serial driver's IRQ handler, and this lock may be busy long
> > > enough -- as long as that IRQ handler transmits/receives chars. but
> > > that's not the point.  
> > 
> > Well. This is what we had for 20 years.
> 
> But for the last 20 years we were not booting on machines with over 200
> CPUs. Well, we were, but those had custom kernels (which probably
> dismantled printk).

Well, not a problem. Just find a solution that works for dual core
machines as well as it did for 20 years.

2 seconds timeout as proposed earlier should work well for big
machines, and with no regressions on small machines.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 15:13                               ` Sergey Senozhatsky
  2017-04-07 15:23                                 ` Peter Zijlstra
@ 2017-04-09 10:12                                 ` Pavel Machek
  2017-04-10  4:53                                   ` Sergey Senozhatsky
  1 sibling, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-09 10:12 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman, Ye Xiaolong,
	Steven Rostedt, Petr Mladek, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

[-- Attachment #1: Type: text/plain, Size: 2819 bytes --]

On Sat 2017-04-08 00:13:06, Sergey Senozhatsky wrote:
> On (04/07/17 14:44), Pavel Machek wrote:
> [..]
> > > [..]
> > > > I believe "spend at most 2 seconds in printk(), then print a warning
> > > > and offload" is a solution closer to what we had before.
> > > 
> > > a warning here can be very noisy.
> > 
> > Well, on normally-configured it should be ok. We don't commonly see
> > printk problems... If it is too noisy, perhaps we should increase from
> > 2 seconds, but I don't think it will be problem.
> 
> we are looking at different typical setups :) serial console being 45
> seconds behind logbuf does not surprise me anymore.
> 
> [..]
> > > what we have been thinking about is something like printk-stall detection.
> > > we probably (there are some if-s) can detect in printk() that offloading
> > > does not work and we must automatically switch to printk_emergency mode.
> > > that, in theory, can relax our dependency on printk_emergency_begin/end
> > > being in the right place at the right time. need to think more about it.
> > 
> > So... I don't really like the begin/end interface. I would rather have
> > printk_emergency(KERN_ ...).
> 
> you mean a single printk_emergency() switches printk to emergency mode
> or printk_emergency(KERN_ ... ) is a single message that must be printed
> in emergency mode?

The latter. Having state is ugly.

> printk() depends on console_trylock(). we can't expect printk_emergency(KERN_ ...)
> to always do more than just log_store().
> 
> the idea behind begin/end interface is that you can do
> 
> 	emergency_begin
> 	printk
> 	pr_cont
> 	pr_cont
> 	pr_cont
> 	printk
> 	dump_stack
> 	emergency_end
> 
> with out the need of rewriting dump_stack() or anything else to use
> printk_emergency(). we, for example, do this in sysrq patch from this
> series.

Well.. I guess it is less work to include emergency_begin/end() but I
also believe result will state-less solution will be cleaner.

> > Second... I don't think "stuck detector" is that helpful. What I
> > usually seen was some rather innocent kernel message followed by
> > hard-lock. That's where "message delayed" is useful..
> 
> a side note,
> that's rather unclear to me how would "message delayed" really help.
> if your system hard-lockup so badly and there are no printk messages
> even from NMI watchdog, then we won't be able to print that message.

We are talking about

   printk("unusual condition");
   do_something_clever(); /* Which unfortunately hard-crashes the machine */

that works with my proposal, but not with yours. Seen it happen many
times before.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places
  2017-04-06 17:20   ` Pavel Machek
@ 2017-04-09 10:59     ` Andreas Mohr
  2017-04-10 12:20       ` Petr Mladek
  0 siblings, 1 reply; 75+ messages in thread
From: Andreas Mohr @ 2017-04-09 10:59 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Steven Rostedt, Petr Mladek, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, Sergey Senozhatsky

On Thu, Apr 06, 2017 at 07:20:52PM +0200, Pavel Machek wrote:
> On Wed 2017-03-29 18:25:07, Sergey Senozhatsky wrote:
> > It's not always possible/safe to wake_up() printk kernel
> > thread. For example, late suspend/early resume may printk()
> > while timekeeping is not initialized yet, so calling into the
> > scheduler may result in recursive warnings.
> > 
> > Another thing to notice is the fact PM at some point
> > freezes user space and kernel threads: freeze_processes()
> > and freeze_kernel_threads(), correspondingly. Thus we need
> > printk() to operate in old mode there and attempt to
> > immediately flush pending kernel message to the console.
> > 
> > This patch adds printk_emergency_begin/on sections.
> > 
> > Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> 
> I don't like this. It is symptom of printk getting much more fragile
> now.


Sergey has mentioned it already:
"at some point freezes user space and kernel threads".
Well, this is the action which is *itself* causing thoroughly disrupting consequences,
which I'd think thus ought to be responsible to
ensure *itself* that all resulting consequences actually can be dealt with properly,
rather than having
weird *completely-unrelated-dependency* crap
("there happens to be some functionality called printk, and we need to bend it,
since we need to bend it, since otherwise it would not be bent" - ahem...)
leak into ("layer violation" keyword)
pm handling implementation specifics.
IOW, I would think that for any relevant kthread use in API user code,
such code ought to be able to
register kthread-API-provided callbacks (observer pattern, or whatever)
where the (back to current case:) printk kthread would then be able to
*implicitly*/*invisibly* switch the entire printk operation interface
(e.g. via a global interface struct) to
the "dumb"/"safe" fallback variant.
Potential interface: kthread_notify(callback_func, kthread_notification_type);

That way it could (hopefully) be ensured that
people could use a consistent "printk" *interface* universally regardless of which
"special" conditions happen to be in place at the moment.
(IOW, keep interface behaviour which is required/expected at user code
definitely isolated from
awkward "implementation aspects" necessity which is currently poisoning user code implementation).


Put differently,
handling preferrably ought to get consistently adapted (i.e., switched) *centrally*,
rather than
requiring weird helpers (printk_emergency_X()) at all user code sites.

...or so goes the theory.
(quite possibly such thoughts may hit roadblocks e.g. due to locking/atomicity issues)

HTH,

Andreas Mohr

-- 
GNU/Linux. It's not the software that's free, it's you.

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 15:40                                   ` Sergey Senozhatsky
@ 2017-04-09 18:21                                     ` Eric W. Biederman
  2017-04-10  4:46                                       ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Eric W. Biederman @ 2017-04-09 18:21 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Peter Zijlstra, Pavel Machek, Sergey Senozhatsky, Jan Kara,
	Ye Xiaolong, Steven Rostedt, Petr Mladek, Andrew Morton,
	Linus Torvalds, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

Sergey Senozhatsky <sergey.senozhatsky@gmail.com> writes:

> On (04/07/17 17:23), Peter Zijlstra wrote:
> [..]
>> > we are looking at different typical setups :) serial console being 45
>> > seconds behind logbuf does not surprise me anymore.
>> 
>> That does sound like you're doing something wrong and should look at
>> reducing printk() more than anything else.
>
> yeah, 45sec is an extreme case that simply doesn't surprise me anymore ;)
> that's not a normal/usual delay, of course, we are not this mad. on average
> it's much better and may be not so far 2 seconds after all. a massive OOM
> report, of course, appends logbuf messages at a much higher rate than UART
> serial console can swallow, so the delay is getting larger, expectedly.
> and, no, I don't add any printk-s, I'm looking at the lockup reports

Are you running your serial consoles at 9600 baud?

I would think the first thing to do would be to up your serial console
baud rate to 115200 or at least 38400.

Similarly anything the kernel is certain to survive I would set loglevel
such that it is logging somewhere with syslog rather than printk.

Of course my expectation on a production machine is to have panic on oom
set, to print the huge OOM message and then reboot.  So I don't possibly
see how offloading to another thread and then switching right back to
emergency mode is at all practical to solve the delay for a serious
situation like OOM.

It sounds like you are blaming printk when the problem is a very slow
logging device.

Eric

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-09 18:21                                     ` Eric W. Biederman
@ 2017-04-10  4:46                                       ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-10  4:46 UTC (permalink / raw)
  To: Eric W. Biederman
  Cc: Sergey Senozhatsky, Peter Zijlstra, Pavel Machek,
	Sergey Senozhatsky, Jan Kara, Ye Xiaolong, Steven Rostedt,
	Petr Mladek, Andrew Morton, Linus Torvalds, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

Hello Eric,

On (04/09/17 13:21), Eric W. Biederman wrote:
[..]
> It sounds like you are blaming printk when the problem is a very slow
> logging device.

sure, slow logging device definitely adds up to the problem. if there
is no delay in call_console_driver() then printk()->console_unlock()
take no time. anything (uart, fbdev, etc.) that makes call_console_drivers()
slower makes printk() slower. the patch set is not about offloading during
panic(), when offloading make no sense, as you mentioned, or about
uncommon/extreme/impossible cases of 45sec delays in printk. no.

but about the fact that printk() called from inappropriate context
can introduce delays/timeouts/stalls and lockups. several CPUs may call
printk simultaneously, but we don't have any mechanism that would grant
console_sem ownership to a CPU in !atomic context. the winner (the CPU
that first acquires console_sem) prints it all, as long as there are
pending messages.

e.g.
 lkml.kernel.org/r/20160701165959.GR12473@ubuntu
e.g.
 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/kernel/printk/printk.c?id=8d91f8b15361dfb438ab6eb3b319e2ded43458ff

and so on.

I've even seen when printk->console_unlock() invoked from RCU read side
caused OOM condition (some RCU protected objects are small, but some are
big -- e.g. slab pages: kmem_rcu_free()). that's very rare, but I've seen
it.

so there are too many uncertainties and too many inappropriate contexts
for printk.

but yes, you are right, if there is nothing that makes call_console_driver()
slow, then there is no issue.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-09 10:12                                 ` Pavel Machek
@ 2017-04-10  4:53                                   ` Sergey Senozhatsky
  2017-04-10 11:54                                     ` Petr Mladek
  2017-04-10 18:48                                     ` Pavel Machek
  0 siblings, 2 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-10  4:53 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Sergey Senozhatsky, Jan Kara,
	Eric W. Biederman, Ye Xiaolong, Steven Rostedt, Petr Mladek,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, lkp

On (04/09/17 12:12), Pavel Machek wrote:
[..]
> > a side note,
> > that's rather unclear to me how would "message delayed" really help.
> > if your system hard-lockup so badly and there are no printk messages
> > even from NMI watchdog, then we won't be able to print that message.
> 
> We are talking about
> 
>    printk("unusual condition");
>    do_something_clever(); /* Which unfortunately hard-crashes the machine */
> 
> that works with my proposal, but not with yours. Seen it happen many
> times before.

I see your point, sure.
I can't completely agree on "that works with my proposal, but not with yours."

on SMP system this would be true only if no other CPU holds the console_sem
at the time we call printk(). (skipping irrelevant cases when we have suspended
console or !online CPU and !CON_ANYTIME console). and there is nothing that
makes "no other CPU holds the console_sem" always true on SMP system at any
given point in time. so no, "A always works, B never works" is not accurate.

but, once again, I see your point.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-10  4:53                                   ` Sergey Senozhatsky
@ 2017-04-10 11:54                                     ` Petr Mladek
  2017-04-10 15:08                                       ` Sergey Senozhatsky
  2017-04-10 18:48                                     ` Pavel Machek
  1 sibling, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-04-10 11:54 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Pavel Machek, Sergey Senozhatsky, Jan Kara, Eric W. Biederman,
	Ye Xiaolong, Steven Rostedt, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

On Mon 2017-04-10 13:53:39, Sergey Senozhatsky wrote:
> On (04/09/17 12:12), Pavel Machek wrote:
> [..]
> > > a side note,
> > > that's rather unclear to me how would "message delayed" really help.
> > > if your system hard-lockup so badly and there are no printk messages
> > > even from NMI watchdog, then we won't be able to print that message.
> > 
> > We are talking about
> > 
> >    printk("unusual condition");
> >    do_something_clever(); /* Which unfortunately hard-crashes the machine */
> > 
> > that works with my proposal, but not with yours. Seen it happen many
> > times before.
> 
> I see your point, sure.
> I can't completely agree on "that works with my proposal, but not with yours."
> 
> on SMP system this would be true only if no other CPU holds the console_sem
> at the time we call printk(). (skipping irrelevant cases when we have suspended
> console or !online CPU and !CON_ANYTIME console). and there is nothing that
> makes "no other CPU holds the console_sem" always true on SMP system at any
> given point in time. so no, "A always works, B never works" is not accurate.
> 
> but, once again, I see your point.

A compromise might be to move the offloading from vprintk_emit() to
console_unlock(). By other words, the printk could always try to
flush some messages to the console. The console might trigger
the offload (wakeup kthread) after few lines or when the printing
takes too long.

We could go even furter. We could replace the cond_resched() in
console_unlock() with a need_resched() check. Then we could avoid
sleeping with console_sem taken.

It will avoid the softlockups caused by printk(). It should
work pretty well in most critical situations.

Of course, it will not guarantee that we will see all messages when
there is a flood of messages from many CPUs. But it was never
guaranteed.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places
  2017-04-09 10:59     ` Andreas Mohr
@ 2017-04-10 12:20       ` Petr Mladek
  2017-04-10 14:38         ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-04-10 12:20 UTC (permalink / raw)
  To: Andreas Mohr
  Cc: Pavel Machek, Sergey Senozhatsky, Steven Rostedt, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, Sergey Senozhatsky

On Sun 2017-04-09 12:59:18, Andreas Mohr wrote:
> On Thu, Apr 06, 2017 at 07:20:52PM +0200, Pavel Machek wrote:
> > On Wed 2017-03-29 18:25:07, Sergey Senozhatsky wrote:
> > > It's not always possible/safe to wake_up() printk kernel
> > > thread. For example, late suspend/early resume may printk()
> > > while timekeeping is not initialized yet, so calling into the
> > > scheduler may result in recursive warnings.
> > > 
> > > Another thing to notice is the fact PM at some point
> > > freezes user space and kernel threads: freeze_processes()
> > > and freeze_kernel_threads(), correspondingly. Thus we need
> > > printk() to operate in old mode there and attempt to
> > > immediately flush pending kernel message to the console.
> > > 
> Sergey has mentioned it already:
> "at some point freezes user space and kernel threads".
> Well, this is the action which is *itself* causing thoroughly disrupting consequences,
> which I'd think thus ought to be responsible to
> ensure *itself* that all resulting consequences actually can be dealt with properly,
> rather than having
> weird *completely-unrelated-dependency* crap
> ("there happens to be some functionality called printk, and we need to bend it,
> since we need to bend it, since otherwise it would not be bent" - ahem...)
> leak into ("layer violation" keyword)
> pm handling implementation specifics.
> IOW, I would think that for any relevant kthread use in API user code,
> such code ought to be able to
> register kthread-API-provided callbacks (observer pattern, or whatever)
> where the (back to current case:) printk kthread would then be able to
> *implicitly*/*invisibly* switch the entire printk operation interface
> (e.g. via a global interface struct) to
> the "dumb"/"safe" fallback variant.
> Potential interface: kthread_notify(callback_func, kthread_notification_type);

Interesting idea. The power management area probably can be solved
by the existing notifiers framework, see register_pm_notifier().

I haven't checked it but if the notifiers are called on right
locations, it would be cleaner than adding the calls into
the pm code.


> That way it could (hopefully) be ensured that
> people could use a consistent "printk" *interface* universally regardless of which
> "special" conditions happen to be in place at the moment.
> (IOW, keep interface behaviour which is required/expected at user code
> definitely isolated from
> awkward "implementation aspects" necessity which is currently poisoning user code implementation).

> Put differently,
> handling preferrably ought to get consistently adapted (i.e., switched) *centrally*,
> rather than
> requiring weird helpers (printk_emergency_X()) at all user code sites.

Note that there already all many printk/console related "hacks"
in sensitive code paths. For example, see the use of
pm_prepare_console(), suspend_console(), console_level.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places
  2017-04-10 12:20       ` Petr Mladek
@ 2017-04-10 14:38         ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-10 14:38 UTC (permalink / raw)
  To: Andreas Mohr, Petr Mladek
  Cc: Pavel Machek, Sergey Senozhatsky, Steven Rostedt, Jan Kara,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Eric Biederman, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, Sergey Senozhatsky

On (04/10/17 14:20), Petr Mladek wrote:
[..]
> > Sergey has mentioned it already:
> > "at some point freezes user space and kernel threads".
> > Well, this is the action which is *itself* causing thoroughly disrupting consequences,
> > which I'd think thus ought to be responsible to
> > ensure *itself* that all resulting consequences actually can be dealt with properly,
> > rather than having
> > weird *completely-unrelated-dependency* crap
> > ("there happens to be some functionality called printk, and we need to bend it,
> > since we need to bend it, since otherwise it would not be bent" - ahem...)
> > leak into ("layer violation" keyword)
> > pm handling implementation specifics.
> > IOW, I would think that for any relevant kthread use in API user code,
> > such code ought to be able to
> > register kthread-API-provided callbacks (observer pattern, or whatever)
> > where the (back to current case:) printk kthread would then be able to
> > *implicitly*/*invisibly* switch the entire printk operation interface
> > (e.g. via a global interface struct) to
> > the "dumb"/"safe" fallback variant.
> > Potential interface: kthread_notify(callback_func, kthread_notification_type);
> 
> Interesting idea. The power management area probably can be solved
> by the existing notifiers framework.

good idea indeed.

wish we also had kexec and sysrq notifiers :) there is a
`panic_notifier_list', but that's not exactly what we need.

[..]
> > Put differently,
> > handling preferrably ought to get consistently adapted (i.e., switched) *centrally*,
> > rather than
> > requiring weird helpers (printk_emergency_X()) at all user code sites.
> 
> Note that there already all many printk/console related "hacks"
> in sensitive code paths. For example, see the use of
> pm_prepare_console(), suspend_console(), console_level.

yep. I wonder if some of those can be moved to printk pm notifiers.
but that's out of the scope of this patch set.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-10 11:54                                     ` Petr Mladek
@ 2017-04-10 15:08                                       ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-10 15:08 UTC (permalink / raw)
  To: Petr Mladek, Pavel Machek
  Cc: Sergey Senozhatsky, Sergey Senozhatsky, Jan Kara,
	Eric W. Biederman, Ye Xiaolong, Steven Rostedt, Andrew Morton,
	Linus Torvalds, Peter Zijlstra, Rafael J . Wysocki,
	Greg Kroah-Hartman, Jiri Slaby, Len Brown, linux-kernel, lkp

On (04/10/17 13:54), Petr Mladek wrote:
[..]
> > > that works with my proposal, but not with yours. Seen it happen many
> > > times before.
> > 
> > I see your point, sure.
> > I can't completely agree on "that works with my proposal, but not with yours."
> > 
> > on SMP system this would be true only if no other CPU holds the console_sem
> > at the time we call printk(). (skipping irrelevant cases when we have suspended
> > console or !online CPU and !CON_ANYTIME console). and there is nothing that
> > makes "no other CPU holds the console_sem" always true on SMP system at any
> > given point in time. so no, "A always works, B never works" is not accurate.
> > 
> > but, once again, I see your point.
> 
> A compromise might be to move the offloading from vprintk_emit() to
> console_unlock(). By other words, the printk could always try to
> flush some messages to the console. The console might trigger
> the offload (wakeup kthread) after few lines

yep, that's the proposal.


hm, this also should align with one more thing.

we briefly discussed it before, and it was on my list, that
wake_up(printk_kthread) _eventually_ better be moved to console_unlock()
[1] (I also had it in the slides at KS, but I believe we didn't have much
time back then).

vprintk_emit() is not the only console_lock() caller. user space does
console_lock() and console_unlock() calls, and in some cases a user
space process can stuck in system call printing kernel messages to a
potentially slow console [2]. it can be unpleasant, but far less dramatic
than doing console_unlock() from IRQ, or under spin_lock. so it was
moved down the list. seems we have one more reason to reshuffle the
list and do offloading from console_unlock() from the beginning.

will take a look.

/* in our "in-house" kernels we do 'async' console_unlock(). not
exactly the way it's shown in [1], but quite similar. */

[1] https://marc.info/?l=linux-kernel&m=145750373530161
[2] https://marc.info/?l=linux-kernel&m=145762735308470

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-10  4:53                                   ` Sergey Senozhatsky
  2017-04-10 11:54                                     ` Petr Mladek
@ 2017-04-10 18:48                                     ` Pavel Machek
  2017-04-11  1:46                                       ` Sergey Senozhatsky
  1 sibling, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-10 18:48 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Sergey Senozhatsky, Jan Kara, Eric W. Biederman, Ye Xiaolong,
	Steven Rostedt, Petr Mladek, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

On Mon 2017-04-10 13:53:39, Sergey Senozhatsky wrote:
> On (04/09/17 12:12), Pavel Machek wrote:
> [..]
> > > a side note,
> > > that's rather unclear to me how would "message delayed" really help.
> > > if your system hard-lockup so badly and there are no printk messages
> > > even from NMI watchdog, then we won't be able to print that message.
> > 
> > We are talking about
> > 
> >    printk("unusual condition");
> >    do_something_clever(); /* Which unfortunately hard-crashes the machine */
> > 
> > that works with my proposal, but not with yours. Seen it happen many
> > times before.
> 
> I see your point, sure.
> I can't completely agree on "that works with my proposal, but not with yours."
> 
> on SMP system this would be true only if no other CPU holds the console_sem
> at the time we call printk(). (skipping irrelevant cases when we have suspended
> console or !online CPU and !CON_ANYTIME console). and there is nothing that
> makes "no other CPU holds the console_sem" always true on SMP system at any
> given point in time. so no, "A always works, B never works" is not
> accurate.

Ok, you are right. OTOH the common case is console_sem is unlocked (at
least on systems I develop on). 

> but, once again, I see your point.

Good. Does that mean that the next version of patches will work ok in
that case?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-10 18:48                                     ` Pavel Machek
@ 2017-04-11  1:46                                       ` Sergey Senozhatsky
  2017-04-11 16:19                                         ` Sergey Senozhatsky
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-11  1:46 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Sergey Senozhatsky, Jan Kara,
	Eric W. Biederman, Ye Xiaolong, Steven Rostedt, Petr Mladek,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, lkp

On (04/10/17 20:48), Pavel Machek wrote:
[..]
> > but, once again, I see your point.
> 
> Good. Does that mean that the next version of patches will work ok in
> that case?

yes.

we also likely will get rid of printk_begin/end in PM code.
but we still need to have printk_emergency hints from the
outside world in the rest of the places (sysrq etc.)    :(

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-11  1:46                                       ` Sergey Senozhatsky
@ 2017-04-11 16:19                                         ` Sergey Senozhatsky
  2017-04-12 18:43                                           ` Pavel Machek
                                                             ` (2 more replies)
  0 siblings, 3 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-11 16:19 UTC (permalink / raw)
  To: Petr Mladek, Pavel Machek
  Cc: Jan Kara, Eric W. Biederman, Ye Xiaolong, Steven Rostedt,
	Petr Mladek, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, Sergey Senozhatsky, Sergey Senozhatsky

On (04/11/17 10:46), Sergey Senozhatsky wrote:
> On (04/10/17 20:48), Pavel Machek wrote:
> [..]
> > > but, once again, I see your point.
> > 
> > Good. Does that mean that the next version of patches will work ok in
> > that case?
> 
> yes.

ok... so I'm looking at something like below right now.
not really tested yet.

I put some comments into the code.

it does offloading after X printed lines by the same process.
if we reschedule, then the counter resets. which is probably OK,
we don't really want any process, except for printk_kthread, to
stay in console_unlock() forever. "number of lines printed" is
probably easier to understand (easily converted to the number of
pageup/pagedown you need to press, terminal buffer history size,
etc.) than seconds we spent on printing (which doesn't even
correspond to messages' timestamps in general case).

when the limit of "number of lines printed" is 0, then no
offloading takes place.

it also has some simple mechanism to handle cases when
we try to wake_up() printk_kthread, but it never becomes alive.
it's a bit simple minded, probably.

wake_up is done from printk_safe context, so warnings/printks
from there should do no harm (in fact, we even do pr_crit()
error reporting, when we enforce printk_emergency mode).

I'll do more tests tomorrow, and will take a closer look.
this code is basically just compiles, boots and passes some
trivial tests. quite possible I've missed something important.

once verified, then the next question will be -- do we even
need printk_emergency_begin/end or we can leave without it.

// given that printk_emergency enforcement works properly


but, once again, the code might be stupid and wrong.
and I need some sleep.

---
 include/linux/console.h |   3 +
 kernel/printk/printk.c  | 200 ++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 198 insertions(+), 5 deletions(-)

diff --git a/include/linux/console.h b/include/linux/console.h
index 5949d1855589..f1a86944072e 100644
--- a/include/linux/console.h
+++ b/include/linux/console.h
@@ -187,6 +187,9 @@ extern bool console_suspend_enabled;
 extern void suspend_console(void);
 extern void resume_console(void);
 
+extern void printk_emergency_begin(void);
+extern void printk_emergency_end(void);
+
 int mda_console_init(void);
 void prom_con_init(void);
 
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 6cf756dbee39..c0075e8b3a09 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -48,6 +48,7 @@
 #include <linux/sched/clock.h>
 #include <linux/sched/debug.h>
 #include <linux/sched/task_stack.h>
+#include <linux/kthread.h>
 
 #include <linux/uaccess.h>
 #include <asm/sections.h>
@@ -402,7 +403,8 @@ DEFINE_RAW_SPINLOCK(logbuf_lock);
 	} while (0)
 
 /*
- * Delayed printk version, for scheduler-internal messages:
+ * Used both for deferred printk version (scheduler-internal messages)
+ * and printk_kthread control.
  */
 #define PRINTK_PENDING_WAKEUP	0x01
 #define PRINTK_PENDING_OUTPUT	0x02
@@ -445,6 +447,48 @@ static char __log_buf[__LOG_BUF_LEN] __aligned(LOG_ALIGN);
 static char *log_buf = __log_buf;
 static u32 log_buf_len = __LOG_BUF_LEN;
 
+static struct task_struct *printk_kthread __read_mostly;
+/*
+ * We can't call into the scheduler (wake_up() printk kthread) during
+ * suspend/kexec/etc. This temporarily switches printk to old behaviour.
+ */
+static atomic_t printk_emergency __read_mostly;
+/*
+ * Disable printk_kthread permanently. Unlike `oops_in_progress'
+ * it doesn't go back to 0.
+ */
+static bool printk_enforce_emergency __read_mostly;
+
+static unsigned int atomic_print_limit = 10000;
+
+module_param_named(atomic_print_limit, atomic_print_limit, uint, 0644);
+MODULE_PARM_DESC(atomic_print_limit,
+		 "max lines to print before offloading to printk kthread");
+
+static inline bool printk_kthread_enabled(void)
+{
+	return !printk_enforce_emergency &&
+		printk_kthread && atomic_read(&printk_emergency) == 0;
+}
+
+/*
+ * This disables printing offloading and instead attempts
+ * to do the usual console_trylock()->console_unlock().
+ *
+ * Note, this does not stop the printk_kthread if it's already
+ * printing logbuf messages.
+ */
+void printk_emergency_begin(void)
+{
+	atomic_inc(&printk_emergency);
+}
+
+/* This re-enables printk_kthread offloading. */
+void printk_emergency_end(void)
+{
+	atomic_dec(&printk_emergency);
+}
+
 /* Return log buffer address */
 char *log_buf_addr_get(void)
 {
@@ -1765,6 +1809,15 @@ asmlinkage int vprintk_emit(int facility, int level,
 
 	printed_len += log_output(facility, level, lflags, dict, dictlen, text, text_len);
 
+	/*
+	 * Emergency level indicates that the system is unstable and, thus,
+	 * we better stop relying on wake_up(printk_kthread) and try to do
+	 * a direct printing.
+	 */
+	if (level == LOGLEVEL_EMERG)
+		printk_enforce_emergency = true;
+
+	set_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 	logbuf_unlock_irqrestore(flags);
 
 	/* If called from the scheduler, we can not call up(). */
@@ -1882,6 +1935,9 @@ static size_t msg_print_text(const struct printk_log *msg,
 			     bool syslog, char *buf, size_t size) { return 0; }
 static bool suppress_message_printing(int level) { return false; }
 
+void printk_emergency_begin(void) {}
+void printk_emergency_end(void) {}
+
 #endif /* CONFIG_PRINTK */
 
 #ifdef CONFIG_EARLY_PRINTK
@@ -2141,6 +2197,89 @@ static inline int can_use_console(void)
 	return cpu_online(raw_smp_processor_id()) || have_callable_console();
 }
 
+/*
+ * Under heavy printing load/slow serial console/etc console_unlock() can
+ * stall CPUs, which can result in soft/hard-lockups, lost interrupts, RCU
+ * stalls, etc. Therefore we attempt to print the messages to console from
+ * a dedicated printk_kthread, which always runs in schedulable context.
+ *
+ * There are several possible scenarios:
+ *
+ * a) When we got a large number of pending messages to print.
+ * e.g.
+ *    vprintk_emit() or console_lock()
+ *     console_unlock()
+ *      <<massive dump>>
+ *
+ * b) When we printk() a large number of messages.
+ * e.g.
+ *   vprintk_emit()->console_unlock()   <<print 1 message>>
+ *   vprintk_emit()->console_unlock()   <<print 1 message>>
+ *   ...
+ *   vprintk_emit()->console_unlock()   <<print 1 message>>
+ *
+ * In all those cases we can be in atomic context, we need to offload
+ * printing at some point.
+ *
+ * This function must be called from 'printk_safe' context.
+ */
+static inline bool console_offload_printing(void)
+{
+	static struct task_struct *printing_task = NULL;
+	static unsigned long lines_printed = 0;
+
+	if (!atomic_print_limit || !printk_kthread_enabled())
+		return false;
+
+	/* We rescheduled - reset the counters. */
+	if (printing_task != current) {
+		lines_printed = 0;
+		printing_task = current;
+		return false;
+	}
+
+	if (current == printk_kthread)
+		return false;
+
+	/*
+	 * Don't reset the counter, let CPU overrun the limit.
+	 * The idea is that
+	 *
+	 *   a) woken up printk_kthread (if succeeded)
+	 * or
+	 *   b) concurrent printk from another CPU (if any)
+	 *
+	 * will change `printing_task' and reset the counter. This also
+	 * let us to introduce additional policies later, for instance,
+	 * if we can't wakeup printk_kthread for Y times, e.g.
+	 *
+	 *            lines_printed > 2 * atomic_print_limit
+	 *
+	 * then we can declare emergency and stop relying on printk_kthread.
+	 *
+	 * If neither a) nor b) happens - we continue printing from
+	 * current process. Which is bad and can be risky, but we can't
+	 * wake_up() printk_kthread, so things already don't look normal.
+	 */
+	lines_printed++;
+	if (lines_printed < atomic_print_limit)
+		return false;
+
+	/*
+	 * A trivial emergency enforcement.
+	 * Assumes that `atomic_print_limit' is large enough.
+	 */
+	if (lines_printed > 2 * (unsigned long)atomic_print_limit) {
+		printk_enforce_emergency = true;
+		pr_crit("Declaring printk emergency mode.\n");
+		return false;
+	}
+
+	/* Must be executed in 'printk_safe' context. */
+	wake_up_process(printk_kthread);
+	return true;
+}
+
 /**
  * console_unlock - unlock the console system
  *
@@ -2163,8 +2302,17 @@ void console_unlock(void)
 	unsigned long flags;
 	bool wake_klogd = false;
 	bool do_cond_resched, retry;
+	bool did_offload;
 
 	if (console_suspended) {
+		/*
+		 * Here and later, we need to clear the PENDING_OUTPUT bit
+		 * in order to avoid an infinite loop in printk_kthread
+		 * function when console_unlock() cannot flush messages
+		 * because we suspended consoles. Someone else will print
+		 * the messages from resume_console().
+		 */
+		clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 		up_console_sem();
 		return;
 	}
@@ -2186,6 +2334,7 @@ void console_unlock(void)
 	do_cond_resched = console_may_schedule;
 again:
 	console_may_schedule = 0;
+	did_offload = 0;
 
 	/*
 	 * We released the console_sem lock, so we need to recheck if
@@ -2193,6 +2342,7 @@ void console_unlock(void)
 	 * console.
 	 */
 	if (!can_use_console()) {
+		clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 		console_locked = 0;
 		up_console_sem();
 		return;
@@ -2221,7 +2371,7 @@ void console_unlock(void)
 			len = 0;
 		}
 skip:
-		if (console_seq == log_next_seq)
+		if (did_offload || console_seq == log_next_seq)
 			break;
 
 		msg = log_from_idx(console_idx);
@@ -2253,9 +2403,28 @@ void console_unlock(void)
 		stop_critical_timings();	/* don't trace print latency */
 		call_console_drivers(ext_text, ext_len, text, len);
 		start_critical_timings();
+
+		/*
+		 * Sometimes we may lock console_sem before printk_kthread.
+		 * In this case we will jump to `again' label (if there are
+		 * pending messages), print one more line from current
+		 * process, break out of printing loop (we don't reset the
+		 * counter of printed lines) and do up_console_sem() to
+		 * wakeup printk_kthread again.
+		 *
+		 * If printk_kthread never wakes up (which may indicate that
+		 * the system is unstable or something weird is going on),
+		 * then we will keep jumping to `again' label and printing
+		 * one message from the logbuf. This is a bit ugly, but at
+		 * least we will print out the logbuf.
+		 *
+		 * If such condition occurs, console_offload_printing() can
+		 * declare `printk_emergency' at some point.
+		 */
+		did_offload = console_offload_printing();
 		printk_safe_exit_irqrestore(flags);
 
-		if (do_cond_resched)
+		if (!did_offload && do_cond_resched)
 			cond_resched();
 	}
 	console_locked = 0;
@@ -2276,6 +2445,8 @@ void console_unlock(void)
 	 */
 	raw_spin_lock(&logbuf_lock);
 	retry = console_seq != log_next_seq;
+	if (!retry)
+		clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
 	raw_spin_unlock(&logbuf_lock);
 	printk_safe_exit_irqrestore(flags);
 
@@ -2669,8 +2840,11 @@ late_initcall(printk_late_init);
 #if defined CONFIG_PRINTK
 static void wake_up_klogd_work_func(struct irq_work *irq_work)
 {
-	if (test_and_clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending)) {
-		/* If trylock fails, someone else is doing the printing */
+	if (test_bit(PRINTK_PENDING_OUTPUT, &printk_pending)) {
+		/*
+		 * If trylock fails, someone else is doing the printing.
+		 * PRINTK_PENDING_OUTPUT bit is cleared by console_unlock().
+		 */
 		if (console_trylock())
 			console_unlock();
 	}
@@ -2684,6 +2858,22 @@ static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) = {
 	.flags = IRQ_WORK_LAZY,
 };
 
+static int printk_kthread_func(void *data)
+{
+	while (1) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		if (!test_bit(PRINTK_PENDING_OUTPUT, &printk_pending))
+			schedule();
+
+		__set_current_state(TASK_RUNNING);
+
+		console_lock();
+		console_unlock();
+	}
+
+	return 0;
+}
+
 void wake_up_klogd(void)
 {
 	preempt_disable();
-- 
2.12.2

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-11 16:19                                         ` Sergey Senozhatsky
@ 2017-04-12 18:43                                           ` Pavel Machek
  2017-04-13  4:34                                             ` Sergey Senozhatsky
  2017-04-13  5:50                                           ` Sergey Senozhatsky
  2017-04-13 14:03                                           ` Petr Mladek
  2 siblings, 1 reply; 75+ messages in thread
From: Pavel Machek @ 2017-04-12 18:43 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Petr Mladek, Jan Kara, Eric W. Biederman, Ye Xiaolong,
	Steven Rostedt, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, Sergey Senozhatsky

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

On Wed 2017-04-12 01:19:53, Sergey Senozhatsky wrote:
> On (04/11/17 10:46), Sergey Senozhatsky wrote:
> > On (04/10/17 20:48), Pavel Machek wrote:
> > [..]
> > > > but, once again, I see your point.
> > > 
> > > Good. Does that mean that the next version of patches will work ok in
> > > that case?
> > 
> > yes.
> 
> ok... so I'm looking at something like below right now.
> not really tested yet.
> 
> I put some comments into the code.
> 
> it does offloading after X printed lines by the same process.
> if we reschedule, then the counter resets. which is probably OK,
> we don't really want any process, except for printk_kthread, to
> stay in console_unlock() forever. "number of lines printed" is
> probably easier to understand (easily converted to the number of
> pageup/pagedown you need to press, terminal buffer history size,
> etc.) than seconds we spent on printing (which doesn't even
> correspond to messages' timestamps in general case).

Design looks good to me... certainly better than previous version :-).
								

> when the limit of "number of lines printed" is 0, then no
> offloading takes place.

And with "number of lines printed" set to 999999, it will get us
previous behaviour, right? 

Thanks,
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-12 18:43                                           ` Pavel Machek
@ 2017-04-13  4:34                                             ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-13  4:34 UTC (permalink / raw)
  To: Pavel Machek
  Cc: Sergey Senozhatsky, Petr Mladek, Jan Kara, Eric W. Biederman,
	Ye Xiaolong, Steven Rostedt, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, Sergey Senozhatsky

On (04/12/17 20:43), Pavel Machek wrote:
[..]
> > when the limit of "number of lines printed" is 0, then no
> > offloading takes place.
> 
> And with "number of lines printed" set to 999999, it will get us
> previous behaviour, right? 

`atomic_print_limit' set to zero disables offloading explicitly.
at the same time, an unreasonably high `atomic_print_limit' value
makes offloading less possible and starting from some value,
basically, disables it implicitly.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-11 16:19                                         ` Sergey Senozhatsky
  2017-04-12 18:43                                           ` Pavel Machek
@ 2017-04-13  5:50                                           ` Sergey Senozhatsky
  2017-04-13  8:19                                             ` Sergey Senozhatsky
  2017-04-13 14:03                                           ` Petr Mladek
  2 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-13  5:50 UTC (permalink / raw)
  To: Petr Mladek, Pavel Machek
  Cc: Jan Kara, Eric W. Biederman, Ye Xiaolong, Steven Rostedt,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, Sergey Senozhatsky, Sergey Senozhatsky

On (04/12/17 01:19), Sergey Senozhatsky wrote:
[..]
> it does offloading after X printed lines by the same process.
> if we reschedule, then the counter resets. which is probably OK,
> we don't really want any process, except for printk_kthread, to
> stay in console_unlock() forever.

may be this can be changed. we don't want even printk_kthread to keep
console_sem locked for too long, because other process that might want
to lock console_sem have to sleep in TASK_UNINTERRUPTIBLE as long as
printing thread has pending messages to print. so may be the rule can
be "every process prints up to `atomic_print_limit' lines and then
offloads printing - wake_up()s printk_kthread and up()s console_sem".
some other process (printk_kthread or a process from console_sem wait
list, let them compete for console_sem) will eventually down()
console_sem and print the next `atomic_print_limit' lines, while
current process will have a chance to return from console_unlock() and
do something else.

[..]
> the next question will be -- do we even need printk_emergency_begin/end
> or we can leave without it.

what I meant here -- drop sysrq and kexec printk_emergency_begin/end
patches, but keep printk_emergency_begin/end API and do
printk_emergency_begin/end in console_suspend()/resume().
PM already calls console_suspend()/resume(). something like that...

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-13  5:50                                           ` Sergey Senozhatsky
@ 2017-04-13  8:19                                             ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-13  8:19 UTC (permalink / raw)
  To: Petr Mladek, Pavel Machek
  Cc: Jan Kara, Eric W. Biederman, Ye Xiaolong, Steven Rostedt,
	Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, Sergey Senozhatsky, Sergey Senozhatsky

On (04/13/17 14:50), Sergey Senozhatsky wrote:
[..]
> On (04/12/17 01:19), Sergey Senozhatsky wrote:
> [..]
> > it does offloading after X printed lines by the same process.
> > if we reschedule, then the counter resets. which is probably OK,
> > we don't really want any process, except for printk_kthread, to
> > stay in console_unlock() forever.
> 
> may be this can be changed. we don't want even printk_kthread to keep
> console_sem locked for too long, because other process that might want
> to lock console_sem have to sleep in TASK_UNINTERRUPTIBLE as long as
> printing thread has pending messages to print. so may be the rule can
> be "every process prints up to `atomic_print_limit' lines and then
> offloads printing - wake_up()s printk_kthread and up()s console_sem".
> some other process (printk_kthread or a process from console_sem wait
> list, let them compete for console_sem) will eventually down()
> console_sem and print the next `atomic_print_limit' lines, while
> current process will have a chance to return from console_unlock() and
> do something else.

something like this, perhaps.

static inline bool console_offload_printing(void)
{
	static struct task_struct *printing_task;
	static unsigned long lines_printed;
	static bool did_wakeup;

	if (!atomic_print_limit || !printk_kthread_enabled())
		return false;

	/* We rescheduled - reset the counters. */
	if (printing_task != current) {
		did_wakeup = false;
		lines_printed = 0;
		printing_task = current;
		return false;
	}

	/*
	 * Don't reset the counter, let CPU overrun the limit.
	 * The idea is that
	 *
	 *   a) woken up printk_kthread (if succeeded)
	 * or
	 *   b) concurrent printk from another CPU (if any)
	 *
	 * will change `printing_task' and reset the counter.
	 * If neither a) nor b) happens - we continue printing from
	 * current process. Which is bad and can be risky, but we can't
	 * wake_up() printk_kthread, so things already don't look normal.
	 */
	lines_printed++;
	if (lines_printed < atomic_print_limit)
		return false;

	if (current == printk_kthread) {
		/*
		 * Reset the counter, just in case if printk_kthread is the
		 * only process left that would down() console_sem.
		 */
		lines_printed = 0;
		return true;
	}

	/*
	 * A trivial emergency enforcement - give up on printk_kthread if
	 * we can't wake it up. This assumes that `atomic_print_limit' is
	 * large enough.
	 */
	if (lines_printed > 2 * (unsigned long long)atomic_print_limit) {
		printk_enforce_emergency = true;
		pr_crit("Declaring printk emergency mode.\n");
		return false;
	}

	/*
	 * Must be executed in 'printk_safe' context. Call into the
	 * scheduler just once, in case if it backfires on us with
	 * warnings and backtraces.
	 */
	if (!did_wakeup) {
		did_wakeup = true;
		wake_up_process(printk_kthread);
	}
	return true;
}

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-11 16:19                                         ` Sergey Senozhatsky
  2017-04-12 18:43                                           ` Pavel Machek
  2017-04-13  5:50                                           ` Sergey Senozhatsky
@ 2017-04-13 14:03                                           ` Petr Mladek
  2017-04-14  4:42                                             ` Sergey Senozhatsky
  2 siblings, 1 reply; 75+ messages in thread
From: Petr Mladek @ 2017-04-13 14:03 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Pavel Machek, Jan Kara, Eric W. Biederman, Ye Xiaolong,
	Steven Rostedt, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, Sergey Senozhatsky

On Wed 2017-04-12 01:19:53, Sergey Senozhatsky wrote:
> On (04/11/17 10:46), Sergey Senozhatsky wrote:
> > On (04/10/17 20:48), Pavel Machek wrote:
> > [..]
> > > > but, once again, I see your point.
> > > 
> > > Good. Does that mean that the next version of patches will work ok in
> > > that case?
> > 
> > yes.
> 
> ok... so I'm looking at something like below right now.
> not really tested yet.
> 
> I put some comments into the code.
> 
> it does offloading after X printed lines by the same process.
> if we reschedule, then the counter resets. which is probably OK,
> we don't really want any process, except for printk_kthread, to
> stay in console_unlock() forever. "number of lines printed" is
> probably easier to understand (easily converted to the number of
> pageup/pagedown you need to press, terminal buffer history size,
> etc.) than seconds we spent on printing (which doesn't even
> correspond to messages' timestamps in general case).
> 
> when the limit of "number of lines printed" is 0, then no
> offloading takes place.
> 
> it also has some simple mechanism to handle cases when
> we try to wake_up() printk_kthread, but it never becomes alive.
> it's a bit simple minded, probably.
> 
> wake_up is done from printk_safe context, so warnings/printks
> from there should do no harm (in fact, we even do pr_crit()
> error reporting, when we enforce printk_emergency mode).
> 
> I'll do more tests tomorrow, and will take a closer look.
> this code is basically just compiles, boots and passes some
> trivial tests. quite possible I've missed something important.
> 
> once verified, then the next question will be -- do we even
> need printk_emergency_begin/end or we can leave without it.
> 
> // given that printk_emergency enforcement works properly
> 
> 
> but, once again, the code might be stupid and wrong.
> and I need some sleep.
> 
> ---
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 6cf756dbee39..c0075e8b3a09 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -2141,6 +2197,89 @@ static inline int can_use_console(void)
>  	return cpu_online(raw_smp_processor_id()) || have_callable_console();
>  }
>  
> +/*
> + * Under heavy printing load/slow serial console/etc console_unlock() can
> + * stall CPUs, which can result in soft/hard-lockups, lost interrupts, RCU
> + * stalls, etc. Therefore we attempt to print the messages to console from
> + * a dedicated printk_kthread, which always runs in schedulable context.
> + *
> + * There are several possible scenarios:
> + *
> + * a) When we got a large number of pending messages to print.
> + * e.g.
> + *    vprintk_emit() or console_lock()
> + *     console_unlock()
> + *      <<massive dump>>
> + *
> + * b) When we printk() a large number of messages.
> + * e.g.
> + *   vprintk_emit()->console_unlock()   <<print 1 message>>
> + *   vprintk_emit()->console_unlock()   <<print 1 message>>
> + *   ...
> + *   vprintk_emit()->console_unlock()   <<print 1 message>>
> + *
> + * In all those cases we can be in atomic context, we need to offload
> + * printing at some point.
> + *
> + * This function must be called from 'printk_safe' context.
> + */
> +static inline bool console_offload_printing(void)
> +{
> +	static struct task_struct *printing_task = NULL;
> +	static unsigned long lines_printed = 0;
> +
> +	if (!atomic_print_limit || !printk_kthread_enabled())
> +		return false;
> +
> +	/* We rescheduled - reset the counters. */
> +	if (printing_task != current) {
> +		lines_printed = 0;
> +		printing_task = current;
> +		return false;
> +	}

If we want to check that the process rescheduled, we should
store/check also current->nvcsw + current->nivcsw.

This might be even more important in the detection of
the emergency situation, see below.


> +	if (current == printk_kthread)
> +		return false;

Yup, printk_kthread is special. You suggest another solution
in the other reply.

IMHO, the best solution would be if printk_kthread calls
console_unlock() with disabled preemption and does the offload
(releases console_sem) when need_resched() returns true.

By other words, printk_kthread should use its allotted time
as much as possible. But it should not block the console_sem
when sleeping.

My only fear is that it is getting more and more complicated.
On the other hand, any partial solution is asking for
troubles and complains.

> +
> +	/*
> +	 * Don't reset the counter, let CPU overrun the limit.
> +	 * The idea is that
> +	 *
> +	 *   a) woken up printk_kthread (if succeeded)
> +	 * or
> +	 *   b) concurrent printk from another CPU (if any)
> +	 *
> +	 * will change `printing_task' and reset the counter. This also
> +	 * let us to introduce additional policies later, for instance,
> +	 * if we can't wakeup printk_kthread for Y times, e.g.
> +	 *
> +	 *            lines_printed > 2 * atomic_print_limit
> +	 *
> +	 * then we can declare emergency and stop relying on printk_kthread.
> +	 *
> +	 * If neither a) nor b) happens - we continue printing from
> +	 * current process. Which is bad and can be risky, but we can't
> +	 * wake_up() printk_kthread, so things already don't look normal.
> +	 */
> +	lines_printed++;
> +	if (lines_printed < atomic_print_limit)
> +		return false;
> +
> +	/*
> +	 * A trivial emergency enforcement.
> +	 * Assumes that `atomic_print_limit' is large enough.
> +	 */
> +	if (lines_printed > 2 * (unsigned long)atomic_print_limit) {
> +		printk_enforce_emergency = true;
> +		pr_crit("Declaring printk emergency mode.\n");
> +		return false;
> +	}

The only messages that are printed on my workstation are the same
few lines everytime I connect my phone over USB to get it charged.
I am not sure if they are printed by the same process. But I would
get scared if printk switches to the emergency mode just because
there is only one process "regularly" producing messages.

It might help to check the number of process switch counts as
suggested above.


> +	/* Must be executed in 'printk_safe' context. */
> +	wake_up_process(printk_kthread);
> +	return true;
> +}
> +
>  /**
>   * console_unlock - unlock the console system
>   *
> @@ -2163,8 +2302,17 @@ void console_unlock(void)
>  	unsigned long flags;
>  	bool wake_klogd = false;
>  	bool do_cond_resched, retry;
> +	bool did_offload;
>  
>  	if (console_suspended) {
> +		/*
> +		 * Here and later, we need to clear the PENDING_OUTPUT bit
> +		 * in order to avoid an infinite loop in printk_kthread
> +		 * function when console_unlock() cannot flush messages
> +		 * because we suspended consoles. Someone else will print
> +		 * the messages from resume_console().
> +		 */
> +		clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
>  		up_console_sem();
>  		return;
>  	}
> @@ -2186,6 +2334,7 @@ void console_unlock(void)
>  	do_cond_resched = console_may_schedule;
>  again:
>  	console_may_schedule = 0;
> +	did_offload = 0;

It would make more sense to clear the variable before the again:
target. In fact, there is a logic mistake, see below.
>  
>  	/*
>  	 * We released the console_sem lock, so we need to recheck if
> @@ -2193,6 +2342,7 @@ void console_unlock(void)
>  	 * console.
>  	 */
>  	if (!can_use_console()) {
> +		clear_bit(PRINTK_PENDING_OUTPUT, &printk_pending);
>  		console_locked = 0;
>  		up_console_sem();
>  		return;
> @@ -2221,7 +2371,7 @@ void console_unlock(void)
>  			len = 0;
>  		}
>  skip:
> -		if (console_seq == log_next_seq)
> +		if (did_offload || console_seq == log_next_seq)
>  			break;
>  
>  		msg = log_from_idx(console_idx);
> @@ -2253,9 +2403,28 @@ void console_unlock(void)
>  		stop_critical_timings();	/* don't trace print latency */
>  		call_console_drivers(ext_text, ext_len, text, len);
>  		start_critical_timings();
> +
> +		/*
> +		 * Sometimes we may lock console_sem before printk_kthread.
> +		 * In this case we will jump to `again' label (if there are
> +		 * pending messages), print one more line from current
> +		 * process, break out of printing loop (we don't reset the
> +		 * counter of printed lines) and do up_console_sem() to
> +		 * wakeup printk_kthread again.
> +		 *
> +		 * If printk_kthread never wakes up (which may indicate that
> +		 * the system is unstable or something weird is going on),
> +		 * then we will keep jumping to `again' label and printing
> +		 * one message from the logbuf. This is a bit ugly, but at
> +		 * least we will print out the logbuf.
> +		 *
> +		 * If such condition occurs, console_offload_printing() can
> +		 * declare `printk_emergency' at some point.

I am a bit confused by the comment above. The again goto target is
used only when there is a race between leaving the loop and releasing
the console_sem. It is a corner case.

If there are messages from other CPUs, they most likely appear during
the slow call_console_drivers(). They are handled inside the
for(;;) cycle unless we reach the atomic_print_limit and
wake up the printk_kthread.

IMHO, we should just mention here that the jobs might get offloaded
to prevent softlookups when console_unlock() is called in atomic
context.

> +		 */
> +		did_offload = console_offload_printing();
>  		printk_safe_exit_irqrestore(flags);
>  
> -		if (do_cond_resched)
> +		if (!did_offload && do_cond_resched)
>  			cond_resched();
>  	}
>  	console_locked = 0;
> @@ -2276,6 +2445,8 @@ void console_unlock(void)
>  	 */
>  	raw_spin_lock(&logbuf_lock);
>  	retry = console_seq != log_next_seq;

This should be:

	retry = console_seq != log_next_seq && !did_offload;

Otherwise, it would never leave.

I know that this was only proposal and you did it late until
late night.

Anyway, I like the proposal. I have a good feeling about
this way. Also I really like how the main logic is localized
in console_offload_printing().

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-13 14:03                                           ` Petr Mladek
@ 2017-04-14  4:42                                             ` Sergey Senozhatsky
  0 siblings, 0 replies; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-04-14  4:42 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Sergey Senozhatsky, Pavel Machek, Jan Kara, Eric W. Biederman,
	Ye Xiaolong, Steven Rostedt, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, Sergey Senozhatsky

Hello Petr,

thanks for taking a look!

On (04/13/17 16:03), Petr Mladek wrote:
> > +static inline bool console_offload_printing(void)
> > +{
> > +	static struct task_struct *printing_task = NULL;
> > +	static unsigned long lines_printed = 0;
> > +
> > +	if (!atomic_print_limit || !printk_kthread_enabled())
> > +		return false;
> > +
> > +	/* We rescheduled - reset the counters. */
> > +	if (printing_task != current) {
> > +		lines_printed = 0;
> > +		printing_task = current;
> > +		return false;
> > +	}
> 
> If we want to check that the process rescheduled, we should
> store/check also current->nvcsw + current->nivcsw.

ok.

[..]
> My only fear is that it is getting more and more complicated.
> On the other hand, any partial solution is asking for
> troubles and complains.

yeah. we have to aim slightly different and conflicting targets - introducing
a new printk behavior, while preserving an already existing guarantees. which
is a bit tricky.


[..]
> > +	if (lines_printed > 2 * (unsigned long)atomic_print_limit) {
> > +		printk_enforce_emergency = true;
> > +		pr_crit("Declaring printk emergency mode.\n");
> > +		return false;
> > +	}
> 
> The only messages that are printed on my workstation are the same
> few lines everytime I connect my phone over USB to get it charged.

you are right. this is a known and yet to be resolved issue.


> It might help to check the number of process switch counts as
> suggested above.

will take a look at your 'current->nvcsw + current->nivcsw' idea.
thanks.


[..]
> > +		/*
> > +		 * Sometimes we may lock console_sem before printk_kthread.
> > +		 * In this case we will jump to `again' label (if there are
> > +		 * pending messages), print one more line from current
> > +		 * process, break out of printing loop (we don't reset the
> > +		 * counter of printed lines) and do up_console_sem() to
> > +		 * wakeup printk_kthread again.
> > +		 *
> > +		 * If printk_kthread never wakes up (which may indicate that
> > +		 * the system is unstable or something weird is going on),
> > +		 * then we will keep jumping to `again' label and printing
> > +		 * one message from the logbuf. This is a bit ugly, but at
> > +		 * least we will print out the logbuf.
> > +		 *
> > +		 * If such condition occurs, console_offload_printing() can
> > +		 * declare `printk_emergency' at some point.
> 
> I am a bit confused by the comment above. The again goto target is
> used only when there is a race between leaving the loop and releasing
> the console_sem. It is a corner case.

not really.

this is the part where "preserve printk guarantees" jumps in.

when we limit the number of lines we can print we have to leave this loop
with not fully flushed logbuf. so `goto again' is not solely for corner
case anymore. when we prematurely leave the printing loop, we wake_up
printk_kthread, unlock console_sem... and then we have no idea if
printk_kthread going to wake_up at all, and, if it's going to, how much
time will it take. at the same time we have a task that is already in
console_unlock() and, probably, we still have pending messages in the logbuf.
that's why the process that just has left the printing loop [and there easily
might be pending messages in the logbuf] does the whole 'retry' thing. we can
have a misbehaving high priority process or something, that would prevent
printk_kthread from becoming running just when we need it. so, at least
sometimes, the printing process (the one that breaks ouf of printing loop
and wakes up printk_kthread) can re-acquire console_sem and print one more
line, then it up() console_sem, which, hopefully, will wake_up printk_kthread.
if printk_kthread did become running then if would be in console_sem wait
list at this point. if it didn't - then we

	a) wake up some other process that is probably in console_sem list
	   (hopefully there is one)

or

	b) continue printing from the current process. because printk_kthread
	   is still not running and there are no other processes that want to
	   console_lock(). not much we can do at this point.


so in expected/normal scenario, we fail to re-acquire the console_sem lock
(console_trylock()), which means that either printk_kthread or some other
process from console_sem wait list acquired the console_sem and will take
over printing.


I do something like this

---
@@ -2427,6 +2427,8 @@ void console_unlock(void)
                console_seq++;
                raw_spin_unlock(&logbuf_lock);
 
+               sprintf(text + 7, "{%s}", current->comm);
+
                stop_critical_timings();        /* don't trace print latency */
                call_console_drivers(ext_text, ext_len, text, len);
                start_critical_timings();
---


and fire up some silly printk tests (I don't care what exactly it prints,
I'm curious what process prints it). it sort of makes it easier to observe
the behaviour.



> If there are messages from other CPUs, they most likely appear during
> the slow call_console_drivers(). They are handled inside the
> for(;;) cycle unless we reach the atomic_print_limit and
> wake up the printk_kthread.

but `atomic_print_limit' can be quite restrictive. we break out of the
printing loop then `atomic_print_limit' tells us to do so, not when we
the logbuf is empty.


[..]
> > @@ -2276,6 +2445,8 @@ void console_unlock(void)
> >  	 */
> >  	raw_spin_lock(&logbuf_lock);
> >  	retry = console_seq != log_next_seq;
> 
> This should be:
> 
> 	retry = console_seq != log_next_seq && !did_offload;
> 
> Otherwise, it would never leave.

it is expected to leave at some point, when we would know for sure that
a successful offloading took place. otherwise we probably shouldn't leave.
we invoked wake_up_process(), but that does not really buy us anything
(from printk guarantees POV). we trust one thing only - a process holding
the console_sem lock. because that's "a fact"; wake_up_process() is nothing
but "a promise".


so, in short, the basic idea is:
- the worse the situation is, the closer printk behavior to the original one.


the bigger the scheduling latencies are, the more time we spent
in console_unlock(), printing more than `atomic_print_limit' chars.


in more details:

once we cross the `atomic_print_limit' we start this thing


@again label:

1) call_console_drivers()	- print a single logbuf entry

1) up()				- "somebody please take over" (we expect that at
				  least one process will do this - printk_kthread.
				  but we don't know when, and not entirely sure
				  if it will. so we are increasing our chances
				  by waking up printk_kthread, but we don't
				  completely count on it).

2) console_trylock()		- if successful
					then "eeehhh, OK... I'll print one more line"
					goto @again
				  else
					return, someone console_lock()-ed

yes, in absolutely bad scenario -- even printk_kthread can't lock
console_sem - this means old printk behaviour. but that's sort of exactly
what we want to do with the printk_emergency_begin/end annotations anyway.
in exchange we print out the logbuf eventually.


I'm trying to find a compromise that would make everyone happy. may be
I'm missing an easier/better solution, wouldn't be the first time ever :)

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-04-07 14:40                               ` Steven Rostedt
@ 2017-05-08  6:37                                 ` Sergey Senozhatsky
  2017-05-17 13:13                                   ` Petr Mladek
  0 siblings, 1 reply; 75+ messages in thread
From: Sergey Senozhatsky @ 2017-05-08  6:37 UTC (permalink / raw)
  To: Steven Rostedt, Petr Mladek, Jan Kara, Pavel Machek
  Cc: Eric W. Biederman, Ye Xiaolong, Andrew Morton, Linus Torvalds,
	Peter Zijlstra, Rafael J . Wysocki, Greg Kroah-Hartman,
	Jiri Slaby, Len Brown, linux-kernel, lkp, Sergey Senozhatsky,
	Sergey Senozhatsky

Hello,

On (04/07/17 10:40), Steven Rostedt wrote:
[..]
> On Fri, 7 Apr 2017 14:44:55 +0200
> Pavel Machek <pavel@ucw.cz> wrote:
> 
> > Well, I have to agree here. This is 20 years worth of mess :-(.
> 
> Maybe someone should propose a micro-conf at Linux Plumbers where we
> can brain storm a way to re-invent printk()? Seems it can do with a
> completely new rewrite. ;-)

So I've been thinking about it... I'm somewhat limited in budget this
year and can do either LPC or KS* - purely depending on which one can
"attract" a required critical mass. From _this_ point of view KS,
perhaps, would be more appropriate, especially given that it's in EU
this year, if I'm not mistaken. But LPC has its own merits too, of
course. LPC's microconference sounds good enough.

What do you guys think?


* I obviously don't expect to be invited to the KS, all I said is that
  I can be in Prague around that time so we can sit somewhere and talk.

	-ss

^ permalink raw reply	[flat|nested] 75+ messages in thread

* Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
  2017-05-08  6:37                                 ` Sergey Senozhatsky
@ 2017-05-17 13:13                                   ` Petr Mladek
  0 siblings, 0 replies; 75+ messages in thread
From: Petr Mladek @ 2017-05-17 13:13 UTC (permalink / raw)
  To: Sergey Senozhatsky
  Cc: Steven Rostedt, Jan Kara, Pavel Machek, Eric W. Biederman,
	Ye Xiaolong, Andrew Morton, Linus Torvalds, Peter Zijlstra,
	Rafael J . Wysocki, Greg Kroah-Hartman, Jiri Slaby, Len Brown,
	linux-kernel, lkp, Sergey Senozhatsky

On Mon 2017-05-08 15:37:41, Sergey Senozhatsky wrote:
> Hello,
> 
> On (04/07/17 10:40), Steven Rostedt wrote:
> [..]
> > On Fri, 7 Apr 2017 14:44:55 +0200
> > Pavel Machek <pavel@ucw.cz> wrote:
> > 
> > > Well, I have to agree here. This is 20 years worth of mess :-(.
> > 
> > Maybe someone should propose a micro-conf at Linux Plumbers where we
> > can brain storm a way to re-invent printk()? Seems it can do with a
> > completely new rewrite. ;-)
> 
> So I've been thinking about it... I'm somewhat limited in budget this
> year and can do either LPC or KS* - purely depending on which one can
> "attract" a required critical mass. From _this_ point of view KS,
> perhaps, would be more appropriate, especially given that it's in EU
> this year, if I'm not mistaken. But LPC has its own merits too, of
> course. LPC's microconference sounds good enough.
> 
> What do you guys think?
> 
> 
> * I obviously don't expect to be invited to the KS, all I said is that
>   I can be in Prague around that time so we can sit somewhere and talk.

I live in Prague and will be happy to discuss printk issues with
interested people.

I think about rewriting printk from time to time. Some brainstorming
might be helpful. On the other hand, I have become aware of many new printk
limits and deficiencies last year. The more I know the less I am sure
that I know enough for making a good design. I would personally prefer
to give it some longer time to gather information. Especially I am
interested into more feedback about the printk kthread and console
work offloading. Also I still do not have enough experience about
how different consoles are used and behave.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 75+ messages in thread

end of thread, other threads:[~2017-05-17 13:13 UTC | newest]

Thread overview: 75+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu Sergey Senozhatsky
2017-03-31 13:09   ` Petr Mladek
2017-03-31 13:33     ` Peter Zijlstra
2017-04-03 11:23       ` Sergey Senozhatsky
2017-04-03 12:43         ` Petr Mladek
2017-03-29  9:25 ` [RFC][PATCHv2 2/8] printk: introduce printing kernel thread Sergey Senozhatsky
2017-04-04  9:01   ` Petr Mladek
2017-04-04  9:36     ` Sergey Senozhatsky
2017-04-06 17:14   ` Pavel Machek
2017-04-07  5:12     ` Sergey Senozhatsky
2017-04-07  7:21       ` Pavel Machek
2017-04-07  8:15         ` Sergey Senozhatsky
2017-04-07 12:06           ` Pavel Machek
2017-03-29  9:25 ` [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func() Sergey Senozhatsky
2017-03-31 14:56   ` Petr Mladek
2017-04-04 16:15     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places Sergey Senozhatsky
2017-03-31 15:06   ` Petr Mladek
2017-04-06 17:20   ` Pavel Machek
2017-04-09 10:59     ` Andreas Mohr
2017-04-10 12:20       ` Petr Mladek
2017-04-10 14:38         ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 5/8] sysrq: " Sergey Senozhatsky
2017-03-31 15:37   ` Petr Mladek
2017-04-01  0:04     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 6/8] kexec: " Sergey Senozhatsky
2017-03-31 15:39   ` Petr Mladek
2017-03-29  9:25 ` [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter Sergey Senozhatsky
2017-04-03 15:29   ` Petr Mladek
2017-04-04  8:29     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 8/8] printk: enable printk offloading Sergey Senozhatsky
2017-03-30 21:38   ` [printk] fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage kernel test robot
2017-03-31  2:35     ` Sergey Senozhatsky
2017-03-31  4:04       ` Sergey Senozhatsky
2017-03-31  6:39         ` Ye Xiaolong
2017-03-31 14:47           ` Sergey Senozhatsky
2017-03-31 15:28             ` Eric W. Biederman
2017-04-03  9:31               ` Jan Kara
2017-04-03 10:06                 ` Petr Mladek
2017-04-06 17:33                 ` Pavel Machek
2017-04-07  4:44                   ` Sergey Senozhatsky
2017-04-07  7:15                     ` Pavel Machek
2017-04-07  7:46                       ` Sergey Senozhatsky
2017-04-07  8:14                         ` Pavel Machek
2017-04-07 12:10                           ` Sergey Senozhatsky
2017-04-07 12:44                             ` Pavel Machek
2017-04-07 14:40                               ` Steven Rostedt
2017-05-08  6:37                                 ` Sergey Senozhatsky
2017-05-17 13:13                                   ` Petr Mladek
2017-04-07 15:13                               ` Sergey Senozhatsky
2017-04-07 15:23                                 ` Peter Zijlstra
2017-04-07 15:40                                   ` Sergey Senozhatsky
2017-04-09 18:21                                     ` Eric W. Biederman
2017-04-10  4:46                                       ` Sergey Senozhatsky
2017-04-09 10:12                                 ` Pavel Machek
2017-04-10  4:53                                   ` Sergey Senozhatsky
2017-04-10 11:54                                     ` Petr Mladek
2017-04-10 15:08                                       ` Sergey Senozhatsky
2017-04-10 18:48                                     ` Pavel Machek
2017-04-11  1:46                                       ` Sergey Senozhatsky
2017-04-11 16:19                                         ` Sergey Senozhatsky
2017-04-12 18:43                                           ` Pavel Machek
2017-04-13  4:34                                             ` Sergey Senozhatsky
2017-04-13  5:50                                           ` Sergey Senozhatsky
2017-04-13  8:19                                             ` Sergey Senozhatsky
2017-04-13 14:03                                           ` Petr Mladek
2017-04-14  4:42                                             ` Sergey Senozhatsky
2017-04-07 14:29                           ` Steven Rostedt
2017-04-09  9:57                             ` Pavel Machek
2017-04-03 10:51               ` Sergey Senozhatsky
2017-04-05  7:29           ` Ye Xiaolong
2017-04-05  8:40             ` Sergey Senozhatsky
2017-04-03 15:42   ` [RFC][PATCHv2 8/8] printk: enable printk offloading Petr Mladek
2017-04-04 13:20     ` Sergey Senozhatsky

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.