All of lore.kernel.org
 help / color / mirror / Atom feed
From: Josh Poimboeuf <jpoimboe@redhat.com>
To: Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@kernel.org>, "H . Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org, linux-kernel@vger.kernel.org,
	Andy Lutomirski <luto@amacapital.net>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	Brian Gerst <brgerst@gmail.com>,
	Kees Cook <keescook@chromium.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	Byungchul Park <byungchul.park@lge.com>,
	Nilay Vaish <nilayvaish@gmail.com>
Subject: [PATCH v4 40/57] x86/dumpstack: convert show_trace_log_lvl() to use the new unwinder
Date: Thu, 18 Aug 2016 08:06:20 -0500	[thread overview]
Message-ID: <69aa8b6c20a12948b3fb9bd6e9e712010cc4e901.1471525031.git.jpoimboe@redhat.com> (raw)
In-Reply-To: <cover.1471525031.git.jpoimboe@redhat.com>

Convert show_trace_log_lvl() to use the new unwinder.  dump_trace() has
been deprecated.

show_trace_log_lvl() is special compared to other users of the unwinder.
It's the only place where both reliable *and* unreliable addresses are
needed.  With frame pointers enabled, most callers of the unwinder don't
want to know about unreliable addresses.  But in this case, when we're
dumping the stack to the console because something presumably went
wrong, the unreliable addresses are useful:

- They show stale data on the stack which can provide useful clues.

- If something goes wrong with the unwinder, or if frame pointers are
  corrupt or missing, all the stack addresses still get shown.

So in order to show all addresses on the stack, and at the same time
figure out which addresses are reliable, we have to do the scanning and
the unwinding in parallel.

The scanning is done with the help of get_stack_info() to traverse the
stacks.  The unwinding is done separately by the new unwinder.

In theory we could simplify show_trace_log_lvl() by instead pushing some
of this logic into the unwind code.  But then we would need some kind of
"fake" frame logic in the unwinder which would add a lot of complexity
and wouldn't be worth it in order to support only one user.

Another benefit of this approach is that once we have a DWARF unwinder,
we should be able to just plug it in with minimal impact to this code.

Another change here is that callers of show_trace_log_lvl() don't need
to provide the 'bp' argument.  The unwinder already finds the relevant
frame pointer by unwinding until it reaches the first frame after the
provided stack pointer.

Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
---
 arch/x86/include/asm/stacktrace.h |  10 ++-
 arch/x86/kernel/dumpstack.c       | 130 +++++++++++++++++++++++++++++---------
 arch/x86/kernel/dumpstack_32.c    |   6 +-
 arch/x86/kernel/dumpstack_64.c    |  10 +--
 4 files changed, 111 insertions(+), 45 deletions(-)

diff --git a/arch/x86/include/asm/stacktrace.h b/arch/x86/include/asm/stacktrace.h
index e564b1d..4f9630d 100644
--- a/arch/x86/include/asm/stacktrace.h
+++ b/arch/x86/include/asm/stacktrace.h
@@ -119,13 +119,11 @@ get_stack_pointer(struct task_struct *task, struct pt_regs *regs)
 	return (unsigned long *)task->thread.sp;
 }
 
-extern void
-show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
-		   unsigned long *stack, unsigned long bp, char *log_lvl);
+void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
+			unsigned long *stack, char *log_lvl);
 
-extern void
-show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
-		   unsigned long *sp, unsigned long bp, char *log_lvl);
+void show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
+			unsigned long *sp, char *log_lvl);
 
 extern unsigned int code_bytes;
 
diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 960eef6..b27df04 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -17,7 +17,7 @@
 #include <linux/sysfs.h>
 
 #include <asm/stacktrace.h>
-
+#include <asm/unwind.h>
 
 int panic_on_unrecovered_nmi;
 int panic_on_io_nmi;
@@ -142,54 +142,122 @@ print_context_stack_bp(struct task_struct *task,
 }
 EXPORT_SYMBOL_GPL(print_context_stack_bp);
 
-static int print_trace_stack(void *data, const char *name)
+void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
+			unsigned long *stack, char *log_lvl)
 {
-	printk("%s <%s> ", (char *)data, name);
-	return 0;
-}
+	struct unwind_state state;
+	struct stack_info stack_info = {0};
+	unsigned long visit_mask = 0;
+	int graph_idx = 0;
 
-/*
- * Print one address/symbol entries per line.
- */
-static int print_trace_address(void *data, unsigned long addr, int reliable)
-{
-	printk_stack_address(addr, reliable, data);
-	return 0;
-}
+	printk("%sCall Trace:\n", log_lvl);
 
-static const struct stacktrace_ops print_trace_ops = {
-	.stack			= print_trace_stack,
-	.address		= print_trace_address,
-	.walk_stack		= print_context_stack,
-};
+	stack = stack ? : get_stack_pointer(task, regs);
+	if (!task)
+		task = current;
 
-void
-show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
-		unsigned long *stack, unsigned long bp, char *log_lvl)
-{
-	printk("%sCall Trace:\n", log_lvl);
-	dump_trace(task, regs, stack, bp, &print_trace_ops, log_lvl);
+	unwind_start(&state, task, regs, stack);
+
+	/*
+	 * Iterate through the stacks, starting with the current stack pointer.
+	 * Each stack has a pointer to the next one.
+	 *
+	 * x86-64 can have several stacks:
+	 * - task stack
+	 * - interrupt stack
+	 * - HW exception stacks (double fault, nmi, debug, mce)
+	 *
+	 * x86-32 can have up to three stacks:
+	 * - task stack
+	 * - softirq stack
+	 * - hardirq stack
+	 */
+	for (; stack; stack = stack_info.next_sp) {
+		const char *str_begin, *str_end;
+
+		/*
+		 * If we overflowed the task stack into a guard page, jump back
+		 * to the bottom of the usable stack.
+		 */
+		if (task_stack_page(task) - (void *)stack < PAGE_SIZE)
+			stack = task_stack_page(task);
+
+		if (get_stack_info(stack, task, &stack_info, &visit_mask))
+			break;
+
+		stack_type_str(stack_info.type, &str_begin, &str_end);
+		if (str_begin)
+			printk("%s <%s> ", log_lvl, str_begin);
+
+		/*
+		 * Scan the stack, printing any text addresses we find.  At the
+		 * same time, follow proper stack frames with the unwinder.
+		 *
+		 * Addresses found during the scan which are not reported by
+		 * the unwinder are considered to be additional clues which are
+		 * sometimes useful for debugging and are prefixed with '?'.
+		 * This also serves as a failsafe option in case the unwinder
+		 * goes off in the weeds.
+		 */
+		for (; stack < stack_info.end; stack++) {
+			unsigned long real_addr;
+			int reliable = 0;
+			unsigned long addr = *stack;
+			unsigned long *ret_addr_p =
+				unwind_get_return_address_ptr(&state);
+
+			if (!__kernel_text_address(addr))
+				continue;
+
+			if (stack == ret_addr_p)
+				reliable = 1;
+
+			/*
+			 * When function graph tracing is enabled for a
+			 * function, its return address on the stack is
+			 * replaced with the address of an ftrace handler
+			 * (return_to_handler).  In that case, before printing
+			 * the "real" address, we want to print the handler
+			 * address as an "unreliable" hint that function graph
+			 * tracing was involved.
+			 */
+			real_addr = ftrace_graph_ret_addr(task, &graph_idx,
+							  addr, stack);
+			if (real_addr != addr)
+				printk_stack_address(addr, 0, log_lvl);
+			printk_stack_address(real_addr, reliable, log_lvl);
+
+			if (!reliable)
+				continue;
+
+			/*
+			 * Get the next frame from the unwinder.  No need to
+			 * check for an error: if anything goes wrong, the rest
+			 * of the addresses will just be printed as unreliable.
+			 */
+			unwind_next_frame(&state);
+		}
+
+		if (str_end)
+			printk("%s <%s> ", log_lvl, str_end);
+	}
 }
 
 void show_stack(struct task_struct *task, unsigned long *sp)
 {
-	unsigned long bp = 0;
-
 	/*
 	 * Stack frames below this one aren't interesting.  Don't show them
 	 * if we're printing for %current.
 	 */
-	if (!sp && (!task || task == current)) {
+	if (!sp && (!task || task == current))
 		sp = get_stack_pointer(current, NULL);
-		bp = (unsigned long)get_frame_pointer(current, NULL);
-	}
 
-	show_stack_log_lvl(task, NULL, sp, bp, "");
+	show_stack_log_lvl(task, NULL, sp, "");
 }
 
 void show_stack_regs(struct pt_regs *regs)
 {
-	show_stack_log_lvl(current, regs, NULL, 0, "");
+	show_stack_log_lvl(NULL, regs, NULL, "");
 }
 
 static arch_spinlock_t die_lock = __ARCH_SPIN_LOCK_UNLOCKED;
diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c
index c233f93..94c5ed5 100644
--- a/arch/x86/kernel/dumpstack_32.c
+++ b/arch/x86/kernel/dumpstack_32.c
@@ -150,7 +150,7 @@ EXPORT_SYMBOL(dump_trace);
 
 void
 show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
-		   unsigned long *sp, unsigned long bp, char *log_lvl)
+		   unsigned long *sp, char *log_lvl)
 {
 	unsigned long *stack;
 	int i;
@@ -170,7 +170,7 @@ show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
 		touch_nmi_watchdog();
 	}
 	pr_cont("\n");
-	show_trace_log_lvl(task, regs, sp, bp, log_lvl);
+	show_trace_log_lvl(task, regs, sp, log_lvl);
 }
 
 
@@ -192,7 +192,7 @@ void show_regs(struct pt_regs *regs)
 		u8 *ip;
 
 		pr_emerg("Stack:\n");
-		show_stack_log_lvl(NULL, regs, NULL, 0, KERN_EMERG);
+		show_stack_log_lvl(NULL, regs, NULL, KERN_EMERG);
 
 		pr_emerg("Code:");
 
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index 7a4029d..6690322 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -15,6 +15,7 @@
 #include <linux/nmi.h>
 
 #include <asm/stacktrace.h>
+#include <asm/unwind.h>
 
 static char *exception_stack_names[N_EXCEPTION_STACKS] = {
 		[ DOUBLEFAULT_STACK-1	]	= "#DF",
@@ -205,9 +206,8 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 }
 EXPORT_SYMBOL(dump_trace);
 
-void
-show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
-		   unsigned long *sp, unsigned long bp, char *log_lvl)
+void show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
+			unsigned long *sp, char *log_lvl)
 {
 	unsigned long *irq_stack, *irq_stack_end;
 	unsigned long *stack;
@@ -247,7 +247,7 @@ show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
 	}
 
 	pr_cont("\n");
-	show_trace_log_lvl(task, regs, sp, bp, log_lvl);
+	show_trace_log_lvl(task, regs, sp, log_lvl);
 }
 
 void show_regs(struct pt_regs *regs)
@@ -268,7 +268,7 @@ void show_regs(struct pt_regs *regs)
 		u8 *ip;
 
 		printk(KERN_DEFAULT "Stack:\n");
-		show_stack_log_lvl(NULL, regs, NULL, 0, KERN_DEFAULT);
+		show_stack_log_lvl(NULL, regs, NULL, KERN_DEFAULT);
 
 		printk(KERN_DEFAULT "Code: ");
 
-- 
2.7.4

  parent reply	other threads:[~2016-08-18 13:21 UTC|newest]

Thread overview: 107+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-08-18 13:05 [PATCH v4 00/57] x86/dumpstack: rewrite x86 stack dump code Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 01/57] x86/dumpstack: remove show_trace() Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 02/57] x86/asm/head: remove unused init_rsp variable extern Josh Poimboeuf
2016-08-18 16:22   ` Sebastian Andrzej Siewior
2016-08-18 13:05 ` [PATCH v4 03/57] x86/asm/head: rename 'stack_start' -> 'initial_stack' Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 04/57] x86/asm/head: use a common function for starting CPUs Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 05/57] x86/dumpstack: make printk_stack_address() more generally useful Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 06/57] x86/dumpstack: add IRQ_USABLE_STACK_SIZE define Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 07/57] x86/dumpstack: remove extra brackets around "<EOE>" Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 08/57] x86/dumpstack: fix irq stack bounds calculation in show_stack_log_lvl() Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 09/57] x86/dumpstack: fix x86_32 kernel_stack_pointer() previous stack access Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 10/57] x86/dumpstack: add get_stack_pointer() and get_frame_pointer() Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 11/57] x86/dumpstack: remove unnecessary stack pointer arguments Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 12/57] x86: move _stext marker to before head code Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 13/57] x86/head: remove useless zeroed word Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 14/57] x86/head: put real return address on idle task stack Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 15/57] x86/head: fix the end of the stack for idle tasks Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 16/57] x86/entry/32: fix the end of the stack for newly forked tasks Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 17/57] x86/head/32: fix the end of the stack for idle tasks Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 18/57] x86/smp: fix initial idle stack location on 32-bit Josh Poimboeuf
2016-08-18 13:05 ` [PATCH v4 19/57] x86/entry/head/32: use local labels Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 20/57] x86/entry/32: rename 'error_code' to 'common_exception' Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 21/57] perf/x86: check perf_callchain_store() error Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 22/57] oprofile/x86: add regs->ip to oprofile trace Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 23/57] proc: fix return address printk conversion specifer in /proc/<pid>/stack Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 24/57] ftrace: remove CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST from config Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 25/57] ftrace: only allocate the ret_stack 'fp' field when needed Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 26/57] ftrace: add return address pointer to ftrace_ret_stack Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 27/57] ftrace: add ftrace_graph_ret_addr() stack unwinding helpers Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 28/57] x86/dumpstack/ftrace: convert dump_trace() callbacks to use ftrace_graph_ret_addr() Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 29/57] ftrace/x86: implement HAVE_FUNCTION_GRAPH_RET_ADDR_PTR Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 30/57] x86/dumpstack/ftrace: mark function graph handler function as unreliable Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 31/57] x86/dumpstack/ftrace: don't print unreliable addresses in print_context_stack_bp() Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 32/57] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace() Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 33/57] x86/dumpstack: simplify in_exception_stack() Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 34/57] x86/dumpstack: add get_stack_info() interface Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 35/57] x86/dumpstack: add recursion checking for all stacks Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 36/57] x86/unwind: add new unwind interface and implementations Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 37/57] perf/x86: convert perf_callchain_kernel() to use the new unwinder Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 38/57] x86/stacktrace: convert save_stack_trace_*() " Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 39/57] oprofile/x86: convert x86_backtrace() " Josh Poimboeuf
2016-08-18 13:06 ` Josh Poimboeuf [this message]
2016-08-18 13:06 ` [PATCH v4 41/57] x86/dumpstack: remove dump_trace() and related callbacks Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 42/57] x86/entry/unwind: create stack frames for saved interrupt registers Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 43/57] x86/unwind: create stack frames for saved syscall registers Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 44/57] x86/dumpstack: print stack identifier on its own line Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 45/57] x86/dumpstack: print any pt_regs found on the stack Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 46/57] x86/dumpstack: fix duplicate RIP address display in __show_regs() Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 47/57] x86/dumpstack: print orig_ax " Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 48/57] x86: remove 64-byte gap at end of irq stack Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 49/57] x86/unwind: warn on kernel stack corruption Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 50/57] x86/unwind: warn on bad stack return address Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 51/57] x86/unwind: warn if stack grows up Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 52/57] x86/dumpstack: warn on stack recursion Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 53/57] x86/mm: move arch_within_stack_frames() to usercopy.c Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 54/57] x86/mm: convert arch_within_stack_frames() to use the new unwinder Josh Poimboeuf
2016-08-19 18:27   ` Kees Cook
2016-08-19 21:55     ` Josh Poimboeuf
2016-08-22 20:27       ` Josh Poimboeuf
2016-08-22 23:33         ` Josh Poimboeuf
2016-08-23  0:59           ` Kees Cook
2016-08-23  4:21             ` Josh Poimboeuf
2016-08-22 22:11   ` Linus Torvalds
2016-08-23  1:27     ` Kees Cook
2016-08-23 16:21       ` Josh Poimboeuf
2016-08-23 18:47       ` Linus Torvalds
2016-08-23 16:06     ` Josh Poimboeuf
2016-08-23 19:28       ` [PATCH 1/2] mm/usercopy: get rid of "provably correct" warnings Josh Poimboeuf
2016-08-24  2:36         ` Kees Cook
2016-08-23 19:28       ` [PATCH 2/2] mm/usercopy: enable usercopy size checking for modern versions of gcc Josh Poimboeuf
2016-08-24  2:37         ` Kees Cook
2016-08-25 20:47           ` Josh Poimboeuf
2016-08-26  2:14             ` Kees Cook
2016-08-26  3:27               ` Josh Poimboeuf
2016-08-26 13:42                 ` Kees Cook
2016-08-26 13:55                   ` Josh Poimboeuf
2016-08-26 20:56                     ` Josh Poimboeuf
2016-08-26 21:00                       ` Josh Poimboeuf
2016-08-27  0:37                       ` Linus Torvalds
2016-08-29 14:48                         ` Josh Poimboeuf
2016-08-29 15:36                           ` Linus Torvalds
2016-08-29 17:08                             ` [PATCH v2] mm/usercopy: get rid of CONFIG_DEBUG_STRICT_USER_COPY_CHECKS Josh Poimboeuf
2016-08-29 17:59                               ` Josh Poimboeuf
2016-08-30 13:04                               ` [PATCH v3] " Josh Poimboeuf
2016-08-30 17:02                                 ` Linus Torvalds
2016-08-30 18:12                                   ` Al Viro
2016-08-30 18:13                                     ` Linus Torvalds
2016-08-30 18:15                                   ` Kees Cook
2016-08-30 19:09                                     ` Josh Poimboeuf
2016-08-30 19:20                                       ` Kees Cook
2016-08-30 20:13                                     ` Al Viro
2016-08-30 22:20                                       ` Kees Cook
2016-08-31  9:43                                       ` Mark Rutland
2016-08-30 18:33                           ` [PATCH 2/2] mm/usercopy: enable usercopy size checking for modern versions of gcc Kees Cook
2016-08-23 20:31     ` [PATCH v4 54/57] x86/mm: convert arch_within_stack_frames() to use the new unwinder Andy Lutomirski
2016-08-23 21:06       ` Linus Torvalds
2016-08-23 21:08       ` Josh Poimboeuf
2016-08-24  1:37         ` Kees Cook
2016-08-18 13:06 ` [PATCH v4 55/57] x86/mm: simplify starting frame logic for hardened usercopy Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 56/57] x86/mm: removed unused arch_within_stack_frames() arguments Josh Poimboeuf
2016-08-18 13:06 ` [PATCH v4 57/57] mm: re-enable gcc frame address warning Josh Poimboeuf
2016-08-18 13:25 ` [PATCH v4 00/57] x86/dumpstack: rewrite x86 stack dump code Frederic Weisbecker
2016-08-18 13:39   ` Ingo Molnar
2016-08-18 14:31     ` Josh Poimboeuf
2016-08-18 14:41       ` Steven Rostedt
2016-08-18 16:36       ` Ingo Molnar
2016-08-18 14:34     ` Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=69aa8b6c20a12948b3fb9bd6e9e712010cc4e901.1471525031.git.jpoimboe@redhat.com \
    --to=jpoimboe@redhat.com \
    --cc=brgerst@gmail.com \
    --cc=byungchul.park@lge.com \
    --cc=fweisbec@gmail.com \
    --cc=hpa@zytor.com \
    --cc=keescook@chromium.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=luto@amacapital.net \
    --cc=mingo@kernel.org \
    --cc=nilayvaish@gmail.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=tglx@linutronix.de \
    --cc=torvalds@linux-foundation.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.