[PATCH v2 0/2] Stop Intel Processor Trace logging on panic

[PATCH v2 0/2] Stop Intel Processor Trace logging on panic

[Takao Indoh]

Hi all,

These patch series provide a feature to stop Intel Processor Trace
(Intel PT) logging and save its registers in the memory on panic.

Intel PT is a new feature of Intel CPU "Broadwell", it captures
information about program execution flow. Here is a article about Intel
PT.
https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing

Once Intel PT is enabled, the events which change program flow, like
branch instructions, exceptions, interruptions, traps and so on are
logged in the memory. This is very useful for debugging because we can
know the detailed behavior of software.

When kernel panic occurs while you are running a perf command with Intel
PT (with -e intel_pt// option), these patches disable Intel PT and save
its registers in the memory. After crash dump is captured by kdump, you
can retrieve Intel PT log buffer from vmcore and investigate kernel
behavior. I have not made a tool yet to salvage Intel PT log buffer from
vmcore, but I'll do once these patches are accepted.

changelog:
v2:
- Define function in intel_pt.h with static inline

v1:
https://lkml.org/lkml/2015/10/28/136


Background:
These patches are a part of patch series I posted before, the original
discussion is bellow.

x86: Intel Processor Trace Logger
v1: https://lkml.org/lkml/2015/7/29/6
v2: https://lkml.org/lkml/2015/9/8/24

The purpose of the original patches is introducing in-kernel logger
using Intel PT. To implement it I need to add some APIs to control perf
counter and ring buffer in kernel. Alexander Shishkin is working on such
APIs for his work to make use of Intel PT for process core dump. Apart
from such APIs, the feature to save Intel PT registers on panic is
helpful for normal perf command user as I described above, therefore I
separate the feature from original patches.

Takao Indoh (2):
  perf/x86/intel/pt: Add interface to stop Intel PT logging
  x86: Stop Intel PT before kdump starts

 arch/x86/include/asm/intel_pt.h           |   10 ++++++++++
 arch/x86/kernel/cpu/perf_event_intel_pt.c |    9 +++++++++
 arch/x86/kernel/crash.c                   |   11 +++++++++++
 3 files changed, 30 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/include/asm/intel_pt.h

[PATCH v2 1/2] perf/x86/intel/pt: Add interface to stop Intel PT logging

[Takao Indoh]

This patch add a function for external component to stop Intel PT.
Basically this function is used when kernel panic occurs. When it is
called, intel_pt driver disables Intel PT and saves its registers using
pt_event_stop, which is also used by pmu.stop handler. This function
stops Intel PT on the cpu where it is working, therefore user need to
call it for each cpu to stop all logging.

Signed-off-by: Takao Indoh <indou.takao@jp.fujitsu.com>
---
 arch/x86/include/asm/intel_pt.h           |   10 ++++++++++
 arch/x86/kernel/cpu/perf_event_intel_pt.c |    9 +++++++++
 2 files changed, 19 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/include/asm/intel_pt.h

diff --git a/arch/x86/include/asm/intel_pt.h b/arch/x86/include/asm/intel_pt.h
new file mode 100644
index 0000000..e1a4117
--- /dev/null
+++ b/arch/x86/include/asm/intel_pt.h
@@ -0,0 +1,10 @@
+#ifndef _ASM_X86_INTEL_PT_H
+#define _ASM_X86_INTEL_PT_H
+
+#if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_CPU_SUP_INTEL)
+void cpu_emergency_stop_pt(void);
+#else
+static inline void cpu_emergency_stop_pt(void) {}
+#endif
+
+#endif /* _ASM_X86_INTEL_PT_H */
diff --git a/arch/x86/kernel/cpu/perf_event_intel_pt.c b/arch/x86/kernel/cpu/perf_event_intel_pt.c
index 4216928..a638b5b 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_pt.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_pt.c
@@ -27,6 +27,7 @@
 #include <asm/perf_event.h>
 #include <asm/insn.h>
 #include <asm/io.h>
+#include <asm/intel_pt.h>
 
 #include "perf_event.h"
 #include "intel_pt.h"
@@ -1125,6 +1126,14 @@ static int pt_event_init(struct perf_event *event)
 	return 0;
 }
 
+void cpu_emergency_stop_pt(void)
+{
+	struct pt *pt = this_cpu_ptr(&pt_ctx);
+
+	if (pt->handle.event)
+		pt_event_stop(pt->handle.event, PERF_EF_UPDATE);
+}
+
 static __init int pt_init(void)
 {
 	int ret, cpu, prior_warn = 0;
-- 
1.7.1

[PATCH v2 2/2] x86: Stop Intel PT before kdump starts

[Takao Indoh]

This patch stops Intel PT logging and saves its registers in the memory
before kdump is started. This feature is needed to prevent Intel PT from
overwrite its log buffer after panic, and saved registers are needed to
find the last position where Intel PT wrote data. After crash dump is
captured by kdump, user can retrieve the log buffer from vmcore and use
it to investigate kernel behavior.

Signed-off-by: Takao Indoh <indou.takao@jp.fujitsu.com>
---
 arch/x86/kernel/crash.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 74ca2fe..5f383d2 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -35,6 +35,7 @@
 #include <asm/cpu.h>
 #include <asm/reboot.h>
 #include <asm/virtext.h>
+#include <asm/intel_pt.h>
 
 /* Alignment required for elf header segment */
 #define ELF_CORE_HEADER_ALIGN   4096
@@ -127,6 +128,11 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
 	cpu_emergency_vmxoff();
 	cpu_emergency_svm_disable();
 
+	/*
+	 * Disable Intel PT to stop its logging
+	 */
+	cpu_emergency_stop_pt();
+
 	disable_local_APIC();
 }
 
@@ -172,6 +178,11 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	cpu_emergency_vmxoff();
 	cpu_emergency_svm_disable();
 
+	/*
+	 * Disable Intel PT to stop its logging
+	 */
+	cpu_emergency_stop_pt();
+
 #ifdef CONFIG_X86_IO_APIC
 	/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
 	ioapic_zap_locks();
-- 
1.7.1

Re: [PATCH v2 0/2] Stop Intel Processor Trace logging on panic

[Takao Indoh]

Ping, any comments on these patches?

Thanks,
Takao Indoh

On 2015/11/04 14:22, Takao Indoh wrote:
> Hi all,
> 
> These patch series provide a feature to stop Intel Processor Trace
> (Intel PT) logging and save its registers in the memory on panic.
> 
> Intel PT is a new feature of Intel CPU "Broadwell", it captures
> information about program execution flow. Here is a article about Intel
> PT.
> https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing
> 
> Once Intel PT is enabled, the events which change program flow, like
> branch instructions, exceptions, interruptions, traps and so on are
> logged in the memory. This is very useful for debugging because we can
> know the detailed behavior of software.
> 
> When kernel panic occurs while you are running a perf command with Intel
> PT (with -e intel_pt// option), these patches disable Intel PT and save
> its registers in the memory. After crash dump is captured by kdump, you
> can retrieve Intel PT log buffer from vmcore and investigate kernel
> behavior. I have not made a tool yet to salvage Intel PT log buffer from
> vmcore, but I'll do once these patches are accepted.
> 
> changelog:
> v2:
> - Define function in intel_pt.h with static inline
> 
> v1:
> https://lkml.org/lkml/2015/10/28/136
> 
> 
> Background:
> These patches are a part of patch series I posted before, the original
> discussion is bellow.
> 
> x86: Intel Processor Trace Logger
> v1: https://lkml.org/lkml/2015/7/29/6
> v2: https://lkml.org/lkml/2015/9/8/24
> 
> The purpose of the original patches is introducing in-kernel logger
> using Intel PT. To implement it I need to add some APIs to control perf
> counter and ring buffer in kernel. Alexander Shishkin is working on such
> APIs for his work to make use of Intel PT for process core dump. Apart
> from such APIs, the feature to save Intel PT registers on panic is
> helpful for normal perf command user as I described above, therefore I
> separate the feature from original patches.
> 
> Takao Indoh (2):
>    perf/x86/intel/pt: Add interface to stop Intel PT logging
>    x86: Stop Intel PT before kdump starts
> 
>   arch/x86/include/asm/intel_pt.h           |   10 ++++++++++
>   arch/x86/kernel/cpu/perf_event_intel_pt.c |    9 +++++++++
>   arch/x86/kernel/crash.c                   |   11 +++++++++++
>   3 files changed, 30 insertions(+), 0 deletions(-)
>   create mode 100644 arch/x86/include/asm/intel_pt.h
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [PATCH v2 0/2] Stop Intel Processor Trace logging on panic

[Peter Zijlstra]

On Thu, Nov 12, 2015 at 09:05:11PM +0900, Takao Indoh wrote:
> Ping, any comments on these patches?
> 

I've taken them, they should appear in tip sometime after the merge
window closes.

Re: [PATCH v2 0/2] Stop Intel Processor Trace logging on panic

[Takao Indoh]

On 2015/11/12 21:38, Peter Zijlstra wrote:
> On Thu, Nov 12, 2015 at 09:05:11PM +0900, Takao Indoh wrote:
>> Ping, any comments on these patches?
>>
> 
> I've taken them, they should appear in tip sometime after the merge
> window closes.
> 

Ok, thanks.

Thanks,
Takao Indoh

[tip:perf/core] perf/x86/intel/pt: Add interface to stop Intel PT logging

[tip-bot for Takao Indoh]

Commit-ID:  24cc12b17679f8e9046746f92fd377f589efc163
Gitweb:     http://git.kernel.org/tip/24cc12b17679f8e9046746f92fd377f589efc163
Author:     Takao Indoh <indou.takao@jp.fujitsu.com>
AuthorDate: Wed, 4 Nov 2015 14:22:32 +0900
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 23 Nov 2015 09:58:26 +0100

perf/x86/intel/pt: Add interface to stop Intel PT logging

This patch add a function for external components to stop Intel PT.
Basically this function is used when kernel panic occurs. When it is
called, the intel_pt driver disables Intel PT and saves its registers
using pt_event_stop(), which is also used by pmu.stop handler.

This function stops Intel PT on the CPU where it is working, therefore
users of it need to call it for each CPU to stop all logging.

Signed-off-by: Takao Indoh <indou.takao@jp.fujitsu.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Alexander Shishkin<alexander.shishkin@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: H.Peter Anvin <hpa@zytor.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vince Weaver <vincent.weaver@maine.edu>
Cc: Vivek Goyal <vgoyal@redhat.com>
Link: http://lkml.kernel.org/r/1446614553-6072-2-git-send-email-indou.takao@jp.fujitsu.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/include/asm/intel_pt.h           | 10 ++++++++++
 arch/x86/kernel/cpu/perf_event_intel_pt.c |  9 +++++++++
 2 files changed, 19 insertions(+)

diff --git a/arch/x86/include/asm/intel_pt.h b/arch/x86/include/asm/intel_pt.h
new file mode 100644
index 0000000..e1a4117
--- /dev/null
+++ b/arch/x86/include/asm/intel_pt.h
@@ -0,0 +1,10 @@
+#ifndef _ASM_X86_INTEL_PT_H
+#define _ASM_X86_INTEL_PT_H
+
+#if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_CPU_SUP_INTEL)
+void cpu_emergency_stop_pt(void);
+#else
+static inline void cpu_emergency_stop_pt(void) {}
+#endif
+
+#endif /* _ASM_X86_INTEL_PT_H */
diff --git a/arch/x86/kernel/cpu/perf_event_intel_pt.c b/arch/x86/kernel/cpu/perf_event_intel_pt.c
index 868e119..c0bbd10 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_pt.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_pt.c
@@ -27,6 +27,7 @@
 #include <asm/perf_event.h>
 #include <asm/insn.h>
 #include <asm/io.h>
+#include <asm/intel_pt.h>
 
 #include "perf_event.h"
 #include "intel_pt.h"
@@ -1122,6 +1123,14 @@ static int pt_event_init(struct perf_event *event)
 	return 0;
 }
 
+void cpu_emergency_stop_pt(void)
+{
+	struct pt *pt = this_cpu_ptr(&pt_ctx);
+
+	if (pt->handle.event)
+		pt_event_stop(pt->handle.event, PERF_EF_UPDATE);
+}
+
 static __init int pt_init(void)
 {
 	int ret, cpu, prior_warn = 0;

[tip:perf/core] perf, x86: Stop Intel PT before kdump starts

[tip-bot for Takao Indoh]

Commit-ID:  da06a43d3f3f3df87416f654fe15d29fecb5e321
Gitweb:     http://git.kernel.org/tip/da06a43d3f3f3df87416f654fe15d29fecb5e321
Author:     Takao Indoh <indou.takao@jp.fujitsu.com>
AuthorDate: Wed, 4 Nov 2015 14:22:33 +0900
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 23 Nov 2015 09:58:26 +0100

perf, x86: Stop Intel PT before kdump starts

This patch stops Intel PT logging and saves its registers in memory
before kdump is started. This feature is needed to prevent Intel PT from
overwriting its log buffer after panic, and saved registers are needed to
find the last position where Intel PT wrote data.

After the crash dump is captured by kdump, users can retrieve the log buffer
from the vmcore and use it to investigate bad kernel behavior.

Signed-off-by: Takao Indoh <indou.takao@jp.fujitsu.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Alexander Shishkin<alexander.shishkin@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: H.Peter Anvin <hpa@zytor.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vince Weaver <vincent.weaver@maine.edu>
Cc: Vivek Goyal <vgoyal@redhat.com>
Link: http://lkml.kernel.org/r/1446614553-6072-3-git-send-email-indou.takao@jp.fujitsu.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/kernel/crash.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 2c1910f..58f3431 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -35,6 +35,7 @@
 #include <asm/cpu.h>
 #include <asm/reboot.h>
 #include <asm/virtext.h>
+#include <asm/intel_pt.h>
 
 /* Alignment required for elf header segment */
 #define ELF_CORE_HEADER_ALIGN   4096
@@ -125,6 +126,11 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
 	cpu_emergency_vmxoff();
 	cpu_emergency_svm_disable();
 
+	/*
+	 * Disable Intel PT to stop its logging
+	 */
+	cpu_emergency_stop_pt();
+
 	disable_local_APIC();
 }
 
@@ -169,6 +175,11 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	cpu_emergency_vmxoff();
 	cpu_emergency_svm_disable();
 
+	/*
+	 * Disable Intel PT to stop its logging
+	 */
+	cpu_emergency_stop_pt();
+
 #ifdef CONFIG_X86_IO_APIC
 	/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
 	ioapic_zap_locks();