[PATCH v1 0/4] coresight: support panic dump functionality

[PATCH v1 0/4] coresight: support panic dump functionality

[Leo Yan]

### Introduction ###

Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
usually has buffer size from 2KB to 8KB. These data has been used for
profiling and this has been well implemented in coresight driver.

This patch set is to explore ETB RAM data for postmortem debugging.
We could consider ETB RAM data is quite useful for postmortem debugging,
especially if the hardware design with local ETB buffer (ARM DDI 0461B)
chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
dump the hang CPU ETB RAM. Then we can quickly get to know what's the
exact execution flow before its hang.

Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
then the trace data for causing error is easily to be overwritten by
other PEs; but even so sometimes we still have chance to go through the
trace data to assist debugging panic issues.

### Implementation ###

Firstly we need provide a unified APIs for panic dump functionality, so
it can be easily extended to enable panic dump for multiple drivers. This
is finished by patch 0001, it registers panic notifier, and provide the
general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
functions so any coresight device can add into dump list or delete itself
as needed.

Generally all the panic dump specific stuff are related to the sinks
devices, so this initial version code it only supports sink devices; and
Patch 0002 is to add and remove panic callback for sink devices.

Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
drivers; so these two drivers can save specific trace data when panic
happens.

NOTE: patch 0003 for tmc driver panic callback which has been verified on
Hikey board. patch 0004 for etb10 has not been tested due lack hardware
in hand.

### Usage ###

Below are the example for how to use panic dump functionality on 96boards
Hikey, the brief flow is: when the panic happens the ETB panic callback
function saves trace data into memory, then relies on kdump to use
recovery kernel to save DDR content as kernel core dump file; after we
transfer kernel core dump file from board to host PC, use 'crash' tool to
extract the coresight ETB trace data; finally we can use python script
to generate perf format compatible file and use 'perf' to output the
readable execution flow.

- Save trace data into memory with kdump on Hikey:

  ARM64's kdump supports to use the same kernel image both for main
  kernel and dump-capture kernel; so we can simply to load dump-capture
  kernel with below command:
  ./kexec -p vmlinux --dtb=hi6220-hikey.dtb --append="root=/dev/mmcblk0p9
  rw  maxcpus=1 reset_devices earlycon=pl011,0xf7113000 nohlt
  initcall_debug console=tty0 console=ttyAMA3,115200 clk_ignore_unused"

  Enable the coresight path for ETB device:
  echo 1 > /sys/bus/coresight/devices/f6402000.etf/enable_sink
  echo 1 > /sys/bus/coresight/devices/f659c000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f659d000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f659e000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f659f000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65dc000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65dd000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65de000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65df000.etm/enable_source

- After kernel panic happens, the kdump launches dump-capture kernel;
  so we need save kernel's dump file on target:
  cp /proc/vmcore ./vmcore

  After we download vmcore file from Hikey board to host PC, we can
  use 'crash' tool to check coresight dump info and extract trace data:
  crash vmlinux vmcore
  crash> log
  [   37.559337] coresight f6402000.etf: invoke panic dump...
  [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000@0xffff80003b8da180
  crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin

- Use python script perf_cs_dump_wrapper.py to wrap trace data for
  perf format compatible file and finally use perf to output CPU
  execution flow:

  On host PC run python script, please note now this script is not flexbile
  to support all kinds of coresight topologies, this script still has hard coded
  info related with coresight specific topology in Hikey:
  python perf_cs_dump_wrapper.py -i cs_etb_trace.bin -o perf.data

  On Hikey board:
  ./perf script -v -F cpu,event,ip,sym,symoff --kallsyms ksymbol -i perf.data -k vmlinux

  [002]          instructions:  ffff0000087d1d60 psci_cpu_suspend_enter+0x48
  [002]          instructions:  ffff000008093400 cpu_suspend+0x0
  [002]          instructions:  ffff000008093210 __cpu_suspend_enter+0x0
  [002]          instructions:  ffff000008099970 cpu_do_suspend+0x0
  [002]          instructions:  ffff000008093294 __cpu_suspend_enter+0x84
  [002]          instructions:  ffff000008093428 cpu_suspend+0x28
  [002]          instructions:  ffff00000809342c cpu_suspend+0x2c
  [002]          instructions:  ffff0000087d1968 psci_suspend_finisher+0x0
  [002]          instructions:  ffff0000087d1768 psci_cpu_suspend+0x0
  [002]          instructions:  ffff0000087d19f0 __invoke_psci_fn_smc+0x0

Have uploaded related tools into folder:
http://people.linaro.org/~leo.yan/debug/coresight_dump/

Changes from RFC:
* Follow Mathieu's suggestion, use general framework to support dump
  functionality.
* Changed to use perf to analyse trace data.

Leo Yan (4):
  coresight: support panic dump functionality
  coresight: add and remove panic callback for sink
  coresight: tmc: hook panic callback for ETB/ETF
  coresight: etb10: hook panic callback

 drivers/hwtracing/coresight/Kconfig                |  10 ++
 drivers/hwtracing/coresight/Makefile               |   1 +
 drivers/hwtracing/coresight/coresight-etb10.c      |  16 +++
 drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
 drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
 drivers/hwtracing/coresight/coresight-tmc-etf.c    |  26 +++++
 drivers/hwtracing/coresight/coresight.c            |  11 ++
 include/linux/coresight.h                          |   2 +
 8 files changed, 206 insertions(+)
 create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c

-- 
2.7.4

[PATCH v1 0/4] coresight: support panic dump functionality

[Leo Yan]

### Introduction ###

Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
usually has buffer size from 2KB to 8KB. These data has been used for
profiling and this has been well implemented in coresight driver.

This patch set is to explore ETB RAM data for postmortem debugging.
We could consider ETB RAM data is quite useful for postmortem debugging,
especially if the hardware design with local ETB buffer (ARM DDI 0461B)
chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
dump the hang CPU ETB RAM. Then we can quickly get to know what's the
exact execution flow before its hang.

Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
then the trace data for causing error is easily to be overwritten by
other PEs; but even so sometimes we still have chance to go through the
trace data to assist debugging panic issues.

### Implementation ###

Firstly we need provide a unified APIs for panic dump functionality, so
it can be easily extended to enable panic dump for multiple drivers. This
is finished by patch 0001, it registers panic notifier, and provide the
general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
functions so any coresight device can add into dump list or delete itself
as needed.

Generally all the panic dump specific stuff are related to the sinks
devices, so this initial version code it only supports sink devices; and
Patch 0002 is to add and remove panic callback for sink devices.

Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
drivers; so these two drivers can save specific trace data when panic
happens.

NOTE: patch 0003 for tmc driver panic callback which has been verified on
Hikey board. patch 0004 for etb10 has not been tested due lack hardware
in hand.

### Usage ###

Below are the example for how to use panic dump functionality on 96boards
Hikey, the brief flow is: when the panic happens the ETB panic callback
function saves trace data into memory, then relies on kdump to use
recovery kernel to save DDR content as kernel core dump file; after we
transfer kernel core dump file from board to host PC, use 'crash' tool to
extract the coresight ETB trace data; finally we can use python script
to generate perf format compatible file and use 'perf' to output the
readable execution flow.

- Save trace data into memory with kdump on Hikey:

  ARM64's kdump supports to use the same kernel image both for main
  kernel and dump-capture kernel; so we can simply to load dump-capture
  kernel with below command:
  ./kexec -p vmlinux --dtb=hi6220-hikey.dtb --append="root=/dev/mmcblk0p9
  rw  maxcpus=1 reset_devices earlycon=pl011,0xf7113000 nohlt
  initcall_debug console=tty0 console=ttyAMA3,115200 clk_ignore_unused"

  Enable the coresight path for ETB device:
  echo 1 > /sys/bus/coresight/devices/f6402000.etf/enable_sink
  echo 1 > /sys/bus/coresight/devices/f659c000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f659d000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f659e000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f659f000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65dc000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65dd000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65de000.etm/enable_source
  echo 1 > /sys/bus/coresight/devices/f65df000.etm/enable_source

- After kernel panic happens, the kdump launches dump-capture kernel;
  so we need save kernel's dump file on target:
  cp /proc/vmcore ./vmcore

  After we download vmcore file from Hikey board to host PC, we can
  use 'crash' tool to check coresight dump info and extract trace data:
  crash vmlinux vmcore
  crash> log
  [   37.559337] coresight f6402000.etf: invoke panic dump...
  [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000 at 0xffff80003b8da180
  crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin

- Use python script perf_cs_dump_wrapper.py to wrap trace data for
  perf format compatible file and finally use perf to output CPU
  execution flow:

  On host PC run python script, please note now this script is not flexbile
  to support all kinds of coresight topologies, this script still has hard coded
  info related with coresight specific topology in Hikey:
  python perf_cs_dump_wrapper.py -i cs_etb_trace.bin -o perf.data

  On Hikey board:
  ./perf script -v -F cpu,event,ip,sym,symoff --kallsyms ksymbol -i perf.data -k vmlinux

  [002]          instructions:  ffff0000087d1d60 psci_cpu_suspend_enter+0x48
  [002]          instructions:  ffff000008093400 cpu_suspend+0x0
  [002]          instructions:  ffff000008093210 __cpu_suspend_enter+0x0
  [002]          instructions:  ffff000008099970 cpu_do_suspend+0x0
  [002]          instructions:  ffff000008093294 __cpu_suspend_enter+0x84
  [002]          instructions:  ffff000008093428 cpu_suspend+0x28
  [002]          instructions:  ffff00000809342c cpu_suspend+0x2c
  [002]          instructions:  ffff0000087d1968 psci_suspend_finisher+0x0
  [002]          instructions:  ffff0000087d1768 psci_cpu_suspend+0x0
  [002]          instructions:  ffff0000087d19f0 __invoke_psci_fn_smc+0x0

Have uploaded related tools into folder:
http://people.linaro.org/~leo.yan/debug/coresight_dump/

Changes from RFC:
* Follow Mathieu's suggestion, use general framework to support dump
  functionality.
* Changed to use perf to analyse trace data.

Leo Yan (4):
  coresight: support panic dump functionality
  coresight: add and remove panic callback for sink
  coresight: tmc: hook panic callback for ETB/ETF
  coresight: etb10: hook panic callback

 drivers/hwtracing/coresight/Kconfig                |  10 ++
 drivers/hwtracing/coresight/Makefile               |   1 +
 drivers/hwtracing/coresight/coresight-etb10.c      |  16 +++
 drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
 drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
 drivers/hwtracing/coresight/coresight-tmc-etf.c    |  26 +++++
 drivers/hwtracing/coresight/coresight.c            |  11 ++
 include/linux/coresight.h                          |   2 +
 8 files changed, 206 insertions(+)
 create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c

-- 
2.7.4

[PATCH v1 1/4] coresight: support panic dump functionality

[Leo Yan]

After kernel panic happens, coresight has many useful info can be used
for analysis. For example, the trace info from ETB RAM can be used to
check the CPU execution flows before crash. So we can save the tracing
data from sink devices, and rely on kdump to extract them from vmcore
file.

This patch is to add a simple framework to support panic dump
functionality; it registers panic notifier, and provide the general APIs
{coresight_add_panic_cb|coresight_del_panic_cb} as helper functions so
any coresight device can add itself into dump list or delete as needed;
usually these two functions can be used when a session is started or
when it ends. When kernel panic happened, the panic notifier iterates
dump list and calls every node for the device callback function to dump
device specific info. Generally all the panic dump specific stuff are
related to the sinks devices, so this initial version code it only
supports sink devices.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/Kconfig                |  10 ++
 drivers/hwtracing/coresight/Makefile               |   1 +
 drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
 drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
 include/linux/coresight.h                          |   2 +
 5 files changed, 153 insertions(+)
 create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c

diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig
index 8d55d6d..8890023 100644
--- a/drivers/hwtracing/coresight/Kconfig
+++ b/drivers/hwtracing/coresight/Kconfig
@@ -103,4 +103,14 @@ config CORESIGHT_CPU_DEBUG
 	  properly, please refer Documentation/trace/coresight-cpu-debug.txt
 	  for detailed description and the example for usage.
 
+config CORESIGHT_PANIC_DUMP
+	bool "CoreSight Panic Dump driver"
+	depends on ARM || ARM64
+	depends on CORESIGHT_LINKS_AND_SINKS
+	help
+	  This driver provides panic dump functionality for coresight
+	  devices; when kernel panic happens, we can use callback functions
+	  to save trace data to memory. Finally can rely on kdump to extract
+	  out these trace data from kernel dump file.
+
 endif
diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile
index 433d590..0ac3216 100644
--- a/drivers/hwtracing/coresight/Makefile
+++ b/drivers/hwtracing/coresight/Makefile
@@ -17,3 +17,4 @@ obj-$(CONFIG_CORESIGHT_SOURCE_ETM4X) += coresight-etm4x.o \
 obj-$(CONFIG_CORESIGHT_QCOM_REPLICATOR) += coresight-replicator-qcom.o
 obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o
 obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o
+obj-$(CONFIG_CORESIGHT_PANIC_DUMP) += coresight-panic-dump.o
diff --git a/drivers/hwtracing/coresight/coresight-panic-dump.c b/drivers/hwtracing/coresight/coresight-panic-dump.c
new file mode 100644
index 0000000..23869ff
--- /dev/null
+++ b/drivers/hwtracing/coresight/coresight-panic-dump.c
@@ -0,0 +1,130 @@
+/*
+ * Copyright(C) 2017 Linaro Limited. All rights reserved.
+ * Author: Leo Yan <leo.yan@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/coresight.h>
+#include <linux/coresight-pmu.h>
+#include <linux/cpumask.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/init.h>
+#include <linux/perf_event.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+static DEFINE_MUTEX(coresight_panic_lock);
+static struct list_head coresight_panic_list;
+static struct notifier_block coresight_panic_nb;
+
+struct coresight_panic_node {
+	char *name;
+	struct coresight_device *csdev;
+	struct list_head list;
+};
+
+static int coresight_panic_notify(struct notifier_block *nb,
+				  unsigned long mode, void *_unused)
+{
+	int ret = 0, err;
+	struct coresight_panic_node *node;
+	struct coresight_device *csdev;
+	u32 type;
+
+	mutex_lock(&coresight_panic_lock);
+
+	list_for_each_entry(node, &coresight_panic_list, list) {
+		csdev = node->csdev;
+		type = csdev->type;
+
+		dev_info(&csdev->dev, "invoke panic dump...\n");
+
+		switch (type) {
+		case CORESIGHT_DEV_TYPE_SINK:
+		case CORESIGHT_DEV_TYPE_LINKSINK:
+			err = sink_ops(csdev)->panic_cb(csdev);
+			if (err)
+				ret = err;
+			break;
+		default:
+			dev_err(&csdev->dev,
+				"Unsupported type for panic dump\n");
+			break;
+		}
+	}
+
+	mutex_unlock(&coresight_panic_lock);
+	return ret;
+}
+
+int coresight_add_panic_cb(struct coresight_device *csdev)
+{
+	struct coresight_panic_node *node;
+
+	node = kzalloc(sizeof(struct coresight_panic_node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	node->name = kstrndup(dev_name(&csdev->dev), 16, GFP_KERNEL);
+	if (!node->name) {
+		kfree(node);
+		return -ENOMEM;
+	}
+	node->csdev = csdev;
+
+	mutex_lock(&coresight_panic_lock);
+	list_add_tail(&node->list, &coresight_panic_list);
+	mutex_unlock(&coresight_panic_lock);
+
+	return 0;
+}
+
+void coresight_del_panic_cb(struct coresight_device *csdev)
+{
+	struct coresight_panic_node *node;
+
+	mutex_lock(&coresight_panic_lock);
+
+	list_for_each_entry(node, &coresight_panic_list, list) {
+		if (node->csdev == csdev) {
+			list_del(&node->list);
+			kfree(node->name);
+			kfree(node);
+			mutex_unlock(&coresight_panic_lock);
+			return;
+		}
+	}
+
+	dev_err(&csdev->dev, "Failed to find panic node.\n");
+	mutex_unlock(&coresight_panic_lock);
+}
+
+static int __init coresight_panic_init(void)
+{
+	int ret;
+
+	INIT_LIST_HEAD(&coresight_panic_list);
+
+	coresight_panic_nb.notifier_call = coresight_panic_notify;
+	ret = atomic_notifier_chain_register(&panic_notifier_list,
+					     &coresight_panic_nb);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+subsys_initcall(coresight_panic_init);
diff --git a/drivers/hwtracing/coresight/coresight-priv.h b/drivers/hwtracing/coresight/coresight-priv.h
index 5f662d8..e6ed3e9 100644
--- a/drivers/hwtracing/coresight/coresight-priv.h
+++ b/drivers/hwtracing/coresight/coresight-priv.h
@@ -122,4 +122,14 @@ static inline int etm_readl_cp14(u32 off, unsigned int *val) { return 0; }
 static inline int etm_writel_cp14(u32 off, u32 val) { return 0; }
 #endif
 
+#ifdef CONFIG_CORESIGHT_PANIC_DUMP
+extern int coresight_add_panic_cb(struct coresight_device *csdev);
+extern void coresight_del_panic_cb(struct coresight_device *csdev);
+#else
+static inline int coresight_add_panic_cb(struct coresight_device *csdev)
+{ return 0; }
+static inline void coresight_del_panic_cb(struct coresight_device *csdev)
+{ return; }
+#endif
+
 #endif
diff --git a/include/linux/coresight.h b/include/linux/coresight.h
index d950dad..31fcaeb 100644
--- a/include/linux/coresight.h
+++ b/include/linux/coresight.h
@@ -189,6 +189,7 @@ struct coresight_device {
  * @set_buffer:		initialises buffer mechanic before a trace session.
  * @reset_buffer:	finalises buffer mechanic after a trace session.
  * @update_buffer:	update buffer pointers after a trace session.
+ * @panic_cb:		hooks callback function for panic notifier.
  */
 struct coresight_ops_sink {
 	int (*enable)(struct coresight_device *csdev, u32 mode);
@@ -205,6 +206,7 @@ struct coresight_ops_sink {
 	void (*update_buffer)(struct coresight_device *csdev,
 			      struct perf_output_handle *handle,
 			      void *sink_config);
+	int (*panic_cb)(struct coresight_device *csdev);
 };
 
 /**
-- 
2.7.4

[PATCH v1 1/4] coresight: support panic dump functionality

[Leo Yan]

After kernel panic happens, coresight has many useful info can be used
for analysis. For example, the trace info from ETB RAM can be used to
check the CPU execution flows before crash. So we can save the tracing
data from sink devices, and rely on kdump to extract them from vmcore
file.

This patch is to add a simple framework to support panic dump
functionality; it registers panic notifier, and provide the general APIs
{coresight_add_panic_cb|coresight_del_panic_cb} as helper functions so
any coresight device can add itself into dump list or delete as needed;
usually these two functions can be used when a session is started or
when it ends. When kernel panic happened, the panic notifier iterates
dump list and calls every node for the device callback function to dump
device specific info. Generally all the panic dump specific stuff are
related to the sinks devices, so this initial version code it only
supports sink devices.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/Kconfig                |  10 ++
 drivers/hwtracing/coresight/Makefile               |   1 +
 drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
 drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
 include/linux/coresight.h                          |   2 +
 5 files changed, 153 insertions(+)
 create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c

diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig
index 8d55d6d..8890023 100644
--- a/drivers/hwtracing/coresight/Kconfig
+++ b/drivers/hwtracing/coresight/Kconfig
@@ -103,4 +103,14 @@ config CORESIGHT_CPU_DEBUG
 	  properly, please refer Documentation/trace/coresight-cpu-debug.txt
 	  for detailed description and the example for usage.
 
+config CORESIGHT_PANIC_DUMP
+	bool "CoreSight Panic Dump driver"
+	depends on ARM || ARM64
+	depends on CORESIGHT_LINKS_AND_SINKS
+	help
+	  This driver provides panic dump functionality for coresight
+	  devices; when kernel panic happens, we can use callback functions
+	  to save trace data to memory. Finally can rely on kdump to extract
+	  out these trace data from kernel dump file.
+
 endif
diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile
index 433d590..0ac3216 100644
--- a/drivers/hwtracing/coresight/Makefile
+++ b/drivers/hwtracing/coresight/Makefile
@@ -17,3 +17,4 @@ obj-$(CONFIG_CORESIGHT_SOURCE_ETM4X) += coresight-etm4x.o \
 obj-$(CONFIG_CORESIGHT_QCOM_REPLICATOR) += coresight-replicator-qcom.o
 obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o
 obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o
+obj-$(CONFIG_CORESIGHT_PANIC_DUMP) += coresight-panic-dump.o
diff --git a/drivers/hwtracing/coresight/coresight-panic-dump.c b/drivers/hwtracing/coresight/coresight-panic-dump.c
new file mode 100644
index 0000000..23869ff
--- /dev/null
+++ b/drivers/hwtracing/coresight/coresight-panic-dump.c
@@ -0,0 +1,130 @@
+/*
+ * Copyright(C) 2017 Linaro Limited. All rights reserved.
+ * Author: Leo Yan <leo.yan@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/coresight.h>
+#include <linux/coresight-pmu.h>
+#include <linux/cpumask.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/init.h>
+#include <linux/perf_event.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+static DEFINE_MUTEX(coresight_panic_lock);
+static struct list_head coresight_panic_list;
+static struct notifier_block coresight_panic_nb;
+
+struct coresight_panic_node {
+	char *name;
+	struct coresight_device *csdev;
+	struct list_head list;
+};
+
+static int coresight_panic_notify(struct notifier_block *nb,
+				  unsigned long mode, void *_unused)
+{
+	int ret = 0, err;
+	struct coresight_panic_node *node;
+	struct coresight_device *csdev;
+	u32 type;
+
+	mutex_lock(&coresight_panic_lock);
+
+	list_for_each_entry(node, &coresight_panic_list, list) {
+		csdev = node->csdev;
+		type = csdev->type;
+
+		dev_info(&csdev->dev, "invoke panic dump...\n");
+
+		switch (type) {
+		case CORESIGHT_DEV_TYPE_SINK:
+		case CORESIGHT_DEV_TYPE_LINKSINK:
+			err = sink_ops(csdev)->panic_cb(csdev);
+			if (err)
+				ret = err;
+			break;
+		default:
+			dev_err(&csdev->dev,
+				"Unsupported type for panic dump\n");
+			break;
+		}
+	}
+
+	mutex_unlock(&coresight_panic_lock);
+	return ret;
+}
+
+int coresight_add_panic_cb(struct coresight_device *csdev)
+{
+	struct coresight_panic_node *node;
+
+	node = kzalloc(sizeof(struct coresight_panic_node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	node->name = kstrndup(dev_name(&csdev->dev), 16, GFP_KERNEL);
+	if (!node->name) {
+		kfree(node);
+		return -ENOMEM;
+	}
+	node->csdev = csdev;
+
+	mutex_lock(&coresight_panic_lock);
+	list_add_tail(&node->list, &coresight_panic_list);
+	mutex_unlock(&coresight_panic_lock);
+
+	return 0;
+}
+
+void coresight_del_panic_cb(struct coresight_device *csdev)
+{
+	struct coresight_panic_node *node;
+
+	mutex_lock(&coresight_panic_lock);
+
+	list_for_each_entry(node, &coresight_panic_list, list) {
+		if (node->csdev == csdev) {
+			list_del(&node->list);
+			kfree(node->name);
+			kfree(node);
+			mutex_unlock(&coresight_panic_lock);
+			return;
+		}
+	}
+
+	dev_err(&csdev->dev, "Failed to find panic node.\n");
+	mutex_unlock(&coresight_panic_lock);
+}
+
+static int __init coresight_panic_init(void)
+{
+	int ret;
+
+	INIT_LIST_HEAD(&coresight_panic_list);
+
+	coresight_panic_nb.notifier_call = coresight_panic_notify;
+	ret = atomic_notifier_chain_register(&panic_notifier_list,
+					     &coresight_panic_nb);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+subsys_initcall(coresight_panic_init);
diff --git a/drivers/hwtracing/coresight/coresight-priv.h b/drivers/hwtracing/coresight/coresight-priv.h
index 5f662d8..e6ed3e9 100644
--- a/drivers/hwtracing/coresight/coresight-priv.h
+++ b/drivers/hwtracing/coresight/coresight-priv.h
@@ -122,4 +122,14 @@ static inline int etm_readl_cp14(u32 off, unsigned int *val) { return 0; }
 static inline int etm_writel_cp14(u32 off, u32 val) { return 0; }
 #endif
 
+#ifdef CONFIG_CORESIGHT_PANIC_DUMP
+extern int coresight_add_panic_cb(struct coresight_device *csdev);
+extern void coresight_del_panic_cb(struct coresight_device *csdev);
+#else
+static inline int coresight_add_panic_cb(struct coresight_device *csdev)
+{ return 0; }
+static inline void coresight_del_panic_cb(struct coresight_device *csdev)
+{ return; }
+#endif
+
 #endif
diff --git a/include/linux/coresight.h b/include/linux/coresight.h
index d950dad..31fcaeb 100644
--- a/include/linux/coresight.h
+++ b/include/linux/coresight.h
@@ -189,6 +189,7 @@ struct coresight_device {
  * @set_buffer:		initialises buffer mechanic before a trace session.
  * @reset_buffer:	finalises buffer mechanic after a trace session.
  * @update_buffer:	update buffer pointers after a trace session.
+ * @panic_cb:		hooks callback function for panic notifier.
  */
 struct coresight_ops_sink {
 	int (*enable)(struct coresight_device *csdev, u32 mode);
@@ -205,6 +206,7 @@ struct coresight_ops_sink {
 	void (*update_buffer)(struct coresight_device *csdev,
 			      struct perf_output_handle *handle,
 			      void *sink_config);
+	int (*panic_cb)(struct coresight_device *csdev);
 };
 
 /**
-- 
2.7.4

[PATCH v1 2/4] coresight: add and remove panic callback for sink

[Leo Yan]

If the sink device has panic callback function, add the panic callback
node for coresight panic dump list when the sink device is enabled;
also cleanup the node when the sink device is disabled.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
index 0c37356..5928886 100644
--- a/drivers/hwtracing/coresight/coresight.c
+++ b/drivers/hwtracing/coresight/coresight.c
@@ -138,6 +138,13 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
 			if (ret)
 				return ret;
 		}
+
+		/* Add kernel panic callback */
+		if (sink_ops(csdev)->panic_cb) {
+			ret = coresight_add_panic_cb(csdev);
+			if (ret)
+				return ret;
+		}
 		csdev->enable = true;
 	}
 
@@ -149,6 +156,10 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
 static void coresight_disable_sink(struct coresight_device *csdev)
 {
 	if (atomic_dec_return(csdev->refcnt) == 0) {
+		/* Remove kernel panic callback */
+		if (sink_ops(csdev)->panic_cb)
+			coresight_del_panic_cb(csdev);
+
 		if (sink_ops(csdev)->disable) {
 			sink_ops(csdev)->disable(csdev);
 			csdev->enable = false;
-- 
2.7.4

[PATCH v1 2/4] coresight: add and remove panic callback for sink

[Leo Yan]

If the sink device has panic callback function, add the panic callback
node for coresight panic dump list when the sink device is enabled;
also cleanup the node when the sink device is disabled.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
index 0c37356..5928886 100644
--- a/drivers/hwtracing/coresight/coresight.c
+++ b/drivers/hwtracing/coresight/coresight.c
@@ -138,6 +138,13 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
 			if (ret)
 				return ret;
 		}
+
+		/* Add kernel panic callback */
+		if (sink_ops(csdev)->panic_cb) {
+			ret = coresight_add_panic_cb(csdev);
+			if (ret)
+				return ret;
+		}
 		csdev->enable = true;
 	}
 
@@ -149,6 +156,10 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
 static void coresight_disable_sink(struct coresight_device *csdev)
 {
 	if (atomic_dec_return(csdev->refcnt) == 0) {
+		/* Remove kernel panic callback */
+		if (sink_ops(csdev)->panic_cb)
+			coresight_del_panic_cb(csdev);
+
 		if (sink_ops(csdev)->disable) {
 			sink_ops(csdev)->disable(csdev);
 			csdev->enable = false;
-- 
2.7.4

[PATCH v1 3/4] coresight: tmc: hook panic callback for ETB/ETF

[Leo Yan]

The panic dump functionality has been ready, this patch is to hook
panic callback function for ETB/ETF. Because the driver data structure
has allocated buffer when the session started, so simply save ETB/ETF
trace data into the buffer when panic happens.

Below is simple usage when we connect kdump to extract trace data:

- During kernel panic, tmc driver prints out below log.
  Dump ETB buffer 0x2000@0xffff80003bd5a180

- After get kernel dump file 'vmcore', use below command to extract
  ETB/ETF trace data.
  crash vmlinux vmcore
  rd 0xffff80003bd5a180 0x2000 -r cs_etb_trace.bin

- Generate perf formatted file, so can be analyzed by perf tool:
  python perf_cs_dump_wrapper.py -i cs_etb_trace.bin -o perf.data
  ./perf script -v -F cpu,event,ip,sym,symoff --kallsyms ksymbol -i perf.data -k vmlinux

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-tmc-etf.c | 26 +++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index e3b9fb8..4cadbb1 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -476,6 +476,31 @@ static void tmc_update_etf_buffer(struct coresight_device *csdev,
 	CS_LOCK(drvdata->base);
 }
 
+static int tmc_panic_cb(struct coresight_device *csdev)
+{
+	struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+	unsigned long flags;
+
+	if (WARN_ON_ONCE(drvdata->config_type != TMC_CONFIG_TYPE_ETB &&
+			 drvdata->config_type != TMC_CONFIG_TYPE_ETF))
+		return -EINVAL;
+
+	spin_lock_irqsave(&drvdata->spinlock, flags);
+
+	CS_UNLOCK(drvdata->base);
+
+	tmc_flush_and_stop(drvdata);
+	tmc_etb_dump_hw(drvdata);
+
+	CS_LOCK(drvdata->base);
+
+	dev_err(drvdata->dev, "Dump ETB buffer 0x%x@0x%p\n",
+		drvdata->len, drvdata->buf);
+
+	spin_unlock_irqrestore(&drvdata->spinlock, flags);
+	return 0;
+}
+
 static const struct coresight_ops_sink tmc_etf_sink_ops = {
 	.enable		= tmc_enable_etf_sink,
 	.disable	= tmc_disable_etf_sink,
@@ -484,6 +509,7 @@ static const struct coresight_ops_sink tmc_etf_sink_ops = {
 	.set_buffer	= tmc_set_etf_buffer,
 	.reset_buffer	= tmc_reset_etf_buffer,
 	.update_buffer	= tmc_update_etf_buffer,
+	.panic_cb	= tmc_panic_cb,
 };
 
 static const struct coresight_ops_link tmc_etf_link_ops = {
-- 
2.7.4

[PATCH v1 3/4] coresight: tmc: hook panic callback for ETB/ETF

[Leo Yan]

The panic dump functionality has been ready, this patch is to hook
panic callback function for ETB/ETF. Because the driver data structure
has allocated buffer when the session started, so simply save ETB/ETF
trace data into the buffer when panic happens.

Below is simple usage when we connect kdump to extract trace data:

- During kernel panic, tmc driver prints out below log.
  Dump ETB buffer 0x2000 at 0xffff80003bd5a180

- After get kernel dump file 'vmcore', use below command to extract
  ETB/ETF trace data.
  crash vmlinux vmcore
  rd 0xffff80003bd5a180 0x2000 -r cs_etb_trace.bin

- Generate perf formatted file, so can be analyzed by perf tool:
  python perf_cs_dump_wrapper.py -i cs_etb_trace.bin -o perf.data
  ./perf script -v -F cpu,event,ip,sym,symoff --kallsyms ksymbol -i perf.data -k vmlinux

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-tmc-etf.c | 26 +++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index e3b9fb8..4cadbb1 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -476,6 +476,31 @@ static void tmc_update_etf_buffer(struct coresight_device *csdev,
 	CS_LOCK(drvdata->base);
 }
 
+static int tmc_panic_cb(struct coresight_device *csdev)
+{
+	struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+	unsigned long flags;
+
+	if (WARN_ON_ONCE(drvdata->config_type != TMC_CONFIG_TYPE_ETB &&
+			 drvdata->config_type != TMC_CONFIG_TYPE_ETF))
+		return -EINVAL;
+
+	spin_lock_irqsave(&drvdata->spinlock, flags);
+
+	CS_UNLOCK(drvdata->base);
+
+	tmc_flush_and_stop(drvdata);
+	tmc_etb_dump_hw(drvdata);
+
+	CS_LOCK(drvdata->base);
+
+	dev_err(drvdata->dev, "Dump ETB buffer 0x%x at 0x%p\n",
+		drvdata->len, drvdata->buf);
+
+	spin_unlock_irqrestore(&drvdata->spinlock, flags);
+	return 0;
+}
+
 static const struct coresight_ops_sink tmc_etf_sink_ops = {
 	.enable		= tmc_enable_etf_sink,
 	.disable	= tmc_disable_etf_sink,
@@ -484,6 +509,7 @@ static const struct coresight_ops_sink tmc_etf_sink_ops = {
 	.set_buffer	= tmc_set_etf_buffer,
 	.reset_buffer	= tmc_reset_etf_buffer,
 	.update_buffer	= tmc_update_etf_buffer,
+	.panic_cb	= tmc_panic_cb,
 };
 
 static const struct coresight_ops_link tmc_etf_link_ops = {
-- 
2.7.4

[PATCH v1 4/4] coresight: etb10: hook panic callback

[Leo Yan]

This patch is to hook panic callback for etb10 driver; so after panic
can dump trace data from it.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-etb10.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-etb10.c b/drivers/hwtracing/coresight/coresight-etb10.c
index 979ea6e..c9ea281 100644
--- a/drivers/hwtracing/coresight/coresight-etb10.c
+++ b/drivers/hwtracing/coresight/coresight-etb10.c
@@ -473,6 +473,21 @@ static void etb_update_buffer(struct coresight_device *csdev,
 	CS_LOCK(drvdata->base);
 }
 
+static int etb_panic_cb(struct coresight_device *csdev)
+{
+	struct etb_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+	unsigned long flags;
+
+	spin_lock_irqsave(&drvdata->spinlock, flags);
+	etb_disable_hw(drvdata);
+	etb_dump_hw(drvdata);
+	spin_unlock_irqrestore(&drvdata->spinlock, flags);
+
+	dev_err(drvdata->dev, "Dump ETB buffer 0x%x@0x%p\n",
+		drvdata->buffer_depth, drvdata->buf);
+	return 0;
+}
+
 static const struct coresight_ops_sink etb_sink_ops = {
 	.enable		= etb_enable,
 	.disable	= etb_disable,
@@ -481,6 +496,7 @@ static const struct coresight_ops_sink etb_sink_ops = {
 	.set_buffer	= etb_set_buffer,
 	.reset_buffer	= etb_reset_buffer,
 	.update_buffer	= etb_update_buffer,
+	.panic_cb	= etb_panic_cb,
 };
 
 static const struct coresight_ops etb_cs_ops = {
-- 
2.7.4

[PATCH v1 4/4] coresight: etb10: hook panic callback

[Leo Yan]

This patch is to hook panic callback for etb10 driver; so after panic
can dump trace data from it.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-etb10.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-etb10.c b/drivers/hwtracing/coresight/coresight-etb10.c
index 979ea6e..c9ea281 100644
--- a/drivers/hwtracing/coresight/coresight-etb10.c
+++ b/drivers/hwtracing/coresight/coresight-etb10.c
@@ -473,6 +473,21 @@ static void etb_update_buffer(struct coresight_device *csdev,
 	CS_LOCK(drvdata->base);
 }
 
+static int etb_panic_cb(struct coresight_device *csdev)
+{
+	struct etb_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+	unsigned long flags;
+
+	spin_lock_irqsave(&drvdata->spinlock, flags);
+	etb_disable_hw(drvdata);
+	etb_dump_hw(drvdata);
+	spin_unlock_irqrestore(&drvdata->spinlock, flags);
+
+	dev_err(drvdata->dev, "Dump ETB buffer 0x%x at 0x%p\n",
+		drvdata->buffer_depth, drvdata->buf);
+	return 0;
+}
+
 static const struct coresight_ops_sink etb_sink_ops = {
 	.enable		= etb_enable,
 	.disable	= etb_disable,
@@ -481,6 +496,7 @@ static const struct coresight_ops_sink etb_sink_ops = {
 	.set_buffer	= etb_set_buffer,
 	.reset_buffer	= etb_reset_buffer,
 	.update_buffer	= etb_update_buffer,
+	.panic_cb	= etb_panic_cb,
 };
 
 static const struct coresight_ops etb_cs_ops = {
-- 
2.7.4

Re: [PATCH v1 0/4] coresight: support panic dump functionality

[Suzuki K Poulose]

On 03/06/17 15:42, Leo Yan wrote:
> ### Introduction ###
>
> Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
> usually has buffer size from 2KB to 8KB. These data has been used for
> profiling and this has been well implemented in coresight driver.
>
> This patch set is to explore ETB RAM data for postmortem debugging.
> We could consider ETB RAM data is quite useful for postmortem debugging,
> especially if the hardware design with local ETB buffer (ARM DDI 0461B)
> chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
> dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
> dump the hang CPU ETB RAM. Then we can quickly get to know what's the
> exact execution flow before its hang.
>
> Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
> then the trace data for causing error is easily to be overwritten by
> other PEs; but even so sometimes we still have chance to go through the
> trace data to assist debugging panic issues.
>
> ### Implementation ###
>
> Firstly we need provide a unified APIs for panic dump functionality, so
> it can be easily extended to enable panic dump for multiple drivers. This
> is finished by patch 0001, it registers panic notifier, and provide the
> general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
> functions so any coresight device can add into dump list or delete itself
> as needed.
>
> Generally all the panic dump specific stuff are related to the sinks
> devices, so this initial version code it only supports sink devices; and
> Patch 0002 is to add and remove panic callback for sink devices.
>
> Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
> drivers; so these two drivers can save specific trace data when panic
> happens.
>
> NOTE: patch 0003 for tmc driver panic callback which has been verified on
> Hikey board. patch 0004 for etb10 has not been tested due lack hardware
> in hand.
>

> - After kernel panic happens, the kdump launches dump-capture kernel;
>   so we need save kernel's dump file on target:
>   cp /proc/vmcore ./vmcore


>   After we download vmcore file from Hikey board to host PC, we can
>   use 'crash' tool to check coresight dump info and extract trace data:
>   crash vmlinux vmcore
>   crash> log
>   [   37.559337] coresight f6402000.etf: invoke panic dump...
>   [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000@0xffff80003b8da180
>   crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
>

Have you explored appending the above information as a vmcoreinfo parameter via
vmcoreinfo_append_str() ? That would make it easier to list all the information
above and if needed, we may be able to extend the makedumpfile to dump the ETB
dumps from a given vmcore.
  
Suzuki

[PATCH v1 0/4] coresight: support panic dump functionality

[Suzuki K Poulose]

On 03/06/17 15:42, Leo Yan wrote:
> ### Introduction ###
>
> Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
> usually has buffer size from 2KB to 8KB. These data has been used for
> profiling and this has been well implemented in coresight driver.
>
> This patch set is to explore ETB RAM data for postmortem debugging.
> We could consider ETB RAM data is quite useful for postmortem debugging,
> especially if the hardware design with local ETB buffer (ARM DDI 0461B)
> chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
> dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
> dump the hang CPU ETB RAM. Then we can quickly get to know what's the
> exact execution flow before its hang.
>
> Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
> then the trace data for causing error is easily to be overwritten by
> other PEs; but even so sometimes we still have chance to go through the
> trace data to assist debugging panic issues.
>
> ### Implementation ###
>
> Firstly we need provide a unified APIs for panic dump functionality, so
> it can be easily extended to enable panic dump for multiple drivers. This
> is finished by patch 0001, it registers panic notifier, and provide the
> general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
> functions so any coresight device can add into dump list or delete itself
> as needed.
>
> Generally all the panic dump specific stuff are related to the sinks
> devices, so this initial version code it only supports sink devices; and
> Patch 0002 is to add and remove panic callback for sink devices.
>
> Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
> drivers; so these two drivers can save specific trace data when panic
> happens.
>
> NOTE: patch 0003 for tmc driver panic callback which has been verified on
> Hikey board. patch 0004 for etb10 has not been tested due lack hardware
> in hand.
>

> - After kernel panic happens, the kdump launches dump-capture kernel;
>   so we need save kernel's dump file on target:
>   cp /proc/vmcore ./vmcore


>   After we download vmcore file from Hikey board to host PC, we can
>   use 'crash' tool to check coresight dump info and extract trace data:
>   crash vmlinux vmcore
>   crash> log
>   [   37.559337] coresight f6402000.etf: invoke panic dump...
>   [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000 at 0xffff80003b8da180
>   crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
>

Have you explored appending the above information as a vmcoreinfo parameter via
vmcoreinfo_append_str() ? That would make it easier to list all the information
above and if needed, we may be able to extend the makedumpfile to dump the ETB
dumps from a given vmcore.
  
Suzuki

Re: [PATCH v1 2/4] coresight: add and remove panic callback for sink

[Suzuki K Poulose]

On 03/06/17 15:42, Leo Yan wrote:
> If the sink device has panic callback function, add the panic callback
> node for coresight panic dump list when the sink device is enabled;
> also cleanup the node when the sink device is disabled.
>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>
> ---
>  drivers/hwtracing/coresight/coresight.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> index 0c37356..5928886 100644
> --- a/drivers/hwtracing/coresight/coresight.c
> +++ b/drivers/hwtracing/coresight/coresight.c
> @@ -138,6 +138,13 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
>  			if (ret)
>  				return ret;
>  		}
> +
> +		/* Add kernel panic callback */
> +		if (sink_ops(csdev)->panic_cb) {
> +			ret = coresight_add_panic_cb(csdev);
> +			if (ret)
> +				return ret;
> +		}
>  		csdev->enable = true;
>  	}
>
> @@ -149,6 +156,10 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
>  static void coresight_disable_sink(struct coresight_device *csdev)
>  {
>  	if (atomic_dec_return(csdev->refcnt) == 0) {
> +		/* Remove kernel panic callback */
> +		if (sink_ops(csdev)->panic_cb)
> +			coresight_del_panic_cb(csdev);
> +
>  		if (sink_ops(csdev)->disable) {
>  			sink_ops(csdev)->disable(csdev);
>  			csdev->enable = false;
>

Just a thought, instead of adding/deleting every time the sink is enabled,
could we add/del the device once for-ever and let the panic_cb decide to dump
it based on whether the device was active or not ?

Suzuki

[PATCH v1 2/4] coresight: add and remove panic callback for sink

[Suzuki K Poulose]

On 03/06/17 15:42, Leo Yan wrote:
> If the sink device has panic callback function, add the panic callback
> node for coresight panic dump list when the sink device is enabled;
> also cleanup the node when the sink device is disabled.
>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>
> ---
>  drivers/hwtracing/coresight/coresight.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> index 0c37356..5928886 100644
> --- a/drivers/hwtracing/coresight/coresight.c
> +++ b/drivers/hwtracing/coresight/coresight.c
> @@ -138,6 +138,13 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
>  			if (ret)
>  				return ret;
>  		}
> +
> +		/* Add kernel panic callback */
> +		if (sink_ops(csdev)->panic_cb) {
> +			ret = coresight_add_panic_cb(csdev);
> +			if (ret)
> +				return ret;
> +		}
>  		csdev->enable = true;
>  	}
>
> @@ -149,6 +156,10 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
>  static void coresight_disable_sink(struct coresight_device *csdev)
>  {
>  	if (atomic_dec_return(csdev->refcnt) == 0) {
> +		/* Remove kernel panic callback */
> +		if (sink_ops(csdev)->panic_cb)
> +			coresight_del_panic_cb(csdev);
> +
>  		if (sink_ops(csdev)->disable) {
>  			sink_ops(csdev)->disable(csdev);
>  			csdev->enable = false;
>

Just a thought, instead of adding/deleting every time the sink is enabled,
could we add/del the device once for-ever and let the panic_cb decide to dump
it based on whether the device was active or not ?

Suzuki

Re: [PATCH v1 0/4] coresight: support panic dump functionality

[Leo Yan]

On Mon, Jun 05, 2017 at 09:57:39AM +0100, Suzuki K Poulose wrote:
> On 03/06/17 15:42, Leo Yan wrote:
> >### Introduction ###
> >
> >Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
> >usually has buffer size from 2KB to 8KB. These data has been used for
> >profiling and this has been well implemented in coresight driver.
> >
> >This patch set is to explore ETB RAM data for postmortem debugging.
> >We could consider ETB RAM data is quite useful for postmortem debugging,
> >especially if the hardware design with local ETB buffer (ARM DDI 0461B)
> >chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
> >dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
> >dump the hang CPU ETB RAM. Then we can quickly get to know what's the
> >exact execution flow before its hang.
> >
> >Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
> >then the trace data for causing error is easily to be overwritten by
> >other PEs; but even so sometimes we still have chance to go through the
> >trace data to assist debugging panic issues.
> >
> >### Implementation ###
> >
> >Firstly we need provide a unified APIs for panic dump functionality, so
> >it can be easily extended to enable panic dump for multiple drivers. This
> >is finished by patch 0001, it registers panic notifier, and provide the
> >general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
> >functions so any coresight device can add into dump list or delete itself
> >as needed.
> >
> >Generally all the panic dump specific stuff are related to the sinks
> >devices, so this initial version code it only supports sink devices; and
> >Patch 0002 is to add and remove panic callback for sink devices.
> >
> >Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
> >drivers; so these two drivers can save specific trace data when panic
> >happens.
> >
> >NOTE: patch 0003 for tmc driver panic callback which has been verified on
> >Hikey board. patch 0004 for etb10 has not been tested due lack hardware
> >in hand.
> >
> 
> >- After kernel panic happens, the kdump launches dump-capture kernel;
> >  so we need save kernel's dump file on target:
> >  cp /proc/vmcore ./vmcore
> 
> 
> >  After we download vmcore file from Hikey board to host PC, we can
> >  use 'crash' tool to check coresight dump info and extract trace data:
> >  crash vmlinux vmcore
> >  crash> log
> >  [   37.559337] coresight f6402000.etf: invoke panic dump...
> >  [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000@0xffff80003b8da180
> >  crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
> >
> 
> Have you explored appending the above information as a vmcoreinfo parameter via
> vmcoreinfo_append_str() ? That would make it easier to list all the information
> above and if needed, we may be able to extend the makedumpfile to dump the ETB
> dumps from a given vmcore.

Thanks for good suggestion, Suzuki. After you pointed out
vmcoreinfo_append_str() I look at it a bit just now, will add it in
next version and verify on Hikey.

Thanks,
Leo Yan

[PATCH v1 0/4] coresight: support panic dump functionality

[Leo Yan]

On Mon, Jun 05, 2017 at 09:57:39AM +0100, Suzuki K Poulose wrote:
> On 03/06/17 15:42, Leo Yan wrote:
> >### Introduction ###
> >
> >Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
> >usually has buffer size from 2KB to 8KB. These data has been used for
> >profiling and this has been well implemented in coresight driver.
> >
> >This patch set is to explore ETB RAM data for postmortem debugging.
> >We could consider ETB RAM data is quite useful for postmortem debugging,
> >especially if the hardware design with local ETB buffer (ARM DDI 0461B)
> >chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
> >dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
> >dump the hang CPU ETB RAM. Then we can quickly get to know what's the
> >exact execution flow before its hang.
> >
> >Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
> >then the trace data for causing error is easily to be overwritten by
> >other PEs; but even so sometimes we still have chance to go through the
> >trace data to assist debugging panic issues.
> >
> >### Implementation ###
> >
> >Firstly we need provide a unified APIs for panic dump functionality, so
> >it can be easily extended to enable panic dump for multiple drivers. This
> >is finished by patch 0001, it registers panic notifier, and provide the
> >general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
> >functions so any coresight device can add into dump list or delete itself
> >as needed.
> >
> >Generally all the panic dump specific stuff are related to the sinks
> >devices, so this initial version code it only supports sink devices; and
> >Patch 0002 is to add and remove panic callback for sink devices.
> >
> >Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
> >drivers; so these two drivers can save specific trace data when panic
> >happens.
> >
> >NOTE: patch 0003 for tmc driver panic callback which has been verified on
> >Hikey board. patch 0004 for etb10 has not been tested due lack hardware
> >in hand.
> >
> 
> >- After kernel panic happens, the kdump launches dump-capture kernel;
> >  so we need save kernel's dump file on target:
> >  cp /proc/vmcore ./vmcore
> 
> 
> >  After we download vmcore file from Hikey board to host PC, we can
> >  use 'crash' tool to check coresight dump info and extract trace data:
> >  crash vmlinux vmcore
> >  crash> log
> >  [   37.559337] coresight f6402000.etf: invoke panic dump...
> >  [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000 at 0xffff80003b8da180
> >  crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
> >
> 
> Have you explored appending the above information as a vmcoreinfo parameter via
> vmcoreinfo_append_str() ? That would make it easier to list all the information
> above and if needed, we may be able to extend the makedumpfile to dump the ETB
> dumps from a given vmcore.

Thanks for good suggestion, Suzuki. After you pointed out
vmcoreinfo_append_str() I look at it a bit just now, will add it in
next version and verify on Hikey.

Thanks,
Leo Yan

Re: [PATCH v1 2/4] coresight: add and remove panic callback for sink

[Leo Yan]

On Mon, Jun 05, 2017 at 10:24:41AM +0100, Suzuki K Poulose wrote:
> On 03/06/17 15:42, Leo Yan wrote:
> >If the sink device has panic callback function, add the panic callback
> >node for coresight panic dump list when the sink device is enabled;
> >also cleanup the node when the sink device is disabled.
> >
> >Signed-off-by: Leo Yan <leo.yan@linaro.org>
> >---
> > drivers/hwtracing/coresight/coresight.c | 11 +++++++++++
> > 1 file changed, 11 insertions(+)
> >
> >diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> >index 0c37356..5928886 100644
> >--- a/drivers/hwtracing/coresight/coresight.c
> >+++ b/drivers/hwtracing/coresight/coresight.c
> >@@ -138,6 +138,13 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
> > 			if (ret)
> > 				return ret;
> > 		}
> >+
> >+		/* Add kernel panic callback */
> >+		if (sink_ops(csdev)->panic_cb) {
> >+			ret = coresight_add_panic_cb(csdev);
> >+			if (ret)
> >+				return ret;
> >+		}
> > 		csdev->enable = true;
> > 	}
> >
> >@@ -149,6 +156,10 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
> > static void coresight_disable_sink(struct coresight_device *csdev)
> > {
> > 	if (atomic_dec_return(csdev->refcnt) == 0) {
> >+		/* Remove kernel panic callback */
> >+		if (sink_ops(csdev)->panic_cb)
> >+			coresight_del_panic_cb(csdev);
> >+
> > 		if (sink_ops(csdev)->disable) {
> > 			sink_ops(csdev)->disable(csdev);
> > 			csdev->enable = false;
> >
> 
> Just a thought, instead of adding/deleting every time the sink is enabled,
> could we add/del the device once for-ever and let the panic_cb decide to dump
> it based on whether the device was active or not ?

Yeah, we can check 'if (drvdata->mode == CS_MODE_DISABLED)' and
directly bail out if the module is disabled for panic dumping. Will
fix with this way.

Thanks,
Leo Yan

[PATCH v1 2/4] coresight: add and remove panic callback for sink

[Leo Yan]

On Mon, Jun 05, 2017 at 10:24:41AM +0100, Suzuki K Poulose wrote:
> On 03/06/17 15:42, Leo Yan wrote:
> >If the sink device has panic callback function, add the panic callback
> >node for coresight panic dump list when the sink device is enabled;
> >also cleanup the node when the sink device is disabled.
> >
> >Signed-off-by: Leo Yan <leo.yan@linaro.org>
> >---
> > drivers/hwtracing/coresight/coresight.c | 11 +++++++++++
> > 1 file changed, 11 insertions(+)
> >
> >diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> >index 0c37356..5928886 100644
> >--- a/drivers/hwtracing/coresight/coresight.c
> >+++ b/drivers/hwtracing/coresight/coresight.c
> >@@ -138,6 +138,13 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
> > 			if (ret)
> > 				return ret;
> > 		}
> >+
> >+		/* Add kernel panic callback */
> >+		if (sink_ops(csdev)->panic_cb) {
> >+			ret = coresight_add_panic_cb(csdev);
> >+			if (ret)
> >+				return ret;
> >+		}
> > 		csdev->enable = true;
> > 	}
> >
> >@@ -149,6 +156,10 @@ static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
> > static void coresight_disable_sink(struct coresight_device *csdev)
> > {
> > 	if (atomic_dec_return(csdev->refcnt) == 0) {
> >+		/* Remove kernel panic callback */
> >+		if (sink_ops(csdev)->panic_cb)
> >+			coresight_del_panic_cb(csdev);
> >+
> > 		if (sink_ops(csdev)->disable) {
> > 			sink_ops(csdev)->disable(csdev);
> > 			csdev->enable = false;
> >
> 
> Just a thought, instead of adding/deleting every time the sink is enabled,
> could we add/del the device once for-ever and let the panic_cb decide to dump
> it based on whether the device was active or not ?

Yeah, we can check 'if (drvdata->mode == CS_MODE_DISABLED)' and
directly bail out if the module is disabled for panic dumping. Will
fix with this way.

Thanks,
Leo Yan

Re: [PATCH v1 1/4] coresight: support panic dump functionality

[Mathieu Poirier]

On 3 June 2017 at 08:42, Leo Yan <leo.yan@linaro.org> wrote:
> After kernel panic happens, coresight has many useful info can be used
> for analysis. For example, the trace info from ETB RAM can be used to
> check the CPU execution flows before crash. So we can save the tracing
> data from sink devices, and rely on kdump to extract them from vmcore
> file.
>
> This patch is to add a simple framework to support panic dump
> functionality; it registers panic notifier, and provide the general APIs
> {coresight_add_panic_cb|coresight_del_panic_cb} as helper functions so
> any coresight device can add itself into dump list or delete as needed;
> usually these two functions can be used when a session is started or
> when it ends. When kernel panic happened, the panic notifier iterates
> dump list and calls every node for the device callback function to dump
> device specific info. Generally all the panic dump specific stuff are
> related to the sinks devices, so this initial version code it only
> supports sink devices.
>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>

Hi Leo,

> ---
>  drivers/hwtracing/coresight/Kconfig                |  10 ++
>  drivers/hwtracing/coresight/Makefile               |   1 +
>  drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
>  drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
>  include/linux/coresight.h                          |   2 +
>  5 files changed, 153 insertions(+)
>  create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c
>
> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig
> index 8d55d6d..8890023 100644
> --- a/drivers/hwtracing/coresight/Kconfig
> +++ b/drivers/hwtracing/coresight/Kconfig
> @@ -103,4 +103,14 @@ config CORESIGHT_CPU_DEBUG
>           properly, please refer Documentation/trace/coresight-cpu-debug.txt
>           for detailed description and the example for usage.
>
> +config CORESIGHT_PANIC_DUMP
> +       bool "CoreSight Panic Dump driver"
> +       depends on ARM || ARM64
> +       depends on CORESIGHT_LINKS_AND_SINKS
> +       help
> +         This driver provides panic dump functionality for coresight
> +         devices; when kernel panic happens, we can use callback functions
> +         to save trace data to memory. Finally can rely on kdump to extract
> +         out these trace data from kernel dump file.

This driver provides panic dump functionality for CoreSight
devices.  When a kernel panic happen a device supplied callback function
is used to save trace data to memory. From there we rely on kdump to extract
the trace data from kernel dump file.



> +
>  endif
> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile
> index 433d590..0ac3216 100644
> --- a/drivers/hwtracing/coresight/Makefile
> +++ b/drivers/hwtracing/coresight/Makefile
> @@ -17,3 +17,4 @@ obj-$(CONFIG_CORESIGHT_SOURCE_ETM4X) += coresight-etm4x.o \
>  obj-$(CONFIG_CORESIGHT_QCOM_REPLICATOR) += coresight-replicator-qcom.o
>  obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o
>  obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o
> +obj-$(CONFIG_CORESIGHT_PANIC_DUMP) += coresight-panic-dump.o
> diff --git a/drivers/hwtracing/coresight/coresight-panic-dump.c b/drivers/hwtracing/coresight/coresight-panic-dump.c
> new file mode 100644
> index 0000000..23869ff
> --- /dev/null
> +++ b/drivers/hwtracing/coresight/coresight-panic-dump.c
> @@ -0,0 +1,130 @@
> +/*
> + * Copyright(C) 2017 Linaro Limited. All rights reserved.
> + * Author: Leo Yan <leo.yan@linaro.org>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published by
> + * the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/coresight.h>
> +#include <linux/coresight-pmu.h>
> +#include <linux/cpumask.h>
> +#include <linux/device.h>
> +#include <linux/list.h>
> +#include <linux/mm.h>
> +#include <linux/init.h>

Alphabetical order if possible.

> +#include <linux/perf_event.h>
> +#include <linux/slab.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +static DEFINE_MUTEX(coresight_panic_lock);
> +static struct list_head coresight_panic_list;
> +static struct notifier_block coresight_panic_nb;
> +
> +struct coresight_panic_node {
> +       char *name;

There is no need for another copy of the device name as it is already
available in csdev->dev.

> +       struct coresight_device *csdev;
> +       struct list_head list;
> +};
> +
> +static int coresight_panic_notify(struct notifier_block *nb,
> +                                 unsigned long mode, void *_unused)
> +{
> +       int ret = 0, err;
> +       struct coresight_panic_node *node;
> +       struct coresight_device *csdev;
> +       u32 type;
> +
> +       mutex_lock(&coresight_panic_lock);
> +
> +       list_for_each_entry(node, &coresight_panic_list, list) {
> +               csdev = node->csdev;
> +               type = csdev->type;
> +
> +               dev_info(&csdev->dev, "invoke panic dump...\n");
> +
> +               switch (type) {
> +               case CORESIGHT_DEV_TYPE_SINK:
> +               case CORESIGHT_DEV_TYPE_LINKSINK:
> +                       err = sink_ops(csdev)->panic_cb(csdev);
> +                       if (err)
> +                               ret = err;
> +                       break;
> +               default:
> +                       dev_err(&csdev->dev,
> +                               "Unsupported type for panic dump\n");
> +                       break;
> +               }
> +       }
> +
> +       mutex_unlock(&coresight_panic_lock);
> +       return ret;
> +}

This should be called in coresight_register() if a csdev has a  panic
callback.  I'm backing Suzuki's suggestion of executing the callbacks
only if a device has been enabled.  That way the only thing CS devices
have to do is provide a panic_cb function.

> +
> +int coresight_add_panic_cb(struct coresight_device *csdev)
> +{
> +       struct coresight_panic_node *node;
> +
> +       node = kzalloc(sizeof(struct coresight_panic_node), GFP_KERNEL);

The trend in the kernel is so use "sizeof(*node)"

> +       if (!node)
> +               return -ENOMEM;
> +
> +       node->name = kstrndup(dev_name(&csdev->dev), 16, GFP_KERNEL);
> +       if (!node->name) {
> +               kfree(node);
> +               return -ENOMEM;
> +       }
> +       node->csdev = csdev;
> +
> +       mutex_lock(&coresight_panic_lock);
> +       list_add_tail(&node->list, &coresight_panic_list);
> +       mutex_unlock(&coresight_panic_lock);
> +
> +       return 0;
> +}

Same as above.

> +
> +void coresight_del_panic_cb(struct coresight_device *csdev)
> +{
> +       struct coresight_panic_node *node;
> +
> +       mutex_lock(&coresight_panic_lock);
> +
> +       list_for_each_entry(node, &coresight_panic_list, list) {
> +               if (node->csdev == csdev) {
> +                       list_del(&node->list);
> +                       kfree(node->name);
> +                       kfree(node);
> +                       mutex_unlock(&coresight_panic_lock);
> +                       return;
> +               }
> +       }
> +
> +       dev_err(&csdev->dev, "Failed to find panic node.\n");
> +       mutex_unlock(&coresight_panic_lock);
> +}
> +
> +static int __init coresight_panic_init(void)
> +{

This function should go in function coresight_init().

> +       int ret;
> +
> +       INIT_LIST_HEAD(&coresight_panic_list);
> +
> +       coresight_panic_nb.notifier_call = coresight_panic_notify;
> +       ret = atomic_notifier_chain_register(&panic_notifier_list,
> +                                            &coresight_panic_nb);
> +       if (ret)
> +               return ret;
> +
> +       return 0;
> +}
> +subsys_initcall(coresight_panic_init);
> diff --git a/drivers/hwtracing/coresight/coresight-priv.h b/drivers/hwtracing/coresight/coresight-priv.h
> index 5f662d8..e6ed3e9 100644
> --- a/drivers/hwtracing/coresight/coresight-priv.h
> +++ b/drivers/hwtracing/coresight/coresight-priv.h
> @@ -122,4 +122,14 @@ static inline int etm_readl_cp14(u32 off, unsigned int *val) { return 0; }
>  static inline int etm_writel_cp14(u32 off, u32 val) { return 0; }
>  #endif
>
> +#ifdef CONFIG_CORESIGHT_PANIC_DUMP
> +extern int coresight_add_panic_cb(struct coresight_device *csdev);
> +extern void coresight_del_panic_cb(struct coresight_device *csdev);
> +#else
> +static inline int coresight_add_panic_cb(struct coresight_device *csdev)
> +{ return 0; }
> +static inline void coresight_del_panic_cb(struct coresight_device *csdev)
> +{ return; }
> +#endif
> +
>  #endif
> diff --git a/include/linux/coresight.h b/include/linux/coresight.h
> index d950dad..31fcaeb 100644
> --- a/include/linux/coresight.h
> +++ b/include/linux/coresight.h
> @@ -189,6 +189,7 @@ struct coresight_device {
>   * @set_buffer:                initialises buffer mechanic before a trace session.
>   * @reset_buffer:      finalises buffer mechanic after a trace session.
>   * @update_buffer:     update buffer pointers after a trace session.
> + * @panic_cb:          hooks callback function for panic notifier.

"hook function for panic notifier"

Thanks,
Mathieu

>   */
>  struct coresight_ops_sink {
>         int (*enable)(struct coresight_device *csdev, u32 mode);
> @@ -205,6 +206,7 @@ struct coresight_ops_sink {
>         void (*update_buffer)(struct coresight_device *csdev,
>                               struct perf_output_handle *handle,
>                               void *sink_config);
> +       int (*panic_cb)(struct coresight_device *csdev);
>  };
>
>  /**
> --
> 2.7.4
>

[PATCH v1 1/4] coresight: support panic dump functionality

[Mathieu Poirier]

On 3 June 2017 at 08:42, Leo Yan <leo.yan@linaro.org> wrote:
> After kernel panic happens, coresight has many useful info can be used
> for analysis. For example, the trace info from ETB RAM can be used to
> check the CPU execution flows before crash. So we can save the tracing
> data from sink devices, and rely on kdump to extract them from vmcore
> file.
>
> This patch is to add a simple framework to support panic dump
> functionality; it registers panic notifier, and provide the general APIs
> {coresight_add_panic_cb|coresight_del_panic_cb} as helper functions so
> any coresight device can add itself into dump list or delete as needed;
> usually these two functions can be used when a session is started or
> when it ends. When kernel panic happened, the panic notifier iterates
> dump list and calls every node for the device callback function to dump
> device specific info. Generally all the panic dump specific stuff are
> related to the sinks devices, so this initial version code it only
> supports sink devices.
>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>

Hi Leo,

> ---
>  drivers/hwtracing/coresight/Kconfig                |  10 ++
>  drivers/hwtracing/coresight/Makefile               |   1 +
>  drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
>  drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
>  include/linux/coresight.h                          |   2 +
>  5 files changed, 153 insertions(+)
>  create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c
>
> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig
> index 8d55d6d..8890023 100644
> --- a/drivers/hwtracing/coresight/Kconfig
> +++ b/drivers/hwtracing/coresight/Kconfig
> @@ -103,4 +103,14 @@ config CORESIGHT_CPU_DEBUG
>           properly, please refer Documentation/trace/coresight-cpu-debug.txt
>           for detailed description and the example for usage.
>
> +config CORESIGHT_PANIC_DUMP
> +       bool "CoreSight Panic Dump driver"
> +       depends on ARM || ARM64
> +       depends on CORESIGHT_LINKS_AND_SINKS
> +       help
> +         This driver provides panic dump functionality for coresight
> +         devices; when kernel panic happens, we can use callback functions
> +         to save trace data to memory. Finally can rely on kdump to extract
> +         out these trace data from kernel dump file.

This driver provides panic dump functionality for CoreSight
devices.  When a kernel panic happen a device supplied callback function
is used to save trace data to memory. From there we rely on kdump to extract
the trace data from kernel dump file.



> +
>  endif
> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile
> index 433d590..0ac3216 100644
> --- a/drivers/hwtracing/coresight/Makefile
> +++ b/drivers/hwtracing/coresight/Makefile
> @@ -17,3 +17,4 @@ obj-$(CONFIG_CORESIGHT_SOURCE_ETM4X) += coresight-etm4x.o \
>  obj-$(CONFIG_CORESIGHT_QCOM_REPLICATOR) += coresight-replicator-qcom.o
>  obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o
>  obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o
> +obj-$(CONFIG_CORESIGHT_PANIC_DUMP) += coresight-panic-dump.o
> diff --git a/drivers/hwtracing/coresight/coresight-panic-dump.c b/drivers/hwtracing/coresight/coresight-panic-dump.c
> new file mode 100644
> index 0000000..23869ff
> --- /dev/null
> +++ b/drivers/hwtracing/coresight/coresight-panic-dump.c
> @@ -0,0 +1,130 @@
> +/*
> + * Copyright(C) 2017 Linaro Limited. All rights reserved.
> + * Author: Leo Yan <leo.yan@linaro.org>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published by
> + * the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/coresight.h>
> +#include <linux/coresight-pmu.h>
> +#include <linux/cpumask.h>
> +#include <linux/device.h>
> +#include <linux/list.h>
> +#include <linux/mm.h>
> +#include <linux/init.h>

Alphabetical order if possible.

> +#include <linux/perf_event.h>
> +#include <linux/slab.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +static DEFINE_MUTEX(coresight_panic_lock);
> +static struct list_head coresight_panic_list;
> +static struct notifier_block coresight_panic_nb;
> +
> +struct coresight_panic_node {
> +       char *name;

There is no need for another copy of the device name as it is already
available in csdev->dev.

> +       struct coresight_device *csdev;
> +       struct list_head list;
> +};
> +
> +static int coresight_panic_notify(struct notifier_block *nb,
> +                                 unsigned long mode, void *_unused)
> +{
> +       int ret = 0, err;
> +       struct coresight_panic_node *node;
> +       struct coresight_device *csdev;
> +       u32 type;
> +
> +       mutex_lock(&coresight_panic_lock);
> +
> +       list_for_each_entry(node, &coresight_panic_list, list) {
> +               csdev = node->csdev;
> +               type = csdev->type;
> +
> +               dev_info(&csdev->dev, "invoke panic dump...\n");
> +
> +               switch (type) {
> +               case CORESIGHT_DEV_TYPE_SINK:
> +               case CORESIGHT_DEV_TYPE_LINKSINK:
> +                       err = sink_ops(csdev)->panic_cb(csdev);
> +                       if (err)
> +                               ret = err;
> +                       break;
> +               default:
> +                       dev_err(&csdev->dev,
> +                               "Unsupported type for panic dump\n");
> +                       break;
> +               }
> +       }
> +
> +       mutex_unlock(&coresight_panic_lock);
> +       return ret;
> +}

This should be called in coresight_register() if a csdev has a  panic
callback.  I'm backing Suzuki's suggestion of executing the callbacks
only if a device has been enabled.  That way the only thing CS devices
have to do is provide a panic_cb function.

> +
> +int coresight_add_panic_cb(struct coresight_device *csdev)
> +{
> +       struct coresight_panic_node *node;
> +
> +       node = kzalloc(sizeof(struct coresight_panic_node), GFP_KERNEL);

The trend in the kernel is so use "sizeof(*node)"

> +       if (!node)
> +               return -ENOMEM;
> +
> +       node->name = kstrndup(dev_name(&csdev->dev), 16, GFP_KERNEL);
> +       if (!node->name) {
> +               kfree(node);
> +               return -ENOMEM;
> +       }
> +       node->csdev = csdev;
> +
> +       mutex_lock(&coresight_panic_lock);
> +       list_add_tail(&node->list, &coresight_panic_list);
> +       mutex_unlock(&coresight_panic_lock);
> +
> +       return 0;
> +}

Same as above.

> +
> +void coresight_del_panic_cb(struct coresight_device *csdev)
> +{
> +       struct coresight_panic_node *node;
> +
> +       mutex_lock(&coresight_panic_lock);
> +
> +       list_for_each_entry(node, &coresight_panic_list, list) {
> +               if (node->csdev == csdev) {
> +                       list_del(&node->list);
> +                       kfree(node->name);
> +                       kfree(node);
> +                       mutex_unlock(&coresight_panic_lock);
> +                       return;
> +               }
> +       }
> +
> +       dev_err(&csdev->dev, "Failed to find panic node.\n");
> +       mutex_unlock(&coresight_panic_lock);
> +}
> +
> +static int __init coresight_panic_init(void)
> +{

This function should go in function coresight_init().

> +       int ret;
> +
> +       INIT_LIST_HEAD(&coresight_panic_list);
> +
> +       coresight_panic_nb.notifier_call = coresight_panic_notify;
> +       ret = atomic_notifier_chain_register(&panic_notifier_list,
> +                                            &coresight_panic_nb);
> +       if (ret)
> +               return ret;
> +
> +       return 0;
> +}
> +subsys_initcall(coresight_panic_init);
> diff --git a/drivers/hwtracing/coresight/coresight-priv.h b/drivers/hwtracing/coresight/coresight-priv.h
> index 5f662d8..e6ed3e9 100644
> --- a/drivers/hwtracing/coresight/coresight-priv.h
> +++ b/drivers/hwtracing/coresight/coresight-priv.h
> @@ -122,4 +122,14 @@ static inline int etm_readl_cp14(u32 off, unsigned int *val) { return 0; }
>  static inline int etm_writel_cp14(u32 off, u32 val) { return 0; }
>  #endif
>
> +#ifdef CONFIG_CORESIGHT_PANIC_DUMP
> +extern int coresight_add_panic_cb(struct coresight_device *csdev);
> +extern void coresight_del_panic_cb(struct coresight_device *csdev);
> +#else
> +static inline int coresight_add_panic_cb(struct coresight_device *csdev)
> +{ return 0; }
> +static inline void coresight_del_panic_cb(struct coresight_device *csdev)
> +{ return; }
> +#endif
> +
>  #endif
> diff --git a/include/linux/coresight.h b/include/linux/coresight.h
> index d950dad..31fcaeb 100644
> --- a/include/linux/coresight.h
> +++ b/include/linux/coresight.h
> @@ -189,6 +189,7 @@ struct coresight_device {
>   * @set_buffer:                initialises buffer mechanic before a trace session.
>   * @reset_buffer:      finalises buffer mechanic after a trace session.
>   * @update_buffer:     update buffer pointers after a trace session.
> + * @panic_cb:          hooks callback function for panic notifier.

"hook function for panic notifier"

Thanks,
Mathieu

>   */
>  struct coresight_ops_sink {
>         int (*enable)(struct coresight_device *csdev, u32 mode);
> @@ -205,6 +206,7 @@ struct coresight_ops_sink {
>         void (*update_buffer)(struct coresight_device *csdev,
>                               struct perf_output_handle *handle,
>                               void *sink_config);
> +       int (*panic_cb)(struct coresight_device *csdev);
>  };
>
>  /**
> --
> 2.7.4
>

Re: [PATCH v1 0/4] coresight: support panic dump functionality

[Mathieu Poirier]

On 5 June 2017 at 02:57, Suzuki K Poulose <Suzuki.Poulose@arm.com> wrote:
> On 03/06/17 15:42, Leo Yan wrote:
>>
>> ### Introduction ###
>>
>> Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
>> usually has buffer size from 2KB to 8KB. These data has been used for
>> profiling and this has been well implemented in coresight driver.
>>
>> This patch set is to explore ETB RAM data for postmortem debugging.
>> We could consider ETB RAM data is quite useful for postmortem debugging,
>> especially if the hardware design with local ETB buffer (ARM DDI 0461B)
>> chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
>> dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
>> dump the hang CPU ETB RAM. Then we can quickly get to know what's the
>> exact execution flow before its hang.
>>
>> Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
>> then the trace data for causing error is easily to be overwritten by
>> other PEs; but even so sometimes we still have chance to go through the
>> trace data to assist debugging panic issues.
>>
>> ### Implementation ###
>>
>> Firstly we need provide a unified APIs for panic dump functionality, so
>> it can be easily extended to enable panic dump for multiple drivers. This
>> is finished by patch 0001, it registers panic notifier, and provide the
>> general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
>> functions so any coresight device can add into dump list or delete itself
>> as needed.
>>
>> Generally all the panic dump specific stuff are related to the sinks
>> devices, so this initial version code it only supports sink devices; and
>> Patch 0002 is to add and remove panic callback for sink devices.
>>
>> Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
>> drivers; so these two drivers can save specific trace data when panic
>> happens.
>>
>> NOTE: patch 0003 for tmc driver panic callback which has been verified on
>> Hikey board. patch 0004 for etb10 has not been tested due lack hardware
>> in hand.
>>
>
>> - After kernel panic happens, the kdump launches dump-capture kernel;
>>   so we need save kernel's dump file on target:
>>   cp /proc/vmcore ./vmcore
>
>
>
>>   After we download vmcore file from Hikey board to host PC, we can
>>   use 'crash' tool to check coresight dump info and extract trace data:
>>   crash vmlinux vmcore
>>   crash> log
>>   [   37.559337] coresight f6402000.etf: invoke panic dump...
>>   [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer
>> 0x2000@0xffff80003b8da180
>>   crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
>>
>
> Have you explored appending the above information as a vmcoreinfo parameter
> via
> vmcoreinfo_append_str() ? That would make it easier to list all the
> information
> above and if needed, we may be able to extend the makedumpfile to dump the
> ETB
> dumps from a given vmcore.
>  Suzuki

One thing this patchset doesn't address is the tracer configuration
(metadata).  I'm thinking we can use the same mechanism to store the
relevant information in memory in the same format done for perf.

[PATCH v1 0/4] coresight: support panic dump functionality

[Mathieu Poirier]

On 5 June 2017 at 02:57, Suzuki K Poulose <Suzuki.Poulose@arm.com> wrote:
> On 03/06/17 15:42, Leo Yan wrote:
>>
>> ### Introduction ###
>>
>> Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
>> usually has buffer size from 2KB to 8KB. These data has been used for
>> profiling and this has been well implemented in coresight driver.
>>
>> This patch set is to explore ETB RAM data for postmortem debugging.
>> We could consider ETB RAM data is quite useful for postmortem debugging,
>> especially if the hardware design with local ETB buffer (ARM DDI 0461B)
>> chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
>> dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
>> dump the hang CPU ETB RAM. Then we can quickly get to know what's the
>> exact execution flow before its hang.
>>
>> Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
>> then the trace data for causing error is easily to be overwritten by
>> other PEs; but even so sometimes we still have chance to go through the
>> trace data to assist debugging panic issues.
>>
>> ### Implementation ###
>>
>> Firstly we need provide a unified APIs for panic dump functionality, so
>> it can be easily extended to enable panic dump for multiple drivers. This
>> is finished by patch 0001, it registers panic notifier, and provide the
>> general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
>> functions so any coresight device can add into dump list or delete itself
>> as needed.
>>
>> Generally all the panic dump specific stuff are related to the sinks
>> devices, so this initial version code it only supports sink devices; and
>> Patch 0002 is to add and remove panic callback for sink devices.
>>
>> Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
>> drivers; so these two drivers can save specific trace data when panic
>> happens.
>>
>> NOTE: patch 0003 for tmc driver panic callback which has been verified on
>> Hikey board. patch 0004 for etb10 has not been tested due lack hardware
>> in hand.
>>
>
>> - After kernel panic happens, the kdump launches dump-capture kernel;
>>   so we need save kernel's dump file on target:
>>   cp /proc/vmcore ./vmcore
>
>
>
>>   After we download vmcore file from Hikey board to host PC, we can
>>   use 'crash' tool to check coresight dump info and extract trace data:
>>   crash vmlinux vmcore
>>   crash> log
>>   [   37.559337] coresight f6402000.etf: invoke panic dump...
>>   [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer
>> 0x2000 at 0xffff80003b8da180
>>   crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
>>
>
> Have you explored appending the above information as a vmcoreinfo parameter
> via
> vmcoreinfo_append_str() ? That would make it easier to list all the
> information
> above and if needed, we may be able to extend the makedumpfile to dump the
> ETB
> dumps from a given vmcore.
>  Suzuki

One thing this patchset doesn't address is the tracer configuration
(metadata).  I'm thinking we can use the same mechanism to store the
relevant information in memory in the same format done for perf.

Re: [PATCH v1 0/4] coresight: support panic dump functionality

[Mathieu Poirier]

On 3 June 2017 at 08:42, Leo Yan <leo.yan@linaro.org> wrote:
> ### Introduction ###

Good day Leo,

>
> Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
> usually has buffer size from 2KB to 8KB. These data has been used for
> profiling and this has been well implemented in coresight driver.
>
> This patch set is to explore ETB RAM data for postmortem debugging.
> We could consider ETB RAM data is quite useful for postmortem debugging,
> especially if the hardware design with local ETB buffer (ARM DDI 0461B)
> chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
> dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
> dump the hang CPU ETB RAM. Then we can quickly get to know what's the
> exact execution flow before its hang.
>
> Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
> then the trace data for causing error is easily to be overwritten by
> other PEs; but even so sometimes we still have chance to go through the
> trace data to assist debugging panic issues.
>
> ### Implementation ###
>
> Firstly we need provide a unified APIs for panic dump functionality, so
> it can be easily extended to enable panic dump for multiple drivers. This
> is finished by patch 0001, it registers panic notifier, and provide the
> general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
> functions so any coresight device can add into dump list or delete itself
> as needed.
>
> Generally all the panic dump specific stuff are related to the sinks
> devices, so this initial version code it only supports sink devices; and
> Patch 0002 is to add and remove panic callback for sink devices.
>
> Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
> drivers; so these two drivers can save specific trace data when panic
> happens.
>
> NOTE: patch 0003 for tmc driver panic callback which has been verified on
> Hikey board. patch 0004 for etb10 has not been tested due lack hardware
> in hand.
>
> ### Usage ###

On top of my comments in the patches I think this section is
interesting and worth its own text file under Documentation.  We
already have coresight.txt and coresight-cpu-debug.txt...  As such I
suggest you add a new "coresight" directory under Documentation/trace
and move coresight.txt and coresight-cpu-debug.txt there.  Once that
is done you can add coresight-panic-dump.txt there.

>
> Below are the example for how to use panic dump functionality on 96boards
> Hikey, the brief flow is: when the panic happens the ETB panic callback
> function saves trace data into memory, then relies on kdump to use
> recovery kernel to save DDR content as kernel core dump file; after we
> transfer kernel core dump file from board to host PC, use 'crash' tool to
> extract the coresight ETB trace data; finally we can use python script
> to generate perf format compatible file and use 'perf' to output the
> readable execution flow.
>
> - Save trace data into memory with kdump on Hikey:
>
>   ARM64's kdump supports to use the same kernel image both for main
>   kernel and dump-capture kernel; so we can simply to load dump-capture
>   kernel with below command:
>   ./kexec -p vmlinux --dtb=hi6220-hikey.dtb --append="root=/dev/mmcblk0p9
>   rw  maxcpus=1 reset_devices earlycon=pl011,0xf7113000 nohlt
>   initcall_debug console=tty0 console=ttyAMA3,115200 clk_ignore_unused"
>
>   Enable the coresight path for ETB device:
>   echo 1 > /sys/bus/coresight/devices/f6402000.etf/enable_sink
>   echo 1 > /sys/bus/coresight/devices/f659c000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f659d000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f659e000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f659f000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65dc000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65dd000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65de000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65df000.etm/enable_source
>
> - After kernel panic happens, the kdump launches dump-capture kernel;
>   so we need save kernel's dump file on target:
>   cp /proc/vmcore ./vmcore
>
>   After we download vmcore file from Hikey board to host PC, we can
>   use 'crash' tool to check coresight dump info and extract trace data:
>   crash vmlinux vmcore
>   crash> log
>   [   37.559337] coresight f6402000.etf: invoke panic dump...
>   [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000@0xffff80003b8da180
>   crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
>
> - Use python script perf_cs_dump_wrapper.py to wrap trace data for
>   perf format compatible file and finally use perf to output CPU
>   execution flow:
>
>   On host PC run python script, please note now this script is not flexbile
>   to support all kinds of coresight topologies, this script still has hard coded
>   info related with coresight specific topology in Hikey:
>   python perf_cs_dump_wrapper.py -i cs_etb_trace.bin -o perf.data

I'm not sure what we'll do with "perf_cs_dump_wrapper.py" yet...  I
suspect openCSD on github will be a good place for it but let's see
about that later.

Regards,
Mathieu

>
>   On Hikey board:
>   ./perf script -v -F cpu,event,ip,sym,symoff --kallsyms ksymbol -i perf.data -k vmlinux
>
>   [002]          instructions:  ffff0000087d1d60 psci_cpu_suspend_enter+0x48
>   [002]          instructions:  ffff000008093400 cpu_suspend+0x0
>   [002]          instructions:  ffff000008093210 __cpu_suspend_enter+0x0
>   [002]          instructions:  ffff000008099970 cpu_do_suspend+0x0
>   [002]          instructions:  ffff000008093294 __cpu_suspend_enter+0x84
>   [002]          instructions:  ffff000008093428 cpu_suspend+0x28
>   [002]          instructions:  ffff00000809342c cpu_suspend+0x2c
>   [002]          instructions:  ffff0000087d1968 psci_suspend_finisher+0x0
>   [002]          instructions:  ffff0000087d1768 psci_cpu_suspend+0x0
>   [002]          instructions:  ffff0000087d19f0 __invoke_psci_fn_smc+0x0
>
> Have uploaded related tools into folder:
> http://people.linaro.org/~leo.yan/debug/coresight_dump/
>
> Changes from RFC:
> * Follow Mathieu's suggestion, use general framework to support dump
>   functionality.
> * Changed to use perf to analyse trace data.
>
> Leo Yan (4):
>   coresight: support panic dump functionality
>   coresight: add and remove panic callback for sink
>   coresight: tmc: hook panic callback for ETB/ETF
>   coresight: etb10: hook panic callback
>
>  drivers/hwtracing/coresight/Kconfig                |  10 ++
>  drivers/hwtracing/coresight/Makefile               |   1 +
>  drivers/hwtracing/coresight/coresight-etb10.c      |  16 +++
>  drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
>  drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
>  drivers/hwtracing/coresight/coresight-tmc-etf.c    |  26 +++++
>  drivers/hwtracing/coresight/coresight.c            |  11 ++
>  include/linux/coresight.h                          |   2 +
>  8 files changed, 206 insertions(+)
>  create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c
>
> --
> 2.7.4
>

[PATCH v1 0/4] coresight: support panic dump functionality

[Mathieu Poirier]

On 3 June 2017 at 08:42, Leo Yan <leo.yan@linaro.org> wrote:
> ### Introduction ###

Good day Leo,

>
> Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
> usually has buffer size from 2KB to 8KB. These data has been used for
> profiling and this has been well implemented in coresight driver.
>
> This patch set is to explore ETB RAM data for postmortem debugging.
> We could consider ETB RAM data is quite useful for postmortem debugging,
> especially if the hardware design with local ETB buffer (ARM DDI 0461B)
> chapter 1.2.7. 'Local ETF', with this kind design every CPU has one
> dedicated ETB RAM. So it's quite handy that we can use alive CPU to help
> dump the hang CPU ETB RAM. Then we can quickly get to know what's the
> exact execution flow before its hang.
>
> Due ETB RAM buffer has small size, if all CPUs shared one ETB buffer
> then the trace data for causing error is easily to be overwritten by
> other PEs; but even so sometimes we still have chance to go through the
> trace data to assist debugging panic issues.
>
> ### Implementation ###
>
> Firstly we need provide a unified APIs for panic dump functionality, so
> it can be easily extended to enable panic dump for multiple drivers. This
> is finished by patch 0001, it registers panic notifier, and provide the
> general APIs {coresight_add_panic_cb|coresight_del_panic_cb} as helper
> functions so any coresight device can add into dump list or delete itself
> as needed.
>
> Generally all the panic dump specific stuff are related to the sinks
> devices, so this initial version code it only supports sink devices; and
> Patch 0002 is to add and remove panic callback for sink devices.
>
> Patch 0003 and 0004 are to add panic callback functions for tmc and etb10
> drivers; so these two drivers can save specific trace data when panic
> happens.
>
> NOTE: patch 0003 for tmc driver panic callback which has been verified on
> Hikey board. patch 0004 for etb10 has not been tested due lack hardware
> in hand.
>
> ### Usage ###

On top of my comments in the patches I think this section is
interesting and worth its own text file under Documentation.  We
already have coresight.txt and coresight-cpu-debug.txt...  As such I
suggest you add a new "coresight" directory under Documentation/trace
and move coresight.txt and coresight-cpu-debug.txt there.  Once that
is done you can add coresight-panic-dump.txt there.

>
> Below are the example for how to use panic dump functionality on 96boards
> Hikey, the brief flow is: when the panic happens the ETB panic callback
> function saves trace data into memory, then relies on kdump to use
> recovery kernel to save DDR content as kernel core dump file; after we
> transfer kernel core dump file from board to host PC, use 'crash' tool to
> extract the coresight ETB trace data; finally we can use python script
> to generate perf format compatible file and use 'perf' to output the
> readable execution flow.
>
> - Save trace data into memory with kdump on Hikey:
>
>   ARM64's kdump supports to use the same kernel image both for main
>   kernel and dump-capture kernel; so we can simply to load dump-capture
>   kernel with below command:
>   ./kexec -p vmlinux --dtb=hi6220-hikey.dtb --append="root=/dev/mmcblk0p9
>   rw  maxcpus=1 reset_devices earlycon=pl011,0xf7113000 nohlt
>   initcall_debug console=tty0 console=ttyAMA3,115200 clk_ignore_unused"
>
>   Enable the coresight path for ETB device:
>   echo 1 > /sys/bus/coresight/devices/f6402000.etf/enable_sink
>   echo 1 > /sys/bus/coresight/devices/f659c000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f659d000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f659e000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f659f000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65dc000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65dd000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65de000.etm/enable_source
>   echo 1 > /sys/bus/coresight/devices/f65df000.etm/enable_source
>
> - After kernel panic happens, the kdump launches dump-capture kernel;
>   so we need save kernel's dump file on target:
>   cp /proc/vmcore ./vmcore
>
>   After we download vmcore file from Hikey board to host PC, we can
>   use 'crash' tool to check coresight dump info and extract trace data:
>   crash vmlinux vmcore
>   crash> log
>   [   37.559337] coresight f6402000.etf: invoke panic dump...
>   [   37.565460] coresight-tmc f6402000.etf: Dump ETB buffer 0x2000 at 0xffff80003b8da180
>   crash> rd 0xffff80003b8da180 0x2000 -r cs_etb_trace.bin
>
> - Use python script perf_cs_dump_wrapper.py to wrap trace data for
>   perf format compatible file and finally use perf to output CPU
>   execution flow:
>
>   On host PC run python script, please note now this script is not flexbile
>   to support all kinds of coresight topologies, this script still has hard coded
>   info related with coresight specific topology in Hikey:
>   python perf_cs_dump_wrapper.py -i cs_etb_trace.bin -o perf.data

I'm not sure what we'll do with "perf_cs_dump_wrapper.py" yet...  I
suspect openCSD on github will be a good place for it but let's see
about that later.

Regards,
Mathieu

>
>   On Hikey board:
>   ./perf script -v -F cpu,event,ip,sym,symoff --kallsyms ksymbol -i perf.data -k vmlinux
>
>   [002]          instructions:  ffff0000087d1d60 psci_cpu_suspend_enter+0x48
>   [002]          instructions:  ffff000008093400 cpu_suspend+0x0
>   [002]          instructions:  ffff000008093210 __cpu_suspend_enter+0x0
>   [002]          instructions:  ffff000008099970 cpu_do_suspend+0x0
>   [002]          instructions:  ffff000008093294 __cpu_suspend_enter+0x84
>   [002]          instructions:  ffff000008093428 cpu_suspend+0x28
>   [002]          instructions:  ffff00000809342c cpu_suspend+0x2c
>   [002]          instructions:  ffff0000087d1968 psci_suspend_finisher+0x0
>   [002]          instructions:  ffff0000087d1768 psci_cpu_suspend+0x0
>   [002]          instructions:  ffff0000087d19f0 __invoke_psci_fn_smc+0x0
>
> Have uploaded related tools into folder:
> http://people.linaro.org/~leo.yan/debug/coresight_dump/
>
> Changes from RFC:
> * Follow Mathieu's suggestion, use general framework to support dump
>   functionality.
> * Changed to use perf to analyse trace data.
>
> Leo Yan (4):
>   coresight: support panic dump functionality
>   coresight: add and remove panic callback for sink
>   coresight: tmc: hook panic callback for ETB/ETF
>   coresight: etb10: hook panic callback
>
>  drivers/hwtracing/coresight/Kconfig                |  10 ++
>  drivers/hwtracing/coresight/Makefile               |   1 +
>  drivers/hwtracing/coresight/coresight-etb10.c      |  16 +++
>  drivers/hwtracing/coresight/coresight-panic-dump.c | 130 +++++++++++++++++++++
>  drivers/hwtracing/coresight/coresight-priv.h       |  10 ++
>  drivers/hwtracing/coresight/coresight-tmc-etf.c    |  26 +++++
>  drivers/hwtracing/coresight/coresight.c            |  11 ++
>  include/linux/coresight.h                          |   2 +
>  8 files changed, 206 insertions(+)
>  create mode 100644 drivers/hwtracing/coresight/coresight-panic-dump.c
>
> --
> 2.7.4
>