linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH RFC 0/4] coresight: support dump ETB RAM
@ 2017-04-11  9:10 Leo Yan
  2017-04-11  9:10 ` [PATCH RFC 1/4] coresight: tmc: check dump buffer is overflow Leo Yan
                   ` (4 more replies)
  0 siblings, 5 replies; 8+ messages in thread
From: Leo Yan @ 2017-04-11  9:10 UTC (permalink / raw)
  To: Mathieu Poirier, linux-arm-kernel, linux-kernel; +Cc: Leo Yan


### Introduction ###

Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
usually has buffer size from 2KB to 8KB. These data has been used for
profiling and this has been well implemented in coresight driver.

This patch is to explore ETB RAM data for postmortem debugging. Due ETB
RAM buffer has small size, so the real trace data caused error is
easily to be overwritten by other PEs; but we could consider ETB RAM
data is quite useful for postmortem debugging with below scenarios:

Case 1: if system is bus lockup and CPU pipeline stalls for bus
accessing, CPUs have no more chance to fill enough data into ETB RAM
so after analyze ETB RAM we can quickly get to know the culprit if bus
lock is caused by improper programs, one often example is wrongly to
access the module without enable the module's clock. For this case,
we can rely on watchdog to trigger SoC reset and if lucky the ETB RAM
can survive after reset. So for this case, after system reboot we can
save ETB RAM before any new data input into it.

Case 2: There also has another hardware design with local ETB buffer
(ARM DDI 0461B) chapter 1.2.7. Local ETF, with this kind design every
CPU may has one dedicated ETB RAM. So it's quite handy that we can use
alive CPU to help dump the hang CPU ETB RAM. Then we can quickly get
to know what's the last point the CPU has executed before its hang.


### Implementation ###

Based on current Coresight ETB driver, we only needs some minor
enhancement so can support dump ETB RAM with two methods.

Patches 0001/0002 are minor fixes so can support more scenarios for ETB
RAM dumping.

Patch 0003 is to dump ETB RAM after system reboot, this is for the
platforms which use watchdog reset and ETB RAM can survive.

Patch 0004 is to dump ETB RAM when panic happens, so we can save ETB RAM
into memory. If we connect this with Kdump, then we can easily extract
the ETB RAM from vmcore.


### Usage ###

To dump ETB RAM after reboot, simply use below command:
# dd if=/dev/f6402000.etf of=cstrace.bin

To dump ETB RAM for kernel panic, we need add "crash_kexec_post_notifiers"
into kernel command line so let kernel call panic notifiers before launch
dump kernel. After dump kernel has booted up, we need use below methods
to ETB RAM offline analysis:

On the target:
# cp /proc/vmcore ./vmcore
# scp ./vmcore your@hostpc

On the host PC:
# ./crash vmcore vmlinux

crash> log
[...]
[  112.600051] coresight-tmc f6402000.etf: Flush ETB buffer 0x2000@0xffff800038300080
[  112.614743] Starting crashdump kernel...
[  112.618681] Bye!
crash> rd 0xffff800038300080 0x2000 -r /tmp/cstrace.bin
8192 bytes copied from 0xffff800038300080 to /tmp/cstrace.bin

After we get cstrace.bin data, we can use OpenCSD snapshot method to parse
ETB trace data. These two methods have been verified on Hikey, For Hikey
snapshot config files you can refer [1]. For total kernel patches for
integration Kdump and Coresight, you can refer [2].

[1] http://people.linaro.org/~leo.yan/opencsd_hikey/hikey_snapshot.tgz
[2] https://git.linaro.org/people/leo.yan/linux-debug-workshop.git/log/?h=coresight_etb_dump


### TODO ###

Need work for ETB1.0 driver, this is based on review and comments
for this patch set.


Leo Yan (4):
  coresight: tmc: check dump buffer is overflow
  coresight: tmc: set read pointer before dump RAM
  coresight: tmc: dump RAM when device is disabled
  coresight: tmc: dump RAM for panic

 drivers/hwtracing/coresight/coresight-tmc-etf.c | 86 ++++++++++++++++++++++++-
 drivers/hwtracing/coresight/coresight-tmc.h     |  2 +
 2 files changed, 85 insertions(+), 3 deletions(-)

-- 
2.7.4

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH RFC 1/4] coresight: tmc: check dump buffer is overflow
  2017-04-11  9:10 [PATCH RFC 0/4] coresight: support dump ETB RAM Leo Yan
@ 2017-04-11  9:10 ` Leo Yan
  2017-04-11  9:10 ` [PATCH RFC 2/4] coresight: tmc: set read pointer before dump RAM Leo Yan
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 8+ messages in thread
From: Leo Yan @ 2017-04-11  9:10 UTC (permalink / raw)
  To: Mathieu Poirier, linux-arm-kernel, linux-kernel
  Cc: Leo Yan, Mike Leach, Suzuki K Poulose

In the code dumping ETB buffer is terminated when read back value
0xFFFF_FFFF. This is not safe and introduce infinite looping for some
cases, e.g. when reset the CPU and read out ETB RAM.

Cc: Mathieu Poirier <mathieu.poirier@linaro.org>
Cc: Mike Leach <mike.leach@linaro.org>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-tmc-etf.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index 3a1c181..6150dac 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -54,6 +54,13 @@ static void tmc_etb_dump_hw(struct tmc_drvdata *drvdata)
 			read_data = readl_relaxed(drvdata->base + TMC_RRD);
 			if (read_data == 0xFFFFFFFF)
 				return;
+
+			/* Check if allocated buffer is overflow */
+			if (drvdata->len >= drvdata->size) {
+				dev_info(drvdata->dev, "TMC dump overflow!\n");
+				return;
+			}
+
 			memcpy(bufp, &read_data, 4);
 			bufp += 4;
 			drvdata->len += 4;
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH RFC 2/4] coresight: tmc: set read pointer before dump RAM
  2017-04-11  9:10 [PATCH RFC 0/4] coresight: support dump ETB RAM Leo Yan
  2017-04-11  9:10 ` [PATCH RFC 1/4] coresight: tmc: check dump buffer is overflow Leo Yan
@ 2017-04-11  9:10 ` Leo Yan
  2017-04-11 11:25   ` Chunyan Zhang
  2017-04-11  9:10 ` [PATCH RFC 3/4] coresight: tmc: dump RAM when device is disabled Leo Yan
                   ` (2 subsequent siblings)
  4 siblings, 1 reply; 8+ messages in thread
From: Leo Yan @ 2017-04-11  9:10 UTC (permalink / raw)
  To: Mathieu Poirier, linux-arm-kernel, linux-kernel
  Cc: Leo Yan, Mike Leach, Suzuki K Poulose

When dump RAM, we need set read pointer so can make sure every time read
the consistent content. If the RAM is full by checking status register
(STS), so set the read pointer to same value with write pointer,
otherwise set read point to 0 so can read from the start of RAM.

Cc: Mathieu Poirier <mathieu.poirier@linaro.org>
Cc: Mike Leach <mike.leach@linaro.org>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-tmc-etf.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index 6150dac..43cfeaa 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -43,10 +43,28 @@ static void tmc_etb_enable_hw(struct tmc_drvdata *drvdata)
 
 static void tmc_etb_dump_hw(struct tmc_drvdata *drvdata)
 {
+	u32 write_ptr, status;
 	char *bufp;
 	u32 read_data;
 	int i;
 
+	write_ptr = readl_relaxed(drvdata->base + TMC_RWP);
+
+	/*
+	 * Get a hold of the status register and see if a wrap around
+	 * has occurred.  If so adjust things accordingly.
+	 */
+	status = readl_relaxed(drvdata->base + TMC_STS);
+	if (status & TMC_STS_FULL)
+		/* Tell the HW the reading start point */
+		writel_relaxed(write_ptr, drvdata->base + TMC_RRP);
+	else
+		/*
+		 * In case this is not first time to read ETB RAM,
+		 * always write 0 for reading pointer.
+		 */
+		writel_relaxed(0x0, drvdata->base + TMC_RRP);
+
 	bufp = drvdata->buf;
 	drvdata->len = 0;
 	while (1) {
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH RFC 3/4] coresight: tmc: dump RAM when device is disabled
  2017-04-11  9:10 [PATCH RFC 0/4] coresight: support dump ETB RAM Leo Yan
  2017-04-11  9:10 ` [PATCH RFC 1/4] coresight: tmc: check dump buffer is overflow Leo Yan
  2017-04-11  9:10 ` [PATCH RFC 2/4] coresight: tmc: set read pointer before dump RAM Leo Yan
@ 2017-04-11  9:10 ` Leo Yan
  2017-04-11  9:10 ` [PATCH RFC 4/4] coresight: tmc: dump RAM for panic Leo Yan
  2017-04-20 17:45 ` [PATCH RFC 0/4] coresight: support dump ETB RAM Mathieu Poirier
  4 siblings, 0 replies; 8+ messages in thread
From: Leo Yan @ 2017-04-11  9:10 UTC (permalink / raw)
  To: Mathieu Poirier, linux-arm-kernel, linux-kernel
  Cc: Leo Yan, Mike Leach, Suzuki K Poulose

Coresight framework creates /dev nodes for ETB/ETF devices, these nodes
cannot read out data if without enable the coresight path. Finally we
lose chance to dump RAM if there have some old data kept in it.

So this patch is to add support to dump RAM when device is disabled,
with this small change we can easily read out ETB RAM data before there
have new data has been sent to it. These dump data may come from the
previous system running, so it will be very helpful to analyze some hang
issue before system dead.

Cc: Mathieu Poirier <mathieu.poirier@linaro.org>
Cc: Mike Leach <mike.leach@linaro.org>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-tmc-etf.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index 43cfeaa..5e709af 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -95,7 +95,7 @@ static void tmc_etb_disable_hw(struct tmc_drvdata *drvdata)
 	 * When operating in sysFS mode the content of the buffer needs to be
 	 * read before the TMC is disabled.
 	 */
-	if (drvdata->mode == CS_MODE_SYSFS)
+	if (drvdata->mode != CS_MODE_PERF)
 		tmc_etb_dump_hw(drvdata);
 	tmc_disable_hw(drvdata);
 
@@ -557,6 +557,10 @@ int tmc_read_prepare_etb(struct tmc_drvdata *drvdata)
 		goto out;
 	}
 
+	/* Try to dump RAM even for disabled state */
+	if (drvdata->mode == CS_MODE_DISABLED)
+		drvdata->buf = kzalloc(drvdata->size, GFP_KERNEL);
+
 	/* If drvdata::buf is NULL the trace data has been read already */
 	if (drvdata->buf == NULL) {
 		ret = -EINVAL;
@@ -564,8 +568,7 @@ int tmc_read_prepare_etb(struct tmc_drvdata *drvdata)
 	}
 
 	/* Disable the TMC if need be */
-	if (drvdata->mode == CS_MODE_SYSFS)
-		tmc_etb_disable_hw(drvdata);
+	tmc_etb_disable_hw(drvdata);
 
 	drvdata->reading = true;
 out:
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH RFC 4/4] coresight: tmc: dump RAM for panic
  2017-04-11  9:10 [PATCH RFC 0/4] coresight: support dump ETB RAM Leo Yan
                   ` (2 preceding siblings ...)
  2017-04-11  9:10 ` [PATCH RFC 3/4] coresight: tmc: dump RAM when device is disabled Leo Yan
@ 2017-04-11  9:10 ` Leo Yan
  2017-04-20 17:45 ` [PATCH RFC 0/4] coresight: support dump ETB RAM Mathieu Poirier
  4 siblings, 0 replies; 8+ messages in thread
From: Leo Yan @ 2017-04-11  9:10 UTC (permalink / raw)
  To: Mathieu Poirier, linux-arm-kernel, linux-kernel
  Cc: Leo Yan, Mike Leach, Suzuki K Poulose

If coresight ETB/ETF have been enabled by 'sysfs' or 'perf' modes and we
can rely on panic notifier to save all ETB RAM data into memory. Relies
on Kdump, finally the ETB RAM data can be extracted from kernel's
vmcore, so we can analyse panic reason from dumped ETB RAM data.

Cc: Mathieu Poirier <mathieu.poirier@linaro.org>
Cc: Mike Leach <mike.leach@linaro.org>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
 drivers/hwtracing/coresight/coresight-tmc-etf.c | 52 +++++++++++++++++++++++++
 drivers/hwtracing/coresight/coresight-tmc.h     |  2 +
 2 files changed, 54 insertions(+)

diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index 5e709af..bc52cad 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -102,6 +102,47 @@ static void tmc_etb_disable_hw(struct tmc_drvdata *drvdata)
 	CS_LOCK(drvdata->base);
 }
 
+static int tmc_etb_notify(struct notifier_block *nb,
+			  unsigned long mode, void *_unused)
+{
+	struct tmc_drvdata *drvdata = container_of(nb, struct tmc_drvdata,
+						   panic_nb);
+	unsigned long flags;
+	int ret = 0;
+
+	if (WARN_ON_ONCE(drvdata->config_type != TMC_CONFIG_TYPE_ETB &&
+			 drvdata->config_type != TMC_CONFIG_TYPE_ETF))
+		return -EINVAL;
+
+	spin_lock_irqsave(&drvdata->spinlock, flags);
+
+	/* There is no point in reading a TMC in HW FIFO mode */
+	mode = readl_relaxed(drvdata->base + TMC_MODE);
+	if (mode != TMC_MODE_CIRCULAR_BUFFER) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* If drvdata::buf is NULL the trace data has been read already */
+	if (drvdata->buf == NULL) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	CS_UNLOCK(drvdata->base);
+
+	tmc_flush_and_stop(drvdata);
+	tmc_etb_dump_hw(drvdata);
+
+	dev_info(drvdata->dev, "Flush ETB buffer 0x%x@0x%p\n",
+			drvdata->len, drvdata->buf);
+
+	CS_LOCK(drvdata->base);
+out:
+	spin_unlock_irqrestore(&drvdata->spinlock, flags);
+	return ret;
+}
+
 static void tmc_etf_enable_hw(struct tmc_drvdata *drvdata)
 {
 	CS_UNLOCK(drvdata->base);
@@ -245,6 +286,12 @@ static int tmc_enable_etf_sink(struct coresight_device *csdev, u32 mode)
 	if (ret)
 		return ret;
 
+	drvdata->panic_nb.notifier_call = tmc_etb_notify;
+	ret = atomic_notifier_chain_register(&panic_notifier_list,
+					     &drvdata->panic_nb);
+	if (ret)
+		return ret;
+
 	dev_info(drvdata->dev, "TMC-ETB/ETF enabled\n");
 	return 0;
 }
@@ -262,6 +309,11 @@ static void tmc_disable_etf_sink(struct coresight_device *csdev)
 
 	/* Disable the TMC only if it needs to */
 	if (drvdata->mode != CS_MODE_DISABLED) {
+		/* Unregister panic notifier */
+		drvdata->panic_nb.notifier_call = NULL;
+		atomic_notifier_chain_unregister(&panic_notifier_list,
+				&drvdata->panic_nb);
+
 		tmc_etb_disable_hw(drvdata);
 		drvdata->mode = CS_MODE_DISABLED;
 	}
diff --git a/drivers/hwtracing/coresight/coresight-tmc.h b/drivers/hwtracing/coresight/coresight-tmc.h
index 51c0185..bb563b8 100644
--- a/drivers/hwtracing/coresight/coresight-tmc.h
+++ b/drivers/hwtracing/coresight/coresight-tmc.h
@@ -93,6 +93,7 @@ enum tmc_mem_intf_width {
  * @base:	memory mapped base address for this component.
  * @dev:	the device entity associated to this component.
  * @csdev:	component vitals needed by the framework.
+ * @panic_nb:	notifier callback for panic.
  * @miscdev:	specifics to handle "/dev/xyz.tmc" entry.
  * @spinlock:	only one at a time pls.
  * @buf:	area of memory where trace data get sent.
@@ -109,6 +110,7 @@ struct tmc_drvdata {
 	void __iomem		*base;
 	struct device		*dev;
 	struct coresight_device	*csdev;
+	struct notifier_block	panic_nb;
 	struct miscdevice	miscdev;
 	spinlock_t		spinlock;
 	bool			reading;
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH RFC 2/4] coresight: tmc: set read pointer before dump RAM
  2017-04-11  9:10 ` [PATCH RFC 2/4] coresight: tmc: set read pointer before dump RAM Leo Yan
@ 2017-04-11 11:25   ` Chunyan Zhang
  2017-04-11 13:46     ` Leo Yan
  0 siblings, 1 reply; 8+ messages in thread
From: Chunyan Zhang @ 2017-04-11 11:25 UTC (permalink / raw)
  To: Leo Yan
  Cc: Mathieu Poirier, linux-arm-kernel, linux-kernel, Mike Leach,
	Suzuki K Poulose

On 11 April 2017 at 17:10, Leo Yan <leo.yan@linaro.org> wrote:
> When dump RAM, we need set read pointer so can make sure every time read
> the consistent content. If the RAM is full by checking status register
> (STS), so set the read pointer to same value with write pointer,
> otherwise set read point to 0 so can read from the start of RAM.
>
> Cc: Mathieu Poirier <mathieu.poirier@linaro.org>
> Cc: Mike Leach <mike.leach@linaro.org>
> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>
> ---
>  drivers/hwtracing/coresight/coresight-tmc-etf.c | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
>
> diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
> index 6150dac..43cfeaa 100644
> --- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
> +++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
> @@ -43,10 +43,28 @@ static void tmc_etb_enable_hw(struct tmc_drvdata *drvdata)
>
>  static void tmc_etb_dump_hw(struct tmc_drvdata *drvdata)
>  {
> +       u32 write_ptr, status;
>         char *bufp;
>         u32 read_data;
>         int i;
>
> +       write_ptr = readl_relaxed(drvdata->base + TMC_RWP);

Since 'write_ptr' is only used in the below 'if' branch, I would
suggest to move this statement into that.

Regards,
Chunyan

> +
> +       /*
> +        * Get a hold of the status register and see if a wrap around
> +        * has occurred.  If so adjust things accordingly.
> +        */
> +       status = readl_relaxed(drvdata->base + TMC_STS);
> +       if (status & TMC_STS_FULL)
> +               /* Tell the HW the reading start point */
> +               writel_relaxed(write_ptr, drvdata->base + TMC_RRP);
> +       else
> +               /*
> +                * In case this is not first time to read ETB RAM,
> +                * always write 0 for reading pointer.
> +                */
> +               writel_relaxed(0x0, drvdata->base + TMC_RRP);
> +
>         bufp = drvdata->buf;
>         drvdata->len = 0;
>         while (1) {
> --
> 2.7.4
>

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH RFC 2/4] coresight: tmc: set read pointer before dump RAM
  2017-04-11 11:25   ` Chunyan Zhang
@ 2017-04-11 13:46     ` Leo Yan
  0 siblings, 0 replies; 8+ messages in thread
From: Leo Yan @ 2017-04-11 13:46 UTC (permalink / raw)
  To: Chunyan Zhang
  Cc: Mathieu Poirier, linux-arm-kernel, linux-kernel, Mike Leach,
	Suzuki K Poulose

On Tue, Apr 11, 2017 at 07:25:30PM +0800, Chunyan Zhang wrote:
> On 11 April 2017 at 17:10, Leo Yan <leo.yan@linaro.org> wrote:
> > When dump RAM, we need set read pointer so can make sure every time read
> > the consistent content. If the RAM is full by checking status register
> > (STS), so set the read pointer to same value with write pointer,
> > otherwise set read point to 0 so can read from the start of RAM.
> >
> > Cc: Mathieu Poirier <mathieu.poirier@linaro.org>
> > Cc: Mike Leach <mike.leach@linaro.org>
> > Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
> > Signed-off-by: Leo Yan <leo.yan@linaro.org>
> > ---
> >  drivers/hwtracing/coresight/coresight-tmc-etf.c | 18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> >
> > diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
> > index 6150dac..43cfeaa 100644
> > --- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
> > +++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
> > @@ -43,10 +43,28 @@ static void tmc_etb_enable_hw(struct tmc_drvdata *drvdata)
> >
> >  static void tmc_etb_dump_hw(struct tmc_drvdata *drvdata)
> >  {
> > +       u32 write_ptr, status;
> >         char *bufp;
> >         u32 read_data;
> >         int i;
> >
> > +       write_ptr = readl_relaxed(drvdata->base + TMC_RWP);
> 
> Since 'write_ptr' is only used in the below 'if' branch, I would
> suggest to move this statement into that.

Thanks, Chunyan. Agree, will fix.

> > +
> > +       /*
> > +        * Get a hold of the status register and see if a wrap around
> > +        * has occurred.  If so adjust things accordingly.
> > +        */
> > +       status = readl_relaxed(drvdata->base + TMC_STS);
> > +       if (status & TMC_STS_FULL)
> > +               /* Tell the HW the reading start point */
> > +               writel_relaxed(write_ptr, drvdata->base + TMC_RRP);
> > +       else
> > +               /*
> > +                * In case this is not first time to read ETB RAM,
> > +                * always write 0 for reading pointer.
> > +                */
> > +               writel_relaxed(0x0, drvdata->base + TMC_RRP);
> > +
> >         bufp = drvdata->buf;
> >         drvdata->len = 0;
> >         while (1) {
> > --
> > 2.7.4
> >

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH RFC 0/4] coresight: support dump ETB RAM
  2017-04-11  9:10 [PATCH RFC 0/4] coresight: support dump ETB RAM Leo Yan
                   ` (3 preceding siblings ...)
  2017-04-11  9:10 ` [PATCH RFC 4/4] coresight: tmc: dump RAM for panic Leo Yan
@ 2017-04-20 17:45 ` Mathieu Poirier
  4 siblings, 0 replies; 8+ messages in thread
From: Mathieu Poirier @ 2017-04-20 17:45 UTC (permalink / raw)
  To: Leo Yan; +Cc: linux-arm-kernel, linux-kernel

On 11 April 2017 at 03:10, Leo Yan <leo.yan@linaro.org> wrote:
>
> ### Introduction ###
>
> Embedded Trace Buffer (ETB) provides on-chip storage of trace data,
> usually has buffer size from 2KB to 8KB. These data has been used for
> profiling and this has been well implemented in coresight driver.
>
> This patch is to explore ETB RAM data for postmortem debugging. Due ETB
> RAM buffer has small size, so the real trace data caused error is
> easily to be overwritten by other PEs; but we could consider ETB RAM
> data is quite useful for postmortem debugging with below scenarios:
>
> Case 1: if system is bus lockup and CPU pipeline stalls for bus
> accessing, CPUs have no more chance to fill enough data into ETB RAM
> so after analyze ETB RAM we can quickly get to know the culprit if bus
> lock is caused by improper programs, one often example is wrongly to
> access the module without enable the module's clock. For this case,
> we can rely on watchdog to trigger SoC reset and if lucky the ETB RAM
> can survive after reset. So for this case, after system reboot we can
> save ETB RAM before any new data input into it.
>
> Case 2: There also has another hardware design with local ETB buffer
> (ARM DDI 0461B) chapter 1.2.7. Local ETF, with this kind design every
> CPU may has one dedicated ETB RAM. So it's quite handy that we can use
> alive CPU to help dump the hang CPU ETB RAM. Then we can quickly get
> to know what's the last point the CPU has executed before its hang.
>
>
> ### Implementation ###
>
> Based on current Coresight ETB driver, we only needs some minor
> enhancement so can support dump ETB RAM with two methods.
>
> Patches 0001/0002 are minor fixes so can support more scenarios for ETB
> RAM dumping.
>
> Patch 0003 is to dump ETB RAM after system reboot, this is for the
> platforms which use watchdog reset and ETB RAM can survive.
>
> Patch 0004 is to dump ETB RAM when panic happens, so we can save ETB RAM
> into memory. If we connect this with Kdump, then we can easily extract
> the ETB RAM from vmcore.
>
>
> ### Usage ###
>
> To dump ETB RAM after reboot, simply use below command:
> # dd if=/dev/f6402000.etf of=cstrace.bin
>
> To dump ETB RAM for kernel panic, we need add "crash_kexec_post_notifiers"
> into kernel command line so let kernel call panic notifiers before launch
> dump kernel. After dump kernel has booted up, we need use below methods
> to ETB RAM offline analysis:
>
> On the target:
> # cp /proc/vmcore ./vmcore
> # scp ./vmcore your@hostpc
>
> On the host PC:
> # ./crash vmcore vmlinux
>
> crash> log
> [...]
> [  112.600051] coresight-tmc f6402000.etf: Flush ETB buffer 0x2000@0xffff800038300080
> [  112.614743] Starting crashdump kernel...
> [  112.618681] Bye!
> crash> rd 0xffff800038300080 0x2000 -r /tmp/cstrace.bin
> 8192 bytes copied from 0xffff800038300080 to /tmp/cstrace.bin
>
> After we get cstrace.bin data, we can use OpenCSD snapshot method to parse
> ETB trace data. These two methods have been verified on Hikey, For Hikey
> snapshot config files you can refer [1]. For total kernel patches for
> integration Kdump and Coresight, you can refer [2].
>
> [1] http://people.linaro.org/~leo.yan/opencsd_hikey/hikey_snapshot.tgz
> [2] https://git.linaro.org/people/leo.yan/linux-debug-workshop.git/log/?h=coresight_etb_dump
>
>
> ### TODO ###
>
> Need work for ETB1.0 driver, this is based on review and comments
> for this patch set.

Hi Leo and thank you for this first stab.

The first thing to do is drop the case where trace data are salvaged
from ETB memory after a crash.  This method is not reliable and the
trace data is almost guaranteed to have some sort of corruption since
the debug power domain will be reset by the architecture.  On top of
things it only applies to the ETB.

Also function tmc_enable/disable_etf_sink() can be called hundreds of
times during a trace session.  Inserting and removing the panic
notifier is too much overhead.  The notifier should be added when a
session is started and removed when it ends.

Your patchset doesn't deal with trace configuration, and that is a
serious problem.  Trace data can't be decoded without them.  What we
have for perf [1] is already working well and I would like to avoid
having to parse two different header format.  The header could be
inserted at the beginning of the file that is retreived after a crash
dump.

Last but not least we need to come up with an API to deal with the
kernel crash dump functionality. From there sinks could chose to
simply call the API when they are ready.  All the crash dump specific
stuff happens in the coresight crash dump code while everything
related to the sinks (of any kind) happens in the driver.  Look at
coresight-etm-perf.c for an idea of what I mean.

Regards,
Mathieu

[1]. http://lxr.free-electrons.com/source/tools/perf/util/cs-etm.h
>
>
> Leo Yan (4):
>   coresight: tmc: check dump buffer is overflow
>   coresight: tmc: set read pointer before dump RAM
>   coresight: tmc: dump RAM when device is disabled
>   coresight: tmc: dump RAM for panic
>
>  drivers/hwtracing/coresight/coresight-tmc-etf.c | 86 ++++++++++++++++++++++++-
>  drivers/hwtracing/coresight/coresight-tmc.h     |  2 +
>  2 files changed, 85 insertions(+), 3 deletions(-)
>
> --
> 2.7.4
>

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2017-04-20 17:45 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-11  9:10 [PATCH RFC 0/4] coresight: support dump ETB RAM Leo Yan
2017-04-11  9:10 ` [PATCH RFC 1/4] coresight: tmc: check dump buffer is overflow Leo Yan
2017-04-11  9:10 ` [PATCH RFC 2/4] coresight: tmc: set read pointer before dump RAM Leo Yan
2017-04-11 11:25   ` Chunyan Zhang
2017-04-11 13:46     ` Leo Yan
2017-04-11  9:10 ` [PATCH RFC 3/4] coresight: tmc: dump RAM when device is disabled Leo Yan
2017-04-11  9:10 ` [PATCH RFC 4/4] coresight: tmc: dump RAM for panic Leo Yan
2017-04-20 17:45 ` [PATCH RFC 0/4] coresight: support dump ETB RAM Mathieu Poirier

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).