[PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Anshuman Khandual]

Currently TMC-ETR automatically selects the buffer mode from all available
methods in the following sequentially fallback manner - also in that order.

1. FLAT mode with or without IOMMU
2. TMC-ETR-SG (scatter gather) mode when available
3. CATU mode when available

But this order might not be ideal for all situations. For example if there
is a CATU connected to ETR, it may be better to use TMC-ETR scatter gather
method, rather than CATU. But hard coding such order changes will prevent
us from testing or using a particular mode. This change provides following
new sysfs tunables for the user to control TMC-ETR buffer mode explicitly,
if required. This adds following new sysfs files for buffer mode selection
purpose explicitly in the user space.

/sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
/sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred

$ cat buf_modes_available
auto flat tmc-sg catu	------------------> Supported TMC-ETR buffer modes

$ echo catu > buf_mode_preferred   -------> Explicit buffer mode request

But explicit user request has to be within supported ETR buffer modes only.
These sysfs interface files are exclussive to ETR, and hence these are not
available for other TMC devices such as ETB or ETF etc.

A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback to the
existing default behaviour, when user provided preferred buffer mode fails.
ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred modes.

Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: Mike Leach <mike.leach@linaro.org>
Cc: James Clark <james.clark@arm.com>
Cc: Leo Yan <leo.yan@linaro.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: coresight@lists.linaro.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
This applies on v6.5-rc6

Changes in V2:

- Renamed sysfs file etr_buf_modes_available as buf_modes_available
- Renamed sysfs file buf_mode_current as buf_mode_preferred
- Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
- Renamed coresight_tmc_groups[] as coresight_etf_groups[]
- Reused coresight_tmc_group[] for trigger_cntr and buffer_size
- Fallback trying ETR_MODE_AUTO when user preferred mode fails
- Moved ETR sysfs details into coresight-tmc-etr.c
- Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT in sysfs
- Moved struct etr_buf_hw inside coresight-tmc-etr.c
- Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside coresight-tmc-etr.c
- Updated month in Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc

Changes in V1:

https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/

 .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
 .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
 .../hwtracing/coresight/coresight-tmc-etr.c   | 111 ++++++++++++++++--
 drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
 4 files changed, 131 insertions(+), 14 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc b/Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
index 6aa527296c71..50833ce97300 100644
--- a/Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
+++ b/Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
@@ -91,3 +91,19 @@ Contact:	Mathieu Poirier <mathieu.poirier@linaro.org>
 Description:	(RW) Size of the trace buffer for TMC-ETR when used in SYSFS
 		mode. Writable only for TMC-ETR configurations. The value
 		should be aligned to the kernel pagesize.
+
+What:		/sys/bus/coresight/devices/<memory_map>.tmc/buf_modes_available
+Date:		August 2023
+KernelVersion:	6.6
+Contact:	Anshuman Khandual <anshuman.khandual@arm.com>
+Description:	(Read) Shows all supported Coresight TMC-ETR buffer modes available
+		for the users to configure explicitly. This file is avaialble only
+		for TMC ETR devices.
+
+What:		/sys/bus/coresight/devices/<memory_map>.tmc/buf_mode_preferred
+Date:		August 2023
+KernelVersion:	6.6
+Contact:	Anshuman Khandual <anshuman.khandual@arm.com>
+Description:	(RW) Current Coresight TMC-ETR buffer mode selected. But user could
+		only provide a mode which is supported for a given ETR device. This
+		file is available only for TMC ETR devices.
diff --git a/drivers/hwtracing/coresight/coresight-tmc-core.c b/drivers/hwtracing/coresight/coresight-tmc-core.c
index c106d142e632..7ec5365e2b64 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-core.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-core.c
@@ -10,6 +10,7 @@
 #include <linux/device.h>
 #include <linux/idr.h>
 #include <linux/io.h>
+#include <linux/iommu.h>
 #include <linux/err.h>
 #include <linux/fs.h>
 #include <linux/miscdevice.h>
@@ -344,7 +345,14 @@ static const struct attribute_group coresight_tmc_mgmt_group = {
 	.name = "mgmt",
 };
 
-static const struct attribute_group *coresight_tmc_groups[] = {
+static const struct attribute_group *coresight_etf_groups[] = {
+	&coresight_tmc_group,
+	&coresight_tmc_mgmt_group,
+	NULL,
+};
+
+static const struct attribute_group *coresight_etr_groups[] = {
+	&coresight_etr_group,
 	&coresight_tmc_group,
 	&coresight_tmc_mgmt_group,
 	NULL,
@@ -465,6 +473,7 @@ static int tmc_probe(struct amba_device *adev, const struct amba_id *id)
 	drvdata->memwidth = tmc_get_memwidth(devid);
 	/* This device is not associated with a session */
 	drvdata->pid = -1;
+	drvdata->etr_mode = ETR_MODE_AUTO;
 
 	if (drvdata->config_type == TMC_CONFIG_TYPE_ETR) {
 		drvdata->size = tmc_etr_get_default_buffer_size(dev);
@@ -474,16 +483,17 @@ static int tmc_probe(struct amba_device *adev, const struct amba_id *id)
 	}
 
 	desc.dev = dev;
-	desc.groups = coresight_tmc_groups;
 
 	switch (drvdata->config_type) {
 	case TMC_CONFIG_TYPE_ETB:
+		desc.groups = coresight_etf_groups;
 		desc.type = CORESIGHT_DEV_TYPE_SINK;
 		desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_BUFFER;
 		desc.ops = &tmc_etb_cs_ops;
 		dev_list = &etb_devs;
 		break;
 	case TMC_CONFIG_TYPE_ETR:
+		desc.groups = coresight_etr_groups;
 		desc.type = CORESIGHT_DEV_TYPE_SINK;
 		desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_SYSMEM;
 		desc.ops = &tmc_etr_cs_ops;
@@ -496,6 +506,7 @@ static int tmc_probe(struct amba_device *adev, const struct amba_id *id)
 		dev_list = &etr_devs;
 		break;
 	case TMC_CONFIG_TYPE_ETF:
+		desc.groups = coresight_etf_groups;
 		desc.type = CORESIGHT_DEV_TYPE_LINKSINK;
 		desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_BUFFER;
 		desc.subtype.link_subtype = CORESIGHT_DEV_SUBTYPE_LINK_FIFO;
diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c
index 766325de0e29..52ce24afa323 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
@@ -26,6 +26,12 @@ struct etr_flat_buf {
 	size_t		size;
 };
 
+struct etr_buf_hw {
+	bool	has_iommu;
+	bool	has_etr_sg;
+	bool	has_catu;
+};
+
 /*
  * etr_perf_buffer - Perf buffer used for ETR
  * @drvdata		- The ETR drvdaga this buffer has been allocated for.
@@ -828,6 +834,22 @@ static inline int tmc_etr_mode_alloc_buf(int mode,
 	}
 }
 
+static void get_etr_buf_hw(struct device *dev, struct etr_buf_hw *buf_hw)
+{
+	struct tmc_drvdata *drvdata = dev_get_drvdata(dev->parent);
+
+	buf_hw->has_iommu = iommu_get_domain_for_dev(dev->parent);
+	buf_hw->has_etr_sg = tmc_etr_has_cap(drvdata, TMC_ETR_SG);
+	buf_hw->has_catu = !!tmc_etr_get_catu_device(drvdata);
+}
+
+static bool etr_can_use_flat_mode(struct etr_buf_hw *buf_hw, ssize_t etr_buf_size)
+{
+	bool has_sg = buf_hw->has_catu || buf_hw->has_etr_sg;
+
+	return !has_sg || buf_hw->has_iommu || etr_buf_size < SZ_1M;
+}
+
 /*
  * tmc_alloc_etr_buf: Allocate a buffer use by ETR.
  * @drvdata	: ETR device details.
@@ -841,23 +863,22 @@ static struct etr_buf *tmc_alloc_etr_buf(struct tmc_drvdata *drvdata,
 					 int node, void **pages)
 {
 	int rc = -ENOMEM;
-	bool has_etr_sg, has_iommu;
-	bool has_sg, has_catu;
 	struct etr_buf *etr_buf;
+	struct etr_buf_hw buf_hw;
 	struct device *dev = &drvdata->csdev->dev;
 
-	has_etr_sg = tmc_etr_has_cap(drvdata, TMC_ETR_SG);
-	has_iommu = iommu_get_domain_for_dev(dev->parent);
-	has_catu = !!tmc_etr_get_catu_device(drvdata);
-
-	has_sg = has_catu || has_etr_sg;
-
+	get_etr_buf_hw(dev, &buf_hw);
 	etr_buf = kzalloc(sizeof(*etr_buf), GFP_KERNEL);
 	if (!etr_buf)
 		return ERR_PTR(-ENOMEM);
 
 	etr_buf->size = size;
 
+	/* If there is user directive for buffer mode, try that first */
+	if (drvdata->etr_mode != ETR_MODE_AUTO)
+		rc = tmc_etr_mode_alloc_buf(drvdata->etr_mode, drvdata,
+					    etr_buf, node, pages);
+
 	/*
 	 * If we have to use an existing list of pages, we cannot reliably
 	 * use a contiguous DMA memory (even if we have an IOMMU). Otherwise,
@@ -870,14 +891,13 @@ static struct etr_buf *tmc_alloc_etr_buf(struct tmc_drvdata *drvdata,
 	 * Fallback to available mechanisms.
 	 *
 	 */
-	if (!pages &&
-	    (!has_sg || has_iommu || size < SZ_1M))
+	if (rc && !pages && etr_can_use_flat_mode(&buf_hw, size))
 		rc = tmc_etr_mode_alloc_buf(ETR_MODE_FLAT, drvdata,
 					    etr_buf, node, pages);
-	if (rc && has_etr_sg)
+	if (rc && buf_hw.has_etr_sg)
 		rc = tmc_etr_mode_alloc_buf(ETR_MODE_ETR_SG, drvdata,
 					    etr_buf, node, pages);
-	if (rc && has_catu)
+	if (rc && buf_hw.has_catu)
 		rc = tmc_etr_mode_alloc_buf(ETR_MODE_CATU, drvdata,
 					    etr_buf, node, pages);
 	if (rc) {
@@ -1800,3 +1820,70 @@ int tmc_read_unprepare_etr(struct tmc_drvdata *drvdata)
 
 	return 0;
 }
+
+static const char *const buf_modes_str[] = {
+	[ETR_MODE_FLAT]		= "flat",
+	[ETR_MODE_ETR_SG]	= "tmc-sg",
+	[ETR_MODE_CATU]		= "catu",
+	[ETR_MODE_AUTO]		= "auto",
+};
+
+static ssize_t buf_modes_available_show(struct device *dev,
+					    struct device_attribute *attr, char *buf)
+{
+	struct etr_buf_hw buf_hw;
+	ssize_t size = 0;
+
+	get_etr_buf_hw(dev, &buf_hw);
+	size += sysfs_emit(buf, "%s ", buf_modes_str[ETR_MODE_AUTO]);
+	size += sysfs_emit_at(buf, size, "%s ", buf_modes_str[ETR_MODE_FLAT]);
+	if (buf_hw.has_etr_sg)
+		size += sysfs_emit_at(buf, size, "%s ", buf_modes_str[ETR_MODE_ETR_SG]);
+
+	if (buf_hw.has_catu)
+		size += sysfs_emit_at(buf, size, "%s ", buf_modes_str[ETR_MODE_CATU]);
+
+	size += sysfs_emit_at(buf, size, "\n");
+	return size;
+}
+static DEVICE_ATTR_RO(buf_modes_available);
+
+static ssize_t buf_mode_preferred_show(struct device *dev,
+					 struct device_attribute *attr, char *buf)
+{
+	struct tmc_drvdata *drvdata = dev_get_drvdata(dev->parent);
+
+	return sysfs_emit(buf, "%s\n", buf_modes_str[drvdata->etr_mode]);
+}
+
+static ssize_t buf_mode_preferred_store(struct device *dev,
+					  struct device_attribute *attr,
+					  const char *buf, size_t size)
+{
+	struct tmc_drvdata *drvdata = dev_get_drvdata(dev->parent);
+	struct etr_buf_hw buf_hw;
+
+	get_etr_buf_hw(dev, &buf_hw);
+	if (sysfs_streq(buf, buf_modes_str[ETR_MODE_FLAT]))
+		drvdata->etr_mode = ETR_MODE_FLAT;
+	else if (sysfs_streq(buf, buf_modes_str[ETR_MODE_ETR_SG]) && buf_hw.has_etr_sg)
+		drvdata->etr_mode = ETR_MODE_ETR_SG;
+	else if (sysfs_streq(buf, buf_modes_str[ETR_MODE_CATU]) && buf_hw.has_catu)
+		drvdata->etr_mode = ETR_MODE_CATU;
+	else if (sysfs_streq(buf, buf_modes_str[ETR_MODE_AUTO]))
+		drvdata->etr_mode = ETR_MODE_AUTO;
+	else
+		return -EINVAL;
+	return size;
+}
+static DEVICE_ATTR_RW(buf_mode_preferred);
+
+static struct attribute *coresight_etr_attrs[] = {
+	&dev_attr_buf_modes_available.attr,
+	&dev_attr_buf_mode_preferred.attr,
+	NULL,
+};
+
+const struct attribute_group coresight_etr_group = {
+	.attrs = coresight_etr_attrs,
+};
diff --git a/drivers/hwtracing/coresight/coresight-tmc.h b/drivers/hwtracing/coresight/coresight-tmc.h
index b97da39652d2..e59d423a9749 100644
--- a/drivers/hwtracing/coresight/coresight-tmc.h
+++ b/drivers/hwtracing/coresight/coresight-tmc.h
@@ -135,6 +135,7 @@ enum etr_mode {
 	ETR_MODE_FLAT,		/* Uses contiguous flat buffer */
 	ETR_MODE_ETR_SG,	/* Uses in-built TMC ETR SG mechanism */
 	ETR_MODE_CATU,		/* Use SG mechanism in CATU */
+	ETR_MODE_AUTO,		/* Use the default mechanism */
 };
 
 struct etr_buf_operations;
@@ -207,6 +208,7 @@ struct tmc_drvdata {
 	enum tmc_mem_intf_width	memwidth;
 	u32			trigger_cntr;
 	u32			etr_caps;
+	enum etr_mode		etr_mode;
 	struct idr		idr;
 	struct mutex		idr_mutex;
 	struct etr_buf		*sysfs_buf;
@@ -334,5 +336,6 @@ void tmc_etr_set_catu_ops(const struct etr_buf_operations *catu);
 void tmc_etr_remove_catu_ops(void);
 struct etr_buf *tmc_etr_get_buffer(struct coresight_device *csdev,
 				   enum cs_mode mode, void *data);
+extern const struct attribute_group coresight_etr_group;
 
 #endif
-- 
2.25.1

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

Cc: Steve

Steve,

Are you able to test this with CATU ?


On 18/08/2023 09:21, Anshuman Khandual wrote:
> Currently TMC-ETR automatically selects the buffer mode from all available
> methods in the following sequentially fallback manner - also in that order.
> 
> 1. FLAT mode with or without IOMMU
> 2. TMC-ETR-SG (scatter gather) mode when available
> 3. CATU mode when available
> 
> But this order might not be ideal for all situations. For example if there
> is a CATU connected to ETR, it may be better to use TMC-ETR scatter gather
> method, rather than CATU. But hard coding such order changes will prevent
> us from testing or using a particular mode. This change provides following
> new sysfs tunables for the user to control TMC-ETR buffer mode explicitly,
> if required. This adds following new sysfs files for buffer mode selection
> purpose explicitly in the user space.
> 
> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
> 
> $ cat buf_modes_available
> auto flat tmc-sg catu	------------------> Supported TMC-ETR buffer modes
> 
> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode request
> 
> But explicit user request has to be within supported ETR buffer modes only.
> These sysfs interface files are exclussive to ETR, and hence these are not
> available for other TMC devices such as ETB or ETF etc.
> 
> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback to the
> existing default behaviour, when user provided preferred buffer mode fails.
> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred modes.
> 
> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
> Cc: Mike Leach <mike.leach@linaro.org>
> Cc: James Clark <james.clark@arm.com>
> Cc: Leo Yan <leo.yan@linaro.org>
> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Cc: coresight@lists.linaro.org
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> ---
> This applies on v6.5-rc6
> 
> Changes in V2:
> 
> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
> - Renamed sysfs file buf_mode_current as buf_mode_preferred
> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
> - Moved ETR sysfs details into coresight-tmc-etr.c
> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT in sysfs
> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside coresight-tmc-etr.c
> - Updated month in Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
> 
> Changes in V1:
> 
> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
> 
>   .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>   .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>   .../hwtracing/coresight/coresight-tmc-etr.c   | 111 ++++++++++++++++--
>   drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>   4 files changed, 131 insertions(+), 14 deletions(-)


Looks good to me.

Suzuki

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger OS]

Hi Suzuki,

I may be able to test it this week. You've already pointed me at the
patch thread(s). The main holdup is I need to merge the 6.6 pending
platform work in order to use the Ampere ACPI. I couldn't get these
patches to apply directly to 6.4 last I tried.

Steve C.

On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
> Cc: Steve
> 
> Steve,
> 
> Are you able to test this with CATU ?
> 
> 
> On 18/08/2023 09:21, Anshuman Khandual wrote:
>> Currently TMC-ETR automatically selects the buffer mode from all
>> available
>> methods in the following sequentially fallback manner - also in that
>> order.
>>
>> 1. FLAT mode with or without IOMMU
>> 2. TMC-ETR-SG (scatter gather) mode when available
>> 3. CATU mode when available
>>
>> But this order might not be ideal for all situations. For example if
>> there
>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>> gather
>> method, rather than CATU. But hard coding such order changes will prevent
>> us from testing or using a particular mode. This change provides
>> following
>> new sysfs tunables for the user to control TMC-ETR buffer mode
>> explicitly,
>> if required. This adds following new sysfs files for buffer mode
>> selection
>> purpose explicitly in the user space.
>>
>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>
>> $ cat buf_modes_available
>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>> modes
>>
>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode request
>>
>> But explicit user request has to be within supported ETR buffer modes
>> only.
>> These sysfs interface files are exclussive to ETR, and hence these are
>> not
>> available for other TMC devices such as ETB or ETF etc.
>>
>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>> to the
>> existing default behaviour, when user provided preferred buffer mode
>> fails.
>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred modes.
>>
>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>> Cc: Mike Leach <mike.leach@linaro.org>
>> Cc: James Clark <james.clark@arm.com>
>> Cc: Leo Yan <leo.yan@linaro.org>
>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>> Cc: coresight@lists.linaro.org
>> Cc: linux-arm-kernel@lists.infradead.org
>> Cc: linux-kernel@vger.kernel.org
>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>> ---
>> This applies on v6.5-rc6
>>
>> Changes in V2:
>>
>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>> - Moved ETR sysfs details into coresight-tmc-etr.c
>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>> in sysfs
>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>> coresight-tmc-etr.c
>> - Updated month in
>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>
>> Changes in V1:
>>
>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>
>>   .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>   .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>   .../hwtracing/coresight/coresight-tmc-etr.c   | 111 ++++++++++++++++--
>>   drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>   4 files changed, 131 insertions(+), 14 deletions(-)
> 
> 
> Looks good to me.
> 
> Suzuki
> 
> 

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger]

Here's some quick feedback. My system shows two modes available; auto catu

etr_buf_mode_current is writable. I expected to see tmc-sg (former
default) listed in etr_buf_modes_available but it doesn't show up.

Note that both the auto and catu etr_buf_mode_current settings default
to catu. My understanding is auto should revert to the default behavior.
On my system the default was tmc-sg.

More later.

[root@sut01sys-b212 kernel]# cat
/sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available

auto catu
[root@sut01sys-b212 kernel]# cat
/sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
catu
[root@sut01sys-b212 kernel]# echo "catu" >
/sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
[root@sut01sys-b212 kernel]# cat
/sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
catu

Steve C.


On 8/21/2023 12:40 PM, Steve Clevenger wrote:
> 
> Hi Suzuki,
> 
> I may be able to test it this week. You've already pointed me at the
> patch thread(s). The main holdup is I need to merge the 6.6 pending
> platform work in order to use the Ampere ACPI. I couldn't get these
> patches to apply directly to 6.4 last I tried.
> 
> Steve C.
> 
> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>> Cc: Steve
>>
>> Steve,
>>
>> Are you able to test this with CATU ?
>>
>>
>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>> Currently TMC-ETR automatically selects the buffer mode from all
>>> available
>>> methods in the following sequentially fallback manner - also in that
>>> order.
>>>
>>> 1. FLAT mode with or without IOMMU
>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>> 3. CATU mode when available
>>>
>>> But this order might not be ideal for all situations. For example if
>>> there
>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>> gather
>>> method, rather than CATU. But hard coding such order changes will prevent
>>> us from testing or using a particular mode. This change provides
>>> following
>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>> explicitly,
>>> if required. This adds following new sysfs files for buffer mode
>>> selection
>>> purpose explicitly in the user space.
>>>
>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>
>>> $ cat buf_modes_available
>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>>> modes
>>>
>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode request
>>>
>>> But explicit user request has to be within supported ETR buffer modes
>>> only.
>>> These sysfs interface files are exclussive to ETR, and hence these are
>>> not
>>> available for other TMC devices such as ETB or ETF etc.
>>>
>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>>> to the
>>> existing default behaviour, when user provided preferred buffer mode
>>> fails.
>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred modes.
>>>
>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>> Cc: Mike Leach <mike.leach@linaro.org>
>>> Cc: James Clark <james.clark@arm.com>
>>> Cc: Leo Yan <leo.yan@linaro.org>
>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>> Cc: coresight@lists.linaro.org
>>> Cc: linux-arm-kernel@lists.infradead.org
>>> Cc: linux-kernel@vger.kernel.org
>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>> ---
>>> This applies on v6.5-rc6
>>>
>>> Changes in V2:
>>>
>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>>> in sysfs
>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>> coresight-tmc-etr.c
>>> - Updated month in
>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>
>>> Changes in V1:
>>>
>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>
>>>   .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>   .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>   .../hwtracing/coresight/coresight-tmc-etr.c   | 111 ++++++++++++++++--
>>>   drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>   4 files changed, 131 insertions(+), 14 deletions(-)
>>
>>
>> Looks good to me.
>>
>> Suzuki
>>
>>

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger]

Unfortunately, I tested with the original patch not [PATCH V2]. I've
remedied this. My results below:

[root@sut01sys-b212 linux]# cat
/sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
auto flat catu
[root@sut01sys-b212 linux]# cat
/sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
auto
[root@sut01sys-b212 linux]# echo "catu" >
/sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
[root@sut01sys-b212 linux]# cat
/sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
catu

As with the V1 patch, auto defaults to catu.

I expected to see tmc-sg (former default) as an available mode, but do
not. As I recall, the buffer mode defaulted to ETR scatter-gather prior
to this patch. Must this capability now be explicitly advertised? I've
seen this done as "arm,scatter-gather" in device trees, but not used by
Ampere. Perhaps someone can enlighten me.

Steve C.

On 8/23/2023 4:10 PM, Steve Clevenger wrote:
> 
> Here's some quick feedback. My system shows two modes available; auto catu
> 
> etr_buf_mode_current is writable. I expected to see tmc-sg (former
> default) listed in etr_buf_modes_available but it doesn't show up.
> 
> Note that both the auto and catu etr_buf_mode_current settings default
> to catu. My understanding is auto should revert to the default behavior.
> On my system the default was tmc-sg.
> 
> More later.
> 
> [root@sut01sys-b212 kernel]# cat
> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
> 
> auto catu
> [root@sut01sys-b212 kernel]# cat
> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
> catu
> [root@sut01sys-b212 kernel]# echo "catu" >
> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
> [root@sut01sys-b212 kernel]# cat
> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
> catu
> 
> Steve C.
> 
> 
> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>
>> Hi Suzuki,
>>
>> I may be able to test it this week. You've already pointed me at the
>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>> platform work in order to use the Ampere ACPI. I couldn't get these
>> patches to apply directly to 6.4 last I tried.
>>
>> Steve C.
>>
>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>> Cc: Steve
>>>
>>> Steve,
>>>
>>> Are you able to test this with CATU ?
>>>
>>>
>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>> available
>>>> methods in the following sequentially fallback manner - also in that
>>>> order.
>>>>
>>>> 1. FLAT mode with or without IOMMU
>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>> 3. CATU mode when available
>>>>
>>>> But this order might not be ideal for all situations. For example if
>>>> there
>>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>>> gather
>>>> method, rather than CATU. But hard coding such order changes will prevent
>>>> us from testing or using a particular mode. This change provides
>>>> following
>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>> explicitly,
>>>> if required. This adds following new sysfs files for buffer mode
>>>> selection
>>>> purpose explicitly in the user space.
>>>>
>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>
>>>> $ cat buf_modes_available
>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>>>> modes
>>>>
>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode request
>>>>
>>>> But explicit user request has to be within supported ETR buffer modes
>>>> only.
>>>> These sysfs interface files are exclussive to ETR, and hence these are
>>>> not
>>>> available for other TMC devices such as ETB or ETF etc.
>>>>
>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>>>> to the
>>>> existing default behaviour, when user provided preferred buffer mode
>>>> fails.
>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred modes.
>>>>
>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>> Cc: James Clark <james.clark@arm.com>
>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>> Cc: coresight@lists.linaro.org
>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>> Cc: linux-kernel@vger.kernel.org
>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>> ---
>>>> This applies on v6.5-rc6
>>>>
>>>> Changes in V2:
>>>>
>>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>>>> in sysfs
>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>> coresight-tmc-etr.c
>>>> - Updated month in
>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>
>>>> Changes in V1:
>>>>
>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>
>>>>   .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>   .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>   .../hwtracing/coresight/coresight-tmc-etr.c   | 111 ++++++++++++++++--
>>>>   drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>   4 files changed, 131 insertions(+), 14 deletions(-)
>>>
>>>
>>> Looks good to me.
>>>
>>> Suzuki
>>>
>>>

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

Hi Steve

On 26/08/2023 01:14, Steve Clevenger wrote:
> 
> Unfortunately, I tested with the original patch not [PATCH V2]. I've
> remedied this. My results below:
> 
> [root@sut01sys-b212 linux]# cat
> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
> auto flat catu
> [root@sut01sys-b212 linux]# cat
> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
> auto
> [root@sut01sys-b212 linux]# echo "catu" >
> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
> [root@sut01sys-b212 linux]# cat
> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
> catu
> 
> As with the V1 patch, auto defaults to catu.
> 
> I expected to see tmc-sg (former default) as an available mode, but do
> not. As I recall, the buffer mode defaulted to ETR scatter-gather prior
> to this patch. Must this capability now be explicitly advertised? I've
> seen this done as "arm,scatter-gather" in device trees, but not used by
> Ampere. Perhaps someone can enlighten me.

Yes, you must add that property to the TMC-ETR node (for both DT and
ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
locked up the system while using the SG mode (due to the interconnect
issues, something to do with the transaction). Thus, we decided to
add a property explicitly enabling this for a given platform.

When you mentioned, it was using TMC-ETR SG mode, how did you verify
this ? Please be aware that the table allocation code etc are shared
by both TMC-SG and CATU.

Kind regards
Suzuki

> 
> Steve C.
> 
> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>
>> Here's some quick feedback. My system shows two modes available; auto catu
>>
>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>> default) listed in etr_buf_modes_available but it doesn't show up.
>>
>> Note that both the auto and catu etr_buf_mode_current settings default
>> to catu. My understanding is auto should revert to the default behavior.
>> On my system the default was tmc-sg.
>>
>> More later.
>>
>> [root@sut01sys-b212 kernel]# cat
>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>
>> auto catu
>> [root@sut01sys-b212 kernel]# cat
>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>> catu
>> [root@sut01sys-b212 kernel]# echo "catu" >
>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>> [root@sut01sys-b212 kernel]# cat
>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>> catu
>>
>> Steve C.
>>
>>
>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>
>>> Hi Suzuki,
>>>
>>> I may be able to test it this week. You've already pointed me at the
>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>> patches to apply directly to 6.4 last I tried.
>>>
>>> Steve C.
>>>
>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>> Cc: Steve
>>>>
>>>> Steve,
>>>>
>>>> Are you able to test this with CATU ?
>>>>
>>>>
>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>> available
>>>>> methods in the following sequentially fallback manner - also in that
>>>>> order.
>>>>>
>>>>> 1. FLAT mode with or without IOMMU
>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>> 3. CATU mode when available
>>>>>
>>>>> But this order might not be ideal for all situations. For example if
>>>>> there
>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>>>> gather
>>>>> method, rather than CATU. But hard coding such order changes will prevent
>>>>> us from testing or using a particular mode. This change provides
>>>>> following
>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>> explicitly,
>>>>> if required. This adds following new sysfs files for buffer mode
>>>>> selection
>>>>> purpose explicitly in the user space.
>>>>>
>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>
>>>>> $ cat buf_modes_available
>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>>>>> modes
>>>>>
>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode request
>>>>>
>>>>> But explicit user request has to be within supported ETR buffer modes
>>>>> only.
>>>>> These sysfs interface files are exclussive to ETR, and hence these are
>>>>> not
>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>
>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>>>>> to the
>>>>> existing default behaviour, when user provided preferred buffer mode
>>>>> fails.
>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred modes.
>>>>>
>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>>> Cc: James Clark <james.clark@arm.com>
>>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>> Cc: coresight@lists.linaro.org
>>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>>> Cc: linux-kernel@vger.kernel.org
>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>>> ---
>>>>> This applies on v6.5-rc6
>>>>>
>>>>> Changes in V2:
>>>>>
>>>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>>>>> in sysfs
>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>> coresight-tmc-etr.c
>>>>> - Updated month in
>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>
>>>>> Changes in V1:
>>>>>
>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>
>>>>>    .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>    .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>    .../hwtracing/coresight/coresight-tmc-etr.c   | 111 ++++++++++++++++--
>>>>>    drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>    4 files changed, 131 insertions(+), 14 deletions(-)
>>>>
>>>>
>>>> Looks good to me.
>>>>
>>>> Suzuki
>>>>
>>>>

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger]

Hi Suzuki,

On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
> Hi Steve
> 
> On 26/08/2023 01:14, Steve Clevenger wrote:
>>
>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>> remedied this. My results below:
>>
>> [root@sut01sys-b212 linux]# cat
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>> auto flat catu
>> [root@sut01sys-b212 linux]# cat
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>> auto
>> [root@sut01sys-b212 linux]# echo "catu" >
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>> [root@sut01sys-b212 linux]# cat
>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>> catu
>>
>> As with the V1 patch, auto defaults to catu.
>>
>> I expected to see tmc-sg (former default) as an available mode, but do
>> not. As I recall, the buffer mode defaulted to ETR scatter-gather prior
>> to this patch. Must this capability now be explicitly advertised? I've
>> seen this done as "arm,scatter-gather" in device trees, but not used by
>> Ampere. Perhaps someone can enlighten me.
> 
> Yes, you must add that property to the TMC-ETR node (for both DT and
> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
> locked up the system while using the SG mode (due to the interconnect
> issues, something to do with the transaction). Thus, we decided to
> add a property explicitly enabling this for a given platform.
> 
> When you mentioned, it was using TMC-ETR SG mode, how did you verify
> this ? Please be aware that the table allocation code etc are shared
> by both TMC-SG and CATU.
> 

You might recall how this started. I had no way to test the CATU due to
the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
purposes, I programmatically swapped the ETR-SG/CATU order and could
then verify CATU operation by the driver calling into CATU code. This
suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
prior to this hack. This didn't offer the user any control, hence my
feature request. Note that most of the early Ampere self-hosted trace
collection used ETR-SG. Now I can't select it.

How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
Platform Design Document" (DEN0067) doesn't describe this.

Thanks,
Steve


> Kind regards
> Suzuki
> 
>>
>> Steve C.
>>
>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>
>>> Here's some quick feedback. My system shows two modes available; auto
>>> catu
>>>
>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>
>>> Note that both the auto and catu etr_buf_mode_current settings default
>>> to catu. My understanding is auto should revert to the default behavior.
>>> On my system the default was tmc-sg.
>>>
>>> More later.
>>>
>>> [root@sut01sys-b212 kernel]# cat
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>
>>> auto catu
>>> [root@sut01sys-b212 kernel]# cat
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>> catu
>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>> [root@sut01sys-b212 kernel]# cat
>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>> catu
>>>
>>> Steve C.
>>>
>>>
>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>
>>>> Hi Suzuki,
>>>>
>>>> I may be able to test it this week. You've already pointed me at the
>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>> patches to apply directly to 6.4 last I tried.
>>>>
>>>> Steve C.
>>>>
>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>> Cc: Steve
>>>>>
>>>>> Steve,
>>>>>
>>>>> Are you able to test this with CATU ?
>>>>>
>>>>>
>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>> available
>>>>>> methods in the following sequentially fallback manner - also in that
>>>>>> order.
>>>>>>
>>>>>> 1. FLAT mode with or without IOMMU
>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>> 3. CATU mode when available
>>>>>>
>>>>>> But this order might not be ideal for all situations. For example if
>>>>>> there
>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>>>>> gather
>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>> prevent
>>>>>> us from testing or using a particular mode. This change provides
>>>>>> following
>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>> explicitly,
>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>> selection
>>>>>> purpose explicitly in the user space.
>>>>>>
>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>
>>>>>> $ cat buf_modes_available
>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>>>>>> modes
>>>>>>
>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>> request
>>>>>>
>>>>>> But explicit user request has to be within supported ETR buffer modes
>>>>>> only.
>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>> are
>>>>>> not
>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>
>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>>>>>> to the
>>>>>> existing default behaviour, when user provided preferred buffer mode
>>>>>> fails.
>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>> modes.
>>>>>>
>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>>>> Cc: James Clark <james.clark@arm.com>
>>>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>>> Cc: coresight@lists.linaro.org
>>>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>>>> Cc: linux-kernel@vger.kernel.org
>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>>>> ---
>>>>>> This applies on v6.5-rc6
>>>>>>
>>>>>> Changes in V2:
>>>>>>
>>>>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>>>>>> in sysfs
>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>> coresight-tmc-etr.c
>>>>>> - Updated month in
>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>
>>>>>> Changes in V1:
>>>>>>
>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>
>>>>>>    .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>    .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>    .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>> ++++++++++++++++--
>>>>>>    drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>    4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>
>>>>>
>>>>> Looks good to me.
>>>>>
>>>>> Suzuki
>>>>>
>>>>>
> 

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

Cc: Sudeep

Hi Steve

On 28/08/2023 17:35, Steve Clevenger wrote:
> 
> Hi Suzuki,
> 
> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>> Hi Steve
>>
>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>
>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>> remedied this. My results below:
>>>
>>> [root@sut01sys-b212 linux]# cat
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>> auto flat catu
>>> [root@sut01sys-b212 linux]# cat
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>> auto
>>> [root@sut01sys-b212 linux]# echo "catu" >
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>> [root@sut01sys-b212 linux]# cat
>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>> catu
>>>
>>> As with the V1 patch, auto defaults to catu.
>>>
>>> I expected to see tmc-sg (former default) as an available mode, but do
>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather prior
>>> to this patch. Must this capability now be explicitly advertised? I've
>>> seen this done as "arm,scatter-gather" in device trees, but not used by
>>> Ampere. Perhaps someone can enlighten me.
>>
>> Yes, you must add that property to the TMC-ETR node (for both DT and
>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>> locked up the system while using the SG mode (due to the interconnect
>> issues, something to do with the transaction). Thus, we decided to
>> add a property explicitly enabling this for a given platform.
>>
>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>> this ? Please be aware that the table allocation code etc are shared
>> by both TMC-SG and CATU.
>>
> 
> You might recall how this started. I had no way to test the CATU due to
> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
> purposes, I programmatically swapped the ETR-SG/CATU order and could
> then verify CATU operation by the driver calling into CATU code. This

So, were you using the DT based boot for the above runs ?

> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
> prior to this hack. This didn't offer the user any control, hence my
> feature request. Note that most of the early Ampere self-hosted trace
> collection used ETR-SG. Now I can't select it.


> 
> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
> Platform Design Document" (DEN0067) doesn't describe this.

This is not specified in the ACPI platform design document. I can get
it fixed. Ideally we need a property describing that the scatter-gather
mode is safe to use.

DT uses "arm,scatter-gather" property [0] and this is what we now expect
in the ACPI based systems too.

https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73

Does it sound fine ?

Suzuki

> 
> Thanks,
> Steve
> 
> 
>> Kind regards
>> Suzuki
>>
>>>
>>> Steve C.
>>>
>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>
>>>> Here's some quick feedback. My system shows two modes available; auto
>>>> catu
>>>>
>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>
>>>> Note that both the auto and catu etr_buf_mode_current settings default
>>>> to catu. My understanding is auto should revert to the default behavior.
>>>> On my system the default was tmc-sg.
>>>>
>>>> More later.
>>>>
>>>> [root@sut01sys-b212 kernel]# cat
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>
>>>> auto catu
>>>> [root@sut01sys-b212 kernel]# cat
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>> catu
>>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>> [root@sut01sys-b212 kernel]# cat
>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>> catu
>>>>
>>>> Steve C.
>>>>
>>>>
>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>
>>>>> Hi Suzuki,
>>>>>
>>>>> I may be able to test it this week. You've already pointed me at the
>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>> patches to apply directly to 6.4 last I tried.
>>>>>
>>>>> Steve C.
>>>>>
>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>> Cc: Steve
>>>>>>
>>>>>> Steve,
>>>>>>
>>>>>> Are you able to test this with CATU ?
>>>>>>
>>>>>>
>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>> available
>>>>>>> methods in the following sequentially fallback manner - also in that
>>>>>>> order.
>>>>>>>
>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>> 3. CATU mode when available
>>>>>>>
>>>>>>> But this order might not be ideal for all situations. For example if
>>>>>>> there
>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>>>>>> gather
>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>> prevent
>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>> following
>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>> explicitly,
>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>> selection
>>>>>>> purpose explicitly in the user space.
>>>>>>>
>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>
>>>>>>> $ cat buf_modes_available
>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR buffer
>>>>>>> modes
>>>>>>>
>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>> request
>>>>>>>
>>>>>>> But explicit user request has to be within supported ETR buffer modes
>>>>>>> only.
>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>> are
>>>>>>> not
>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>
>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback
>>>>>>> to the
>>>>>>> existing default behaviour, when user provided preferred buffer mode
>>>>>>> fails.
>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>> modes.
>>>>>>>
>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>>>>> Cc: James Clark <james.clark@arm.com>
>>>>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>>>>> Cc: linux-kernel@vger.kernel.org
>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>>>>> ---
>>>>>>> This applies on v6.5-rc6
>>>>>>>
>>>>>>> Changes in V2:
>>>>>>>
>>>>>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>> - Dropped etr_can_use_flat_mode() check while offering ETR_MODE_FLAT
>>>>>>> in sysfs
>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>> coresight-tmc-etr.c
>>>>>>> - Updated month in
>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>
>>>>>>> Changes in V1:
>>>>>>>
>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>>
>>>>>>>     .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>     .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>     .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>> ++++++++++++++++--
>>>>>>>     drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>     4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>
>>>>>>
>>>>>> Looks good to me.
>>>>>>
>>>>>> Suzuki
>>>>>>
>>>>>>
>>

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

Hi Steve

On 30/08/2023 17:04, Suzuki K Poulose wrote:
> Cc: Sudeep
> 
> Hi Steve
> 
> On 28/08/2023 17:35, Steve Clevenger wrote:
>>
>> Hi Suzuki,
>>
>> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>>> Hi Steve
>>>
>>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>>
>>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>>> remedied this. My results below:
>>>>
>>>> [root@sut01sys-b212 linux]# cat
>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>>> auto flat catu
>>>> [root@sut01sys-b212 linux]# cat
>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>> auto
>>>> [root@sut01sys-b212 linux]# echo "catu" >
>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>> [root@sut01sys-b212 linux]# cat
>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>> catu
>>>>
>>>> As with the V1 patch, auto defaults to catu.
>>>>
>>>> I expected to see tmc-sg (former default) as an available mode, but do
>>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather prior
>>>> to this patch. Must this capability now be explicitly advertised? I've
>>>> seen this done as "arm,scatter-gather" in device trees, but not used by
>>>> Ampere. Perhaps someone can enlighten me.
>>>
>>> Yes, you must add that property to the TMC-ETR node (for both DT and
>>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>>> locked up the system while using the SG mode (due to the interconnect
>>> issues, something to do with the transaction). Thus, we decided to
>>> add a property explicitly enabling this for a given platform.
>>>
>>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>>> this ? Please be aware that the table allocation code etc are shared
>>> by both TMC-SG and CATU.
>>>
>>
>> You might recall how this started. I had no way to test the CATU due to
>> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
>> purposes, I programmatically swapped the ETR-SG/CATU order and could
>> then verify CATU operation by the driver calling into CATU code. This
> 
> So, were you using the DT based boot for the above runs ?
> 
>> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
>> prior to this hack. This didn't offer the user any control, hence my
>> feature request. Note that most of the early Ampere self-hosted trace
>> collection used ETR-SG. Now I can't select it.
> 
> 
>>
>> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
>> Platform Design Document" (DEN0067) doesn't describe this.
> 
> This is not specified in the ACPI platform design document. I can get
> it fixed. Ideally we need a property describing that the scatter-gather
> mode is safe to use.

Looks like this is not straight forward copying of DT property. We are
investigating this on our side and will get back to you.

Suzuki


> 
> DT uses "arm,scatter-gather" property [0] and this is what we now expect
> in the ACPI based systems too.
> 
> https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73
> 
> Does it sound fine ?
> 
> Suzuki
> 
>>
>> Thanks,
>> Steve
>>
>>
>>> Kind regards
>>> Suzuki
>>>
>>>>
>>>> Steve C.
>>>>
>>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>>
>>>>> Here's some quick feedback. My system shows two modes available; auto
>>>>> catu
>>>>>
>>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>>
>>>>> Note that both the auto and catu etr_buf_mode_current settings default
>>>>> to catu. My understanding is auto should revert to the default 
>>>>> behavior.
>>>>> On my system the default was tmc-sg.
>>>>>
>>>>> More later.
>>>>>
>>>>> [root@sut01sys-b212 kernel]# cat
>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>>
>>>>> auto catu
>>>>> [root@sut01sys-b212 kernel]# cat
>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>> catu
>>>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>> [root@sut01sys-b212 kernel]# cat
>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>> catu
>>>>>
>>>>> Steve C.
>>>>>
>>>>>
>>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>>
>>>>>> Hi Suzuki,
>>>>>>
>>>>>> I may be able to test it this week. You've already pointed me at the
>>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>>> patches to apply directly to 6.4 last I tried.
>>>>>>
>>>>>> Steve C.
>>>>>>
>>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>>> Cc: Steve
>>>>>>>
>>>>>>> Steve,
>>>>>>>
>>>>>>> Are you able to test this with CATU ?
>>>>>>>
>>>>>>>
>>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>>> available
>>>>>>>> methods in the following sequentially fallback manner - also in 
>>>>>>>> that
>>>>>>>> order.
>>>>>>>>
>>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>>> 3. CATU mode when available
>>>>>>>>
>>>>>>>> But this order might not be ideal for all situations. For 
>>>>>>>> example if
>>>>>>>> there
>>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR scatter
>>>>>>>> gather
>>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>>> prevent
>>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>>> following
>>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>>> explicitly,
>>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>>> selection
>>>>>>>> purpose explicitly in the user space.
>>>>>>>>
>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>>
>>>>>>>> $ cat buf_modes_available
>>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR 
>>>>>>>> buffer
>>>>>>>> modes
>>>>>>>>
>>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>>> request
>>>>>>>>
>>>>>>>> But explicit user request has to be within supported ETR buffer 
>>>>>>>> modes
>>>>>>>> only.
>>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>>> are
>>>>>>>> not
>>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>>
>>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help 
>>>>>>>> fallback
>>>>>>>> to the
>>>>>>>> existing default behaviour, when user provided preferred buffer 
>>>>>>>> mode
>>>>>>>> fails.
>>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>>> modes.
>>>>>>>>
>>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>>>>>> Cc: James Clark <james.clark@arm.com>
>>>>>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>>>>>> Cc: linux-kernel@vger.kernel.org
>>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>>>>>> ---
>>>>>>>> This applies on v6.5-rc6
>>>>>>>>
>>>>>>>> Changes in V2:
>>>>>>>>
>>>>>>>> - Renamed sysfs file etr_buf_modes_available as buf_modes_available
>>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>>> - Dropped etr_can_use_flat_mode() check while offering 
>>>>>>>> ETR_MODE_FLAT
>>>>>>>> in sysfs
>>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>>> coresight-tmc-etr.c
>>>>>>>> - Updated month in
>>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>>
>>>>>>>> Changes in V1:
>>>>>>>>
>>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>>>
>>>>>>>>     .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>>     .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>>     .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>>> ++++++++++++++++--
>>>>>>>>     drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>>     4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>>
>>>>>>>
>>>>>>> Looks good to me.
>>>>>>>
>>>>>>> Suzuki
>>>>>>>
>>>>>>>
>>>
> 

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger]

Hi Suzuki,

On 9/1/2023 5:22 AM, Suzuki K Poulose wrote:
> Hi Steve
> 
> On 30/08/2023 17:04, Suzuki K Poulose wrote:
>> Cc: Sudeep
>>
>> Hi Steve
>>
>> On 28/08/2023 17:35, Steve Clevenger wrote:
>>>
>>> Hi Suzuki,
>>>
>>> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>>>> Hi Steve
>>>>
>>>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>>>
>>>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>>>> remedied this. My results below:
>>>>>
>>>>> [root@sut01sys-b212 linux]# cat
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>>>> auto flat catu
>>>>> [root@sut01sys-b212 linux]# cat
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>> auto
>>>>> [root@sut01sys-b212 linux]# echo "catu" >
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>> [root@sut01sys-b212 linux]# cat
>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>> catu
>>>>>
>>>>> As with the V1 patch, auto defaults to catu.
>>>>>
>>>>> I expected to see tmc-sg (former default) as an available mode, but do
>>>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather
>>>>> prior
>>>>> to this patch. Must this capability now be explicitly advertised? I've
>>>>> seen this done as "arm,scatter-gather" in device trees, but not
>>>>> used by
>>>>> Ampere. Perhaps someone can enlighten me.
>>>>
>>>> Yes, you must add that property to the TMC-ETR node (for both DT and
>>>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>>>> locked up the system while using the SG mode (due to the interconnect
>>>> issues, something to do with the transaction). Thus, we decided to
>>>> add a property explicitly enabling this for a given platform.
>>>>
>>>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>>>> this ? Please be aware that the table allocation code etc are shared
>>>> by both TMC-SG and CATU.
>>>>
>>>
>>> You might recall how this started. I had no way to test the CATU due to
>>> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
>>> purposes, I programmatically swapped the ETR-SG/CATU order and could
>>> then verify CATU operation by the driver calling into CATU code. This
>>
>> So, were you using the DT based boot for the above runs ?
>>
>>> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
>>> prior to this hack. This didn't offer the user any control, hence my
>>> feature request. Note that most of the early Ampere self-hosted trace
>>> collection used ETR-SG. Now I can't select it.
>>
>>
>>>
>>> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
>>> Platform Design Document" (DEN0067) doesn't describe this.
>>
>> This is not specified in the ACPI platform design document. I can get
>> it fixed. Ideally we need a property describing that the scatter-gather
>> mode is safe to use.
> 
> Looks like this is not straight forward copying of DT property. We are
> investigating this on our side and will get back to you.
>

The intent behind my request was to have a way to SysFS configure the
(available) ETR mode. Unless there's a change to the ACPI, the CATU is
the only SG option for Ampere in the near term.

Thanks,
Steve


> Suzuki
> 
> 
>>
>> DT uses "arm,scatter-gather" property [0] and this is what we now expect
>> in the ACPI based systems too.
>>
>> https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73
>>
>> Does it sound fine ?
>>
>> Suzuki
>>
>>>
>>> Thanks,
>>> Steve
>>>
>>>
>>>> Kind regards
>>>> Suzuki
>>>>
>>>>>
>>>>> Steve C.
>>>>>
>>>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>>>
>>>>>> Here's some quick feedback. My system shows two modes available; auto
>>>>>> catu
>>>>>>
>>>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>>>
>>>>>> Note that both the auto and catu etr_buf_mode_current settings
>>>>>> default
>>>>>> to catu. My understanding is auto should revert to the default
>>>>>> behavior.
>>>>>> On my system the default was tmc-sg.
>>>>>>
>>>>>> More later.
>>>>>>
>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>>>
>>>>>> auto catu
>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>> catu
>>>>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>> catu
>>>>>>
>>>>>> Steve C.
>>>>>>
>>>>>>
>>>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>>>
>>>>>>> Hi Suzuki,
>>>>>>>
>>>>>>> I may be able to test it this week. You've already pointed me at the
>>>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>>>> patches to apply directly to 6.4 last I tried.
>>>>>>>
>>>>>>> Steve C.
>>>>>>>
>>>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>>>> Cc: Steve
>>>>>>>>
>>>>>>>> Steve,
>>>>>>>>
>>>>>>>> Are you able to test this with CATU ?
>>>>>>>>
>>>>>>>>
>>>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>>>> available
>>>>>>>>> methods in the following sequentially fallback manner - also in
>>>>>>>>> that
>>>>>>>>> order.
>>>>>>>>>
>>>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>>>> 3. CATU mode when available
>>>>>>>>>
>>>>>>>>> But this order might not be ideal for all situations. For
>>>>>>>>> example if
>>>>>>>>> there
>>>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR
>>>>>>>>> scatter
>>>>>>>>> gather
>>>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>>>> prevent
>>>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>>>> following
>>>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>>>> explicitly,
>>>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>>>> selection
>>>>>>>>> purpose explicitly in the user space.
>>>>>>>>>
>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>>>
>>>>>>>>> $ cat buf_modes_available
>>>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR
>>>>>>>>> buffer
>>>>>>>>> modes
>>>>>>>>>
>>>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>>>> request
>>>>>>>>>
>>>>>>>>> But explicit user request has to be within supported ETR buffer
>>>>>>>>> modes
>>>>>>>>> only.
>>>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>>>> are
>>>>>>>>> not
>>>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>>>
>>>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help
>>>>>>>>> fallback
>>>>>>>>> to the
>>>>>>>>> existing default behaviour, when user provided preferred buffer
>>>>>>>>> mode
>>>>>>>>> fails.
>>>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>>>> modes.
>>>>>>>>>
>>>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>>>>>>> Cc: James Clark <james.clark@arm.com>
>>>>>>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>>>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>>>>>>> Cc: linux-kernel@vger.kernel.org
>>>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>>>>>>> ---
>>>>>>>>> This applies on v6.5-rc6
>>>>>>>>>
>>>>>>>>> Changes in V2:
>>>>>>>>>
>>>>>>>>> - Renamed sysfs file etr_buf_modes_available as
>>>>>>>>> buf_modes_available
>>>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>>>> - Dropped etr_can_use_flat_mode() check while offering
>>>>>>>>> ETR_MODE_FLAT
>>>>>>>>> in sysfs
>>>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>>>> coresight-tmc-etr.c
>>>>>>>>> - Updated month in
>>>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>>>
>>>>>>>>> Changes in V1:
>>>>>>>>>
>>>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>>>>
>>>>>>>>>     .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>>>     .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>>>     .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>>>> ++++++++++++++++--
>>>>>>>>>     drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>>>     4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>>>
>>>>>>>>
>>>>>>>> Looks good to me.
>>>>>>>>
>>>>>>>> Suzuki
>>>>>>>>
>>>>>>>>
>>>>
>>
> 

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Anshuman Khandual]

On 8/18/23 13:51, Anshuman Khandual wrote:
> Currently TMC-ETR automatically selects the buffer mode from all available
> methods in the following sequentially fallback manner - also in that order.
> 
> 1. FLAT mode with or without IOMMU
> 2. TMC-ETR-SG (scatter gather) mode when available
> 3. CATU mode when available
> 
> But this order might not be ideal for all situations. For example if there
> is a CATU connected to ETR, it may be better to use TMC-ETR scatter gather
> method, rather than CATU. But hard coding such order changes will prevent
> us from testing or using a particular mode. This change provides following
> new sysfs tunables for the user to control TMC-ETR buffer mode explicitly,
> if required. This adds following new sysfs files for buffer mode selection
> purpose explicitly in the user space.
> 
> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
> 
> $ cat buf_modes_available
> auto flat tmc-sg catu	------------------> Supported TMC-ETR buffer modes
> 
> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode request
> 
> But explicit user request has to be within supported ETR buffer modes only.
> These sysfs interface files are exclussive to ETR, and hence these are not
> available for other TMC devices such as ETB or ETF etc.
> 
> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help fallback to the
> existing default behaviour, when user provided preferred buffer mode fails.
> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred modes.
> 
> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
> Cc: Mike Leach <mike.leach@linaro.org>
> Cc: James Clark <james.clark@arm.com>
> Cc: Leo Yan <leo.yan@linaro.org>
> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Cc: coresight@lists.linaro.org
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> ---
> This applies on v6.5-rc6
> 
> Changes in V2:

Hi Suzuki,

This patch still applies cleanly on current coresight/for-next/queue.
But in case, this needs any changes, please do let me know. Thanks !

- Anshuman

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

On Fri, 18 Aug 2023 13:51:12 +0530, Anshuman Khandual wrote:
> Currently TMC-ETR automatically selects the buffer mode from all available
> methods in the following sequentially fallback manner - also in that order.
> 
> 1. FLAT mode with or without IOMMU
> 2. TMC-ETR-SG (scatter gather) mode when available
> 3. CATU mode when available
> 
> [...]

Applied, thanks!

[1/1] coresight: tmc: Make etr buffer mode user configurable from sysfs
      https://git.kernel.org/coresight/c/efb05212a230

Best regards,
-- 
Suzuki K Poulose <suzuki.poulose@arm.com>

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Steve Clevenger]

Hi Suzuki,

On 9/7/2023 2:31 PM, Steve Clevenger wrote:
> 
> Hi Suzuki,
> 
> On 9/1/2023 5:22 AM, Suzuki K Poulose wrote:
>> Hi Steve
>>
>> On 30/08/2023 17:04, Suzuki K Poulose wrote:
>>> Cc: Sudeep
>>>
>>> Hi Steve
>>>
>>> On 28/08/2023 17:35, Steve Clevenger wrote:
>>>>
>>>> Hi Suzuki,
>>>>
>>>> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>>>>> Hi Steve
>>>>>
>>>>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>>>>
>>>>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>>>>> remedied this. My results below:
>>>>>>
>>>>>> [root@sut01sys-b212 linux]# cat
>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>>>>> auto flat catu
>>>>>> [root@sut01sys-b212 linux]# cat
>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>> auto
>>>>>> [root@sut01sys-b212 linux]# echo "catu" >
>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>> [root@sut01sys-b212 linux]# cat
>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>> catu
>>>>>>
>>>>>> As with the V1 patch, auto defaults to catu.
>>>>>>
>>>>>> I expected to see tmc-sg (former default) as an available mode, but do
>>>>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather
>>>>>> prior
>>>>>> to this patch. Must this capability now be explicitly advertised? I've
>>>>>> seen this done as "arm,scatter-gather" in device trees, but not
>>>>>> used by
>>>>>> Ampere. Perhaps someone can enlighten me.
>>>>>
>>>>> Yes, you must add that property to the TMC-ETR node (for both DT and
>>>>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>>>>> locked up the system while using the SG mode (due to the interconnect
>>>>> issues, something to do with the transaction). Thus, we decided to
>>>>> add a property explicitly enabling this for a given platform.
>>>>>
>>>>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>>>>> this ? Please be aware that the table allocation code etc are shared
>>>>> by both TMC-SG and CATU.
>>>>>
>>>>
>>>> You might recall how this started. I had no way to test the CATU due to
>>>> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
>>>> purposes, I programmatically swapped the ETR-SG/CATU order and could
>>>> then verify CATU operation by the driver calling into CATU code. This
>>>
>>> So, were you using the DT based boot for the above runs ?
>>>
>>>> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
>>>> prior to this hack. This didn't offer the user any control, hence my
>>>> feature request. Note that most of the early Ampere self-hosted trace
>>>> collection used ETR-SG. Now I can't select it.
>>>
>>>
>>>>
>>>> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
>>>> Platform Design Document" (DEN0067) doesn't describe this.
>>>
>>> This is not specified in the ACPI platform design document. I can get
>>> it fixed. Ideally we need a property describing that the scatter-gather
>>> mode is safe to use.
>>
>> Looks like this is not straight forward copying of DT property. We are
>> investigating this on our side and will get back to you.

I noticed this work is queued for 6.7 (coresight-next-6.7). Do you have
an update to the ACPI platform design document Ampere can use to base an
update so the scatter-gather mode can be used?

Thanks,
Steve

>>
> 
> The intent behind my request was to have a way to SysFS configure the
> (available) ETR mode. Unless there's a change to the ACPI, the CATU is
> the only SG option for Ampere in the near term.
> 
> Thanks,
> Steve
> 
> 
>> Suzuki
>>
>>
>>>
>>> DT uses "arm,scatter-gather" property [0] and this is what we now expect
>>> in the ACPI based systems too.
>>>
>>> https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73
>>>
>>> Does it sound fine ?
>>>
>>> Suzuki
>>>
>>>>
>>>> Thanks,
>>>> Steve
>>>>
>>>>
>>>>> Kind regards
>>>>> Suzuki
>>>>>
>>>>>>
>>>>>> Steve C.
>>>>>>
>>>>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>>>>
>>>>>>> Here's some quick feedback. My system shows two modes available; auto
>>>>>>> catu
>>>>>>>
>>>>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>>>>
>>>>>>> Note that both the auto and catu etr_buf_mode_current settings
>>>>>>> default
>>>>>>> to catu. My understanding is auto should revert to the default
>>>>>>> behavior.
>>>>>>> On my system the default was tmc-sg.
>>>>>>>
>>>>>>> More later.
>>>>>>>
>>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>>>>
>>>>>>> auto catu
>>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>> catu
>>>>>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>> catu
>>>>>>>
>>>>>>> Steve C.
>>>>>>>
>>>>>>>
>>>>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>>>>
>>>>>>>> Hi Suzuki,
>>>>>>>>
>>>>>>>> I may be able to test it this week. You've already pointed me at the
>>>>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>>>>> patches to apply directly to 6.4 last I tried.
>>>>>>>>
>>>>>>>> Steve C.
>>>>>>>>
>>>>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>>>>> Cc: Steve
>>>>>>>>>
>>>>>>>>> Steve,
>>>>>>>>>
>>>>>>>>> Are you able to test this with CATU ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>>>>> available
>>>>>>>>>> methods in the following sequentially fallback manner - also in
>>>>>>>>>> that
>>>>>>>>>> order.
>>>>>>>>>>
>>>>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>>>>> 3. CATU mode when available
>>>>>>>>>>
>>>>>>>>>> But this order might not be ideal for all situations. For
>>>>>>>>>> example if
>>>>>>>>>> there
>>>>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR
>>>>>>>>>> scatter
>>>>>>>>>> gather
>>>>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>>>>> prevent
>>>>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>>>>> following
>>>>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>>>>> explicitly,
>>>>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>>>>> selection
>>>>>>>>>> purpose explicitly in the user space.
>>>>>>>>>>
>>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>>>>
>>>>>>>>>> $ cat buf_modes_available
>>>>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR
>>>>>>>>>> buffer
>>>>>>>>>> modes
>>>>>>>>>>
>>>>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>>>>> request
>>>>>>>>>>
>>>>>>>>>> But explicit user request has to be within supported ETR buffer
>>>>>>>>>> modes
>>>>>>>>>> only.
>>>>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>>>>> are
>>>>>>>>>> not
>>>>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>>>>
>>>>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help
>>>>>>>>>> fallback
>>>>>>>>>> to the
>>>>>>>>>> existing default behaviour, when user provided preferred buffer
>>>>>>>>>> mode
>>>>>>>>>> fails.
>>>>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>>>>> modes.
>>>>>>>>>>
>>>>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>>>>>>>> Cc: James Clark <james.clark@arm.com>
>>>>>>>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>>>>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>>>>>>>> Cc: linux-kernel@vger.kernel.org
>>>>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>>>>>>>> ---
>>>>>>>>>> This applies on v6.5-rc6
>>>>>>>>>>
>>>>>>>>>> Changes in V2:
>>>>>>>>>>
>>>>>>>>>> - Renamed sysfs file etr_buf_modes_available as
>>>>>>>>>> buf_modes_available
>>>>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>>>>> - Dropped etr_can_use_flat_mode() check while offering
>>>>>>>>>> ETR_MODE_FLAT
>>>>>>>>>> in sysfs
>>>>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>>>>> coresight-tmc-etr.c
>>>>>>>>>> - Updated month in
>>>>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>>>>
>>>>>>>>>> Changes in V1:
>>>>>>>>>>
>>>>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>>>>>
>>>>>>>>>>     .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>>>>     .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>>>>     .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>>>>> ++++++++++++++++--
>>>>>>>>>>     drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>>>>     4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Looks good to me.
>>>>>>>>>
>>>>>>>>> Suzuki
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>
>>

Re: [PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

Hi Steve

On 04/12/2023 21:41, Steve Clevenger wrote:
> 
> Hi Suzuki,
> 
> On 9/7/2023 2:31 PM, Steve Clevenger wrote:
>>
>> Hi Suzuki,
>>
>> On 9/1/2023 5:22 AM, Suzuki K Poulose wrote:
>>> Hi Steve
>>>
>>> On 30/08/2023 17:04, Suzuki K Poulose wrote:
>>>> Cc: Sudeep
>>>>
>>>> Hi Steve
>>>>
>>>> On 28/08/2023 17:35, Steve Clevenger wrote:
>>>>>
>>>>> Hi Suzuki,
>>>>>
>>>>> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>>>>>> Hi Steve
>>>>>>
>>>>>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>>>>>
>>>>>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>>>>>> remedied this. My results below:
>>>>>>>
>>>>>>> [root@sut01sys-b212 linux]# cat
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>>>>>> auto flat catu
>>>>>>> [root@sut01sys-b212 linux]# cat
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>>> auto
>>>>>>> [root@sut01sys-b212 linux]# echo "catu" >
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>>> [root@sut01sys-b212 linux]# cat
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>>> catu
>>>>>>>
>>>>>>> As with the V1 patch, auto defaults to catu.
>>>>>>>
>>>>>>> I expected to see tmc-sg (former default) as an available mode, but do
>>>>>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather
>>>>>>> prior
>>>>>>> to this patch. Must this capability now be explicitly advertised? I've
>>>>>>> seen this done as "arm,scatter-gather" in device trees, but not
>>>>>>> used by
>>>>>>> Ampere. Perhaps someone can enlighten me.
>>>>>>
>>>>>> Yes, you must add that property to the TMC-ETR node (for both DT and
>>>>>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>>>>>> locked up the system while using the SG mode (due to the interconnect
>>>>>> issues, something to do with the transaction). Thus, we decided to
>>>>>> add a property explicitly enabling this for a given platform.
>>>>>>
>>>>>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>>>>>> this ? Please be aware that the table allocation code etc are shared
>>>>>> by both TMC-SG and CATU.
>>>>>>
>>>>>
>>>>> You might recall how this started. I had no way to test the CATU due to
>>>>> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
>>>>> purposes, I programmatically swapped the ETR-SG/CATU order and could
>>>>> then verify CATU operation by the driver calling into CATU code. This
>>>>
>>>> So, were you using the DT based boot for the above runs ?
>>>>
>>>>> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
>>>>> prior to this hack. This didn't offer the user any control, hence my
>>>>> feature request. Note that most of the early Ampere self-hosted trace
>>>>> collection used ETR-SG. Now I can't select it.
>>>>
>>>>
>>>>>
>>>>> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
>>>>> Platform Design Document" (DEN0067) doesn't describe this.
>>>>
>>>> This is not specified in the ACPI platform design document. I can get
>>>> it fixed. Ideally we need a property describing that the scatter-gather
>>>> mode is safe to use.
>>>
>>> Looks like this is not straight forward copying of DT property. We are
>>> investigating this on our side and will get back to you.
> 
> I noticed this work is queued for 6.7 (coresight-next-6.7). Do you have
> an update to the ACPI platform design document Ampere can use to base an
> update so the scatter-gather mode can be used?

I have chased this with the specification team, and we should have
something published soon. Apologies for the delay.

Suzuki

> 
> Thanks,
> Steve
> 
>>>
>>
>> The intent behind my request was to have a way to SysFS configure the
>> (available) ETR mode. Unless there's a change to the ACPI, the CATU is
>> the only SG option for Ampere in the near term.
>>
>> Thanks,
>> Steve
>>
>>
>>> Suzuki
>>>
>>>
>>>>
>>>> DT uses "arm,scatter-gather" property [0] and this is what we now expect
>>>> in the ACPI based systems too.
>>>>
>>>> https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73
>>>>
>>>> Does it sound fine ?
>>>>
>>>> Suzuki
>>>>
>>>>>
>>>>> Thanks,
>>>>> Steve
>>>>>
>>>>>
>>>>>> Kind regards
>>>>>> Suzuki
>>>>>>
>>>>>>>
>>>>>>> Steve C.
>>>>>>>
>>>>>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>>>>>
>>>>>>>> Here's some quick feedback. My system shows two modes available; auto
>>>>>>>> catu
>>>>>>>>
>>>>>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>>>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>>>>>
>>>>>>>> Note that both the auto and catu etr_buf_mode_current settings
>>>>>>>> default
>>>>>>>> to catu. My understanding is auto should revert to the default
>>>>>>>> behavior.
>>>>>>>> On my system the default was tmc-sg.
>>>>>>>>
>>>>>>>> More later.
>>>>>>>>
>>>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>>>>>
>>>>>>>> auto catu
>>>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>>> catu
>>>>>>>> [root@sut01sys-b212 kernel]# echo "catu" >
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>>> [root@sut01sys-b212 kernel]# cat
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>>> catu
>>>>>>>>
>>>>>>>> Steve C.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>>>>>
>>>>>>>>> Hi Suzuki,
>>>>>>>>>
>>>>>>>>> I may be able to test it this week. You've already pointed me at the
>>>>>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>>>>>> patches to apply directly to 6.4 last I tried.
>>>>>>>>>
>>>>>>>>> Steve C.
>>>>>>>>>
>>>>>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>>>>>> Cc: Steve
>>>>>>>>>>
>>>>>>>>>> Steve,
>>>>>>>>>>
>>>>>>>>>> Are you able to test this with CATU ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>>>>>> available
>>>>>>>>>>> methods in the following sequentially fallback manner - also in
>>>>>>>>>>> that
>>>>>>>>>>> order.
>>>>>>>>>>>
>>>>>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>>>>>> 3. CATU mode when available
>>>>>>>>>>>
>>>>>>>>>>> But this order might not be ideal for all situations. For
>>>>>>>>>>> example if
>>>>>>>>>>> there
>>>>>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR
>>>>>>>>>>> scatter
>>>>>>>>>>> gather
>>>>>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>>>>>> prevent
>>>>>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>>>>>> following
>>>>>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>>>>>> explicitly,
>>>>>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>>>>>> selection
>>>>>>>>>>> purpose explicitly in the user space.
>>>>>>>>>>>
>>>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>>>>>
>>>>>>>>>>> $ cat buf_modes_available
>>>>>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR
>>>>>>>>>>> buffer
>>>>>>>>>>> modes
>>>>>>>>>>>
>>>>>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>>>>>> request
>>>>>>>>>>>
>>>>>>>>>>> But explicit user request has to be within supported ETR buffer
>>>>>>>>>>> modes
>>>>>>>>>>> only.
>>>>>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>>>>>> are
>>>>>>>>>>> not
>>>>>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>>>>>
>>>>>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help
>>>>>>>>>>> fallback
>>>>>>>>>>> to the
>>>>>>>>>>> existing default behaviour, when user provided preferred buffer
>>>>>>>>>>> mode
>>>>>>>>>>> fails.
>>>>>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>>>>>> modes.
>>>>>>>>>>>
>>>>>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
>>>>>>>>>>> Cc: Mike Leach <mike.leach@linaro.org>
>>>>>>>>>>> Cc: James Clark <james.clark@arm.com>
>>>>>>>>>>> Cc: Leo Yan <leo.yan@linaro.org>
>>>>>>>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>>>>>>>> Cc: coresight@lists.linaro.org
>>>>>>>>>>> Cc: linux-arm-kernel@lists.infradead.org
>>>>>>>>>>> Cc: linux-kernel@vger.kernel.org
>>>>>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>>>>>>>>>> ---
>>>>>>>>>>> This applies on v6.5-rc6
>>>>>>>>>>>
>>>>>>>>>>> Changes in V2:
>>>>>>>>>>>
>>>>>>>>>>> - Renamed sysfs file etr_buf_modes_available as
>>>>>>>>>>> buf_modes_available
>>>>>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>>>>>> - Dropped etr_can_use_flat_mode() check while offering
>>>>>>>>>>> ETR_MODE_FLAT
>>>>>>>>>>> in sysfs
>>>>>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>>>>>> coresight-tmc-etr.c
>>>>>>>>>>> - Updated month in
>>>>>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>>>>>
>>>>>>>>>>> Changes in V1:
>>>>>>>>>>>
>>>>>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>>>>>>
>>>>>>>>>>>      .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>>>>>      .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>>>>>      .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>>>>>> ++++++++++++++++--
>>>>>>>>>>>      drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>>>>>      4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Looks good to me.
>>>>>>>>>>
>>>>>>>>>> Suzuki
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>
>>>