linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device
@ 2022-04-07 12:58 Yicong Yang
  2022-04-07 12:58 ` [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
                   ` (7 more replies)
  0 siblings, 8 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
integrated Endpoint (RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic (tune),
and trace the TLP headers (trace).

PTT tune is designed for monitoring and adjusting PCIe link parameters.
We provide several parameters of the PCIe link. Through the driver,
user can adjust the value of certain parameter to affect the PCIe link
for the purpose of enhancing the performance in certian situation.

PTT trace is designed for dumping the TLP headers to the memory, which
can be used to analyze the transactions and usage condition of the PCIe
Link. Users can choose filters to trace headers, by either requester
ID, or those downstream of a set of Root Ports on the same core of the
PTT device. It's also supported to trace the headers of certain type and
of certain direction.

The driver registers a PMU device for each PTT device. The trace can
be used through `perf record` and the traced headers can be decoded
by `perf report`. The perf command support for the device is also
added in this patchset. The tune can be used through the sysfs
attributes of related PMU device. See the documentation for the
detailed usage.

Change since v6:
- Fix W=1 errors reported by lkp test, thanks

Change since v5:
- Squash the PMU patch into PATCH 2 suggested by John
- refine the commit message of PATCH 1 and some comments
Link: https://lore.kernel.org/lkml/20220308084930.5142-1-yangyicong@hisilicon.com/

Change since v4:
Address the comments from Jonathan, John and Ma Ca, thanks.
- Use devm* also for allocating the DMA buffers
- Remove the IRQ handler stub in Patch 2
- Make functions waiting for hardware state return boolean
- Manual remove the PMU device as it should be removed first
- Modifier the orders in probe and removal to make them matched well
- Make available {directions,type,format} array const and non-global
- Using the right filter list in filters show and well protect the
  list with mutex
- Record the trace status with a boolean @started rather than enum
- Optimize the process of finding the PTT devices of the perf-tool
Link: https://lore.kernel.org/linux-pci/20220221084307.33712-1-yangyicong@hisilicon.com/

Change since v3:
Address the comments from Jonathan and John, thanks.
- drop members in the common struct which can be get on the fly
- reduce buffer struct and organize the buffers with array instead of list
- reduce the DMA reset wait time to avoid long time busy loop
- split the available_filters sysfs attribute into two files, for root port
  and requester respectively. Update the documentation accordingly
- make IOMMU mapping check earlier in probe to avoid race condition. Also
  make IOMMU quirk patch prior to driver in the series
- Cleanups and typos fixes from John and Jonathan
Link: https://lore.kernel.org/linux-pci/20220124131118.17887-1-yangyicong@hisilicon.com/

Change since v2:
- address the comments from Mathieu, thanks.
  - rename the directory to ptt to match the function of the device
  - spinoff the declarations to a separate header
  - split the trace function to several patches
  - some other comments.
- make default smmu domain type of PTT device to identity
  Drop the RMR as it's not recommended and use an iommu_def_domain_type
  quirk to passthrough the device DMA as suggested by Robin. 
Link: https://lore.kernel.org/linux-pci/20211116090625.53702-1-yangyicong@hisilicon.com/

Change since v1:
- switch the user interface of trace to perf from debugfs
- switch the user interface of tune to sysfs from debugfs
- add perf tool support to start trace and decode the trace data
- address the comments of documentation from Bjorn
- add RMR[1] support of the device as trace works in RMR mode or
  direct DMA mode. RMR support is achieved by common APIs rather
  than the APIs implemented in [1].
Link: https://lore.kernel.org/lkml/1618654631-42454-1-git-send-email-yangyicong@hisilicon.com/
[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/

Qi Liu (1):
  perf tool: Add support for HiSilicon PCIe Tune and Trace device driver

Yicong Yang (6):
  iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to
    identity
  hwtracing: Add trace function support for HiSilicon PCIe Tune and
    Trace device
  hisi_ptt: Add support for dynamically updating the filter list
  hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace
    device
  docs: Add HiSilicon PTT device driver documentation
  MAINTAINERS: Add maintainer for HiSilicon PTT driver

 Documentation/trace/hisi-ptt.rst              |  303 +++++
 MAINTAINERS                                   |    7 +
 drivers/Makefile                              |    1 +
 drivers/hwtracing/Kconfig                     |    2 +
 drivers/hwtracing/ptt/Kconfig                 |   12 +
 drivers/hwtracing/ptt/Makefile                |    2 +
 drivers/hwtracing/ptt/hisi_ptt.c              | 1161 +++++++++++++++++
 drivers/hwtracing/ptt/hisi_ptt.h              |  220 ++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |   16 +
 tools/perf/arch/arm/util/auxtrace.c           |   76 +-
 tools/perf/arch/arm/util/pmu.c                |    3 +
 tools/perf/arch/arm64/util/Build              |    2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         |  195 +++
 tools/perf/util/Build                         |    2 +
 tools/perf/util/auxtrace.c                    |    4 +
 tools/perf/util/auxtrace.h                    |    1 +
 tools/perf/util/hisi-ptt-decoder/Build        |    1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   |  170 +++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |   28 +
 tools/perf/util/hisi_ptt.c                    |  218 ++++
 tools/perf/util/hisi_ptt.h                    |   28 +
 21 files changed, 2448 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/trace/hisi-ptt.rst
 create mode 100644 drivers/hwtracing/ptt/Kconfig
 create mode 100644 drivers/hwtracing/ptt/Makefile
 create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
 create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

-- 
2.24.0


^ permalink raw reply	[flat|nested] 30+ messages in thread

* [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
@ 2022-04-07 12:58 ` Yicong Yang
  2022-05-10 11:23   ` Will Deacon
  2022-05-11 11:42   ` John Garry
  2022-04-07 12:58 ` [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device Yicong Yang
                   ` (6 subsequent siblings)
  7 siblings, 2 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

The DMA operations of HiSilicon PTT device can only work properly with
identical mappings. So add a quirk for the device to force the domain
as passthrough.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 627a3ed5ee8f..5ec15ae2a9b1 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2839,6 +2839,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
 	}
 }
 
+#define IS_HISI_PTT_DEVICE(pdev)	((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
+					 (pdev)->device == 0xa12e)
+
+static int arm_smmu_def_domain_type(struct device *dev)
+{
+	if (dev_is_pci(dev)) {
+		struct pci_dev *pdev = to_pci_dev(dev);
+
+		if (IS_HISI_PTT_DEVICE(pdev))
+			return IOMMU_DOMAIN_IDENTITY;
+	}
+
+	return 0;
+}
+
 static struct iommu_ops arm_smmu_ops = {
 	.capable		= arm_smmu_capable,
 	.domain_alloc		= arm_smmu_domain_alloc,
@@ -2856,6 +2871,7 @@ static struct iommu_ops arm_smmu_ops = {
 	.sva_unbind		= arm_smmu_sva_unbind,
 	.sva_get_pasid		= arm_smmu_sva_get_pasid,
 	.page_response		= arm_smmu_page_response,
+	.def_domain_type	= arm_smmu_def_domain_type,
 	.pgsize_bitmap		= -1UL, /* Restricted during device attach */
 	.owner			= THIS_MODULE,
 	.default_domain_ops = &(const struct iommu_domain_ops) {
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
  2022-04-07 12:58 ` [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
@ 2022-04-07 12:58 ` Yicong Yang
  2022-04-11 17:02   ` John Garry
                     ` (2 more replies)
  2022-04-07 12:58 ` [PATCH v7 3/7] hisi_ptt: Add support for dynamically updating the filter list Yicong Yang
                   ` (5 subsequent siblings)
  7 siblings, 3 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
Endpoint(RCiEP) device, providing the capability to dynamically monitor and
tune the PCIe traffic, and trace the TLP headers.

Add the driver for the device to enable the trace function. Register PMU
device of PTT trace, then users can use trace through perf command. The
driver makes use of perf AUX trace and support following events to
configure the trace:

- filter: select Root port or Endpoint to trace
- type: select the type of traced TLP headers
- direction: select the direction of traced TLP headers
- format: select the data format of the traced TLP headers

This patch adds the driver part of PTT trace. The perf command support of
PTT trace is added in the following patch.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 drivers/Makefile                 |   1 +
 drivers/hwtracing/Kconfig        |   2 +
 drivers/hwtracing/ptt/Kconfig    |  12 +
 drivers/hwtracing/ptt/Makefile   |   2 +
 drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
 drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
 6 files changed, 1057 insertions(+)
 create mode 100644 drivers/hwtracing/ptt/Kconfig
 create mode 100644 drivers/hwtracing/ptt/Makefile
 create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
 create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h

diff --git a/drivers/Makefile b/drivers/Makefile
index 020780b6b4d2..662d50599467 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
 obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
 obj-y				+= hwtracing/intel_th/
 obj-$(CONFIG_STM)		+= hwtracing/stm/
+obj-$(CONFIG_HISI_PTT)		+= hwtracing/ptt/
 obj-$(CONFIG_ANDROID)		+= android/
 obj-$(CONFIG_NVMEM)		+= nvmem/
 obj-$(CONFIG_FPGA)		+= fpga/
diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
index 13085835a636..911ee977103c 100644
--- a/drivers/hwtracing/Kconfig
+++ b/drivers/hwtracing/Kconfig
@@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
 
 source "drivers/hwtracing/intel_th/Kconfig"
 
+source "drivers/hwtracing/ptt/Kconfig"
+
 endmenu
diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
new file mode 100644
index 000000000000..8902a6f27563
--- /dev/null
+++ b/drivers/hwtracing/ptt/Kconfig
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config HISI_PTT
+	tristate "HiSilicon PCIe Tune and Trace Device"
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
+	help
+	  HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
+	  device, and it provides support for PCIe traffic tuning and
+	  tracing TLP headers to the memory.
+
+	  This driver can also be built as a module. If so, the module
+	  will be called hisi_ptt.
diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
new file mode 100644
index 000000000000..908c09a98161
--- /dev/null
+++ b/drivers/hwtracing/ptt/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
new file mode 100644
index 000000000000..242b41870380
--- /dev/null
+++ b/drivers/hwtracing/ptt/hisi_ptt.c
@@ -0,0 +1,874 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for HiSilicon PCIe tune and trace device
+ *
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ * Author: Yicong Yang <yangyicong@hisilicon.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/dma-iommu.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iommu.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/sysfs.h>
+#include <linux/vmalloc.h>
+
+#include "hisi_ptt.h"
+
+static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
+{
+	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
+		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
+
+	return PCI_DEVID(pdev->bus->number, pdev->devfn);
+}
+
+static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
+					  val, val & HISI_PTT_TRACE_IDLE,
+					  HISI_PTT_WAIT_POLL_INTERVAL_US,
+					  HISI_PTT_WAIT_TRACE_TIMEOUT_US);
+}
+
+static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
+					  val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
+					  HISI_PTT_RESET_TIMEOUT_US);
+}
+
+static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	int i;
+
+	if (!ctrl->trace_buf)
+		return;
+
+	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
+		if (ctrl->trace_buf[i].addr)
+			dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
+					   ctrl->trace_buf[i].addr,
+					   ctrl->trace_buf[i].dma);
+	}
+
+	devm_kfree(dev, ctrl->trace_buf);
+	ctrl->trace_buf = NULL;
+}
+
+static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	int i;
+
+	hisi_ptt->trace_ctrl.buf_index = 0;
+
+	/* If the trace buffer has already been allocated, zero it. */
+	if (ctrl->trace_buf) {
+		for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
+			memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
+		return 0;
+	}
+
+	ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
+				       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
+	if (!ctrl->trace_buf)
+		return -ENOMEM;
+
+	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
+		ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
+							     &ctrl->trace_buf[i].dma,
+							     GFP_KERNEL);
+		if (!ctrl->trace_buf[i].addr) {
+			hisi_ptt_free_trace_buf(hisi_ptt);
+			return -ENOMEM;
+		}
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
+{
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	hisi_ptt->trace_ctrl.started = false;
+}
+
+static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	u32 val;
+	int i;
+
+	/* Check device idle before start trace */
+	if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
+		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
+		return -EBUSY;
+	}
+
+	ctrl->started = true;
+
+	/* Reset the DMA before start tracing */
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val |= HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	hisi_ptt_wait_dma_reset_done(hisi_ptt);
+
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val &= ~HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	/* Clear the interrupt status */
+	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
+
+	/* Configure the trace DMA buffer */
+	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
+		writel(lower_32_bits(ctrl->trace_buf[i].dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
+		       i * HISI_PTT_TRACE_ADDR_STRIDE);
+		writel(upper_32_bits(ctrl->trace_buf[i].dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
+		       i * HISI_PTT_TRACE_ADDR_STRIDE);
+	}
+	writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
+
+	/* Set the trace control register */
+	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
+	if (!hisi_ptt->trace_ctrl.is_port)
+		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
+
+	/* Start the Trace */
+	val |= HISI_PTT_TRACE_CTRL_EN;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	return 0;
+}
+
+static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct perf_output_handle *handle = &ctrl->handle;
+	struct perf_event *event = handle->event;
+	struct hisi_ptt_pmu_buf *buf;
+	void *addr;
+
+	buf = perf_get_aux(handle);
+	if (!buf || !handle->size)
+		return -EINVAL;
+
+	addr = ctrl->trace_buf[ctrl->buf_index].addr;
+
+	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
+	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);
+	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
+
+	if (stop) {
+		perf_aux_output_end(handle, buf->pos);
+	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
+		perf_aux_output_skip(handle, buf->length - buf->pos);
+		perf_aux_output_end(handle, buf->pos);
+
+		buf = perf_aux_output_begin(handle, event);
+		if (!buf)
+			return -EINVAL;
+
+		buf->pos = handle->head % buf->length;
+		if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
+			perf_aux_output_end(handle, 0);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static irqreturn_t hisi_ptt_isr(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status, buf_idx;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	buf_idx = ffs(status) - 1;
+
+	/* Clear the interrupt status of buffer @buf_idx */
+	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+
+	/*
+	 * Update the AUX buffer and cache the current buffer index,
+	 * as we need to know this and save the data when the trace
+	 * is ended out of the interrupt handler. End the trace
+	 * if the updating fails.
+	 */
+	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
+		hisi_ptt_trace_end(hisi_ptt);
+	else
+		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUF_CNT;
+
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t hisi_ptt_irq(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
+		return IRQ_NONE;
+
+	return IRQ_WAKE_THREAD;
+}
+
+static void hisi_ptt_irq_free_vectors(void *pdev)
+{
+	pci_free_irq_vectors(pdev);
+}
+
+static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	int ret;
+
+	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
+	if (ret < 0) {
+		pci_err(pdev, "failed to allocate irq vector, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
+	if (ret < 0)
+		return ret;
+
+	ret = devm_request_threaded_irq(&pdev->dev,
+					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
+					hisi_ptt_irq, hisi_ptt_isr, 0,
+					DRV_NAME, hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to request irq %d, ret = %d\n",
+			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+{
+	struct hisi_ptt_filter_desc *filter;
+	struct hisi_ptt *hisi_ptt = data;
+	struct list_head *target_list;
+
+	target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
+		      &hisi_ptt->port_filters : &hisi_ptt->req_filters;
+
+	filter = kzalloc(sizeof(*filter), GFP_KERNEL);
+	if (!filter) {
+		pci_err(hisi_ptt->pdev, "failed to add filter %s\n", pci_name(pdev));
+		return -ENOMEM;
+	}
+
+	filter->pdev = pdev;
+	list_add_tail(&filter->list, target_list);
+
+	/* Update the available port mask */
+	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
+		hisi_ptt->port_mask |= hisi_ptt_get_filter_val(pdev);
+
+	return 0;
+}
+
+static void hisi_ptt_release_filters(void *data)
+{
+	struct hisi_ptt_filter_desc *filter, *tfilter;
+	struct hisi_ptt *hisi_ptt = data;
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+}
+
+static int hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	struct pci_bus *bus;
+	int ret;
+	u32 reg;
+
+	INIT_LIST_HEAD(&hisi_ptt->port_filters);
+	INIT_LIST_HEAD(&hisi_ptt->req_filters);
+
+	/*
+	 * The device range register provides the information about the
+	 * root ports which the RCiEP can control and trace. The RCiEP
+	 * and the root ports it support are on the same PCIe core, with
+	 * same domain number but maybe different bus number. The device
+	 * range register will tell us which root ports we can support,
+	 * Bit[31:16] indicates the upper BDF numbers of the root port,
+	 * while Bit[15:0] indicates the lower.
+	 */
+	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
+	hisi_ptt->upper = FIELD_GET(HISI_PTT_DEVICE_RANGE_UPPER, reg);
+	hisi_ptt->lower = FIELD_GET(HISI_PTT_DEVICE_RANGE_LOWER, reg);
+
+	bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
+	if (bus)
+		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
+
+	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_release_filters, hisi_ptt);
+	if (ret)
+		return ret;
+
+	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
+
+	return 0;
+}
+
+#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
+#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
+#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
+#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
+#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
+
+static ssize_t available_root_port_filters_show(struct device *dev,
+						struct device_attribute *attr,
+						char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct hisi_ptt_filter_desc *filter;
+	int pos = 0;
+
+	mutex_lock(&hisi_ptt->mutex);
+	if (list_empty(&hisi_ptt->port_filters)) {
+		pos = sysfs_emit(buf, "\n");
+		goto out;
+	}
+
+	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev) |
+				     HISI_PTT_PMU_FILTER_IS_PORT);
+
+out:
+	mutex_unlock(&hisi_ptt->mutex);
+	return pos;
+}
+static DEVICE_ATTR_ADMIN_RO(available_root_port_filters);
+
+static ssize_t available_requester_filters_show(struct device *dev,
+						struct device_attribute *attr,
+						char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct hisi_ptt_filter_desc *filter;
+	int pos = 0;
+
+	mutex_lock(&hisi_ptt->mutex);
+	if (list_empty(&hisi_ptt->req_filters)) {
+		pos = sysfs_emit(buf, "\n");
+		goto out;
+	}
+
+	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev));
+
+out:
+	mutex_unlock(&hisi_ptt->mutex);
+	return pos;
+}
+static DEVICE_ATTR_ADMIN_RO(available_requester_filters);
+
+PMU_FORMAT_ATTR(filter,		"config:0-19");
+PMU_FORMAT_ATTR(direction,	"config:20-23");
+PMU_FORMAT_ATTR(type,		"config:24-31");
+PMU_FORMAT_ATTR(format,		"config:32-35");
+
+static struct attribute *hisi_ptt_pmu_format_attrs[] = {
+	&format_attr_filter.attr,
+	&format_attr_direction.attr,
+	&format_attr_type.attr,
+	&format_attr_format.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_format_group = {
+	.name = "format",
+	.attrs = hisi_ptt_pmu_format_attrs,
+};
+
+static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
+	&dev_attr_available_root_port_filters.attr,
+	&dev_attr_available_requester_filters.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_filter_group = {
+	.attrs = hisi_ptt_pmu_filter_attrs,
+};
+
+static const struct attribute_group *hisi_ptt_pmu_groups[] = {
+	&hisi_ptt_pmu_format_group,
+	&hisi_ptt_pmu_filter_group,
+	NULL
+};
+
+/*
+ * Check whether the config is valid or not. Some configs are multi-selectable
+ * and can be set simultaneously, while some are single selectable (onehot).
+ * Use this function to check the non-onehot configs while
+ * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
+ */
+static int hisi_ptt_trace_valid_config(u32 val, const u32 *available_list, u32 list_size)
+{
+	int i;
+
+	/* The non-onehot configs cannot be 0. */
+	if (!val)
+		return -EINVAL;
+
+	/*
+	 * Walk the available list and clear the valid bits of
+	 * the config. If there is any resident bit after the
+	 * walk then the config is invalid.
+	 */
+	for (i = 0; i < list_size; i++)
+		val &= ~available_list[i];
+
+	return val ? -EINVAL : 0;
+}
+
+static int hisi_ptt_trace_valid_config_onehot(u32 val, const u32 *available_list, u32 list_size)
+{
+	int i;
+
+	for (i = 0; i < list_size; i++)
+		if (val == available_list[i])
+			return 0;
+
+	return -EINVAL;
+}
+
+static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
+{
+	unsigned long val, port_mask = hisi_ptt->port_mask;
+	struct hisi_ptt_filter_desc *filter;
+	int ret = -EINVAL;
+
+	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
+	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
+
+	/*
+	 * Port filters are defined as bit mask. For port filters, check
+	 * the bits in the @val are within the range of hisi_ptt->port_mask
+	 * and whether it's empty or not, otherwise user has specified
+	 * some unsupported root ports.
+	 *
+	 * For Requester ID filters, walk the available filter list to see
+	 * whether we have one matched.
+	 */
+	if (!hisi_ptt->trace_ctrl.is_port) {
+		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+			if (val == hisi_ptt_get_filter_val(filter->pdev)) {
+				ret = 0;
+				break;
+			}
+	} else if (bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
+		ret = 0;
+	}
+
+	if (ret)
+		return ret;
+
+	hisi_ptt->trace_ctrl.filter = val;
+	return 0;
+}
+
+static int hisi_ptt_pmu_event_init(struct perf_event *event)
+{
+	/*
+	 * The supported value of the direction parameter. See hisi_ptt.rst
+	 * documentation for more details.
+	 */
+	static const u32 hisi_ptt_trace_available_direction[] = {
+		0,
+		1,
+		2,
+		3,
+	};
+	/* Different types can be set simultaneously */
+	static const u32 hisi_ptt_trace_available_type[] = {
+		1,	/* posted_request */
+		2,	/* non-posted_request */
+		4,	/* completion */
+	};
+	static const u32 hisi_ptt_trace_availble_format[] = {
+		0,	/* 4DW */
+		1,	/* 8DW */
+	};
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	int ret;
+	u32 val;
+
+	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
+		return -ENOENT;
+
+	mutex_lock(&hisi_ptt->mutex);
+
+	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
+	if (ret < 0)
+		goto out;
+
+	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
+						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
+	if (ret < 0)
+		goto out;
+	ctrl->direction = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
+					  ARRAY_SIZE(hisi_ptt_trace_available_type));
+	if (ret < 0)
+		goto out;
+	ctrl->type = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
+						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
+	if (ret < 0)
+		goto out;
+	ctrl->format = val;
+
+out:
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret;
+}
+
+static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
+				    int nr_pages, bool overwrite)
+{
+	struct hisi_ptt_pmu_buf *buf;
+	struct page **pagelist;
+	int i;
+
+	if (overwrite) {
+		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
+		return NULL;
+	}
+
+	/* If the pages size less than buffers, we cannot start trace */
+	if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
+		return NULL;
+
+	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
+	if (!buf)
+		return NULL;
+
+	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
+	if (!pagelist) {
+		kfree(buf);
+		return NULL;
+	}
+
+	for (i = 0; i < nr_pages; i++)
+		pagelist[i] = virt_to_page(pages[i]);
+
+	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
+	if (!buf->base) {
+		kfree(pagelist);
+		kfree(buf);
+		return NULL;
+	}
+
+	buf->nr_pages = nr_pages;
+	buf->length = nr_pages * PAGE_SIZE;
+	buf->pos = 0;
+
+	kfree(pagelist);
+	return buf;
+}
+
+static void hisi_ptt_pmu_free_aux(void *aux)
+{
+	struct hisi_ptt_pmu_buf *buf = aux;
+
+	vunmap(buf->base);
+	kfree(buf);
+}
+
+static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+	struct hw_perf_event *hwc = &event->hw;
+	struct hisi_ptt_pmu_buf *buf;
+	int cpu = event->cpu;
+	int ret;
+
+	hwc->state = 0;
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.started) {
+		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
+		goto stop;
+	}
+
+	if (cpu == -1)
+		cpu = hisi_ptt->trace_ctrl.default_cpu;
+
+	/*
+	 * Handle the interrupt on the same cpu which starts the trace to avoid
+	 * context mismatch. Otherwise we'll trigger the WARN from the perf
+	 * core in event_function_local().
+	 */
+	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
+				 cpumask_of(cpu)));
+
+	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
+		goto stop;
+	}
+
+	buf = perf_aux_output_begin(handle, event);
+	if (!buf) {
+		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
+		goto stop;
+	}
+
+	buf->pos = handle->head % buf->length;
+
+	ret = hisi_ptt_trace_start(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
+		perf_aux_output_end(handle, 0);
+		goto stop;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return;
+stop:
+	event->hw.state |= PERF_HES_STOPPED;
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+
+	if (hwc->state & PERF_HES_STOPPED)
+		return;
+
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.started) {
+		hisi_ptt_trace_end(hisi_ptt);
+		WARN(!hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
+		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
+	}
+	mutex_unlock(&hisi_ptt->mutex);
+
+	hwc->state |= PERF_HES_STOPPED;
+	perf_event_update_userpage(event);
+	hwc->state |= PERF_HES_UPTODATE;
+}
+
+static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+	int cpu = event->cpu;
+
+	/*
+	 * Only allow the default cpu to add the event if user doesn't specify
+	 * the cpus.
+	 */
+	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
+		return 0;
+
+	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+
+	if (flags & PERF_EF_START) {
+		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
+		if (hwc->state & PERF_HES_STOPPED)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
+{
+	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
+}
+
+static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
+{
+	u16 core_id, sicl_id;
+	char *pmu_name;
+	u32 reg;
+
+	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
+		.module		= THIS_MODULE,
+		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
+		.task_ctx_nr	= perf_sw_context,
+		.attr_groups	= hisi_ptt_pmu_groups,
+		.event_init	= hisi_ptt_pmu_event_init,
+		.setup_aux	= hisi_ptt_pmu_setup_aux,
+		.free_aux	= hisi_ptt_pmu_free_aux,
+		.start		= hisi_ptt_pmu_start,
+		.stop		= hisi_ptt_pmu_stop,
+		.add		= hisi_ptt_pmu_add,
+		.del		= hisi_ptt_pmu_del,
+	};
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
+	core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
+	sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
+
+	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
+				  sicl_id, core_id);
+	if (!pmu_name)
+		return -ENOMEM;
+
+	return perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
+}
+
+/*
+ * The DMA of PTT trace can only use direct mapping, due to some
+ * hardware restriction. Check whether there is an IOMMU or the
+ * policy of the IOMMU domain is passthrough, otherwise the trace
+ * cannot work.
+ *
+ * The PTT device is supposed to behind the ARM SMMUv3, which
+ * should have passthrough the device by a quirk.
+ */
+static int hisi_ptt_check_iommu_mapping(struct pci_dev *pdev)
+{
+	struct iommu_domain *iommu_domain;
+
+	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
+	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
+		return 0;
+
+	return -EOPNOTSUPP;
+}
+
+static int hisi_ptt_probe(struct pci_dev *pdev,
+			  const struct pci_device_id *id)
+{
+	struct hisi_ptt *hisi_ptt;
+	int ret;
+
+	ret = hisi_ptt_check_iommu_mapping(pdev);
+	if (ret) {
+		pci_err(pdev, "requires direct DMA mappings\n");
+		return ret;
+	}
+
+	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
+	if (!hisi_ptt)
+		return -ENOMEM;
+
+	mutex_init(&hisi_ptt->mutex);
+	hisi_ptt->pdev = pdev;
+	pci_set_drvdata(pdev, hisi_ptt);
+
+	ret = pcim_enable_device(pdev);
+	if (ret) {
+		pci_err(pdev, "failed to enable device, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = pcim_iomap_regions(pdev, BIT(2), DRV_NAME);
+	if (ret) {
+		pci_err(pdev, "failed to remap io memory, ret = %d\n", ret);
+		return ret;
+	}
+
+	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
+
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret) {
+		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d\n", ret);
+		return ret;
+	}
+
+	pci_set_master(pdev);
+
+	ret = hisi_ptt_register_irq(hisi_ptt);
+	if (ret)
+		return ret;
+
+	ret = hisi_ptt_init_ctrls(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to init controls, ret = %d\n", ret);
+		return ret;
+	}
+
+	ret = hisi_ptt_register_pmu(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to register PMU device, ret = %d", ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_remove(struct pci_dev *pdev)
+{
+	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
+
+	/*
+	 * We have to manually unregister the PMU device rather than make it
+	 * devres managed to keep order that the PMU device's unregistration
+	 * is prior to the release of DMA buffers. As the DMA buffers are
+	 * devm allocated when necessary which is after the registration of
+	 * the PMU device.
+	 */
+	perf_pmu_unregister(&hisi_ptt->hisi_ptt_pmu);
+}
+
+static const struct pci_device_id hisi_ptt_id_tbl[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
+	{ }
+};
+MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
+
+static struct pci_driver hisi_ptt_driver = {
+	.name = DRV_NAME,
+	.id_table = hisi_ptt_id_tbl,
+	.probe = hisi_ptt_probe,
+	.remove = hisi_ptt_remove,
+};
+module_pci_driver(hisi_ptt_driver);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
+MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
new file mode 100644
index 000000000000..29b6d6293d6c
--- /dev/null
+++ b/drivers/hwtracing/ptt/hisi_ptt.h
@@ -0,0 +1,166 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Driver for HiSilicon PCIe tune and trace device
+ *
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ * Author: Yicong Yang <yangyicong@hisilicon.com>
+ */
+
+#ifndef _HISI_PTT_H
+#define _HISI_PTT_H
+
+#include <linux/bits.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/perf_event.h>
+#include <linux/types.h>
+
+#define DRV_NAME "hisi_ptt"
+
+/*
+ * The definition of the device registers and register fields.
+ */
+#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
+#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
+#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
+#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
+#define HISI_PTT_TRACE_CTRL		0x0850
+#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
+#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
+#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
+#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
+#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
+#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
+#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
+#define HISI_PTT_TRACE_INT_STAT		0x0890
+#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
+#define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TRACE_WR_STS		0x08a0
+#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
+#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
+#define HISI_PTT_TRACE_STS		0x08b0
+#define   HISI_PTT_TRACE_IDLE		BIT(0)
+#define HISI_PTT_DEVICE_RANGE		0x0fe0
+#define   HISI_PTT_DEVICE_RANGE_UPPER	GENMASK(31, 16)
+#define   HISI_PTT_DEVICE_RANGE_LOWER	GENMASK(15, 0)
+#define HISI_PTT_LOCATION		0x0fe8
+#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
+#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
+
+/* Parameters of PTT trace DMA part. */
+#define HISI_PTT_TRACE_DMA_IRQ			0
+#define HISI_PTT_TRACE_BUF_CNT			4
+#define HISI_PTT_TRACE_BUF_SIZE			SZ_4M
+#define HISI_PTT_TRACE_TOTAL_BUF_SIZE		(HISI_PTT_TRACE_BUF_SIZE * \
+						 HISI_PTT_TRACE_BUF_CNT)
+/* Wait time for hardware DMA to reset */
+#define HISI_PTT_RESET_TIMEOUT_US	10UL
+#define HISI_PTT_RESET_POLL_INTERVAL_US	1UL
+/* Poll timeout and interval for waiting hardware work to finish */
+#define HISI_PTT_WAIT_TRACE_TIMEOUT_US	100UL
+#define HISI_PTT_WAIT_POLL_INTERVAL_US	10UL
+
+#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
+
+/**
+ * struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
+ *                              The detail of the data format is described
+ *                              in the documentation of PTT device.
+ * @dma:   DMA address of this buffer visible to the device
+ * @addr:  virtual address of this buffer visible to the cpu
+ */
+struct hisi_ptt_dma_buffer {
+	dma_addr_t dma;
+	void *addr;
+};
+
+/**
+ * struct hisi_ptt_trace_ctrl - control and status of PTT trace
+ * @trace_buf:   array of the trace buffers for holding the trace data.
+ *               the length will be HISI_PTT_TRACE_BUF_CNT.
+ * @handle:      perf output handle of current trace session
+ * @default_cpu: default cpu to start the trace session
+ * @buf_index:   the index of current using trace buffer
+ * @started:     current trace status, true for started
+ * @is_port:     whether we're tracing root port or not
+ * @direction:   direction of the TLP headers to trace
+ * @filter:      filter value for tracing the TLP headers
+ * @format:      format of the TLP headers to trace
+ * @type:        type of the TLP headers to trace
+ */
+struct hisi_ptt_trace_ctrl {
+	struct hisi_ptt_dma_buffer *trace_buf;
+	struct perf_output_handle handle;
+	int default_cpu;
+	u32 buf_index;
+	bool started;
+	bool is_port;
+	u32 direction:2;
+	u32 filter:16;
+	u32 format:1;
+	u32 type:4;
+};
+
+/**
+ * struct hisi_ptt_filter_desc - descriptor of the PTT trace filter
+ * @list: entry of this descriptor in the filter list
+ * @pdev: pci_dev related to this filter
+ */
+struct hisi_ptt_filter_desc {
+	struct list_head list;
+	struct pci_dev *pdev;
+};
+
+
+/**
+ * struct hisi_ptt_pmu_buf - descriptor of the AUX buffer of PTT trace
+ * @length:   size of the AUX buffer
+ * @nr_pages: number of pages of the AUX buffer
+ * @base:     start address of AUX buffer
+ * @pos:      position in the AUX buffer to commit traced data
+ */
+struct hisi_ptt_pmu_buf {
+	size_t length;
+	int nr_pages;
+	void *base;
+	long pos;
+};
+
+/**
+ * struct hisi_ptt - per PTT device data
+ * @trace_ctrl:   the control information of PTT trace
+ * @hisi_ptt_pmu: the pum device of trace
+ * @iobase:       base IO address of the device
+ * @pdev:         pci_dev of this PTT device
+ * @mutex:        mutex to protect the filter list and serialize the perf process.
+ * @upper:        the upper BDF range of the PCI devices managed by this PTT device
+ * @lower:        the lower BDF range of the PCI devices managed by this PTT device
+ * @port_filters: the filter list of root ports
+ * @req_filters:  the filter list of requester ID
+ * @port_mask:    port mask of the managed root ports
+ */
+struct hisi_ptt {
+	struct hisi_ptt_trace_ctrl trace_ctrl;
+	struct pmu hisi_ptt_pmu;
+	void __iomem *iobase;
+	struct pci_dev *pdev;
+	struct mutex mutex;
+	u32 upper;
+	u32 lower;
+
+	/*
+	 * The trace TLP headers can either be filtered by certain
+	 * root port, or by the requester ID. Organize the filters
+	 * by @port_filters and @req_filters here. The mask of all
+	 * the valid ports is also cached for doing sanity check
+	 * of user input.
+	 */
+	struct list_head port_filters;
+	struct list_head req_filters;
+	u16 port_mask;
+};
+
+#define to_hisi_ptt(pmu) container_of(pmu, struct hisi_ptt, hisi_ptt_pmu)
+
+#endif /* _HISI_PTT_H */
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [PATCH v7 3/7] hisi_ptt: Add support for dynamically updating the filter list
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
  2022-04-07 12:58 ` [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
  2022-04-07 12:58 ` [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device Yicong Yang
@ 2022-04-07 12:58 ` Yicong Yang
  2022-04-07 12:58 ` [PATCH v7 4/7] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device Yicong Yang
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

The PCIe devices supported by the PTT trace can be removed/rescanned by
hotplug or through sysfs.  Add support for dynamically updating the
available filter list by registering a PCI bus notifier block. Then user
can always get latest information about available tracing filters and
driver can block the invalid filters of which related devices no longer
exist in the system.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 drivers/hwtracing/ptt/hisi_ptt.c | 159 ++++++++++++++++++++++++++++---
 drivers/hwtracing/ptt/hisi_ptt.h |  34 +++++++
 2 files changed, 180 insertions(+), 13 deletions(-)

diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
index 242b41870380..b1958ac20372 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.c
+++ b/drivers/hwtracing/ptt/hisi_ptt.c
@@ -270,27 +270,121 @@ static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
 	return 0;
 }
 
-static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+static void hisi_ptt_update_filters(struct work_struct *work)
 {
+	struct delayed_work *delayed_work = to_delayed_work(work);
+	struct hisi_ptt_filter_update_info info;
 	struct hisi_ptt_filter_desc *filter;
-	struct hisi_ptt *hisi_ptt = data;
 	struct list_head *target_list;
+	struct hisi_ptt *hisi_ptt;
 
-	target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
-		      &hisi_ptt->port_filters : &hisi_ptt->req_filters;
+	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
 
-	filter = kzalloc(sizeof(*filter), GFP_KERNEL);
-	if (!filter) {
-		pci_err(hisi_ptt->pdev, "failed to add filter %s\n", pci_name(pdev));
-		return -ENOMEM;
+	if (!mutex_trylock(&hisi_ptt->mutex)) {
+		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
+		return;
 	}
 
-	filter->pdev = pdev;
-	list_add_tail(&filter->list, target_list);
+	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
+		bool is_port = pci_pcie_type(info.pdev) == PCI_EXP_TYPE_ROOT_PORT;
+		u16 val = hisi_ptt_get_filter_val(info.pdev);
 
-	/* Update the available port mask */
-	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
-		hisi_ptt->port_mask |= hisi_ptt_get_filter_val(pdev);
+		target_list = is_port ? &hisi_ptt->port_filters : &hisi_ptt->req_filters;
+
+		if (info.is_add) {
+			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
+			if (!filter) {
+				pci_err(hisi_ptt->pdev, "failed to add filter %s\n",
+					pci_name(info.pdev));
+				continue;
+			}
+
+			filter->pdev = info.pdev;
+			list_add_tail(&filter->list, target_list);
+		} else {
+			list_for_each_entry(filter, target_list, list)
+				if (hisi_ptt_get_filter_val(filter->pdev) == val) {
+					list_del(&filter->list);
+					kfree(filter);
+					break;
+				}
+		}
+
+		/* Update the available port mask */
+		if (!is_port)
+			continue;
+
+		if (info.is_add)
+			hisi_ptt->port_mask |= val;
+		else
+			hisi_ptt->port_mask &= ~val;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
+				    struct hisi_ptt_filter_update_info *info)
+{
+	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
+	u32 port_devid;
+
+	if (!root_port)
+		return;
+
+	port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
+	if (port_devid < hisi_ptt->lower ||
+	    port_devid > hisi_ptt->upper)
+		return;
+
+	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
+				&hisi_ptt->filter_update_lock))
+		schedule_delayed_work(&hisi_ptt->work, 0);
+	else
+		pci_warn(hisi_ptt->pdev,
+			 "filter update fifo overflow for target %s\n",
+			 pci_name(info->pdev));
+}
+
+/*
+ * A PCI bus notifier is used here for dynamically updating the filter
+ * list.
+ */
+static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
+				  void *data)
+{
+	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
+	struct hisi_ptt_filter_update_info info;
+	struct device *dev = data;
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	info.pdev = pdev;
+
+	switch (action) {
+	case BUS_NOTIFY_ADD_DEVICE:
+		info.is_add = true;
+		break;
+	case BUS_NOTIFY_DEL_DEVICE:
+		info.is_add = false;
+		break;
+	default:
+		return 0;
+	}
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+	return 0;
+}
+
+static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+{
+	struct hisi_ptt_filter_update_info info = {
+		.pdev = pdev,
+		.is_add = true,
+	};
+	struct hisi_ptt *hisi_ptt = data;
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
 
 	return 0;
 }
@@ -318,6 +412,9 @@ static int hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
 	int ret;
 	u32 reg;
 
+	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
+	spin_lock_init(&hisi_ptt->filter_update_lock);
+	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
 	INIT_LIST_HEAD(&hisi_ptt->port_filters);
 	INIT_LIST_HEAD(&hisi_ptt->req_filters);
 
@@ -334,6 +431,13 @@ static int hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
 	hisi_ptt->upper = FIELD_GET(HISI_PTT_DEVICE_RANGE_UPPER, reg);
 	hisi_ptt->lower = FIELD_GET(HISI_PTT_DEVICE_RANGE_LOWER, reg);
 
+	/*
+	 * No need to fail if the bus is NULL here as the device
+	 * maybe hotplugged after the PTT driver probe, in which
+	 * case we can detect the event and update the list as
+	 * we register a bus notifier for dynamically updating
+	 * the filter list.
+	 */
 	bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
 	if (bus)
 		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
@@ -760,6 +864,31 @@ static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
 	return perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
 }
 
+static void hisi_ptt_unregister_filter_update_notifier(void *data)
+{
+	struct hisi_ptt *hisi_ptt = data;
+
+	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+
+	/* Cancel any work that has been queued */
+	cancel_delayed_work_sync(&hisi_ptt->work);
+}
+
+/* Register the bus notifier for dynamically updating the filter list */
+static int hisi_ptt_register_filter_update_notifier(struct hisi_ptt *hisi_ptt)
+{
+	int ret;
+
+	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
+	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+	if (ret)
+		return ret;
+
+	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
+					hisi_ptt_unregister_filter_update_notifier,
+					hisi_ptt);
+}
+
 /*
  * The DMA of PTT trace can only use direct mapping, due to some
  * hardware restriction. Check whether there is an IOMMU or the
@@ -832,6 +961,10 @@ static int hisi_ptt_probe(struct pci_dev *pdev,
 		return ret;
 	}
 
+	ret = hisi_ptt_register_filter_update_notifier(hisi_ptt);
+	if (ret)
+		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
+
 	ret = hisi_ptt_register_pmu(hisi_ptt);
 	if (ret) {
 		pci_err(pdev, "failed to register PMU device, ret = %d", ret);
diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
index 29b6d6293d6c..53805f6c9bc2 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.h
+++ b/drivers/hwtracing/ptt/hisi_ptt.h
@@ -10,11 +10,15 @@
 #define _HISI_PTT_H
 
 #include <linux/bits.h>
+#include <linux/kfifo.h>
 #include <linux/list.h>
 #include <linux/mutex.h>
+#include <linux/notifier.h>
 #include <linux/pci.h>
 #include <linux/perf_event.h>
+#include <linux/spinlock.h>
 #include <linux/types.h>
+#include <linux/workqueue.h>
 
 #define DRV_NAME "hisi_ptt"
 
@@ -57,6 +61,12 @@
 /* Wait time for hardware DMA to reset */
 #define HISI_PTT_RESET_TIMEOUT_US	10UL
 #define HISI_PTT_RESET_POLL_INTERVAL_US	1UL
+/* FIFO size for dynamically updating the PTT trace filter list. */
+#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
+/* Delay time for filter updating work */
+#define HISI_PTT_WORK_DELAY_MS		100UL
+/* Wait time for DMA hardware to reset */
+#define HISI_PTT_RESET_WAIT_MS		1000UL
 /* Poll timeout and interval for waiting hardware work to finish */
 #define HISI_PTT_WAIT_TRACE_TIMEOUT_US	100UL
 #define HISI_PTT_WAIT_POLL_INTERVAL_US	10UL
@@ -112,6 +122,15 @@ struct hisi_ptt_filter_desc {
 	struct pci_dev *pdev;
 };
 
+/**
+ * struct hisi_ptt_filter_update_info - information for PTT filter updating
+ * @pdev:       the PCI device to update in the filter list
+ * @is_add:     adding to the filter or not
+ */
+struct hisi_ptt_filter_update_info {
+	struct pci_dev *pdev;
+	bool is_add;
+};
 
 /**
  * struct hisi_ptt_pmu_buf - descriptor of the AUX buffer of PTT trace
@@ -139,9 +158,13 @@ struct hisi_ptt_pmu_buf {
  * @port_filters: the filter list of root ports
  * @req_filters:  the filter list of requester ID
  * @port_mask:    port mask of the managed root ports
+ * @work:         delayed work for filter updating
+ * @filter_update_lock: spinlock to protect the filter update fifo
+ * @filter_update_fifo: fifo of the filters waiting to update the filter list
  */
 struct hisi_ptt {
 	struct hisi_ptt_trace_ctrl trace_ctrl;
+	struct notifier_block hisi_ptt_nb;
 	struct pmu hisi_ptt_pmu;
 	void __iomem *iobase;
 	struct pci_dev *pdev;
@@ -159,6 +182,17 @@ struct hisi_ptt {
 	struct list_head port_filters;
 	struct list_head req_filters;
 	u16 port_mask;
+
+	/*
+	 * We use a delayed work here to avoid indefinitely waiting for
+	 * the hisi_ptt->mutex which protecting the filter list. The
+	 * work will be delayed only if the mutex can not be held,
+	 * otherwise no delay will be applied.
+	 */
+	struct delayed_work work;
+	spinlock_t filter_update_lock;
+	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
+		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
 };
 
 #define to_hisi_ptt(pmu) container_of(pmu, struct hisi_ptt, hisi_ptt_pmu)
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [PATCH v7 4/7] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
                   ` (2 preceding siblings ...)
  2022-04-07 12:58 ` [PATCH v7 3/7] hisi_ptt: Add support for dynamically updating the filter list Yicong Yang
@ 2022-04-07 12:58 ` Yicong Yang
  2022-04-07 12:58 ` [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

Add tune function for the HiSilicon Tune and Trace device. The interface
of tune is exposed through sysfs attributes of PTT PMU device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 drivers/hwtracing/ptt/hisi_ptt.c | 154 +++++++++++++++++++++++++++++++
 drivers/hwtracing/ptt/hisi_ptt.h |  20 ++++
 2 files changed, 174 insertions(+)

diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
index b1958ac20372..331c8e43cd17 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.c
+++ b/drivers/hwtracing/ptt/hisi_ptt.c
@@ -21,6 +21,159 @@
 
 #include "hisi_ptt.h"
 
+static bool hisi_ptt_wait_tuning_finish(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return !readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TUNING_INT_STAT,
+				  val, !(val & HISI_PTT_TUNING_INT_STAT_MASK),
+				  HISI_PTT_WAIT_POLL_INTERVAL_US,
+				  HISI_PTT_WAIT_TUNE_TIMEOUT_US);
+}
+
+static int hisi_ptt_tune_data_get(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 *data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	/* Write all 1 to indicates it's the read process */
+	writel(~0U, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (!hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+	reg &= HISI_PTT_TUNING_DATA_VAL_MASK;
+	*data = FIELD_GET(HISI_PTT_TUNING_DATA_VAL_MASK, reg);
+
+	return 0;
+}
+
+static int hisi_ptt_tune_data_set(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	writel(FIELD_PREP(HISI_PTT_TUNING_DATA_VAL_MASK, data),
+	       hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (!hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	return 0;
+}
+
+static ssize_t hisi_ptt_tune_attr_show(struct device *dev,
+				       struct device_attribute *attr,
+				       char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_get(hisi_ptt, desc->event_code, &val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : sysfs_emit(buf, "%u\n", val);
+}
+
+static ssize_t hisi_ptt_tune_attr_store(struct device *dev,
+					struct device_attribute *attr,
+					const char *buf, size_t count)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (kstrtou16(buf, 10, &val))
+		return -EINVAL;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_set(hisi_ptt, desc->event_code, val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : count;
+}
+
+#define HISI_PTT_TUNE_ATTR(_name, _val, _show, _store)			\
+	static struct hisi_ptt_tune_desc _name##_desc = {		\
+		.name = #_name,						\
+		.event_code = _val,					\
+	};								\
+	static struct dev_ext_attribute hisi_ptt_##_name##_attr = {	\
+		.attr	= __ATTR(_name, 0600, _show, _store),		\
+		.var	= &_name##_desc,				\
+	}
+
+#define HISI_PTT_TUNE_ATTR_COMMON(_name, _val)		\
+	HISI_PTT_TUNE_ATTR(_name, _val,			\
+			   hisi_ptt_tune_attr_show,	\
+			   hisi_ptt_tune_attr_store)
+
+/*
+ * The value of the tuning event are composed of two parts: main event code in bit[0,15] and
+ * subevent code in bit[16,23]. For example, qox_tx_cpl is a subevent of 'Tx path QoS control'
+ * which for tuning the weight of Tx completion TLPs. See hisi_ptt.rst documentation for
+ * more information.
+ */
+#define HISI_PTT_TUNE_QOS_TX_CPL				(0x4 | (3 << 16))
+#define HISI_PTT_TUNE_QOS_TX_NP					(0x4 | (4 << 16))
+#define HISI_PTT_TUNE_QOS_TX_P					(0x4 | (5 << 16))
+#define HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL	(0x5 | (6 << 16))
+#define HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL		(0x5 | (7 << 16))
+
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_cpl,
+			  HISI_PTT_TUNE_QOS_TX_CPL);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_np,
+			  HISI_PTT_TUNE_QOS_TX_NP);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_p,
+			  HISI_PTT_TUNE_QOS_TX_P);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_iob_rx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_tx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL);
+
+static struct attribute *hisi_ptt_tune_attrs[] = {
+	&hisi_ptt_qos_tx_cpl_attr.attr.attr,
+	&hisi_ptt_qos_tx_np_attr.attr.attr,
+	&hisi_ptt_qos_tx_p_attr.attr.attr,
+	&hisi_ptt_tx_path_iob_rx_req_alloc_buf_level_attr.attr.attr,
+	&hisi_ptt_tx_path_tx_req_alloc_buf_level_attr.attr.attr,
+	NULL,
+};
+
+static struct attribute_group hisi_ptt_tune_group = {
+	.attrs	= hisi_ptt_tune_attrs,
+	.name	= "tune",
+};
+
 static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
 {
 	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
@@ -539,6 +692,7 @@ static struct attribute_group hisi_ptt_pmu_filter_group = {
 static const struct attribute_group *hisi_ptt_pmu_groups[] = {
 	&hisi_ptt_pmu_format_group,
 	&hisi_ptt_pmu_filter_group,
+	&hisi_ptt_tune_group,
 	NULL
 };
 
diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
index 53805f6c9bc2..d88c326e9289 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.h
+++ b/drivers/hwtracing/ptt/hisi_ptt.h
@@ -25,6 +25,11 @@
 /*
  * The definition of the device registers and register fields.
  */
+#define HISI_PTT_TUNING_CTRL		0x0000
+#define   HISI_PTT_TUNING_CTRL_CODE	GENMASK(15, 0)
+#define   HISI_PTT_TUNING_CTRL_SUB	GENMASK(23, 16)
+#define HISI_PTT_TUNING_DATA		0x0004
+#define   HISI_PTT_TUNING_DATA_VAL_MASK	GENMASK(15, 0)
 #define HISI_PTT_TRACE_ADDR_SIZE	0x0800
 #define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
 #define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
@@ -40,6 +45,8 @@
 #define HISI_PTT_TRACE_INT_STAT		0x0890
 #define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
 #define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TUNING_INT_STAT	0x0898
+#define   HISI_PTT_TUNING_INT_STAT_MASK	BIT(0)
 #define HISI_PTT_TRACE_WR_STS		0x08a0
 #define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
 #define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
@@ -68,11 +75,24 @@
 /* Wait time for DMA hardware to reset */
 #define HISI_PTT_RESET_WAIT_MS		1000UL
 /* Poll timeout and interval for waiting hardware work to finish */
+#define HISI_PTT_WAIT_TUNE_TIMEOUT_US	1000000UL
 #define HISI_PTT_WAIT_TRACE_TIMEOUT_US	100UL
 #define HISI_PTT_WAIT_POLL_INTERVAL_US	10UL
 
 #define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
 
+/**
+ * struct hisi_ptt_tune_desc - describe tune event for PTT tune
+ * @hisi_ptt:   PTT device this tune event belongs to
+ * @name:       name of this event
+ * @event_code: code of the event
+ */
+struct hisi_ptt_tune_desc {
+	struct hisi_ptt *hisi_ptt;
+	const char *name;
+	u32 event_code;
+};
+
 /**
  * struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
  *                              The detail of the data format is described
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
                   ` (3 preceding siblings ...)
  2022-04-07 12:58 ` [PATCH v7 4/7] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device Yicong Yang
@ 2022-04-07 12:58 ` Yicong Yang
  2022-04-11 17:19   ` John Garry
                     ` (2 more replies)
  2022-04-07 12:58 ` [PATCH v7 6/7] docs: Add HiSilicon PTT device driver documentation Yicong Yang
                   ` (2 subsequent siblings)
  7 siblings, 3 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

From: Qi Liu <liuqi115@huawei.com>

'perf record' and 'perf report --dump-raw-trace' supported in this
patch.

Example usage:

Output will contain raw PTT data and its textual representation, such
as:

0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
.
. ... HISI PTT data: size 4194304 bytes
.  00000000: 00 00 00 00                                 Prefix
.  00000004: 08 20 00 60                                 Header DW0
.  00000008: ff 02 00 01                                 Header DW1
.  0000000c: 20 08 00 00                                 Header DW2
.  00000010: 10 e7 44 ab                                 Header DW3
.  00000014: 2a a8 1e 01                                 Time
.  00000020: 00 00 00 00                                 Prefix
.  00000024: 01 00 00 60                                 Header DW0
.  00000028: 0f 1e 00 01                                 Header DW1
.  0000002c: 04 00 00 00                                 Header DW2
.  00000030: 40 00 81 02                                 Header DW3
.  00000034: ee 02 00 00                                 Time
....

Signed-off-by: Qi Liu <liuqi115@huawei.com>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 tools/perf/arch/arm/util/auxtrace.c           |  76 +++++-
 tools/perf/arch/arm/util/pmu.c                |   3 +
 tools/perf/arch/arm64/util/Build              |   2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         | 195 ++++++++++++++++
 tools/perf/util/Build                         |   2 +
 tools/perf/util/auxtrace.c                    |   4 +
 tools/perf/util/auxtrace.h                    |   1 +
 tools/perf/util/hisi-ptt-decoder/Build        |   1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 ++++++++++++++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
 tools/perf/util/hisi_ptt.c                    | 218 ++++++++++++++++++
 tools/perf/util/hisi_ptt.h                    |  28 +++
 12 files changed, 724 insertions(+), 4 deletions(-)
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
index 5fc6a2a3dbc5..393f5757c039 100644
--- a/tools/perf/arch/arm/util/auxtrace.c
+++ b/tools/perf/arch/arm/util/auxtrace.c
@@ -4,9 +4,11 @@
  * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
  */
 
+#include <dirent.h>
 #include <stdbool.h>
 #include <linux/coresight-pmu.h>
 #include <linux/zalloc.h>
+#include <api/fs/fs.h>
 
 #include "../../../util/auxtrace.h"
 #include "../../../util/debug.h"
@@ -14,6 +16,7 @@
 #include "../../../util/pmu.h"
 #include "cs-etm.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 
 static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 {
@@ -50,6 +53,58 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 	return arm_spe_pmus;
 }
 
+static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
+{
+	const char *sysfs = sysfs__mountpoint();
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+	struct dirent *dent;
+	char path[PATH_MAX];
+	DIR *dir = NULL;
+	int idx = 0;
+
+	snprintf(path, PATH_MAX, "%s" EVENT_SOURCE_DEVICE_PATH, sysfs);
+	dir = opendir(path);
+	if (!dir) {
+		pr_err("can't read directory '%s'\n", EVENT_SOURCE_DEVICE_PATH);
+		*err = -EINVAL;
+		goto out;
+	}
+
+	while ((dent = readdir(dir))) {
+		if (strstr(dent->d_name, HISI_PTT_PMU_NAME))
+			(*nr_ptts)++;
+	}
+
+	if (!(*nr_ptts))
+		goto out;
+
+	hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * (*nr_ptts));
+	if (!hisi_ptt_pmus) {
+		pr_err("hisi_ptt alloc failed\n");
+		*err = -ENOMEM;
+		goto out;
+	}
+
+	rewinddir(dir);
+	while ((dent = readdir(dir))) {
+		if (strstr(dent->d_name, HISI_PTT_PMU_NAME) && idx < (*nr_ptts)) {
+			hisi_ptt_pmus[idx] = perf_pmu__find(dent->d_name);
+			if (hisi_ptt_pmus[idx]) {
+				pr_debug2("%s %d: hisi_ptt_pmu %d type %d name %s\n",
+					__func__, __LINE__, idx,
+					hisi_ptt_pmus[idx]->type,
+					hisi_ptt_pmus[idx]->name);
+					idx++;
+			}
+
+		}
+	}
+
+out:
+	closedir(dir);
+	return hisi_ptt_pmus;
+}
+
 struct auxtrace_record
 *auxtrace_record__init(struct evlist *evlist, int *err)
 {
@@ -57,8 +112,12 @@ struct auxtrace_record
 	struct evsel *evsel;
 	bool found_etm = false;
 	struct perf_pmu *found_spe = NULL;
+	struct perf_pmu *found_ptt = NULL;
 	struct perf_pmu **arm_spe_pmus = NULL;
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+
 	int nr_spes = 0;
+	int nr_ptts = 0;
 	int i = 0;
 
 	if (!evlist)
@@ -66,13 +125,14 @@ struct auxtrace_record
 
 	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
 	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
+	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
 
 	evlist__for_each_entry(evlist, evsel) {
 		if (cs_etm_pmu &&
 		    evsel->core.attr.type == cs_etm_pmu->type)
 			found_etm = true;
 
-		if (!nr_spes || found_spe)
+		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
 			continue;
 
 		for (i = 0; i < nr_spes; i++) {
@@ -81,11 +141,18 @@ struct auxtrace_record
 				break;
 			}
 		}
+
+		for (i = 0; i < nr_ptts; i++) {
+			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
+				found_ptt = hisi_ptt_pmus[i];
+				break;
+			}
+		}
 	}
 	free(arm_spe_pmus);
 
-	if (found_etm && found_spe) {
-		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
+	if (found_etm && found_spe && found_ptt) {
+		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
 		*err = -EOPNOTSUPP;
 		return NULL;
 	}
@@ -96,6 +163,9 @@ struct auxtrace_record
 #if defined(__aarch64__)
 	if (found_spe)
 		return arm_spe_recording_init(err, found_spe);
+
+	if (found_ptt)
+		return hisi_ptt_recording_init(err, found_ptt);
 #endif
 
 	/*
diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
index b8b23b9dc598..89a3cedb4557 100644
--- a/tools/perf/arch/arm/util/pmu.c
+++ b/tools/perf/arch/arm/util/pmu.c
@@ -10,6 +10,7 @@
 #include <linux/string.h>
 
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "../../../util/pmu.h"
 
 struct perf_event_attr
@@ -22,6 +23,8 @@ struct perf_event_attr
 #if defined(__aarch64__)
 	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
 		return arm_spe_pmu_default_config(pmu);
+	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
+		pmu->selectable = true;
 #endif
 	}
 
diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
index 9fcb4e68add9..8b7fd1dc9f37 100644
--- a/tools/perf/arch/arm64/util/Build
+++ b/tools/perf/arch/arm64/util/Build
@@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
 perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
 			      ../../arm/util/auxtrace.o \
 			      ../../arm/util/cs-etm.o \
-			      arm-spe.o mem-events.o
+			      arm-spe.o mem-events.o hisi_ptt.o
diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
new file mode 100644
index 000000000000..de3a3523ffd5
--- /dev/null
+++ b/tools/perf/arch/arm64/util/hisi_ptt.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/log2.h>
+#include <linux/zalloc.h>
+#include <time.h>
+
+#include <internal/lib.h> // page_size
+#include "../../../util/auxtrace.h"
+#include "../../../util/cpumap.h"
+#include "../../../util/debug.h"
+#include "../../../util/event.h"
+#include "../../../util/evlist.h"
+#include "../../../util/evsel.h"
+#include "../../../util/hisi_ptt.h"
+#include "../../../util/pmu.h"
+#include "../../../util/record.h"
+#include "../../../util/session.h"
+#include "../../../util/tsc.h"
+
+#define DEFAULT_PAGE_SIZE 1024
+#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
+#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
+
+struct hisi_ptt_recording {
+	struct auxtrace_record	itr;
+	struct perf_pmu *hisi_ptt_pmu;
+	struct evlist *evlist;
+};
+
+static size_t
+hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
+			struct evlist *evlist __maybe_unused)
+{
+	return HISI_PTT_AUXTRACE_PRIV_SIZE;
+}
+
+static int hisi_ptt_info_fill(struct auxtrace_record *itr,
+			      struct perf_session *session,
+			      struct perf_record_auxtrace_info *auxtrace_info,
+			      size_t priv_size)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+
+	if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
+		return -EINVAL;
+
+	if (!session->evlist->core.nr_mmaps)
+		return -EINVAL;
+
+	auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
+	auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
+
+	return 0;
+}
+
+static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
+{
+	bool privileged = perf_event_paranoid_check(-1);
+
+	if (!opts->full_auxtrace)
+		return 0;
+
+	if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
+		if (privileged) {
+			opts->auxtrace_mmap_pages = MiB(16) / page_size;
+		} else {
+			opts->auxtrace_mmap_pages = KiB(128) / page_size;
+			if (opts->mmap_pages == UINT_MAX)
+				opts->mmap_pages = KiB(256) / page_size;
+		}
+	}
+
+	/* Validate auxtrace_mmap_pages */
+	if (opts->auxtrace_mmap_pages) {
+		size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
+		size_t min_sz = KiB(8);
+
+		if (sz < min_sz || !is_power_of_2(sz)) {
+			pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
+			       min_sz / DEFAULT_PAGE_SIZE);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_recording_options(struct auxtrace_record *itr,
+				      struct evlist *evlist,
+				      struct record_opts *opts)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+	struct perf_cpu_map *cpus = evlist->core.cpus;
+	struct evsel *evsel, *hisi_ptt_evsel = NULL;
+	struct evsel *tracking_evsel;
+	int err;
+
+	pttr->evlist = evlist;
+	evlist__for_each_entry(evlist, evsel) {
+		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
+			if (hisi_ptt_evsel) {
+				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
+				return -EINVAL;
+			}
+			evsel->core.attr.freq = 0;
+			evsel->core.attr.sample_period = 1;
+			hisi_ptt_evsel = evsel;
+			opts->full_auxtrace = true;
+		}
+	}
+
+	err = hisi_ptt_set_auxtrace_mmap_page(opts);
+	if (err)
+		return err;
+	/*
+	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
+	 * must come first.
+	 */
+	evlist__to_front(evlist, hisi_ptt_evsel);
+
+	if (!perf_cpu_map__empty(cpus)) {
+		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
+		evsel__set_sample_bit(hisi_ptt_evsel, CPU);
+	}
+
+	/* Add dummy event to keep tracking */
+	err = parse_events(evlist, "dummy:u", NULL);
+	if (err)
+		return err;
+
+	tracking_evsel = evlist__last(evlist);
+	evlist__set_tracking_event(evlist, tracking_evsel);
+
+	tracking_evsel->core.attr.freq = 0;
+	tracking_evsel->core.attr.sample_period = 1;
+
+	if (!perf_cpu_map__empty(cpus))
+		evsel__set_sample_bit(tracking_evsel, TIME);
+
+	return 0;
+}
+
+static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
+{
+	return rdtsc();
+}
+
+static void hisi_ptt_recording_free(struct auxtrace_record *itr)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+
+	free(pttr);
+}
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu)
+{
+	struct hisi_ptt_recording *pttr;
+
+	if (!hisi_ptt_pmu) {
+		*err = -ENODEV;
+		return NULL;
+	}
+
+	pttr = zalloc(sizeof(struct hisi_ptt_recording));
+	if (!pttr) {
+		*err = -ENOMEM;
+		return NULL;
+	}
+
+	pttr->hisi_ptt_pmu = hisi_ptt_pmu;
+	pttr->itr.pmu = hisi_ptt_pmu;
+	pttr->itr.recording_options = hisi_ptt_recording_options;
+	pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
+	pttr->itr.info_fill = hisi_ptt_info_fill;
+	pttr->itr.free = hisi_ptt_recording_free;
+	pttr->itr.reference = hisi_ptt_reference;
+	pttr->itr.read_finish = auxtrace_record__read_finish;
+	pttr->itr.alignment = 0;
+
+	*err = 0;
+	return &pttr->itr;
+}
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 9a7209a99e16..95d2d76c52da 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -116,6 +116,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
 perf-$(CONFIG_AUXTRACE) += intel-bts.o
 perf-$(CONFIG_AUXTRACE) += arm-spe.o
 perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
+perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
 perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
 
 ifdef CONFIG_LIBOPENCSD
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index df1c5bbbaa0d..f4841ebaf314 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -51,6 +51,7 @@
 #include "intel-pt.h"
 #include "intel-bts.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "s390-cpumsf.h"
 #include "util/mmap.h"
 
@@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
 	case PERF_AUXTRACE_S390_CPUMSF:
 		err = s390_cpumsf_process_auxtrace_info(event, session);
 		break;
+	case PERF_AUXTRACE_HISI_PTT:
+		err = hisi_ptt_process_auxtrace_info(event, session);
+		break;
 	case PERF_AUXTRACE_UNKNOWN:
 	default:
 		return -EINVAL;
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index dc38b6f57232..3858c5752ead 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -48,6 +48,7 @@ enum auxtrace_type {
 	PERF_AUXTRACE_CS_ETM,
 	PERF_AUXTRACE_ARM_SPE,
 	PERF_AUXTRACE_S390_CPUMSF,
+	PERF_AUXTRACE_HISI_PTT,
 };
 
 enum itrace_period_type {
diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
new file mode 100644
index 000000000000..db3db8b75033
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/Build
@@ -0,0 +1 @@
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
new file mode 100644
index 000000000000..3fb7a6949209
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <endian.h>
+#include <byteswap.h>
+#include <linux/bitops.h>
+#include <stdarg.h>
+
+#include "../color.h"
+#include "hisi-ptt-pkt-decoder.h"
+
+/*
+ * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
+ * used to distinguish the data format.
+ * 8DW format is like:
+ *   bits [                 31:11                 ][       10:0       ]
+ *        |---------------------------------------|-------------------|
+ *    DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+ *    DW1 [                       Prefix                              ]
+ *    DW2 [                     Header DW0                            ]
+ *    DW3 [                     Header DW1                            ]
+ *    DW4 [                     Header DW2                            ]
+ *    DW5 [                     Header DW3                            ]
+ *    DW6 [                   Reserved (0x0)                          ]
+ *    DW7 [                        Time                               ]
+ *
+ * 4DW format is like:
+ *   bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+ *        |-----|---------|---|---|---|---|-------------|-------------|
+ *    DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+ *    DW1 [                     Header DW1                            ]
+ *    DW2 [                     Header DW2                            ]
+ *    DW3 [                     Header DW3                            ]
+ */
+
+enum hisi_ptt_8dw_pkt_field_type {
+	HISI_PTT_8DW_PREFIX,
+	HISI_PTT_8DW_HEAD0,
+	HISI_PTT_8DW_HEAD1,
+	HISI_PTT_8DW_HEAD2,
+	HISI_PTT_8DW_HEAD3,
+	HISI_PTT_8DW_TIME,
+	HISI_PTT_8DW_TYPE_MAX
+};
+
+enum hisi_ptt_4dw_pkt_field_type {
+	HISI_PTT_4DW_HEAD1,
+	HISI_PTT_4DW_HEAD2,
+	HISI_PTT_4DW_HEAD3,
+	HISI_PTT_4DW_TYPE_MAX
+};
+
+static const char * const hisi_ptt_8dw_pkt_field_name[] = {
+	[HISI_PTT_8DW_PREFIX]	= "Prefix",
+	[HISI_PTT_8DW_HEAD0]	= "Header DW0",
+	[HISI_PTT_8DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_8DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_8DW_HEAD3]	= "Header DW3",
+	[HISI_PTT_8DW_TIME]	= "Time",
+};
+
+static const char * const hisi_ptt_4dw_pkt_field_name[] = {
+	[HISI_PTT_4DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_4DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_4DW_HEAD3]	= "Header DW3",
+};
+
+/* offset of each member is determined by format of 8dw packet. */
+static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
+	[HISI_PTT_8DW_PREFIX]	= 4,
+	[HISI_PTT_8DW_HEAD0]	= 4,
+	[HISI_PTT_8DW_HEAD1]	= 4,
+	[HISI_PTT_8DW_HEAD2]	= 4,
+	[HISI_PTT_8DW_HEAD3]	= 4,
+	[HISI_PTT_8DW_TIME]	= 8,
+};
+
+union hisi_ptt_4dw {
+	struct {
+		uint32_t format : 2;
+		uint32_t type : 5;
+		uint32_t t9 : 1;
+		uint32_t t8 : 1;
+		uint32_t th : 1;
+		uint32_t so : 1;
+		uint32_t len : 10;
+		uint32_t time : 11;
+	};
+	uint32_t value;
+};
+
+static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	int i;
+
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+	color_fprintf(stdout, color, "  %s\n", desc);
+	*pos += field_len;
+}
+
+static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
+		buf += hisi_ptt_8dw_pkt_field_offset[i];
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_8DW_PKT_SIZE;
+}
+
+static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	union hisi_ptt_4dw dw0;
+	int i;
+
+	dw0.value = *(uint32_t *)buf;
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+
+	color_fprintf(stdout, color,
+		      "  %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
+		      "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
+		      "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
+		      dw0.len, "Time", dw0.time);
+
+	*pos += field_len;
+}
+
+static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	hisi_ptt_4dw_print_dw0(buf, &pos);
+
+	for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
+		buf += sizeof(uint32_t);
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_4DW_PKT_SIZE;
+}
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
+{
+	if (type == HISI_PTT_8DW_PKT)
+		return hisi_ptt_8dw_kpt_desc(buf, pos);
+
+	return hisi_ptt_4dw_kpt_desc(buf, pos);
+}
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
new file mode 100644
index 000000000000..20c51f230cac
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
+#define INCLUDE__HISI_PTT_PKT_DECODER_H__
+
+#include <stddef.h>
+#include <stdint.h>
+
+#define HISI_PTT_PKT_DESC_MAX		256
+#define HISI_PTT_NEED_MORE_BYTES	-1
+#define HISI_PTT_8DW_CHECK_MASK		GENMASK(31, 11)
+#define HISI_PTT_IS_8DW_PKT		GENMASK(31, 11)
+#define HISI_PTT_8DW_PKT_SIZE		32
+#define HISI_PTT_4DW_PKT_SIZE		16
+#define HISI_PTT_MAX_SPACE_LEN		10
+
+enum hisi_ptt_pkt_type {
+	HISI_PTT_4DW_PKT,
+	HISI_PTT_8DW_PKT,
+};
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
+
+#endif
diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
new file mode 100644
index 000000000000..e2934214763c
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.c
@@ -0,0 +1,218 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <byteswap.h>
+#include <endian.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/types.h>
+#include <linux/zalloc.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "auxtrace.h"
+#include "color.h"
+#include "debug.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "hisi_ptt.h"
+#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
+#include "machine.h"
+#include "session.h"
+#include "symbol.h"
+#include "tool.h"
+#include "util/synthetic-events.h"
+
+struct hisi_ptt {
+	struct auxtrace auxtrace;
+	struct auxtrace_queues queues;
+	u32 auxtrace_type;
+	struct perf_session *session;
+	struct machine *machine;
+	u32 pmu_type;
+};
+
+struct hisi_ptt_queue {
+	struct hisi_ptt *ptt;
+	struct auxtrace_buffer *buffer;
+};
+
+static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
+{
+	uint32_t head = *(uint32_t *)buf;
+
+	if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
+		return HISI_PTT_8DW_PKT;
+
+	return HISI_PTT_4DW_PKT;
+}
+
+static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
+			  unsigned char *buf, size_t len)
+{
+	const char *color = PERF_COLOR_BLUE;
+	enum hisi_ptt_pkt_type type;
+	size_t pos = 0;
+	int pkt_len;
+
+	color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
+		      len);
+
+	type = hisi_ptt_check_packet_type(buf);
+	while (len) {
+		pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
+		if (!pkt_len)
+			color_fprintf(stdout, color, " Bad packet!\n");
+
+		pos += pkt_len;
+		buf += pkt_len;
+		len -= pkt_len;
+	}
+}
+
+static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
+				size_t len)
+{
+	printf(".\n");
+
+	hisi_ptt_dump(ptt, buf, len);
+}
+
+static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
+				  union perf_event *event __maybe_unused,
+				  struct perf_sample *sample __maybe_unused,
+				  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
+					   union perf_event *event,
+					   struct perf_tool *tool __maybe_unused)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_buffer *buffer;
+	off_t data_offset;
+	int fd = perf_data__fd(session->data);
+	int err;
+
+	if (perf_data__is_pipe(session->data)) {
+		data_offset = 0;
+	} else {
+		data_offset = lseek(fd, 0, SEEK_CUR);
+		if (data_offset == -1)
+			return -errno;
+	}
+
+	err = auxtrace_queues__add_event(&ptt->queues, session, event,
+					 data_offset, &buffer);
+	if (err)
+		return err;
+
+	if (dump_trace) {
+		if (auxtrace_buffer__get_data(buffer, fd)) {
+			hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
+			auxtrace_buffer__put_data(buffer);
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
+			  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static void hisi_ptt_free_events(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_queues *queues = &ptt->queues;
+	unsigned int i;
+
+	for (i = 0; i < queues->nr_queues; i++) {
+		free(queues->queue_array[i].priv);
+		queues->queue_array[i].priv = NULL;
+	}
+	auxtrace_queues__free(queues);
+}
+
+static void hisi_ptt_free(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+
+	hisi_ptt_free_events(session);
+	session->auxtrace = NULL;
+	free(ptt);
+}
+
+static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
+				       struct evsel *evsel)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
+
+	return evsel->core.attr.type == ptt->pmu_type;
+}
+
+static const char * const hisi_ptt_info_fmts[] = {
+	[HISI_PTT_PMU_TYPE]		= "  PMU Type           %" PRId64 "\n",
+};
+
+static void hisi_ptt_print_info(__u64 *arr)
+{
+	if (!dump_trace)
+		return;
+
+	fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
+}
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session)
+{
+	struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
+	struct hisi_ptt *ptt;
+	int err;
+
+	if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
+				sizeof(struct perf_record_auxtrace_info))
+		return -EINVAL;
+
+	ptt = zalloc(sizeof(struct hisi_ptt));
+	if (!ptt)
+		return -ENOMEM;
+
+	err = auxtrace_queues__init(&ptt->queues);
+	if (err)
+		goto err_free;
+
+	ptt->session = session;
+	ptt->machine = &session->machines.host; /* No kvm support */
+	ptt->auxtrace_type = auxtrace_info->type;
+	ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
+
+	ptt->auxtrace.process_event = hisi_ptt_process_event;
+	ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
+	ptt->auxtrace.flush_events = hisi_ptt_flush;
+	ptt->auxtrace.free_events = hisi_ptt_free_events;
+	ptt->auxtrace.free = hisi_ptt_free;
+	ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
+	session->auxtrace = &ptt->auxtrace;
+
+	hisi_ptt_print_info(&auxtrace_info->priv[0]);
+
+	return 0;
+
+err_free:
+	free(ptt);
+	return err;
+}
diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
new file mode 100644
index 000000000000..c0b6cbde1221
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#ifndef INCLUDE__PERF_HISI_PTT_H__
+#define INCLUDE__PERF_HISI_PTT_H__
+
+#define HISI_PTT_PMU_NAME "hisi_ptt"
+enum {
+	HISI_PTT_PMU_TYPE,
+	HISI_PTT_PER_CPU_MMAPS,
+	HISI_PTT_AUXTRACE_PRIV_MAX,
+};
+
+#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
+union perf_event;
+struct perf_session;
+struct perf_pmu;
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu);
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session);
+
+#endif
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [PATCH v7 6/7] docs: Add HiSilicon PTT device driver documentation
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
                   ` (4 preceding siblings ...)
  2022-04-07 12:58 ` [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
@ 2022-04-07 12:58 ` Yicong Yang
  2022-04-07 12:58 ` [PATCH v7 7/7] MAINTAINERS: Add maintainer for HiSilicon PTT driver Yicong Yang
  2022-04-14 13:20 ` [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
  7 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

Document the introduction and usage of HiSilicon PTT device driver.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 Documentation/trace/hisi-ptt.rst | 303 +++++++++++++++++++++++++++++++
 1 file changed, 303 insertions(+)
 create mode 100644 Documentation/trace/hisi-ptt.rst

diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
new file mode 100644
index 000000000000..13677705ee1f
--- /dev/null
+++ b/Documentation/trace/hisi-ptt.rst
@@ -0,0 +1,303 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+HiSilicon PCIe Tune and Trace device
+======================================
+
+Introduction
+============
+
+HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
+integrated Endpoint (RCiEP) device, providing the capability
+to dynamically monitor and tune the PCIe link's events (tune),
+and trace the TLP headers (trace). The two functions are independent,
+but is recommended to use them together to analyze and enhance the
+PCIe link's performance.
+
+On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
+PCIe cores. Each PCIe core includes several Root Ports and a PTT
+RCiEP, like below. The PTT device is capable of tuning and
+tracing the links of the PCIe core.
+::
+          +--------------Core 0-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+    Root Complex  |------Core 1-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[ Switch ]---[Endpoint]
+          |       |       [Root Port]---[Endpoint] `-[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          +---------------------------+
+
+The PTT device driver registers one PMU device for each PTT device.
+The name of each PTT device is composed of 'hisi_ptt' prefix with
+the id of the SICL and the Core where it locates. The Kunpeng 930
+SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
+IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
+Complex for each SICL.
+::
+    /sys/devices/hisi_ptt<sicl_id>_<core_id>
+
+Tune
+====
+
+PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
+Currently we support events in 4 classes. The scope of the events
+covers the PCIe core to which the PTT device belongs.
+
+Each event is presented as a file under $(PTT PMU dir)/tune, and
+a simple open/read/write/close cycle will be used to tune the event.
+::
+    $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
+    $ ls
+    qos_tx_cpl    qos_tx_np    qos_tx_p
+    tx_path_rx_req_alloc_buf_level
+    tx_path_tx_req_alloc_buf_level
+    $ cat qos_tx_dp
+    1
+    $ echo 2 > qos_tx_dp
+    $ cat qos_tx_dp
+    2
+
+Current value (numerical value) of the event can be simply read
+from the file, and the desired value written to the file to tune.
+
+1. Tx path QoS control
+------------------------
+
+The following files are provided to tune the QoS of the tx path of
+the PCIe core.
+
+- qos_tx_cpl: weight of Tx completion TLPs
+- qos_tx_np: weight of Tx non-posted TLPs
+- qos_tx_p: weight of Tx posted TLPs
+
+The weight influences the proportion of certain packets on the PCIe link.
+For example, for the storage scenario, increase the proportion
+of the completion packets on the link to enhance the performance as
+more completions are consumed.
+
+The available tune data of these events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+2. Tx path buffer control
+-------------------------
+
+Following files are provided to tune the buffer of tx path of the PCIe core.
+
+- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
+- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
+
+These events influence the watermark of the buffer allocated for each
+type. Rx means the inbound while Tx means outbound. The packets will
+be stored in the buffer first and then transmitted either when the
+watermark reached or when timed out. For a busy direction, you should
+increase the related buffer watermark to avoid frequently posting and
+thus enhance the performance. In most cases just keep the default value.
+
+The available tune data of above events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+Trace
+=====
+
+PTT trace is designed for dumping the TLP headers to the memory, which
+can be used to analyze the transactions and usage condition of the PCIe
+Link. You can choose to filter the traced headers by either requester ID,
+or those downstream of a set of Root Ports on the same core of the PTT
+device. It's also supported to trace the headers of certain type and of
+certain direction.
+
+You can use the perf command `perf record` to set the parameters, start
+trace and get the data. It's also supported to decode the trace
+data with `perf report`. The control parameters for trace is inputted
+as event code for each events, which will be further illustrated later.
+An example usage is like
+::
+    $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
+      format=1/ -- sleep 5
+
+This will trace the TLP headers downstream root port 0000:00:10.1 (event
+code for event 'filter' is 0x80001) with type of posted TLP requests,
+direction of inbound and traced data format of 8DW.
+
+1. filter
+---------
+
+The TLP headers to trace can be filtered by the Root Ports or the requester
+ID of the endpoints, which are located on the same core of the PTT device.
+You can set the filter by specifying the `filter` parameter which is required
+to start the trace. The parameter value is 20 bit. The supported filters and
+related values are outputted through `available_root_port_filters` and
+`available_requester_filters` sysfs attributes for Root Ports and Requesters
+respectively.
+::
+    $ cat available_root_port_filters
+    0000:00:10.0	0x80001
+    0000:00:11.0	0x80004
+    $ cat available_requester_filters
+    0000:01:00.0	0x00100
+    0000:01:00.1	0x00101
+
+Note that multiple Root Ports can be specified at one time, but only
+one Endpoint function can be specified in one trace. Specifying both
+Root Port and function at the same time is not supported.
+
+If no filter is available, reading the related filter sysfs attribute
+will get an empty string.
+::
+    $ cat available_root_port_filters
+
+    $ cat available_requester_filters
+
+The available filters can be dynamically updated, which means you can always
+get correct filter information when hotplug events happen, or when you manually
+remove/rescan the devices.
+
+2. type
+-------
+
+You can trace the TLP headers of certain types by specifying the `type`
+parameter, which is required to start the trace. The parameter value is
+8 bit. Current supported types and related values are shown below:
+
+8'b00000001: posted requests (P)
+8'b00000010: non-posted requests (NP)
+8'b00000100: completions (CPL)
+
+You can specify multiple types when tracing inbound TLP headers, but can only
+specify one when tracing outbound TLP headers.
+
+3. direction
+------------
+
+You can trace the TLP headers from certain direction, which is relative
+to the Root Port or the PCIe core, by specifying the `direction` parameter.
+This is optional and the default parameter is inbound. The parameter value
+is 4 bit. When the desired format is 4DW, directions and related values
+supported are shown below:
+
+4'b0000: inbound TLPs (P, NP, CPL)
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
+4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
+
+When the desired format is 8DW, directions and related values supported are
+shown below:
+
+4'b0000: reserved
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: inbound TLPs (P, NP, CPL B)
+4'b0011: inbound TLPs (CPL A)
+
+Inbound completions are classified into two types:
+
+completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
+completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
+
+4. format
+--------------
+
+You can change the format of the traced TLP headers by specifying the
+`format` parameter. The default format is 4DW. The parameter value is 4 bit.
+Current supported formats and related values are shown below:
+
+4'b0000: 4DW length per TLP header
+4'b0001: 8DW length per TLP header
+
+The traced TLP header format is different from the PCIe standard.
+
+When using the 8DW data format, the entire TLP header is logged
+(Header DW0-3 shown below). For example, the TLP header for Memory
+Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
+the header for Configuration Requests is shown in Figure 2.20, etc.
+
+In addition, 8DW trace buffer entries contain a timestamp and
+possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
+Otherwise this field will be all 0.
+
+The bit[31:11] of DW0 is always 0x1fffff, which can be
+used to distinguish the data format. 8DW format is like
+::
+    bits [                 31:11                 ][       10:0       ]
+         |---------------------------------------|-------------------|
+     DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+     DW1 [                       Prefix                              ]
+     DW2 [                     Header DW0                            ]
+     DW3 [                     Header DW1                            ]
+     DW4 [                     Header DW2                            ]
+     DW5 [                     Header DW3                            ]
+     DW6 [                   Reserved (0x0)                          ]
+     DW7 [                        Time                               ]
+
+When using the 4DW data format, DW0 of the trace buffer entry
+contains selected fields of DW0 of the TLP, together with a
+timestamp.  DW1-DW3 of the trace buffer entry contain DW1-DW3
+directly from the TLP header.
+
+4DW format is like
+::
+    bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+         |-----|---------|---|---|---|---|-------------|-------------|
+     DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+     DW1 [                     Header DW1                            ]
+     DW2 [                     Header DW2                            ]
+     DW3 [                     Header DW3                            ]
+
+5. memory management
+--------------------
+
+The traced TLP headers will be written to the memory allocated
+by the driver. The hardware accepts 4 DMA address with same size,
+and writes the buffer sequentially like below. If DMA addr 3 is
+finished and the trace is still on, it will return to addr 0.
+::
+    +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
+    +---------------------------------------------------------+
+
+Driver will allocate each DMA buffer of 4MiB. The finished buffer
+will be copied to the perf AUX buffer allocated by the perf core.
+Once the AUX buffer is full while the trace is still on, driver
+will commit the AUX buffer first and then apply for a new one with
+the same size. The size of AUX buffer is default to 16MiB. User can
+adjust the size by specifying the `-m` parameter of the perf command.
+
+Note that there is a gap between committing the old AUX buffer and
+applying a new one, which means the trace is stopped during the
+moment and TLPs transferred in the moment cannot be traced. To avoid
+this situation, you should begin the trace with large AUX buffer
+enough to avoid this gap.
+
+6. decoding
+-----------
+
+You can decode the traced data with `perf report -D` command (currently
+only support to dump the raw trace data). The traced data will be decoded
+according to the format described previously (take 8DW as an example):
+::
+    [...perf headers and other information]
+    . ... HISI PTT data: size 4194304 bytes
+    .  00000000: 00 00 00 00                                 Prefix
+    .  00000004: 01 00 00 60                                 Header DW0
+    .  00000008: 0f 1e 00 01                                 Header DW1
+    .  0000000c: 04 00 00 00                                 Header DW2
+    .  00000010: 40 00 81 02                                 Header DW3
+    .  00000014: 33 c0 04 00                                 Time
+    .  00000020: 00 00 00 00                                 Prefix
+    .  00000024: 01 00 00 60                                 Header DW0
+    .  00000028: 0f 1e 00 01                                 Header DW1
+    .  0000002c: 04 00 00 00                                 Header DW2
+    .  00000030: 40 00 81 02                                 Header DW3
+    .  00000034: 02 00 00 00                                 Time
+    .  00000040: 00 00 00 00                                 Prefix
+    .  00000044: 01 00 00 60                                 Header DW0
+    .  00000048: 0f 1e 00 01                                 Header DW1
+    .  0000004c: 04 00 00 00                                 Header DW2
+    .  00000050: 40 00 81 02                                 Header DW3
+    [...]
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [PATCH v7 7/7] MAINTAINERS: Add maintainer for HiSilicon PTT driver
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
                   ` (5 preceding siblings ...)
  2022-04-07 12:58 ` [PATCH v7 6/7] docs: Add HiSilicon PTT device driver documentation Yicong Yang
@ 2022-04-07 12:58 ` Yicong Yang
  2022-04-14 13:20 ` [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
  7 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-07 12:58 UTC (permalink / raw)
  To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong

Add maintainer for driver and documentation of HiSilicon PTT device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index fd768d43e048..d30a1698251c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8858,6 +8858,13 @@ F:	Documentation/admin-guide/perf/hisi-pcie-pmu.rst
 F:	Documentation/admin-guide/perf/hisi-pmu.rst
 F:	drivers/perf/hisilicon
 
+HISILICON PTT DRIVER
+M:	Yicong Yang <yangyicong@hisilicon.com>
+L:	linux-kernel@vger.kernel.org
+S:	Maintained
+F:	Documentation/trace/hisi-ptt.rst
+F:	drivers/hwtracing/ptt/
+
 HISILICON QM AND ZIP Controller DRIVER
 M:	Zhou Wang <wangzhou1@hisilicon.com>
 L:	linux-crypto@vger.kernel.org
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-07 12:58 ` [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device Yicong Yang
@ 2022-04-11 17:02   ` John Garry
  2022-04-12  7:36     ` Yicong Yang
  2022-04-29 16:00   ` Leo Yan
  2022-05-10  9:46   ` James Clark
  2 siblings, 1 reply; 30+ messages in thread
From: John Garry @ 2022-04-11 17:02 UTC (permalink / raw)
  To: Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

On 07/04/2022 13:58, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
> tune the PCIe traffic, and trace the TLP headers.
> 
> Add the driver for the device to enable the trace function. Register PMU
> device of PTT trace, then users can use trace through perf command. The
> driver makes use of perf AUX trace and support following events to

"The driver makes use of perf AUX trace function and supports the 
following events to .."

> configure the trace:
> 
> - filter: select Root port or Endpoint to trace
> - type: select the type of traced TLP headers
> - direction: select the direction of traced TLP headers
> - format: select the data format of the traced TLP headers
> 
> This patch adds the driver part of PTT trace. The perf command support of
> PTT trace is added in the following patch.

I would not mention "following patch" and the like. Just have "initially 
add a basic driver"

> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>   drivers/Makefile                 |   1 +
>   drivers/hwtracing/Kconfig        |   2 +
>   drivers/hwtracing/ptt/Kconfig    |  12 +
>   drivers/hwtracing/ptt/Makefile   |   2 +
>   drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
>   drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
>   6 files changed, 1057 insertions(+)
>   create mode 100644 drivers/hwtracing/ptt/Kconfig
>   create mode 100644 drivers/hwtracing/ptt/Makefile
>   create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>   create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
> 
> diff --git a/drivers/Makefile b/drivers/Makefile
> index 020780b6b4d2..662d50599467 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>   obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>   obj-y				+= hwtracing/intel_th/
>   obj-$(CONFIG_STM)		+= hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT)		+= hwtracing/ptt/
>   obj-$(CONFIG_ANDROID)		+= android/
>   obj-$(CONFIG_NVMEM)		+= nvmem/
>   obj-$(CONFIG_FPGA)		+= fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..911ee977103c 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>   
>   source "drivers/hwtracing/intel_th/Kconfig"
>   
> +source "drivers/hwtracing/ptt/Kconfig"
> +
>   endmenu
> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
> new file mode 100644
> index 000000000000..8902a6f27563
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Kconfig
> @@ -0,0 +1,12 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> +	tristate "HiSilicon PCIe Tune and Trace Device"
> +	depends on ARM64 || (COMPILE_TEST && 64BIT)
> +	depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
> +	help
> +	  HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
> +	  device, and it provides support for PCIe traffic tuning and
> +	  tracing TLP headers to the memory.
> +
> +	  This driver can also be built as a module. If so, the module
> +	  will be called hisi_ptt.
> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
> new file mode 100644
> index 000000000000..242b41870380
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
> @@ -0,0 +1,874 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/module.h>
> +#include <linux/sysfs.h>
> +#include <linux/vmalloc.h>
> +
> +#include "hisi_ptt.h"
> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
> +					  val, val & HISI_PTT_TRACE_IDLE,
> +					  HISI_PTT_WAIT_POLL_INTERVAL_US,
> +					  HISI_PTT_WAIT_TRACE_TIMEOUT_US);
> +}
> +
> +static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
> +					  val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
> +					  HISI_PTT_RESET_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	int i;
> +
> +	if (!ctrl->trace_buf)
> +		return;
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
> +		if (ctrl->trace_buf[i].addr)
> +			dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
> +					   ctrl->trace_buf[i].addr,
> +					   ctrl->trace_buf[i].dma);
> +	}
> +
> +	devm_kfree(dev, ctrl->trace_buf);
> +	ctrl->trace_buf = NULL;
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	int i;
> +
> +	hisi_ptt->trace_ctrl.buf_index = 0;
> +
> +	/* If the trace buffer has already been allocated, zero it. */

I am not sure why this is not called from the probe

> +	if (ctrl->trace_buf) {
> +		for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
> +			memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
> +		return 0;
> +	}
> +
> +	ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
> +				       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);

sizeof(*ctrl->trace_buf) may be better

> +	if (!ctrl->trace_buf)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
> +		ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
> +							     &ctrl->trace_buf[i].dma,
> +							     GFP_KERNEL);
> +		if (!ctrl->trace_buf[i].addr) {
> +			hisi_ptt_free_trace_buf(hisi_ptt);
> +			return -ENOMEM;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	hisi_ptt->trace_ctrl.started = false;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	u32 val;
> +	int i;
> +
> +	/* Check device idle before start trace */
> +	if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
> +		return -EBUSY;
> +	}
> +
> +	ctrl->started = true;
> +
> +	/* Reset the DMA before start tracing */
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val |= HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	hisi_ptt_wait_dma_reset_done(hisi_ptt);
> +
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/* Clear the interrupt status */
> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> +	/* Configure the trace DMA buffer */

I am not sure why this sort of thing is done outside probing

> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
> +		writel(lower_32_bits(ctrl->trace_buf[i].dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
> +		writel(upper_32_bits(ctrl->trace_buf[i].dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
> +	}
> +	writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> +	/* Set the trace control register */
> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> +	if (!hisi_ptt->trace_ctrl.is_port)
> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> +	/* Start the Trace */
> +	val |= HISI_PTT_TRACE_CTRL_EN;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct perf_output_handle *handle = &ctrl->handle;
> +	struct perf_event *event = handle->event;
> +	struct hisi_ptt_pmu_buf *buf;
> +	void *addr;
> +
> +	buf = perf_get_aux(handle);
> +	if (!buf || !handle->size)
> +		return -EINVAL;
> +
> +	addr = ctrl->trace_buf[ctrl->buf_index].addr;
> +
> +	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
> +	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);
> +	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
> +
> +	if (stop) {
> +		perf_aux_output_end(handle, buf->pos);
> +	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> +		perf_aux_output_end(handle, buf->pos);
> +
> +		buf = perf_aux_output_begin(handle, event);
> +		if (!buf)
> +			return -EINVAL;
> +
> +		buf->pos = handle->head % buf->length;
> +		if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
> +			perf_aux_output_end(handle, 0);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status, buf_idx;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	buf_idx = ffs(status) - 1;
> +
> +	/* Clear the interrupt status of buffer @buf_idx */
> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> +	/*
> +	 * Update the AUX buffer and cache the current buffer index,
> +	 * as we need to know this and save the data when the trace
> +	 * is ended out of the interrupt handler. End the trace
> +	 * if the updating fails.
> +	 */
> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> +		hisi_ptt_trace_end(hisi_ptt);
> +	else
> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUF_CNT;
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> +		return IRQ_NONE;
> +
> +	return IRQ_WAKE_THREAD;
> +}
> +
> +static void hisi_ptt_irq_free_vectors(void *pdev)
> +{
> +	pci_free_irq_vectors(pdev);
> +}
> +
> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	int ret;
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> +	if (ret < 0) {
> +		pci_err(pdev, "failed to allocate irq vector, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = devm_request_threaded_irq(&pdev->dev,
> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> +					DRV_NAME, hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to request irq %d, ret = %d\n",
> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> +{
> +	struct hisi_ptt_filter_desc *filter;
> +	struct hisi_ptt *hisi_ptt = data;
> +	struct list_head *target_list;
> +
> +	target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
> +		      &hisi_ptt->port_filters : &hisi_ptt->req_filters;
> +
> +	filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> +	if (!filter) {
> +		pci_err(hisi_ptt->pdev, "failed to add filter %s\n", pci_name(pdev));
> +		return -ENOMEM;
> +	}
> +
> +	filter->pdev = pdev;
> +	list_add_tail(&filter->list, target_list);
> +
> +	/* Update the available port mask */
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		hisi_ptt->port_mask |= hisi_ptt_get_filter_val(pdev);
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_release_filters(void *data)
> +{
> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> +	struct hisi_ptt *hisi_ptt = data;
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +}
> +
> +static int hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct pci_bus *bus;
> +	int ret;
> +	u32 reg;
> +
> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> +	/*
> +	 * The device range register provides the information about the
> +	 * root ports which the RCiEP can control and trace. The RCiEP
> +	 * and the root ports it support are on the same PCIe core, with
> +	 * same domain number but maybe different bus number. The device
> +	 * range register will tell us which root ports we can support,
> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> +	 * while Bit[15:0] indicates the lower.
> +	 */
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> +	hisi_ptt->upper = FIELD_GET(HISI_PTT_DEVICE_RANGE_UPPER, reg);
> +	hisi_ptt->lower = FIELD_GET(HISI_PTT_DEVICE_RANGE_LOWER, reg);
> +
> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
> +	if (bus)
> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_release_filters, hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +
> +	return 0;
> +}
> +
> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> +
> +static ssize_t available_root_port_filters_show(struct device *dev,
> +						struct device_attribute *attr,
> +						char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (list_empty(&hisi_ptt->port_filters)) {
> +		pos = sysfs_emit(buf, "\n");
> +		goto out;
> +	}
> +
> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",

\t can be used for a formatting tab

> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev) |
> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_root_port_filters);
> +
> +static ssize_t available_requester_filters_show(struct device *dev,
> +						struct device_attribute *attr,
> +						char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (list_empty(&hisi_ptt->req_filters)) {
> +		pos = sysfs_emit(buf, "\n");
> +		goto out;
> +	}
> +
> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev));
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_requester_filters);
> +
> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> +PMU_FORMAT_ATTR(type,		"config:24-31");
> +PMU_FORMAT_ATTR(format,		"config:32-35");
> +
> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> +	&format_attr_filter.attr,
> +	&format_attr_direction.attr,
> +	&format_attr_type.attr,
> +	&format_attr_format.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_format_group = {
> +	.name = "format",
> +	.attrs = hisi_ptt_pmu_format_attrs,
> +};
> +
> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> +	&dev_attr_available_root_port_filters.attr,
> +	&dev_attr_available_requester_filters.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> +	.attrs = hisi_ptt_pmu_filter_attrs,
> +};
> +
> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> +	&hisi_ptt_pmu_format_group,
> +	&hisi_ptt_pmu_filter_group,
> +	NULL
> +};
> +
> +/*
> + * Check whether the config is valid or not. Some configs are multi-selectable
> + * and can be set simultaneously, while some are single selectable (onehot).
> + * Use this function to check the non-onehot configs while
> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> + */
> +static int hisi_ptt_trace_valid_config(u32 val, const u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	/* The non-onehot configs cannot be 0. */
> +	if (!val)
> +		return -EINVAL;
> +
> +	/*
> +	 * Walk the available list and clear the valid bits of
> +	 * the config. If there is any resident bit after the
> +	 * walk then the config is invalid.
> +	 */
> +	for (i = 0; i < list_size; i++)
> +		val &= ~available_list[i];
> +
> +	return val ? -EINVAL : 0;
> +}
> +
> +static int hisi_ptt_trace_valid_config_onehot(u32 val, const u32 *available_list, u32 list_size)

line too long

> +{
> +	int i;
> +
> +	for (i = 0; i < list_size; i++)
> +		if (val == available_list[i])
> +			return 0;
> +
> +	return -EINVAL;
> +}
> +
> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> +{
> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> +	struct hisi_ptt_filter_desc *filter;
> +	int ret = -EINVAL;
> +
> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> +
> +	/*
> +	 * Port filters are defined as bit mask. For port filters, check
> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> +	 * and whether it's empty or not, otherwise user has specified
> +	 * some unsupported root ports.
> +	 *
> +	 * For Requester ID filters, walk the available filter list to see
> +	 * whether we have one matched.
> +	 */
> +	if (!hisi_ptt->trace_ctrl.is_port) {
> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)


use {} for this as well. the formatting looks strange

> +			if (val == hisi_ptt_get_filter_val(filter->pdev)) {
> +				ret = 0;
> +				break;
> +			}
> +	} else if (bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> +		ret = 0;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.filter = val;
> +	return 0;
> +}
> +
> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> +{
> +	/*
> +	 * The supported value of the direction parameter. See hisi_ptt.rst
> +	 * documentation for more details.
> +	 */
> +	static const u32 hisi_ptt_trace_available_direction[] = {
> +		0,
> +		1,
> +		2,
> +		3,
> +	};
> +	/* Different types can be set simultaneously */
> +	static const u32 hisi_ptt_trace_available_type[] = {
> +		1,	/* posted_request */
> +		2,	/* non-posted_request */
> +		4,	/* completion */
> +	};
> +	static const u32 hisi_ptt_trace_availble_format[] = {
> +		0,	/* 4DW */
> +		1,	/* 8DW */
> +	};
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	int ret;
> +	u32 val;
> +
> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> +		return -ENOENT;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +
> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> +	if (ret < 0)
> +		goto out;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,

how about put all those arrays in hisi_ptt_trace_valid_config_onehot() 
and pass some flag to say which array you want to use? Or something like 
that. Passing the arrays in this fashion is messy


> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->direction = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->type = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->format = val;
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return ret;
> +}
> +
> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> +				    int nr_pages, bool overwrite)
> +{
> +	struct hisi_ptt_pmu_buf *buf;
> +	struct page **pagelist;
> +	int i;
> +
> +	if (overwrite) {
> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> +		return NULL;
> +	}
> +
> +	/* If the pages size less than buffers, we cannot start trace */
> +	if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
> +		return NULL;
> +
> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> +	if (!buf)
> +		return NULL;
> +
> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> +	if (!pagelist) {
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++)
> +		pagelist[i] = virt_to_page(pages[i]);
> +
> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> +	if (!buf->base) {
> +		kfree(pagelist);
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	buf->nr_pages = nr_pages;
> +	buf->length = nr_pages * PAGE_SIZE;
> +	buf->pos = 0;
> +
> +	kfree(pagelist);
> +	return buf;
> +}
> +
> +static void hisi_ptt_pmu_free_aux(void *aux)
> +{
> +	struct hisi_ptt_pmu_buf *buf = aux;
> +
> +	vunmap(buf->base);
> +	kfree(buf);
> +}
> +
> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct hw_perf_event *hwc = &event->hw;
> +	struct hisi_ptt_pmu_buf *buf;
> +	int cpu = event->cpu;
> +	int ret;
> +
> +	hwc->state = 0;
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.started) {
> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");

doesn't perf core guard against this sort of thing?

> +		goto stop;
> +	}
> +
> +	if (cpu == -1)
> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> +
> +	/*
> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> +	 * core in event_function_local().
> +	 */
> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> +				 cpumask_of(cpu)));
> +
> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> +		goto stop;
> +	}
> +
> +	buf = perf_aux_output_begin(handle, event);
> +	if (!buf) {
> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> +		goto stop;
> +	}
> +
> +	buf->pos = handle->head % buf->length;
> +
> +	ret = hisi_ptt_trace_start(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> +		perf_aux_output_end(handle, 0);
> +		goto stop;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return;
> +stop:
> +	event->hw.state |= PERF_HES_STOPPED;
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +
> +	if (hwc->state & PERF_HES_STOPPED)
> +		return;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.started) {
> +		hisi_ptt_trace_end(hisi_ptt);
> +		WARN(!hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> +	}
> +	mutex_unlock(&hisi_ptt->mutex);
> +
> +	hwc->state |= PERF_HES_STOPPED;
> +	perf_event_update_userpage(event);
> +	hwc->state |= PERF_HES_UPTODATE;
> +}
> +
> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +	int cpu = event->cpu;
> +
> +	/*
> +	 * Only allow the default cpu to add the event if user doesn't specify
> +	 * the cpus.
> +	 */
> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> +		return 0;
> +
> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +
> +	if (flags & PERF_EF_START) {
> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> +		if (hwc->state & PERF_HES_STOPPED)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> +{
> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> +}
> +
> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> +{
> +	u16 core_id, sicl_id;
> +	char *pmu_name;
> +	u32 reg;
> +
> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> +		.module		= THIS_MODULE,
> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> +		.task_ctx_nr	= perf_sw_context,
> +		.attr_groups	= hisi_ptt_pmu_groups,
> +		.event_init	= hisi_ptt_pmu_event_init,
> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> +		.free_aux	= hisi_ptt_pmu_free_aux,
> +		.start		= hisi_ptt_pmu_start,
> +		.stop		= hisi_ptt_pmu_stop,
> +		.add		= hisi_ptt_pmu_add,
> +		.del		= hisi_ptt_pmu_del,
> +	};
> +
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> +	core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> +	sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> +				  sicl_id, core_id);
> +	if (!pmu_name)
> +		return -ENOMEM;
> +
> +	return perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> +}
> +
> +/*
> + * The DMA of PTT trace can only use direct mapping, due to some
> + * hardware restriction. Check whether there is an IOMMU or the
> + * policy of the IOMMU domain is passthrough, otherwise the trace
> + * cannot work.
> + *
> + * The PTT device is supposed to behind the ARM SMMUv3, which

/s/ the ARM SMMUv3/an ARM SMMUv3/

> + * should have passthrough the device by a quirk.
> + */
> +static int hisi_ptt_check_iommu_mapping(struct pci_dev *pdev)
> +{
> +	struct iommu_domain *iommu_domain;
> +
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	return -EOPNOTSUPP;
> +}
> +
> +static int hisi_ptt_probe(struct pci_dev *pdev,
> +			  const struct pci_device_id *id)
> +{
> +	struct hisi_ptt *hisi_ptt;
> +	int ret;
> +
> +	ret = hisi_ptt_check_iommu_mapping(pdev);
> +	if (ret) {
> +		pci_err(pdev, "requires direct DMA mappings\n");
> +		return ret;
> +	}
> +
> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> +	if (!hisi_ptt)
> +		return -ENOMEM;
> +
> +	mutex_init(&hisi_ptt->mutex);
> +	hisi_ptt->pdev = pdev;
> +	pci_set_drvdata(pdev, hisi_ptt);
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		pci_err(pdev, "failed to enable device, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(2), DRV_NAME);
> +	if (ret) {
> +		pci_err(pdev, "failed to remap io memory, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> +
> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> +	if (ret) {
> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	pci_set_master(pdev);
> +
> +	ret = hisi_ptt_register_irq(hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	ret = hisi_ptt_init_ctrls(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to init controls, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to register PMU device, ret = %d", ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_remove(struct pci_dev *pdev)
> +{
> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> +
> +	/*
> +	 * We have to manually unregister the PMU device rather than make it
> +	 * devres managed to keep order that the PMU device's unregistration
> +	 * is prior to the release of DMA buffers. As the DMA buffers are
> +	 * devm allocated when necessary which is after the registration of
> +	 * the PMU device.
> +	 */

do you really need to mention all this?

> +	perf_pmu_unregister(&hisi_ptt->hisi_ptt_pmu);
> +}
> +
> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> +	{ }
> +};
> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> +
> +static struct pci_driver hisi_ptt_driver = {
> +	.name = DRV_NAME,
> +	.id_table = hisi_ptt_id_tbl,
> +	.probe = hisi_ptt_probe,
> +	.remove = hisi_ptt_remove,
> +};
> +module_pci_driver(hisi_ptt_driver);
> +
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
> new file mode 100644
> index 000000000000..29b6d6293d6c
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.h
> @@ -0,0 +1,166 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#ifndef _HISI_PTT_H
> +#define _HISI_PTT_H
> +
> +#include <linux/bits.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/perf_event.h>
> +#include <linux/types.h>
> +
> +#define DRV_NAME "hisi_ptt"
> +
> +/*
> + * The definition of the device registers and register fields.
> + */
> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> +#define HISI_PTT_TRACE_CTRL		0x0850
> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS		0x08b0
> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> +#define   HISI_PTT_DEVICE_RANGE_UPPER	GENMASK(31, 16)
> +#define   HISI_PTT_DEVICE_RANGE_LOWER	GENMASK(15, 0)
> +#define HISI_PTT_LOCATION		0x0fe8
> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> +
> +/* Parameters of PTT trace DMA part. */
> +#define HISI_PTT_TRACE_DMA_IRQ			0
> +#define HISI_PTT_TRACE_BUF_CNT			4
> +#define HISI_PTT_TRACE_BUF_SIZE			SZ_4M
> +#define HISI_PTT_TRACE_TOTAL_BUF_SIZE		(HISI_PTT_TRACE_BUF_SIZE * \
> +						 HISI_PTT_TRACE_BUF_CNT)
> +/* Wait time for hardware DMA to reset */
> +#define HISI_PTT_RESET_TIMEOUT_US	10UL
> +#define HISI_PTT_RESET_POLL_INTERVAL_US	1UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TRACE_TIMEOUT_US	100UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	10UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> +
> +/**
> + * struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
> + *                              The detail of the data format is described
> + *                              in the documentation of PTT device.
> + * @dma:   DMA address of this buffer visible to the device
> + * @addr:  virtual address of this buffer visible to the cpu
> + */
> +struct hisi_ptt_dma_buffer {
> +	dma_addr_t dma;
> +	void *addr;
> +};
> +
> +/**
> + * struct hisi_ptt_trace_ctrl - control and status of PTT trace
> + * @trace_buf:   array of the trace buffers for holding the trace data.
> + *               the length will be HISI_PTT_TRACE_BUF_CNT.
> + * @handle:      perf output handle of current trace session
> + * @default_cpu: default cpu to start the trace session
> + * @buf_index:   the index of current using trace buffer
> + * @started:     current trace status, true for started
> + * @is_port:     whether we're tracing root port or not
> + * @direction:   direction of the TLP headers to trace
> + * @filter:      filter value for tracing the TLP headers
> + * @format:      format of the TLP headers to trace
> + * @type:        type of the TLP headers to trace
> + */
> +struct hisi_ptt_trace_ctrl {
> +	struct hisi_ptt_dma_buffer *trace_buf;
> +	struct perf_output_handle handle;
> +	int default_cpu;
> +	u32 buf_index;
> +	bool started;
> +	bool is_port;
> +	u32 direction:2;
> +	u32 filter:16;
> +	u32 format:1;
> +	u32 type:4;
> +};
> +
> +/**
> + * struct hisi_ptt_filter_desc - descriptor of the PTT trace filter
> + * @list: entry of this descriptor in the filter list
> + * @pdev: pci_dev related to this filter
> + */
> +struct hisi_ptt_filter_desc {
> +	struct list_head list;
> +	struct pci_dev *pdev;
> +};
> +
> +
> +/**
> + * struct hisi_ptt_pmu_buf - descriptor of the AUX buffer of PTT trace
> + * @length:   size of the AUX buffer
> + * @nr_pages: number of pages of the AUX buffer
> + * @base:     start address of AUX buffer
> + * @pos:      position in the AUX buffer to commit traced data
> + */
> +struct hisi_ptt_pmu_buf {
> +	size_t length;
> +	int nr_pages;
> +	void *base;
> +	long pos;
> +};
> +
> +/**
> + * struct hisi_ptt - per PTT device data
> + * @trace_ctrl:   the control information of PTT trace
> + * @hisi_ptt_pmu: the pum device of trace
> + * @iobase:       base IO address of the device
> + * @pdev:         pci_dev of this PTT device
> + * @mutex:        mutex to protect the filter list and serialize the perf process.
> + * @upper:        the upper BDF range of the PCI devices managed by this PTT device
> + * @lower:        the lower BDF range of the PCI devices managed by this PTT device
> + * @port_filters: the filter list of root ports
> + * @req_filters:  the filter list of requester ID
> + * @port_mask:    port mask of the managed root ports
> + */
> +struct hisi_ptt {
> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> +	struct pmu hisi_ptt_pmu;
> +	void __iomem *iobase;
> +	struct pci_dev *pdev;
> +	struct mutex mutex;
> +	u32 upper;

upper_bdf would be a better name

people don't want to always refer to comments to check the meaning

> +	u32 lower;
> +
> +	/*
> +	 * The trace TLP headers can either be filtered by certain
> +	 * root port, or by the requester ID. Organize the filters
> +	 * by @port_filters and @req_filters here. The mask of all
> +	 * the valid ports is also cached for doing sanity check
> +	 * of user input.
> +	 */
> +	struct list_head port_filters;
> +	struct list_head req_filters;
> +	u16 port_mask;
> +};
> +
> +#define to_hisi_ptt(pmu) container_of(pmu, struct hisi_ptt, hisi_ptt_pmu)
> +
> +#endif /* _HISI_PTT_H */


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-04-07 12:58 ` [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
@ 2022-04-11 17:19   ` John Garry
  2022-04-12  7:41     ` Yicong Yang
  2022-04-30  7:35   ` Leo Yan
  2022-05-10 10:14   ` James Clark
  2 siblings, 1 reply; 30+ messages in thread
From: John Garry @ 2022-04-11 17:19 UTC (permalink / raw)
  To: Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

On 07/04/2022 13:58, Yicong Yang wrote:
> From: Qi Liu <liuqi115@huawei.com>
> 
> 'perf record' and 'perf report --dump-raw-trace' supported in this
> patch.
> 
> Example usage:
> 
> Output will contain raw PTT data and its textual representation, such
> as:
> 
> 0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
> ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
> .
> . ... HISI PTT data: size 4194304 bytes
> .  00000000: 00 00 00 00                                 Prefix
> .  00000004: 08 20 00 60                                 Header DW0
> .  00000008: ff 02 00 01                                 Header DW1
> .  0000000c: 20 08 00 00                                 Header DW2
> .  00000010: 10 e7 44 ab                                 Header DW3
> .  00000014: 2a a8 1e 01                                 Time
> .  00000020: 00 00 00 00                                 Prefix
> .  00000024: 01 00 00 60                                 Header DW0
> .  00000028: 0f 1e 00 01                                 Header DW1
> .  0000002c: 04 00 00 00                                 Header DW2
> .  00000030: 40 00 81 02                                 Header DW3
> .  00000034: ee 02 00 00                                 Time
> ....
> 
> Signed-off-by: Qi Liu <liuqi115@huawei.com>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>   tools/perf/arch/arm/util/auxtrace.c           |  76 +++++-
>   tools/perf/arch/arm/util/pmu.c                |   3 +
>   tools/perf/arch/arm64/util/Build              |   2 +-
>   tools/perf/arch/arm64/util/hisi_ptt.c         | 195 ++++++++++++++++
>   tools/perf/util/Build                         |   2 +
>   tools/perf/util/auxtrace.c                    |   4 +
>   tools/perf/util/auxtrace.h                    |   1 +
>   tools/perf/util/hisi-ptt-decoder/Build        |   1 +
>   .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 ++++++++++++++
>   .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
>   tools/perf/util/hisi_ptt.c                    | 218 ++++++++++++++++++
>   tools/perf/util/hisi_ptt.h                    |  28 +++
>   12 files changed, 724 insertions(+), 4 deletions(-)
>   create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
>   create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
>   create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
>   create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
>   create mode 100644 tools/perf/util/hisi_ptt.c
>   create mode 100644 tools/perf/util/hisi_ptt.h
> 
> diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
> index 5fc6a2a3dbc5..393f5757c039 100644
> --- a/tools/perf/arch/arm/util/auxtrace.c
> +++ b/tools/perf/arch/arm/util/auxtrace.c
> @@ -4,9 +4,11 @@
>    * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
>    */
>   
> +#include <dirent.h>
>   #include <stdbool.h>
>   #include <linux/coresight-pmu.h>
>   #include <linux/zalloc.h>
> +#include <api/fs/fs.h>
>   
>   #include "../../../util/auxtrace.h"
>   #include "../../../util/debug.h"
> @@ -14,6 +16,7 @@
>   #include "../../../util/pmu.h"
>   #include "cs-etm.h"
>   #include "arm-spe.h"
> +#include "hisi_ptt.h"
>   
>   static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>   {
> @@ -50,6 +53,58 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>   	return arm_spe_pmus;
>   }
>   
> +static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
> +{
> +	const char *sysfs = sysfs__mountpoint();
> +	struct perf_pmu **hisi_ptt_pmus = NULL;
> +	struct dirent *dent;
> +	char path[PATH_MAX];
> +	DIR *dir = NULL;
> +	int idx = 0;
> +
> +	snprintf(path, PATH_MAX, "%s" EVENT_SOURCE_DEVICE_PATH, sysfs);
> +	dir = opendir(path);
> +	if (!dir) {
> +		pr_err("can't read directory '%s'\n", EVENT_SOURCE_DEVICE_PATH);
> +		*err = -EINVAL;
> +		goto out;
> +	}
> +
> +	while ((dent = readdir(dir))) {
> +		if (strstr(dent->d_name, HISI_PTT_PMU_NAME))
> +			(*nr_ptts)++;
> +	}
> +
> +	if (!(*nr_ptts))
> +		goto out;
> +
> +	hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * (*nr_ptts));
> +	if (!hisi_ptt_pmus) {
> +		pr_err("hisi_ptt alloc failed\n");
> +		*err = -ENOMEM;

using PTR_ERR seems better, if possible

> +		goto out;
> +	}
> +
> +	rewinddir(dir);
> +	while ((dent = readdir(dir))) {
> +		if (strstr(dent->d_name, HISI_PTT_PMU_NAME) && idx < (*nr_ptts)) {
> +			hisi_ptt_pmus[idx] = perf_pmu__find(dent->d_name);
> +			if (hisi_ptt_pmus[idx]) {
> +				pr_debug2("%s %d: hisi_ptt_pmu %d type %d name %s\n",

do you really need this?

> +					__func__, __LINE__, idx,
> +					hisi_ptt_pmus[idx]->type,
> +					hisi_ptt_pmus[idx]->name);
> +					idx++;
> +			}
> +
> +		}
> +	}
> +
> +out:
> +	closedir(dir);
> +	return hisi_ptt_pmus;
> +}
> +
>   struct auxtrace_record
>   *auxtrace_record__init(struct evlist *evlist, int *err)
>   {
> @@ -57,8 +112,12 @@ struct auxtrace_record
>   	struct evsel *evsel;
>   	bool found_etm = false;
>   	struct perf_pmu *found_spe = NULL;
> +	struct perf_pmu *found_ptt = NULL;
>   	struct perf_pmu **arm_spe_pmus = NULL;
> +	struct perf_pmu **hisi_ptt_pmus = NULL;
> +
>   	int nr_spes = 0;
> +	int nr_ptts = 0;
>   	int i = 0;
>   
>   	if (!evlist)
> @@ -66,13 +125,14 @@ struct auxtrace_record
>   
>   	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
>   	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
> +	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
>   
>   	evlist__for_each_entry(evlist, evsel) {
>   		if (cs_etm_pmu &&
>   		    evsel->core.attr.type == cs_etm_pmu->type)
>   			found_etm = true;
>   
> -		if (!nr_spes || found_spe)
> +		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
>   			continue;
>   
>   		for (i = 0; i < nr_spes; i++) {
> @@ -81,11 +141,18 @@ struct auxtrace_record
>   				break;
>   			}
>   		}
> +
> +		for (i = 0; i < nr_ptts; i++) {
> +			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
> +				found_ptt = hisi_ptt_pmus[i];
> +				break;
> +			}
> +		}
>   	}
>   	free(arm_spe_pmus);

do you need to do something similar for hisi ptt?

>   
> -	if (found_etm && found_spe) {
> -		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
> +	if (found_etm && found_spe && found_ptt) {
> +		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
>   		*err = -EOPNOTSUPP;
>   		return NULL;
>   	}
> @@ -96,6 +163,9 @@ struct auxtrace_record
>   #if defined(__aarch64__)
>   	if (found_spe)
>   		return arm_spe_recording_init(err, found_spe);
> +
> +	if (found_ptt)
> +		return hisi_ptt_recording_init(err, found_ptt);
>   #endif
>   
>   	/*
> diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
> index b8b23b9dc598..89a3cedb4557 100644
> --- a/tools/perf/arch/arm/util/pmu.c
> +++ b/tools/perf/arch/arm/util/pmu.c
> @@ -10,6 +10,7 @@
>   #include <linux/string.h>
>   
>   #include "arm-spe.h"
> +#include "hisi_ptt.h"
>   #include "../../../util/pmu.h"
>   
>   struct perf_event_attr
> @@ -22,6 +23,8 @@ struct perf_event_attr
>   #if defined(__aarch64__)
>   	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
>   		return arm_spe_pmu_default_config(pmu);
> +	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
> +		pmu->selectable = true;
>   #endif
>   	}
>   
> diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
> index 9fcb4e68add9..8b7fd1dc9f37 100644
> --- a/tools/perf/arch/arm64/util/Build
> +++ b/tools/perf/arch/arm64/util/Build
> @@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
>   perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
>   			      ../../arm/util/auxtrace.o \
>   			      ../../arm/util/cs-etm.o \
> -			      arm-spe.o mem-events.o
> +			      arm-spe.o mem-events.o hisi_ptt.o

nit: others use '-' in the name


I'll check the rest tomorrow.

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-11 17:02   ` John Garry
@ 2022-04-12  7:36     ` Yicong Yang
  2022-04-12  8:39       ` John Garry
  0 siblings, 1 reply; 30+ messages in thread
From: Yicong Yang @ 2022-04-12  7:36 UTC (permalink / raw)
  To: John Garry, Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

Hi John,

Thanks for the comments! some questions replied below.

On 2022/4/12 1:02, John Garry wrote:
> On 07/04/2022 13:58, Yicong Yang wrote:
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
>> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
>> tune the PCIe traffic, and trace the TLP headers.
>>
>> Add the driver for the device to enable the trace function. Register PMU
>> device of PTT trace, then users can use trace through perf command. The
>> driver makes use of perf AUX trace and support following events to
> 
> "The driver makes use of perf AUX trace function and supports the following events to .."
> 
>> configure the trace:
>>
>> - filter: select Root port or Endpoint to trace
>> - type: select the type of traced TLP headers
>> - direction: select the direction of traced TLP headers
>> - format: select the data format of the traced TLP headers
>>
>> This patch adds the driver part of PTT trace. The perf command support of
>> PTT trace is added in the following patch.
> 
> I would not mention "following patch" and the like. Just have "initially add a basic driver"
> 

ok. will refine the commit here and above.

>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
>>   drivers/Makefile                 |   1 +
>>   drivers/hwtracing/Kconfig        |   2 +
>>   drivers/hwtracing/ptt/Kconfig    |  12 +
>>   drivers/hwtracing/ptt/Makefile   |   2 +
>>   drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
>>   drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
>>   6 files changed, 1057 insertions(+)
>>   create mode 100644 drivers/hwtracing/ptt/Kconfig
>>   create mode 100644 drivers/hwtracing/ptt/Makefile
>>   create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>>   create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>>
>> diff --git a/drivers/Makefile b/drivers/Makefile
>> index 020780b6b4d2..662d50599467 100644
>> --- a/drivers/Makefile
>> +++ b/drivers/Makefile
>> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)        += thunderbolt/
>>   obj-$(CONFIG_CORESIGHT)        += hwtracing/coresight/
>>   obj-y                += hwtracing/intel_th/
>>   obj-$(CONFIG_STM)        += hwtracing/stm/
>> +obj-$(CONFIG_HISI_PTT)        += hwtracing/ptt/
>>   obj-$(CONFIG_ANDROID)        += android/
>>   obj-$(CONFIG_NVMEM)        += nvmem/
>>   obj-$(CONFIG_FPGA)        += fpga/
>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>> index 13085835a636..911ee977103c 100644
>> --- a/drivers/hwtracing/Kconfig
>> +++ b/drivers/hwtracing/Kconfig
>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>     source "drivers/hwtracing/intel_th/Kconfig"
>>   +source "drivers/hwtracing/ptt/Kconfig"
>> +
>>   endmenu
>> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
>> new file mode 100644
>> index 000000000000..8902a6f27563
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/Kconfig
>> @@ -0,0 +1,12 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config HISI_PTT
>> +    tristate "HiSilicon PCIe Tune and Trace Device"
>> +    depends on ARM64 || (COMPILE_TEST && 64BIT)
>> +    depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
>> +    help
>> +      HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
>> +      device, and it provides support for PCIe traffic tuning and
>> +      tracing TLP headers to the memory.
>> +
>> +      This driver can also be built as a module. If so, the module
>> +      will be called hisi_ptt.
>> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
>> new file mode 100644
>> index 000000000000..908c09a98161
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
>> new file mode 100644
>> index 000000000000..242b41870380
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
>> @@ -0,0 +1,874 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for HiSilicon PCIe tune and trace device
>> + *
>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>> + * Author: Yicong Yang <yangyicong@hisilicon.com>
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/bitops.h>
>> +#include <linux/delay.h>
>> +#include <linux/dma-iommu.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io.h>
>> +#include <linux/iommu.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/module.h>
>> +#include <linux/sysfs.h>
>> +#include <linux/vmalloc.h>
>> +
>> +#include "hisi_ptt.h"
>> +
>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>> +{
>> +    if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>> +        return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>> +
>> +    return PCI_DEVID(pdev->bus->number, pdev->devfn);
>> +}
>> +
>> +static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>> +{
>> +    u32 val;
>> +
>> +    return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
>> +                      val, val & HISI_PTT_TRACE_IDLE,
>> +                      HISI_PTT_WAIT_POLL_INTERVAL_US,
>> +                      HISI_PTT_WAIT_TRACE_TIMEOUT_US);
>> +}
>> +
>> +static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
>> +{
>> +    u32 val;
>> +
>> +    return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
>> +                      val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
>> +                      HISI_PTT_RESET_TIMEOUT_US);
>> +}
>> +
>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +    struct device *dev = &hisi_ptt->pdev->dev;
>> +    int i;
>> +
>> +    if (!ctrl->trace_buf)
>> +        return;
>> +
>> +    for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>> +        if (ctrl->trace_buf[i].addr)
>> +            dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>> +                       ctrl->trace_buf[i].addr,
>> +                       ctrl->trace_buf[i].dma);
>> +    }
>> +
>> +    devm_kfree(dev, ctrl->trace_buf);
>> +    ctrl->trace_buf = NULL;
>> +}
>> +
>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +    struct device *dev = &hisi_ptt->pdev->dev;
>> +    int i;
>> +
>> +    hisi_ptt->trace_ctrl.buf_index = 0;
>> +
>> +    /* If the trace buffer has already been allocated, zero it. */
> 
> I am not sure why this is not called from the probe
> 

The buffer allocation is done when necessary as driver will probe the device on booting but
the user may never use it. In this condition it's a waste of memory if we allocate the buffers
in probe. Currently we'll allocate 16M memory for 4 buffers.

So this function is called every time before we start trace. In the first time it will allocate
the DMA buffers and it the other times it just zero the buffers to clear the data of last time.

>> +    if (ctrl->trace_buf) {
>> +        for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
>> +            memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>> +        return 0;
>> +    }
>> +
>> +    ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
>> +                       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
> 
> sizeof(*ctrl->trace_buf) may be better
>

ok.

>> +    if (!ctrl->trace_buf)
>> +        return -ENOMEM;
>> +
>> +    for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>> +        ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>> +                                 &ctrl->trace_buf[i].dma,
>> +                                 GFP_KERNEL);
>> +        if (!ctrl->trace_buf[i].addr) {
>> +            hisi_ptt_free_trace_buf(hisi_ptt);
>> +            return -ENOMEM;
>> +        }
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>> +{
>> +    writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +    hisi_ptt->trace_ctrl.started = false;
>> +}
>> +
>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>> +{
>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +    u32 val;
>> +    int i;
>> +
>> +    /* Check device idle before start trace */
>> +    if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>> +        pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
>> +        return -EBUSY;
>> +    }
>> +
>> +    ctrl->started = true;
>> +
>> +    /* Reset the DMA before start tracing */
>> +    val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +    val |= HISI_PTT_TRACE_CTRL_RST;
>> +    writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +    hisi_ptt_wait_dma_reset_done(hisi_ptt);
>> +
>> +    val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +    val &= ~HISI_PTT_TRACE_CTRL_RST;
>> +    writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +    /* Clear the interrupt status */
>> +    writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +    writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>> +
>> +    /* Configure the trace DMA buffer */
> 
> I am not sure why this sort of thing is done outside probing
> 

Since the buffers are allocated on necessary as illustrated above, we cannot configure the
buffer address in probe. As illustrated above.

>> +    for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>> +        writel(lower_32_bits(ctrl->trace_buf[i].dma),
>> +               hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>> +               i * HISI_PTT_TRACE_ADDR_STRIDE);
>> +        writel(upper_32_bits(ctrl->trace_buf[i].dma),
>> +               hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>> +               i * HISI_PTT_TRACE_ADDR_STRIDE);
>> +    }
>> +    writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>> +
>> +    /* Set the trace control register */
>> +    val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>> +    val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>> +    val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>> +    val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>> +    if (!hisi_ptt->trace_ctrl.is_port)
>> +        val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>> +
>> +    /* Start the Trace */
>> +    val |= HISI_PTT_TRACE_CTRL_EN;
>> +    writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +    return 0;
>> +}
[...]
>> +static ssize_t available_root_port_filters_show(struct device *dev,
>> +                        struct device_attribute *attr,
>> +                        char *buf)
>> +{
>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>> +    struct hisi_ptt_filter_desc *filter;
>> +    int pos = 0;
>> +
>> +    mutex_lock(&hisi_ptt->mutex);
>> +    if (list_empty(&hisi_ptt->port_filters)) {
>> +        pos = sysfs_emit(buf, "\n");
>> +        goto out;
>> +    }
>> +
>> +    list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>> +        pos += sysfs_emit_at(buf, pos, "%s    0x%05lx\n",
> 
> \t can be used for a formatting tab
> 

actually I used a tab here. maybe I can make it obvious like:

       pos += sysfs_emit_at(buf, pos, "%s\t0x%05lx\n",

>> +                     pci_name(filter->pdev),
>> +                     hisi_ptt_get_filter_val(filter->pdev) |
>> +                     HISI_PTT_PMU_FILTER_IS_PORT);
>> +
>> +out:
>> +    mutex_unlock(&hisi_ptt->mutex);
>> +    return pos;
>> +}
>> +static DEVICE_ATTR_ADMIN_RO(available_root_port_filters);
>> +
>> +static ssize_t available_requester_filters_show(struct device *dev,
>> +                        struct device_attribute *attr,
>> +                        char *buf)
>> +{
>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>> +    struct hisi_ptt_filter_desc *filter;
>> +    int pos = 0;
>> +
>> +    mutex_lock(&hisi_ptt->mutex);
>> +    if (list_empty(&hisi_ptt->req_filters)) {
>> +        pos = sysfs_emit(buf, "\n");
>> +        goto out;
>> +    }
>> +
>> +    list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> +        pos += sysfs_emit_at(buf, pos, "%s    0x%05x\n",
>> +                     pci_name(filter->pdev),
>> +                     hisi_ptt_get_filter_val(filter->pdev));
>> +
>> +out:
>> +    mutex_unlock(&hisi_ptt->mutex);
>> +    return pos;
>> +}
>> +static DEVICE_ATTR_ADMIN_RO(available_requester_filters);
>> +
>> +PMU_FORMAT_ATTR(filter,        "config:0-19");
>> +PMU_FORMAT_ATTR(direction,    "config:20-23");
>> +PMU_FORMAT_ATTR(type,        "config:24-31");
>> +PMU_FORMAT_ATTR(format,        "config:32-35");
>> +
>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>> +    &format_attr_filter.attr,
>> +    &format_attr_direction.attr,
>> +    &format_attr_type.attr,
>> +    &format_attr_format.attr,
>> +    NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>> +    .name = "format",
>> +    .attrs = hisi_ptt_pmu_format_attrs,
>> +};
>> +
>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>> +    &dev_attr_available_root_port_filters.attr,
>> +    &dev_attr_available_requester_filters.attr,
>> +    NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>> +    .attrs = hisi_ptt_pmu_filter_attrs,
>> +};
>> +
>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>> +    &hisi_ptt_pmu_format_group,
>> +    &hisi_ptt_pmu_filter_group,
>> +    NULL
>> +};
>> +
>> +/*
>> + * Check whether the config is valid or not. Some configs are multi-selectable
>> + * and can be set simultaneously, while some are single selectable (onehot).
>> + * Use this function to check the non-onehot configs while
>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>> + */
>> +static int hisi_ptt_trace_valid_config(u32 val, const u32 *available_list, u32 list_size)
>> +{
>> +    int i;
>> +
>> +    /* The non-onehot configs cannot be 0. */
>> +    if (!val)
>> +        return -EINVAL;
>> +
>> +    /*
>> +     * Walk the available list and clear the valid bits of
>> +     * the config. If there is any resident bit after the
>> +     * walk then the config is invalid.
>> +     */
>> +    for (i = 0; i < list_size; i++)
>> +        val &= ~available_list[i];
>> +
>> +    return val ? -EINVAL : 0;
>> +}
>> +
>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, const u32 *available_list, u32 list_size)
> 
> line too long
> 

will wrap it.

>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < list_size; i++)
>> +        if (val == available_list[i])
>> +            return 0;
>> +
>> +    return -EINVAL;
>> +}
>> +
>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>> +{
>> +    unsigned long val, port_mask = hisi_ptt->port_mask;
>> +    struct hisi_ptt_filter_desc *filter;
>> +    int ret = -EINVAL;
>> +
>> +    hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>> +    val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>> +
>> +    /*
>> +     * Port filters are defined as bit mask. For port filters, check
>> +     * the bits in the @val are within the range of hisi_ptt->port_mask
>> +     * and whether it's empty or not, otherwise user has specified
>> +     * some unsupported root ports.
>> +     *
>> +     * For Requester ID filters, walk the available filter list to see
>> +     * whether we have one matched.
>> +     */
>> +    if (!hisi_ptt->trace_ctrl.is_port) {
>> +        list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> 
> 
> use {} for this as well. the formatting looks strange
> 

ok.

>> +            if (val == hisi_ptt_get_filter_val(filter->pdev)) {
>> +                ret = 0;
>> +                break;
>> +            }
>> +    } else if (bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>> +        ret = 0;
>> +    }
>> +
>> +    if (ret)
>> +        return ret;
>> +
>> +    hisi_ptt->trace_ctrl.filter = val;
>> +    return 0;
>> +}
>> +
>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>> +{
>> +    /*
>> +     * The supported value of the direction parameter. See hisi_ptt.rst
>> +     * documentation for more details.
>> +     */
>> +    static const u32 hisi_ptt_trace_available_direction[] = {
>> +        0,
>> +        1,
>> +        2,
>> +        3,
>> +    };
>> +    /* Different types can be set simultaneously */
>> +    static const u32 hisi_ptt_trace_available_type[] = {
>> +        1,    /* posted_request */
>> +        2,    /* non-posted_request */
>> +        4,    /* completion */
>> +    };
>> +    static const u32 hisi_ptt_trace_availble_format[] = {
>> +        0,    /* 4DW */
>> +        1,    /* 8DW */
>> +    };
>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +    int ret;
>> +    u32 val;
>> +
>> +    if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>> +        return -ENOENT;
>> +
>> +    mutex_lock(&hisi_ptt->mutex);
>> +
>> +    ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>> +    if (ret < 0)
>> +        goto out;
>> +
>> +    val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>> +    ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> 
> how about put all those arrays in hisi_ptt_trace_valid_config_onehot() and pass some flag to say which array you want to use? Or something like that. Passing the arrays in this fashion is messy
> 

Since there are 3 configs (type, direction, format) with different available range and setting method (onehot, non-onehot),
moving the arrays into the valid checking function means we need to recognize the config types (passed by the caller but need
to know the available value array) and the checking method together. That may make the code more complex than now: 1st picking
up the right array and judge wich checking method this array applied and 2nd do the checking.

Currently it's designed to decouple the checking method and the available value array. The hisi_ptt_trace_valid_config{_onehot}()
won't care about which array to use since caller take responsibilty for this. So perhaps current approach is simple and clear
enough.

> 
>> +                         ARRAY_SIZE(hisi_ptt_trace_available_direction));
>> +    if (ret < 0)
>> +        goto out;
>> +    ctrl->direction = val;
>> +
>> +    val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>> +    ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>> +                      ARRAY_SIZE(hisi_ptt_trace_available_type));
>> +    if (ret < 0)
>> +        goto out;
>> +    ctrl->type = val;
>> +
>> +    val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>> +    ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>> +                         ARRAY_SIZE(hisi_ptt_trace_availble_format));
>> +    if (ret < 0)
>> +        goto out;
>> +    ctrl->format = val;
>> +
>> +out:
>> +    mutex_unlock(&hisi_ptt->mutex);
>> +    return ret;
>> +}
>> +
>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>> +                    int nr_pages, bool overwrite)
>> +{
>> +    struct hisi_ptt_pmu_buf *buf;
>> +    struct page **pagelist;
>> +    int i;
>> +
>> +    if (overwrite) {
>> +        dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>> +        return NULL;
>> +    }
>> +
>> +    /* If the pages size less than buffers, we cannot start trace */
>> +    if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
>> +        return NULL;
>> +
>> +    buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>> +    if (!buf)
>> +        return NULL;
>> +
>> +    pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>> +    if (!pagelist) {
>> +        kfree(buf);
>> +        return NULL;
>> +    }
>> +
>> +    for (i = 0; i < nr_pages; i++)
>> +        pagelist[i] = virt_to_page(pages[i]);
>> +
>> +    buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>> +    if (!buf->base) {
>> +        kfree(pagelist);
>> +        kfree(buf);
>> +        return NULL;
>> +    }
>> +
>> +    buf->nr_pages = nr_pages;
>> +    buf->length = nr_pages * PAGE_SIZE;
>> +    buf->pos = 0;
>> +
>> +    kfree(pagelist);
>> +    return buf;
>> +}
>> +
>> +static void hisi_ptt_pmu_free_aux(void *aux)
>> +{
>> +    struct hisi_ptt_pmu_buf *buf = aux;
>> +
>> +    vunmap(buf->base);
>> +    kfree(buf);
>> +}
>> +
>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>> +{
>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +    struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +    struct hw_perf_event *hwc = &event->hw;
>> +    struct hisi_ptt_pmu_buf *buf;
>> +    int cpu = event->cpu;
>> +    int ret;
>> +
>> +    hwc->state = 0;
>> +    mutex_lock(&hisi_ptt->mutex);
>> +    if (hisi_ptt->trace_ctrl.started) {
>> +        pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> 
> doesn't perf core guard against this sort of thing?
> 

Maybe not as tested. The perf core will start the events 1)on the cpus user specified or
2)on all the cpus, but the PTT trace is intended to start once on one cpu.

For the 2) case, the driver will make default cpu to start the trace and block others
in pmu::add(). For the 1) case we'll met the condition here. So the started status is
test here to avoid a second start.

>> +        goto stop;
>> +    }
>> +
>> +    if (cpu == -1)
>> +        cpu = hisi_ptt->trace_ctrl.default_cpu;
>> +
>> +    /*
>> +     * Handle the interrupt on the same cpu which starts the trace to avoid
>> +     * context mismatch. Otherwise we'll trigger the WARN from the perf
>> +     * core in event_function_local().
>> +     */
>> +    WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +                 cpumask_of(cpu)));
>> +
>> +    ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>> +    if (ret) {
>> +        pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>> +        goto stop;
>> +    }
>> +
>> +    buf = perf_aux_output_begin(handle, event);
>> +    if (!buf) {
>> +        pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>> +        goto stop;
>> +    }
>> +
>> +    buf->pos = handle->head % buf->length;
>> +
>> +    ret = hisi_ptt_trace_start(hisi_ptt);
>> +    if (ret) {
>> +        pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>> +        perf_aux_output_end(handle, 0);
>> +        goto stop;
>> +    }
>> +
>> +    mutex_unlock(&hisi_ptt->mutex);
>> +    return;
>> +stop:
>> +    event->hw.state |= PERF_HES_STOPPED;
>> +    mutex_unlock(&hisi_ptt->mutex);
>> +}
>> +
>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>> +{
>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +    struct hw_perf_event *hwc = &event->hw;
>> +
>> +    if (hwc->state & PERF_HES_STOPPED)
>> +        return;
>> +
>> +    mutex_lock(&hisi_ptt->mutex);
>> +    if (hisi_ptt->trace_ctrl.started) {
>> +        hisi_ptt_trace_end(hisi_ptt);
>> +        WARN(!hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>> +        hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>> +    }
>> +    mutex_unlock(&hisi_ptt->mutex);
>> +
>> +    hwc->state |= PERF_HES_STOPPED;
>> +    perf_event_update_userpage(event);
>> +    hwc->state |= PERF_HES_UPTODATE;
>> +}
>> +
>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>> +{
>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +    struct hw_perf_event *hwc = &event->hw;
>> +    int cpu = event->cpu;
>> +
>> +    /*
>> +     * Only allow the default cpu to add the event if user doesn't specify
>> +     * the cpus.
>> +     */
>> +    if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>> +        return 0;
>> +
>> +    hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>> +
>> +    if (flags & PERF_EF_START) {
>> +        hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>> +        if (hwc->state & PERF_HES_STOPPED)
>> +            return -EINVAL;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>> +{
>> +    hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>> +}
>> +
>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>> +{
>> +    u16 core_id, sicl_id;
>> +    char *pmu_name;
>> +    u32 reg;
>> +
>> +    hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>> +        .module        = THIS_MODULE,
>> +        .capabilities    = PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>> +        .task_ctx_nr    = perf_sw_context,
>> +        .attr_groups    = hisi_ptt_pmu_groups,
>> +        .event_init    = hisi_ptt_pmu_event_init,
>> +        .setup_aux    = hisi_ptt_pmu_setup_aux,
>> +        .free_aux    = hisi_ptt_pmu_free_aux,
>> +        .start        = hisi_ptt_pmu_start,
>> +        .stop        = hisi_ptt_pmu_stop,
>> +        .add        = hisi_ptt_pmu_add,
>> +        .del        = hisi_ptt_pmu_del,
>> +    };
>> +
>> +    reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>> +    core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>> +    sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>> +
>> +    pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>> +                  sicl_id, core_id);
>> +    if (!pmu_name)
>> +        return -ENOMEM;
>> +
>> +    return perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>> +}
>> +
>> +/*
>> + * The DMA of PTT trace can only use direct mapping, due to some
>> + * hardware restriction. Check whether there is an IOMMU or the
>> + * policy of the IOMMU domain is passthrough, otherwise the trace
>> + * cannot work.
>> + *
>> + * The PTT device is supposed to behind the ARM SMMUv3, which
> 
> /s/ the ARM SMMUv3/an ARM SMMUv3/
> 
ok.
>> + * should have passthrough the device by a quirk.
>> + */
>> +static int hisi_ptt_check_iommu_mapping(struct pci_dev *pdev)
>> +{
>> +    struct iommu_domain *iommu_domain;
>> +
>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> +        return 0;
>> +
>> +    return -EOPNOTSUPP;
>> +}
>> +
>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>> +              const struct pci_device_id *id)
>> +{
>> +    struct hisi_ptt *hisi_ptt;
>> +    int ret;
>> +
>> +    ret = hisi_ptt_check_iommu_mapping(pdev);
>> +    if (ret) {
>> +        pci_err(pdev, "requires direct DMA mappings\n");
>> +        return ret;
>> +    }
>> +
>> +    hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>> +    if (!hisi_ptt)
>> +        return -ENOMEM;
>> +
>> +    mutex_init(&hisi_ptt->mutex);
>> +    hisi_ptt->pdev = pdev;
>> +    pci_set_drvdata(pdev, hisi_ptt);
>> +
>> +    ret = pcim_enable_device(pdev);
>> +    if (ret) {
>> +        pci_err(pdev, "failed to enable device, ret = %d\n", ret);
>> +        return ret;
>> +    }
>> +
>> +    ret = pcim_iomap_regions(pdev, BIT(2), DRV_NAME);
>> +    if (ret) {
>> +        pci_err(pdev, "failed to remap io memory, ret = %d\n", ret);
>> +        return ret;
>> +    }
>> +
>> +    hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>> +
>> +    ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>> +    if (ret) {
>> +        pci_err(pdev, "failed to set 64 bit dma mask, ret = %d\n", ret);
>> +        return ret;
>> +    }
>> +
>> +    pci_set_master(pdev);
>> +
>> +    ret = hisi_ptt_register_irq(hisi_ptt);
>> +    if (ret)
>> +        return ret;
>> +
>> +    ret = hisi_ptt_init_ctrls(hisi_ptt);
>> +    if (ret) {
>> +        pci_err(pdev, "failed to init controls, ret = %d\n", ret);
>> +        return ret;
>> +    }
>> +
>> +    ret = hisi_ptt_register_pmu(hisi_ptt);
>> +    if (ret) {
>> +        pci_err(pdev, "failed to register PMU device, ret = %d", ret);
>> +        return ret;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +static void hisi_ptt_remove(struct pci_dev *pdev)
>> +{
>> +    struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>> +
>> +    /*
>> +     * We have to manually unregister the PMU device rather than make it
>> +     * devres managed to keep order that the PMU device's unregistration
>> +     * is prior to the release of DMA buffers. As the DMA buffers are
>> +     * devm allocated when necessary which is after the registration of
>> +     * the PMU device.
>> +     */
> 
> do you really need to mention all this?
> 

I think yes. Otherwise people may ask why not register PMU device in managed
way as well.

https://lore.kernel.org/lkml/20220308102157.00003725@Huawei.com/#t

>> +    perf_pmu_unregister(&hisi_ptt->hisi_ptt_pmu);
>> +}
>> +
[...]
>> +/**
>> + * struct hisi_ptt - per PTT device data
>> + * @trace_ctrl:   the control information of PTT trace
>> + * @hisi_ptt_pmu: the pum device of trace
>> + * @iobase:       base IO address of the device
>> + * @pdev:         pci_dev of this PTT device
>> + * @mutex:        mutex to protect the filter list and serialize the perf process.
>> + * @upper:        the upper BDF range of the PCI devices managed by this PTT device
>> + * @lower:        the lower BDF range of the PCI devices managed by this PTT device
>> + * @port_filters: the filter list of root ports
>> + * @req_filters:  the filter list of requester ID
>> + * @port_mask:    port mask of the managed root ports
>> + */
>> +struct hisi_ptt {
>> +    struct hisi_ptt_trace_ctrl trace_ctrl;
>> +    struct pmu hisi_ptt_pmu;
>> +    void __iomem *iobase;
>> +    struct pci_dev *pdev;
>> +    struct mutex mutex;
>> +    u32 upper;
> 
> upper_bdf would be a better name
> 
> people don't want to always refer to comments to check the meaning
> 

will change to upper_bdf and lower_bdf.

Thanks for the comments.

Regards,
Yicong


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-04-11 17:19   ` John Garry
@ 2022-04-12  7:41     ` Yicong Yang
  2022-04-14 13:00       ` John Garry
  0 siblings, 1 reply; 30+ messages in thread
From: Yicong Yang @ 2022-04-12  7:41 UTC (permalink / raw)
  To: John Garry, Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

On 2022/4/12 1:19, John Garry wrote:
> On 07/04/2022 13:58, Yicong Yang wrote:
>> From: Qi Liu <liuqi115@huawei.com>
>>
>> 'perf record' and 'perf report --dump-raw-trace' supported in this
>> patch.
>>
>> Example usage:
>>
>> Output will contain raw PTT data and its textual representation, such
>> as:
>>
>> 0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
>> ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
>> .
>> . ... HISI PTT data: size 4194304 bytes
>> .  00000000: 00 00 00 00                                 Prefix
>> .  00000004: 08 20 00 60                                 Header DW0
>> .  00000008: ff 02 00 01                                 Header DW1
>> .  0000000c: 20 08 00 00                                 Header DW2
>> .  00000010: 10 e7 44 ab                                 Header DW3
>> .  00000014: 2a a8 1e 01                                 Time
>> .  00000020: 00 00 00 00                                 Prefix
>> .  00000024: 01 00 00 60                                 Header DW0
>> .  00000028: 0f 1e 00 01                                 Header DW1
>> .  0000002c: 04 00 00 00                                 Header DW2
>> .  00000030: 40 00 81 02                                 Header DW3
>> .  00000034: ee 02 00 00                                 Time
>> ....
>>
>> Signed-off-by: Qi Liu <liuqi115@huawei.com>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>>   tools/perf/arch/arm/util/auxtrace.c           |  76 +++++-
>>   tools/perf/arch/arm/util/pmu.c                |   3 +
>>   tools/perf/arch/arm64/util/Build              |   2 +-
>>   tools/perf/arch/arm64/util/hisi_ptt.c         | 195 ++++++++++++++++
>>   tools/perf/util/Build                         |   2 +
>>   tools/perf/util/auxtrace.c                    |   4 +
>>   tools/perf/util/auxtrace.h                    |   1 +
>>   tools/perf/util/hisi-ptt-decoder/Build        |   1 +
>>   .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 ++++++++++++++
>>   .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
>>   tools/perf/util/hisi_ptt.c                    | 218 ++++++++++++++++++
>>   tools/perf/util/hisi_ptt.h                    |  28 +++
>>   12 files changed, 724 insertions(+), 4 deletions(-)
>>   create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
>>   create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
>>   create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
>>   create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
>>   create mode 100644 tools/perf/util/hisi_ptt.c
>>   create mode 100644 tools/perf/util/hisi_ptt.h
>>
>> diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
>> index 5fc6a2a3dbc5..393f5757c039 100644
>> --- a/tools/perf/arch/arm/util/auxtrace.c
>> +++ b/tools/perf/arch/arm/util/auxtrace.c
>> @@ -4,9 +4,11 @@
>>    * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
>>    */
>>   +#include <dirent.h>
>>   #include <stdbool.h>
>>   #include <linux/coresight-pmu.h>
>>   #include <linux/zalloc.h>
>> +#include <api/fs/fs.h>
>>     #include "../../../util/auxtrace.h"
>>   #include "../../../util/debug.h"
>> @@ -14,6 +16,7 @@
>>   #include "../../../util/pmu.h"
>>   #include "cs-etm.h"
>>   #include "arm-spe.h"
>> +#include "hisi_ptt.h"
>>     static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>>   {
>> @@ -50,6 +53,58 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>>       return arm_spe_pmus;
>>   }
>>   +static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
>> +{
>> +    const char *sysfs = sysfs__mountpoint();
>> +    struct perf_pmu **hisi_ptt_pmus = NULL;
>> +    struct dirent *dent;
>> +    char path[PATH_MAX];
>> +    DIR *dir = NULL;
>> +    int idx = 0;
>> +
>> +    snprintf(path, PATH_MAX, "%s" EVENT_SOURCE_DEVICE_PATH, sysfs);
>> +    dir = opendir(path);
>> +    if (!dir) {
>> +        pr_err("can't read directory '%s'\n", EVENT_SOURCE_DEVICE_PATH);
>> +        *err = -EINVAL;
>> +        goto out;
>> +    }
>> +
>> +    while ((dent = readdir(dir))) {
>> +        if (strstr(dent->d_name, HISI_PTT_PMU_NAME))
>> +            (*nr_ptts)++;
>> +    }
>> +
>> +    if (!(*nr_ptts))
>> +        goto out;
>> +
>> +    hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * (*nr_ptts));
>> +    if (!hisi_ptt_pmus) {
>> +        pr_err("hisi_ptt alloc failed\n");
>> +        *err = -ENOMEM;
> 
> using PTR_ERR seems better, if possible
> 

ok will change to that. *err = -ENOMEM is used here to keep consistence with
what spe does.

>> +        goto out;
>> +    }
>> +
>> +    rewinddir(dir);
>> +    while ((dent = readdir(dir))) {
>> +        if (strstr(dent->d_name, HISI_PTT_PMU_NAME) && idx < (*nr_ptts)) {
>> +            hisi_ptt_pmus[idx] = perf_pmu__find(dent->d_name);
>> +            if (hisi_ptt_pmus[idx]) {
>> +                pr_debug2("%s %d: hisi_ptt_pmu %d type %d name %s\n",
> 
> do you really need this?
> 

I asked Qi maybe it's unnecessary, but to keep consistence like arm spe.

>> +                    __func__, __LINE__, idx,
>> +                    hisi_ptt_pmus[idx]->type,
>> +                    hisi_ptt_pmus[idx]->name);
>> +                    idx++;
>> +            }
>> +
>> +        }
>> +    }
>> +
>> +out:
>> +    closedir(dir);
>> +    return hisi_ptt_pmus;
>> +}
>> +
>>   struct auxtrace_record
>>   *auxtrace_record__init(struct evlist *evlist, int *err)
>>   {
>> @@ -57,8 +112,12 @@ struct auxtrace_record
>>       struct evsel *evsel;
>>       bool found_etm = false;
>>       struct perf_pmu *found_spe = NULL;
>> +    struct perf_pmu *found_ptt = NULL;
>>       struct perf_pmu **arm_spe_pmus = NULL;
>> +    struct perf_pmu **hisi_ptt_pmus = NULL;
>> +
>>       int nr_spes = 0;
>> +    int nr_ptts = 0;
>>       int i = 0;
>>         if (!evlist)
>> @@ -66,13 +125,14 @@ struct auxtrace_record
>>         cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
>>       arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
>> +    hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
>>         evlist__for_each_entry(evlist, evsel) {
>>           if (cs_etm_pmu &&
>>               evsel->core.attr.type == cs_etm_pmu->type)
>>               found_etm = true;
>>   -        if (!nr_spes || found_spe)
>> +        if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
>>               continue;
>>             for (i = 0; i < nr_spes; i++) {
>> @@ -81,11 +141,18 @@ struct auxtrace_record
>>                   break;
>>               }
>>           }
>> +
>> +        for (i = 0; i < nr_ptts; i++) {
>> +            if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
>> +                found_ptt = hisi_ptt_pmus[i];
>> +                break;
>> +            }
>> +        }
>>       }
>>       free(arm_spe_pmus);
> 
> do you need to do something similar for hisi ptt?
> 

thanks for catching this. it's missed. will fix.

>>   -    if (found_etm && found_spe) {
>> -        pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
>> +    if (found_etm && found_spe && found_ptt) {
>> +        pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
>>           *err = -EOPNOTSUPP;
>>           return NULL;
>>       }
>> @@ -96,6 +163,9 @@ struct auxtrace_record
>>   #if defined(__aarch64__)
>>       if (found_spe)
>>           return arm_spe_recording_init(err, found_spe);
>> +
>> +    if (found_ptt)
>> +        return hisi_ptt_recording_init(err, found_ptt);
>>   #endif
>>         /*
>> diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
>> index b8b23b9dc598..89a3cedb4557 100644
>> --- a/tools/perf/arch/arm/util/pmu.c
>> +++ b/tools/perf/arch/arm/util/pmu.c
>> @@ -10,6 +10,7 @@
>>   #include <linux/string.h>
>>     #include "arm-spe.h"
>> +#include "hisi_ptt.h"
>>   #include "../../../util/pmu.h"
>>     struct perf_event_attr
>> @@ -22,6 +23,8 @@ struct perf_event_attr
>>   #if defined(__aarch64__)
>>       } else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
>>           return arm_spe_pmu_default_config(pmu);
>> +    } else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
>> +        pmu->selectable = true;
>>   #endif
>>       }
>>   diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
>> index 9fcb4e68add9..8b7fd1dc9f37 100644
>> --- a/tools/perf/arch/arm64/util/Build
>> +++ b/tools/perf/arch/arm64/util/Build
>> @@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
>>   perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
>>                     ../../arm/util/auxtrace.o \
>>                     ../../arm/util/cs-etm.o \
>> -                  arm-spe.o mem-events.o
>> +                  arm-spe.o mem-events.o hisi_ptt.o
> 
> nit: others use '-' in the name
> 

will follow the convention.

> 
> I'll check the rest tomorrow.

Thanks,
Yicong



^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-12  7:36     ` Yicong Yang
@ 2022-04-12  8:39       ` John Garry
  2022-04-12 12:52         ` Yicong Yang
  0 siblings, 1 reply; 30+ messages in thread
From: John Garry @ 2022-04-12  8:39 UTC (permalink / raw)
  To: Yicong Yang, Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> +    struct device *dev = &hisi_ptt->pdev->dev;
>>> +    int i;
>>> +
>>> +    hisi_ptt->trace_ctrl.buf_index = 0;
>>> +
>>> +    /* If the trace buffer has already been allocated, zero it. */
>>
>> I am not sure why this is not called from the probe
>>
> 
> The buffer allocation is done when necessary as driver will probe the device on booting but
> the user may never use it. In this condition it's a waste of memory if we allocate the buffers
> in probe. Currently we'll allocate 16M memory for 4 buffers.
> 

But that's just not how we do things. We setup the driver fully to be 
used in the probe. If the user cannot really afford the memory then 
he/she should not load the driver.

In addition, this driver would be used in a machine which will have 
gigbytes of memory, so I think that the memory mentioned here is 
relatively insignificant.

> So this function is called every time before we start trace. In the first time it will allocate
> the DMA buffers and it the other times it just zero the buffers to clear the data of last time.
> 
>>> +    if (ctrl->trace_buf) {
>>> +        for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
>>> +            memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>>> +        return 0;
>>> +    }
>>> +
>>> +    ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
>>> +                       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
>>
>> sizeof(*ctrl->trace_buf) may be better
>>
> 
> ok.
> 
>>> +    if (!ctrl->trace_buf)
>>> +        return -ENOMEM;
>>> +
>>> +    for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>>> +        ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>>> +                                 &ctrl->trace_buf[i].dma,
>>> +                                 GFP_KERNEL);
>>> +        if (!ctrl->trace_buf[i].addr) {
>>> +            hisi_ptt_free_trace_buf(hisi_ptt);
>>> +            return -ENOMEM;
>>> +        }
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +    writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +    hisi_ptt->trace_ctrl.started = false;
>>> +}
>>> +
>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> +    u32 val;
>>> +    int i;
>>> +
>>> +    /* Check device idle before start trace */
>>> +    if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>> +        pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
>>> +        return -EBUSY;
>>> +    }
>>> +
>>> +    ctrl->started = true;
>>> +
>>> +    /* Reset the DMA before start tracing */
>>> +    val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +    val |= HISI_PTT_TRACE_CTRL_RST;
>>> +    writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> +    hisi_ptt_wait_dma_reset_done(hisi_ptt);
>>> +
>>> +    val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +    val &= ~HISI_PTT_TRACE_CTRL_RST;
>>> +    writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> +    /* Clear the interrupt status */
>>> +    writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>> +    writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>> +
>>> +    /* Configure the trace DMA buffer */
>>
>> I am not sure why this sort of thing is done outside probing
>>

...

>>> +
>>> +    val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>> +    ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>
>> how about put all those arrays in hisi_ptt_trace_valid_config_onehot() and pass some flag to say which array you want to use? Or something like that. Passing the arrays in this fashion is messy
>>
> 
> Since there are 3 configs (type, direction, format) with different available range and setting method (onehot, non-onehot),
> moving the arrays into the valid checking function means we need to recognize the config types (passed by the caller but need
> to know the available value array) and the checking method together. That may make the code more complex than now: 1st picking
> up the right array and judge wich checking method this array applied and 2nd do the checking.
> 
> Currently it's designed to decouple the checking method and the available value array. The hisi_ptt_trace_valid_config{_onehot}()
> won't care about which array to use since caller take responsibilty for this. So perhaps current approach is simple and clear
> enough.


A couple of points:
- hisi_ptt_trace_valid_config_type() only has 1x caller so can make it 
dedicated for that caller
- there is not much code in hisi_ptt_trace_valid_config_onshot(), so ok 
to duplicate if makes overall code look better

So I think dedicated functions make the code simpler, easier to follow, 
and maintain:

static int hisi_ptt_trace_valid_config_dir(u32 val)
{
	int i;
	/*
	 * The supported value of the direction parameter. See hisi_ptt.rst
	 * documentation for more details.
	 */
	static const u32 hisi_ptt_trace_available_direction[] = {
		0,
		1,
		2,
		3,
	};

	for (i = 0; i < ARRAY_SIZE(hisi_ptt_trace_available_direction); i++)
		if (val == hisi_ptt_trace_available_direction[i])
			return 0;

	return -EINVAL;
}

static int hisi_ptt_trace_valid_config_format(u32 val)
{
	int i;
	static const u32 hisi_ptt_trace_availble_format[] = {
		0,	/* 4DW */
		1,	/* 8DW */
	};

	for (i = 0; i < ARRAY_SIZE(hisi_ptt_trace_availble_format); i++)
		if (val == hisi_ptt_trace_availble_format[i])
			return 0;

	return -EINVAL;
}

static int hisi_ptt_trace_valid_config_type(u32 val)
{
	int i;
	/* Different types can be set simultaneously */
	static const u32 hisi_ptt_trace_available_type[] = {
		1,	/* posted_request */
		2,	/* non-posted_request */
		4,	/* completion */
	};

	for (i = 0; i < ARRAY_SIZE(hisi_ptt_trace_available_type); i++)
		val &= ~hisi_ptt_trace_available_type[i];

	if (val)
		return -EINVAL;
	return 0;
}

...

static int hisi_ptt_pmu_event_init(struct perf_event *event)
{

...

	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
	ret = hisi_ptt_trace_valid_config_dir(val);
	if (ret < 0)
		goto out;
	ctrl->direction = val;

	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
	ret = hisi_ptt_trace_valid_config_type(val);
	if (ret < 0)
		goto out;
	ctrl->type = val;

	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
	ret = hisi_ptt_trace_valid_config_format(val);
	if (ret < 0)
		goto out;
	ctrl->format = val;

	...
}
> 
>>
>>> +                         ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>> +    if (ret < 0)
>>> +        goto out;
>>> +    ctrl->direction = val;
>>> +
>>> +    val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>> +    ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>> +                      ARRAY_SIZE(hisi_ptt_trace_available_type));
>>> +    if (ret < 0)
>>> +        goto out;
>>> +    ctrl->type = val;
>>> +
>>> +    val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>> +    ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>> +                         ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>> +    if (ret < 0)
>>> +        goto out;
>>> +    ctrl->format = val;
>>> +
>>> +out:
>>> +    mutex_unlock(&hisi_ptt->mutex);
>>> +    return ret;
>>> +}
>>> +
>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>> +                    int nr_pages, bool overwrite)
>>> +{
>>> +    struct hisi_ptt_pmu_buf *buf;
>>> +    struct page **pagelist;
>>> +    int i;
>>> +
>>> +    if (overwrite) {
>>> +        dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>> +        return NULL;
>>> +    }
>>> +
>>> +    /* If the pages size less than buffers, we cannot start trace */
>>> +    if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
>>> +        return NULL;
>>> +
>>> +    buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>> +    if (!buf)
>>> +        return NULL;
>>> +
>>> +    pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>> +    if (!pagelist) {
>>> +        kfree(buf);
>>> +        return NULL;
>>> +    }
>>> +
>>> +    for (i = 0; i < nr_pages; i++)
>>> +        pagelist[i] = virt_to_page(pages[i]);
>>> +
>>> +    buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>> +    if (!buf->base) {
>>> +        kfree(pagelist);
>>> +        kfree(buf);
>>> +        return NULL;
>>> +    }
>>> +
>>> +    buf->nr_pages = nr_pages;
>>> +    buf->length = nr_pages * PAGE_SIZE;
>>> +    buf->pos = 0;
>>> +
>>> +    kfree(pagelist);
>>> +    return buf;
>>> +}
>>> +
>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>> +{
>>> +    struct hisi_ptt_pmu_buf *buf = aux;
>>> +
>>> +    vunmap(buf->base);
>>> +    kfree(buf);
>>> +}
>>> +
>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>> +{
>>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>> +    struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>> +    struct hw_perf_event *hwc = &event->hw;
>>> +    struct hisi_ptt_pmu_buf *buf;
>>> +    int cpu = event->cpu;
>>> +    int ret;
>>> +
>>> +    hwc->state = 0;
>>> +    mutex_lock(&hisi_ptt->mutex);
>>> +    if (hisi_ptt->trace_ctrl.started) {
>>> +        pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>
>> doesn't perf core guard against this sort of thing?
>>
> 
> Maybe not as tested. The perf core will start the events 1)on the cpus user specified or
> 2)on all the cpus, but the PTT trace is intended to start once on one cpu.
> 
> For the 2) case, the driver will make default cpu to start the trace and block others
> in pmu::add(). For the 1) case we'll met the condition here. So the started status is
> test here to avoid a second start.

if this is a realistic and sensible usecase then it would be nice to 
handle in core perf code at some stage

> 
>>> +        goto stop;
>>> +    }
>>> +
>>> +    if (cpu == -1)
>>> +        cpu = hisi_ptt->trace_ctrl.default_cpu;
>>> +
>>> +    /*
>>> +     * Handle the interrupt on the same cpu which starts the trace to avoid
>>> +     * context mismatch. Otherwise we'll trigger the WARN from the perf
>>> +     * core in event_function_local().
>>> +     */
>>> +    WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>> +                 cpumask_of(cpu)));
>>> +
>>> +    ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>> +    if (ret) {
>>> +        pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>> +        goto stop;
>>> +    }
>>> +
>>> +    buf = perf_aux_output_begin(handle, event);
>>> +    if (!buf) {
>>> +        pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>> +        goto stop;
>>> +    }
>>> +
>>> +    buf->pos = handle->head % buf->length;
>>> +
>>> +    ret = hisi_ptt_trace_start(hisi_ptt);
>>> +    if (ret) {
>>> +        pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>> +        perf_aux_output_end(handle, 0);
>>> +        goto stop;
>>> +    }
>>> +
>>> +    mutex_unlock(&hisi_ptt->mutex);
>>> +    return;
>>> +stop:
>>> +    event->hw.state |= PERF_HES_STOPPED;
>>> +    mutex_unlock(&hisi_ptt->mutex);
>>> +}
>>> +
>>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>>> +{
>>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>> +    struct hw_perf_event *hwc = &event->hw;
>>> +
>>> +    if (hwc->state & PERF_HES_STOPPED)
>>> +        return;
>>> +
>>> +    mutex_lock(&hisi_ptt->mutex);
>>> +    if (hisi_ptt->trace_ctrl.started) {
>>> +        hisi_ptt_trace_end(hisi_ptt);
>>> +        WARN(!hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>>> +        hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>>> +    }
>>> +    mutex_unlock(&hisi_ptt->mutex);
>>> +
>>> +    hwc->state |= PERF_HES_STOPPED;
>>> +    perf_event_update_userpage(event);
>>> +    hwc->state |= PERF_HES_UPTODATE;
>>> +}
>>> +
>>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>>> +{
>>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>> +    struct hw_perf_event *hwc = &event->hw;
>>> +    int cpu = event->cpu;
>>> +
>>> +    /*
>>> +     * Only allow the default cpu to add the event if user doesn't specify
>>> +     * the cpus.
>>> +     */
>>> +    if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>>> +        return 0;
>>> +
>>> +    hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>>> +
>>> +    if (flags & PERF_EF_START) {
>>> +        hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>>> +        if (hwc->state & PERF_HES_STOPPED)
>>> +            return -EINVAL;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>>> +{
>>> +    hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>>> +}
>>> +
>>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +    u16 core_id, sicl_id;
>>> +    char *pmu_name;
>>> +    u32 reg;
>>> +
>>> +    hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>>> +        .module        = THIS_MODULE,
>>> +        .capabilities    = PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>>> +        .task_ctx_nr    = perf_sw_context,
>>> +        .attr_groups    = hisi_ptt_pmu_groups,
>>> +        .event_init    = hisi_ptt_pmu_event_init,
>>> +        .setup_aux    = hisi_ptt_pmu_setup_aux,
>>> +        .free_aux    = hisi_ptt_pmu_free_aux,
>>> +        .start        = hisi_ptt_pmu_start,
>>> +        .stop        = hisi_ptt_pmu_stop,
>>> +        .add        = hisi_ptt_pmu_add,
>>> +        .del        = hisi_ptt_pmu_del,
>>> +    };
>>> +
>>> +    reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>> +    core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>> +    sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>> +
>>> +    pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>>> +                  sicl_id, core_id);
>>> +    if (!pmu_name)
>>> +        return -ENOMEM;
>>> +
>>> +    return perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>>> +}
>>> +
>>> +/*
>>> + * The DMA of PTT trace can only use direct mapping, due to some
>>> + * hardware restriction. Check whether there is an IOMMU or the
>>> + * policy of the IOMMU domain is passthrough, otherwise the trace
>>> + * cannot work.
>>> + *
>>> + * The PTT device is supposed to behind the ARM SMMUv3, which
>>
>> /s/ the ARM SMMUv3/an ARM SMMUv3/
>>
> ok.
>>> + * should have passthrough the device by a quirk.
>>> + */
>>> +static int hisi_ptt_check_iommu_mapping(struct pci_dev *pdev)
>>> +{
>>> +    struct iommu_domain *iommu_domain;
>>> +
>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>> +        return 0;
>>> +
>>> +    return -EOPNOTSUPP;
>>> +}
>>> +
>>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>>> +              const struct pci_device_id *id)
>>> +{
>>> +    struct hisi_ptt *hisi_ptt;
>>> +    int ret;
>>> +
>>> +    ret = hisi_ptt_check_iommu_mapping(pdev);
>>> +    if (ret) {
>>> +        pci_err(pdev, "requires direct DMA mappings\n");
>>> +        return ret;
>>> +    }
>>> +
>>> +    hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>>> +    if (!hisi_ptt)
>>> +        return -ENOMEM;
>>> +
>>> +    mutex_init(&hisi_ptt->mutex);
>>> +    hisi_ptt->pdev = pdev;
>>> +    pci_set_drvdata(pdev, hisi_ptt);
>>> +
>>> +    ret = pcim_enable_device(pdev);
>>> +    if (ret) {
>>> +        pci_err(pdev, "failed to enable device, ret = %d\n", ret);
>>> +        return ret;
>>> +    }
>>> +
>>> +    ret = pcim_iomap_regions(pdev, BIT(2), DRV_NAME);
>>> +    if (ret) {
>>> +        pci_err(pdev, "failed to remap io memory, ret = %d\n", ret);
>>> +        return ret;
>>> +    }
>>> +
>>> +    hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>>> +
>>> +    ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>>> +    if (ret) {
>>> +        pci_err(pdev, "failed to set 64 bit dma mask, ret = %d\n", ret);
>>> +        return ret;
>>> +    }
>>> +
>>> +    pci_set_master(pdev);
>>> +
>>> +    ret = hisi_ptt_register_irq(hisi_ptt);
>>> +    if (ret)
>>> +        return ret;
>>> +
>>> +    ret = hisi_ptt_init_ctrls(hisi_ptt);
>>> +    if (ret) {
>>> +        pci_err(pdev, "failed to init controls, ret = %d\n", ret);
>>> +        return ret;
>>> +    }
>>> +
>>> +    ret = hisi_ptt_register_pmu(hisi_ptt);
>>> +    if (ret) {
>>> +        pci_err(pdev, "failed to register PMU device, ret = %d", ret);
>>> +        return ret;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void hisi_ptt_remove(struct pci_dev *pdev)
>>> +{
>>> +    struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>> +
>>> +    /*
>>> +     * We have to manually unregister the PMU device rather than make it
>>> +     * devres managed to keep order that the PMU device's unregistration
>>> +     * is prior to the release of DMA buffers. As the DMA buffers are
>>> +     * devm allocated when necessary which is after the registration of
>>> +     * the PMU device.
>>> +     */
>>
>> do you really need to mention all this?
>>
> 
> I think yes. Otherwise people may ask why not register PMU device in managed
> way as well.

I expect devres work to be done after hisi_ptt_remove() so I would know 
this...


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-12  8:39       ` John Garry
@ 2022-04-12 12:52         ` Yicong Yang
  0 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-12 12:52 UTC (permalink / raw)
  To: John Garry, Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

On 2022/4/12 16:39, John Garry wrote:
>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +    struct device *dev = &hisi_ptt->pdev->dev;
>>>> +    int i;
>>>> +
>>>> +    hisi_ptt->trace_ctrl.buf_index = 0;
>>>> +
>>>> +    /* If the trace buffer has already been allocated, zero it. */
>>>
>>> I am not sure why this is not called from the probe
>>>
>>
>> The buffer allocation is done when necessary as driver will probe the device on booting but
>> the user may never use it. In this condition it's a waste of memory if we allocate the buffers
>> in probe. Currently we'll allocate 16M memory for 4 buffers.
>>
> 
> But that's just not how we do things. We setup the driver fully to be used in the probe. If the user cannot really afford the memory then he/she should not load the driver.
> 

I think in most time user don't need to manually load the driver because of udev.

> In addition, this driver would be used in a machine which will have gigbytes of memory, so I think that the memory mentioned here is relatively insignificant.
> 

ok, yes it'll be used in the server so the memory will be enough in most time. considering this maybe it's proper to move the buffer allocation and the configuration to the driver probe then.

>> So this function is called every time before we start trace. In the first time it will allocate
>> the DMA buffers and it the other times it just zero the buffers to clear the data of last time.
>>
>>>> +    if (ctrl->trace_buf) {
>>>> +        for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
>>>> +            memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>>>> +        return 0;
>>>> +    }
>>>> +
>>>> +    ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
>>>> +                       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
>>>
>>> sizeof(*ctrl->trace_buf) may be better
>>>
>>
>> ok.
>>
>>>> +    if (!ctrl->trace_buf)
>>>> +        return -ENOMEM;
>>>> +
>>>> +    for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>>>> +        ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>>>> +                                 &ctrl->trace_buf[i].dma,
>>>> +                                 GFP_KERNEL);
>>>> +        if (!ctrl->trace_buf[i].addr) {
>>>> +            hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +            return -ENOMEM;
>>>> +        }
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +    writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +    hisi_ptt->trace_ctrl.started = false;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +    struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +    u32 val;
>>>> +    int i;
>>>> +
>>>> +    /* Check device idle before start trace */
>>>> +    if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>> +        pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
>>>> +        return -EBUSY;
>>>> +    }
>>>> +
>>>> +    ctrl->started = true;
>>>> +
>>>> +    /* Reset the DMA before start tracing */
>>>> +    val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +    val |= HISI_PTT_TRACE_CTRL_RST;
>>>> +    writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +    hisi_ptt_wait_dma_reset_done(hisi_ptt);
>>>> +
>>>> +    val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +    val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>> +    writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +    /* Clear the interrupt status */
>>>> +    writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +    writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>> +
>>>> +    /* Configure the trace DMA buffer */
>>>
>>> I am not sure why this sort of thing is done outside probing
>>>
> 
> ...
> 
>>>> +
>>>> +    val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>>> +    ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>>
>>> how about put all those arrays in hisi_ptt_trace_valid_config_onehot() and pass some flag to say which array you want to use? Or something like that. Passing the arrays in this fashion is messy
>>>
>>
>> Since there are 3 configs (type, direction, format) with different available range and setting method (onehot, non-onehot),
>> moving the arrays into the valid checking function means we need to recognize the config types (passed by the caller but need
>> to know the available value array) and the checking method together. That may make the code more complex than now: 1st picking
>> up the right array and judge wich checking method this array applied and 2nd do the checking.
>>
>> Currently it's designed to decouple the checking method and the available value array. The hisi_ptt_trace_valid_config{_onehot}()
>> won't care about which array to use since caller take responsibilty for this. So perhaps current approach is simple and clear
>> enough.
> 
> 
> A couple of points:
> - hisi_ptt_trace_valid_config_type() only has 1x caller so can make it dedicated for that caller
> - there is not much code in hisi_ptt_trace_valid_config_onshot(), so ok to duplicate if makes overall code look better
> 
> So I think dedicated functions make the code simpler, easier to follow, and maintain:
> 

ok then it won't make it more complex. I thought you mean something like hisi_ptt_trace_config(u32 val, enum CONFIG_TYPE type, bool onehot)
as mentioned as "pass some flag to say which array you want to use". The implementation of that will be more complex. Thanks for the great
snippet!

> static int hisi_ptt_trace_valid_config_dir(u32 val)
> {
>     int i;
>     /*
>      * The supported value of the direction parameter. See hisi_ptt.rst
>      * documentation for more details.
>      */
>     static const u32 hisi_ptt_trace_available_direction[] = {
>         0,
>         1,
>         2,
>         3,
>     };
> 
>     for (i = 0; i < ARRAY_SIZE(hisi_ptt_trace_available_direction); i++)
>         if (val == hisi_ptt_trace_available_direction[i])
>             return 0;
> 
>     return -EINVAL;
> }
> 
> static int hisi_ptt_trace_valid_config_format(u32 val)
> {
>     int i;
>     static const u32 hisi_ptt_trace_availble_format[] = {
>         0,    /* 4DW */
>         1,    /* 8DW */
>     };
> 
>     for (i = 0; i < ARRAY_SIZE(hisi_ptt_trace_availble_format); i++)
>         if (val == hisi_ptt_trace_availble_format[i])
>             return 0;
> 
>     return -EINVAL;
> }
> 
> static int hisi_ptt_trace_valid_config_type(u32 val)
> {
>     int i;
>     /* Different types can be set simultaneously */
>     static const u32 hisi_ptt_trace_available_type[] = {
>         1,    /* posted_request */
>         2,    /* non-posted_request */
>         4,    /* completion */
>     };
> 
>     for (i = 0; i < ARRAY_SIZE(hisi_ptt_trace_available_type); i++)
>         val &= ~hisi_ptt_trace_available_type[i];
> 
>     if (val)
>         return -EINVAL;
>     return 0;
> }
> 
> ...
> 
> static int hisi_ptt_pmu_event_init(struct perf_event *event)
> {
> 
> ...
> 
>     val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>     ret = hisi_ptt_trace_valid_config_dir(val);
>     if (ret < 0)
>         goto out;
>     ctrl->direction = val;
> 
>     val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>     ret = hisi_ptt_trace_valid_config_type(val);
>     if (ret < 0)
>         goto out;
>     ctrl->type = val;
> 
>     val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>     ret = hisi_ptt_trace_valid_config_format(val);
>     if (ret < 0)
>         goto out;
>     ctrl->format = val;
> 
>     ...
> }
>>
>>>
>>>> +                         ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>>> +    if (ret < 0)
>>>> +        goto out;
>>>> +    ctrl->direction = val;
>>>> +
>>>> +    val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>>> +    ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>>> +                      ARRAY_SIZE(hisi_ptt_trace_available_type));
>>>> +    if (ret < 0)
>>>> +        goto out;
>>>> +    ctrl->type = val;
>>>> +
>>>> +    val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>>> +    ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>>> +                         ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>>> +    if (ret < 0)
>>>> +        goto out;
>>>> +    ctrl->format = val;
>>>> +
>>>> +out:
>>>> +    mutex_unlock(&hisi_ptt->mutex);
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>>> +                    int nr_pages, bool overwrite)
>>>> +{
>>>> +    struct hisi_ptt_pmu_buf *buf;
>>>> +    struct page **pagelist;
>>>> +    int i;
>>>> +
>>>> +    if (overwrite) {
>>>> +        dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>>> +        return NULL;
>>>> +    }
>>>> +
>>>> +    /* If the pages size less than buffers, we cannot start trace */
>>>> +    if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
>>>> +        return NULL;
>>>> +
>>>> +    buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>>> +    if (!buf)
>>>> +        return NULL;
>>>> +
>>>> +    pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>>> +    if (!pagelist) {
>>>> +        kfree(buf);
>>>> +        return NULL;
>>>> +    }
>>>> +
>>>> +    for (i = 0; i < nr_pages; i++)
>>>> +        pagelist[i] = virt_to_page(pages[i]);
>>>> +
>>>> +    buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>>> +    if (!buf->base) {
>>>> +        kfree(pagelist);
>>>> +        kfree(buf);
>>>> +        return NULL;
>>>> +    }
>>>> +
>>>> +    buf->nr_pages = nr_pages;
>>>> +    buf->length = nr_pages * PAGE_SIZE;
>>>> +    buf->pos = 0;
>>>> +
>>>> +    kfree(pagelist);
>>>> +    return buf;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>>> +{
>>>> +    struct hisi_ptt_pmu_buf *buf = aux;
>>>> +
>>>> +    vunmap(buf->base);
>>>> +    kfree(buf);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>>> +{
>>>> +    struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +    struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>> +    struct hw_perf_event *hwc = &event->hw;
>>>> +    struct hisi_ptt_pmu_buf *buf;
>>>> +    int cpu = event->cpu;
>>>> +    int ret;
>>>> +
>>>> +    hwc->state = 0;
>>>> +    mutex_lock(&hisi_ptt->mutex);
>>>> +    if (hisi_ptt->trace_ctrl.started) {
>>>> +        pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>>
>>> doesn't perf core guard against this sort of thing?
>>>
>>
>> Maybe not as tested. The perf core will start the events 1)on the cpus user specified or
>> 2)on all the cpus, but the PTT trace is intended to start once on one cpu.
>>
>> For the 2) case, the driver will make default cpu to start the trace and block others
>> in pmu::add(). For the 1) case we'll met the condition here. So the started status is
>> test here to avoid a second start.
> 
> if this is a realistic and sensible usecase then it would be nice to handle in core perf code at some stage
> 

I think at least PTT trace meet this condition but seems we're the only uncore tracing PMU now, so
maybe it's still not common.

>>
>>>> +        goto stop;
>>>> +    }
>>>> +
>>>> +    if (cpu == -1)
>>>> +        cpu = hisi_ptt->trace_ctrl.default_cpu;
>>>> +
>>>> +    /*
>>>> +     * Handle the interrupt on the same cpu which starts the trace to avoid
>>>> +     * context mismatch. Otherwise we'll trigger the WARN from the perf
>>>> +     * core in event_function_local().
>>>> +     */
>>>> +    WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>> +                 cpumask_of(cpu)));
>>>> +
>>>> +    ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>>> +    if (ret) {
>>>> +        pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>>> +        goto stop;
>>>> +    }
>>>> +
>>>> +    buf = perf_aux_output_begin(handle, event);
>>>> +    if (!buf) {
>>>> +        pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>>> +        goto stop;
>>>> +    }
>>>> +
>>>> +    buf->pos = handle->head % buf->length;
>>>> +
>>>> +    ret = hisi_ptt_trace_start(hisi_ptt);
>>>> +    if (ret) {
>>>> +        pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>>> +        perf_aux_output_end(handle, 0);
>>>> +        goto stop;
>>>> +    }
>>>> +
>>>> +    mutex_unlock(&hisi_ptt->mutex);
>>>> +    return;
>>>> +stop:
>>>> +    event->hw.state |= PERF_HES_STOPPED;
>>>> +    mutex_unlock(&hisi_ptt->mutex);
>>>> +}
[...]
>>>> +
>>>> +static void hisi_ptt_remove(struct pci_dev *pdev)
>>>> +{
>>>> +    struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>>> +
>>>> +    /*
>>>> +     * We have to manually unregister the PMU device rather than make it
>>>> +     * devres managed to keep order that the PMU device's unregistration
>>>> +     * is prior to the release of DMA buffers. As the DMA buffers are
>>>> +     * devm allocated when necessary which is after the registration of
>>>> +     * the PMU device.
>>>> +     */
>>>
>>> do you really need to mention all this?
>>>
>>
>> I think yes. Otherwise people may ask why not register PMU device in managed
>> way as well.
> 
> I expect devres work to be done after hisi_ptt_remove() so I would know this...
> 

yes, but the problem is that the buffers are devm allocated outside probe so the
order is not obvious and maybe a bit hard to understand.

But as suggested to allocate the buffers in the probe stage we can get rid of it
and make PMU registration devres managed as well.

Thanks,
Yicong



^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-04-12  7:41     ` Yicong Yang
@ 2022-04-14 13:00       ` John Garry
  0 siblings, 0 replies; 30+ messages in thread
From: John Garry @ 2022-04-14 13:00 UTC (permalink / raw)
  To: Yicong Yang, Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

On 12/04/2022 08:41, Yicong Yang wrote:
>>> +    hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * (*nr_ptts));
>>> +    if (!hisi_ptt_pmus) {
>>> +        pr_err("hisi_ptt alloc failed\n");
>>> +        *err = -ENOMEM;
>> using PTR_ERR seems better, if possible
>>
> ok will change to that. *err = -ENOMEM is used here to keep consistence with
> what spe does.
> 

Ah, I see that we are contrained by the interface of 
auxtrace_record_init() to pass err as a pointer, so I suppose the code 
in this patch is ok to fit into that.

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device
  2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
                   ` (6 preceding siblings ...)
  2022-04-07 12:58 ` [PATCH v7 7/7] MAINTAINERS: Add maintainer for HiSilicon PTT driver Yicong Yang
@ 2022-04-14 13:20 ` Yicong Yang
  7 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-04-14 13:20 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, Yicong Yang,
	gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu

Hi Bjorn,

Since it's a device about tuning and analyzing PCIe link in your realm and you've given
helpful comments in RFC and v1 version, looking forward to see your opnion on this
driver as the user interface has changed to perf. Also to confirm that the hotplug problem
mentioned in RFC[1] has been resolved in a proper way.

[1] https://lore.kernel.org/linux-pci/20200710230913.GA90375@bjorn-Precision-5520/
Thanks.

On 2022/4/7 20:58, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
> integrated Endpoint (RCiEP) device, providing the capability
> to dynamically monitor and tune the PCIe traffic (tune),
> and trace the TLP headers (trace).
> 
> PTT tune is designed for monitoring and adjusting PCIe link parameters.
> We provide several parameters of the PCIe link. Through the driver,
> user can adjust the value of certain parameter to affect the PCIe link
> for the purpose of enhancing the performance in certian situation.
> 
> PTT trace is designed for dumping the TLP headers to the memory, which
> can be used to analyze the transactions and usage condition of the PCIe
> Link. Users can choose filters to trace headers, by either requester
> ID, or those downstream of a set of Root Ports on the same core of the
> PTT device. It's also supported to trace the headers of certain type and
> of certain direction.
> 
> The driver registers a PMU device for each PTT device. The trace can
> be used through `perf record` and the traced headers can be decoded
> by `perf report`. The perf command support for the device is also
> added in this patchset. The tune can be used through the sysfs
> attributes of related PMU device. See the documentation for the
> detailed usage.
> 
> Change since v6:
> - Fix W=1 errors reported by lkp test, thanks
> 
> Change since v5:
> - Squash the PMU patch into PATCH 2 suggested by John
> - refine the commit message of PATCH 1 and some comments
> Link: https://lore.kernel.org/lkml/20220308084930.5142-1-yangyicong@hisilicon.com/
> 
> Change since v4:
> Address the comments from Jonathan, John and Ma Ca, thanks.
> - Use devm* also for allocating the DMA buffers
> - Remove the IRQ handler stub in Patch 2
> - Make functions waiting for hardware state return boolean
> - Manual remove the PMU device as it should be removed first
> - Modifier the orders in probe and removal to make them matched well
> - Make available {directions,type,format} array const and non-global
> - Using the right filter list in filters show and well protect the
>   list with mutex
> - Record the trace status with a boolean @started rather than enum
> - Optimize the process of finding the PTT devices of the perf-tool
> Link: https://lore.kernel.org/linux-pci/20220221084307.33712-1-yangyicong@hisilicon.com/
> 
> Change since v3:
> Address the comments from Jonathan and John, thanks.
> - drop members in the common struct which can be get on the fly
> - reduce buffer struct and organize the buffers with array instead of list
> - reduce the DMA reset wait time to avoid long time busy loop
> - split the available_filters sysfs attribute into two files, for root port
>   and requester respectively. Update the documentation accordingly
> - make IOMMU mapping check earlier in probe to avoid race condition. Also
>   make IOMMU quirk patch prior to driver in the series
> - Cleanups and typos fixes from John and Jonathan
> Link: https://lore.kernel.org/linux-pci/20220124131118.17887-1-yangyicong@hisilicon.com/
> 
> Change since v2:
> - address the comments from Mathieu, thanks.
>   - rename the directory to ptt to match the function of the device
>   - spinoff the declarations to a separate header
>   - split the trace function to several patches
>   - some other comments.
> - make default smmu domain type of PTT device to identity
>   Drop the RMR as it's not recommended and use an iommu_def_domain_type
>   quirk to passthrough the device DMA as suggested by Robin. 
> Link: https://lore.kernel.org/linux-pci/20211116090625.53702-1-yangyicong@hisilicon.com/
> 
> Change since v1:
> - switch the user interface of trace to perf from debugfs
> - switch the user interface of tune to sysfs from debugfs
> - add perf tool support to start trace and decode the trace data
> - address the comments of documentation from Bjorn
> - add RMR[1] support of the device as trace works in RMR mode or
>   direct DMA mode. RMR support is achieved by common APIs rather
>   than the APIs implemented in [1].
> Link: https://lore.kernel.org/lkml/1618654631-42454-1-git-send-email-yangyicong@hisilicon.com/
> [1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/
> 
> Qi Liu (1):
>   perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
> 
> Yicong Yang (6):
>   iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to
>     identity
>   hwtracing: Add trace function support for HiSilicon PCIe Tune and
>     Trace device
>   hisi_ptt: Add support for dynamically updating the filter list
>   hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace
>     device
>   docs: Add HiSilicon PTT device driver documentation
>   MAINTAINERS: Add maintainer for HiSilicon PTT driver
> 
>  Documentation/trace/hisi-ptt.rst              |  303 +++++
>  MAINTAINERS                                   |    7 +
>  drivers/Makefile                              |    1 +
>  drivers/hwtracing/Kconfig                     |    2 +
>  drivers/hwtracing/ptt/Kconfig                 |   12 +
>  drivers/hwtracing/ptt/Makefile                |    2 +
>  drivers/hwtracing/ptt/hisi_ptt.c              | 1161 +++++++++++++++++
>  drivers/hwtracing/ptt/hisi_ptt.h              |  220 ++++
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |   16 +
>  tools/perf/arch/arm/util/auxtrace.c           |   76 +-
>  tools/perf/arch/arm/util/pmu.c                |    3 +
>  tools/perf/arch/arm64/util/Build              |    2 +-
>  tools/perf/arch/arm64/util/hisi_ptt.c         |  195 +++
>  tools/perf/util/Build                         |    2 +
>  tools/perf/util/auxtrace.c                    |    4 +
>  tools/perf/util/auxtrace.h                    |    1 +
>  tools/perf/util/hisi-ptt-decoder/Build        |    1 +
>  .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   |  170 +++
>  .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |   28 +
>  tools/perf/util/hisi_ptt.c                    |  218 ++++
>  tools/perf/util/hisi_ptt.h                    |   28 +
>  21 files changed, 2448 insertions(+), 4 deletions(-)
>  create mode 100644 Documentation/trace/hisi-ptt.rst
>  create mode 100644 drivers/hwtracing/ptt/Kconfig
>  create mode 100644 drivers/hwtracing/ptt/Makefile
>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>  create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
>  create mode 100644 tools/perf/util/hisi_ptt.c
>  create mode 100644 tools/perf/util/hisi_ptt.h
> 

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-07 12:58 ` [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device Yicong Yang
  2022-04-11 17:02   ` John Garry
@ 2022-04-29 16:00   ` Leo Yan
  2022-05-05 12:13     ` Yicong Yang
  2022-05-10  9:46   ` James Clark
  2 siblings, 1 reply; 30+ messages in thread
From: Leo Yan @ 2022-04-29 16:00 UTC (permalink / raw)
  To: Yicong Yang
  Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu, zhangshaokun, liuqi115, linuxarm,
	prime.zeng

On Thu, Apr 07, 2022 at 08:58:36PM +0800, Yicong Yang via iommu wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
> tune the PCIe traffic, and trace the TLP headers.
> 
> Add the driver for the device to enable the trace function. Register PMU
> device of PTT trace, then users can use trace through perf command. The
> driver makes use of perf AUX trace and support following events to
> configure the trace:
> 
> - filter: select Root port or Endpoint to trace
> - type: select the type of traced TLP headers
> - direction: select the direction of traced TLP headers
> - format: select the data format of the traced TLP headers
> 
> This patch adds the driver part of PTT trace. The perf command support of
> PTT trace is added in the following patch.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>  drivers/Makefile                 |   1 +
>  drivers/hwtracing/Kconfig        |   2 +
>  drivers/hwtracing/ptt/Kconfig    |  12 +
>  drivers/hwtracing/ptt/Makefile   |   2 +
>  drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
>  drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
>  6 files changed, 1057 insertions(+)
>  create mode 100644 drivers/hwtracing/ptt/Kconfig
>  create mode 100644 drivers/hwtracing/ptt/Makefile
>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
> 
> diff --git a/drivers/Makefile b/drivers/Makefile
> index 020780b6b4d2..662d50599467 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>  obj-y				+= hwtracing/intel_th/
>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT)		+= hwtracing/ptt/
>  obj-$(CONFIG_ANDROID)		+= android/
>  obj-$(CONFIG_NVMEM)		+= nvmem/
>  obj-$(CONFIG_FPGA)		+= fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..911ee977103c 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>  
>  source "drivers/hwtracing/intel_th/Kconfig"
>  
> +source "drivers/hwtracing/ptt/Kconfig"
> +
>  endmenu
> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
> new file mode 100644
> index 000000000000..8902a6f27563
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Kconfig
> @@ -0,0 +1,12 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> +	tristate "HiSilicon PCIe Tune and Trace Device"
> +	depends on ARM64 || (COMPILE_TEST && 64BIT)
> +	depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
> +	help
> +	  HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
> +	  device, and it provides support for PCIe traffic tuning and
> +	  tracing TLP headers to the memory.
> +
> +	  This driver can also be built as a module. If so, the module
> +	  will be called hisi_ptt.
> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
> new file mode 100644
> index 000000000000..242b41870380
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
> @@ -0,0 +1,874 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/module.h>
> +#include <linux/sysfs.h>
> +#include <linux/vmalloc.h>
> +
> +#include "hisi_ptt.h"
> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
> +					  val, val & HISI_PTT_TRACE_IDLE,
> +					  HISI_PTT_WAIT_POLL_INTERVAL_US,
> +					  HISI_PTT_WAIT_TRACE_TIMEOUT_US);
> +}
> +
> +static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
> +					  val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
> +					  HISI_PTT_RESET_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	int i;
> +
> +	if (!ctrl->trace_buf)
> +		return;
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
> +		if (ctrl->trace_buf[i].addr)
> +			dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
> +					   ctrl->trace_buf[i].addr,
> +					   ctrl->trace_buf[i].dma);
> +	}
> +
> +	devm_kfree(dev, ctrl->trace_buf);
> +	ctrl->trace_buf = NULL;
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	int i;
> +
> +	hisi_ptt->trace_ctrl.buf_index = 0;
> +
> +	/* If the trace buffer has already been allocated, zero it. */
> +	if (ctrl->trace_buf) {
> +		for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
> +			memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
> +		return 0;
> +	}
> +
> +	ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
> +				       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
> +	if (!ctrl->trace_buf)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
> +		ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
> +							     &ctrl->trace_buf[i].dma,
> +							     GFP_KERNEL);
> +		if (!ctrl->trace_buf[i].addr) {
> +			hisi_ptt_free_trace_buf(hisi_ptt);
> +			return -ENOMEM;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	hisi_ptt->trace_ctrl.started = false;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	u32 val;
> +	int i;
> +
> +	/* Check device idle before start trace */
> +	if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
> +		return -EBUSY;
> +	}
> +
> +	ctrl->started = true;
> +
> +	/* Reset the DMA before start tracing */
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val |= HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	hisi_ptt_wait_dma_reset_done(hisi_ptt);
> +
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/* Clear the interrupt status */
> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> +	/* Configure the trace DMA buffer */
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
> +		writel(lower_32_bits(ctrl->trace_buf[i].dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
> +		writel(upper_32_bits(ctrl->trace_buf[i].dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
> +	}
> +	writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> +	/* Set the trace control register */
> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> +	if (!hisi_ptt->trace_ctrl.is_port)
> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> +	/* Start the Trace */
> +	val |= HISI_PTT_TRACE_CTRL_EN;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct perf_output_handle *handle = &ctrl->handle;
> +	struct perf_event *event = handle->event;
> +	struct hisi_ptt_pmu_buf *buf;
> +	void *addr;
> +
> +	buf = perf_get_aux(handle);
> +	if (!buf || !handle->size)
> +		return -EINVAL;
> +
> +	addr = ctrl->trace_buf[ctrl->buf_index].addr;
> +
> +	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
> +	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);

I am a bit worry buffer usages, at least for below aspects:

The first thing is for memset(), which cleans up the buffer and the
buffer size is 4MiB, this means it will consume much CPU time to
cleanup the buffer, and trace_buf is mapped as non-cacheable, the
performance would be get worse.

The second thing is here it always copies the trace data with size
HISI_PTT_TRACE_BUF_SIZE, usually, the trace module can provide a read
pointer register, so you can know the trace data length based on the
delta value between write and read pointers.

The last thing is the ctrl->trace_buf[] works as bounce buffer, so it
means actually there have an extra copy from bounce buffer to AUX
buffer, is it possible to directly output PTT trace data to AUX buffer?

Sorry if I bring up duplicate questions and before have the simliar
discussion when reviewed the patch.

> +	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
> +
> +	if (stop) {
> +		perf_aux_output_end(handle, buf->pos);
> +	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> +		perf_aux_output_end(handle, buf->pos);
> +
> +		buf = perf_aux_output_begin(handle, event);
> +		if (!buf)
> +			return -EINVAL;
> +
> +		buf->pos = handle->head % buf->length;
> +		if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
> +			perf_aux_output_end(handle, 0);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status, buf_idx;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	buf_idx = ffs(status) - 1;
> +
> +	/* Clear the interrupt status of buffer @buf_idx */
> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> +	/*
> +	 * Update the AUX buffer and cache the current buffer index,
> +	 * as we need to know this and save the data when the trace
> +	 * is ended out of the interrupt handler. End the trace
> +	 * if the updating fails.
> +	 */
> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> +		hisi_ptt_trace_end(hisi_ptt);
> +	else
> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUF_CNT;
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> +		return IRQ_NONE;
> +
> +	return IRQ_WAKE_THREAD;
> +}
> +
> +static void hisi_ptt_irq_free_vectors(void *pdev)
> +{
> +	pci_free_irq_vectors(pdev);
> +}
> +
> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	int ret;
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> +	if (ret < 0) {
> +		pci_err(pdev, "failed to allocate irq vector, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = devm_request_threaded_irq(&pdev->dev,
> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> +					DRV_NAME, hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to request irq %d, ret = %d\n",
> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> +{
> +	struct hisi_ptt_filter_desc *filter;
> +	struct hisi_ptt *hisi_ptt = data;
> +	struct list_head *target_list;
> +
> +	target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
> +		      &hisi_ptt->port_filters : &hisi_ptt->req_filters;
> +
> +	filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> +	if (!filter) {
> +		pci_err(hisi_ptt->pdev, "failed to add filter %s\n", pci_name(pdev));
> +		return -ENOMEM;
> +	}
> +
> +	filter->pdev = pdev;
> +	list_add_tail(&filter->list, target_list);
> +
> +	/* Update the available port mask */
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		hisi_ptt->port_mask |= hisi_ptt_get_filter_val(pdev);
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_release_filters(void *data)
> +{
> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> +	struct hisi_ptt *hisi_ptt = data;
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +}
> +
> +static int hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct pci_bus *bus;
> +	int ret;
> +	u32 reg;
> +
> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> +	/*
> +	 * The device range register provides the information about the
> +	 * root ports which the RCiEP can control and trace. The RCiEP
> +	 * and the root ports it support are on the same PCIe core, with
> +	 * same domain number but maybe different bus number. The device
> +	 * range register will tell us which root ports we can support,
> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> +	 * while Bit[15:0] indicates the lower.
> +	 */
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> +	hisi_ptt->upper = FIELD_GET(HISI_PTT_DEVICE_RANGE_UPPER, reg);
> +	hisi_ptt->lower = FIELD_GET(HISI_PTT_DEVICE_RANGE_LOWER, reg);
> +
> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
> +	if (bus)
> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_release_filters, hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +
> +	return 0;
> +}
> +
> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> +
> +static ssize_t available_root_port_filters_show(struct device *dev,
> +						struct device_attribute *attr,
> +						char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (list_empty(&hisi_ptt->port_filters)) {
> +		pos = sysfs_emit(buf, "\n");
> +		goto out;
> +	}
> +
> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev) |
> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_root_port_filters);
> +
> +static ssize_t available_requester_filters_show(struct device *dev,
> +						struct device_attribute *attr,
> +						char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (list_empty(&hisi_ptt->req_filters)) {
> +		pos = sysfs_emit(buf, "\n");
> +		goto out;
> +	}
> +
> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev));
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_requester_filters);
> +
> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> +PMU_FORMAT_ATTR(type,		"config:24-31");
> +PMU_FORMAT_ATTR(format,		"config:32-35");
> +
> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> +	&format_attr_filter.attr,
> +	&format_attr_direction.attr,
> +	&format_attr_type.attr,
> +	&format_attr_format.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_format_group = {
> +	.name = "format",
> +	.attrs = hisi_ptt_pmu_format_attrs,
> +};
> +
> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> +	&dev_attr_available_root_port_filters.attr,
> +	&dev_attr_available_requester_filters.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> +	.attrs = hisi_ptt_pmu_filter_attrs,
> +};
> +
> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> +	&hisi_ptt_pmu_format_group,
> +	&hisi_ptt_pmu_filter_group,
> +	NULL
> +};
> +
> +/*
> + * Check whether the config is valid or not. Some configs are multi-selectable
> + * and can be set simultaneously, while some are single selectable (onehot).
> + * Use this function to check the non-onehot configs while
> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> + */
> +static int hisi_ptt_trace_valid_config(u32 val, const u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	/* The non-onehot configs cannot be 0. */
> +	if (!val)
> +		return -EINVAL;
> +
> +	/*
> +	 * Walk the available list and clear the valid bits of
> +	 * the config. If there is any resident bit after the
> +	 * walk then the config is invalid.
> +	 */
> +	for (i = 0; i < list_size; i++)
> +		val &= ~available_list[i];
> +
> +	return val ? -EINVAL : 0;
> +}
> +
> +static int hisi_ptt_trace_valid_config_onehot(u32 val, const u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	for (i = 0; i < list_size; i++)
> +		if (val == available_list[i])
> +			return 0;
> +
> +	return -EINVAL;
> +}
> +
> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> +{
> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> +	struct hisi_ptt_filter_desc *filter;
> +	int ret = -EINVAL;
> +
> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> +
> +	/*
> +	 * Port filters are defined as bit mask. For port filters, check
> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> +	 * and whether it's empty or not, otherwise user has specified
> +	 * some unsupported root ports.
> +	 *
> +	 * For Requester ID filters, walk the available filter list to see
> +	 * whether we have one matched.
> +	 */
> +	if (!hisi_ptt->trace_ctrl.is_port) {
> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +			if (val == hisi_ptt_get_filter_val(filter->pdev)) {
> +				ret = 0;
> +				break;
> +			}
> +	} else if (bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> +		ret = 0;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.filter = val;
> +	return 0;
> +}
> +
> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> +{
> +	/*
> +	 * The supported value of the direction parameter. See hisi_ptt.rst
> +	 * documentation for more details.
> +	 */
> +	static const u32 hisi_ptt_trace_available_direction[] = {
> +		0,
> +		1,
> +		2,
> +		3,
> +	};
> +	/* Different types can be set simultaneously */
> +	static const u32 hisi_ptt_trace_available_type[] = {
> +		1,	/* posted_request */
> +		2,	/* non-posted_request */
> +		4,	/* completion */
> +	};
> +	static const u32 hisi_ptt_trace_availble_format[] = {
> +		0,	/* 4DW */
> +		1,	/* 8DW */
> +	};
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	int ret;
> +	u32 val;
> +
> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> +		return -ENOENT;

Does PTT support per-task mode?  Usually PMU not binding to CPU
cannot support per-task mode, if this is the case, please add checking
like below (copied code from arm_smmuv3_pmu.c):

         if (event->cpu < 0) {
                 dev_dbg(dev, "Per-task mode not supported\n");
                 return -EOPNOTSUPP;
         }

> +
> +	mutex_lock(&hisi_ptt->mutex);

Here really need mutex for protection?  I think the question can be
extended as "if the mutex is necessary for the whole driver"?

The reason is the perf core layer has provided the APIs
perf_event_ctx_lock() and perf_event_ctx_unlock() to protect event
context, and perf core layer has the mutex 'event->mmap_mutex' to
protect setup_aux().

Please note, I think init_event() callback is not protected by lock,
so you could see many PMU drivers use init_event() callback to
validate if the event parameters (e.g. if can work as group events
with other hardware events, support stack track, etc).

This means below initialise functions should be moved to add()
or setup_aux() callbacks, since these functions have been protected by
mutex in perf core layer.

> +
> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> +	if (ret < 0)
> +		goto out;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->direction = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->type = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->format = val;
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return ret;
> +}
> +
> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> +				    int nr_pages, bool overwrite)
> +{
> +	struct hisi_ptt_pmu_buf *buf;
> +	struct page **pagelist;
> +	int i;
> +
> +	if (overwrite) {
> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> +		return NULL;
> +	}
> +
> +	/* If the pages size less than buffers, we cannot start trace */
> +	if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
> +		return NULL;
> +
> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> +	if (!buf)
> +		return NULL;
> +
> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> +	if (!pagelist) {
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++)
> +		pagelist[i] = virt_to_page(pages[i]);
> +
> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> +	if (!buf->base) {
> +		kfree(pagelist);
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	buf->nr_pages = nr_pages;
> +	buf->length = nr_pages * PAGE_SIZE;
> +	buf->pos = 0;
> +
> +	kfree(pagelist);
> +	return buf;
> +}
> +
> +static void hisi_ptt_pmu_free_aux(void *aux)
> +{
> +	struct hisi_ptt_pmu_buf *buf = aux;
> +
> +	vunmap(buf->base);
> +	kfree(buf);
> +}
> +
> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct hw_perf_event *hwc = &event->hw;
> +	struct hisi_ptt_pmu_buf *buf;
> +	int cpu = event->cpu;
> +	int ret;
> +
> +	hwc->state = 0;
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.started) {
> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");

Essentialy this is not a PCI driver, it's good to use dev_dbg()
variant in the whole driver.

> +		goto stop;
> +	}
> +
> +	if (cpu == -1)
> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> +
> +	/*
> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> +	 * core in event_function_local().
> +	 */
> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> +				 cpumask_of(cpu)));

Nitpick: Can set the irq affinity in the probe phase?  Seems to me,
the driver has enough info to set irq affinity in the probe phase,
it's not very necessary to defer to set irq affinity at here.

> +
> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> +		goto stop;
> +	}

It's not a good idea to allocate buffer in the start().  The reason is
start() function might be invoked for multiple times in a perf
session.  I understand the PTT driver doesn't support per-thread mode
and snapshot mode, so the start() callback will be not invoked for
multiple times, but you might need to consider if later support snapshot
mode.

I think it's better to move the buffer allocation into setup_aux().

Will continue to review the perf userspace patch.

Thanks,
Leo

> +
> +	buf = perf_aux_output_begin(handle, event);
> +	if (!buf) {
> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> +		goto stop;
> +	}
> +
> +	buf->pos = handle->head % buf->length;
> +
> +	ret = hisi_ptt_trace_start(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> +		perf_aux_output_end(handle, 0);
> +		goto stop;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return;
> +stop:
> +	event->hw.state |= PERF_HES_STOPPED;
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +
> +	if (hwc->state & PERF_HES_STOPPED)
> +		return;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.started) {
> +		hisi_ptt_trace_end(hisi_ptt);
> +		WARN(!hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> +	}
> +	mutex_unlock(&hisi_ptt->mutex);
> +
> +	hwc->state |= PERF_HES_STOPPED;
> +	perf_event_update_userpage(event);
> +	hwc->state |= PERF_HES_UPTODATE;
> +}
> +
> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +	int cpu = event->cpu;
> +
> +	/*
> +	 * Only allow the default cpu to add the event if user doesn't specify
> +	 * the cpus.
> +	 */
> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> +		return 0;
> +
> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +
> +	if (flags & PERF_EF_START) {
> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> +		if (hwc->state & PERF_HES_STOPPED)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> +{
> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> +}
> +
> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> +{
> +	u16 core_id, sicl_id;
> +	char *pmu_name;
> +	u32 reg;
> +
> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> +		.module		= THIS_MODULE,
> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> +		.task_ctx_nr	= perf_sw_context,
> +		.attr_groups	= hisi_ptt_pmu_groups,
> +		.event_init	= hisi_ptt_pmu_event_init,
> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> +		.free_aux	= hisi_ptt_pmu_free_aux,
> +		.start		= hisi_ptt_pmu_start,
> +		.stop		= hisi_ptt_pmu_stop,
> +		.add		= hisi_ptt_pmu_add,
> +		.del		= hisi_ptt_pmu_del,
> +	};
> +
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> +	core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> +	sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> +				  sicl_id, core_id);
> +	if (!pmu_name)
> +		return -ENOMEM;
> +
> +	return perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> +}
> +
> +/*
> + * The DMA of PTT trace can only use direct mapping, due to some
> + * hardware restriction. Check whether there is an IOMMU or the
> + * policy of the IOMMU domain is passthrough, otherwise the trace
> + * cannot work.
> + *
> + * The PTT device is supposed to behind the ARM SMMUv3, which
> + * should have passthrough the device by a quirk.
> + */
> +static int hisi_ptt_check_iommu_mapping(struct pci_dev *pdev)
> +{
> +	struct iommu_domain *iommu_domain;
> +
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	return -EOPNOTSUPP;
> +}
> +
> +static int hisi_ptt_probe(struct pci_dev *pdev,
> +			  const struct pci_device_id *id)
> +{
> +	struct hisi_ptt *hisi_ptt;
> +	int ret;
> +
> +	ret = hisi_ptt_check_iommu_mapping(pdev);
> +	if (ret) {
> +		pci_err(pdev, "requires direct DMA mappings\n");
> +		return ret;
> +	}
> +
> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> +	if (!hisi_ptt)
> +		return -ENOMEM;
> +
> +	mutex_init(&hisi_ptt->mutex);
> +	hisi_ptt->pdev = pdev;
> +	pci_set_drvdata(pdev, hisi_ptt);
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		pci_err(pdev, "failed to enable device, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(2), DRV_NAME);
> +	if (ret) {
> +		pci_err(pdev, "failed to remap io memory, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> +
> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> +	if (ret) {
> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	pci_set_master(pdev);
> +
> +	ret = hisi_ptt_register_irq(hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	ret = hisi_ptt_init_ctrls(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to init controls, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to register PMU device, ret = %d", ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_remove(struct pci_dev *pdev)
> +{
> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> +
> +	/*
> +	 * We have to manually unregister the PMU device rather than make it
> +	 * devres managed to keep order that the PMU device's unregistration
> +	 * is prior to the release of DMA buffers. As the DMA buffers are
> +	 * devm allocated when necessary which is after the registration of
> +	 * the PMU device.
> +	 */
> +	perf_pmu_unregister(&hisi_ptt->hisi_ptt_pmu);
> +}
> +
> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> +	{ }
> +};
> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> +
> +static struct pci_driver hisi_ptt_driver = {
> +	.name = DRV_NAME,
> +	.id_table = hisi_ptt_id_tbl,
> +	.probe = hisi_ptt_probe,
> +	.remove = hisi_ptt_remove,
> +};
> +module_pci_driver(hisi_ptt_driver);
> +
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
> new file mode 100644
> index 000000000000..29b6d6293d6c
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.h
> @@ -0,0 +1,166 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#ifndef _HISI_PTT_H
> +#define _HISI_PTT_H
> +
> +#include <linux/bits.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/perf_event.h>
> +#include <linux/types.h>
> +
> +#define DRV_NAME "hisi_ptt"
> +
> +/*
> + * The definition of the device registers and register fields.
> + */
> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> +#define HISI_PTT_TRACE_CTRL		0x0850
> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS		0x08b0
> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> +#define   HISI_PTT_DEVICE_RANGE_UPPER	GENMASK(31, 16)
> +#define   HISI_PTT_DEVICE_RANGE_LOWER	GENMASK(15, 0)
> +#define HISI_PTT_LOCATION		0x0fe8
> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> +
> +/* Parameters of PTT trace DMA part. */
> +#define HISI_PTT_TRACE_DMA_IRQ			0
> +#define HISI_PTT_TRACE_BUF_CNT			4
> +#define HISI_PTT_TRACE_BUF_SIZE			SZ_4M
> +#define HISI_PTT_TRACE_TOTAL_BUF_SIZE		(HISI_PTT_TRACE_BUF_SIZE * \
> +						 HISI_PTT_TRACE_BUF_CNT)
> +/* Wait time for hardware DMA to reset */
> +#define HISI_PTT_RESET_TIMEOUT_US	10UL
> +#define HISI_PTT_RESET_POLL_INTERVAL_US	1UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TRACE_TIMEOUT_US	100UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	10UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> +
> +/**
> + * struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
> + *                              The detail of the data format is described
> + *                              in the documentation of PTT device.
> + * @dma:   DMA address of this buffer visible to the device
> + * @addr:  virtual address of this buffer visible to the cpu
> + */
> +struct hisi_ptt_dma_buffer {
> +	dma_addr_t dma;
> +	void *addr;
> +};
> +
> +/**
> + * struct hisi_ptt_trace_ctrl - control and status of PTT trace
> + * @trace_buf:   array of the trace buffers for holding the trace data.
> + *               the length will be HISI_PTT_TRACE_BUF_CNT.
> + * @handle:      perf output handle of current trace session
> + * @default_cpu: default cpu to start the trace session
> + * @buf_index:   the index of current using trace buffer
> + * @started:     current trace status, true for started
> + * @is_port:     whether we're tracing root port or not
> + * @direction:   direction of the TLP headers to trace
> + * @filter:      filter value for tracing the TLP headers
> + * @format:      format of the TLP headers to trace
> + * @type:        type of the TLP headers to trace
> + */
> +struct hisi_ptt_trace_ctrl {
> +	struct hisi_ptt_dma_buffer *trace_buf;
> +	struct perf_output_handle handle;
> +	int default_cpu;
> +	u32 buf_index;
> +	bool started;
> +	bool is_port;
> +	u32 direction:2;
> +	u32 filter:16;
> +	u32 format:1;
> +	u32 type:4;
> +};
> +
> +/**
> + * struct hisi_ptt_filter_desc - descriptor of the PTT trace filter
> + * @list: entry of this descriptor in the filter list
> + * @pdev: pci_dev related to this filter
> + */
> +struct hisi_ptt_filter_desc {
> +	struct list_head list;
> +	struct pci_dev *pdev;
> +};
> +
> +
> +/**
> + * struct hisi_ptt_pmu_buf - descriptor of the AUX buffer of PTT trace
> + * @length:   size of the AUX buffer
> + * @nr_pages: number of pages of the AUX buffer
> + * @base:     start address of AUX buffer
> + * @pos:      position in the AUX buffer to commit traced data
> + */
> +struct hisi_ptt_pmu_buf {
> +	size_t length;
> +	int nr_pages;
> +	void *base;
> +	long pos;
> +};
> +
> +/**
> + * struct hisi_ptt - per PTT device data
> + * @trace_ctrl:   the control information of PTT trace
> + * @hisi_ptt_pmu: the pum device of trace
> + * @iobase:       base IO address of the device
> + * @pdev:         pci_dev of this PTT device
> + * @mutex:        mutex to protect the filter list and serialize the perf process.
> + * @upper:        the upper BDF range of the PCI devices managed by this PTT device
> + * @lower:        the lower BDF range of the PCI devices managed by this PTT device
> + * @port_filters: the filter list of root ports
> + * @req_filters:  the filter list of requester ID
> + * @port_mask:    port mask of the managed root ports
> + */
> +struct hisi_ptt {
> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> +	struct pmu hisi_ptt_pmu;
> +	void __iomem *iobase;
> +	struct pci_dev *pdev;
> +	struct mutex mutex;
> +	u32 upper;
> +	u32 lower;
> +
> +	/*
> +	 * The trace TLP headers can either be filtered by certain
> +	 * root port, or by the requester ID. Organize the filters
> +	 * by @port_filters and @req_filters here. The mask of all
> +	 * the valid ports is also cached for doing sanity check
> +	 * of user input.
> +	 */
> +	struct list_head port_filters;
> +	struct list_head req_filters;
> +	u16 port_mask;
> +};
> +
> +#define to_hisi_ptt(pmu) container_of(pmu, struct hisi_ptt, hisi_ptt_pmu)
> +
> +#endif /* _HISI_PTT_H */
> -- 
> 2.24.0
> 
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-04-07 12:58 ` [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
  2022-04-11 17:19   ` John Garry
@ 2022-04-30  7:35   ` Leo Yan
  2022-05-05 11:30     ` liuqi (BA)
  2022-05-10 10:14   ` James Clark
  2 siblings, 1 reply; 30+ messages in thread
From: Leo Yan @ 2022-04-30  7:35 UTC (permalink / raw)
  To: Yicong Yang
  Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu, zhangshaokun, liuqi115, linuxarm,
	prime.zeng

On Thu, Apr 07, 2022 at 08:58:39PM +0800, Yicong Yang via iommu wrote:
> From: Qi Liu <liuqi115@huawei.com>
> 
> 'perf record' and 'perf report --dump-raw-trace' supported in this
> patch.
> 
> Example usage:
> 
> Output will contain raw PTT data and its textual representation, such
> as:
> 
> 0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
> ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
> .
> . ... HISI PTT data: size 4194304 bytes
> .  00000000: 00 00 00 00                                 Prefix
> .  00000004: 08 20 00 60                                 Header DW0
> .  00000008: ff 02 00 01                                 Header DW1
> .  0000000c: 20 08 00 00                                 Header DW2
> .  00000010: 10 e7 44 ab                                 Header DW3
> .  00000014: 2a a8 1e 01                                 Time
> .  00000020: 00 00 00 00                                 Prefix
> .  00000024: 01 00 00 60                                 Header DW0
> .  00000028: 0f 1e 00 01                                 Header DW1
> .  0000002c: 04 00 00 00                                 Header DW2
> .  00000030: 40 00 81 02                                 Header DW3
> .  00000034: ee 02 00 00                                 Time
> ....
> 
> Signed-off-by: Qi Liu <liuqi115@huawei.com>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  tools/perf/arch/arm/util/auxtrace.c           |  76 +++++-
>  tools/perf/arch/arm/util/pmu.c                |   3 +
>  tools/perf/arch/arm64/util/Build              |   2 +-
>  tools/perf/arch/arm64/util/hisi_ptt.c         | 195 ++++++++++++++++
>  tools/perf/util/Build                         |   2 +
>  tools/perf/util/auxtrace.c                    |   4 +
>  tools/perf/util/auxtrace.h                    |   1 +
>  tools/perf/util/hisi-ptt-decoder/Build        |   1 +
>  .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 ++++++++++++++
>  .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
>  tools/perf/util/hisi_ptt.c                    | 218 ++++++++++++++++++
>  tools/perf/util/hisi_ptt.h                    |  28 +++

It's good to divide the big patch into smaller patches, e.g. one patch
is to add PTT auxtrace (so mainly for perf record), and the second
patch is to add PTT decoder for perf decoding.

>  12 files changed, 724 insertions(+), 4 deletions(-)
>  create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
>  create mode 100644 tools/perf/util/hisi_ptt.c
>  create mode 100644 tools/perf/util/hisi_ptt.h
> 
> diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
> index 5fc6a2a3dbc5..393f5757c039 100644
> --- a/tools/perf/arch/arm/util/auxtrace.c
> +++ b/tools/perf/arch/arm/util/auxtrace.c
> @@ -4,9 +4,11 @@
>   * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
>   */
>  
> +#include <dirent.h>
>  #include <stdbool.h>
>  #include <linux/coresight-pmu.h>
>  #include <linux/zalloc.h>
> +#include <api/fs/fs.h>
>  
>  #include "../../../util/auxtrace.h"
>  #include "../../../util/debug.h"
> @@ -14,6 +16,7 @@
>  #include "../../../util/pmu.h"
>  #include "cs-etm.h"
>  #include "arm-spe.h"
> +#include "hisi_ptt.h"
>  
>  static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>  {
> @@ -50,6 +53,58 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>  	return arm_spe_pmus;
>  }
>  
> +static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
> +{
> +	const char *sysfs = sysfs__mountpoint();
> +	struct perf_pmu **hisi_ptt_pmus = NULL;
> +	struct dirent *dent;
> +	char path[PATH_MAX];
> +	DIR *dir = NULL;
> +	int idx = 0;
> +
> +	snprintf(path, PATH_MAX, "%s" EVENT_SOURCE_DEVICE_PATH, sysfs);
> +	dir = opendir(path);
> +	if (!dir) {
> +		pr_err("can't read directory '%s'\n", EVENT_SOURCE_DEVICE_PATH);
> +		*err = -EINVAL;
> +		goto out;
> +	}
> +
> +	while ((dent = readdir(dir))) {
> +		if (strstr(dent->d_name, HISI_PTT_PMU_NAME))
> +			(*nr_ptts)++;
> +	}
> +
> +	if (!(*nr_ptts))
> +		goto out;
> +
> +	hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * (*nr_ptts));
> +	if (!hisi_ptt_pmus) {
> +		pr_err("hisi_ptt alloc failed\n");
> +		*err = -ENOMEM;
> +		goto out;
> +	}
> +
> +	rewinddir(dir);
> +	while ((dent = readdir(dir))) {
> +		if (strstr(dent->d_name, HISI_PTT_PMU_NAME) && idx < (*nr_ptts)) {
> +			hisi_ptt_pmus[idx] = perf_pmu__find(dent->d_name);
> +			if (hisi_ptt_pmus[idx]) {
> +				pr_debug2("%s %d: hisi_ptt_pmu %d type %d name %s\n",
> +					__func__, __LINE__, idx,
> +					hisi_ptt_pmus[idx]->type,
> +					hisi_ptt_pmus[idx]->name);
> +					idx++;

Indentation for "idx++" is not right.

> +			}
> +

Redundant new line.

> +		}
> +	}
> +
> +out:
> +	closedir(dir);
> +	return hisi_ptt_pmus;
> +}
> +
>  struct auxtrace_record
>  *auxtrace_record__init(struct evlist *evlist, int *err)
>  {
> @@ -57,8 +112,12 @@ struct auxtrace_record
>  	struct evsel *evsel;
>  	bool found_etm = false;
>  	struct perf_pmu *found_spe = NULL;
> +	struct perf_pmu *found_ptt = NULL;
>  	struct perf_pmu **arm_spe_pmus = NULL;
> +	struct perf_pmu **hisi_ptt_pmus = NULL;
> +
>  	int nr_spes = 0;
> +	int nr_ptts = 0;
>  	int i = 0;
>  
>  	if (!evlist)
> @@ -66,13 +125,14 @@ struct auxtrace_record
>  
>  	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
>  	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
> +	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
>  
>  	evlist__for_each_entry(evlist, evsel) {
>  		if (cs_etm_pmu &&
>  		    evsel->core.attr.type == cs_etm_pmu->type)
>  			found_etm = true;
>  
> -		if (!nr_spes || found_spe)
> +		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
>  			continue;
>  
>  		for (i = 0; i < nr_spes; i++) {
> @@ -81,11 +141,18 @@ struct auxtrace_record
>  				break;
>  			}
>  		}
> +
> +		for (i = 0; i < nr_ptts; i++) {
> +			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
> +				found_ptt = hisi_ptt_pmus[i];
> +				break;
> +			}
> +		}
>  	}

For the loop of evlist__for_each_entry, it's good to refactor the
event list iteration, we can refine code like below:

static struct perf_pmu *find_pmu_for_event(struct perf_pmu **pmus,
                                           int pmu_nr, struct evsel *evsel)
{
        int i;
        struct perf_pmu *found = NULL;

        if (!pmus)
                return NULL;

        for (i = 0; i < pmu_nr; i++) {
                if (evsel->core.attr.type == pmus[i]->type) {
                        found = pmus[i];
                        break;
                }
        }

        return found;
}

struct auxtrace_record
*auxtrace_record__init(struct evlist *evlist, int *err)
{
    ...

    evlist__for_each_entry(evlist, evsel) {
            if (cs_etm_pmu && !found_etm)
		    found_etm = find_pmu_for_event(&cs_etm_pmu, 1, evsel);

            if (arm_spe_pmus && !found_spe)
		    found_etm = find_pmu_for_event(arm_spe_pmus, nr_spe, evsel);

            if (hisi_ptt_pmus && !found_ptt)
		    found_ptt = find_pmu_for_event(hisi_ptt_pmus, nr_ptt, evsel);
    }

    ...

Please use a separate patch for the refactoring, and then based on it
you could add PTT PMU related finding code.

}


>  	free(arm_spe_pmus);

Add:

     free(hisi_ptt_pmus);

to avoid memory leaking.

>  
> -	if (found_etm && found_spe) {
> -		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
> +	if (found_etm && found_spe && found_ptt) {

This logic is not right; actually we want the logic is:

        int auxtrace_event_cnt = 0;
        if (found_etm)
                auxtrace_event_cnt++;
        if (found_spe)
                auxtrace_event_cnt++;
        if (found_ptt)
                auxtrace_event_cnt++;

        if (auxtrace_event_cnt > 1) {
                pr_err("Concurrent AUX trace operation isn't supported: found etm %d spe %d ptt %d\n",
                       found_etm, found_spe, found_ptt);
                *err = -EOPNOTSUPP;
                return NULL;
        }

> +		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
>  		*err = -EOPNOTSUPP;
>  		return NULL;
>  	}
> @@ -96,6 +163,9 @@ struct auxtrace_record
>  #if defined(__aarch64__)
>  	if (found_spe)
>  		return arm_spe_recording_init(err, found_spe);
> +
> +	if (found_ptt)
> +		return hisi_ptt_recording_init(err, found_ptt);
>  #endif
>  
>  	/*
> diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
> index b8b23b9dc598..89a3cedb4557 100644
> --- a/tools/perf/arch/arm/util/pmu.c
> +++ b/tools/perf/arch/arm/util/pmu.c
> @@ -10,6 +10,7 @@
>  #include <linux/string.h>
>  
>  #include "arm-spe.h"
> +#include "hisi_ptt.h"
>  #include "../../../util/pmu.h"
>  
>  struct perf_event_attr
> @@ -22,6 +23,8 @@ struct perf_event_attr
>  #if defined(__aarch64__)
>  	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
>  		return arm_spe_pmu_default_config(pmu);
> +	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
> +		pmu->selectable = true;
>  #endif
>  	}
>  
> diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
> index 9fcb4e68add9..8b7fd1dc9f37 100644
> --- a/tools/perf/arch/arm64/util/Build
> +++ b/tools/perf/arch/arm64/util/Build
> @@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
>  perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
>  			      ../../arm/util/auxtrace.o \
>  			      ../../arm/util/cs-etm.o \
> -			      arm-spe.o mem-events.o
> +			      arm-spe.o mem-events.o hisi_ptt.o
> diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
> new file mode 100644
> index 000000000000..de3a3523ffd5
> --- /dev/null
> +++ b/tools/perf/arch/arm64/util/hisi_ptt.c
> @@ -0,0 +1,195 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/types.h>
> +#include <linux/bitops.h>
> +#include <linux/log2.h>
> +#include <linux/zalloc.h>
> +#include <time.h>
> +
> +#include <internal/lib.h> // page_size
> +#include "../../../util/auxtrace.h"
> +#include "../../../util/cpumap.h"
> +#include "../../../util/debug.h"
> +#include "../../../util/event.h"
> +#include "../../../util/evlist.h"
> +#include "../../../util/evsel.h"
> +#include "../../../util/hisi_ptt.h"
> +#include "../../../util/pmu.h"
> +#include "../../../util/record.h"
> +#include "../../../util/session.h"
> +#include "../../../util/tsc.h"
> +
> +#define DEFAULT_PAGE_SIZE 1024

It's a bit confusion to define PAGE_SIZE as 1024.  Here can simply
define:

#define KiB(x) ((x) * 1024)

> +#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
> +#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)


> +
> +struct hisi_ptt_recording {
> +	struct auxtrace_record	itr;
> +	struct perf_pmu *hisi_ptt_pmu;
> +	struct evlist *evlist;
> +};
> +
> +static size_t
> +hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
> +			struct evlist *evlist __maybe_unused)
> +{
> +	return HISI_PTT_AUXTRACE_PRIV_SIZE;
> +}
> +
> +static int hisi_ptt_info_fill(struct auxtrace_record *itr,
> +			      struct perf_session *session,
> +			      struct perf_record_auxtrace_info *auxtrace_info,
> +			      size_t priv_size)
> +{
> +	struct hisi_ptt_recording *pttr =
> +			container_of(itr, struct hisi_ptt_recording, itr);
> +	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
> +
> +	if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
> +		return -EINVAL;
> +
> +	if (!session->evlist->core.nr_mmaps)
> +		return -EINVAL;
> +
> +	auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
> +	auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
> +{
> +	bool privileged = perf_event_paranoid_check(-1);
> +
> +	if (!opts->full_auxtrace)
> +		return 0;
> +
> +	if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
> +		if (privileged) {
> +			opts->auxtrace_mmap_pages = MiB(16) / page_size;
> +		} else {
> +			opts->auxtrace_mmap_pages = KiB(128) / page_size;
> +			if (opts->mmap_pages == UINT_MAX)
> +				opts->mmap_pages = KiB(256) / page_size;
> +		}
> +	}
> +
> +	/* Validate auxtrace_mmap_pages */
> +	if (opts->auxtrace_mmap_pages) {
> +		size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
> +		size_t min_sz = KiB(8);
> +
> +		if (sz < min_sz || !is_power_of_2(sz)) {
> +			pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
> +			       min_sz / DEFAULT_PAGE_SIZE);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_recording_options(struct auxtrace_record *itr,
> +				      struct evlist *evlist,
> +				      struct record_opts *opts)
> +{
> +	struct hisi_ptt_recording *pttr =
> +			container_of(itr, struct hisi_ptt_recording, itr);
> +	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
> +	struct perf_cpu_map *cpus = evlist->core.cpus;
> +	struct evsel *evsel, *hisi_ptt_evsel = NULL;
> +	struct evsel *tracking_evsel;
> +	int err;
> +
> +	pttr->evlist = evlist;
> +	evlist__for_each_entry(evlist, evsel) {
> +		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
> +			if (hisi_ptt_evsel) {
> +				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
> +				return -EINVAL;
> +			}
> +			evsel->core.attr.freq = 0;
> +			evsel->core.attr.sample_period = 1;
> +			hisi_ptt_evsel = evsel;
> +			opts->full_auxtrace = true;
> +		}
> +	}
> +
> +	err = hisi_ptt_set_auxtrace_mmap_page(opts);
> +	if (err)
> +		return err;
> +	/*
> +	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
> +	 * must come first.
> +	 */
> +	evlist__to_front(evlist, hisi_ptt_evsel);
> +
> +	if (!perf_cpu_map__empty(cpus)) {
> +		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
> +		evsel__set_sample_bit(hisi_ptt_evsel, CPU);

It needs to set CPU bit in sample type for per-cpu mmaps.  IIUC, PTT
is only used for system wide tracing?  If so, you don't need set CPU
bit.

To be honest, I am also confused set the sample bits.  Actually, there
have two different sample types for AUX trace, one is here set the
sample type for AUX event, and in PTT decoding code it needs to set
sample type for synthesized samples.

> +	}
> +
> +	/* Add dummy event to keep tracking */
> +	err = parse_events(evlist, "dummy:u", NULL);
> +	if (err)
> +		return err;
> +
> +	tracking_evsel = evlist__last(evlist);
> +	evlist__set_tracking_event(evlist, tracking_evsel);
> +
> +	tracking_evsel->core.attr.freq = 0;
> +	tracking_evsel->core.attr.sample_period = 1;
> +
> +	if (!perf_cpu_map__empty(cpus))
> +		evsel__set_sample_bit(tracking_evsel, TIME);
> +
> +	return 0;
> +}
> +
> +static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
> +{
> +	return rdtsc();
> +}
>
> +
> +static void hisi_ptt_recording_free(struct auxtrace_record *itr)
> +{
> +	struct hisi_ptt_recording *pttr =
> +			container_of(itr, struct hisi_ptt_recording, itr);
> +
> +	free(pttr);
> +}
> +
> +struct auxtrace_record *hisi_ptt_recording_init(int *err,
> +						struct perf_pmu *hisi_ptt_pmu)
> +{
> +	struct hisi_ptt_recording *pttr;
> +
> +	if (!hisi_ptt_pmu) {
> +		*err = -ENODEV;
> +		return NULL;
> +	}
> +
> +	pttr = zalloc(sizeof(struct hisi_ptt_recording));
> +	if (!pttr) {
> +		*err = -ENOMEM;
> +		return NULL;
> +	}
> +
> +	pttr->hisi_ptt_pmu = hisi_ptt_pmu;
> +	pttr->itr.pmu = hisi_ptt_pmu;
> +	pttr->itr.recording_options = hisi_ptt_recording_options;
> +	pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
> +	pttr->itr.info_fill = hisi_ptt_info_fill;
> +	pttr->itr.free = hisi_ptt_recording_free;
> +	pttr->itr.reference = hisi_ptt_reference;
> +	pttr->itr.read_finish = auxtrace_record__read_finish;
> +	pttr->itr.alignment = 0;
> +
> +	*err = 0;
> +	return &pttr->itr;
> +}
> diff --git a/tools/perf/util/Build b/tools/perf/util/Build
> index 9a7209a99e16..95d2d76c52da 100644
> --- a/tools/perf/util/Build
> +++ b/tools/perf/util/Build
> @@ -116,6 +116,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
>  perf-$(CONFIG_AUXTRACE) += intel-bts.o
>  perf-$(CONFIG_AUXTRACE) += arm-spe.o
>  perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
> +perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
> +perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
>  perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
>  
>  ifdef CONFIG_LIBOPENCSD
> diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
> index df1c5bbbaa0d..f4841ebaf314 100644
> --- a/tools/perf/util/auxtrace.c
> +++ b/tools/perf/util/auxtrace.c
> @@ -51,6 +51,7 @@
>  #include "intel-pt.h"
>  #include "intel-bts.h"
>  #include "arm-spe.h"
> +#include "hisi_ptt.h"
>  #include "s390-cpumsf.h"
>  #include "util/mmap.h"
>  
> @@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
>  	case PERF_AUXTRACE_S390_CPUMSF:
>  		err = s390_cpumsf_process_auxtrace_info(event, session);
>  		break;
> +	case PERF_AUXTRACE_HISI_PTT:
> +		err = hisi_ptt_process_auxtrace_info(event, session);
> +		break;
>  	case PERF_AUXTRACE_UNKNOWN:
>  	default:
>  		return -EINVAL;
> diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
> index dc38b6f57232..3858c5752ead 100644
> --- a/tools/perf/util/auxtrace.h
> +++ b/tools/perf/util/auxtrace.h
> @@ -48,6 +48,7 @@ enum auxtrace_type {
>  	PERF_AUXTRACE_CS_ETM,
>  	PERF_AUXTRACE_ARM_SPE,
>  	PERF_AUXTRACE_S390_CPUMSF,
> +	PERF_AUXTRACE_HISI_PTT,
>  };
>  
>  enum itrace_period_type {
> diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
> new file mode 100644
> index 000000000000..db3db8b75033
> --- /dev/null
> +++ b/tools/perf/util/hisi-ptt-decoder/Build
> @@ -0,0 +1 @@
> +perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
> diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
> new file mode 100644
> index 000000000000..3fb7a6949209
> --- /dev/null
> +++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
> @@ -0,0 +1,170 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#include <stdlib.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <endian.h>
> +#include <byteswap.h>
> +#include <linux/bitops.h>
> +#include <stdarg.h>
> +
> +#include "../color.h"
> +#include "hisi-ptt-pkt-decoder.h"
> +
> +/*
> + * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
> + * used to distinguish the data format.
> + * 8DW format is like:
> + *   bits [                 31:11                 ][       10:0       ]
> + *        |---------------------------------------|-------------------|
> + *    DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
> + *    DW1 [                       Prefix                              ]
> + *    DW2 [                     Header DW0                            ]
> + *    DW3 [                     Header DW1                            ]
> + *    DW4 [                     Header DW2                            ]
> + *    DW5 [                     Header DW3                            ]
> + *    DW6 [                   Reserved (0x0)                          ]
> + *    DW7 [                        Time                               ]
> + *
> + * 4DW format is like:
> + *   bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
> + *        |-----|---------|---|---|---|---|-------------|-------------|
> + *    DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
> + *    DW1 [                     Header DW1                            ]
> + *    DW2 [                     Header DW2                            ]
> + *    DW3 [                     Header DW3                            ]
> + */
> +
> +enum hisi_ptt_8dw_pkt_field_type {
> +	HISI_PTT_8DW_PREFIX,
> +	HISI_PTT_8DW_HEAD0,
> +	HISI_PTT_8DW_HEAD1,
> +	HISI_PTT_8DW_HEAD2,
> +	HISI_PTT_8DW_HEAD3,
> +	HISI_PTT_8DW_TIME,
> +	HISI_PTT_8DW_TYPE_MAX
> +};
> +
> +enum hisi_ptt_4dw_pkt_field_type {
> +	HISI_PTT_4DW_HEAD1,
> +	HISI_PTT_4DW_HEAD2,
> +	HISI_PTT_4DW_HEAD3,
> +	HISI_PTT_4DW_TYPE_MAX
> +};
> +
> +static const char * const hisi_ptt_8dw_pkt_field_name[] = {
> +	[HISI_PTT_8DW_PREFIX]	= "Prefix",
> +	[HISI_PTT_8DW_HEAD0]	= "Header DW0",
> +	[HISI_PTT_8DW_HEAD1]	= "Header DW1",
> +	[HISI_PTT_8DW_HEAD2]	= "Header DW2",
> +	[HISI_PTT_8DW_HEAD3]	= "Header DW3",
> +	[HISI_PTT_8DW_TIME]	= "Time",
> +};
> +
> +static const char * const hisi_ptt_4dw_pkt_field_name[] = {
> +	[HISI_PTT_4DW_HEAD1]	= "Header DW1",
> +	[HISI_PTT_4DW_HEAD2]	= "Header DW2",
> +	[HISI_PTT_4DW_HEAD3]	= "Header DW3",
> +};
> +
> +/* offset of each member is determined by format of 8dw packet. */
> +static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
> +	[HISI_PTT_8DW_PREFIX]	= 4,
> +	[HISI_PTT_8DW_HEAD0]	= 4,
> +	[HISI_PTT_8DW_HEAD1]	= 4,
> +	[HISI_PTT_8DW_HEAD2]	= 4,
> +	[HISI_PTT_8DW_HEAD3]	= 4,
> +	[HISI_PTT_8DW_TIME]	= 8,
> +};

You could define a structure hisi_ptt_8dw (just like hisi_ptt_4dw) so
that can avoid to define this field offset structure.

It's unusal to define data structure for offset, another way is to
define macros for offset values.

> +
> +union hisi_ptt_4dw {
> +	struct {
> +		uint32_t format : 2;
> +		uint32_t type : 5;
> +		uint32_t t9 : 1;
> +		uint32_t t8 : 1;
> +		uint32_t th : 1;
> +		uint32_t so : 1;
> +		uint32_t len : 10;
> +		uint32_t time : 11;
> +	};
> +	uint32_t value;
> +};
> +
> +static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
> +{
> +	const char *color = PERF_COLOR_BLUE;
> +	int field_len = sizeof(uint32_t);
> +	int i;
> +
> +	printf(".");
> +	color_fprintf(stdout, color, "  %08x: ", *pos);
> +	for (i = 0; i < field_len; i++)
> +		color_fprintf(stdout, color, "%02x ", buf[i]);
> +	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
> +		color_fprintf(stdout, color, "   ");
> +	color_fprintf(stdout, color, "  %s\n", desc);
> +	*pos += field_len;
> +}
> +
> +static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
> +{
> +	int i;
> +
> +	for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
> +		buf += hisi_ptt_8dw_pkt_field_offset[i];
> +		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
> +	}
> +
> +	return HISI_PTT_8DW_PKT_SIZE;
> +}
> +
> +static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
> +{
> +	const char *color = PERF_COLOR_BLUE;
> +	int field_len = sizeof(uint32_t);
> +	union hisi_ptt_4dw dw0;
> +	int i;
> +
> +	dw0.value = *(uint32_t *)buf;
> +	printf(".");
> +	color_fprintf(stdout, color, "  %08x: ", *pos);
> +	for (i = 0; i < field_len; i++)
> +		color_fprintf(stdout, color, "%02x ", buf[i]);
> +	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
> +		color_fprintf(stdout, color, "   ");
> +
> +	color_fprintf(stdout, color,
> +		      "  %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
> +		      "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
> +		      "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
> +		      dw0.len, "Time", dw0.time);
> +
> +	*pos += field_len;
> +}
> +
> +static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
> +{
> +	int i;
> +
> +	hisi_ptt_4dw_print_dw0(buf, &pos);
> +
> +	for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
> +		buf += sizeof(uint32_t);
> +		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
> +	}
> +
> +	return HISI_PTT_4DW_PKT_SIZE;
> +}
> +
> +int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
> +{
> +	if (type == HISI_PTT_8DW_PKT)
> +		return hisi_ptt_8dw_kpt_desc(buf, pos);
> +
> +	return hisi_ptt_4dw_kpt_desc(buf, pos);
> +}
> diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
> new file mode 100644
> index 000000000000..20c51f230cac
> --- /dev/null
> +++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
> +#define INCLUDE__HISI_PTT_PKT_DECODER_H__
> +
> +#include <stddef.h>
> +#include <stdint.h>
> +
> +#define HISI_PTT_PKT_DESC_MAX		256
> +#define HISI_PTT_NEED_MORE_BYTES	-1
> +#define HISI_PTT_8DW_CHECK_MASK		GENMASK(31, 11)
> +#define HISI_PTT_IS_8DW_PKT		GENMASK(31, 11)
> +#define HISI_PTT_8DW_PKT_SIZE		32
> +#define HISI_PTT_4DW_PKT_SIZE		16
> +#define HISI_PTT_MAX_SPACE_LEN		10
> +
> +enum hisi_ptt_pkt_type {
> +	HISI_PTT_4DW_PKT,
> +	HISI_PTT_8DW_PKT,
> +};
> +
> +int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
> +
> +#endif
> diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
> new file mode 100644
> index 000000000000..e2934214763c
> --- /dev/null
> +++ b/tools/perf/util/hisi_ptt.c
> @@ -0,0 +1,218 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#include <byteswap.h>
> +#include <endian.h>
> +#include <errno.h>
> +#include <inttypes.h>
> +#include <linux/bitops.h>
> +#include <linux/kernel.h>
> +#include <linux/log2.h>
> +#include <linux/types.h>
> +#include <linux/zalloc.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +#include "auxtrace.h"
> +#include "color.h"
> +#include "debug.h"
> +#include "evlist.h"
> +#include "evsel.h"
> +#include "hisi_ptt.h"
> +#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
> +#include "machine.h"
> +#include "session.h"
> +#include "symbol.h"
> +#include "tool.h"
> +#include "util/synthetic-events.h"
> +
> +struct hisi_ptt {
> +	struct auxtrace auxtrace;
> +	struct auxtrace_queues queues;
> +	u32 auxtrace_type;
> +	struct perf_session *session;
> +	struct machine *machine;
> +	u32 pmu_type;
> +};
> +
> +struct hisi_ptt_queue {
> +	struct hisi_ptt *ptt;
> +	struct auxtrace_buffer *buffer;
> +};
> +
> +static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
> +{
> +	uint32_t head = *(uint32_t *)buf;
> +
> +	if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
> +		return HISI_PTT_8DW_PKT;
> +
> +	return HISI_PTT_4DW_PKT;
> +}
> +
> +static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
> +			  unsigned char *buf, size_t len)
> +{
> +	const char *color = PERF_COLOR_BLUE;
> +	enum hisi_ptt_pkt_type type;
> +	size_t pos = 0;
> +	int pkt_len;
> +
> +	color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
> +		      len);
> +
> +	type = hisi_ptt_check_packet_type(buf);
> +	while (len) {

It's good to use condition "while (len > 0)".

> +		pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
> +		if (!pkt_len)
> +			color_fprintf(stdout, color, " Bad packet!\n");
> +
> +		pos += pkt_len;
> +		buf += pkt_len;
> +		len -= pkt_len;
> +	}
> +}
> +
> +static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
> +				size_t len)
> +{
> +	printf(".\n");
> +
> +	hisi_ptt_dump(ptt, buf, len);
> +}
> +
> +static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
> +				  union perf_event *event __maybe_unused,
> +				  struct perf_sample *sample __maybe_unused,
> +				  struct perf_tool *tool __maybe_unused)
> +{
> +	return 0;
> +}
> +
> +static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
> +					   union perf_event *event,
> +					   struct perf_tool *tool __maybe_unused)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
> +					    auxtrace);
> +	struct auxtrace_buffer *buffer;
> +	off_t data_offset;
> +	int fd = perf_data__fd(session->data);
> +	int err;
> +
> +	if (perf_data__is_pipe(session->data)) {
> +		data_offset = 0;
> +	} else {
> +		data_offset = lseek(fd, 0, SEEK_CUR);
> +		if (data_offset == -1)
> +			return -errno;
> +	}
> +
> +	err = auxtrace_queues__add_event(&ptt->queues, session, event,
> +					 data_offset, &buffer);
> +	if (err)
> +		return err;
> +
> +	if (dump_trace) {
> +		if (auxtrace_buffer__get_data(buffer, fd)) {
> +			hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
> +			auxtrace_buffer__put_data(buffer);
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
> +			  struct perf_tool *tool __maybe_unused)
> +{
> +	return 0;
> +}
> +
> +static void hisi_ptt_free_events(struct perf_session *session)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
> +					    auxtrace);
> +	struct auxtrace_queues *queues = &ptt->queues;
> +	unsigned int i;
> +
> +	for (i = 0; i < queues->nr_queues; i++) {
> +		free(queues->queue_array[i].priv);
> +		queues->queue_array[i].priv = NULL;
> +	}
> +	auxtrace_queues__free(queues);
> +}
> +
> +static void hisi_ptt_free(struct perf_session *session)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
> +					    auxtrace);
> +
> +	hisi_ptt_free_events(session);
> +	session->auxtrace = NULL;
> +	free(ptt);
> +}
> +
> +static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
> +				       struct evsel *evsel)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
> +
> +	return evsel->core.attr.type == ptt->pmu_type;
> +}
> +
> +static const char * const hisi_ptt_info_fmts[] = {
> +	[HISI_PTT_PMU_TYPE]		= "  PMU Type           %" PRId64 "\n",
> +};
> +
> +static void hisi_ptt_print_info(__u64 *arr)
> +{
> +	if (!dump_trace)
> +		return;
> +
> +	fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
> +}
> +
> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
> +				   struct perf_session *session)
> +{
> +	struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
> +	struct hisi_ptt *ptt;
> +	int err;
> +
> +	if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
> +				sizeof(struct perf_record_auxtrace_info))
> +		return -EINVAL;
> +
> +	ptt = zalloc(sizeof(struct hisi_ptt));
> +	if (!ptt)
> +		return -ENOMEM;
> +
> +	err = auxtrace_queues__init(&ptt->queues);
> +	if (err)
> +		goto err_free;
> +
> +	ptt->session = session;
> +	ptt->machine = &session->machines.host; /* No kvm support */
> +	ptt->auxtrace_type = auxtrace_info->type;
> +	ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
> +
> +	ptt->auxtrace.process_event = hisi_ptt_process_event;
> +	ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
> +	ptt->auxtrace.flush_events = hisi_ptt_flush;
> +	ptt->auxtrace.free_events = hisi_ptt_free_events;
> +	ptt->auxtrace.free = hisi_ptt_free;
> +	ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
> +	session->auxtrace = &ptt->auxtrace;
> +
> +	hisi_ptt_print_info(&auxtrace_info->priv[0]);
> +
> +	return 0;
> +
> +err_free:
> +	free(ptt);
> +	return err;
> +}
> diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
> new file mode 100644
> index 000000000000..c0b6cbde1221
> --- /dev/null
> +++ b/tools/perf/util/hisi_ptt.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#ifndef INCLUDE__PERF_HISI_PTT_H__
> +#define INCLUDE__PERF_HISI_PTT_H__
> +
> +#define HISI_PTT_PMU_NAME "hisi_ptt"
> +enum {
> +	HISI_PTT_PMU_TYPE,
> +	HISI_PTT_PER_CPU_MMAPS,

HISI_PTT_PER_CPU_MMAPS is not used, so can remove it.

Thanks,
Leo

> +	HISI_PTT_AUXTRACE_PRIV_MAX,
> +};
> +
> +#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
> +union perf_event;
> +struct perf_session;
> +struct perf_pmu;
> +
> +struct auxtrace_record *hisi_ptt_recording_init(int *err,
> +						struct perf_pmu *hisi_ptt_pmu);
> +
> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
> +				   struct perf_session *session);
> +
> +#endif
> -- 
> 2.24.0
> 
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-04-30  7:35   ` Leo Yan
@ 2022-05-05 11:30     ` liuqi (BA)
  0 siblings, 0 replies; 30+ messages in thread
From: liuqi (BA) @ 2022-05-05 11:30 UTC (permalink / raw)
  To: Leo Yan, Yicong Yang
  Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu, zhangshaokun, linuxarm, prime.zeng


Hi Leo,

Thanks for your review, some replies below.

On 2022/4/30 15:35, Leo Yan wrote:
> On Thu, Apr 07, 2022 at 08:58:39PM +0800, Yicong Yang via iommu wrote:
>> From: Qi Liu <liuqi115@huawei.com>
>>
>> 'perf record' and 'perf report --dump-raw-trace' supported in this
>> patch.
>>
>> Example usage:
>>
>> Output will contain raw PTT data and its textual representation, such
>> as:
>>
>> 0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
>> ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
>> .
>> . ... HISI PTT data: size 4194304 bytes
>> .  00000000: 00 00 00 00                                 Prefix
>> .  00000004: 08 20 00 60                                 Header DW0
>> .  00000008: ff 02 00 01                                 Header DW1
>> .  0000000c: 20 08 00 00                                 Header DW2
>> .  00000010: 10 e7 44 ab                                 Header DW3
>> .  00000014: 2a a8 1e 01                                 Time
>> .  00000020: 00 00 00 00                                 Prefix
>> .  00000024: 01 00 00 60                                 Header DW0
>> .  00000028: 0f 1e 00 01                                 Header DW1
>> .  0000002c: 04 00 00 00                                 Header DW2
>> .  00000030: 40 00 81 02                                 Header DW3
>> .  00000034: ee 02 00 00                                 Time
>> ....
>>
>> Signed-off-by: Qi Liu <liuqi115@huawei.com>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>>   tools/perf/arch/arm/util/auxtrace.c           |  76 +++++-
>>   tools/perf/arch/arm/util/pmu.c                |   3 +
>>   tools/perf/arch/arm64/util/Build              |   2 +-
>>   tools/perf/arch/arm64/util/hisi_ptt.c         | 195 ++++++++++++++++
>>   tools/perf/util/Build                         |   2 +
>>   tools/perf/util/auxtrace.c                    |   4 +
>>   tools/perf/util/auxtrace.h                    |   1 +
>>   tools/perf/util/hisi-ptt-decoder/Build        |   1 +
>>   .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 ++++++++++++++
>>   .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
>>   tools/perf/util/hisi_ptt.c                    | 218 ++++++++++++++++++
>>   tools/perf/util/hisi_ptt.h                    |  28 +++
> 
> It's good to divide the big patch into smaller patches, e.g. one patch
> is to add PTT auxtrace (so mainly for perf record), and the second
> patch is to add PTT decoder for perf decoding.
> 
got it, I'll do this, thanks.

>>   12 files changed, 724 insertions(+), 4 deletions(-)
>>   create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
>>   create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
>>   create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
>>   create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
>>   create mode 100644 tools/perf/util/hisi_ptt.c
>>   create mode 100644 tools/perf/util/hisi_ptt.h
>>
>> diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
>> index 5fc6a2a3dbc5..393f5757c039 100644
>> --- a/tools/perf/arch/arm/util/auxtrace.c
>> +++ b/tools/perf/arch/arm/util/auxtrace.c
>> @@ -4,9 +4,11 @@
>>    * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
>>    */
[...]
>> +
>> +	rewinddir(dir);
>> +	while ((dent = readdir(dir))) {
>> +		if (strstr(dent->d_name, HISI_PTT_PMU_NAME) && idx < (*nr_ptts)) {
>> +			hisi_ptt_pmus[idx] = perf_pmu__find(dent->d_name);
>> +			if (hisi_ptt_pmus[idx]) {
>> +				pr_debug2("%s %d: hisi_ptt_pmu %d type %d name %s\n",
>> +					__func__, __LINE__, idx,
>> +					hisi_ptt_pmus[idx]->type,
>> +					hisi_ptt_pmus[idx]->name);
>> +					idx++;
> 
> Indentation for "idx++" is not right.
will fix this, thanks.

> 
>> +			}
>> +
> 
> Redundant new line.
will fix this, thanks.
> 
>> +		}
>> +	}
>> +
>> +out:
>> +	closedir(dir);
>> +	return hisi_ptt_pmus;
>> +}
>> +
>>   struct auxtrace_record
>>   *auxtrace_record__init(struct evlist *evlist, int *err)
>>   {
>> @@ -57,8 +112,12 @@ struct auxtrace_record
>>   	struct evsel *evsel;
>>   	bool found_etm = false;
>>   	struct perf_pmu *found_spe = NULL;
>> +	struct perf_pmu *found_ptt = NULL;
>>   	struct perf_pmu **arm_spe_pmus = NULL;
>> +	struct perf_pmu **hisi_ptt_pmus = NULL;
>> +
>>   	int nr_spes = 0;
>> +	int nr_ptts = 0;
>>   	int i = 0;
>>   
>>   	if (!evlist)
>> @@ -66,13 +125,14 @@ struct auxtrace_record
>>   
>>   	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
>>   	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
>> +	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
>>   
>>   	evlist__for_each_entry(evlist, evsel) {
>>   		if (cs_etm_pmu &&
>>   		    evsel->core.attr.type == cs_etm_pmu->type)
>>   			found_etm = true;
>>   
>> -		if (!nr_spes || found_spe)
>> +		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
>>   			continue;
>>   
>>   		for (i = 0; i < nr_spes; i++) {
>> @@ -81,11 +141,18 @@ struct auxtrace_record
>>   				break;
>>   			}
>>   		}
>> +
>> +		for (i = 0; i < nr_ptts; i++) {
>> +			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
>> +				found_ptt = hisi_ptt_pmus[i];
>> +				break;
>> +			}
>> +		}
>>   	}
> 
> For the loop of evlist__for_each_entry, it's good to refactor the
> event list iteration, we can refine code like below:
> 
> static struct perf_pmu *find_pmu_for_event(struct perf_pmu **pmus,
>                                             int pmu_nr, struct evsel *evsel)
> {
>          int i;
>          struct perf_pmu *found = NULL;
> 
>          if (!pmus)
>                  return NULL;
> 
>          for (i = 0; i < pmu_nr; i++) {
>                  if (evsel->core.attr.type == pmus[i]->type) {
>                          found = pmus[i];
>                          break;
>                  }
>          }
> 
>          return found;
> }
> 
> struct auxtrace_record
> *auxtrace_record__init(struct evlist *evlist, int *err)
> {
>      ...
> 
>      evlist__for_each_entry(evlist, evsel) {
>              if (cs_etm_pmu && !found_etm)
> 		    found_etm = find_pmu_for_event(&cs_etm_pmu, 1, evsel);
> 
>              if (arm_spe_pmus && !found_spe)
> 		    found_etm = find_pmu_for_event(arm_spe_pmus, nr_spe, evsel);
> 
>              if (hisi_ptt_pmus && !found_ptt)
> 		    found_ptt = find_pmu_for_event(hisi_ptt_pmus, nr_ptt, evsel);
>      }
> 
>      ...
> 
> Please use a separate patch for the refactoring, and then based on it
> you could add PTT PMU related finding code.
> 
> }
> 

got it, will do this refactoring in next version, thanks.
> 
>>   	free(arm_spe_pmus);
> 
> Add:
> 
>       free(hisi_ptt_pmus);
> 
> to avoid memory leaking.
> 

will fix it, thanks.

>>   
>> -	if (found_etm && found_spe) {
>> -		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
>> +	if (found_etm && found_spe && found_ptt) {
> 
> This logic is not right; actually we want the logic is:
> 
>          int auxtrace_event_cnt = 0;
>          if (found_etm)
>                  auxtrace_event_cnt++;
>          if (found_spe)
>                  auxtrace_event_cnt++;
>          if (found_ptt)
>                  auxtrace_event_cnt++;
> 
>          if (auxtrace_event_cnt > 1) {
>                  pr_err("Concurrent AUX trace operation isn't supported: found etm %d spe %d ptt %d\n",
>                         found_etm, found_spe, found_ptt);
>                  *err = -EOPNOTSUPP;
>                  return NULL;
>          }
> 
got it, will fix it, thanks.
>> +		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
>>   		*err = -EOPNOTSUPP;
>>   		return NULL;
>>   	}
>> @@ -96,6 +163,9 @@ struct auxtrace_record
>>   #if defined(__aarch64__)
>>   	if (found_spe)
>>   		return arm_spe_recording_init(err, found_spe);
>> +
>> +	if (found_ptt)
>> +		return hisi_ptt_recording_init(err, found_ptt);
>>   #endif
>>   
>>   	/*
>> diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
>> index b8b23b9dc598..89a3cedb4557 100644
>> --- a/tools/perf/arch/arm/util/pmu.c
>> +++ b/tools/perf/arch/arm/util/pmu.c
>> @@ -10,6 +10,7 @@
>>   #include <linux/string.h>
>>   
>>   #include "arm-spe.h"
>> +#include "hisi_ptt.h"
>>   #include "../../../util/pmu.h"
>>   
>>   struct perf_event_attr
>> @@ -22,6 +23,8 @@ struct perf_event_attr
>>   #if defined(__aarch64__)
>>   	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
>>   		return arm_spe_pmu_default_config(pmu);
>> +	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
>> +		pmu->selectable = true;
>>   #endif
>>   	}
>>   
>> diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
>> index 9fcb4e68add9..8b7fd1dc9f37 100644
>> --- a/tools/perf/arch/arm64/util/Build
>> +++ b/tools/perf/arch/arm64/util/Build
>> @@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
[...]
>> +
>> +#define DEFAULT_PAGE_SIZE 1024
> 
> It's a bit confusion to define PAGE_SIZE as 1024.  Here can simply
> define:
> 
> #define KiB(x) ((x) * 1024)
> 
>> +#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
>> +#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
> 
> 
ok, I'll change it, thanks.
>> +
>> +struct hisi_ptt_recording {
>> +	struct auxtrace_record	itr;
>> +	struct perf_pmu *hisi_ptt_pmu;
>> +	struct evlist *evlist;
>> +};
>> +
[...]
>> +}
>> +
>> +static int hisi_ptt_recording_options(struct auxtrace_record *itr,
>> +				      struct evlist *evlist,
>> +				      struct record_opts *opts)
>> +{
>> +	struct hisi_ptt_recording *pttr =
>> +			container_of(itr, struct hisi_ptt_recording, itr);
>> +	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
>> +	struct perf_cpu_map *cpus = evlist->core.cpus;
>> +	struct evsel *evsel, *hisi_ptt_evsel = NULL;
>> +	struct evsel *tracking_evsel;
>> +	int err;
>> +
>> +	pttr->evlist = evlist;
>> +	evlist__for_each_entry(evlist, evsel) {
>> +		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
>> +			if (hisi_ptt_evsel) {
>> +				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
>> +				return -EINVAL;
>> +			}
>> +			evsel->core.attr.freq = 0;
>> +			evsel->core.attr.sample_period = 1;
>> +			hisi_ptt_evsel = evsel;
>> +			opts->full_auxtrace = true;
>> +		}
>> +	}
>> +
>> +	err = hisi_ptt_set_auxtrace_mmap_page(opts);
>> +	if (err)
>> +		return err;
>> +	/*
>> +	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
>> +	 * must come first.
>> +	 */
>> +	evlist__to_front(evlist, hisi_ptt_evsel);
>> +
>> +	if (!perf_cpu_map__empty(cpus)) {
>> +		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
>> +		evsel__set_sample_bit(hisi_ptt_evsel, CPU);
> 
> It needs to set CPU bit in sample type for per-cpu mmaps.  IIUC, PTT
> is only used for system wide tracing?  If so, you don't need set CPU
> bit.
>
ok, I'll delete set CPU bit.

> To be honest, I am also confused set the sample bits.  Actually, there
> have two different sample types for AUX trace, one is here set the
> sample type for AUX event, and in PTT decoding code it needs to set
> sample type for synthesized samples.

does "synthesized samples" mean something like spe pmu in perf report? 
such as consolidate multiple pieces of information into one event named 
"llc-miss" "remote-access"?

PTT doesn't need to do this, we just parse message in perf.data and show it.

> 
>> +	}
>> +
>> +	/* Add dummy event to keep tracking */
>> +	err = parse_events(evlist, "dummy:u", NULL);
>> +	if (err)
>> +		return err;
>> +
>> +	tracking_evsel = evlist__last(evlist);
>> +	evlist__set_tracking_event(evlist, tracking_evsel);
>> +
>> +	tracking_evsel->core.attr.freq = 0;
>> +	tracking_evsel->core.attr.sample_period = 1;
>> +
>> +	if (!perf_cpu_map__empty(cpus))
>> +		evsel__set_sample_bit(tracking_evsel, TIME);
>> +
>> +	return 0;

[...]
>> +
>> +enum hisi_ptt_8dw_pkt_field_type {
>> +	HISI_PTT_8DW_PREFIX,
>> +	HISI_PTT_8DW_HEAD0,
>> +	HISI_PTT_8DW_HEAD1,
>> +	HISI_PTT_8DW_HEAD2,
>> +	HISI_PTT_8DW_HEAD3,
>> +	HISI_PTT_8DW_TIME,
>> +	HISI_PTT_8DW_TYPE_MAX
>> +};
>> +
>> +enum hisi_ptt_4dw_pkt_field_type {
>> +	HISI_PTT_4DW_HEAD1,
>> +	HISI_PTT_4DW_HEAD2,
>> +	HISI_PTT_4DW_HEAD3,
>> +	HISI_PTT_4DW_TYPE_MAX
>> +};
>> +
>> +static const char * const hisi_ptt_8dw_pkt_field_name[] = {
>> +	[HISI_PTT_8DW_PREFIX]	= "Prefix",
>> +	[HISI_PTT_8DW_HEAD0]	= "Header DW0",
>> +	[HISI_PTT_8DW_HEAD1]	= "Header DW1",
>> +	[HISI_PTT_8DW_HEAD2]	= "Header DW2",
>> +	[HISI_PTT_8DW_HEAD3]	= "Header DW3",
>> +	[HISI_PTT_8DW_TIME]	= "Time",
>> +};
>> +
>> +static const char * const hisi_ptt_4dw_pkt_field_name[] = {
>> +	[HISI_PTT_4DW_HEAD1]	= "Header DW1",
>> +	[HISI_PTT_4DW_HEAD2]	= "Header DW2",
>> +	[HISI_PTT_4DW_HEAD3]	= "Header DW3",
>> +};
>> +
>> +/* offset of each member is determined by format of 8dw packet. */
>> +static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
>> +	[HISI_PTT_8DW_PREFIX]	= 4,
>> +	[HISI_PTT_8DW_HEAD0]	= 4,
>> +	[HISI_PTT_8DW_HEAD1]	= 4,
>> +	[HISI_PTT_8DW_HEAD2]	= 4,
>> +	[HISI_PTT_8DW_HEAD3]	= 4,
>> +	[HISI_PTT_8DW_TIME]	= 8,
>> +};
> 
> You could define a structure hisi_ptt_8dw (just like hisi_ptt_4dw) so
> that can avoid to define this field offset structure.
> 
> It's unusal to define data structure for offset, another way is to
> define macros for offset values.
> 
got it, I'll do this.
>> +
>> +union hisi_ptt_4dw {
>> +	struct {
>> +		uint32_t format : 2;
>> +		uint32_t type : 5;
>> +		uint32_t t9 : 1;
>> +		uint32_t t8 : 1;
>> +		uint32_t th : 1;
>> +		uint32_t so : 1;
>> +		uint32_t len : 10;
>> +		uint32_t time : 11;
>> +	};
>> +	uint32_t value;
>> +};
>> +
[...]

>> +
>> +static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
>> +			  unsigned char *buf, size_t len)
>> +{
>> +	const char *color = PERF_COLOR_BLUE;
>> +	enum hisi_ptt_pkt_type type;
>> +	size_t pos = 0;
>> +	int pkt_len;
>> +
>> +	color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
>> +		      len);
>> +
>> +	type = hisi_ptt_check_packet_type(buf);
>> +	while (len) {
> 
> It's good to use condition "while (len > 0)".
ok, will change this.

> 
>> +		pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
>> +		if (!pkt_len)
>> +			color_fprintf(stdout, color, " Bad packet!\n");
>> +
>> +		pos += pkt_len;
>> +		buf += pkt_len;
>> +		len -= pkt_len;
>> +	}
>> +}
>> +
[...]

>> diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
>> new file mode 100644
>> index 000000000000..c0b6cbde1221
>> --- /dev/null
>> +++ b/tools/perf/util/hisi_ptt.h
>> @@ -0,0 +1,28 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * HiSilicon PCIe Trace and Tuning (PTT) support
>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>> + */
>> +
>> +#ifndef INCLUDE__PERF_HISI_PTT_H__
>> +#define INCLUDE__PERF_HISI_PTT_H__
>> +
>> +#define HISI_PTT_PMU_NAME "hisi_ptt"
>> +enum {
>> +	HISI_PTT_PMU_TYPE,
>> +	HISI_PTT_PER_CPU_MMAPS,
> 
> HISI_PTT_PER_CPU_MMAPS is not used, so can remove it.
> 
will move this.

Thanks,
Qi
> Thanks,
> Leo
> 
>> +	HISI_PTT_AUXTRACE_PRIV_MAX,
>> +};
>> +
>> +#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
>> +union perf_event;
>> +struct perf_session;
>> +struct perf_pmu;
>> +
>> +struct auxtrace_record *hisi_ptt_recording_init(int *err,
>> +						struct perf_pmu *hisi_ptt_pmu);
>> +
>> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
>> +				   struct perf_session *session);
>> +
>> +#endif
>> -- 
>> 2.24.0
>>
>> _______________________________________________
>> iommu mailing list
>> iommu@lists.linux-foundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
> .
> 

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-29 16:00   ` Leo Yan
@ 2022-05-05 12:13     ` Yicong Yang
  0 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-05-05 12:13 UTC (permalink / raw)
  To: Leo Yan
  Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu, zhangshaokun, liuqi115, linuxarm,
	prime.zeng, Yicong Yang

Hi Leo,

Thanks for the comments. Some questions and replies below.

On 2022/4/30 0:00, Leo Yan wrote:
> On Thu, Apr 07, 2022 at 08:58:36PM +0800, Yicong Yang via iommu wrote:
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
>> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
>> tune the PCIe traffic, and trace the TLP headers.
>>
>> Add the driver for the device to enable the trace function. Register PMU
>> device of PTT trace, then users can use trace through perf command. The
>> driver makes use of perf AUX trace and support following events to
>> configure the trace:
>>
>> - filter: select Root port or Endpoint to trace
>> - type: select the type of traced TLP headers
>> - direction: select the direction of traced TLP headers
>> - format: select the data format of the traced TLP headers
>>
>> This patch adds the driver part of PTT trace. The perf command support of
>> PTT trace is added in the following patch.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
[...]
>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct perf_output_handle *handle = &ctrl->handle;
>> +	struct perf_event *event = handle->event;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	void *addr;
>> +
>> +	buf = perf_get_aux(handle);
>> +	if (!buf || !handle->size)
>> +		return -EINVAL;
>> +
>> +	addr = ctrl->trace_buf[ctrl->buf_index].addr;
>> +
>> +	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
>> +	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);
> 
> I am a bit worry buffer usages, at least for below aspects:
> 
> The first thing is for memset(), which cleans up the buffer and the
> buffer size is 4MiB, this means it will consume much CPU time to
> cleanup the buffer, and trace_buf is mapped as non-cacheable, the
> performance would be get worse.
> 
> The second thing is here it always copies the trace data with size
> HISI_PTT_TRACE_BUF_SIZE, usually, the trace module can provide a read
> pointer register, so you can know the trace data length based on the
> delta value between write and read pointers.
> 
> The last thing is the ctrl->trace_buf[] works as bounce buffer, so it
> means actually there have an extra copy from bounce buffer to AUX
> buffer, is it possible to directly output PTT trace data to AUX buffer?
> 
> Sorry if I bring up duplicate questions and before have the simliar
> discussion when reviewed the patch.
> 

The hardware is designed to use 4 DMA buffers and fill the buffer with the
traced data one by one. When one buffer is full the device will notify the
driver by interrupt and continue to trace the PCIe TLPs in the following
buffer without pausing. If the interrupt status bit of the buffer going to
use is uncleared, the trace will be paused until the corresponding interrupt
status bit is cleared. So the buffer size of 4 MiB is calculated that even
at the max flow the driver can handle the traced data and clear the status
in time, so there won't be a data loss.

For the first thing, the performance is acceptable as we only handle 4MiB at
one time. It's in thread context so we won't block others. The hardware can
keep tracing in the following DMA buffer so handling here won't pause the
trace and will not cause data loss.

For the second thing, this function is called in 2 places: 1) in the IRQ thread
2) the trace is going to stop. In the 1st case the data length will always be
HISI_PTT_TRACE_BUF_SIZE as the interrupt only occurs when one buffer is full.
In the 2nd case we may not have HISI_PTT_TRACE_BUF_SIZE data, the unfilled buffer
is zeroed to be distinguished. We keep committing HISI_PTT_TRACE_BUF_SIZE data
to keep consistence with handling in interrupt and make it simpler here.
(HISI_PTT_TRACE_WR_STS register indicates will buffer is currently used and
the offset in the buffer data is currently writing to)

For the third thing, I'm not sure if we can map the AUX buffer as DMA buffer
and by taht way, considering the case the buffer is full and we need to commit
the AUX buffer and apply a new one, the trace is paused during the procedure and
any TLPs on the link will be missed. But by current approach we won't have
this problem as the tracing is still on even when we're committing the AUX
buffer as the hardware is not directly writing to it.

>> +	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
>> +
>> +	if (stop) {
>> +		perf_aux_output_end(handle, buf->pos);
>> +	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>> +		perf_aux_output_end(handle, buf->pos);
>> +
>> +		buf = perf_aux_output_begin(handle, event);
>> +		if (!buf)
>> +			return -EINVAL;
>> +
>> +		buf->pos = handle->head % buf->length;
>> +		if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
>> +			perf_aux_output_end(handle, 0);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
[...]
>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>> +{
>> +	/*
>> +	 * The supported value of the direction parameter. See hisi_ptt.rst
>> +	 * documentation for more details.
>> +	 */
>> +	static const u32 hisi_ptt_trace_available_direction[] = {
>> +		0,
>> +		1,
>> +		2,
>> +		3,
>> +	};
>> +	/* Different types can be set simultaneously */
>> +	static const u32 hisi_ptt_trace_available_type[] = {
>> +		1,	/* posted_request */
>> +		2,	/* non-posted_request */
>> +		4,	/* completion */
>> +	};
>> +	static const u32 hisi_ptt_trace_availble_format[] = {
>> +		0,	/* 4DW */
>> +		1,	/* 8DW */
>> +	};
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	int ret;
>> +	u32 val;
>> +
>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>> +		return -ENOENT;
> 
> Does PTT support per-task mode?  Usually PMU not binding to CPU
> cannot support per-task mode, if this is the case, please add checking
> like below (copied code from arm_smmuv3_pmu.c):
> 

PTT doesn't support per-task mode. It's not relevant to a certain task.
Will add the check here.

>          if (event->cpu < 0) {
>                  dev_dbg(dev, "Per-task mode not supported\n");
>                  return -EOPNOTSUPP;
>          }
> 
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
> 
> Here really need mutex for protection?  I think the question can be
> extended as "if the mutex is necessary for the whole driver"?
> 
> The reason is the perf core layer has provided the APIs
> perf_event_ctx_lock() and perf_event_ctx_unlock() to protect event
> context, and perf core layer has the mutex 'event->mmap_mutex' to
> protect setup_aux().
> 

But I don't think the same things are protected here. For PTT trace part,
the mutex intends to 1)protect the filter lists and 2) serialize perf
process to enter start stage.

For 1), the filters in initialized in probe stage and can be dynamically
updated (implemented in Patch 3/7). So the mutex is used to protect the
filter from concurrent access by perf and the updating process. (In this
patch it maybe unnecessary as the filter is static).

For 2), it's used below in ->start() to serialize the CPUs entering start().
Since we want only one CPU to really start the trace. Without the lock,
the cases below may happen:

CPU0				CPU1
start()
.				start()
.				//trace_ctrl.started=false, go on
.				.
hisi_ptt_trace_start()		.
  trace_ctrl.started=true	.
				hisi_ptt_trace_start()

There're some PMU drivers using driver_struct::pmu_lock to serialize the
access to hardware registers and I think the usage is similar (referred to
qcom_l2_pmu.c and arm_dsu_pmu.c).

One thing here as mentioned that I used the mutex for the whole driver,
maybe it's better to split the lock of protecting the filter list and for
the perf process. And the lock for serializing the perf process should
be spinlock as ->start() is in atomic context, I think.

> Please note, I think init_event() callback is not protected by lock,
> so you could see many PMU drivers use init_event() callback to
> validate if the event parameters (e.g. if can work as group events
> with other hardware events, support stack track, etc).
> 
> This means below initialise functions should be moved to add()
> or setup_aux() callbacks, since these functions have been protected by
> mutex in perf core layer.
> 

Thanks for the note. Then the handling here is not proper. Will check and
validate the configs here in ->event_init() and move the real initialization
to ->add()/->setup_aux().

>> +
>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>> +	if (ret < 0)
>> +		goto out;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->direction = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->type = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->format = val;
>> +
>> +out:
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return ret;
>> +}
>> +
>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>> +				    int nr_pages, bool overwrite)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	struct page **pagelist;
>> +	int i;
>> +
>> +	if (overwrite) {
>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>> +		return NULL;
>> +	}
>> +
>> +	/* If the pages size less than buffers, we cannot start trace */
>> +	if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
>> +		return NULL;
>> +
>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>> +	if (!buf)
>> +		return NULL;
>> +
>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>> +	if (!pagelist) {
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	for (i = 0; i < nr_pages; i++)
>> +		pagelist[i] = virt_to_page(pages[i]);
>> +
>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>> +	if (!buf->base) {
>> +		kfree(pagelist);
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	buf->nr_pages = nr_pages;
>> +	buf->length = nr_pages * PAGE_SIZE;
>> +	buf->pos = 0;
>> +
>> +	kfree(pagelist);
>> +	return buf;
>> +}
>> +
>> +static void hisi_ptt_pmu_free_aux(void *aux)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf = aux;
>> +
>> +	vunmap(buf->base);
>> +	kfree(buf);
>> +}
>> +
>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +	struct hw_perf_event *hwc = &event->hw;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	int cpu = event->cpu;
>> +	int ret;
>> +
>> +	hwc->state = 0;
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	if (hisi_ptt->trace_ctrl.started) {
>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> 
> Essentialy this is not a PCI driver, it's good to use dev_dbg()
> variant in the whole driver.
> 

It does make sense. will change to use dev_dbg().

>> +		goto stop;
>> +	}
>> +
>> +	if (cpu == -1)
>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>> +
>> +	/*
>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>> +	 * core in event_function_local().
>> +	 */
>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +				 cpumask_of(cpu)));
> 
> Nitpick: Can set the irq affinity in the probe phase?  Seems to me,
> the driver has enough info to set irq affinity in the probe phase,
> it's not very necessary to defer to set irq affinity at here.
> 

We incline to use the cpu on the same node with PTT device. But here intends to handle
the case that user has specified a certain cpu with `perf record -C xxx` and we have to
bind the IRQ to that cpu. In which case we cannot handle in the probe stage.

>> +
>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>> +	if (ret) {
>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>> +		goto stop;
>> +	}
> 
> It's not a good idea to allocate buffer in the start().  The reason is
> start() function might be invoked for multiple times in a perf
> session.  I understand the PTT driver doesn't support per-thread mode
> and snapshot mode, so the start() callback will be not invoked for
> multiple times, but you might need to consider if later support snapshot
> mode.
> 
> I think it's better to move the buffer allocation into setup_aux().
> 

Thanks for the suggestion. John has noticed the similar that we don't need to allocate
and configure the buffer every time we start trace. So I'll move the allocation
and configuration of DMA buffers into probe stage, then we only need to do it once
and just clear the buffers every time we start trace.

Thanks,
Yicong



^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-04-07 12:58 ` [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device Yicong Yang
  2022-04-11 17:02   ` John Garry
  2022-04-29 16:00   ` Leo Yan
@ 2022-05-10  9:46   ` James Clark
  2022-05-10 11:18     ` Yicong Yang
  2 siblings, 1 reply; 30+ messages in thread
From: James Clark @ 2022-05-10  9:46 UTC (permalink / raw)
  To: Yicong Yang
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, gregkh, helgaas,
	alexander.shishkin, lorenzo.pieralisi, will, mark.rutland,
	mathieu.poirier, suzuki.poulose, mike.leach, leo.yan,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu



On 07/04/2022 13:58, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
> tune the PCIe traffic, and trace the TLP headers.
> 
> Add the driver for the device to enable the trace function. Register PMU
> device of PTT trace, then users can use trace through perf command. The
> driver makes use of perf AUX trace and support following events to
> configure the trace:
> 
> - filter: select Root port or Endpoint to trace
> - type: select the type of traced TLP headers
> - direction: select the direction of traced TLP headers
> - format: select the data format of the traced TLP headers
> 
> This patch adds the driver part of PTT trace. The perf command support of
> PTT trace is added in the following patch.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>  drivers/Makefile                 |   1 +
>  drivers/hwtracing/Kconfig        |   2 +
>  drivers/hwtracing/ptt/Kconfig    |  12 +
>  drivers/hwtracing/ptt/Makefile   |   2 +
>  drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
>  drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
>  6 files changed, 1057 insertions(+)
>  create mode 100644 drivers/hwtracing/ptt/Kconfig
>  create mode 100644 drivers/hwtracing/ptt/Makefile
>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
> 
> diff --git a/drivers/Makefile b/drivers/Makefile
> index 020780b6b4d2..662d50599467 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>  obj-y				+= hwtracing/intel_th/
>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT)		+= hwtracing/ptt/
>  obj-$(CONFIG_ANDROID)		+= android/
>  obj-$(CONFIG_NVMEM)		+= nvmem/
>  obj-$(CONFIG_FPGA)		+= fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..911ee977103c 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>  
>  source "drivers/hwtracing/intel_th/Kconfig"
>  
> +source "drivers/hwtracing/ptt/Kconfig"
> +
>  endmenu
> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
> new file mode 100644
> index 000000000000..8902a6f27563
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Kconfig
> @@ -0,0 +1,12 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> +	tristate "HiSilicon PCIe Tune and Trace Device"
> +	depends on ARM64 || (COMPILE_TEST && 64BIT)
> +	depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
> +	help
> +	  HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
> +	  device, and it provides support for PCIe traffic tuning and
> +	  tracing TLP headers to the memory.
> +
> +	  This driver can also be built as a module. If so, the module
> +	  will be called hisi_ptt.
> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
> new file mode 100644
> index 000000000000..242b41870380
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
> @@ -0,0 +1,874 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/module.h>
> +#include <linux/sysfs.h>
> +#include <linux/vmalloc.h>
> +
> +#include "hisi_ptt.h"
> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
> +					  val, val & HISI_PTT_TRACE_IDLE,
> +					  HISI_PTT_WAIT_POLL_INTERVAL_US,
> +					  HISI_PTT_WAIT_TRACE_TIMEOUT_US);
> +}
> +
> +static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
> +					  val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
> +					  HISI_PTT_RESET_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	int i;
> +
> +	if (!ctrl->trace_buf)
> +		return;
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
> +		if (ctrl->trace_buf[i].addr)
> +			dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
> +					   ctrl->trace_buf[i].addr,
> +					   ctrl->trace_buf[i].dma);
> +	}
> +
> +	devm_kfree(dev, ctrl->trace_buf);
> +	ctrl->trace_buf = NULL;
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	int i;
> +
> +	hisi_ptt->trace_ctrl.buf_index = 0;
> +
> +	/* If the trace buffer has already been allocated, zero it. */
> +	if (ctrl->trace_buf) {
> +		for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
> +			memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
> +		return 0;
> +	}
> +
> +	ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
> +				       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
> +	if (!ctrl->trace_buf)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
> +		ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
> +							     &ctrl->trace_buf[i].dma,
> +							     GFP_KERNEL);
> +		if (!ctrl->trace_buf[i].addr) {
> +			hisi_ptt_free_trace_buf(hisi_ptt);
> +			return -ENOMEM;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	hisi_ptt->trace_ctrl.started = false;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	u32 val;
> +	int i;
> +
> +	/* Check device idle before start trace */
> +	if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
> +		return -EBUSY;
> +	}
> +
> +	ctrl->started = true;
> +
> +	/* Reset the DMA before start tracing */
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val |= HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	hisi_ptt_wait_dma_reset_done(hisi_ptt);
> +
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/* Clear the interrupt status */
> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> +	/* Configure the trace DMA buffer */
> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
> +		writel(lower_32_bits(ctrl->trace_buf[i].dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
> +		writel(upper_32_bits(ctrl->trace_buf[i].dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
> +	}
> +	writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> +	/* Set the trace control register */
> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> +	if (!hisi_ptt->trace_ctrl.is_port)
> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> +	/* Start the Trace */
> +	val |= HISI_PTT_TRACE_CTRL_EN;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct perf_output_handle *handle = &ctrl->handle;
> +	struct perf_event *event = handle->event;
> +	struct hisi_ptt_pmu_buf *buf;
> +	void *addr;
> +
> +	buf = perf_get_aux(handle);
> +	if (!buf || !handle->size)
> +		return -EINVAL;
> +
> +	addr = ctrl->trace_buf[ctrl->buf_index].addr;
> +
> +	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
> +	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);

Hi Kicong,

I also have the same comment as Leo here, I don't think the memset is
required.

> +	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
> +
> +	if (stop) {
> +		perf_aux_output_end(handle, buf->pos);
> +	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
> +		perf_aux_output_skip(handle, buf->length - buf->pos);

perf_aux_output_skip() can also return an error so should probably also
be checked like perf_aux_output_begin()

I'm also wondering why there is a skip for every output_end()? Is that
to avoid having two memcpy calls to handle the wrap around if the data
to be copied goes past the end of the aux buffer?

For example if your buffers are 4MB each and the aux buffer that the
user picked isn't a multiple of 4 I can see you needing to write the
first part of the 4MB to the end of the aux buffer and then the last
part to the beginning which would be two memcpy() calls. And then a
skip wouldn't be required.

I looked at all the other uses of perf_output_end() and perf_output_skip()
in the kernel and didn't see a pattern like yours so it seems suspicous to
me. Maybe at least some comments around this section are needed.

Thanks
James

> +		perf_aux_output_end(handle, buf->pos);
> +
> +		buf = perf_aux_output_begin(handle, event);
> +		if (!buf)
> +			return -EINVAL;
> +
> +		buf->pos = handle->head % buf->length;
> +		if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
> +			perf_aux_output_end(handle, 0);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status, buf_idx;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	buf_idx = ffs(status) - 1;
> +
> +	/* Clear the interrupt status of buffer @buf_idx */
> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> +	/*
> +	 * Update the AUX buffer and cache the current buffer index,
> +	 * as we need to know this and save the data when the trace
> +	 * is ended out of the interrupt handler. End the trace
> +	 * if the updating fails.
> +	 */
> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> +		hisi_ptt_trace_end(hisi_ptt);
> +	else
> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUF_CNT;
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> +		return IRQ_NONE;
> +
> +	return IRQ_WAKE_THREAD;
> +}
> +
> +static void hisi_ptt_irq_free_vectors(void *pdev)
> +{
> +	pci_free_irq_vectors(pdev);
> +}
> +
> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	int ret;
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> +	if (ret < 0) {
> +		pci_err(pdev, "failed to allocate irq vector, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = devm_request_threaded_irq(&pdev->dev,
> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> +					DRV_NAME, hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to request irq %d, ret = %d\n",
> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> +{
> +	struct hisi_ptt_filter_desc *filter;
> +	struct hisi_ptt *hisi_ptt = data;
> +	struct list_head *target_list;
> +
> +	target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
> +		      &hisi_ptt->port_filters : &hisi_ptt->req_filters;
> +
> +	filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> +	if (!filter) {
> +		pci_err(hisi_ptt->pdev, "failed to add filter %s\n", pci_name(pdev));
> +		return -ENOMEM;
> +	}
> +
> +	filter->pdev = pdev;
> +	list_add_tail(&filter->list, target_list);
> +
> +	/* Update the available port mask */
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		hisi_ptt->port_mask |= hisi_ptt_get_filter_val(pdev);
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_release_filters(void *data)
> +{
> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> +	struct hisi_ptt *hisi_ptt = data;
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +}
> +
> +static int hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct pci_bus *bus;
> +	int ret;
> +	u32 reg;
> +
> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> +	/*
> +	 * The device range register provides the information about the
> +	 * root ports which the RCiEP can control and trace. The RCiEP
> +	 * and the root ports it support are on the same PCIe core, with
> +	 * same domain number but maybe different bus number. The device
> +	 * range register will tell us which root ports we can support,
> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> +	 * while Bit[15:0] indicates the lower.
> +	 */
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> +	hisi_ptt->upper = FIELD_GET(HISI_PTT_DEVICE_RANGE_UPPER, reg);
> +	hisi_ptt->lower = FIELD_GET(HISI_PTT_DEVICE_RANGE_LOWER, reg);
> +
> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
> +	if (bus)
> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_release_filters, hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +
> +	return 0;
> +}
> +
> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> +
> +static ssize_t available_root_port_filters_show(struct device *dev,
> +						struct device_attribute *attr,
> +						char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (list_empty(&hisi_ptt->port_filters)) {
> +		pos = sysfs_emit(buf, "\n");
> +		goto out;
> +	}
> +
> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev) |
> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_root_port_filters);
> +
> +static ssize_t available_requester_filters_show(struct device *dev,
> +						struct device_attribute *attr,
> +						char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (list_empty(&hisi_ptt->req_filters)) {
> +		pos = sysfs_emit(buf, "\n");
> +		goto out;
> +	}
> +
> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev));
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_requester_filters);
> +
> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> +PMU_FORMAT_ATTR(type,		"config:24-31");
> +PMU_FORMAT_ATTR(format,		"config:32-35");
> +
> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> +	&format_attr_filter.attr,
> +	&format_attr_direction.attr,
> +	&format_attr_type.attr,
> +	&format_attr_format.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_format_group = {
> +	.name = "format",
> +	.attrs = hisi_ptt_pmu_format_attrs,
> +};
> +
> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> +	&dev_attr_available_root_port_filters.attr,
> +	&dev_attr_available_requester_filters.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> +	.attrs = hisi_ptt_pmu_filter_attrs,
> +};
> +
> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> +	&hisi_ptt_pmu_format_group,
> +	&hisi_ptt_pmu_filter_group,
> +	NULL
> +};
> +
> +/*
> + * Check whether the config is valid or not. Some configs are multi-selectable
> + * and can be set simultaneously, while some are single selectable (onehot).
> + * Use this function to check the non-onehot configs while
> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> + */
> +static int hisi_ptt_trace_valid_config(u32 val, const u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	/* The non-onehot configs cannot be 0. */
> +	if (!val)
> +		return -EINVAL;
> +
> +	/*
> +	 * Walk the available list and clear the valid bits of
> +	 * the config. If there is any resident bit after the
> +	 * walk then the config is invalid.
> +	 */
> +	for (i = 0; i < list_size; i++)
> +		val &= ~available_list[i];
> +
> +	return val ? -EINVAL : 0;
> +}
> +
> +static int hisi_ptt_trace_valid_config_onehot(u32 val, const u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	for (i = 0; i < list_size; i++)
> +		if (val == available_list[i])
> +			return 0;
> +
> +	return -EINVAL;
> +}
> +
> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> +{
> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> +	struct hisi_ptt_filter_desc *filter;
> +	int ret = -EINVAL;
> +
> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> +
> +	/*
> +	 * Port filters are defined as bit mask. For port filters, check
> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> +	 * and whether it's empty or not, otherwise user has specified
> +	 * some unsupported root ports.
> +	 *
> +	 * For Requester ID filters, walk the available filter list to see
> +	 * whether we have one matched.
> +	 */
> +	if (!hisi_ptt->trace_ctrl.is_port) {
> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +			if (val == hisi_ptt_get_filter_val(filter->pdev)) {
> +				ret = 0;
> +				break;
> +			}
> +	} else if (bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> +		ret = 0;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.filter = val;
> +	return 0;
> +}
> +
> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> +{
> +	/*
> +	 * The supported value of the direction parameter. See hisi_ptt.rst
> +	 * documentation for more details.
> +	 */
> +	static const u32 hisi_ptt_trace_available_direction[] = {
> +		0,
> +		1,
> +		2,
> +		3,
> +	};
> +	/* Different types can be set simultaneously */
> +	static const u32 hisi_ptt_trace_available_type[] = {
> +		1,	/* posted_request */
> +		2,	/* non-posted_request */
> +		4,	/* completion */
> +	};
> +	static const u32 hisi_ptt_trace_availble_format[] = {
> +		0,	/* 4DW */
> +		1,	/* 8DW */
> +	};
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	int ret;
> +	u32 val;
> +
> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> +		return -ENOENT;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +
> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> +	if (ret < 0)
> +		goto out;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->direction = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->type = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->format = val;
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return ret;
> +}
> +
> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> +				    int nr_pages, bool overwrite)
> +{
> +	struct hisi_ptt_pmu_buf *buf;
> +	struct page **pagelist;
> +	int i;
> +
> +	if (overwrite) {
> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> +		return NULL;
> +	}
> +
> +	/* If the pages size less than buffers, we cannot start trace */
> +	if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
> +		return NULL;
> +
> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> +	if (!buf)
> +		return NULL;
> +
> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> +	if (!pagelist) {
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++)
> +		pagelist[i] = virt_to_page(pages[i]);
> +
> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> +	if (!buf->base) {
> +		kfree(pagelist);
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	buf->nr_pages = nr_pages;
> +	buf->length = nr_pages * PAGE_SIZE;
> +	buf->pos = 0;
> +
> +	kfree(pagelist);
> +	return buf;
> +}
> +
> +static void hisi_ptt_pmu_free_aux(void *aux)
> +{
> +	struct hisi_ptt_pmu_buf *buf = aux;
> +
> +	vunmap(buf->base);
> +	kfree(buf);
> +}
> +
> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct hw_perf_event *hwc = &event->hw;
> +	struct hisi_ptt_pmu_buf *buf;
> +	int cpu = event->cpu;
> +	int ret;
> +
> +	hwc->state = 0;
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.started) {
> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> +		goto stop;
> +	}
> +
> +	if (cpu == -1)
> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> +
> +	/*
> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> +	 * core in event_function_local().
> +	 */
> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> +				 cpumask_of(cpu)));
> +
> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> +		goto stop;
> +	}
> +
> +	buf = perf_aux_output_begin(handle, event);
> +	if (!buf) {
> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> +		goto stop;
> +	}
> +
> +	buf->pos = handle->head % buf->length;
> +
> +	ret = hisi_ptt_trace_start(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> +		perf_aux_output_end(handle, 0);
> +		goto stop;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return;
> +stop:
> +	event->hw.state |= PERF_HES_STOPPED;
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +
> +	if (hwc->state & PERF_HES_STOPPED)
> +		return;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.started) {
> +		hisi_ptt_trace_end(hisi_ptt);
> +		WARN(!hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> +	}
> +	mutex_unlock(&hisi_ptt->mutex);
> +
> +	hwc->state |= PERF_HES_STOPPED;
> +	perf_event_update_userpage(event);
> +	hwc->state |= PERF_HES_UPTODATE;
> +}
> +
> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +	int cpu = event->cpu;
> +
> +	/*
> +	 * Only allow the default cpu to add the event if user doesn't specify
> +	 * the cpus.
> +	 */
> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> +		return 0;
> +
> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +
> +	if (flags & PERF_EF_START) {
> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> +		if (hwc->state & PERF_HES_STOPPED)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> +{
> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> +}
> +
> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> +{
> +	u16 core_id, sicl_id;
> +	char *pmu_name;
> +	u32 reg;
> +
> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> +		.module		= THIS_MODULE,
> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> +		.task_ctx_nr	= perf_sw_context,
> +		.attr_groups	= hisi_ptt_pmu_groups,
> +		.event_init	= hisi_ptt_pmu_event_init,
> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> +		.free_aux	= hisi_ptt_pmu_free_aux,
> +		.start		= hisi_ptt_pmu_start,
> +		.stop		= hisi_ptt_pmu_stop,
> +		.add		= hisi_ptt_pmu_add,
> +		.del		= hisi_ptt_pmu_del,
> +	};
> +
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> +	core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> +	sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> +				  sicl_id, core_id);
> +	if (!pmu_name)
> +		return -ENOMEM;
> +
> +	return perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> +}
> +
> +/*
> + * The DMA of PTT trace can only use direct mapping, due to some
> + * hardware restriction. Check whether there is an IOMMU or the
> + * policy of the IOMMU domain is passthrough, otherwise the trace
> + * cannot work.
> + *
> + * The PTT device is supposed to behind the ARM SMMUv3, which
> + * should have passthrough the device by a quirk.
> + */
> +static int hisi_ptt_check_iommu_mapping(struct pci_dev *pdev)
> +{
> +	struct iommu_domain *iommu_domain;
> +
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	return -EOPNOTSUPP;
> +}
> +
> +static int hisi_ptt_probe(struct pci_dev *pdev,
> +			  const struct pci_device_id *id)
> +{
> +	struct hisi_ptt *hisi_ptt;
> +	int ret;
> +
> +	ret = hisi_ptt_check_iommu_mapping(pdev);
> +	if (ret) {
> +		pci_err(pdev, "requires direct DMA mappings\n");
> +		return ret;
> +	}
> +
> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> +	if (!hisi_ptt)
> +		return -ENOMEM;
> +
> +	mutex_init(&hisi_ptt->mutex);
> +	hisi_ptt->pdev = pdev;
> +	pci_set_drvdata(pdev, hisi_ptt);
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		pci_err(pdev, "failed to enable device, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(2), DRV_NAME);
> +	if (ret) {
> +		pci_err(pdev, "failed to remap io memory, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> +
> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> +	if (ret) {
> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	pci_set_master(pdev);
> +
> +	ret = hisi_ptt_register_irq(hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	ret = hisi_ptt_init_ctrls(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to init controls, ret = %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to register PMU device, ret = %d", ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_remove(struct pci_dev *pdev)
> +{
> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> +
> +	/*
> +	 * We have to manually unregister the PMU device rather than make it
> +	 * devres managed to keep order that the PMU device's unregistration
> +	 * is prior to the release of DMA buffers. As the DMA buffers are
> +	 * devm allocated when necessary which is after the registration of
> +	 * the PMU device.
> +	 */
> +	perf_pmu_unregister(&hisi_ptt->hisi_ptt_pmu);
> +}
> +
> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> +	{ }
> +};
> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> +
> +static struct pci_driver hisi_ptt_driver = {
> +	.name = DRV_NAME,
> +	.id_table = hisi_ptt_id_tbl,
> +	.probe = hisi_ptt_probe,
> +	.remove = hisi_ptt_remove,
> +};
> +module_pci_driver(hisi_ptt_driver);
> +
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
> new file mode 100644
> index 000000000000..29b6d6293d6c
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.h
> @@ -0,0 +1,166 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#ifndef _HISI_PTT_H
> +#define _HISI_PTT_H
> +
> +#include <linux/bits.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/perf_event.h>
> +#include <linux/types.h>
> +
> +#define DRV_NAME "hisi_ptt"
> +
> +/*
> + * The definition of the device registers and register fields.
> + */
> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> +#define HISI_PTT_TRACE_CTRL		0x0850
> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS		0x08b0
> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> +#define   HISI_PTT_DEVICE_RANGE_UPPER	GENMASK(31, 16)
> +#define   HISI_PTT_DEVICE_RANGE_LOWER	GENMASK(15, 0)
> +#define HISI_PTT_LOCATION		0x0fe8
> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> +
> +/* Parameters of PTT trace DMA part. */
> +#define HISI_PTT_TRACE_DMA_IRQ			0
> +#define HISI_PTT_TRACE_BUF_CNT			4
> +#define HISI_PTT_TRACE_BUF_SIZE			SZ_4M
> +#define HISI_PTT_TRACE_TOTAL_BUF_SIZE		(HISI_PTT_TRACE_BUF_SIZE * \
> +						 HISI_PTT_TRACE_BUF_CNT)
> +/* Wait time for hardware DMA to reset */
> +#define HISI_PTT_RESET_TIMEOUT_US	10UL
> +#define HISI_PTT_RESET_POLL_INTERVAL_US	1UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TRACE_TIMEOUT_US	100UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	10UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> +
> +/**
> + * struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
> + *                              The detail of the data format is described
> + *                              in the documentation of PTT device.
> + * @dma:   DMA address of this buffer visible to the device
> + * @addr:  virtual address of this buffer visible to the cpu
> + */
> +struct hisi_ptt_dma_buffer {
> +	dma_addr_t dma;
> +	void *addr;
> +};
> +
> +/**
> + * struct hisi_ptt_trace_ctrl - control and status of PTT trace
> + * @trace_buf:   array of the trace buffers for holding the trace data.
> + *               the length will be HISI_PTT_TRACE_BUF_CNT.
> + * @handle:      perf output handle of current trace session
> + * @default_cpu: default cpu to start the trace session
> + * @buf_index:   the index of current using trace buffer
> + * @started:     current trace status, true for started
> + * @is_port:     whether we're tracing root port or not
> + * @direction:   direction of the TLP headers to trace
> + * @filter:      filter value for tracing the TLP headers
> + * @format:      format of the TLP headers to trace
> + * @type:        type of the TLP headers to trace
> + */
> +struct hisi_ptt_trace_ctrl {
> +	struct hisi_ptt_dma_buffer *trace_buf;
> +	struct perf_output_handle handle;
> +	int default_cpu;
> +	u32 buf_index;
> +	bool started;
> +	bool is_port;
> +	u32 direction:2;
> +	u32 filter:16;
> +	u32 format:1;
> +	u32 type:4;
> +};
> +
> +/**
> + * struct hisi_ptt_filter_desc - descriptor of the PTT trace filter
> + * @list: entry of this descriptor in the filter list
> + * @pdev: pci_dev related to this filter
> + */
> +struct hisi_ptt_filter_desc {
> +	struct list_head list;
> +	struct pci_dev *pdev;
> +};
> +
> +
> +/**
> + * struct hisi_ptt_pmu_buf - descriptor of the AUX buffer of PTT trace
> + * @length:   size of the AUX buffer
> + * @nr_pages: number of pages of the AUX buffer
> + * @base:     start address of AUX buffer
> + * @pos:      position in the AUX buffer to commit traced data
> + */
> +struct hisi_ptt_pmu_buf {
> +	size_t length;
> +	int nr_pages;
> +	void *base;
> +	long pos;
> +};
> +
> +/**
> + * struct hisi_ptt - per PTT device data
> + * @trace_ctrl:   the control information of PTT trace
> + * @hisi_ptt_pmu: the pum device of trace
> + * @iobase:       base IO address of the device
> + * @pdev:         pci_dev of this PTT device
> + * @mutex:        mutex to protect the filter list and serialize the perf process.
> + * @upper:        the upper BDF range of the PCI devices managed by this PTT device
> + * @lower:        the lower BDF range of the PCI devices managed by this PTT device
> + * @port_filters: the filter list of root ports
> + * @req_filters:  the filter list of requester ID
> + * @port_mask:    port mask of the managed root ports
> + */
> +struct hisi_ptt {
> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> +	struct pmu hisi_ptt_pmu;
> +	void __iomem *iobase;
> +	struct pci_dev *pdev;
> +	struct mutex mutex;
> +	u32 upper;
> +	u32 lower;
> +
> +	/*
> +	 * The trace TLP headers can either be filtered by certain
> +	 * root port, or by the requester ID. Organize the filters
> +	 * by @port_filters and @req_filters here. The mask of all
> +	 * the valid ports is also cached for doing sanity check
> +	 * of user input.
> +	 */
> +	struct list_head port_filters;
> +	struct list_head req_filters;
> +	u16 port_mask;
> +};
> +
> +#define to_hisi_ptt(pmu) container_of(pmu, struct hisi_ptt, hisi_ptt_pmu)
> +
> +#endif /* _HISI_PTT_H */

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-04-07 12:58 ` [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
  2022-04-11 17:19   ` John Garry
  2022-04-30  7:35   ` Leo Yan
@ 2022-05-10 10:14   ` James Clark
  2022-05-11  2:02     ` liuqi (BA)
  2 siblings, 1 reply; 30+ messages in thread
From: James Clark @ 2022-05-10 10:14 UTC (permalink / raw)
  To: Yicong Yang, Leo Yan
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, gregkh, helgaas,
	alexander.shishkin, lorenzo.pieralisi, will, mark.rutland,
	mathieu.poirier, suzuki.poulose, mike.leach, leo.yan,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu



On 07/04/2022 13:58, Yicong Yang wrote:
> From: Qi Liu <liuqi115@huawei.com>
> 
> 'perf record' and 'perf report --dump-raw-trace' supported in this
> patch.
> 
> Example usage:
> 
> Output will contain raw PTT data and its textual representation, such
> as:
> 
> 0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
> ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
> .
> . ... HISI PTT data: size 4194304 bytes
> .  00000000: 00 00 00 00                                 Prefix
> .  00000004: 08 20 00 60                                 Header DW0
> .  00000008: ff 02 00 01                                 Header DW1
> .  0000000c: 20 08 00 00                                 Header DW2
> .  00000010: 10 e7 44 ab                                 Header DW3
> .  00000014: 2a a8 1e 01                                 Time
> .  00000020: 00 00 00 00                                 Prefix
> .  00000024: 01 00 00 60                                 Header DW0
> .  00000028: 0f 1e 00 01                                 Header DW1
> .  0000002c: 04 00 00 00                                 Header DW2
> .  00000030: 40 00 81 02                                 Header DW3
> .  00000034: ee 02 00 00                                 Time
> ....
> 
> Signed-off-by: Qi Liu <liuqi115@huawei.com>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  tools/perf/arch/arm/util/auxtrace.c           |  76 +++++-
>  tools/perf/arch/arm/util/pmu.c                |   3 +
>  tools/perf/arch/arm64/util/Build              |   2 +-
>  tools/perf/arch/arm64/util/hisi_ptt.c         | 195 ++++++++++++++++
>  tools/perf/util/Build                         |   2 +
>  tools/perf/util/auxtrace.c                    |   4 +
>  tools/perf/util/auxtrace.h                    |   1 +
>  tools/perf/util/hisi-ptt-decoder/Build        |   1 +
>  .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 ++++++++++++++
>  .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
>  tools/perf/util/hisi_ptt.c                    | 218 ++++++++++++++++++
>  tools/perf/util/hisi_ptt.h                    |  28 +++
>  12 files changed, 724 insertions(+), 4 deletions(-)
>  create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
>  create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
>  create mode 100644 tools/perf/util/hisi_ptt.c
>  create mode 100644 tools/perf/util/hisi_ptt.h
> 
> diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
> index 5fc6a2a3dbc5..393f5757c039 100644
> --- a/tools/perf/arch/arm/util/auxtrace.c
> +++ b/tools/perf/arch/arm/util/auxtrace.c
> @@ -4,9 +4,11 @@
>   * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
>   */
>  
> +#include <dirent.h>
>  #include <stdbool.h>
>  #include <linux/coresight-pmu.h>
>  #include <linux/zalloc.h>
> +#include <api/fs/fs.h>
>  
>  #include "../../../util/auxtrace.h"
>  #include "../../../util/debug.h"
> @@ -14,6 +16,7 @@
>  #include "../../../util/pmu.h"
>  #include "cs-etm.h"
>  #include "arm-spe.h"
> +#include "hisi_ptt.h"
>  
>  static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>  {
> @@ -50,6 +53,58 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
>  	return arm_spe_pmus;
>  }
>  
> +static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
> +{
> +	const char *sysfs = sysfs__mountpoint();
> +	struct perf_pmu **hisi_ptt_pmus = NULL;
> +	struct dirent *dent;
> +	char path[PATH_MAX];
> +	DIR *dir = NULL;
> +	int idx = 0;
> +
> +	snprintf(path, PATH_MAX, "%s" EVENT_SOURCE_DEVICE_PATH, sysfs);
> +	dir = opendir(path);
> +	if (!dir) {
> +		pr_err("can't read directory '%s'\n", EVENT_SOURCE_DEVICE_PATH);
> +		*err = -EINVAL;
> +		goto out;
> +	}
> +
> +	while ((dent = readdir(dir))) {
> +		if (strstr(dent->d_name, HISI_PTT_PMU_NAME))
> +			(*nr_ptts)++;
> +	}
> +
> +	if (!(*nr_ptts))
> +		goto out;
> +
> +	hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * (*nr_ptts));
> +	if (!hisi_ptt_pmus) {
> +		pr_err("hisi_ptt alloc failed\n");
> +		*err = -ENOMEM;
> +		goto out;
> +	}
> +
> +	rewinddir(dir);
> +	while ((dent = readdir(dir))) {
> +		if (strstr(dent->d_name, HISI_PTT_PMU_NAME) && idx < (*nr_ptts)) {
> +			hisi_ptt_pmus[idx] = perf_pmu__find(dent->d_name);
> +			if (hisi_ptt_pmus[idx]) {
> +				pr_debug2("%s %d: hisi_ptt_pmu %d type %d name %s\n",
> +					__func__, __LINE__, idx,
> +					hisi_ptt_pmus[idx]->type,
> +					hisi_ptt_pmus[idx]->name);
> +					idx++;
> +			}
> +
> +		}
> +	}
> +
> +out:
> +	closedir(dir);
> +	return hisi_ptt_pmus;
> +}
> +
>  struct auxtrace_record
>  *auxtrace_record__init(struct evlist *evlist, int *err)
>  {
> @@ -57,8 +112,12 @@ struct auxtrace_record
>  	struct evsel *evsel;
>  	bool found_etm = false;
>  	struct perf_pmu *found_spe = NULL;
> +	struct perf_pmu *found_ptt = NULL;
>  	struct perf_pmu **arm_spe_pmus = NULL;
> +	struct perf_pmu **hisi_ptt_pmus = NULL;
> +
>  	int nr_spes = 0;
> +	int nr_ptts = 0;
>  	int i = 0;
>  
>  	if (!evlist)
> @@ -66,13 +125,14 @@ struct auxtrace_record
>  
>  	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
>  	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
> +	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
>  
>  	evlist__for_each_entry(evlist, evsel) {
>  		if (cs_etm_pmu &&
>  		    evsel->core.attr.type == cs_etm_pmu->type)
>  			found_etm = true;
>  
> -		if (!nr_spes || found_spe)
> +		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
>  			continue;
>  
>  		for (i = 0; i < nr_spes; i++) {
> @@ -81,11 +141,18 @@ struct auxtrace_record
>  				break;
>  			}
>  		}
> +
> +		for (i = 0; i < nr_ptts; i++) {
> +			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
> +				found_ptt = hisi_ptt_pmus[i];
> +				break;
> +			}
> +		}
>  	}
>  	free(arm_spe_pmus);
>  
> -	if (found_etm && found_spe) {
> -		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
> +	if (found_etm && found_spe && found_ptt) {
> +		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");

Hi Yicong,

Is that actually a limitation? I don't see why they couldn't work concurrently.


>  		*err = -EOPNOTSUPP;
>  		return NULL;
>  	}
> @@ -96,6 +163,9 @@ struct auxtrace_record
>  #if defined(__aarch64__)
>  	if (found_spe)
>  		return arm_spe_recording_init(err, found_spe);
> +
> +	if (found_ptt)
> +		return hisi_ptt_recording_init(err, found_ptt);
>  #endif
>  
>  	/*
> diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
> index b8b23b9dc598..89a3cedb4557 100644
> --- a/tools/perf/arch/arm/util/pmu.c
> +++ b/tools/perf/arch/arm/util/pmu.c
> @@ -10,6 +10,7 @@
>  #include <linux/string.h>
>  
>  #include "arm-spe.h"
> +#include "hisi_ptt.h"
>  #include "../../../util/pmu.h"
>  
>  struct perf_event_attr
> @@ -22,6 +23,8 @@ struct perf_event_attr
>  #if defined(__aarch64__)
>  	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
>  		return arm_spe_pmu_default_config(pmu);
> +	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
> +		pmu->selectable = true;
>  #endif
>  	}
>  
> diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
> index 9fcb4e68add9..8b7fd1dc9f37 100644
> --- a/tools/perf/arch/arm64/util/Build
> +++ b/tools/perf/arch/arm64/util/Build
> @@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
>  perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
>  			      ../../arm/util/auxtrace.o \
>  			      ../../arm/util/cs-etm.o \
> -			      arm-spe.o mem-events.o
> +			      arm-spe.o mem-events.o hisi_ptt.o
> diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
> new file mode 100644
> index 000000000000..de3a3523ffd5
> --- /dev/null
> +++ b/tools/perf/arch/arm64/util/hisi_ptt.c
> @@ -0,0 +1,195 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/types.h>
> +#include <linux/bitops.h>
> +#include <linux/log2.h>
> +#include <linux/zalloc.h>
> +#include <time.h>
> +
> +#include <internal/lib.h> // page_size
> +#include "../../../util/auxtrace.h"
> +#include "../../../util/cpumap.h"
> +#include "../../../util/debug.h"
> +#include "../../../util/event.h"
> +#include "../../../util/evlist.h"
> +#include "../../../util/evsel.h"
> +#include "../../../util/hisi_ptt.h"
> +#include "../../../util/pmu.h"
> +#include "../../../util/record.h"
> +#include "../../../util/session.h"
> +#include "../../../util/tsc.h"
> +
> +#define DEFAULT_PAGE_SIZE 1024
> +#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
> +#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
> +
> +struct hisi_ptt_recording {
> +	struct auxtrace_record	itr;
> +	struct perf_pmu *hisi_ptt_pmu;
> +	struct evlist *evlist;
> +};
> +
> +static size_t
> +hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
> +			struct evlist *evlist __maybe_unused)
> +{
> +	return HISI_PTT_AUXTRACE_PRIV_SIZE;
> +}
> +
> +static int hisi_ptt_info_fill(struct auxtrace_record *itr,
> +			      struct perf_session *session,
> +			      struct perf_record_auxtrace_info *auxtrace_info,
> +			      size_t priv_size)
> +{
> +	struct hisi_ptt_recording *pttr =
> +			container_of(itr, struct hisi_ptt_recording, itr);
> +	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
> +
> +	if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
> +		return -EINVAL;
> +
> +	if (!session->evlist->core.nr_mmaps)
> +		return -EINVAL;
> +
> +	auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
> +	auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
> +{
> +	bool privileged = perf_event_paranoid_check(-1);
> +
> +	if (!opts->full_auxtrace)
> +		return 0;
> +
> +	if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
> +		if (privileged) {
> +			opts->auxtrace_mmap_pages = MiB(16) / page_size;
> +		} else {
> +			opts->auxtrace_mmap_pages = KiB(128) / page_size;
> +			if (opts->mmap_pages == UINT_MAX)
> +				opts->mmap_pages = KiB(256) / page_size;
> +		}
> +	}
> +
> +	/* Validate auxtrace_mmap_pages */
> +	if (opts->auxtrace_mmap_pages) {
> +		size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
> +		size_t min_sz = KiB(8);
> +
> +		if (sz < min_sz || !is_power_of_2(sz)) {
> +			pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
> +			       min_sz / DEFAULT_PAGE_SIZE);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_recording_options(struct auxtrace_record *itr,
> +				      struct evlist *evlist,
> +				      struct record_opts *opts)
> +{
> +	struct hisi_ptt_recording *pttr =
> +			container_of(itr, struct hisi_ptt_recording, itr);
> +	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
> +	struct perf_cpu_map *cpus = evlist->core.cpus;
> +	struct evsel *evsel, *hisi_ptt_evsel = NULL;
> +	struct evsel *tracking_evsel;
> +	int err;
> +
> +	pttr->evlist = evlist;
> +	evlist__for_each_entry(evlist, evsel) {
> +		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
> +			if (hisi_ptt_evsel) {
> +				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
> +				return -EINVAL;
> +			}
> +			evsel->core.attr.freq = 0;
> +			evsel->core.attr.sample_period = 1;
> +			hisi_ptt_evsel = evsel;
> +			opts->full_auxtrace = true;
> +		}
> +	}
> +
> +	err = hisi_ptt_set_auxtrace_mmap_page(opts);
> +	if (err)
> +		return err;
> +	/*
> +	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
> +	 * must come first.
> +	 */
> +	evlist__to_front(evlist, hisi_ptt_evsel);
> +
> +	if (!perf_cpu_map__empty(cpus)) {
> +		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
> +		evsel__set_sample_bit(hisi_ptt_evsel, CPU);
> +	}

Similar to Leo's comment: CPU isn't required if it's uncore,
and if TIME is useful then add it regardless of whether the
event is opened per-cpu or on a task.

> +
> +	/* Add dummy event to keep tracking */
> +	err = parse_events(evlist, "dummy:u", NULL);
> +	if (err)
> +		return err;
> +
> +	tracking_evsel = evlist__last(evlist);
> +	evlist__set_tracking_event(evlist, tracking_evsel);
> +
> +	tracking_evsel->core.attr.freq = 0;
> +	tracking_evsel->core.attr.sample_period = 1;
> +
> +	if (!perf_cpu_map__empty(cpus))
> +		evsel__set_sample_bit(tracking_evsel, TIME);

Same comment as above.

> +
> +	return 0;
> +}
> +
> +static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
> +{
> +	return rdtsc();
> +}
> +
> +static void hisi_ptt_recording_free(struct auxtrace_record *itr)
> +{
> +	struct hisi_ptt_recording *pttr =
> +			container_of(itr, struct hisi_ptt_recording, itr);
> +
> +	free(pttr);
> +}
> +
> +struct auxtrace_record *hisi_ptt_recording_init(int *err,
> +						struct perf_pmu *hisi_ptt_pmu)
> +{
> +	struct hisi_ptt_recording *pttr;
> +
> +	if (!hisi_ptt_pmu) {
> +		*err = -ENODEV;
> +		return NULL;
> +	}
> +
> +	pttr = zalloc(sizeof(struct hisi_ptt_recording));
> +	if (!pttr) {
> +		*err = -ENOMEM;
> +		return NULL;
> +	}
> +
> +	pttr->hisi_ptt_pmu = hisi_ptt_pmu;
> +	pttr->itr.pmu = hisi_ptt_pmu;
> +	pttr->itr.recording_options = hisi_ptt_recording_options;
> +	pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
> +	pttr->itr.info_fill = hisi_ptt_info_fill;
> +	pttr->itr.free = hisi_ptt_recording_free;
> +	pttr->itr.reference = hisi_ptt_reference;
> +	pttr->itr.read_finish = auxtrace_record__read_finish;
> +	pttr->itr.alignment = 0;
> +
> +	*err = 0;
> +	return &pttr->itr;
> +}
> diff --git a/tools/perf/util/Build b/tools/perf/util/Build
> index 9a7209a99e16..95d2d76c52da 100644
> --- a/tools/perf/util/Build
> +++ b/tools/perf/util/Build
> @@ -116,6 +116,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
>  perf-$(CONFIG_AUXTRACE) += intel-bts.o
>  perf-$(CONFIG_AUXTRACE) += arm-spe.o
>  perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
> +perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
> +perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
>  perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
>  
>  ifdef CONFIG_LIBOPENCSD
> diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
> index df1c5bbbaa0d..f4841ebaf314 100644
> --- a/tools/perf/util/auxtrace.c
> +++ b/tools/perf/util/auxtrace.c
> @@ -51,6 +51,7 @@
>  #include "intel-pt.h"
>  #include "intel-bts.h"
>  #include "arm-spe.h"
> +#include "hisi_ptt.h"
>  #include "s390-cpumsf.h"
>  #include "util/mmap.h"
>  
> @@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
>  	case PERF_AUXTRACE_S390_CPUMSF:
>  		err = s390_cpumsf_process_auxtrace_info(event, session);
>  		break;
> +	case PERF_AUXTRACE_HISI_PTT:
> +		err = hisi_ptt_process_auxtrace_info(event, session);
> +		break;
>  	case PERF_AUXTRACE_UNKNOWN:
>  	default:
>  		return -EINVAL;
> diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
> index dc38b6f57232..3858c5752ead 100644
> --- a/tools/perf/util/auxtrace.h
> +++ b/tools/perf/util/auxtrace.h
> @@ -48,6 +48,7 @@ enum auxtrace_type {
>  	PERF_AUXTRACE_CS_ETM,
>  	PERF_AUXTRACE_ARM_SPE,
>  	PERF_AUXTRACE_S390_CPUMSF,
> +	PERF_AUXTRACE_HISI_PTT,
>  };
>  
>  enum itrace_period_type {
> diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
> new file mode 100644
> index 000000000000..db3db8b75033
> --- /dev/null
> +++ b/tools/perf/util/hisi-ptt-decoder/Build
> @@ -0,0 +1 @@
> +perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
> diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
> new file mode 100644
> index 000000000000..3fb7a6949209
> --- /dev/null
> +++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
> @@ -0,0 +1,170 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#include <stdlib.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <endian.h>
> +#include <byteswap.h>
> +#include <linux/bitops.h>
> +#include <stdarg.h>
> +
> +#include "../color.h"
> +#include "hisi-ptt-pkt-decoder.h"
> +
> +/*
> + * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
> + * used to distinguish the data format.
> + * 8DW format is like:
> + *   bits [                 31:11                 ][       10:0       ]
> + *        |---------------------------------------|-------------------|
> + *    DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
> + *    DW1 [                       Prefix                              ]
> + *    DW2 [                     Header DW0                            ]
> + *    DW3 [                     Header DW1                            ]
> + *    DW4 [                     Header DW2                            ]
> + *    DW5 [                     Header DW3                            ]
> + *    DW6 [                   Reserved (0x0)                          ]
> + *    DW7 [                        Time                               ]
> + *
> + * 4DW format is like:
> + *   bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
> + *        |-----|---------|---|---|---|---|-------------|-------------|
> + *    DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
> + *    DW1 [                     Header DW1                            ]
> + *    DW2 [                     Header DW2                            ]
> + *    DW3 [                     Header DW3                            ]
> + */
> +
> +enum hisi_ptt_8dw_pkt_field_type {
> +	HISI_PTT_8DW_PREFIX,
> +	HISI_PTT_8DW_HEAD0,
> +	HISI_PTT_8DW_HEAD1,
> +	HISI_PTT_8DW_HEAD2,
> +	HISI_PTT_8DW_HEAD3,
> +	HISI_PTT_8DW_TIME,
> +	HISI_PTT_8DW_TYPE_MAX
> +};
> +
> +enum hisi_ptt_4dw_pkt_field_type {
> +	HISI_PTT_4DW_HEAD1,
> +	HISI_PTT_4DW_HEAD2,
> +	HISI_PTT_4DW_HEAD3,
> +	HISI_PTT_4DW_TYPE_MAX
> +};
> +
> +static const char * const hisi_ptt_8dw_pkt_field_name[] = {
> +	[HISI_PTT_8DW_PREFIX]	= "Prefix",
> +	[HISI_PTT_8DW_HEAD0]	= "Header DW0",
> +	[HISI_PTT_8DW_HEAD1]	= "Header DW1",
> +	[HISI_PTT_8DW_HEAD2]	= "Header DW2",
> +	[HISI_PTT_8DW_HEAD3]	= "Header DW3",
> +	[HISI_PTT_8DW_TIME]	= "Time",
> +};
> +
> +static const char * const hisi_ptt_4dw_pkt_field_name[] = {
> +	[HISI_PTT_4DW_HEAD1]	= "Header DW1",
> +	[HISI_PTT_4DW_HEAD2]	= "Header DW2",
> +	[HISI_PTT_4DW_HEAD3]	= "Header DW3",
> +};
> +
> +/* offset of each member is determined by format of 8dw packet. */
> +static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
> +	[HISI_PTT_8DW_PREFIX]	= 4,
> +	[HISI_PTT_8DW_HEAD0]	= 4,
> +	[HISI_PTT_8DW_HEAD1]	= 4,
> +	[HISI_PTT_8DW_HEAD2]	= 4,
> +	[HISI_PTT_8DW_HEAD3]	= 4,
> +	[HISI_PTT_8DW_TIME]	= 8,
> +};
> +
> +union hisi_ptt_4dw {
> +	struct {
> +		uint32_t format : 2;
> +		uint32_t type : 5;
> +		uint32_t t9 : 1;
> +		uint32_t t8 : 1;
> +		uint32_t th : 1;
> +		uint32_t so : 1;
> +		uint32_t len : 10;
> +		uint32_t time : 11;
> +	};
> +	uint32_t value;
> +};
> +
> +static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
> +{
> +	const char *color = PERF_COLOR_BLUE;
> +	int field_len = sizeof(uint32_t);
> +	int i;
> +
> +	printf(".");
> +	color_fprintf(stdout, color, "  %08x: ", *pos);
> +	for (i = 0; i < field_len; i++)
> +		color_fprintf(stdout, color, "%02x ", buf[i]);
> +	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
> +		color_fprintf(stdout, color, "   ");
> +	color_fprintf(stdout, color, "  %s\n", desc);
> +	*pos += field_len;
> +}
> +
> +static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
> +{
> +	int i;
> +
> +	for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
> +		buf += hisi_ptt_8dw_pkt_field_offset[i];
> +		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
> +	}
> +
> +	return HISI_PTT_8DW_PKT_SIZE;
> +}
> +
> +static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
> +{
> +	const char *color = PERF_COLOR_BLUE;
> +	int field_len = sizeof(uint32_t);
> +	union hisi_ptt_4dw dw0;
> +	int i;
> +
> +	dw0.value = *(uint32_t *)buf;
> +	printf(".");
> +	color_fprintf(stdout, color, "  %08x: ", *pos);
> +	for (i = 0; i < field_len; i++)
> +		color_fprintf(stdout, color, "%02x ", buf[i]);
> +	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
> +		color_fprintf(stdout, color, "   ");
> +
> +	color_fprintf(stdout, color,
> +		      "  %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
> +		      "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
> +		      "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
> +		      dw0.len, "Time", dw0.time);
> +
> +	*pos += field_len;
> +}
> +
> +static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
> +{
> +	int i;
> +
> +	hisi_ptt_4dw_print_dw0(buf, &pos);
> +
> +	for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
> +		buf += sizeof(uint32_t);
> +		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
> +	}
> +
> +	return HISI_PTT_4DW_PKT_SIZE;
> +}
> +
> +int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
> +{
> +	if (type == HISI_PTT_8DW_PKT)
> +		return hisi_ptt_8dw_kpt_desc(buf, pos);
> +
> +	return hisi_ptt_4dw_kpt_desc(buf, pos);
> +}
> diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
> new file mode 100644
> index 000000000000..20c51f230cac
> --- /dev/null
> +++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
> +#define INCLUDE__HISI_PTT_PKT_DECODER_H__
> +
> +#include <stddef.h>
> +#include <stdint.h>
> +
> +#define HISI_PTT_PKT_DESC_MAX		256
> +#define HISI_PTT_NEED_MORE_BYTES	-1
> +#define HISI_PTT_8DW_CHECK_MASK		GENMASK(31, 11)
> +#define HISI_PTT_IS_8DW_PKT		GENMASK(31, 11)
> +#define HISI_PTT_8DW_PKT_SIZE		32
> +#define HISI_PTT_4DW_PKT_SIZE		16
> +#define HISI_PTT_MAX_SPACE_LEN		10
> +
> +enum hisi_ptt_pkt_type {
> +	HISI_PTT_4DW_PKT,
> +	HISI_PTT_8DW_PKT,
> +};
> +
> +int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
> +
> +#endif
> diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
> new file mode 100644
> index 000000000000..e2934214763c
> --- /dev/null
> +++ b/tools/perf/util/hisi_ptt.c
> @@ -0,0 +1,218 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#include <byteswap.h>
> +#include <endian.h>
> +#include <errno.h>
> +#include <inttypes.h>
> +#include <linux/bitops.h>
> +#include <linux/kernel.h>
> +#include <linux/log2.h>
> +#include <linux/types.h>
> +#include <linux/zalloc.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +#include "auxtrace.h"
> +#include "color.h"
> +#include "debug.h"
> +#include "evlist.h"
> +#include "evsel.h"
> +#include "hisi_ptt.h"
> +#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
> +#include "machine.h"
> +#include "session.h"
> +#include "symbol.h"
> +#include "tool.h"
> +#include "util/synthetic-events.h"
> +
> +struct hisi_ptt {
> +	struct auxtrace auxtrace;
> +	struct auxtrace_queues queues;
> +	u32 auxtrace_type;
> +	struct perf_session *session;
> +	struct machine *machine;
> +	u32 pmu_type;
> +};
> +
> +struct hisi_ptt_queue {
> +	struct hisi_ptt *ptt;
> +	struct auxtrace_buffer *buffer;
> +};
> +
> +static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
> +{
> +	uint32_t head = *(uint32_t *)buf;
> +
> +	if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
> +		return HISI_PTT_8DW_PKT;
> +
> +	return HISI_PTT_4DW_PKT;
> +}
> +
> +static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
> +			  unsigned char *buf, size_t len)
> +{
> +	const char *color = PERF_COLOR_BLUE;
> +	enum hisi_ptt_pkt_type type;
> +	size_t pos = 0;
> +	int pkt_len;
> +
> +	color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
> +		      len);
> +
> +	type = hisi_ptt_check_packet_type(buf);
> +	while (len) {
> +		pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
> +		if (!pkt_len)
> +			color_fprintf(stdout, color, " Bad packet!\n");
> +
> +		pos += pkt_len;
> +		buf += pkt_len;
> +		len -= pkt_len;
> +	}
> +}
> +
> +static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
> +				size_t len)
> +{
> +	printf(".\n");
> +
> +	hisi_ptt_dump(ptt, buf, len);
> +}
> +
> +static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
> +				  union perf_event *event __maybe_unused,
> +				  struct perf_sample *sample __maybe_unused,
> +				  struct perf_tool *tool __maybe_unused)
> +{
> +	return 0;
> +}
> +
> +static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
> +					   union perf_event *event,
> +					   struct perf_tool *tool __maybe_unused)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
> +					    auxtrace);
> +	struct auxtrace_buffer *buffer;
> +	off_t data_offset;
> +	int fd = perf_data__fd(session->data);
> +	int err;
> +
> +	if (perf_data__is_pipe(session->data)) {
> +		data_offset = 0;
> +	} else {
> +		data_offset = lseek(fd, 0, SEEK_CUR);
> +		if (data_offset == -1)
> +			return -errno;
> +	}
> +
> +	err = auxtrace_queues__add_event(&ptt->queues, session, event,
> +					 data_offset, &buffer);
> +	if (err)
> +		return err;
> +
> +	if (dump_trace) {
> +		if (auxtrace_buffer__get_data(buffer, fd)) {
> +			hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
> +			auxtrace_buffer__put_data(buffer);
> +		}

Technically auxtrace_queues aren't required here because they are more for
supporting trace from multiple CPUs and sorting and re-ordering between them.

If this is new device is uncore and always from a single source you could
just go straight to  hisi_ptt_dump_event() with data_offset and size of the
auxtrace event.

But I suppose it also doesn't hurt to use some of the existing framework
like you have done.

> +	}
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
> +			  struct perf_tool *tool __maybe_unused)
> +{
> +	return 0;
> +}
> +
> +static void hisi_ptt_free_events(struct perf_session *session)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
> +					    auxtrace);
> +	struct auxtrace_queues *queues = &ptt->queues;
> +	unsigned int i;
> +
> +	for (i = 0; i < queues->nr_queues; i++) {
> +		free(queues->queue_array[i].priv);
> +		queues->queue_array[i].priv = NULL;
> +	}
> +	auxtrace_queues__free(queues);
> +}
> +
> +static void hisi_ptt_free(struct perf_session *session)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
> +					    auxtrace);
> +
> +	hisi_ptt_free_events(session);
> +	session->auxtrace = NULL;
> +	free(ptt);
> +}
> +
> +static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
> +				       struct evsel *evsel)
> +{
> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
> +
> +	return evsel->core.attr.type == ptt->pmu_type;
> +}
> +
> +static const char * const hisi_ptt_info_fmts[] = {
> +	[HISI_PTT_PMU_TYPE]		= "  PMU Type           %" PRId64 "\n",
> +};
> +
> +static void hisi_ptt_print_info(__u64 *arr)
> +{
> +	if (!dump_trace)
> +		return;
> +
> +	fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
> +}
> +
> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
> +				   struct perf_session *session)
> +{
> +	struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
> +	struct hisi_ptt *ptt;
> +	int err;
> +
> +	if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
> +				sizeof(struct perf_record_auxtrace_info))
> +		return -EINVAL;
> +
> +	ptt = zalloc(sizeof(struct hisi_ptt));
> +	if (!ptt)
> +		return -ENOMEM;
> +
> +	err = auxtrace_queues__init(&ptt->queues);
> +	if (err)
> +		goto err_free;
> +
> +	ptt->session = session;
> +	ptt->machine = &session->machines.host; /* No kvm support */
> +	ptt->auxtrace_type = auxtrace_info->type;
> +	ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
> +
> +	ptt->auxtrace.process_event = hisi_ptt_process_event;
> +	ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
> +	ptt->auxtrace.flush_events = hisi_ptt_flush;
> +	ptt->auxtrace.free_events = hisi_ptt_free_events;
> +	ptt->auxtrace.free = hisi_ptt_free;
> +	ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
> +	session->auxtrace = &ptt->auxtrace;
> +
> +	hisi_ptt_print_info(&auxtrace_info->priv[0]);
> +
> +	return 0;
> +
> +err_free:
> +	free(ptt);
> +	return err;
> +}
> diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
> new file mode 100644
> index 000000000000..c0b6cbde1221
> --- /dev/null
> +++ b/tools/perf/util/hisi_ptt.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * HiSilicon PCIe Trace and Tuning (PTT) support
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + */
> +
> +#ifndef INCLUDE__PERF_HISI_PTT_H__
> +#define INCLUDE__PERF_HISI_PTT_H__
> +
> +#define HISI_PTT_PMU_NAME "hisi_ptt"
> +enum {
> +	HISI_PTT_PMU_TYPE,
> +	HISI_PTT_PER_CPU_MMAPS,
> +	HISI_PTT_AUXTRACE_PRIV_MAX,
> +};
> +
> +#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
> +union perf_event;
> +struct perf_session;
> +struct perf_pmu;
> +
> +struct auxtrace_record *hisi_ptt_recording_init(int *err,
> +						struct perf_pmu *hisi_ptt_pmu);
> +
> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
> +				   struct perf_session *session);
> +
> +#endif

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-05-10  9:46   ` James Clark
@ 2022-05-10 11:18     ` Yicong Yang
  2022-05-10 12:54       ` James Clark
  0 siblings, 1 reply; 30+ messages in thread
From: Yicong Yang @ 2022-05-10 11:18 UTC (permalink / raw)
  To: James Clark
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, gregkh, helgaas,
	alexander.shishkin, lorenzo.pieralisi, will, mark.rutland,
	mathieu.poirier, suzuki.poulose, mike.leach, leo.yan,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu

On 2022/5/10 17:46, James Clark wrote:
> 
> 
> On 07/04/2022 13:58, Yicong Yang wrote:
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
>> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
>> tune the PCIe traffic, and trace the TLP headers.
>>
>> Add the driver for the device to enable the trace function. Register PMU
>> device of PTT trace, then users can use trace through perf command. The
>> driver makes use of perf AUX trace and support following events to
>> configure the trace:
>>
>> - filter: select Root port or Endpoint to trace
>> - type: select the type of traced TLP headers
>> - direction: select the direction of traced TLP headers
>> - format: select the data format of the traced TLP headers
>>
>> This patch adds the driver part of PTT trace. The perf command support of
>> PTT trace is added in the following patch.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
>>  drivers/Makefile                 |   1 +
>>  drivers/hwtracing/Kconfig        |   2 +
>>  drivers/hwtracing/ptt/Kconfig    |  12 +
>>  drivers/hwtracing/ptt/Makefile   |   2 +
>>  drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
>>  drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
>>  6 files changed, 1057 insertions(+)
>>  create mode 100644 drivers/hwtracing/ptt/Kconfig
>>  create mode 100644 drivers/hwtracing/ptt/Makefile
>>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>>
>> diff --git a/drivers/Makefile b/drivers/Makefile
>> index 020780b6b4d2..662d50599467 100644
>> --- a/drivers/Makefile
>> +++ b/drivers/Makefile
>> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>  obj-y				+= hwtracing/intel_th/
>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>> +obj-$(CONFIG_HISI_PTT)		+= hwtracing/ptt/
>>  obj-$(CONFIG_ANDROID)		+= android/
>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>  obj-$(CONFIG_FPGA)		+= fpga/
>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>> index 13085835a636..911ee977103c 100644
>> --- a/drivers/hwtracing/Kconfig
>> +++ b/drivers/hwtracing/Kconfig
>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>  
>>  source "drivers/hwtracing/intel_th/Kconfig"
>>  
>> +source "drivers/hwtracing/ptt/Kconfig"
>> +
>>  endmenu
>> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
>> new file mode 100644
>> index 000000000000..8902a6f27563
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/Kconfig
>> @@ -0,0 +1,12 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config HISI_PTT
>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>> +	depends on ARM64 || (COMPILE_TEST && 64BIT)
>> +	depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
>> +	help
>> +	  HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
>> +	  device, and it provides support for PCIe traffic tuning and
>> +	  tracing TLP headers to the memory.
>> +
>> +	  This driver can also be built as a module. If so, the module
>> +	  will be called hisi_ptt.
>> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
>> new file mode 100644
>> index 000000000000..908c09a98161
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
>> new file mode 100644
>> index 000000000000..242b41870380
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
>> @@ -0,0 +1,874 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for HiSilicon PCIe tune and trace device
>> + *
>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>> + * Author: Yicong Yang <yangyicong@hisilicon.com>
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/bitops.h>
>> +#include <linux/delay.h>
>> +#include <linux/dma-iommu.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io.h>
>> +#include <linux/iommu.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/module.h>
>> +#include <linux/sysfs.h>
>> +#include <linux/vmalloc.h>
>> +
>> +#include "hisi_ptt.h"
>> +
>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>> +{
>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>> +
>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>> +}
>> +
>> +static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>> +{
>> +	u32 val;
>> +
>> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
>> +					  val, val & HISI_PTT_TRACE_IDLE,
>> +					  HISI_PTT_WAIT_POLL_INTERVAL_US,
>> +					  HISI_PTT_WAIT_TRACE_TIMEOUT_US);
>> +}
>> +
>> +static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
>> +{
>> +	u32 val;
>> +
>> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
>> +					  val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
>> +					  HISI_PTT_RESET_TIMEOUT_US);
>> +}
>> +
>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	int i;
>> +
>> +	if (!ctrl->trace_buf)
>> +		return;
>> +
>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>> +		if (ctrl->trace_buf[i].addr)
>> +			dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>> +					   ctrl->trace_buf[i].addr,
>> +					   ctrl->trace_buf[i].dma);
>> +	}
>> +
>> +	devm_kfree(dev, ctrl->trace_buf);
>> +	ctrl->trace_buf = NULL;
>> +}
>> +
>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	int i;
>> +
>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>> +
>> +	/* If the trace buffer has already been allocated, zero it. */
>> +	if (ctrl->trace_buf) {
>> +		for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
>> +			memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>> +		return 0;
>> +	}
>> +
>> +	ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
>> +				       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
>> +	if (!ctrl->trace_buf)
>> +		return -ENOMEM;
>> +
>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>> +		ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>> +							     &ctrl->trace_buf[i].dma,
>> +							     GFP_KERNEL);
>> +		if (!ctrl->trace_buf[i].addr) {
>> +			hisi_ptt_free_trace_buf(hisi_ptt);
>> +			return -ENOMEM;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>> +{
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	hisi_ptt->trace_ctrl.started = false;
>> +}
>> +
>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	u32 val;
>> +	int i;
>> +
>> +	/* Check device idle before start trace */
>> +	if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
>> +		return -EBUSY;
>> +	}
>> +
>> +	ctrl->started = true;
>> +
>> +	/* Reset the DMA before start tracing */
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	hisi_ptt_wait_dma_reset_done(hisi_ptt);
>> +
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	/* Clear the interrupt status */
>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>> +
>> +	/* Configure the trace DMA buffer */
>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>> +		writel(lower_32_bits(ctrl->trace_buf[i].dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
>> +		writel(upper_32_bits(ctrl->trace_buf[i].dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
>> +	}
>> +	writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>> +
>> +	/* Set the trace control register */
>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>> +	if (!hisi_ptt->trace_ctrl.is_port)
>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>> +
>> +	/* Start the Trace */
>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct perf_output_handle *handle = &ctrl->handle;
>> +	struct perf_event *event = handle->event;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	void *addr;
>> +
>> +	buf = perf_get_aux(handle);
>> +	if (!buf || !handle->size)
>> +		return -EINVAL;
>> +
>> +	addr = ctrl->trace_buf[ctrl->buf_index].addr;
>> +
>> +	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
>> +	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);
> 
> Hi Kicong,
> 
> I also have the same comment as Leo here, I don't think the memset is
> required.
> 

It's necessary in the current approach as we always commit HISI_PTT_TRACE_BUF_SIZE
data but the buffer maybe partly filled (called when perf going to stopp, not by the
interrupt). The buffer is cleared so the unfilled part of the buffer will have
empty data (normal traced TLP headers won't be all 0), then the user can distinguish
the valid part of the data.

I'm trying to only copy the traced data rather than the whole buffer then the
clear operation here will be unnecessary. The hardware provide a register indicating
which offset of which buffer it's currently writing to and it canbe used here.

>> +	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
>> +
>> +	if (stop) {
>> +		perf_aux_output_end(handle, buf->pos);
>> +	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> 
> perf_aux_output_skip() can also return an error so should probably also
> be checked like perf_aux_output_begin()
> 

ok it should be checked.

> I'm also wondering why there is a skip for every output_end()? Is that
> to avoid having two memcpy calls to handle the wrap around if the data
> to be copied goes past the end of the aux buffer?
> 
> For example if your buffers are 4MB each and the aux buffer that the
> user picked isn't a multiple of 4 I can see you needing to write the
> first part of the 4MB to the end of the aux buffer and then the last
> part to the beginning which would be two memcpy() calls. And then a
> skip wouldn't be required.
> 

I intended to handle the case that AUX buffer is not a multiple of 4 MiB.
When the resident AUX buffer size is less than 4MiB, we're not going to
commit data to it and will apply a new AUX buffer instead. I think you're
right that the perf_aux_output_skip() is unnecessary here. Thanks for
catching this.

> I looked at all the other uses of perf_output_end() and perf_output_skip()
> in the kernel and didn't see a pattern like yours so it seems suspicous to
> me. Maybe at least some comments around this section are needed.
> 

Will add some comments of the handling here.

Regards,
Yicong

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
  2022-04-07 12:58 ` [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
@ 2022-05-10 11:23   ` Will Deacon
  2022-05-10 12:43     ` Yicong Yang
  2022-05-11 11:42   ` John Garry
  1 sibling, 1 reply; 30+ messages in thread
From: Will Deacon @ 2022-05-10 11:23 UTC (permalink / raw)
  To: Yicong Yang
  Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
	linuxarm

On Thu, Apr 07, 2022 at 08:58:35PM +0800, Yicong Yang wrote:
> The DMA operations of HiSilicon PTT device can only work properly with
> identical mappings. So add a quirk for the device to force the domain
> as passthrough.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)

I still don't like this being part of the SMMU driver, but given that
(a) Robin doesn't seem to agree with the objection and (b) you've been
refreshing the patch series:

Acked-by: Will Deacon <will@kernel.org>

If you do respin, then:

> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 627a3ed5ee8f..5ec15ae2a9b1 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2839,6 +2839,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>  	}
>  }

It might be worth adding a brief comment here to explain what this device is
and why it needs an identity mapping.

> +#define IS_HISI_PTT_DEVICE(pdev)	((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
> +					 (pdev)->device == 0xa12e)

Will

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
  2022-05-10 11:23   ` Will Deacon
@ 2022-05-10 12:43     ` Yicong Yang
  0 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-05-10 12:43 UTC (permalink / raw)
  To: Will Deacon
  Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
	linuxarm

On 2022/5/10 19:23, Will Deacon wrote:
> On Thu, Apr 07, 2022 at 08:58:35PM +0800, Yicong Yang wrote:
>> The DMA operations of HiSilicon PTT device can only work properly with
>> identical mappings. So add a quirk for the device to force the domain
>> as passthrough.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>>  1 file changed, 16 insertions(+)
> 
> I still don't like this being part of the SMMU driver, but given that
> (a) Robin doesn't seem to agree with the objection and (b) you've been
> refreshing the patch series:
> 
> Acked-by: Will Deacon <will@kernel.org>
> 
> If you do respin, then:
> 
>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>> index 627a3ed5ee8f..5ec15ae2a9b1 100644
>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>> @@ -2839,6 +2839,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>>  	}
>>  }
> 
> It might be worth adding a brief comment here to explain what this device is
> and why it needs an identity mapping.
> 
>> +#define IS_HISI_PTT_DEVICE(pdev)	((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
>> +					 (pdev)->device == 0xa12e)
> 

Will add a brief comment here in next version.

Thanks,
Yicong

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-05-10 11:18     ` Yicong Yang
@ 2022-05-10 12:54       ` James Clark
  2022-05-10 13:17         ` Yicong Yang
  0 siblings, 1 reply; 30+ messages in thread
From: James Clark @ 2022-05-10 12:54 UTC (permalink / raw)
  To: Yicong Yang, Leo Yan
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, gregkh, helgaas,
	alexander.shishkin, lorenzo.pieralisi, will, mark.rutland,
	mathieu.poirier, suzuki.poulose, mike.leach, leo.yan,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu



On 10/05/2022 12:18, Yicong Yang wrote:
> On 2022/5/10 17:46, James Clark wrote:
>>
>>
>> On 07/04/2022 13:58, Yicong Yang wrote:
>>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
>>> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
>>> tune the PCIe traffic, and trace the TLP headers.
>>>
>>> Add the driver for the device to enable the trace function. Register PMU
>>> device of PTT trace, then users can use trace through perf command. The
>>> driver makes use of perf AUX trace and support following events to
>>> configure the trace:
>>>
>>> - filter: select Root port or Endpoint to trace
>>> - type: select the type of traced TLP headers
>>> - direction: select the direction of traced TLP headers
>>> - format: select the data format of the traced TLP headers
>>>
>>> This patch adds the driver part of PTT trace. The perf command support of
>>> PTT trace is added in the following patch.
>>>
>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>> ---
>>>  drivers/Makefile                 |   1 +
>>>  drivers/hwtracing/Kconfig        |   2 +
>>>  drivers/hwtracing/ptt/Kconfig    |  12 +
>>>  drivers/hwtracing/ptt/Makefile   |   2 +
>>>  drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
>>>  drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
>>>  6 files changed, 1057 insertions(+)
>>>  create mode 100644 drivers/hwtracing/ptt/Kconfig
>>>  create mode 100644 drivers/hwtracing/ptt/Makefile
>>>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>>>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>>>
>>> diff --git a/drivers/Makefile b/drivers/Makefile
>>> index 020780b6b4d2..662d50599467 100644
>>> --- a/drivers/Makefile
>>> +++ b/drivers/Makefile
>>> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>>  obj-y				+= hwtracing/intel_th/
>>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>>> +obj-$(CONFIG_HISI_PTT)		+= hwtracing/ptt/
>>>  obj-$(CONFIG_ANDROID)		+= android/
>>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>>  obj-$(CONFIG_FPGA)		+= fpga/
>>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>>> index 13085835a636..911ee977103c 100644
>>> --- a/drivers/hwtracing/Kconfig
>>> +++ b/drivers/hwtracing/Kconfig
>>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>>  
>>>  source "drivers/hwtracing/intel_th/Kconfig"
>>>  
>>> +source "drivers/hwtracing/ptt/Kconfig"
>>> +
>>>  endmenu
>>> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
>>> new file mode 100644
>>> index 000000000000..8902a6f27563
>>> --- /dev/null
>>> +++ b/drivers/hwtracing/ptt/Kconfig
>>> @@ -0,0 +1,12 @@
>>> +# SPDX-License-Identifier: GPL-2.0-only
>>> +config HISI_PTT
>>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>>> +	depends on ARM64 || (COMPILE_TEST && 64BIT)
>>> +	depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
>>> +	help
>>> +	  HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
>>> +	  device, and it provides support for PCIe traffic tuning and
>>> +	  tracing TLP headers to the memory.
>>> +
>>> +	  This driver can also be built as a module. If so, the module
>>> +	  will be called hisi_ptt.
>>> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
>>> new file mode 100644
>>> index 000000000000..908c09a98161
>>> --- /dev/null
>>> +++ b/drivers/hwtracing/ptt/Makefile
>>> @@ -0,0 +1,2 @@
>>> +# SPDX-License-Identifier: GPL-2.0
>>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>>> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
>>> new file mode 100644
>>> index 000000000000..242b41870380
>>> --- /dev/null
>>> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
>>> @@ -0,0 +1,874 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * Driver for HiSilicon PCIe tune and trace device
>>> + *
>>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>>> + * Author: Yicong Yang <yangyicong@hisilicon.com>
>>> + */
>>> +
>>> +#include <linux/bitfield.h>
>>> +#include <linux/bitops.h>
>>> +#include <linux/delay.h>
>>> +#include <linux/dma-iommu.h>
>>> +#include <linux/dma-mapping.h>
>>> +#include <linux/interrupt.h>
>>> +#include <linux/io.h>
>>> +#include <linux/iommu.h>
>>> +#include <linux/iopoll.h>
>>> +#include <linux/module.h>
>>> +#include <linux/sysfs.h>
>>> +#include <linux/vmalloc.h>
>>> +
>>> +#include "hisi_ptt.h"
>>> +
>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>> +{
>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>> +
>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>> +}
>>> +
>>> +static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +	u32 val;
>>> +
>>> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
>>> +					  val, val & HISI_PTT_TRACE_IDLE,
>>> +					  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>> +					  HISI_PTT_WAIT_TRACE_TIMEOUT_US);
>>> +}
>>> +
>>> +static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +	u32 val;
>>> +
>>> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
>>> +					  val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
>>> +					  HISI_PTT_RESET_TIMEOUT_US);
>>> +}
>>> +
>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>> +	int i;
>>> +
>>> +	if (!ctrl->trace_buf)
>>> +		return;
>>> +
>>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>>> +		if (ctrl->trace_buf[i].addr)
>>> +			dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>>> +					   ctrl->trace_buf[i].addr,
>>> +					   ctrl->trace_buf[i].dma);
>>> +	}
>>> +
>>> +	devm_kfree(dev, ctrl->trace_buf);
>>> +	ctrl->trace_buf = NULL;
>>> +}
>>> +
>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>> +	int i;
>>> +
>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>> +
>>> +	/* If the trace buffer has already been allocated, zero it. */
>>> +	if (ctrl->trace_buf) {
>>> +		for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
>>> +			memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>>> +		return 0;
>>> +	}
>>> +
>>> +	ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
>>> +				       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
>>> +	if (!ctrl->trace_buf)
>>> +		return -ENOMEM;
>>> +
>>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>>> +		ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>>> +							     &ctrl->trace_buf[i].dma,
>>> +							     GFP_KERNEL);
>>> +		if (!ctrl->trace_buf[i].addr) {
>>> +			hisi_ptt_free_trace_buf(hisi_ptt);
>>> +			return -ENOMEM;
>>> +		}
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +	hisi_ptt->trace_ctrl.started = false;
>>> +}
>>> +
>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> +	u32 val;
>>> +	int i;
>>> +
>>> +	/* Check device idle before start trace */
>>> +	if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
>>> +		return -EBUSY;
>>> +	}
>>> +
>>> +	ctrl->started = true;
>>> +
>>> +	/* Reset the DMA before start tracing */
>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> +	hisi_ptt_wait_dma_reset_done(hisi_ptt);
>>> +
>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> +	/* Clear the interrupt status */
>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>> +
>>> +	/* Configure the trace DMA buffer */
>>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>>> +		writel(lower_32_bits(ctrl->trace_buf[i].dma),
>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
>>> +		writel(upper_32_bits(ctrl->trace_buf[i].dma),
>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
>>> +	}
>>> +	writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>> +
>>> +	/* Set the trace control register */
>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>> +
>>> +	/* Start the Trace */
>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>> +{
>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> +	struct perf_output_handle *handle = &ctrl->handle;
>>> +	struct perf_event *event = handle->event;
>>> +	struct hisi_ptt_pmu_buf *buf;
>>> +	void *addr;
>>> +
>>> +	buf = perf_get_aux(handle);
>>> +	if (!buf || !handle->size)
>>> +		return -EINVAL;
>>> +
>>> +	addr = ctrl->trace_buf[ctrl->buf_index].addr;
>>> +
>>> +	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
>>> +	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>>
>> Hi Kicong,
>>
>> I also have the same comment as Leo here, I don't think the memset is
>> required.
>>
> 
> It's necessary in the current approach as we always commit HISI_PTT_TRACE_BUF_SIZE
> data but the buffer maybe partly filled (called when perf going to stopp, not by the
> interrupt). The buffer is cleared so the unfilled part of the buffer will have
> empty data (normal traced TLP headers won't be all 0), then the user can distinguish
> the valid part of the data.
> 
> I'm trying to only copy the traced data rather than the whole buffer then the
> clear operation here will be unnecessary. The hardware provide a register indicating
> which offset of which buffer it's currently writing to and it canbe used here.

If only the traced data is copied rather than the full buffer, isn't that what
perf_aux_output_end() is for? Perf will only read up to the point where you
say the buffer is filled to, it won't go and read the zeros if you didn't tell
it to by emitting perf_aux_output_end() for more data than was written.

If you are having to write zeros to detect which bits of the buffer is filled
or not it sounds like those zero parts are making it into the perf file and are
wasting disk space and CPU cycles to copy them.

> 
>>> +	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
>>> +
>>> +	if (stop) {
>>> +		perf_aux_output_end(handle, buf->pos);
>>> +	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>
>> perf_aux_output_skip() can also return an error so should probably also
>> be checked like perf_aux_output_begin()
>>
> 
> ok it should be checked.
> 
>> I'm also wondering why there is a skip for every output_end()? Is that
>> to avoid having two memcpy calls to handle the wrap around if the data
>> to be copied goes past the end of the aux buffer?
>>
>> For example if your buffers are 4MB each and the aux buffer that the
>> user picked isn't a multiple of 4 I can see you needing to write the
>> first part of the 4MB to the end of the aux buffer and then the last
>> part to the beginning which would be two memcpy() calls. And then a
>> skip wouldn't be required.
>>
> 
> I intended to handle the case that AUX buffer is not a multiple of 4 MiB.
> When the resident AUX buffer size is less than 4MiB, we're not going to
> commit data to it and will apply a new AUX buffer instead. I think you're
> right that the perf_aux_output_skip() is unnecessary here. Thanks for
> catching this.
> 
>> I looked at all the other uses of perf_output_end() and perf_output_skip()
>> in the kernel and didn't see a pattern like yours so it seems suspicous to
>> me. Maybe at least some comments around this section are needed.
>>
> 
> Will add some comments of the handling here.
> 
> Regards,
> Yicong

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
  2022-05-10 12:54       ` James Clark
@ 2022-05-10 13:17         ` Yicong Yang
  0 siblings, 0 replies; 30+ messages in thread
From: Yicong Yang @ 2022-05-10 13:17 UTC (permalink / raw)
  To: James Clark, Leo Yan
  Cc: yangyicong, prime.zeng, liuqi115, zhangshaokun, linuxarm, gregkh,
	helgaas, alexander.shishkin, lorenzo.pieralisi, will,
	mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
	jonathan.cameron, daniel.thompson, joro, john.garry,
	shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
	linux-kernel, linux-arm-kernel, coresight, linux-pci,
	linux-perf-users, iommu

On 2022/5/10 20:54, James Clark wrote:
> 
> 
> On 10/05/2022 12:18, Yicong Yang wrote:
>> On 2022/5/10 17:46, James Clark wrote:
>>>
>>>
>>> On 07/04/2022 13:58, Yicong Yang wrote:
>>>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex integrated
>>>> Endpoint(RCiEP) device, providing the capability to dynamically monitor and
>>>> tune the PCIe traffic, and trace the TLP headers.
>>>>
>>>> Add the driver for the device to enable the trace function. Register PMU
>>>> device of PTT trace, then users can use trace through perf command. The
>>>> driver makes use of perf AUX trace and support following events to
>>>> configure the trace:
>>>>
>>>> - filter: select Root port or Endpoint to trace
>>>> - type: select the type of traced TLP headers
>>>> - direction: select the direction of traced TLP headers
>>>> - format: select the data format of the traced TLP headers
>>>>
>>>> This patch adds the driver part of PTT trace. The perf command support of
>>>> PTT trace is added in the following patch.
>>>>
>>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>>> ---
>>>>  drivers/Makefile                 |   1 +
>>>>  drivers/hwtracing/Kconfig        |   2 +
>>>>  drivers/hwtracing/ptt/Kconfig    |  12 +
>>>>  drivers/hwtracing/ptt/Makefile   |   2 +
>>>>  drivers/hwtracing/ptt/hisi_ptt.c | 874 +++++++++++++++++++++++++++++++
>>>>  drivers/hwtracing/ptt/hisi_ptt.h | 166 ++++++
>>>>  6 files changed, 1057 insertions(+)
>>>>  create mode 100644 drivers/hwtracing/ptt/Kconfig
>>>>  create mode 100644 drivers/hwtracing/ptt/Makefile
>>>>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>>>>  create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>>>>
>>>> diff --git a/drivers/Makefile b/drivers/Makefile
>>>> index 020780b6b4d2..662d50599467 100644
>>>> --- a/drivers/Makefile
>>>> +++ b/drivers/Makefile
>>>> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>>>  obj-y				+= hwtracing/intel_th/
>>>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>>>> +obj-$(CONFIG_HISI_PTT)		+= hwtracing/ptt/
>>>>  obj-$(CONFIG_ANDROID)		+= android/
>>>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>>>  obj-$(CONFIG_FPGA)		+= fpga/
>>>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>>>> index 13085835a636..911ee977103c 100644
>>>> --- a/drivers/hwtracing/Kconfig
>>>> +++ b/drivers/hwtracing/Kconfig
>>>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>>>  
>>>>  source "drivers/hwtracing/intel_th/Kconfig"
>>>>  
>>>> +source "drivers/hwtracing/ptt/Kconfig"
>>>> +
>>>>  endmenu
>>>> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..8902a6f27563
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/ptt/Kconfig
>>>> @@ -0,0 +1,12 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +config HISI_PTT
>>>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>>>> +	depends on ARM64 || (COMPILE_TEST && 64BIT)
>>>> +	depends on PCI && HAS_DMA && HAS_IOMEM && PERF_EVENTS
>>>> +	help
>>>> +	  HiSilicon PCIe Tune and Trace Device exists as a PCIe RCiEP
>>>> +	  device, and it provides support for PCIe traffic tuning and
>>>> +	  tracing TLP headers to the memory.
>>>> +
>>>> +	  This driver can also be built as a module. If so, the module
>>>> +	  will be called hisi_ptt.
>>>> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
>>>> new file mode 100644
>>>> index 000000000000..908c09a98161
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/ptt/Makefile
>>>> @@ -0,0 +1,2 @@
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>>>> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
>>>> new file mode 100644
>>>> index 000000000000..242b41870380
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
>>>> @@ -0,0 +1,874 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * Driver for HiSilicon PCIe tune and trace device
>>>> + *
>>>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>>>> + * Author: Yicong Yang <yangyicong@hisilicon.com>
>>>> + */
>>>> +
>>>> +#include <linux/bitfield.h>
>>>> +#include <linux/bitops.h>
>>>> +#include <linux/delay.h>
>>>> +#include <linux/dma-iommu.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/io.h>
>>>> +#include <linux/iommu.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/module.h>
>>>> +#include <linux/sysfs.h>
>>>> +#include <linux/vmalloc.h>
>>>> +
>>>> +#include "hisi_ptt.h"
>>>> +
>>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>>> +{
>>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>>> +
>>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>>> +}
>>>> +
>>>> +static bool hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	u32 val;
>>>> +
>>>> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_STS,
>>>> +					  val, val & HISI_PTT_TRACE_IDLE,
>>>> +					  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>>> +					  HISI_PTT_WAIT_TRACE_TIMEOUT_US);
>>>> +}
>>>> +
>>>> +static bool hisi_ptt_wait_dma_reset_done(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	u32 val;
>>>> +
>>>> +	return !readl_poll_timeout_atomic(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS,
>>>> +					  val, !val, HISI_PTT_RESET_POLL_INTERVAL_US,
>>>> +					  HISI_PTT_RESET_TIMEOUT_US);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	int i;
>>>> +
>>>> +	if (!ctrl->trace_buf)
>>>> +		return;
>>>> +
>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>>>> +		if (ctrl->trace_buf[i].addr)
>>>> +			dmam_free_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>>>> +					   ctrl->trace_buf[i].addr,
>>>> +					   ctrl->trace_buf[i].dma);
>>>> +	}
>>>> +
>>>> +	devm_kfree(dev, ctrl->trace_buf);
>>>> +	ctrl->trace_buf = NULL;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	int i;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>>> +
>>>> +	/* If the trace buffer has already been allocated, zero it. */
>>>> +	if (ctrl->trace_buf) {
>>>> +		for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++)
>>>> +			memset(ctrl->trace_buf[i].addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	ctrl->trace_buf = devm_kcalloc(dev, HISI_PTT_TRACE_BUF_CNT,
>>>> +				       sizeof(struct hisi_ptt_dma_buffer), GFP_KERNEL);
>>>> +	if (!ctrl->trace_buf)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>>>> +		ctrl->trace_buf[i].addr = dmam_alloc_coherent(dev, HISI_PTT_TRACE_BUF_SIZE,
>>>> +							     &ctrl->trace_buf[i].dma,
>>>> +							     GFP_KERNEL);
>>>> +		if (!ctrl->trace_buf[i].addr) {
>>>> +			hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +			return -ENOMEM;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	hisi_ptt->trace_ctrl.started = false;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	u32 val;
>>>> +	int i;
>>>> +
>>>> +	/* Check device idle before start trace */
>>>> +	if (!hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy\n");
>>>> +		return -EBUSY;
>>>> +	}
>>>> +
>>>> +	ctrl->started = true;
>>>> +
>>>> +	/* Reset the DMA before start tracing */
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	hisi_ptt_wait_dma_reset_done(hisi_ptt);
>>>> +
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	/* Clear the interrupt status */
>>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>> +
>>>> +	/* Configure the trace DMA buffer */
>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; i++) {
>>>> +		writel(lower_32_bits(ctrl->trace_buf[i].dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>>> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +		writel(upper_32_bits(ctrl->trace_buf[i].dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>>> +		       i * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +	}
>>>> +	writel(HISI_PTT_TRACE_BUF_SIZE, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>>> +
>>>> +	/* Set the trace control register */
>>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>>> +
>>>> +	/* Start the Trace */
>>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct perf_output_handle *handle = &ctrl->handle;
>>>> +	struct perf_event *event = handle->event;
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +	void *addr;
>>>> +
>>>> +	buf = perf_get_aux(handle);
>>>> +	if (!buf || !handle->size)
>>>> +		return -EINVAL;
>>>> +
>>>> +	addr = ctrl->trace_buf[ctrl->buf_index].addr;
>>>> +
>>>> +	memcpy(buf->base + buf->pos, addr, HISI_PTT_TRACE_BUF_SIZE);
>>>> +	memset(addr, 0, HISI_PTT_TRACE_BUF_SIZE);
>>>
>>> Hi Kicong,
>>>
>>> I also have the same comment as Leo here, I don't think the memset is
>>> required.
>>>
>>
>> It's necessary in the current approach as we always commit HISI_PTT_TRACE_BUF_SIZE
>> data but the buffer maybe partly filled (called when perf going to stopp, not by the
>> interrupt). The buffer is cleared so the unfilled part of the buffer will have
>> empty data (normal traced TLP headers won't be all 0), then the user can distinguish
>> the valid part of the data.
>>
>> I'm trying to only copy the traced data rather than the whole buffer then the
>> clear operation here will be unnecessary. The hardware provide a register indicating
>> which offset of which buffer it's currently writing to and it canbe used here.
> 
> If only the traced data is copied rather than the full buffer, isn't that what
> perf_aux_output_end() is for? Perf will only read up to the point where you
> say the buffer is filled to, it won't go and read the zeros if you didn't tell
> it to by emitting perf_aux_output_end() for more data than was written.
> 
> If you are having to write zeros to detect which bits of the buffer is filled
> or not it sounds like those zero parts are making it into the perf file and are
> wasting disk space and CPU cycles to copy them.
> 

For the approach in this version, you're right that we'll waste CPU cycles to copy
them. But this will only happens when the perf session is going stop, as in the
updating from interrupts we'll always have data of full buffer size (interrupt only
comes when the buffer is filled fully). The data will be like (perf report -D):

[...]
.  00000004: 00 00 00 00                                 Prefix
.  00000008: 01 80 00 60                                 Header DW0
.  0000000c: 0f 1e 00 01                                 Header DW1
.  00000010: c6 00 00 00                                 Header DW2
.  00000014: 40 00 01 60                                 Header DW3
.  0000001c: c5 94 d1 00                                 Time
.  0000005c: 14 00 00 00                                 Time
.  00000064: 00 00 00 00                                 Prefix
.  00000068: 00 00 00 00                                 Header DW0
.  0000006c: 00 00 00 00                                 Header DW1
.  00000070: 00 00 00 00                                 Header DW2
.  0000005c: 00 00 00 00                                 Time
[...zeroed data]

I'll change to only copy the valid data like below in next version. Then we won't
waste disk space and cpu time to copy it.

static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
{
	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
	struct perf_output_handle *handle = &ctrl->handle;
	struct perf_event *event = handle->event;
	struct hisi_ptt_pmu_buf *buf;
	void *addr;
	size_t sz;

	buf = perf_get_aux(handle);
	if (!buf || !handle->size)
		return -EINVAL;

	addr = ctrl->trace_buf[ctrl->buf_index].addr;

	if (stop) {
		u32 reg;

		reg = readl(hisi_ptt->iobase + HISI_PTT_TRACE_WR_STS);
		sz = FIELD_GET(HISI_PTT_TRACE_WR_STS_WRITE, reg);
	} else {
		sz = HISI_PTT_TRACE_BUF_SIZE;
	}

	memcpy(buf->base + buf->pos, addr, sz);
	buf->pos += sz;

	/*
	 * Just commit the traced data if we're going to stop. Otherwise if the
	 * resident AUX buffer cannot contain data of next trace buffer, apply
	 * a new one.
	 */
	if (stop) {
		perf_aux_output_end(handle, buf->pos);
	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
		perf_aux_output_end(handle, buf->pos);

		buf = perf_aux_output_begin(handle, event);
		if (!buf)
			return -EINVAL;

		buf->pos = handle->head % buf->length;
		if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
			perf_aux_output_end(handle, 0);
			return -EINVAL;
		}
	}

	return 0;
}

>>
>>>> +	buf->pos += HISI_PTT_TRACE_BUF_SIZE;
>>>> +
>>>> +	if (stop) {
>>>> +		perf_aux_output_end(handle, buf->pos);
>>>> +	} else if (buf->length - buf->pos < HISI_PTT_TRACE_BUF_SIZE) {
>>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>>
>>> perf_aux_output_skip() can also return an error so should probably also
>>> be checked like perf_aux_output_begin()
>>>
>>
>> ok it should be checked.
>>
>>> I'm also wondering why there is a skip for every output_end()? Is that
>>> to avoid having two memcpy calls to handle the wrap around if the data
>>> to be copied goes past the end of the aux buffer?
>>>
>>> For example if your buffers are 4MB each and the aux buffer that the
>>> user picked isn't a multiple of 4 I can see you needing to write the
>>> first part of the 4MB to the end of the aux buffer and then the last
>>> part to the beginning which would be two memcpy() calls. And then a
>>> skip wouldn't be required.
>>>
>>
>> I intended to handle the case that AUX buffer is not a multiple of 4 MiB.
>> When the resident AUX buffer size is less than 4MiB, we're not going to
>> commit data to it and will apply a new AUX buffer instead. I think you're
>> right that the perf_aux_output_skip() is unnecessary here. Thanks for
>> catching this.
>>
>>> I looked at all the other uses of perf_output_end() and perf_output_skip()
>>> in the kernel and didn't see a pattern like yours so it seems suspicous to
>>> me. Maybe at least some comments around this section are needed.
>>>
>>
>> Will add some comments of the handling here.
>>
>> Regards,
>> Yicong
> .
> 

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-05-10 10:14   ` James Clark
@ 2022-05-11  2:02     ` liuqi (BA)
  2022-05-11  9:18       ` James Clark
  0 siblings, 1 reply; 30+ messages in thread
From: liuqi (BA) @ 2022-05-11  2:02 UTC (permalink / raw)
  To: James Clark, Yicong Yang, Leo Yan
  Cc: prime.zeng, zhangshaokun, linuxarm, gregkh, helgaas,
	alexander.shishkin, lorenzo.pieralisi, will, mark.rutland,
	mathieu.poirier, suzuki.poulose, mike.leach, jonathan.cameron,
	daniel.thompson, joro, john.garry, shameerali.kolothum.thodi,
	robin.murphy, peterz, mingo, acme, linux-kernel,
	linux-arm-kernel, coresight, linux-pci, linux-perf-users, iommu


Hi James,

On 2022/5/10 18:14, James Clark wrote:
> 
> 
> On 07/04/2022 13:58, Yicong Yang wrote:
>> From: Qi Liu <liuqi115@huawei.com>
>>
[...]
>>   struct auxtrace_record
>>   *auxtrace_record__init(struct evlist *evlist, int *err)
>>   {
>> @@ -57,8 +112,12 @@ struct auxtrace_record
>>   	struct evsel *evsel;
>>   	bool found_etm = false;
>>   	struct perf_pmu *found_spe = NULL;
>> +	struct perf_pmu *found_ptt = NULL;
>>   	struct perf_pmu **arm_spe_pmus = NULL;
>> +	struct perf_pmu **hisi_ptt_pmus = NULL;
>> +
>>   	int nr_spes = 0;
>> +	int nr_ptts = 0;
>>   	int i = 0;
>>   
>>   	if (!evlist)
>> @@ -66,13 +125,14 @@ struct auxtrace_record
>>   
>>   	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
>>   	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
>> +	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
>>   
>>   	evlist__for_each_entry(evlist, evsel) {
>>   		if (cs_etm_pmu &&
>>   		    evsel->core.attr.type == cs_etm_pmu->type)
>>   			found_etm = true;
>>   
>> -		if (!nr_spes || found_spe)
>> +		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
>>   			continue;
>>   
>>   		for (i = 0; i < nr_spes; i++) {
>> @@ -81,11 +141,18 @@ struct auxtrace_record
>>   				break;
>>   			}
>>   		}
>> +
>> +		for (i = 0; i < nr_ptts; i++) {
>> +			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
>> +				found_ptt = hisi_ptt_pmus[i];
>> +				break;
>> +			}
>> +		}
>>   	}
>>   	free(arm_spe_pmus);
>>   
>> -	if (found_etm && found_spe) {
>> -		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
>> +	if (found_etm && found_spe && found_ptt) {
>> +		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
> 
> Hi Yicong,
> 
> Is that actually a limitation? I don't see why they couldn't work concurrently.

As Leo said, the logic here should be like this:

         int auxtrace_event_cnt = 0;
         if (found_etm)
                 auxtrace_event_cnt++;
         if (found_spe)
                 auxtrace_event_cnt++;
         if (found_ptt)
                 auxtrace_event_cnt++;

         if (auxtrace_event_cnt > 1) {
                 pr_err("Concurrent AUX trace operation isn't supported: 
found etm %d spe %d ptt %d\n",
                        found_etm, found_spe, found_ptt);
                 *err = -EOPNOTSUPP;
                 return NULL;
         }

which means perf doesn't allow more than one auxtrace event recording at 
the same time.

> 
> 
>>   		*err = -EOPNOTSUPP;
>>   		return NULL;
>>   	}
>> @@ -96,6 +163,9 @@ struct auxtrace_record
>>   #if defined(__aarch64__)
>>   	if (found_spe)
>>   		return arm_spe_recording_init(err, found_spe);
>> +
>> +	if (found_ptt)
>> +		return hisi_ptt_recording_init(err, found_ptt);
>>   #endif
>>   
>>   	/*

[...]
>> +
>> +static int hisi_ptt_recording_options(struct auxtrace_record *itr,
>> +				      struct evlist *evlist,
>> +				      struct record_opts *opts)
>> +{
>> +	struct hisi_ptt_recording *pttr =
>> +			container_of(itr, struct hisi_ptt_recording, itr);
>> +	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
>> +	struct perf_cpu_map *cpus = evlist->core.cpus;
>> +	struct evsel *evsel, *hisi_ptt_evsel = NULL;
>> +	struct evsel *tracking_evsel;
>> +	int err;
>> +
>> +	pttr->evlist = evlist;
>> +	evlist__for_each_entry(evlist, evsel) {
>> +		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
>> +			if (hisi_ptt_evsel) {
>> +				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
>> +				return -EINVAL;
>> +			}
>> +			evsel->core.attr.freq = 0;
>> +			evsel->core.attr.sample_period = 1;
>> +			hisi_ptt_evsel = evsel;
>> +			opts->full_auxtrace = true;
>> +		}
>> +	}
>> +
>> +	err = hisi_ptt_set_auxtrace_mmap_page(opts);
>> +	if (err)
>> +		return err;
>> +	/*
>> +	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
>> +	 * must come first.
>> +	 */
>> +	evlist__to_front(evlist, hisi_ptt_evsel);
>> +
>> +	if (!perf_cpu_map__empty(cpus)) {
>> +		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
>> +		evsel__set_sample_bit(hisi_ptt_evsel, CPU);
>> +	}
> 
> Similar to Leo's comment: CPU isn't required if it's uncore,
> and if TIME is useful then add it regardless of whether the
> event is opened per-cpu or on a task.
> 
got it, will fix this next time.

>> +
>> +	/* Add dummy event to keep tracking */
>> +	err = parse_events(evlist, "dummy:u", NULL);
>> +	if (err)
>> +		return err;
>> +
>> +	tracking_evsel = evlist__last(evlist);
>> +	evlist__set_tracking_event(evlist, tracking_evsel);
>> +
>> +	tracking_evsel->core.attr.freq = 0;
>> +	tracking_evsel->core.attr.sample_period = 1;
>> +
>> +	if (!perf_cpu_map__empty(cpus))
>> +		evsel__set_sample_bit(tracking_evsel, TIME);
> 
> Same comment as above.
> 
got it, thanks.
>> +
>> +	return 0;
>> +}
>> +


>> +
>> +static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
>> +					   union perf_event *event,
>> +					   struct perf_tool *tool __maybe_unused)
>> +{
>> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
>> +					    auxtrace);
>> +	struct auxtrace_buffer *buffer;
>> +	off_t data_offset;
>> +	int fd = perf_data__fd(session->data);
>> +	int err;
>> +
>> +	if (perf_data__is_pipe(session->data)) {
>> +		data_offset = 0;
>> +	} else {
>> +		data_offset = lseek(fd, 0, SEEK_CUR);
>> +		if (data_offset == -1)
>> +			return -errno;
>> +	}
>> +
>> +	err = auxtrace_queues__add_event(&ptt->queues, session, event,
>> +					 data_offset, &buffer);
>> +	if (err)
>> +		return err;
>> +
>> +	if (dump_trace) {
>> +		if (auxtrace_buffer__get_data(buffer, fd)) {
>> +			hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
>> +			auxtrace_buffer__put_data(buffer);
>> +		}
> 
> Technically auxtrace_queues aren't required here because they are more for
> supporting trace from multiple CPUs and sorting and re-ordering between them.
> 
> If this is new device is uncore and always from a single source you could
> just go straight to  hisi_ptt_dump_event() with data_offset and size of the
> auxtrace event.
> 
> But I suppose it also doesn't hurt to use some of the existing framework
> like you have done.
> 
ok, I'll delete the auxtrace_queues next time, thanks.

Qi

>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
>> +			  struct perf_tool *tool __maybe_unused)
>> +{
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_free_events(struct perf_session *session)
>> +{
>> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
>> +					    auxtrace);
>> +	struct auxtrace_queues *queues = &ptt->queues;
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < queues->nr_queues; i++) {
>> +		free(queues->queue_array[i].priv);
>> +		queues->queue_array[i].priv = NULL;
>> +	}
>> +	auxtrace_queues__free(queues);
>> +}
>> +
>> +static void hisi_ptt_free(struct perf_session *session)
>> +{
>> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
>> +					    auxtrace);
>> +
>> +	hisi_ptt_free_events(session);
>> +	session->auxtrace = NULL;
>> +	free(ptt);
>> +}
>> +
>> +static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
>> +				       struct evsel *evsel)
>> +{
>> +	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
>> +
>> +	return evsel->core.attr.type == ptt->pmu_type;
>> +}
>> +
>> +static const char * const hisi_ptt_info_fmts[] = {
>> +	[HISI_PTT_PMU_TYPE]		= "  PMU Type           %" PRId64 "\n",
>> +};
>> +
>> +static void hisi_ptt_print_info(__u64 *arr)
>> +{
>> +	if (!dump_trace)
>> +		return;
>> +
>> +	fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
>> +}
>> +
>> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
>> +				   struct perf_session *session)
>> +{
>> +	struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
>> +	struct hisi_ptt *ptt;
>> +	int err;
>> +
>> +	if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
>> +				sizeof(struct perf_record_auxtrace_info))
>> +		return -EINVAL;
>> +
>> +	ptt = zalloc(sizeof(struct hisi_ptt));
>> +	if (!ptt)
>> +		return -ENOMEM;
>> +
>> +	err = auxtrace_queues__init(&ptt->queues);
>> +	if (err)
>> +		goto err_free;
>> +
>> +	ptt->session = session;
>> +	ptt->machine = &session->machines.host; /* No kvm support */
>> +	ptt->auxtrace_type = auxtrace_info->type;
>> +	ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
>> +
>> +	ptt->auxtrace.process_event = hisi_ptt_process_event;
>> +	ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
>> +	ptt->auxtrace.flush_events = hisi_ptt_flush;
>> +	ptt->auxtrace.free_events = hisi_ptt_free_events;
>> +	ptt->auxtrace.free = hisi_ptt_free;
>> +	ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
>> +	session->auxtrace = &ptt->auxtrace;
>> +
>> +	hisi_ptt_print_info(&auxtrace_info->priv[0]);
>> +
>> +	return 0;
>> +
>> +err_free:
>> +	free(ptt);
>> +	return err;
>> +}
>> diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
>> new file mode 100644
>> index 000000000000..c0b6cbde1221
>> --- /dev/null
>> +++ b/tools/perf/util/hisi_ptt.h
>> @@ -0,0 +1,28 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * HiSilicon PCIe Trace and Tuning (PTT) support
>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>> + */
>> +
>> +#ifndef INCLUDE__PERF_HISI_PTT_H__
>> +#define INCLUDE__PERF_HISI_PTT_H__
>> +
>> +#define HISI_PTT_PMU_NAME "hisi_ptt"
>> +enum {
>> +	HISI_PTT_PMU_TYPE,
>> +	HISI_PTT_PER_CPU_MMAPS,
>> +	HISI_PTT_AUXTRACE_PRIV_MAX,
>> +};
>> +
>> +#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
>> +union perf_event;
>> +struct perf_session;
>> +struct perf_pmu;
>> +
>> +struct auxtrace_record *hisi_ptt_recording_init(int *err,
>> +						struct perf_pmu *hisi_ptt_pmu);
>> +
>> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
>> +				   struct perf_session *session);
>> +
>> +#endif
> .
> 

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
  2022-05-11  2:02     ` liuqi (BA)
@ 2022-05-11  9:18       ` James Clark
  0 siblings, 0 replies; 30+ messages in thread
From: James Clark @ 2022-05-11  9:18 UTC (permalink / raw)
  To: liuqi (BA), Yicong Yang, Leo Yan
  Cc: prime.zeng, zhangshaokun, linuxarm, gregkh, helgaas,
	alexander.shishkin, lorenzo.pieralisi, will, mark.rutland,
	mathieu.poirier, suzuki.poulose, mike.leach, jonathan.cameron,
	daniel.thompson, joro, john.garry, shameerali.kolothum.thodi,
	robin.murphy, peterz, mingo, acme, linux-kernel,
	linux-arm-kernel, coresight, linux-pci, linux-perf-users, iommu



On 11/05/2022 03:02, liuqi (BA) wrote:
> 
> Hi James,
> 
> On 2022/5/10 18:14, James Clark wrote:
>>
>>
>> On 07/04/2022 13:58, Yicong Yang wrote:
>>> From: Qi Liu <liuqi115@huawei.com>
>>>
> [...]
>>>   struct auxtrace_record
>>>   *auxtrace_record__init(struct evlist *evlist, int *err)
>>>   {
>>> @@ -57,8 +112,12 @@ struct auxtrace_record
>>>       struct evsel *evsel;
>>>       bool found_etm = false;
>>>       struct perf_pmu *found_spe = NULL;
>>> +    struct perf_pmu *found_ptt = NULL;
>>>       struct perf_pmu **arm_spe_pmus = NULL;
>>> +    struct perf_pmu **hisi_ptt_pmus = NULL;
>>> +
>>>       int nr_spes = 0;
>>> +    int nr_ptts = 0;
>>>       int i = 0;
>>>         if (!evlist)
>>> @@ -66,13 +125,14 @@ struct auxtrace_record
>>>         cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
>>>       arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
>>> +    hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
>>>         evlist__for_each_entry(evlist, evsel) {
>>>           if (cs_etm_pmu &&
>>>               evsel->core.attr.type == cs_etm_pmu->type)
>>>               found_etm = true;
>>>   -        if (!nr_spes || found_spe)
>>> +        if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
>>>               continue;
>>>             for (i = 0; i < nr_spes; i++) {
>>> @@ -81,11 +141,18 @@ struct auxtrace_record
>>>                   break;
>>>               }
>>>           }
>>> +
>>> +        for (i = 0; i < nr_ptts; i++) {
>>> +            if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
>>> +                found_ptt = hisi_ptt_pmus[i];
>>> +                break;
>>> +            }
>>> +        }
>>>       }
>>>       free(arm_spe_pmus);
>>>   -    if (found_etm && found_spe) {
>>> -        pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
>>> +    if (found_etm && found_spe && found_ptt) {
>>> +        pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
>>
>> Hi Yicong,
>>
>> Is that actually a limitation? I don't see why they couldn't work concurrently.
> 
> As Leo said, the logic here should be like this:
> 
>         int auxtrace_event_cnt = 0;
>         if (found_etm)
>                 auxtrace_event_cnt++;
>         if (found_spe)
>                 auxtrace_event_cnt++;
>         if (found_ptt)
>                 auxtrace_event_cnt++;
> 
>         if (auxtrace_event_cnt > 1) {
>                 pr_err("Concurrent AUX trace operation isn't supported: found etm %d spe %d ptt %d\n",
>                        found_etm, found_spe, found_ptt);
>                 *err = -EOPNOTSUPP;
>                 return NULL;
>         }
> 
> which means perf doesn't allow more than one auxtrace event recording at the same time.

Oh I see that the limitation is actually in perf when decoding the data. I thought it meant
that it wasn't possible to open multiple aux events at the same time, which I think should
work in theory. Makes sense.

> 
>>
>>
>>>           *err = -EOPNOTSUPP;
>>>           return NULL;
>>>       }
>>> @@ -96,6 +163,9 @@ struct auxtrace_record
>>>   #if defined(__aarch64__)
>>>       if (found_spe)
>>>           return arm_spe_recording_init(err, found_spe);
>>> +
>>> +    if (found_ptt)
>>> +        return hisi_ptt_recording_init(err, found_ptt);
>>>   #endif
>>>         /*
> 
> [...]
>>> +
>>> +static int hisi_ptt_recording_options(struct auxtrace_record *itr,
>>> +                      struct evlist *evlist,
>>> +                      struct record_opts *opts)
>>> +{
>>> +    struct hisi_ptt_recording *pttr =
>>> +            container_of(itr, struct hisi_ptt_recording, itr);
>>> +    struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
>>> +    struct perf_cpu_map *cpus = evlist->core.cpus;
>>> +    struct evsel *evsel, *hisi_ptt_evsel = NULL;
>>> +    struct evsel *tracking_evsel;
>>> +    int err;
>>> +
>>> +    pttr->evlist = evlist;
>>> +    evlist__for_each_entry(evlist, evsel) {
>>> +        if (evsel->core.attr.type == hisi_ptt_pmu->type) {
>>> +            if (hisi_ptt_evsel) {
>>> +                pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
>>> +                return -EINVAL;
>>> +            }
>>> +            evsel->core.attr.freq = 0;
>>> +            evsel->core.attr.sample_period = 1;
>>> +            hisi_ptt_evsel = evsel;
>>> +            opts->full_auxtrace = true;
>>> +        }
>>> +    }
>>> +
>>> +    err = hisi_ptt_set_auxtrace_mmap_page(opts);
>>> +    if (err)
>>> +        return err;
>>> +    /*
>>> +     * To obtain the auxtrace buffer file descriptor, the auxtrace event
>>> +     * must come first.
>>> +     */
>>> +    evlist__to_front(evlist, hisi_ptt_evsel);
>>> +
>>> +    if (!perf_cpu_map__empty(cpus)) {
>>> +        evsel__set_sample_bit(hisi_ptt_evsel, TIME);
>>> +        evsel__set_sample_bit(hisi_ptt_evsel, CPU);
>>> +    }
>>
>> Similar to Leo's comment: CPU isn't required if it's uncore,
>> and if TIME is useful then add it regardless of whether the
>> event is opened per-cpu or on a task.
>>
> got it, will fix this next time.
> 
>>> +
>>> +    /* Add dummy event to keep tracking */
>>> +    err = parse_events(evlist, "dummy:u", NULL);
>>> +    if (err)
>>> +        return err;
>>> +
>>> +    tracking_evsel = evlist__last(evlist);
>>> +    evlist__set_tracking_event(evlist, tracking_evsel);
>>> +
>>> +    tracking_evsel->core.attr.freq = 0;
>>> +    tracking_evsel->core.attr.sample_period = 1;
>>> +
>>> +    if (!perf_cpu_map__empty(cpus))
>>> +        evsel__set_sample_bit(tracking_evsel, TIME);
>>
>> Same comment as above.
>>
> got it, thanks.
>>> +
>>> +    return 0;
>>> +}
>>> +
> 
> 
>>> +
>>> +static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
>>> +                       union perf_event *event,
>>> +                       struct perf_tool *tool __maybe_unused)
>>> +{
>>> +    struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
>>> +                        auxtrace);
>>> +    struct auxtrace_buffer *buffer;
>>> +    off_t data_offset;
>>> +    int fd = perf_data__fd(session->data);
>>> +    int err;
>>> +
>>> +    if (perf_data__is_pipe(session->data)) {
>>> +        data_offset = 0;
>>> +    } else {
>>> +        data_offset = lseek(fd, 0, SEEK_CUR);
>>> +        if (data_offset == -1)
>>> +            return -errno;
>>> +    }
>>> +
>>> +    err = auxtrace_queues__add_event(&ptt->queues, session, event,
>>> +                     data_offset, &buffer);
>>> +    if (err)
>>> +        return err;
>>> +
>>> +    if (dump_trace) {
>>> +        if (auxtrace_buffer__get_data(buffer, fd)) {
>>> +            hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
>>> +            auxtrace_buffer__put_data(buffer);
>>> +        }
>>
>> Technically auxtrace_queues aren't required here because they are more for
>> supporting trace from multiple CPUs and sorting and re-ordering between them.
>>
>> If this is new device is uncore and always from a single source you could
>> just go straight to  hisi_ptt_dump_event() with data_offset and size of the
>> auxtrace event.
>>
>> But I suppose it also doesn't hurt to use some of the existing framework
>> like you have done.
>>
> ok, I'll delete the auxtrace_queues next time, thanks.

It's up to you, it might be more work and it's best to leave it as it is now
that you have it working already.

James

> 
> Qi
> 
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
>>> +              struct perf_tool *tool __maybe_unused)
>>> +{
>>> +    return 0;
>>> +}
>>> +
>>> +static void hisi_ptt_free_events(struct perf_session *session)
>>> +{
>>> +    struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
>>> +                        auxtrace);
>>> +    struct auxtrace_queues *queues = &ptt->queues;
>>> +    unsigned int i;
>>> +
>>> +    for (i = 0; i < queues->nr_queues; i++) {
>>> +        free(queues->queue_array[i].priv);
>>> +        queues->queue_array[i].priv = NULL;
>>> +    }
>>> +    auxtrace_queues__free(queues);
>>> +}
>>> +
>>> +static void hisi_ptt_free(struct perf_session *session)
>>> +{
>>> +    struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
>>> +                        auxtrace);
>>> +
>>> +    hisi_ptt_free_events(session);
>>> +    session->auxtrace = NULL;
>>> +    free(ptt);
>>> +}
>>> +
>>> +static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
>>> +                       struct evsel *evsel)
>>> +{
>>> +    struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
>>> +
>>> +    return evsel->core.attr.type == ptt->pmu_type;
>>> +}
>>> +
>>> +static const char * const hisi_ptt_info_fmts[] = {
>>> +    [HISI_PTT_PMU_TYPE]        = "  PMU Type           %" PRId64 "\n",
>>> +};
>>> +
>>> +static void hisi_ptt_print_info(__u64 *arr)
>>> +{
>>> +    if (!dump_trace)
>>> +        return;
>>> +
>>> +    fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
>>> +}
>>> +
>>> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
>>> +                   struct perf_session *session)
>>> +{
>>> +    struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
>>> +    struct hisi_ptt *ptt;
>>> +    int err;
>>> +
>>> +    if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
>>> +                sizeof(struct perf_record_auxtrace_info))
>>> +        return -EINVAL;
>>> +
>>> +    ptt = zalloc(sizeof(struct hisi_ptt));
>>> +    if (!ptt)
>>> +        return -ENOMEM;
>>> +
>>> +    err = auxtrace_queues__init(&ptt->queues);
>>> +    if (err)
>>> +        goto err_free;
>>> +
>>> +    ptt->session = session;
>>> +    ptt->machine = &session->machines.host; /* No kvm support */
>>> +    ptt->auxtrace_type = auxtrace_info->type;
>>> +    ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
>>> +
>>> +    ptt->auxtrace.process_event = hisi_ptt_process_event;
>>> +    ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
>>> +    ptt->auxtrace.flush_events = hisi_ptt_flush;
>>> +    ptt->auxtrace.free_events = hisi_ptt_free_events;
>>> +    ptt->auxtrace.free = hisi_ptt_free;
>>> +    ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
>>> +    session->auxtrace = &ptt->auxtrace;
>>> +
>>> +    hisi_ptt_print_info(&auxtrace_info->priv[0]);
>>> +
>>> +    return 0;
>>> +
>>> +err_free:
>>> +    free(ptt);
>>> +    return err;
>>> +}
>>> diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
>>> new file mode 100644
>>> index 000000000000..c0b6cbde1221
>>> --- /dev/null
>>> +++ b/tools/perf/util/hisi_ptt.h
>>> @@ -0,0 +1,28 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * HiSilicon PCIe Trace and Tuning (PTT) support
>>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>>> + */
>>> +
>>> +#ifndef INCLUDE__PERF_HISI_PTT_H__
>>> +#define INCLUDE__PERF_HISI_PTT_H__
>>> +
>>> +#define HISI_PTT_PMU_NAME "hisi_ptt"
>>> +enum {
>>> +    HISI_PTT_PMU_TYPE,
>>> +    HISI_PTT_PER_CPU_MMAPS,
>>> +    HISI_PTT_AUXTRACE_PRIV_MAX,
>>> +};
>>> +
>>> +#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
>>> +union perf_event;
>>> +struct perf_session;
>>> +struct perf_pmu;
>>> +
>>> +struct auxtrace_record *hisi_ptt_recording_init(int *err,
>>> +                        struct perf_pmu *hisi_ptt_pmu);
>>> +
>>> +int hisi_ptt_process_auxtrace_info(union perf_event *event,
>>> +                   struct perf_session *session);
>>> +
>>> +#endif
>> .
>>

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
  2022-04-07 12:58 ` [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
  2022-05-10 11:23   ` Will Deacon
@ 2022-05-11 11:42   ` John Garry
  1 sibling, 0 replies; 30+ messages in thread
From: John Garry @ 2022-05-11 11:42 UTC (permalink / raw)
  To: Yicong Yang, gregkh, helgaas, alexander.shishkin,
	lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
	suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
	daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
	peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
	linux-pci, linux-perf-users, iommu
  Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm

On 07/04/2022 13:58, Yicong Yang wrote:
> The DMA operations of HiSilicon PTT device can only work properly with
> identical mappings. So add a quirk for the device to force the domain

I'm not sure if you meant to write "identity mappings".

> as passthrough.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>

FWIW,

Reviewed-by: John Garry <john.garry@huawei.com>

> ---
>   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>   1 file changed, 16 insertions(+)
> 
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 627a3ed5ee8f..5ec15ae2a9b1 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2839,6 +2839,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>   	}
>   }
>   
> +#define IS_HISI_PTT_DEVICE(pdev)	((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
> +					 (pdev)->device == 0xa12e)
> +
> +static int arm_smmu_def_domain_type(struct device *dev)
> +{
> +	if (dev_is_pci(dev)) {
> +		struct pci_dev *pdev = to_pci_dev(dev);
> +
> +		if (IS_HISI_PTT_DEVICE(pdev))
> +			return IOMMU_DOMAIN_IDENTITY;
> +	}
> +
> +	return 0;
> +}
> +
>   static struct iommu_ops arm_smmu_ops = {
>   	.capable		= arm_smmu_capable,
>   	.domain_alloc		= arm_smmu_domain_alloc,
> @@ -2856,6 +2871,7 @@ static struct iommu_ops arm_smmu_ops = {
>   	.sva_unbind		= arm_smmu_sva_unbind,
>   	.sva_get_pasid		= arm_smmu_sva_get_pasid,
>   	.page_response		= arm_smmu_page_response,
> +	.def_domain_type	= arm_smmu_def_domain_type,
>   	.pgsize_bitmap		= -1UL, /* Restricted during device attach */
>   	.owner			= THIS_MODULE,
>   	.default_domain_ops = &(const struct iommu_domain_ops) {


^ permalink raw reply	[flat|nested] 30+ messages in thread

end of thread, other threads:[~2022-05-11 11:42 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-07 12:58 [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
2022-04-07 12:58 ` [PATCH v7 1/7] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
2022-05-10 11:23   ` Will Deacon
2022-05-10 12:43     ` Yicong Yang
2022-05-11 11:42   ` John Garry
2022-04-07 12:58 ` [PATCH v7 2/7] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device Yicong Yang
2022-04-11 17:02   ` John Garry
2022-04-12  7:36     ` Yicong Yang
2022-04-12  8:39       ` John Garry
2022-04-12 12:52         ` Yicong Yang
2022-04-29 16:00   ` Leo Yan
2022-05-05 12:13     ` Yicong Yang
2022-05-10  9:46   ` James Clark
2022-05-10 11:18     ` Yicong Yang
2022-05-10 12:54       ` James Clark
2022-05-10 13:17         ` Yicong Yang
2022-04-07 12:58 ` [PATCH v7 3/7] hisi_ptt: Add support for dynamically updating the filter list Yicong Yang
2022-04-07 12:58 ` [PATCH v7 4/7] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device Yicong Yang
2022-04-07 12:58 ` [PATCH v7 5/7] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
2022-04-11 17:19   ` John Garry
2022-04-12  7:41     ` Yicong Yang
2022-04-14 13:00       ` John Garry
2022-04-30  7:35   ` Leo Yan
2022-05-05 11:30     ` liuqi (BA)
2022-05-10 10:14   ` James Clark
2022-05-11  2:02     ` liuqi (BA)
2022-05-11  9:18       ` James Clark
2022-04-07 12:58 ` [PATCH v7 6/7] docs: Add HiSilicon PTT device driver documentation Yicong Yang
2022-04-07 12:58 ` [PATCH v7 7/7] MAINTAINERS: Add maintainer for HiSilicon PTT driver Yicong Yang
2022-04-14 13:20 ` [PATCH v7 0/7] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).