* [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device
@ 2022-01-24 13:11 Yicong Yang
2022-01-24 13:11 ` [PATCH v3 1/8] hwtracing: Add trace function " Yicong Yang
` (8 more replies)
0 siblings, 9 replies; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
integrated Endpoint (RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic (tune),
and trace the TLP headers (trace).
PTT tune is designed for monitoring and adjusting PCIe link parameters.
We provide several parameters of the PCIe link. Through the driver,
user can adjust the value of certain parameter to affect the PCIe link
for the purpose of enhancing the performance in certian situation.
PTT trace is designed for dumping the TLP headers to the memory, which
can be used to analyze the transactions and usage condition of the PCIe
Link. Users can choose filters to trace headers, by either requester
ID, or those downstream of a set of Root Ports on the same core of the
PTT device. It's also supported to trace the headers of certain type and
of certain direction.
The driver registers a PMU device for each PTT device. The trace can
be used through `perf record` and the traced headers can be decoded
by `perf report`. The perf command support for the device is also
added in this patchset. The tune can be used through the sysfs
attributes of related PMU device. See the documentation for the
detailed usage.
Change since v2:
- address the comments from Mathieu
- rename the directory to ptt to match the function of the device
- spinoff the declarations to a separate header
- split the trace function to several patches
- some other comments.
- make default smmu domain type of PTT device to identity
Drop the RMR as it's not recommended and use an iommu_def_domain_type
quirk to passthrough the device DMA as suggested by Robin.
Link: https://lore.kernel.org/linux-pci/20211116090625.53702-1-yangyicong@hisilicon.com/
Change since v1:
- switch the user interface of trace to perf from debugfs
- switch the user interface of tune to sysfs from debugfs
- add perf tool support to start trace and decode the trace data
- address the comments of documentation from Bjorn
- add RMR[1] support of the device as trace works in RMR mode or
direct DMA mode. RMR support is achieved by common APIs rather
than the APIs implemented in [1].
Link: https://lore.kernel.org/lkml/1618654631-42454-1-git-send-email-yangyicong@hisilicon.com/
[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/
Qi Liu (1):
perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
Yicong Yang (7):
hwtracing: Add trace function support for HiSilicon PCIe Tune and
Trace device
hisi_ptt: Register PMU device for PTT trace
hisi_ptt: Add support for dynamically updating the filter list
hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace
device
docs: Add HiSilicon PTT device driver documentation
MAINTAINERS: Add maintainer for HiSilicon PTT driver
iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to
identity
Documentation/trace/hisi-ptt.rst | 304 +++++
MAINTAINERS | 7 +
drivers/Makefile | 1 +
drivers/hwtracing/Kconfig | 2 +
drivers/hwtracing/ptt/Kconfig | 11 +
drivers/hwtracing/ptt/Makefile | 2 +
drivers/hwtracing/ptt/hisi_ptt.c | 1124 +++++++++++++++++
drivers/hwtracing/ptt/hisi_ptt.h | 239 ++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 +
tools/perf/arch/arm/util/auxtrace.c | 56 +-
tools/perf/arch/arm/util/pmu.c | 3 +
tools/perf/arch/arm64/util/Build | 2 +-
tools/perf/arch/arm64/util/hisi_ptt.c | 195 +++
tools/perf/util/Build | 2 +
tools/perf/util/auxtrace.c | 4 +
tools/perf/util/auxtrace.h | 1 +
tools/perf/util/hisi-ptt-decoder/Build | 1 +
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c | 170 +++
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h | 28 +
tools/perf/util/hisi_ptt.c | 228 ++++
tools/perf/util/hisi_ptt.h | 28 +
21 files changed, 2420 insertions(+), 4 deletions(-)
create mode 100644 Documentation/trace/hisi-ptt.rst
create mode 100644 drivers/hwtracing/ptt/Kconfig
create mode 100644 drivers/hwtracing/ptt/Makefile
create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
create mode 100644 tools/perf/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi_ptt.h
--
2.24.0
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH v3 1/8] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-02-07 11:42 ` Jonathan Cameron
2022-02-07 18:11 ` John Garry
2022-01-24 13:11 ` [PATCH v3 2/8] hisi_ptt: Register PMU device for PTT trace Yicong Yang
` (7 subsequent siblings)
8 siblings, 2 replies; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
integrated Endpoint(RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic, and trace
the TLP headers.
Add the driver for the device to enable the trace function.
This patch adds basic function of trace, including the device's
probe and initialization, functions for trace buffer allocation
and trace enable/disable, register an interrupt handler to
simply response to the DMA events. The user interface of trace
will be added in the following patch.
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
drivers/Makefile | 1 +
drivers/hwtracing/Kconfig | 2 +
drivers/hwtracing/ptt/Kconfig | 11 +
drivers/hwtracing/ptt/Makefile | 2 +
drivers/hwtracing/ptt/hisi_ptt.c | 398 +++++++++++++++++++++++++++++++
drivers/hwtracing/ptt/hisi_ptt.h | 159 ++++++++++++
6 files changed, 573 insertions(+)
create mode 100644 drivers/hwtracing/ptt/Kconfig
create mode 100644 drivers/hwtracing/ptt/Makefile
create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
diff --git a/drivers/Makefile b/drivers/Makefile
index a110338c860c..ab3411e4eba5 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -175,6 +175,7 @@ obj-$(CONFIG_USB4) += thunderbolt/
obj-$(CONFIG_CORESIGHT) += hwtracing/coresight/
obj-y += hwtracing/intel_th/
obj-$(CONFIG_STM) += hwtracing/stm/
+obj-$(CONFIG_HISI_PTT) += hwtracing/ptt/
obj-$(CONFIG_ANDROID) += android/
obj-$(CONFIG_NVMEM) += nvmem/
obj-$(CONFIG_FPGA) += fpga/
diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
index 13085835a636..911ee977103c 100644
--- a/drivers/hwtracing/Kconfig
+++ b/drivers/hwtracing/Kconfig
@@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
source "drivers/hwtracing/intel_th/Kconfig"
+source "drivers/hwtracing/ptt/Kconfig"
+
endmenu
diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
new file mode 100644
index 000000000000..4f4f2459ac47
--- /dev/null
+++ b/drivers/hwtracing/ptt/Kconfig
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config HISI_PTT
+ tristate "HiSilicon PCIe Tune and Trace Device"
+ depends on ARM64 && PCI && HAS_DMA && HAS_IOMEM
+ help
+ HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
+ device, provides support for PCIe traffic tuning and
+ tracing TLP headers to the memory.
+
+ This driver can also be built as a module. If so, the module
+ will be called hisi_ptt.
diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
new file mode 100644
index 000000000000..908c09a98161
--- /dev/null
+++ b/drivers/hwtracing/ptt/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
new file mode 100644
index 000000000000..6d0a0ca5c0a9
--- /dev/null
+++ b/drivers/hwtracing/ptt/hisi_ptt.c
@@ -0,0 +1,398 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for HiSilicon PCIe tune and trace device
+ *
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ * Author: Yicong Yang <yangyicong@hisilicon.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/dma-iommu.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iommu.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/sysfs.h>
+
+#include "hisi_ptt.h"
+
+static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
+{
+ if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
+ return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
+
+ return PCI_DEVID(pdev->bus->number, pdev->devfn);
+}
+
+static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
+{
+ u32 val;
+
+ return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
+ val & HISI_PTT_TRACE_IDLE,
+ HISI_PTT_WAIT_POLL_INTERVAL_US,
+ HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+ struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+ struct device *dev = &hisi_ptt->pdev->dev;
+ struct hisi_ptt_dma_buffer *buffer, *tbuffer;
+
+ list_for_each_entry_safe(buffer, tbuffer, &ctrl->trace_buf, list) {
+ list_del(&buffer->list);
+ dma_free_coherent(dev, buffer->size, buffer->addr,
+ buffer->dma);
+ kfree(buffer);
+ }
+}
+
+static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+ struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+ struct device *dev = &hisi_ptt->pdev->dev;
+ struct hisi_ptt_dma_buffer *buffer;
+ int i, ret;
+
+ hisi_ptt->trace_ctrl.buf_index = 0;
+
+ /* Make sure the trace buffer is empty before allocating */
+ if (!list_empty(&ctrl->trace_buf)) {
+ list_for_each_entry(buffer, &ctrl->trace_buf, list)
+ memset(buffer->addr, 0, buffer->size);
+ return 0;
+ }
+
+ for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
+ buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
+ if (!buffer) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ buffer->addr = dma_alloc_coherent(dev, ctrl->buffer_size,
+ &buffer->dma, GFP_KERNEL);
+ if (!buffer->addr) {
+ kfree(buffer);
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ memset(buffer->addr, 0, buffer->size);
+
+ buffer->index = i;
+ buffer->size = ctrl->buffer_size;
+ list_add_tail(&buffer->list, &ctrl->trace_buf);
+ }
+
+ return 0;
+err:
+ hisi_ptt_free_trace_buf(hisi_ptt);
+ return ret;
+}
+
+static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
+{
+ writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+ hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
+}
+
+static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
+{
+ struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+ struct hisi_ptt_dma_buffer *cur;
+ u32 val;
+
+ /* Check device idle before start trace */
+ if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
+ pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
+ return -EBUSY;
+ }
+
+ /* Reset the DMA before start tracing */
+ val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+ val |= HISI_PTT_TRACE_CTRL_RST;
+ writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+ /*
+ * We'll be in the perf context where preemption is disabled,
+ * so use busy loop here.
+ */
+ mdelay(HISI_PTT_RESET_WAIT_MS);
+
+ val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+ val &= ~HISI_PTT_TRACE_CTRL_RST;
+ writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+ /* Clear the interrupt status */
+ writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+ writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
+
+ /* Configure the trace DMA buffer */
+ list_for_each_entry(cur, &ctrl->trace_buf, list) {
+ writel(lower_32_bits(cur->dma),
+ hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
+ cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+ writel(upper_32_bits(cur->dma),
+ hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
+ cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+ }
+ writel(ctrl->buffer_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
+
+ /* Set the trace control register */
+ val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
+ val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
+ val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
+ val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
+ if (!hisi_ptt->trace_ctrl.is_port)
+ val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
+
+ /* Start the Trace */
+ val |= HISI_PTT_TRACE_CTRL_EN;
+ writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+ ctrl->status = HISI_PTT_TRACE_STATUS_ON;
+
+ return 0;
+}
+
+static irqreturn_t hisi_ptt_isr(int irq, void *context)
+{
+ struct hisi_ptt *hisi_ptt = context;
+ u32 status;
+
+ status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+
+ /* Clear the interrupt status of buffer @buf_idx */
+ writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t hisi_ptt_irq(int irq, void *context)
+{
+ struct hisi_ptt *hisi_ptt = context;
+ u32 status;
+
+ status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+ if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
+ return IRQ_NONE;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static void hisi_ptt_irq_free_vectors(void *pdev)
+{
+ pci_free_irq_vectors(pdev);
+}
+
+static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
+{
+ struct pci_dev *pdev = hisi_ptt->pdev;
+ int ret;
+
+ ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
+ if (ret < 0) {
+ pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
+ return ret;
+ }
+
+ ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
+ if (ret < 0)
+ return ret;
+
+ ret = devm_request_threaded_irq(&pdev->dev,
+ pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
+ hisi_ptt_irq, hisi_ptt_isr, 0,
+ "hisi-ptt", hisi_ptt);
+ if (ret) {
+ pci_err(pdev, "failed to request irq %d, ret = %d.\n",
+ pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+{
+ struct hisi_ptt_filter_desc *filter;
+ struct hisi_ptt *hisi_ptt = data;
+ struct list_head *target_list;
+
+ target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
+ &hisi_ptt->port_filters : &hisi_ptt->req_filters;
+
+ filter = kzalloc(sizeof(*filter), GFP_KERNEL);
+ if (!filter)
+ return -ENOMEM;
+
+ filter->pdev = pdev;
+ filter->val = hisi_ptt_get_filter_val(pdev);
+ list_add_tail(&filter->list, target_list);
+
+ /* Update the available port mask */
+ if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
+ hisi_ptt->port_mask |= filter->val;
+
+ return 0;
+}
+
+static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
+{
+ struct hisi_ptt_filter_desc *filter, *tfilter;
+
+ list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
+ list_del(&filter->list);
+ kfree(filter);
+ }
+
+ list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
+ list_del(&filter->list);
+ kfree(filter);
+ }
+}
+
+static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
+{
+ struct pci_dev *pdev = hisi_ptt->pdev;
+ struct pci_bus *bus;
+ u32 reg;
+
+ INIT_LIST_HEAD(&hisi_ptt->port_filters);
+ INIT_LIST_HEAD(&hisi_ptt->req_filters);
+
+ /*
+ * The device range register provides the information about the
+ * root ports which the RCiEP can control and trace. The RCiEP
+ * and the root ports it support are on the same PCIe core, with
+ * same domain number but maybe different bus number. The device
+ * range register will tell us which root ports we can support,
+ * Bit[31:16] indicates the upper BDF numbers of the root port,
+ * while Bit[15:0] indicates the lower.
+ */
+ reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
+ hisi_ptt->upper = reg >> 16;
+ hisi_ptt->lower = reg & 0xffff;
+
+ reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
+ hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
+ hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
+
+ bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
+ if (bus)
+ pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
+
+ /* Initialize trace controls */
+ INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
+ hisi_ptt->trace_ctrl.buffer_size = HISI_PTT_TRACE_BUF_SIZE;
+ hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
+}
+
+/*
+ * The DMA of PTT trace can only use direct mapping, due to some
+ * hardware restriction. Check whether there is an iommu or the
+ * policy of the iommu domain is passthrough, otherwise the trace
+ * cannot work.
+ */
+static int hisi_ptt_check_iommu_mapping(struct hisi_ptt *hisi_ptt)
+{
+ struct pci_dev *pdev = hisi_ptt->pdev;
+ struct iommu_domain *iommu_domain;
+
+ iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
+ if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
+ return 0;
+
+ return -EOPNOTSUPP;
+}
+
+static int hisi_ptt_probe(struct pci_dev *pdev,
+ const struct pci_device_id *id)
+{
+ struct hisi_ptt *hisi_ptt;
+ int ret;
+
+ hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
+ if (!hisi_ptt)
+ return -ENOMEM;
+
+ mutex_init(&hisi_ptt->mutex);
+ hisi_ptt->pdev = pdev;
+
+ /*
+ * Lifetime of pci_dev is longer than hisi_ptt,
+ * so directly reference to the pci name string.
+ */
+ hisi_ptt->name = pci_name(hisi_ptt->pdev);
+ pci_set_drvdata(pdev, hisi_ptt);
+
+ ret = pcim_enable_device(pdev);
+ if (ret) {
+ pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
+ return ret;
+ }
+
+ ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
+ if (ret) {
+ pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
+ return ret;
+ }
+
+ hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
+
+ ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret) {
+ pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
+ return ret;
+ }
+ pci_set_master(pdev);
+
+ ret = hisi_ptt_register_irq(hisi_ptt);
+ if (ret)
+ return ret;
+
+ hisi_ptt_init_ctrls(hisi_ptt);
+
+ ret = hisi_ptt_check_iommu_mapping(hisi_ptt);
+ if (ret) {
+ pci_err(pdev, "cannot work with non-direct DMA mapping.\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+void hisi_ptt_remove(struct pci_dev *pdev)
+{
+ struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
+
+ if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
+ hisi_ptt_trace_end(hisi_ptt);
+
+ hisi_ptt_free_trace_buf(hisi_ptt);
+ hisi_ptt_release_filters(hisi_ptt);
+}
+
+static const struct pci_device_id hisi_ptt_id_tbl[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
+ { }
+};
+MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
+
+static struct pci_driver hisi_ptt_driver = {
+ .name = "hisi_ptt",
+ .id_table = hisi_ptt_id_tbl,
+ .probe = hisi_ptt_probe,
+ .remove = hisi_ptt_remove,
+};
+module_pci_driver(hisi_ptt_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
+MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
new file mode 100644
index 000000000000..8aa5d6b78cc5
--- /dev/null
+++ b/drivers/hwtracing/ptt/hisi_ptt.h
@@ -0,0 +1,159 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Driver for HiSilicon PCIe tune and trace device
+ *
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ * Author: Yicong Yang <yangyicong@hisilicon.com>
+ */
+
+#ifndef _HISI_PTT_H
+#define _HISI_PTT_H
+
+#include <linux/bits.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+
+/*
+ * The definition of the device registers and register fields.
+ */
+#define HISI_PTT_TRACE_ADDR_SIZE 0x0800
+#define HISI_PTT_TRACE_ADDR_BASE_LO_0 0x0810
+#define HISI_PTT_TRACE_ADDR_BASE_HI_0 0x0814
+#define HISI_PTT_TRACE_ADDR_STRIDE 0x8
+#define HISI_PTT_TRACE_CTRL 0x0850
+#define HISI_PTT_TRACE_CTRL_EN BIT(0)
+#define HISI_PTT_TRACE_CTRL_RST BIT(1)
+#define HISI_PTT_TRACE_CTRL_RXTX_SEL GENMASK(3, 2)
+#define HISI_PTT_TRACE_CTRL_TYPE_SEL GENMASK(7, 4)
+#define HISI_PTT_TRACE_CTRL_DATA_FORMAT BIT(14)
+#define HISI_PTT_TRACE_CTRL_FILTER_MODE BIT(15)
+#define HISI_PTT_TRACE_CTRL_TARGET_SEL GENMASK(31, 16)
+#define HISI_PTT_TRACE_INT_STAT 0x0890
+#define HISI_PTT_TRACE_INT_STAT_MASK GENMASK(3, 0)
+#define HISI_PTT_TRACE_INT_MASK 0x0894
+#define HISI_PTT_TRACE_WR_STS 0x08a0
+#define HISI_PTT_TRACE_WR_STS_WRITE GENMASK(27, 0)
+#define HISI_PTT_TRACE_WR_STS_BUFFER GENMASK(29, 28)
+#define HISI_PTT_TRACE_STS 0x08b0
+#define HISI_PTT_TRACE_IDLE BIT(0)
+#define HISI_PTT_DEVICE_RANGE 0x0fe0
+#define HISI_PTT_LOCATION 0x0fe8
+#define HISI_PTT_CORE_ID GENMASK(15, 0)
+#define HISI_PTT_SICL_ID GENMASK(31, 16)
+
+/* Parameters of PTT trace DMA part. */
+#define HISI_PTT_TRACE_DMA_IRQ 0
+#define HISI_PTT_TRACE_BUF_CNT 4
+#define HISI_PTT_TRACE_BUF_SIZE SZ_4M
+#define HISI_PTT_TRACE_TOTAL_BUF_SIZE (HISI_PTT_TRACE_BUF_SIZE * \
+ HISI_PTT_TRACE_BUF_CNT)
+/* Wait time for DMA hardware to reset */
+#define HISI_PTT_RESET_WAIT_MS 1000UL
+/* Poll timeout and interval for waiting hardware work to finish */
+#define HISI_PTT_WAIT_TIMEOUT_US 1000000UL
+#define HISI_PTT_WAIT_POLL_INTERVAL_US 100UL
+
+#define HISI_PCIE_CORE_PORT_ID(devfn) (PCI_FUNC(devfn) << 1)
+
+enum hisi_ptt_trace_status {
+ HISI_PTT_TRACE_STATUS_OFF = 0,
+ HISI_PTT_TRACE_STATUS_ON,
+};
+
+/**
+ * struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
+ * The detail of the data format is described
+ * in the documentation of PTT device.
+ * @list: entry of this buffer in the buffer list
+ * @size: size of this buffer
+ * @dma: DMA address of this buffer visible to the device
+ * @addr: virtual address of this buffer visible to the cpu
+ * @index: index of this buffer in the buffer list
+ */
+struct hisi_ptt_dma_buffer {
+ struct list_head list;
+ unsigned int size;
+ dma_addr_t dma;
+ void *addr;
+ int index;
+};
+
+/**
+ * struct hisi_ptt_trace_ctrl - control and status of PTT trace
+ * @status: current trace status
+ * @trace_buf: the trace buffers for holding the trace data
+ * @default_cpu: default cpu to start the trace session
+ * @buffer_size: the size of each trace buffer in the list
+ * @buf_index: the index of current using trace buffer
+ * @is_port: whether we're tracing root port or not
+ * @direction: direction of the TLP headers to trace
+ * @filter: filter value for tracing the TLP headers
+ * @format: format of the TLP headers to trace
+ * @type: type of the TLP headers to trace
+ */
+struct hisi_ptt_trace_ctrl {
+ enum hisi_ptt_trace_status status;
+ struct list_head trace_buf;
+ int default_cpu;
+ u32 buffer_size;
+ u32 buf_index;
+ bool is_port;
+ u32 direction:2;
+ u32 filter:16;
+ u32 format:1;
+ u32 type:4;
+};
+
+/**
+ * struct hisi_ptt_filter_desc - descriptor of the PTT trace filter
+ * @list: entry of this descriptor in the filter list
+ * @pdev: pci_dev related to this filter
+ * @val: filter value for the device
+ */
+struct hisi_ptt_filter_desc {
+ struct list_head list;
+ struct pci_dev *pdev;
+ u16 val;
+};
+
+/**
+ * struct hisi_ptt - per PTT device data
+ * @trace_ctrl: the control information of PTT trace
+ * @iobase: base IO address of the device
+ * @pdev: pci_dev of this PTT device
+ * @mutex: mutex to protect the filter list and serialize the perf process.
+ * @name: name of the PTT device
+ * @core_id: PCIe core ID this PTT device locates
+ * @sicl_id: SICL ID this PTT device locates
+ * @upper: the upper BDF range of the PCI devices managed by this PTT device
+ * @lower: the lower BDF range of the PCI devices managed by this PTT device
+ * @port_filters: the filter list of root ports
+ * @req_filters: the filter list of requester ID
+ * @port_mask: port mask of the managed root ports
+ */
+struct hisi_ptt {
+ struct hisi_ptt_trace_ctrl trace_ctrl;
+ void __iomem *iobase;
+ struct pci_dev *pdev;
+ struct mutex mutex;
+ const char *name;
+ u16 core_id;
+ u16 sicl_id;
+ u32 upper;
+ u32 lower;
+
+ /*
+ * The trace TLP headers can either be filtered by certain
+ * root port, or by the requester ID. Organize the filters
+ * by @port_filters and @req_filters here. The mask of all
+ * the valid ports is also cached for doing sanity check
+ * of user input.
+ */
+ struct list_head port_filters;
+ struct list_head req_filters;
+ u16 port_mask;
+};
+
+#endif /* _HISI_PTT_H */
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v3 2/8] hisi_ptt: Register PMU device for PTT trace
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
2022-01-24 13:11 ` [PATCH v3 1/8] hwtracing: Add trace function " Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-02-07 11:42 ` Jonathan Cameron
2022-01-24 13:11 ` [PATCH v3 3/8] hisi_ptt: Add support for dynamically updating the filter list Yicong Yang
` (6 subsequent siblings)
8 siblings, 1 reply; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
Register PMU device of PTT trace, then users can use
trace through perf command. The driver makes use of perf
AUX trace and support following events to configure the
trace:
- filter: select Root port or Endpoint to trace
- type: select the type of traced TLP headers
- direction: select the direction of traced TLP headers
- format: select the data format of the traced TLP headers
This patch adds the PMU driver part of PTT trace. The perf
command support of PTT trace is added in the following
patch.
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
drivers/hwtracing/ptt/hisi_ptt.c | 459 ++++++++++++++++++++++++++++++-
drivers/hwtracing/ptt/hisi_ptt.h | 22 ++
2 files changed, 480 insertions(+), 1 deletion(-)
diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
index 6d0a0ca5c0a9..eeb8afc65a56 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.c
+++ b/drivers/hwtracing/ptt/hisi_ptt.c
@@ -17,6 +17,7 @@
#include <linux/iopoll.h>
#include <linux/module.h>
#include <linux/sysfs.h>
+#include <linux/vmalloc.h>
#include "hisi_ptt.h"
@@ -161,16 +162,67 @@ static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
return 0;
}
+static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
+{
+ struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+ struct perf_event *event = handle->event;
+ struct hisi_ptt_dma_buffer *cur;
+ struct hisi_ptt_pmu_buf *buf;
+
+ buf = perf_get_aux(handle);
+ if (!buf || !handle->size)
+ return -EINVAL;
+
+ list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
+ if (cur->index == index)
+ break;
+
+ memcpy(buf->base + buf->pos, cur->addr, cur->size);
+ memset(cur->addr, 0, cur->size);
+ buf->pos += cur->size;
+
+ if (stop) {
+ perf_aux_output_end(handle, buf->pos);
+ } else if (buf->length - buf->pos < cur->size) {
+ perf_aux_output_skip(handle, buf->length - buf->pos);
+ perf_aux_output_end(handle, buf->pos);
+
+ buf = perf_aux_output_begin(handle, event);
+ if (!buf)
+ return -EINVAL;
+
+ buf->pos = handle->head % buf->length;
+ if (buf->length - buf->pos < cur->size) {
+ perf_aux_output_end(handle, 0);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
static irqreturn_t hisi_ptt_isr(int irq, void *context)
{
struct hisi_ptt *hisi_ptt = context;
- u32 status;
+ u32 status, buf_idx;
status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+ buf_idx = ffs(status) - 1;
/* Clear the interrupt status of buffer @buf_idx */
writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+ /*
+ * Update the AUX buffer and cache the current buffer index,
+ * as we need to know this and save the data when the trace
+ * is ended out of the interrupt handler. End the trace
+ * if the updating fails.
+ */
+ if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
+ hisi_ptt_trace_end(hisi_ptt);
+ else
+ hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUF_CNT;
+
return IRQ_HANDLED;
}
@@ -294,6 +346,405 @@ static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
}
+#define HISI_PTT_PMU_FILTER_IS_PORT BIT(19)
+#define HISI_PTT_PMU_FILTER_VAL_MASK GENMASK(15, 0)
+#define HISI_PTT_PMU_DIRECTION_MASK GENMASK(23, 20)
+#define HISI_PTT_PMU_TYPE_MASK GENMASK(31, 24)
+#define HISI_PTT_PMU_FORMAT_MASK GENMASK(35, 32)
+
+static ssize_t available_filters_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+ struct hisi_ptt_filter_desc *filter;
+ int pos = 0;
+
+ if (list_empty(&hisi_ptt->port_filters))
+ return sysfs_emit(buf, "#### No available filter ####\n");
+
+ mutex_lock(&hisi_ptt->mutex);
+ pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
+ list_for_each_entry(filter, &hisi_ptt->port_filters, list)
+ pos += sysfs_emit_at(buf, pos, "%s 0x%05lx\n",
+ pci_name(filter->pdev),
+ hisi_ptt_get_filter_val(filter->pdev) |
+ HISI_PTT_PMU_FILTER_IS_PORT);
+
+ pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
+ list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+ pos += sysfs_emit_at(buf, pos, "%s 0x%05x\n",
+ pci_name(filter->pdev),
+ hisi_ptt_get_filter_val(filter->pdev));
+
+ mutex_unlock(&hisi_ptt->mutex);
+ return pos;
+}
+static DEVICE_ATTR_ADMIN_RO(available_filters);
+
+PMU_FORMAT_ATTR(filter, "config:0-19");
+PMU_FORMAT_ATTR(direction, "config:20-23");
+PMU_FORMAT_ATTR(type, "config:24-31");
+PMU_FORMAT_ATTR(format, "config:32-35");
+
+static struct attribute *hisi_ptt_pmu_format_attrs[] = {
+ &format_attr_filter.attr,
+ &format_attr_direction.attr,
+ &format_attr_type.attr,
+ &format_attr_format.attr,
+ NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_format_group = {
+ .name = "format",
+ .attrs = hisi_ptt_pmu_format_attrs,
+};
+
+static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
+ &dev_attr_available_filters.attr,
+ NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_filter_group = {
+ .attrs = hisi_ptt_pmu_filter_attrs,
+};
+
+static const struct attribute_group *hisi_ptt_pmu_groups[] = {
+ &hisi_ptt_pmu_format_group,
+ &hisi_ptt_pmu_filter_group,
+ NULL
+};
+
+/*
+ * The supported value of the direction parameter. See hisi_ptt.rst
+ * documentation for more details.
+ */
+static u32 hisi_ptt_trace_available_direction[] = {
+ 0,
+ 1,
+ 2,
+ 3,
+};
+
+/* Different types can be set simultaneously */
+static u32 hisi_ptt_trace_available_type[] = {
+ 1, /* posted_request */
+ 2, /* non-posted_request */
+ 4, /* completion */
+};
+
+static u32 hisi_ptt_trace_availble_format[] = {
+ 0, /* 4DW */
+ 1, /* 8DW */
+};
+
+/*
+ * Check whether the config is valid or not. Some configs are multi-selectable
+ * and can be set simultaneously, while some are single selectable (onehot).
+ * Use this function to check the non-onehot configs while
+ * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
+ */
+static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
+{
+ int i;
+
+ /* The non-onehot configs cannot be 0. */
+ if (!val)
+ return -EINVAL;
+
+ /*
+ * Walk the available list and clear the valid bits of
+ * the config. If there is any resident bit after the
+ * walk then the config is invalid.
+ */
+ for (i = 0; i < list_size; i++)
+ val &= ~available_list[i];
+
+ return val ? -EINVAL : 0;
+}
+
+static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
+{
+ int i, ret = -EINVAL;
+
+ for (i = 0; i < list_size; i++)
+ if (val == available_list[i]) {
+ ret = 0;
+ break;
+ }
+
+ return ret;
+}
+
+static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
+{
+ unsigned long val, port_mask = hisi_ptt->port_mask;
+ struct hisi_ptt_filter_desc *filter;
+ int ret = -EINVAL;
+
+ hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
+ val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
+
+ /*
+ * Port filters are defined as bit mask. For port filters, check
+ * the bits in the @val are within the range of hisi_ptt->port_mask
+ * and whether it's empty or not, otherwise user has specified
+ * some unsupported root ports.
+ *
+ * For Requester ID filters, walk the available filter list to see
+ * whether we have one matched.
+ */
+ if (!hisi_ptt->trace_ctrl.is_port) {
+ list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+ if (filter->val == val) {
+ ret = 0;
+ break;
+ }
+ } else if (bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
+ ret = 0;
+ }
+
+ if (ret)
+ return ret;
+
+ hisi_ptt->trace_ctrl.filter = val;
+ return 0;
+}
+
+static int hisi_ptt_pmu_event_init(struct perf_event *event)
+{
+ struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+ struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+ int ret;
+ u32 val;
+
+ if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
+ return -ENOENT;
+
+ mutex_lock(&hisi_ptt->mutex);
+
+ ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
+ if (ret < 0)
+ goto out;
+
+ val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
+ ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
+ ARRAY_SIZE(hisi_ptt_trace_available_direction));
+ if (ret < 0)
+ goto out;
+ ctrl->direction = val;
+
+ val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
+
+ ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
+ ARRAY_SIZE(hisi_ptt_trace_available_type));
+ if (ret < 0)
+ goto out;
+ ctrl->type = val;
+
+ val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
+ ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
+ ARRAY_SIZE(hisi_ptt_trace_availble_format));
+ if (ret < 0)
+ goto out;
+ ctrl->format = val;
+
+out:
+ mutex_unlock(&hisi_ptt->mutex);
+ return ret;
+}
+
+static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
+ int nr_pages, bool overwrite)
+{
+ struct hisi_ptt_pmu_buf *buf;
+ struct page **pagelist;
+ int i;
+
+ if (overwrite) {
+ dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
+ return NULL;
+ }
+
+ /* If the pages size less than buffers, we cannot start trace */
+ if (nr_pages < HISI_PTT_TRACE_TOTAL_BUF_SIZE / PAGE_SIZE)
+ return NULL;
+
+ buf = kzalloc(sizeof(*buf), GFP_KERNEL);
+ if (!buf)
+ return NULL;
+
+ pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
+ if (!pagelist) {
+ kfree(buf);
+ return NULL;
+ }
+
+ for (i = 0; i < nr_pages; i++)
+ pagelist[i] = virt_to_page(pages[i]);
+
+ buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
+ if (!buf->base) {
+ kfree(pagelist);
+ kfree(buf);
+ return NULL;
+ }
+
+ buf->nr_pages = nr_pages;
+ buf->length = nr_pages * PAGE_SIZE;
+ buf->pos = 0;
+
+ kfree(pagelist);
+ return buf;
+}
+
+static void hisi_ptt_pmu_free_aux(void *aux)
+{
+ struct hisi_ptt_pmu_buf *buf = aux;
+
+ vunmap(buf->base);
+ kfree(buf);
+}
+
+static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
+{
+ struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+ struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+ struct hw_perf_event *hwc = &event->hw;
+ struct hisi_ptt_pmu_buf *buf;
+ int cpu = event->cpu;
+ int ret;
+
+ hwc->state = 0;
+ mutex_lock(&hisi_ptt->mutex);
+ if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+ pci_dbg(hisi_ptt->pdev, "trace has already started\n");
+ goto stop;
+ }
+
+ if (cpu == -1)
+ cpu = hisi_ptt->trace_ctrl.default_cpu;
+
+ /*
+ * Handle the interrupt on the same cpu which starts the trace to avoid
+ * context mismatch. Otherwise we'll trigger the WARN from the perf
+ * core in event_function_local().
+ */
+ WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
+ cpumask_of(cpu)));
+
+ ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
+ if (ret) {
+ pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
+ goto stop;
+ }
+
+ buf = perf_aux_output_begin(handle, event);
+ if (!buf) {
+ pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
+ goto stop;
+ }
+
+ buf->pos = handle->head % buf->length;
+
+ ret = hisi_ptt_trace_start(hisi_ptt);
+ if (ret) {
+ pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
+ perf_aux_output_end(handle, 0);
+ goto stop;
+ }
+
+ mutex_unlock(&hisi_ptt->mutex);
+ return;
+stop:
+ event->hw.state |= PERF_HES_STOPPED;
+ mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
+{
+ struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+
+ if (hwc->state & PERF_HES_STOPPED)
+ return;
+
+ mutex_lock(&hisi_ptt->mutex);
+ if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+ hisi_ptt_trace_end(hisi_ptt);
+ WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
+ hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
+ }
+ mutex_unlock(&hisi_ptt->mutex);
+
+ hwc->state |= PERF_HES_STOPPED;
+ perf_event_update_userpage(event);
+ hwc->state |= PERF_HES_UPTODATE;
+}
+
+static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
+{
+ struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ int cpu = event->cpu;
+
+ if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
+ return 0;
+
+ hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+
+ if (flags & PERF_EF_START) {
+ hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
+ if (hwc->state & PERF_HES_STOPPED)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
+{
+ hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
+}
+
+static void hisi_ptt_unregister_pmu(void *priv)
+{
+ perf_pmu_unregister(priv);
+}
+
+static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
+{
+ char *pmu_name;
+ int ret;
+
+ hisi_ptt->hisi_ptt_pmu = (struct pmu) {
+ .module = THIS_MODULE,
+ .capabilities = PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
+ .task_ctx_nr = perf_sw_context,
+ .attr_groups = hisi_ptt_pmu_groups,
+ .event_init = hisi_ptt_pmu_event_init,
+ .setup_aux = hisi_ptt_pmu_setup_aux,
+ .free_aux = hisi_ptt_pmu_free_aux,
+ .start = hisi_ptt_pmu_start,
+ .stop = hisi_ptt_pmu_stop,
+ .add = hisi_ptt_pmu_add,
+ .del = hisi_ptt_pmu_del,
+ };
+
+ pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
+ hisi_ptt->sicl_id, hisi_ptt->core_id);
+ if (!pmu_name)
+ return -ENOMEM;
+
+ ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
+ if (ret)
+ return ret;
+
+ return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
+ hisi_ptt_unregister_pmu,
+ &hisi_ptt->hisi_ptt_pmu);
+}
+
/*
* The DMA of PTT trace can only use direct mapping, due to some
* hardware restriction. Check whether there is an iommu or the
@@ -359,6 +810,12 @@ static int hisi_ptt_probe(struct pci_dev *pdev,
hisi_ptt_init_ctrls(hisi_ptt);
+ ret = hisi_ptt_register_pmu(hisi_ptt);
+ if (ret) {
+ pci_err(pdev, "failed to register pmu device, ret = %d", ret);
+ return ret;
+ }
+
ret = hisi_ptt_check_iommu_mapping(hisi_ptt);
if (ret) {
pci_err(pdev, "cannot work with non-direct DMA mapping.\n");
diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
index 8aa5d6b78cc5..002b64dfa42d 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.h
+++ b/drivers/hwtracing/ptt/hisi_ptt.h
@@ -13,6 +13,7 @@
#include <linux/list.h>
#include <linux/mutex.h>
#include <linux/pci.h>
+#include <linux/perf_event.h>
#include <linux/types.h>
/*
@@ -83,6 +84,7 @@ struct hisi_ptt_dma_buffer {
/**
* struct hisi_ptt_trace_ctrl - control and status of PTT trace
* @status: current trace status
+ * @handle: perf output handle of current trace session
* @trace_buf: the trace buffers for holding the trace data
* @default_cpu: default cpu to start the trace session
* @buffer_size: the size of each trace buffer in the list
@@ -95,6 +97,7 @@ struct hisi_ptt_dma_buffer {
*/
struct hisi_ptt_trace_ctrl {
enum hisi_ptt_trace_status status;
+ struct perf_output_handle handle;
struct list_head trace_buf;
int default_cpu;
u32 buffer_size;
@@ -118,9 +121,25 @@ struct hisi_ptt_filter_desc {
u16 val;
};
+
+/**
+ * struct hisi_ptt_pmu_buf - descriptor of the AUX buffer of PTT trace
+ * @length: size of the AUX buffer
+ * @nr_pages: number of pages of the AUX buffer
+ * @base: start address of AUX buffer
+ * @pos: position in the AUX buffer to commit traced data
+ */
+struct hisi_ptt_pmu_buf {
+ size_t length;
+ int nr_pages;
+ void *base;
+ long pos;
+};
+
/**
* struct hisi_ptt - per PTT device data
* @trace_ctrl: the control information of PTT trace
+ * @hisi_ptt_pmu: the pum device of trace
* @iobase: base IO address of the device
* @pdev: pci_dev of this PTT device
* @mutex: mutex to protect the filter list and serialize the perf process.
@@ -135,6 +154,7 @@ struct hisi_ptt_filter_desc {
*/
struct hisi_ptt {
struct hisi_ptt_trace_ctrl trace_ctrl;
+ struct pmu hisi_ptt_pmu;
void __iomem *iobase;
struct pci_dev *pdev;
struct mutex mutex;
@@ -156,4 +176,6 @@ struct hisi_ptt {
u16 port_mask;
};
+#define to_hisi_ptt(pmu) container_of(pmu, struct hisi_ptt, hisi_ptt_pmu)
+
#endif /* _HISI_PTT_H */
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v3 3/8] hisi_ptt: Add support for dynamically updating the filter list
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
2022-01-24 13:11 ` [PATCH v3 1/8] hwtracing: Add trace function " Yicong Yang
2022-01-24 13:11 ` [PATCH v3 2/8] hisi_ptt: Register PMU device for PTT trace Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-01-24 13:11 ` [PATCH v3 4/8] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device Yicong Yang
` (5 subsequent siblings)
8 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
The PCIe devices supported by the PTT trace can be removed/rescanned
by hotplug or through sysfs. Add support for dynamically updating
the available filter list by registering a PCI bus notifier block.
Then user can always get latest information about available tracing
filters and driver can block the invalid filters of which related
devices no longer exist in the system.
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
drivers/hwtracing/ptt/hisi_ptt.c | 141 ++++++++++++++++++++++++++++---
drivers/hwtracing/ptt/hisi_ptt.h | 39 +++++++++
2 files changed, 167 insertions(+), 13 deletions(-)
diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
index eeb8afc65a56..2994354e690b 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.c
+++ b/drivers/hwtracing/ptt/hisi_ptt.c
@@ -271,26 +271,120 @@ static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
return 0;
}
-static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+static void hisi_ptt_update_filters(struct work_struct *work)
{
+ struct delayed_work *delayed_work = to_delayed_work(work);
+ struct hisi_ptt_filter_update_info info;
struct hisi_ptt_filter_desc *filter;
- struct hisi_ptt *hisi_ptt = data;
struct list_head *target_list;
+ struct hisi_ptt *hisi_ptt;
- target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
- &hisi_ptt->port_filters : &hisi_ptt->req_filters;
+ hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
- filter = kzalloc(sizeof(*filter), GFP_KERNEL);
- if (!filter)
- return -ENOMEM;
+ if (!mutex_trylock(&hisi_ptt->mutex)) {
+ schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
+ return;
+ }
- filter->pdev = pdev;
- filter->val = hisi_ptt_get_filter_val(pdev);
- list_add_tail(&filter->list, target_list);
+ while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
+ target_list = info.is_port ? &hisi_ptt->port_filters :
+ &hisi_ptt->req_filters;
+
+ if (info.is_add) {
+ filter = kzalloc(sizeof(*filter), GFP_KERNEL);
+ if (!filter)
+ continue;
+
+ filter->pdev = info.pdev;
+ filter->val = info.val;
+
+ list_add_tail(&filter->list, target_list);
+ } else {
+ list_for_each_entry(filter, target_list, list)
+ if (filter->val == info.val) {
+ list_del(&filter->list);
+ kfree(filter);
+ break;
+ }
+ }
- /* Update the available port mask */
- if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
- hisi_ptt->port_mask |= filter->val;
+ /* Update the available port mask */
+ if (!info.is_port)
+ continue;
+
+ if (info.is_add)
+ hisi_ptt->port_mask |= info.val;
+ else
+ hisi_ptt->port_mask &= ~info.val;
+ }
+
+ mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
+ struct hisi_ptt_filter_update_info *info)
+{
+ struct pci_dev *root_port = pcie_find_root_port(info->pdev);
+
+ if (!root_port)
+ return;
+
+ info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
+ if (info->port_devid < hisi_ptt->lower ||
+ info->port_devid > hisi_ptt->upper)
+ return;
+
+ info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
+ info->val = hisi_ptt_get_filter_val(info->pdev);
+
+ if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
+ &hisi_ptt->filter_update_lock))
+ schedule_delayed_work(&hisi_ptt->work, 0);
+ else
+ pci_warn(hisi_ptt->pdev,
+ "filter update fifo overflow for target %s\n",
+ pci_name(info->pdev));
+}
+
+/*
+ * A PCI bus notifier is used here for dynamically updating the filter
+ * list.
+ */
+static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
+ void *data)
+{
+ struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
+ struct hisi_ptt_filter_update_info info;
+ struct device *dev = data;
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ info.pdev = pdev;
+
+ switch (action) {
+ case BUS_NOTIFY_ADD_DEVICE:
+ info.is_add = true;
+ break;
+ case BUS_NOTIFY_DEL_DEVICE:
+ info.is_add = false;
+ break;
+ default:
+ return 0;
+ }
+
+ hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+ return 0;
+}
+
+static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+{
+ struct hisi_ptt_filter_update_info info = {
+ .pdev = pdev,
+ .is_add = true,
+ };
+ struct hisi_ptt *hisi_ptt = data;
+
+ hisi_ptt_update_fifo_in(hisi_ptt, &info);
return 0;
}
@@ -316,6 +410,9 @@ static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
struct pci_bus *bus;
u32 reg;
+ INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
+ spin_lock_init(&hisi_ptt->filter_update_lock);
+ INIT_KFIFO(hisi_ptt->filter_update_kfifo);
INIT_LIST_HEAD(&hisi_ptt->port_filters);
INIT_LIST_HEAD(&hisi_ptt->req_filters);
@@ -336,6 +433,13 @@ static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
+ /*
+ * No need to fail if the bus is NULL here as the device
+ * maybe hotplugged after the PTT driver probe, in which
+ * case we can detect the event and update the list as
+ * we register a bus notifier for dynamically updating
+ * the filter list.
+ */
bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
if (bus)
pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
@@ -822,6 +926,12 @@ static int hisi_ptt_probe(struct pci_dev *pdev,
return ret;
}
+ /* Register the bus notifier for dynamically updating the filter list */
+ hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
+ ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+ if (ret)
+ pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
+
return 0;
}
@@ -829,6 +939,11 @@ void hisi_ptt_remove(struct pci_dev *pdev)
{
struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
+ bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+
+ /* Cancel any work that has been queued */
+ cancel_delayed_work_sync(&hisi_ptt->work);
+
if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
hisi_ptt_trace_end(hisi_ptt);
diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
index 002b64dfa42d..151e9e544653 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.h
+++ b/drivers/hwtracing/ptt/hisi_ptt.h
@@ -10,11 +10,15 @@
#define _HISI_PTT_H
#include <linux/bits.h>
+#include <linux/kfifo.h>
#include <linux/list.h>
#include <linux/mutex.h>
+#include <linux/notifier.h>
#include <linux/pci.h>
#include <linux/perf_event.h>
+#include <linux/spinlock.h>
#include <linux/types.h>
+#include <linux/workqueue.h>
/*
* The definition of the device registers and register fields.
@@ -50,6 +54,10 @@
#define HISI_PTT_TRACE_BUF_SIZE SZ_4M
#define HISI_PTT_TRACE_TOTAL_BUF_SIZE (HISI_PTT_TRACE_BUF_SIZE * \
HISI_PTT_TRACE_BUF_CNT)
+/* FIFO size for dynamically updating the PTT trace filter list. */
+#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE 16
+/* Delay time for filter updating work */
+#define HISI_PTT_WORK_DELAY_MS 100UL
/* Wait time for DMA hardware to reset */
#define HISI_PTT_RESET_WAIT_MS 1000UL
/* Poll timeout and interval for waiting hardware work to finish */
@@ -136,6 +144,22 @@ struct hisi_ptt_pmu_buf {
long pos;
};
+/**
+ * struct hisi_ptt_filter_update_info - information for PTT filter updating
+ * @pdev: the PCI device to update in the filter list
+ * @port_devid: the device ID of the PCI device
+ * @is_port: whether the PCI device is a root port
+ * @is_add: adding to the filter or not
+ * @val: the filter value related to the PCI device
+ */
+struct hisi_ptt_filter_update_info {
+ struct pci_dev *pdev;
+ u32 port_devid;
+ bool is_port;
+ bool is_add;
+ u16 val;
+};
+
/**
* struct hisi_ptt - per PTT device data
* @trace_ctrl: the control information of PTT trace
@@ -151,9 +175,13 @@ struct hisi_ptt_pmu_buf {
* @port_filters: the filter list of root ports
* @req_filters: the filter list of requester ID
* @port_mask: port mask of the managed root ports
+ * @work: delayed work for filter updating
+ * @filter_update_lock: spinlock to protect the filter update fifo
+ * @filter_update_fifo: fifo of the filters waiting to update the filter list
*/
struct hisi_ptt {
struct hisi_ptt_trace_ctrl trace_ctrl;
+ struct notifier_block hisi_ptt_nb;
struct pmu hisi_ptt_pmu;
void __iomem *iobase;
struct pci_dev *pdev;
@@ -174,6 +202,17 @@ struct hisi_ptt {
struct list_head port_filters;
struct list_head req_filters;
u16 port_mask;
+
+ /*
+ * We use a delayed work here to avoid indefinitely waiting for
+ * the hisi_ptt->mutex which protecting the filter list. The
+ * work will be delayed only if the mutex can not be held,
+ * otherwise no delay will be applied.
+ */
+ struct delayed_work work;
+ spinlock_t filter_update_lock;
+ DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
+ HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
};
#define to_hisi_ptt(pmu) container_of(pmu, struct hisi_ptt, hisi_ptt_pmu)
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v3 4/8] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
` (2 preceding siblings ...)
2022-01-24 13:11 ` [PATCH v3 3/8] hisi_ptt: Add support for dynamically updating the filter list Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-02-07 11:49 ` Jonathan Cameron
2022-01-24 13:11 ` [PATCH v3 5/8] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
` (4 subsequent siblings)
8 siblings, 1 reply; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
Add tune function for the HiSilicon Tune and Trace device. The interface
of tune is exposed through sysfs attributes of PTT PMU device.
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
drivers/hwtracing/ptt/hisi_ptt.c | 154 +++++++++++++++++++++++++++++++
drivers/hwtracing/ptt/hisi_ptt.h | 19 ++++
2 files changed, 173 insertions(+)
diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
index 2994354e690b..b11e702eb506 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.c
+++ b/drivers/hwtracing/ptt/hisi_ptt.c
@@ -21,6 +21,159 @@
#include "hisi_ptt.h"
+static int hisi_ptt_wait_tuning_finish(struct hisi_ptt *hisi_ptt)
+{
+ u32 val;
+
+ return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TUNING_INT_STAT,
+ val, !(val & HISI_PTT_TUNING_INT_STAT_MASK),
+ HISI_PTT_WAIT_POLL_INTERVAL_US,
+ HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static int hisi_ptt_tune_data_get(struct hisi_ptt *hisi_ptt,
+ u32 event, u16 *data)
+{
+ u32 reg;
+
+ reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+ reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+ reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+ event);
+ writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+ /* Write all 1 to indicates it's the read process */
+ writel(~0UL, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+ if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+ return -ETIMEDOUT;
+
+ reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+ reg &= HISI_PTT_TUNING_DATA_VAL_MASK;
+ *data = (u16)reg;
+
+ return 0;
+}
+
+static int hisi_ptt_tune_data_set(struct hisi_ptt *hisi_ptt,
+ u32 event, u16 data)
+{
+ u32 reg;
+
+ reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+ reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+ reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+ event);
+ writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+ reg = data;
+ writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+ if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+ return -ETIMEDOUT;
+
+ return 0;
+}
+
+static ssize_t hisi_ptt_tune_attr_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+ struct dev_ext_attribute *ext_attr;
+ struct hisi_ptt_tune_desc *desc;
+ int ret;
+ u16 val;
+
+ ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+ desc = ext_attr->var;
+
+ if (!mutex_trylock(&hisi_ptt->mutex))
+ return -EBUSY;
+
+ ret = hisi_ptt_tune_data_get(hisi_ptt, desc->event_code, &val);
+
+ mutex_unlock(&hisi_ptt->mutex);
+ return ret ? ret : sysfs_emit(buf, "%u\n", val);
+}
+
+static ssize_t hisi_ptt_tune_attr_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+ struct dev_ext_attribute *ext_attr;
+ struct hisi_ptt_tune_desc *desc;
+ int ret;
+ u16 val;
+
+ ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+ desc = ext_attr->var;
+
+ if (kstrtou16(buf, 10, &val))
+ return -EINVAL;
+
+ if (!mutex_trylock(&hisi_ptt->mutex))
+ return -EBUSY;
+
+ ret = hisi_ptt_tune_data_set(hisi_ptt, desc->event_code, val);
+
+ mutex_unlock(&hisi_ptt->mutex);
+ return ret ? ret : count;
+}
+
+#define HISI_PTT_TUNE_ATTR(_name, _val, _show, _store) \
+ static struct hisi_ptt_tune_desc _name##_desc = { \
+ .name = #_name, \
+ .event_code = _val, \
+ }; \
+ static struct dev_ext_attribute hisi_ptt_##_name##_attr = { \
+ .attr = __ATTR(_name, 0600, _show, _store), \
+ .var = &_name##_desc, \
+ }
+
+#define HISI_PTT_TUNE_ATTR_COMMON(_name, _val) \
+ HISI_PTT_TUNE_ATTR(_name, _val, \
+ hisi_ptt_tune_attr_show, \
+ hisi_ptt_tune_attr_store)
+
+/*
+ * The value of the tuning event are composed of two parts: main event code in bit[0,15] and
+ * subevent code in bit[16,23]. For example, qox_tx_cpl is a subevent of 'Tx path QoS control'
+ * which for tuning the weight of Tx completion TLPs. See hisi_ptt.rst documentation for
+ * more information.
+ */
+#define HISI_PTT_TUNE_QOS_TX_CPL (0x4 | (3 << 16))
+#define HISI_PTT_TUNE_QOS_TX_NP (0x4 | (4 << 16))
+#define HISI_PTT_TUNE_QOS_TX_P (0x4 | (5 << 16))
+#define HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL (0x5 | (6 << 16))
+#define HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL (0x5 | (7 << 16))
+
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_cpl,
+ HISI_PTT_TUNE_QOS_TX_CPL);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_np,
+ HISI_PTT_TUNE_QOS_TX_NP);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_p,
+ HISI_PTT_TUNE_QOS_TX_P);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_iob_rx_req_alloc_buf_level,
+ HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_tx_req_alloc_buf_level,
+ HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL);
+
+static struct attribute *hisi_ptt_tune_attrs[] = {
+ &hisi_ptt_qos_tx_cpl_attr.attr.attr,
+ &hisi_ptt_qos_tx_np_attr.attr.attr,
+ &hisi_ptt_qos_tx_p_attr.attr.attr,
+ &hisi_ptt_tx_path_iob_rx_req_alloc_buf_level_attr.attr.attr,
+ &hisi_ptt_tx_path_tx_req_alloc_buf_level_attr.attr.attr,
+ NULL,
+};
+
+static struct attribute_group hisi_ptt_tune_group = {
+ .attrs = hisi_ptt_tune_attrs,
+ .name = "tune",
+};
+
static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
{
if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
@@ -516,6 +669,7 @@ static struct attribute_group hisi_ptt_pmu_filter_group = {
static const struct attribute_group *hisi_ptt_pmu_groups[] = {
&hisi_ptt_pmu_format_group,
&hisi_ptt_pmu_filter_group,
+ &hisi_ptt_tune_group,
NULL
};
diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
index 151e9e544653..c350f6d91c69 100644
--- a/drivers/hwtracing/ptt/hisi_ptt.h
+++ b/drivers/hwtracing/ptt/hisi_ptt.h
@@ -23,6 +23,11 @@
/*
* The definition of the device registers and register fields.
*/
+#define HISI_PTT_TUNING_CTRL 0x0000
+#define HISI_PTT_TUNING_CTRL_CODE GENMASK(15, 0)
+#define HISI_PTT_TUNING_CTRL_SUB GENMASK(23, 16)
+#define HISI_PTT_TUNING_DATA 0x0004
+#define HISI_PTT_TUNING_DATA_VAL_MASK GENMASK(15, 0)
#define HISI_PTT_TRACE_ADDR_SIZE 0x0800
#define HISI_PTT_TRACE_ADDR_BASE_LO_0 0x0810
#define HISI_PTT_TRACE_ADDR_BASE_HI_0 0x0814
@@ -38,6 +43,8 @@
#define HISI_PTT_TRACE_INT_STAT 0x0890
#define HISI_PTT_TRACE_INT_STAT_MASK GENMASK(3, 0)
#define HISI_PTT_TRACE_INT_MASK 0x0894
+#define HISI_PTT_TUNING_INT_STAT 0x0898
+#define HISI_PTT_TUNING_INT_STAT_MASK BIT(0)
#define HISI_PTT_TRACE_WR_STS 0x08a0
#define HISI_PTT_TRACE_WR_STS_WRITE GENMASK(27, 0)
#define HISI_PTT_TRACE_WR_STS_BUFFER GENMASK(29, 28)
@@ -71,6 +78,18 @@ enum hisi_ptt_trace_status {
HISI_PTT_TRACE_STATUS_ON,
};
+/**
+ * struct hisi_ptt_tune_desc - describe tune event for PTT tune
+ * @hisi_ptt: PTT device this tune event belongs to
+ * @name: name of this event
+ * @event_code: code of the event
+ */
+struct hisi_ptt_tune_desc {
+ struct hisi_ptt *hisi_ptt;
+ const char *name;
+ u32 event_code;
+};
+
/**
* struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
* The detail of the data format is described
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v3 5/8] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
` (3 preceding siblings ...)
2022-01-24 13:11 ` [PATCH v3 4/8] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-02-07 11:55 ` Jonathan Cameron
2022-01-24 13:11 ` [PATCH v3 6/8] docs: Add HiSilicon PTT device driver documentation Yicong Yang
` (3 subsequent siblings)
8 siblings, 1 reply; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
From: Qi Liu <liuqi115@huawei.com>
'perf record' and 'perf report --dump-raw-trace' supported in this
patch.
Example usage:
Output will contain raw PTT data and its textual representation, such
as:
0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000 offset: 0
ref: 0xa5d50c725 idx: 0 tid: -1 cpu: 0
.
. ... HISI PTT data: size 4194304 bytes
. 00000000: 00 00 00 00 Prefix
. 00000004: 08 20 00 60 Header DW0
. 00000008: ff 02 00 01 Header DW1
. 0000000c: 20 08 00 00 Header DW2
. 00000010: 10 e7 44 ab Header DW3
. 00000014: 2a a8 1e 01 Time
. 00000020: 00 00 00 00 Prefix
. 00000024: 01 00 00 60 Header DW0
. 00000028: 0f 1e 00 01 Header DW1
. 0000002c: 04 00 00 00 Header DW2
. 00000030: 40 00 81 02 Header DW3
. 00000034: ee 02 00 00 Time
....
Signed-off-by: Qi Liu <liuqi115@huawei.com>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
tools/perf/arch/arm/util/auxtrace.c | 56 ++++-
tools/perf/arch/arm/util/pmu.c | 3 +
tools/perf/arch/arm64/util/Build | 2 +-
tools/perf/arch/arm64/util/hisi_ptt.c | 195 +++++++++++++++
tools/perf/util/Build | 2 +
tools/perf/util/auxtrace.c | 4 +
tools/perf/util/auxtrace.h | 1 +
tools/perf/util/hisi-ptt-decoder/Build | 1 +
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c | 170 +++++++++++++
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h | 28 +++
tools/perf/util/hisi_ptt.c | 228 ++++++++++++++++++
tools/perf/util/hisi_ptt.h | 28 +++
12 files changed, 714 insertions(+), 4 deletions(-)
create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
create mode 100644 tools/perf/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi_ptt.h
diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
index 5fc6a2a3dbc5..dc9d2172464e 100644
--- a/tools/perf/arch/arm/util/auxtrace.c
+++ b/tools/perf/arch/arm/util/auxtrace.c
@@ -4,6 +4,7 @@
* Author: Mathieu Poirier <mathieu.poirier@linaro.org>
*/
+#include <dirent.h>
#include <stdbool.h>
#include <linux/coresight-pmu.h>
#include <linux/zalloc.h>
@@ -14,6 +15,7 @@
#include "../../../util/pmu.h"
#include "cs-etm.h"
#include "arm-spe.h"
+#include "hisi_ptt.h"
static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
{
@@ -50,6 +52,39 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
return arm_spe_pmus;
}
+static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
+{
+ struct perf_pmu **hisi_ptt_pmus = NULL;
+ int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+ DIR *dir = NULL;
+ struct dirent *dent;
+
+ hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * nr_cpus);
+ if (!hisi_ptt_pmus) {
+ pr_err("hisi_ptt alloc failed\n");
+ *err = -ENOMEM;
+ return NULL;
+ }
+
+ dir = opendir("/sys/devices");
+ dent = readdir(dir);
+ while (dent) {
+ if (strstr(dent->d_name, HISI_PTT_PMU_NAME)) {
+ hisi_ptt_pmus[*nr_ptts] = perf_pmu__find(dent->d_name);
+ if (hisi_ptt_pmus[*nr_ptts]) {
+ pr_debug2("%s %d: arm_spe_pmu %d type %d name %s\n",
+ __func__, __LINE__, *nr_ptts,
+ hisi_ptt_pmus[*nr_ptts]->type,
+ hisi_ptt_pmus[*nr_ptts]->name);
+ (*nr_ptts)++;
+ }
+ }
+ dent = readdir(dir);
+ }
+
+ return hisi_ptt_pmus;
+}
+
struct auxtrace_record
*auxtrace_record__init(struct evlist *evlist, int *err)
{
@@ -57,8 +92,12 @@ struct auxtrace_record
struct evsel *evsel;
bool found_etm = false;
struct perf_pmu *found_spe = NULL;
+ struct perf_pmu *found_ptt = NULL;
struct perf_pmu **arm_spe_pmus = NULL;
+ struct perf_pmu **hisi_ptt_pmus = NULL;
+
int nr_spes = 0;
+ int nr_ptts = 0;
int i = 0;
if (!evlist)
@@ -66,13 +105,14 @@ struct auxtrace_record
cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
+ hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
evlist__for_each_entry(evlist, evsel) {
if (cs_etm_pmu &&
evsel->core.attr.type == cs_etm_pmu->type)
found_etm = true;
- if (!nr_spes || found_spe)
+ if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
continue;
for (i = 0; i < nr_spes; i++) {
@@ -81,11 +121,18 @@ struct auxtrace_record
break;
}
}
+
+ for (i = 0; i < nr_ptts; i++) {
+ if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
+ found_ptt = hisi_ptt_pmus[i];
+ break;
+ }
+ }
}
free(arm_spe_pmus);
- if (found_etm && found_spe) {
- pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
+ if (found_etm && found_spe && found_ptt) {
+ pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
*err = -EOPNOTSUPP;
return NULL;
}
@@ -96,6 +143,9 @@ struct auxtrace_record
#if defined(__aarch64__)
if (found_spe)
return arm_spe_recording_init(err, found_spe);
+
+ if (found_ptt)
+ return hisi_ptt_recording_init(err, found_ptt);
#endif
/*
diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
index b8b23b9dc598..89a3cedb4557 100644
--- a/tools/perf/arch/arm/util/pmu.c
+++ b/tools/perf/arch/arm/util/pmu.c
@@ -10,6 +10,7 @@
#include <linux/string.h>
#include "arm-spe.h"
+#include "hisi_ptt.h"
#include "../../../util/pmu.h"
struct perf_event_attr
@@ -22,6 +23,8 @@ struct perf_event_attr
#if defined(__aarch64__)
} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
return arm_spe_pmu_default_config(pmu);
+ } else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
+ pmu->selectable = true;
#endif
}
diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
index 9fcb4e68add9..8b7fd1dc9f37 100644
--- a/tools/perf/arch/arm64/util/Build
+++ b/tools/perf/arch/arm64/util/Build
@@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
../../arm/util/auxtrace.o \
../../arm/util/cs-etm.o \
- arm-spe.o mem-events.o
+ arm-spe.o mem-events.o hisi_ptt.o
diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
new file mode 100644
index 000000000000..de3a3523ffd5
--- /dev/null
+++ b/tools/perf/arch/arm64/util/hisi_ptt.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/log2.h>
+#include <linux/zalloc.h>
+#include <time.h>
+
+#include <internal/lib.h> // page_size
+#include "../../../util/auxtrace.h"
+#include "../../../util/cpumap.h"
+#include "../../../util/debug.h"
+#include "../../../util/event.h"
+#include "../../../util/evlist.h"
+#include "../../../util/evsel.h"
+#include "../../../util/hisi_ptt.h"
+#include "../../../util/pmu.h"
+#include "../../../util/record.h"
+#include "../../../util/session.h"
+#include "../../../util/tsc.h"
+
+#define DEFAULT_PAGE_SIZE 1024
+#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
+#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
+
+struct hisi_ptt_recording {
+ struct auxtrace_record itr;
+ struct perf_pmu *hisi_ptt_pmu;
+ struct evlist *evlist;
+};
+
+static size_t
+hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
+ struct evlist *evlist __maybe_unused)
+{
+ return HISI_PTT_AUXTRACE_PRIV_SIZE;
+}
+
+static int hisi_ptt_info_fill(struct auxtrace_record *itr,
+ struct perf_session *session,
+ struct perf_record_auxtrace_info *auxtrace_info,
+ size_t priv_size)
+{
+ struct hisi_ptt_recording *pttr =
+ container_of(itr, struct hisi_ptt_recording, itr);
+ struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+
+ if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
+ return -EINVAL;
+
+ if (!session->evlist->core.nr_mmaps)
+ return -EINVAL;
+
+ auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
+ auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
+
+ return 0;
+}
+
+static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
+{
+ bool privileged = perf_event_paranoid_check(-1);
+
+ if (!opts->full_auxtrace)
+ return 0;
+
+ if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
+ if (privileged) {
+ opts->auxtrace_mmap_pages = MiB(16) / page_size;
+ } else {
+ opts->auxtrace_mmap_pages = KiB(128) / page_size;
+ if (opts->mmap_pages == UINT_MAX)
+ opts->mmap_pages = KiB(256) / page_size;
+ }
+ }
+
+ /* Validate auxtrace_mmap_pages */
+ if (opts->auxtrace_mmap_pages) {
+ size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
+ size_t min_sz = KiB(8);
+
+ if (sz < min_sz || !is_power_of_2(sz)) {
+ pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
+ min_sz / DEFAULT_PAGE_SIZE);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int hisi_ptt_recording_options(struct auxtrace_record *itr,
+ struct evlist *evlist,
+ struct record_opts *opts)
+{
+ struct hisi_ptt_recording *pttr =
+ container_of(itr, struct hisi_ptt_recording, itr);
+ struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+ struct perf_cpu_map *cpus = evlist->core.cpus;
+ struct evsel *evsel, *hisi_ptt_evsel = NULL;
+ struct evsel *tracking_evsel;
+ int err;
+
+ pttr->evlist = evlist;
+ evlist__for_each_entry(evlist, evsel) {
+ if (evsel->core.attr.type == hisi_ptt_pmu->type) {
+ if (hisi_ptt_evsel) {
+ pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
+ return -EINVAL;
+ }
+ evsel->core.attr.freq = 0;
+ evsel->core.attr.sample_period = 1;
+ hisi_ptt_evsel = evsel;
+ opts->full_auxtrace = true;
+ }
+ }
+
+ err = hisi_ptt_set_auxtrace_mmap_page(opts);
+ if (err)
+ return err;
+ /*
+ * To obtain the auxtrace buffer file descriptor, the auxtrace event
+ * must come first.
+ */
+ evlist__to_front(evlist, hisi_ptt_evsel);
+
+ if (!perf_cpu_map__empty(cpus)) {
+ evsel__set_sample_bit(hisi_ptt_evsel, TIME);
+ evsel__set_sample_bit(hisi_ptt_evsel, CPU);
+ }
+
+ /* Add dummy event to keep tracking */
+ err = parse_events(evlist, "dummy:u", NULL);
+ if (err)
+ return err;
+
+ tracking_evsel = evlist__last(evlist);
+ evlist__set_tracking_event(evlist, tracking_evsel);
+
+ tracking_evsel->core.attr.freq = 0;
+ tracking_evsel->core.attr.sample_period = 1;
+
+ if (!perf_cpu_map__empty(cpus))
+ evsel__set_sample_bit(tracking_evsel, TIME);
+
+ return 0;
+}
+
+static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
+{
+ return rdtsc();
+}
+
+static void hisi_ptt_recording_free(struct auxtrace_record *itr)
+{
+ struct hisi_ptt_recording *pttr =
+ container_of(itr, struct hisi_ptt_recording, itr);
+
+ free(pttr);
+}
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+ struct perf_pmu *hisi_ptt_pmu)
+{
+ struct hisi_ptt_recording *pttr;
+
+ if (!hisi_ptt_pmu) {
+ *err = -ENODEV;
+ return NULL;
+ }
+
+ pttr = zalloc(sizeof(struct hisi_ptt_recording));
+ if (!pttr) {
+ *err = -ENOMEM;
+ return NULL;
+ }
+
+ pttr->hisi_ptt_pmu = hisi_ptt_pmu;
+ pttr->itr.pmu = hisi_ptt_pmu;
+ pttr->itr.recording_options = hisi_ptt_recording_options;
+ pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
+ pttr->itr.info_fill = hisi_ptt_info_fill;
+ pttr->itr.free = hisi_ptt_recording_free;
+ pttr->itr.reference = hisi_ptt_reference;
+ pttr->itr.read_finish = auxtrace_record__read_finish;
+ pttr->itr.alignment = 0;
+
+ *err = 0;
+ return &pttr->itr;
+}
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 2a403cefcaf2..a32d1ef8ce29 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -115,6 +115,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
perf-$(CONFIG_AUXTRACE) += intel-bts.o
perf-$(CONFIG_AUXTRACE) += arm-spe.o
perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
+perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
ifdef CONFIG_LIBOPENCSD
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index 825336304a37..7f1ee06538a0 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -51,6 +51,7 @@
#include "intel-pt.h"
#include "intel-bts.h"
#include "arm-spe.h"
+#include "hisi_ptt.h"
#include "s390-cpumsf.h"
#include "util/mmap.h"
@@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
case PERF_AUXTRACE_S390_CPUMSF:
err = s390_cpumsf_process_auxtrace_info(event, session);
break;
+ case PERF_AUXTRACE_HISI_PTT:
+ err = hisi_ptt_process_auxtrace_info(event, session);
+ break;
case PERF_AUXTRACE_UNKNOWN:
default:
return -EINVAL;
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index 19910b9011f3..0f7eaad4d8d0 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -48,6 +48,7 @@ enum auxtrace_type {
PERF_AUXTRACE_CS_ETM,
PERF_AUXTRACE_ARM_SPE,
PERF_AUXTRACE_S390_CPUMSF,
+ PERF_AUXTRACE_HISI_PTT,
};
enum itrace_period_type {
diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
new file mode 100644
index 000000000000..db3db8b75033
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/Build
@@ -0,0 +1 @@
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
new file mode 100644
index 000000000000..3fb7a6949209
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <endian.h>
+#include <byteswap.h>
+#include <linux/bitops.h>
+#include <stdarg.h>
+
+#include "../color.h"
+#include "hisi-ptt-pkt-decoder.h"
+
+/*
+ * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
+ * used to distinguish the data format.
+ * 8DW format is like:
+ * bits [ 31:11 ][ 10:0 ]
+ * |---------------------------------------|-------------------|
+ * DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
+ * DW1 [ Prefix ]
+ * DW2 [ Header DW0 ]
+ * DW3 [ Header DW1 ]
+ * DW4 [ Header DW2 ]
+ * DW5 [ Header DW3 ]
+ * DW6 [ Reserved (0x0) ]
+ * DW7 [ Time ]
+ *
+ * 4DW format is like:
+ * bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
+ * |-----|---------|---|---|---|---|-------------|-------------|
+ * DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
+ * DW1 [ Header DW1 ]
+ * DW2 [ Header DW2 ]
+ * DW3 [ Header DW3 ]
+ */
+
+enum hisi_ptt_8dw_pkt_field_type {
+ HISI_PTT_8DW_PREFIX,
+ HISI_PTT_8DW_HEAD0,
+ HISI_PTT_8DW_HEAD1,
+ HISI_PTT_8DW_HEAD2,
+ HISI_PTT_8DW_HEAD3,
+ HISI_PTT_8DW_TIME,
+ HISI_PTT_8DW_TYPE_MAX
+};
+
+enum hisi_ptt_4dw_pkt_field_type {
+ HISI_PTT_4DW_HEAD1,
+ HISI_PTT_4DW_HEAD2,
+ HISI_PTT_4DW_HEAD3,
+ HISI_PTT_4DW_TYPE_MAX
+};
+
+static const char * const hisi_ptt_8dw_pkt_field_name[] = {
+ [HISI_PTT_8DW_PREFIX] = "Prefix",
+ [HISI_PTT_8DW_HEAD0] = "Header DW0",
+ [HISI_PTT_8DW_HEAD1] = "Header DW1",
+ [HISI_PTT_8DW_HEAD2] = "Header DW2",
+ [HISI_PTT_8DW_HEAD3] = "Header DW3",
+ [HISI_PTT_8DW_TIME] = "Time",
+};
+
+static const char * const hisi_ptt_4dw_pkt_field_name[] = {
+ [HISI_PTT_4DW_HEAD1] = "Header DW1",
+ [HISI_PTT_4DW_HEAD2] = "Header DW2",
+ [HISI_PTT_4DW_HEAD3] = "Header DW3",
+};
+
+/* offset of each member is determined by format of 8dw packet. */
+static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
+ [HISI_PTT_8DW_PREFIX] = 4,
+ [HISI_PTT_8DW_HEAD0] = 4,
+ [HISI_PTT_8DW_HEAD1] = 4,
+ [HISI_PTT_8DW_HEAD2] = 4,
+ [HISI_PTT_8DW_HEAD3] = 4,
+ [HISI_PTT_8DW_TIME] = 8,
+};
+
+union hisi_ptt_4dw {
+ struct {
+ uint32_t format : 2;
+ uint32_t type : 5;
+ uint32_t t9 : 1;
+ uint32_t t8 : 1;
+ uint32_t th : 1;
+ uint32_t so : 1;
+ uint32_t len : 10;
+ uint32_t time : 11;
+ };
+ uint32_t value;
+};
+
+static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
+{
+ const char *color = PERF_COLOR_BLUE;
+ int field_len = sizeof(uint32_t);
+ int i;
+
+ printf(".");
+ color_fprintf(stdout, color, " %08x: ", *pos);
+ for (i = 0; i < field_len; i++)
+ color_fprintf(stdout, color, "%02x ", buf[i]);
+ for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+ color_fprintf(stdout, color, " ");
+ color_fprintf(stdout, color, " %s\n", desc);
+ *pos += field_len;
+}
+
+static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
+{
+ int i;
+
+ for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
+ buf += hisi_ptt_8dw_pkt_field_offset[i];
+ hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
+ }
+
+ return HISI_PTT_8DW_PKT_SIZE;
+}
+
+static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
+{
+ const char *color = PERF_COLOR_BLUE;
+ int field_len = sizeof(uint32_t);
+ union hisi_ptt_4dw dw0;
+ int i;
+
+ dw0.value = *(uint32_t *)buf;
+ printf(".");
+ color_fprintf(stdout, color, " %08x: ", *pos);
+ for (i = 0; i < field_len; i++)
+ color_fprintf(stdout, color, "%02x ", buf[i]);
+ for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+ color_fprintf(stdout, color, " ");
+
+ color_fprintf(stdout, color,
+ " %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
+ "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
+ "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
+ dw0.len, "Time", dw0.time);
+
+ *pos += field_len;
+}
+
+static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
+{
+ int i;
+
+ hisi_ptt_4dw_print_dw0(buf, &pos);
+
+ for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
+ buf += sizeof(uint32_t);
+ hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
+ }
+
+ return HISI_PTT_4DW_PKT_SIZE;
+}
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
+{
+ if (type == HISI_PTT_8DW_PKT)
+ return hisi_ptt_8dw_kpt_desc(buf, pos);
+
+ return hisi_ptt_4dw_kpt_desc(buf, pos);
+}
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
new file mode 100644
index 000000000000..20c51f230cac
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
+#define INCLUDE__HISI_PTT_PKT_DECODER_H__
+
+#include <stddef.h>
+#include <stdint.h>
+
+#define HISI_PTT_PKT_DESC_MAX 256
+#define HISI_PTT_NEED_MORE_BYTES -1
+#define HISI_PTT_8DW_CHECK_MASK GENMASK(31, 11)
+#define HISI_PTT_IS_8DW_PKT GENMASK(31, 11)
+#define HISI_PTT_8DW_PKT_SIZE 32
+#define HISI_PTT_4DW_PKT_SIZE 16
+#define HISI_PTT_MAX_SPACE_LEN 10
+
+enum hisi_ptt_pkt_type {
+ HISI_PTT_4DW_PKT,
+ HISI_PTT_8DW_PKT,
+};
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
+
+#endif
diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
new file mode 100644
index 000000000000..75fa89f3fae3
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.c
@@ -0,0 +1,228 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <byteswap.h>
+#include <endian.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/types.h>
+#include <linux/zalloc.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "auxtrace.h"
+#include "color.h"
+#include "debug.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "hisi_ptt.h"
+#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
+#include "machine.h"
+#include "session.h"
+#include "symbol.h"
+#include "tool.h"
+#include "util/synthetic-events.h"
+
+struct hisi_ptt {
+ struct auxtrace auxtrace;
+ struct auxtrace_queues queues;
+ u32 auxtrace_type;
+ struct perf_session *session;
+ struct machine *machine;
+ u32 pmu_type;
+};
+
+struct hisi_ptt_queue {
+ struct hisi_ptt *ptt;
+ struct auxtrace_buffer *buffer;
+};
+
+static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
+{
+ uint32_t head = *(uint32_t *)buf;
+
+ if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
+ return HISI_PTT_8DW_PKT;
+
+ return HISI_PTT_4DW_PKT;
+}
+
+static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
+ unsigned char *buf, size_t len)
+{
+ const char *color = PERF_COLOR_BLUE;
+ enum hisi_ptt_pkt_type type;
+ size_t pos = 0;
+ int pkt_len;
+
+ color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
+ len);
+
+ type = hisi_ptt_check_packet_type(buf);
+ while (len) {
+ pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
+ if (!pkt_len)
+ color_fprintf(stdout, color, " Bad packet!\n");
+
+ pos += pkt_len;
+ buf += pkt_len;
+ len -= pkt_len;
+ }
+}
+
+static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
+ size_t len)
+{
+ printf(".\n");
+
+ hisi_ptt_dump(ptt, buf, len);
+}
+
+static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
+ union perf_event *event __maybe_unused,
+ struct perf_sample *sample __maybe_unused,
+ struct perf_tool *tool __maybe_unused)
+{
+ return 0;
+}
+
+static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
+ union perf_event *event,
+ struct perf_tool *tool __maybe_unused)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+ auxtrace);
+ struct auxtrace_buffer *buffer;
+ off_t data_offset;
+ int fd = perf_data__fd(session->data);
+ int err;
+
+ if (perf_data__is_pipe(session->data)) {
+ data_offset = 0;
+ } else {
+ data_offset = lseek(fd, 0, SEEK_CUR);
+ if (data_offset == -1)
+ return -errno;
+ }
+
+ err = auxtrace_queues__add_event(&ptt->queues, session, event,
+ data_offset, &buffer);
+ if (err)
+ return err;
+
+ if (dump_trace) {
+ if (auxtrace_buffer__get_data(buffer, fd)) {
+ hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
+ auxtrace_buffer__put_data(buffer);
+ }
+ }
+
+ return 0;
+}
+
+static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
+ struct perf_tool *tool __maybe_unused)
+{
+ return 0;
+}
+
+static void hisi_ptt_free_queue(void *priv)
+{
+ struct hisi_ptt_queue *pttq = priv;
+
+ if (!pttq)
+ return;
+
+ free(pttq);
+}
+
+static void hisi_ptt_free_events(struct perf_session *session)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+ auxtrace);
+ struct auxtrace_queues *queues = &ptt->queues;
+ unsigned int i;
+
+ for (i = 0; i < queues->nr_queues; i++) {
+ hisi_ptt_free_queue(queues->queue_array[i].priv);
+ queues->queue_array[i].priv = NULL;
+ }
+ auxtrace_queues__free(queues);
+}
+
+static void hisi_ptt_free(struct perf_session *session)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+ auxtrace);
+
+ hisi_ptt_free_events(session);
+ session->auxtrace = NULL;
+ free(ptt);
+}
+
+static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
+ struct evsel *evsel)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
+
+ return evsel->core.attr.type == ptt->pmu_type;
+}
+
+static const char * const hisi_ptt_info_fmts[] = {
+ [HISI_PTT_PMU_TYPE] = " PMU Type %" PRId64 "\n",
+};
+
+static void hisi_ptt_print_info(__u64 *arr)
+{
+ if (!dump_trace)
+ return;
+
+ fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
+}
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+ struct perf_session *session)
+{
+ struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
+ struct hisi_ptt *ptt;
+ int err;
+
+ if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
+ sizeof(struct perf_record_auxtrace_info))
+ return -EINVAL;
+
+ ptt = zalloc(sizeof(struct hisi_ptt));
+ if (!ptt)
+ return -ENOMEM;
+
+ err = auxtrace_queues__init(&ptt->queues);
+ if (err)
+ goto err_free;
+
+ ptt->session = session;
+ ptt->machine = &session->machines.host; /* No kvm support */
+ ptt->auxtrace_type = auxtrace_info->type;
+ ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
+
+ ptt->auxtrace.process_event = hisi_ptt_process_event;
+ ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
+ ptt->auxtrace.flush_events = hisi_ptt_flush;
+ ptt->auxtrace.free_events = hisi_ptt_free_events;
+ ptt->auxtrace.free = hisi_ptt_free;
+ ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
+ session->auxtrace = &ptt->auxtrace;
+
+ hisi_ptt_print_info(&auxtrace_info->priv[0]);
+
+ return 0;
+
+err_free:
+ free(ptt);
+ return err;
+}
diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
new file mode 100644
index 000000000000..c0b6cbde1221
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#ifndef INCLUDE__PERF_HISI_PTT_H__
+#define INCLUDE__PERF_HISI_PTT_H__
+
+#define HISI_PTT_PMU_NAME "hisi_ptt"
+enum {
+ HISI_PTT_PMU_TYPE,
+ HISI_PTT_PER_CPU_MMAPS,
+ HISI_PTT_AUXTRACE_PRIV_MAX,
+};
+
+#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
+union perf_event;
+struct perf_session;
+struct perf_pmu;
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+ struct perf_pmu *hisi_ptt_pmu);
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+ struct perf_session *session);
+
+#endif
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v3 6/8] docs: Add HiSilicon PTT device driver documentation
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
` (4 preceding siblings ...)
2022-01-24 13:11 ` [PATCH v3 5/8] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-02-07 12:12 ` Jonathan Cameron
2022-01-24 13:11 ` [PATCH v3 7/8] MAINTAINERS: Add maintainer for HiSilicon PTT driver Yicong Yang
` (2 subsequent siblings)
8 siblings, 1 reply; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
Document the introduction and usage of HiSilicon PTT device driver.
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
Documentation/trace/hisi-ptt.rst | 304 +++++++++++++++++++++++++++++++
1 file changed, 304 insertions(+)
create mode 100644 Documentation/trace/hisi-ptt.rst
diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
new file mode 100644
index 000000000000..f3269b11a2f6
--- /dev/null
+++ b/Documentation/trace/hisi-ptt.rst
@@ -0,0 +1,304 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+HiSilicon PCIe Tune and Trace device
+======================================
+
+Introduction
+============
+
+HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
+integrated Endpoint (RCiEP) device, providing the capability
+to dynamically monitor and tune the PCIe link's events (tune),
+and trace the TLP headers (trace). The two functions are independent,
+but is recommended to use them together to analyze and enhance the
+PCIe link's performance.
+
+On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
+PCIe cores. Each PCIe core includes several Root Ports and a PTT
+RCiEP, like below. The PTT device is capable of tuning and
+tracing the link of the PCIe core.
+::
+ +--------------Core 0-------+
+ | | [ PTT ] |
+ | | [Root Port]---[Endpoint]
+ | | [Root Port]---[Endpoint]
+ | | [Root Port]---[Endpoint]
+ Root Complex |------Core 1-------+
+ | | [ PTT ] |
+ | | [Root Port]---[ Switch ]---[Endpoint]
+ | | [Root Port]---[Endpoint] `-[Endpoint]
+ | | [Root Port]---[Endpoint]
+ +---------------------------+
+
+The PTT device driver registers PMU device for each PTT device.
+The name of each PTT device is composed of 'hisi_ptt' prefix with
+the id of the SICL and the Core where it locates. The Kunpeng 930
+SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
+IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
+Complex for each SICL.
+::
+ /sys/devices/hisi_ptt<sicl_id>_<core_id>
+
+Tune
+====
+
+PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
+Currently we support events in 4 classes. The scope of the events
+covers the PCIe core to which the PTT device belongs.
+
+Each event is presented as a file under $(PTT PMU dir)/tune, and
+mostly a simple open/read/write/close cycle will be used to tune
+the event.
+::
+ $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
+ $ ls
+ qos_tx_cpl qos_tx_np qos_tx_p
+ tx_path_rx_req_alloc_buf_level
+ tx_path_tx_req_alloc_buf_level
+ $ cat qos_tx_dp
+ 1
+ $ echo 2 > qos_tx_dp
+ $ cat qos_tx_dp
+ 2
+
+Current value (numerical value) of the event can be simply read
+from the file, and the desired value written to the file to tune.
+
+1. Tx path QoS control
+------------------------
+
+The following files are provided to tune the QoS of the tx path of
+the PCIe core.
+
+- qos_tx_cpl: weight of Tx completion TLPs
+- qos_tx_np: weight of Tx non-posted TLPs
+- qos_tx_p: weight of Tx posted TLPs
+
+The weight influences the proportion of certain packets on the PCIe link.
+For example, for the storage scenario, increase the proportion
+of the completion packets on the link to enhance the performance as
+more completions are consumed.
+
+The available tune data of these events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+2. Tx path buffer control
+-------------------------
+
+Following files are provided to tune the buffer of tx path of the PCIe core.
+
+- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
+- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
+
+These events influence the watermark of the buffer allocated for each
+type. Rx means the inbound while Tx means outbound. The packets will
+be stored in the buffer first and then posted either when the watermark
+reached or when timed out. For a busy direction, you should increase
+the related buffer watermark to avoid frequently posting and thus
+enhance the performance. In most cases just keep the default value.
+
+The available tune data of above events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+Trace
+=====
+
+PTT trace is designed for dumping the TLP headers to the memory, which
+can be used to analyze the transactions and usage condition of the PCIe
+Link. You can choose to filter the traced headers by either requester ID,
+or those downstream of a set of Root Ports on the same core of the PTT
+device. It's also supported to trace the headers of certain type and of
+certain direction.
+
+You can use the perf command `perf record` to set the parameters, start
+trace and get the data. It's also supported to decode the trace
+data with `perf report`. The control parameters for trace is inputted
+as event code for each events, which will be further illustracted later.
+An example usage is like
+::
+ $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
+ format=1/ -- sleep 5
+
+This will trace the TLP headers downstream root port 0000:00:10.1 (event
+code for event 'filter' is 0x80001) with type of posted TLP requests,
+direction of inbound and traced data format of 8DW.
+
+1. filter
+---------
+
+The TLP headers to trace can be filtered by the Root Ports or the requester
+ID of the endpoints, which are locates on the same core of the PTT device.
+You can set the filter by spedifying the `filter` parameter which is required
+to start the trace. The parameter value is 20 bit. The supported filters and
+related values is outputted through `available_filters` sysfs attribute
+under related PTT PMU directory, classified as Root Ports and Requesters
+respectively.
+::
+ $ cat available_filters
+ #### Root Ports ####
+ 0000:00:10.0 0x80001
+ 0000:00:11.0 0x80004
+ #### Requesters ####
+ 0000:01:00.0 0x00100
+ 0000:01:00.1 0x00101
+
+Note that multiple Root Ports can be specified at one time, but only
+one Endpoint function can be specified in one trace. Specifying both
+Root Port and function at the same time is not supported.
+
+If no filter is available, reading the available_filters will get the hint.
+::
+ $ cat available_filters
+ #### No available filter ####
+
+The available_filters can be dynamically updated, which means you can always
+get correct filter information when hotplug events happen, or when you manually
+remove/rescan the devices.
+
+2. type
+-------
+
+You can trace the TLP headers of certain types by specifying the `type`
+parameter, which is required to start the trace. The parameter value is
+8 bit. Current supported types and related values are shown below:
+
+8'b00000001: posted requests (P)
+8'b00000010: non-posted requests (NP)
+8'b00000100: completions (CPL)
+
+You can specify multiple types when tracing inbound TLP headers, but can only
+specify one when tracing outbound TLP headers.
+
+3. direction
+------------
+
+You can trace the TLP headers from certain direction, which is relative
+to the Root Port or the PCIe core, by specifying the `direction` parameter.
+This is optional and the default parameter is inbound. The parameter value
+is 4 bit. When the desired format is 4DW, directions and related values
+supported are shown below:
+
+4'b0000: inbound TLPs (P, NP, CPL)
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
+4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
+
+When the desired format is 8DW, directions and related values supported are
+shown below:
+
+4'b0000: reserved
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: inbound TLPs (P, NP, CPL B)
+4'b0011: inbound TLPs (CPL A)
+
+Inbound completions are classifed into two types:
+
+completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
+completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
+
+4. format
+--------------
+
+You can change the format of the traced TLP headers by specifying the
+`format` parameter. This is optional and the default format is 4DW.
+The parameter value is 4 bit. Current supported formats and related
+values are shown below:
+
+4'b0000: 4DW length per TLP header
+4'b0001: 8DW length per TLP header
+
+The traced TLP header format is different from the PCIe standard.
+
+When using the 8DW data format, the entire TLP header is logged
+(Header DW0-3 shown below). For example, the TLP header for Memory
+Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
+the header for Configuration Requests is shown in Figure 2.20, etc.
+
+In addition, 8DW trace buffer entries contain a timestamp and
+possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
+Otherwise this field will be all 0.
+
+The bit[31:11] of DW0 is always 0x1fffff, which can be
+used to distinguish the data format. 8DW format is like
+::
+ bits [ 31:11 ][ 10:0 ]
+ |---------------------------------------|-------------------|
+ DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
+ DW1 [ Prefix ]
+ DW2 [ Header DW0 ]
+ DW3 [ Header DW1 ]
+ DW4 [ Header DW2 ]
+ DW5 [ Header DW3 ]
+ DW6 [ Reserved (0x0) ]
+ DW7 [ Time ]
+
+When using the 4DW data format, DW0 of the trace buffer entry
+contains selected fields of DW0 of the TLP, together with a
+timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
+directly from the TLP header.
+
+4DW format is like
+::
+ bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
+ |-----|---------|---|---|---|---|-------------|-------------|
+ DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
+ DW1 [ Header DW1 ]
+ DW2 [ Header DW2 ]
+ DW3 [ Header DW3 ]
+
+5. memory management
+--------------------
+
+The traced TLP headers will be written to the memory allocated
+by the driver. The hardware accepts 4 DMA address with same size,
+and writes the buffer sequentially like below. If DMA addr 3 is
+finished and the trace is still on, it will return to addr 0.
+::
+ +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
+ +---------------------------------------------------------+
+
+Driver will allocate each DMA buffer of 4MiB. The finished buffer
+will be copied to the perf AUX buffer allocated by the perf core.
+Once the AUX buffer is full while the trace is still on, driver
+will commit the AUX buffer first and then apply for a new one with
+the same size. The size of AUX buffer is default to 16MiB. User can
+adjust the size by specifying the `-m` parameter of the perf command.
+
+Note that there is a gap between committing the old AUX buffer and
+applying a new one, which means the trace is stopped during the
+moment and TLPs transferred in the moment cannot be traced. To avoid
+this situation, you should begin the trace with large AUX buffer
+enough to avoid this gap.
+
+6. decoding
+-----------
+
+You can decode the traced data with `perf report -D` command (currently
+only support to dump the raw trace data). The traced data will be decoded
+according to the format described previously (take 8DW as an example):
+::
+ [...perf headers and other information]
+ . ... HISI PTT data: size 4194304 bytes
+ . 00000000: 00 00 00 00 Prefix
+ . 00000004: 01 00 00 60 Header DW0
+ . 00000008: 0f 1e 00 01 Header DW1
+ . 0000000c: 04 00 00 00 Header DW2
+ . 00000010: 40 00 81 02 Header DW3
+ . 00000014: 33 c0 04 00 Time
+ . 00000020: 00 00 00 00 Prefix
+ . 00000024: 01 00 00 60 Header DW0
+ . 00000028: 0f 1e 00 01 Header DW1
+ . 0000002c: 04 00 00 00 Header DW2
+ . 00000030: 40 00 81 02 Header DW3
+ . 00000034: 02 00 00 00 Time
+ . 00000040: 00 00 00 00 Prefix
+ . 00000044: 01 00 00 60 Header DW0
+ . 00000048: 0f 1e 00 01 Header DW1
+ . 0000004c: 04 00 00 00 Header DW2
+ . 00000050: 40 00 81 02 Header DW3
+ [...]
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v3 7/8] MAINTAINERS: Add maintainer for HiSilicon PTT driver
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
` (5 preceding siblings ...)
2022-01-24 13:11 ` [PATCH v3 6/8] docs: Add HiSilicon PTT device driver documentation Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-01-24 13:11 ` [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
2022-02-07 9:40 ` [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
8 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
Add maintainer for driver and documentation of HiSilicon PTT device.
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
MAINTAINERS | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index ea3e6c914384..237c618a74d5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8689,6 +8689,13 @@ F: Documentation/admin-guide/perf/hisi-pcie-pmu.rst
F: Documentation/admin-guide/perf/hisi-pmu.rst
F: drivers/perf/hisilicon
+HISILICON PTT DRIVER
+M: Yicong Yang <yangyicong@hisilicon.com>
+L: linux-kernel@vger.kernel.org
+S: Maintained
+F: Documentation/trace/hisi-ptt.rst
+F: drivers/hwtracing/ptt/
+
HISILICON QM AND ZIP Controller DRIVER
M: Zhou Wang <wangzhou1@hisilicon.com>
L: linux-crypto@vger.kernel.org
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
` (6 preceding siblings ...)
2022-01-24 13:11 ` [PATCH v3 7/8] MAINTAINERS: Add maintainer for HiSilicon PTT driver Yicong Yang
@ 2022-01-24 13:11 ` Yicong Yang
2022-02-08 8:05 ` John Garry
2022-02-14 12:55 ` Yicong Yang
2022-02-07 9:40 ` [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
8 siblings, 2 replies; 32+ messages in thread
From: Yicong Yang @ 2022-01-24 13:11 UTC (permalink / raw)
To: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, jonathan.cameron, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, yangyicong, song.bao.hua
The DMA of HiSilicon PTT device can only work with identical
mapping. So add a quirk for the device to force the domain
passthrough.
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6dc6d8b6b368..6f67a2b1dd27 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
}
}
+#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
+ (pdev)->device == 0xa12e)
+
+static int arm_smmu_def_domain_type(struct device *dev)
+{
+ if (dev_is_pci(dev)) {
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ if (IS_HISI_PTT_DEVICE(pdev))
+ return IOMMU_DOMAIN_IDENTITY;
+ }
+
+ return 0;
+}
+
static struct iommu_ops arm_smmu_ops = {
.capable = arm_smmu_capable,
.domain_alloc = arm_smmu_domain_alloc,
@@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
.sva_unbind = arm_smmu_sva_unbind,
.sva_get_pasid = arm_smmu_sva_get_pasid,
.page_response = arm_smmu_page_response,
+ .def_domain_type = arm_smmu_def_domain_type,
.pgsize_bitmap = -1UL, /* Restricted during device attach */
.owner = THIS_MODULE,
};
--
2.24.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* Re: [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
` (7 preceding siblings ...)
2022-01-24 13:11 ` [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
@ 2022-02-07 9:40 ` Yicong Yang
8 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-07 9:40 UTC (permalink / raw)
To: Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
daniel.thompson, joro, john.garry, shameerali.kolothum.thodi,
robin.murphy, peterz, mingo, acme, linux-kernel,
linux-arm-kernel, coresight, linux-pci, linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, song.bao.hua
Hi perf, ETM and PCI related experts,
a gentle ping ... appreciate for the comments.
thanks.
On 2022/1/24 21:11, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
> integrated Endpoint (RCiEP) device, providing the capability
> to dynamically monitor and tune the PCIe traffic (tune),
> and trace the TLP headers (trace).
>
> PTT tune is designed for monitoring and adjusting PCIe link parameters.
> We provide several parameters of the PCIe link. Through the driver,
> user can adjust the value of certain parameter to affect the PCIe link
> for the purpose of enhancing the performance in certian situation.
>
> PTT trace is designed for dumping the TLP headers to the memory, which
> can be used to analyze the transactions and usage condition of the PCIe
> Link. Users can choose filters to trace headers, by either requester
> ID, or those downstream of a set of Root Ports on the same core of the
> PTT device. It's also supported to trace the headers of certain type and
> of certain direction.
>
> The driver registers a PMU device for each PTT device. The trace can
> be used through `perf record` and the traced headers can be decoded
> by `perf report`. The perf command support for the device is also
> added in this patchset. The tune can be used through the sysfs
> attributes of related PMU device. See the documentation for the
> detailed usage.
>
> Change since v2:
> - address the comments from Mathieu
> - rename the directory to ptt to match the function of the device
> - spinoff the declarations to a separate header
> - split the trace function to several patches
> - some other comments.
> - make default smmu domain type of PTT device to identity
> Drop the RMR as it's not recommended and use an iommu_def_domain_type
> quirk to passthrough the device DMA as suggested by Robin.
> Link: https://lore.kernel.org/linux-pci/20211116090625.53702-1-yangyicong@hisilicon.com/
>
> Change since v1:
> - switch the user interface of trace to perf from debugfs
> - switch the user interface of tune to sysfs from debugfs
> - add perf tool support to start trace and decode the trace data
> - address the comments of documentation from Bjorn
> - add RMR[1] support of the device as trace works in RMR mode or
> direct DMA mode. RMR support is achieved by common APIs rather
> than the APIs implemented in [1].
> Link: https://lore.kernel.org/lkml/1618654631-42454-1-git-send-email-yangyicong@hisilicon.com/
>
> [1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/
>
> Qi Liu (1):
> perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
>
> Yicong Yang (7):
> hwtracing: Add trace function support for HiSilicon PCIe Tune and
> Trace device
> hisi_ptt: Register PMU device for PTT trace
> hisi_ptt: Add support for dynamically updating the filter list
> hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace
> device
> docs: Add HiSilicon PTT device driver documentation
> MAINTAINERS: Add maintainer for HiSilicon PTT driver
> iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to
> identity
>
> Documentation/trace/hisi-ptt.rst | 304 +++++
> MAINTAINERS | 7 +
> drivers/Makefile | 1 +
> drivers/hwtracing/Kconfig | 2 +
> drivers/hwtracing/ptt/Kconfig | 11 +
> drivers/hwtracing/ptt/Makefile | 2 +
> drivers/hwtracing/ptt/hisi_ptt.c | 1124 +++++++++++++++++
> drivers/hwtracing/ptt/hisi_ptt.h | 239 ++++
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 +
> tools/perf/arch/arm/util/auxtrace.c | 56 +-
> tools/perf/arch/arm/util/pmu.c | 3 +
> tools/perf/arch/arm64/util/Build | 2 +-
> tools/perf/arch/arm64/util/hisi_ptt.c | 195 +++
> tools/perf/util/Build | 2 +
> tools/perf/util/auxtrace.c | 4 +
> tools/perf/util/auxtrace.h | 1 +
> tools/perf/util/hisi-ptt-decoder/Build | 1 +
> .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c | 170 +++
> .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h | 28 +
> tools/perf/util/hisi_ptt.c | 228 ++++
> tools/perf/util/hisi_ptt.h | 28 +
> 21 files changed, 2420 insertions(+), 4 deletions(-)
> create mode 100644 Documentation/trace/hisi-ptt.rst
> create mode 100644 drivers/hwtracing/ptt/Kconfig
> create mode 100644 drivers/hwtracing/ptt/Makefile
> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
> create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
> create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
> create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
> create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
> create mode 100644 tools/perf/util/hisi_ptt.c
> create mode 100644 tools/perf/util/hisi_ptt.h
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 1/8] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
2022-01-24 13:11 ` [PATCH v3 1/8] hwtracing: Add trace function " Yicong Yang
@ 2022-02-07 11:42 ` Jonathan Cameron
2022-02-08 11:07 ` Yicong Yang
2022-02-07 18:11 ` John Garry
1 sibling, 1 reply; 32+ messages in thread
From: Jonathan Cameron @ 2022-02-07 11:42 UTC (permalink / raw)
To: Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On Mon, 24 Jan 2022 21:11:11 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> integrated Endpoint(RCiEP) device, providing the capability
> to dynamically monitor and tune the PCIe traffic, and trace
> the TLP headers.
>
> Add the driver for the device to enable the trace function.
> This patch adds basic function of trace, including the device's
> probe and initialization, functions for trace buffer allocation
> and trace enable/disable, register an interrupt handler to
> simply response to the DMA events. The user interface of trace
> will be added in the following patch.
>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Hi Yicong,
I've not been following all the earlier discussion on this driver closely
so I may well raise something that has already been addressed. If so
just ignore the comment.
Thanks,
Jonathan
> ---
> drivers/Makefile | 1 +
> drivers/hwtracing/Kconfig | 2 +
> drivers/hwtracing/ptt/Kconfig | 11 +
> drivers/hwtracing/ptt/Makefile | 2 +
> drivers/hwtracing/ptt/hisi_ptt.c | 398 +++++++++++++++++++++++++++++++
> drivers/hwtracing/ptt/hisi_ptt.h | 159 ++++++++++++
> 6 files changed, 573 insertions(+)
> create mode 100644 drivers/hwtracing/ptt/Kconfig
> create mode 100644 drivers/hwtracing/ptt/Makefile
> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>
> diff --git a/drivers/Makefile b/drivers/Makefile
> index a110338c860c..ab3411e4eba5 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4) += thunderbolt/
> obj-$(CONFIG_CORESIGHT) += hwtracing/coresight/
> obj-y += hwtracing/intel_th/
> obj-$(CONFIG_STM) += hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT) += hwtracing/ptt/
> obj-$(CONFIG_ANDROID) += android/
> obj-$(CONFIG_NVMEM) += nvmem/
> obj-$(CONFIG_FPGA) += fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..911ee977103c 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>
> source "drivers/hwtracing/intel_th/Kconfig"
>
> +source "drivers/hwtracing/ptt/Kconfig"
> +
> endmenu
> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
> new file mode 100644
> index 000000000000..4f4f2459ac47
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Kconfig
> @@ -0,0 +1,11 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> + tristate "HiSilicon PCIe Tune and Trace Device"
> + depends on ARM64 && PCI && HAS_DMA && HAS_IOMEM
> + help
> + HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
> + device, provides support for PCIe traffic tuning and
> + tracing TLP headers to the memory.
> +
> + This driver can also be built as a module. If so, the module
> + will be called hisi_ptt.
> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
> new file mode 100644
> index 000000000000..6d0a0ca5c0a9
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
> @@ -0,0 +1,398 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/module.h>
> +#include <linux/sysfs.h>
> +
> +#include "hisi_ptt.h"
> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> + return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> + return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> + u32 val;
> +
> + return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> + val & HISI_PTT_TRACE_IDLE,
> + HISI_PTT_WAIT_POLL_INTERVAL_US,
> + HISI_PTT_WAIT_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> + struct device *dev = &hisi_ptt->pdev->dev;
> + struct hisi_ptt_dma_buffer *buffer, *tbuffer;
> +
> + list_for_each_entry_safe(buffer, tbuffer, &ctrl->trace_buf, list) {
> + list_del(&buffer->list);
> + dma_free_coherent(dev, buffer->size, buffer->addr,
> + buffer->dma);
> + kfree(buffer);
> + }
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> + struct device *dev = &hisi_ptt->pdev->dev;
> + struct hisi_ptt_dma_buffer *buffer;
> + int i, ret;
> +
> + hisi_ptt->trace_ctrl.buf_index = 0;
> +
> + /* Make sure the trace buffer is empty before allocating */
This comment is misleading as it suggests it not being empty is
a bad thing but the code handles it as an acceptable path.
Perhaps:
/*
* If the trace buffer has already been allocated, zero the
* memory.
*/
> + if (!list_empty(&ctrl->trace_buf)) {
> + list_for_each_entry(buffer, &ctrl->trace_buf, list)
> + memset(buffer->addr, 0, buffer->size);
> + return 0;
> + }
> +
> + for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
> + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
> + if (!buffer) {
> + ret = -ENOMEM;
> + goto err;
> + }
> +
> + buffer->addr = dma_alloc_coherent(dev, ctrl->buffer_size,
> + &buffer->dma, GFP_KERNEL);
> + if (!buffer->addr) {
> + kfree(buffer);
> + ret = -ENOMEM;
> + goto err;
> + }
> +
> + memset(buffer->addr, 0, buffer->size);
See:
https://lore.kernel.org/lkml/20190108130701.14161-4-hch@lst.de/
dma_alloc_coherent() always zeros the memory for us hence there
is no longer a dma_kzalloc_coherent()
> +
> + buffer->index = i;
Carrying an index inside a list which corresponds directly
to the position in the list is not particularly nice.
Why can't we compute this index on the fly where the list
is walked? Or am I misunderstanding and the order of the buffers
is changed in a later patch?
As a side note, is a list actually appropriate when we always
have 4 of these buffers? Feels like an array of buffer
structures might be cheaper.
> + buffer->size = ctrl->buffer_size;
> + list_add_tail(&buffer->list, &ctrl->trace_buf);
> + }
> +
> + return 0;
> +err:
> + hisi_ptt_free_trace_buf(hisi_ptt);
> + return ret;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> + hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> + struct hisi_ptt_dma_buffer *cur;
> + u32 val;
> +
> + /* Check device idle before start trace */
> + if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> + pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> + return -EBUSY;
> + }
> +
> + /* Reset the DMA before start tracing */
> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> + val |= HISI_PTT_TRACE_CTRL_RST;
> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> + /*
> + * We'll be in the perf context where preemption is disabled,
> + * so use busy loop here.
> + */
> + mdelay(HISI_PTT_RESET_WAIT_MS);
Busy look for 1 second? Ouch. If we can reduce this in any way
that would be great or if there is a means to do it before
we disable preemption.
> +
> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> + val &= ~HISI_PTT_TRACE_CTRL_RST;
> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> + /* Clear the interrupt status */
> + writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> + /* Configure the trace DMA buffer */
> + list_for_each_entry(cur, &ctrl->trace_buf, list) {
I comment on the use of cur->index above. Here it would be easy to compute
the index as we go for example assuming we never end up with holes
in the list.
> + writel(lower_32_bits(cur->dma),
> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> + writel(upper_32_bits(cur->dma),
> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> + }
> + writel(ctrl->buffer_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> + /* Set the trace control register */
> + val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> + if (!hisi_ptt->trace_ctrl.is_port)
> + val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> + /* Start the Trace */
> + val |= HISI_PTT_TRACE_CTRL_EN;
> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> + ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> +
> + return 0;
> +}
> +
...
> +
> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> + struct pci_dev *pdev = hisi_ptt->pdev;
> + struct pci_bus *bus;
> + u32 reg;
> +
> + INIT_LIST_HEAD(&hisi_ptt->port_filters);
> + INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> + /*
> + * The device range register provides the information about the
> + * root ports which the RCiEP can control and trace. The RCiEP
> + * and the root ports it support are on the same PCIe core, with
> + * same domain number but maybe different bus number. The device
> + * range register will tell us which root ports we can support,
> + * Bit[31:16] indicates the upper BDF numbers of the root port,
> + * while Bit[15:0] indicates the lower.
> + */
> + reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> + hisi_ptt->upper = reg >> 16;
> + hisi_ptt->lower = reg & 0xffff;
Trivial:
Perhaps worthing define HISI_PTT_DEVICE_RANGE_UPPER_MASK etc adn using
FIELD_GET?
> +
> + reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> + hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> + hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> + bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
> + if (bus)
> + pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> +
> + /* Initialize trace controls */
> + INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> + hisi_ptt->trace_ctrl.buffer_size = HISI_PTT_TRACE_BUF_SIZE;
> + hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +}
> +
...
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
> new file mode 100644
> index 000000000000..8aa5d6b78cc5
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.h
> @@ -0,0 +1,159 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#ifndef _HISI_PTT_H
> +#define _HISI_PTT_H
> +
> +#include <linux/bits.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/types.h>
> +
> +/*
> + * The definition of the device registers and register fields.
> + */
> +#define HISI_PTT_TRACE_ADDR_SIZE 0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0 0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0 0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE 0x8
> +#define HISI_PTT_TRACE_CTRL 0x0850
> +#define HISI_PTT_TRACE_CTRL_EN BIT(0)
> +#define HISI_PTT_TRACE_CTRL_RST BIT(1)
> +#define HISI_PTT_TRACE_CTRL_RXTX_SEL GENMASK(3, 2)
> +#define HISI_PTT_TRACE_CTRL_TYPE_SEL GENMASK(7, 4)
> +#define HISI_PTT_TRACE_CTRL_DATA_FORMAT BIT(14)
> +#define HISI_PTT_TRACE_CTRL_FILTER_MODE BIT(15)
> +#define HISI_PTT_TRACE_CTRL_TARGET_SEL GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT 0x0890
> +#define HISI_PTT_TRACE_INT_STAT_MASK GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK 0x0894
> +#define HISI_PTT_TRACE_WR_STS 0x08a0
> +#define HISI_PTT_TRACE_WR_STS_WRITE GENMASK(27, 0)
> +#define HISI_PTT_TRACE_WR_STS_BUFFER GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS 0x08b0
> +#define HISI_PTT_TRACE_IDLE BIT(0)
> +#define HISI_PTT_DEVICE_RANGE 0x0fe0
> +#define HISI_PTT_LOCATION 0x0fe8
> +#define HISI_PTT_CORE_ID GENMASK(15, 0)
> +#define HISI_PTT_SICL_ID GENMASK(31, 16)
> +
> +/* Parameters of PTT trace DMA part. */
> +#define HISI_PTT_TRACE_DMA_IRQ 0
> +#define HISI_PTT_TRACE_BUF_CNT 4
> +#define HISI_PTT_TRACE_BUF_SIZE SZ_4M
> +#define HISI_PTT_TRACE_TOTAL_BUF_SIZE (HISI_PTT_TRACE_BUF_SIZE * \
> + HISI_PTT_TRACE_BUF_CNT)
> +/* Wait time for DMA hardware to reset */
> +#define HISI_PTT_RESET_WAIT_MS 1000UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TIMEOUT_US 1000000UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US 100UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn) (PCI_FUNC(devfn) << 1)
> +
> +enum hisi_ptt_trace_status {
> + HISI_PTT_TRACE_STATUS_OFF = 0,
> + HISI_PTT_TRACE_STATUS_ON,
> +};
Why not just use a boolean given we only have off and on states?
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 2/8] hisi_ptt: Register PMU device for PTT trace
2022-01-24 13:11 ` [PATCH v3 2/8] hisi_ptt: Register PMU device for PTT trace Yicong Yang
@ 2022-02-07 11:42 ` Jonathan Cameron
2022-02-08 7:41 ` Yicong Yang
0 siblings, 1 reply; 32+ messages in thread
From: Jonathan Cameron @ 2022-02-07 11:42 UTC (permalink / raw)
To: Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On Mon, 24 Jan 2022 21:11:12 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:
> Register PMU device of PTT trace, then users can use
> trace through perf command. The driver makes use of perf
> AUX trace and support following events to configure the
> trace:
>
> - filter: select Root port or Endpoint to trace
> - type: select the type of traced TLP headers
> - direction: select the direction of traced TLP headers
> - format: select the data format of the traced TLP headers
>
> This patch adds the PMU driver part of PTT trace. The perf
> command support of PTT trace is added in the following
> patch.
>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
> @@ -294,6 +346,405 @@ static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> }
>
> +#define HISI_PTT_PMU_FILTER_IS_PORT BIT(19)
> +#define HISI_PTT_PMU_FILTER_VAL_MASK GENMASK(15, 0)
> +#define HISI_PTT_PMU_DIRECTION_MASK GENMASK(23, 20)
> +#define HISI_PTT_PMU_TYPE_MASK GENMASK(31, 24)
> +#define HISI_PTT_PMU_FORMAT_MASK GENMASK(35, 32)
> +
> +static ssize_t available_filters_show(struct device *dev,
> + struct device_attribute *attr,
> + char *buf)
> +{
> + struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> + struct hisi_ptt_filter_desc *filter;
> + int pos = 0;
> +
> + if (list_empty(&hisi_ptt->port_filters))
> + return sysfs_emit(buf, "#### No available filter ####\n");
> +
This is a very unusual sysfs attribute.
They are supposed to be one "thing" per file, so I'd have expected this to
be at least two files
root_ports_available_filters
request_available_filters
and no available filter is indicated by these attribute returning an empty
string.
However you need to match convention for hwtracing drivers so if
this is common approach perhaps you could point me to a similar
example? My grep skills didn't find me one.
> + mutex_lock(&hisi_ptt->mutex);
> + pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> + list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> + pos += sysfs_emit_at(buf, pos, "%s 0x%05lx\n",
> + pci_name(filter->pdev),
> + hisi_ptt_get_filter_val(filter->pdev) |
> + HISI_PTT_PMU_FILTER_IS_PORT);
> +
> + pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> + list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> + pos += sysfs_emit_at(buf, pos, "%s 0x%05x\n",
> + pci_name(filter->pdev),
> + hisi_ptt_get_filter_val(filter->pdev));
> +
> + mutex_unlock(&hisi_ptt->mutex);
> + return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> +
...
> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> +{
> + int i, ret = -EINVAL;
> +
> + for (i = 0; i < list_size; i++)
> + if (val == available_list[i]) {
> + ret = 0;
return 0;
> + break;
> + }
> +
> + return ret;
return -EINVAL;
> +}
> +
> +
> +static void hisi_ptt_pmu_free_aux(void *aux)
> +{
> + struct hisi_ptt_pmu_buf *buf = aux;
> +
> + vunmap(buf->base);
> + kfree(buf);
> +}
> +
...
> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> +{
> + struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> + struct hw_perf_event *hwc = &event->hw;
> + int cpu = event->cpu;
> +
> + if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
This check is not entirely obvious to me. Perhaps a comment would help
readers understand why this condition is successful, but doesn't involve
actually starting the pmu?
> + return 0;
> +
> + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +
> + if (flags & PERF_EF_START) {
> + hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> + if (hwc->state & PERF_HES_STOPPED)
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
...
> /*
> * The DMA of PTT trace can only use direct mapping, due to some
> * hardware restriction. Check whether there is an iommu or the
> @@ -359,6 +810,12 @@ static int hisi_ptt_probe(struct pci_dev *pdev,
>
> hisi_ptt_init_ctrls(hisi_ptt);
>
> + ret = hisi_ptt_register_pmu(hisi_ptt);
> + if (ret) {
> + pci_err(pdev, "failed to register pmu device, ret = %d", ret);
Given I think this exposes userspace interfaces, it should be the very
last thing done in probe(). Otherwise we have a race condition (at least in
theory) where someone starts using it before we then fail the iommu mapping check.
> + return ret;
> + }
> +
> ret = hisi_ptt_check_iommu_mapping(hisi_ptt);
> if (ret) {
> pci_err(pdev, "cannot work with non-direct DMA mapping.\n");
Thanks,
Jonathan
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 4/8] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device
2022-01-24 13:11 ` [PATCH v3 4/8] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device Yicong Yang
@ 2022-02-07 11:49 ` Jonathan Cameron
2022-02-08 7:08 ` Yicong Yang
0 siblings, 1 reply; 32+ messages in thread
From: Jonathan Cameron @ 2022-02-07 11:49 UTC (permalink / raw)
To: Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On Mon, 24 Jan 2022 21:11:14 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:
> Add tune function for the HiSilicon Tune and Trace device. The interface
> of tune is exposed through sysfs attributes of PTT PMU device.
>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
A few trivial things inline, but looks good in general to me.
With those tidied up
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
> drivers/hwtracing/ptt/hisi_ptt.c | 154 +++++++++++++++++++++++++++++++
> drivers/hwtracing/ptt/hisi_ptt.h | 19 ++++
> 2 files changed, 173 insertions(+)
>
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
> index 2994354e690b..b11e702eb506 100644
> --- a/drivers/hwtracing/ptt/hisi_ptt.c
> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
> @@ -21,6 +21,159 @@
>
> #include "hisi_ptt.h"
>
> +static int hisi_ptt_wait_tuning_finish(struct hisi_ptt *hisi_ptt)
> +{
> + u32 val;
> +
> + return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TUNING_INT_STAT,
> + val, !(val & HISI_PTT_TUNING_INT_STAT_MASK),
> + HISI_PTT_WAIT_POLL_INTERVAL_US,
> + HISI_PTT_WAIT_TIMEOUT_US);
> +}
> +
> +static int hisi_ptt_tune_data_get(struct hisi_ptt *hisi_ptt,
> + u32 event, u16 *data)
> +{
> + u32 reg;
> +
> + reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
> + reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
> + reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
> + event);
> + writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
> +
> + /* Write all 1 to indicates it's the read process */
> + writel(~0UL, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
Just to check, this is includes the bits above the DATA_VAL_MASK?
Fine if so, just seems odd to define a field but then write
parts of the register that aren't part of that field.
> +
> + if (hisi_ptt_wait_tuning_finish(hisi_ptt))
> + return -ETIMEDOUT;
> +
> + reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
> + reg &= HISI_PTT_TUNING_DATA_VAL_MASK;
> + *data = (u16)reg;
As below, prefer a FIELD_GET() for this.
> +
> + return 0;
> +}
> +
> +static int hisi_ptt_tune_data_set(struct hisi_ptt *hisi_ptt,
> + u32 event, u16 data)
> +{
> + u32 reg;
> +
> + reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
> + reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
> + reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
> + event);
> + writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
> +
> + reg = data;
Given you defined HISI_PTT_TUNING_DATA_VAL_MASK why not use it here
writel(FIELD_PREP(..), ...)?
> + writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
> +
> + if (hisi_ptt_wait_tuning_finish(hisi_ptt))
> + return -ETIMEDOUT;
> +
> + return 0;
> +}
> +
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 5/8] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
2022-01-24 13:11 ` [PATCH v3 5/8] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
@ 2022-02-07 11:55 ` Jonathan Cameron
0 siblings, 0 replies; 32+ messages in thread
From: Jonathan Cameron @ 2022-02-07 11:55 UTC (permalink / raw)
To: Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On Mon, 24 Jan 2022 21:11:15 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:
> From: Qi Liu <liuqi115@huawei.com>
>
> 'perf record' and 'perf report --dump-raw-trace' supported in this
> patch.
>
> Example usage:
>
> Output will contain raw PTT data and its textual representation, such
> as:
>
> 0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000 offset: 0
> ref: 0xa5d50c725 idx: 0 tid: -1 cpu: 0
> .
> . ... HISI PTT data: size 4194304 bytes
> . 00000000: 00 00 00 00 Prefix
> . 00000004: 08 20 00 60 Header DW0
> . 00000008: ff 02 00 01 Header DW1
> . 0000000c: 20 08 00 00 Header DW2
> . 00000010: 10 e7 44 ab Header DW3
> . 00000014: 2a a8 1e 01 Time
> . 00000020: 00 00 00 00 Prefix
> . 00000024: 01 00 00 60 Header DW0
> . 00000028: 0f 1e 00 01 Header DW1
> . 0000002c: 04 00 00 00 Header DW2
> . 00000030: 40 00 81 02 Header DW3
> . 00000034: ee 02 00 00 Time
> ....
>
> Signed-off-by: Qi Liu <liuqi115@huawei.com>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Hi. This is unfortunately out of my areas of expertise, so I just
took a quick glance and noticed one generic c thing that could be
tidied up.
> diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
> new file mode 100644
> index 000000000000..75fa89f3fae3
> --- /dev/null
> +++ b/tools/perf/util/hisi_ptt.c
...
> +
> +static void hisi_ptt_free_queue(void *priv)
> +{
> + struct hisi_ptt_queue *pttq = priv;
> +
> + if (!pttq)
> + return;
> +
> + free(pttq);
free() is safe against a null ptr, so you don't need the
if (!pttq) return;
See free(3) man page.
> +}
> +
> +static void hisi_ptt_free_events(struct perf_session *session)
> +{
> + struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
> + auxtrace);
> + struct auxtrace_queues *queues = &ptt->queues;
> + unsigned int i;
> +
> + for (i = 0; i < queues->nr_queues; i++) {
> + hisi_ptt_free_queue(queues->queue_array[i].priv);
> + queues->queue_array[i].priv = NULL;
> + }
> + auxtrace_queues__free(queues);
> +}
> +
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 6/8] docs: Add HiSilicon PTT device driver documentation
2022-01-24 13:11 ` [PATCH v3 6/8] docs: Add HiSilicon PTT device driver documentation Yicong Yang
@ 2022-02-07 12:12 ` Jonathan Cameron
2022-02-08 11:09 ` Yicong Yang
0 siblings, 1 reply; 32+ messages in thread
From: Jonathan Cameron @ 2022-02-07 12:12 UTC (permalink / raw)
To: Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On Mon, 24 Jan 2022 21:11:16 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:
> Document the introduction and usage of HiSilicon PTT device driver.
>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
Nice document. A few trivial typos inline.
I would give a RB except I've suggested you change a part of the
sysfs interface which will affect the relevant documentation.
Thanks,
Jonathan
> ---
> Documentation/trace/hisi-ptt.rst | 304 +++++++++++++++++++++++++++++++
> 1 file changed, 304 insertions(+)
> create mode 100644 Documentation/trace/hisi-ptt.rst
>
> diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
> new file mode 100644
> index 000000000000..f3269b11a2f6
> --- /dev/null
> +++ b/Documentation/trace/hisi-ptt.rst
> @@ -0,0 +1,304 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +======================================
> +HiSilicon PCIe Tune and Trace device
> +======================================
> +
> +Introduction
> +============
> +
> +HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
> +integrated Endpoint (RCiEP) device, providing the capability
> +to dynamically monitor and tune the PCIe link's events (tune),
> +and trace the TLP headers (trace). The two functions are independent,
> +but is recommended to use them together to analyze and enhance the
> +PCIe link's performance.
> +
> +On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
> +PCIe cores. Each PCIe core includes several Root Ports and a PTT
> +RCiEP, like below. The PTT device is capable of tuning and
> +tracing the link of the PCIe core.
links
> +::
> + +--------------Core 0-------+
> + | | [ PTT ] |
> + | | [Root Port]---[Endpoint]
> + | | [Root Port]---[Endpoint]
> + | | [Root Port]---[Endpoint]
> + Root Complex |------Core 1-------+
> + | | [ PTT ] |
> + | | [Root Port]---[ Switch ]---[Endpoint]
> + | | [Root Port]---[Endpoint] `-[Endpoint]
> + | | [Root Port]---[Endpoint]
> + +---------------------------+
> +
> +The PTT device driver registers PMU device for each PTT device.
registers one PMU device ..
> +The name of each PTT device is composed of 'hisi_ptt' prefix with
> +the id of the SICL and the Core where it locates. The Kunpeng 930
> +SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
> +IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
> +Complex for each SICL.
> +::
> + /sys/devices/hisi_ptt<sicl_id>_<core_id>
> +
> +Tune
> +====
> +
> +PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
> +Currently we support events in 4 classes. The scope of the events
> +covers the PCIe core to which the PTT device belongs.
> +
> +Each event is presented as a file under $(PTT PMU dir)/tune, and
> +mostly a simple open/read/write/close cycle will be used to tune
drop "mostly" as it doesn't add anything other than potential confusion.
> +the event.
> +::
> + $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
> + $ ls
> + qos_tx_cpl qos_tx_np qos_tx_p
> + tx_path_rx_req_alloc_buf_level
> + tx_path_tx_req_alloc_buf_level
> + $ cat qos_tx_dp
> + 1
> + $ echo 2 > qos_tx_dp
> + $ cat qos_tx_dp
> + 2
> +
> +Current value (numerical value) of the event can be simply read
> +from the file, and the desired value written to the file to tune.
> +
> +1. Tx path QoS control
> +------------------------
> +
> +The following files are provided to tune the QoS of the tx path of
> +the PCIe core.
> +
> +- qos_tx_cpl: weight of Tx completion TLPs
> +- qos_tx_np: weight of Tx non-posted TLPs
> +- qos_tx_p: weight of Tx posted TLPs
> +
> +The weight influences the proportion of certain packets on the PCIe link.
> +For example, for the storage scenario, increase the proportion
> +of the completion packets on the link to enhance the performance as
> +more completions are consumed.
> +
> +The available tune data of these events is [0, 1, 2].
> +Writing a negative value will return an error, and out of range
> +values will be converted to 2. Note that the event value just
> +indicates a probable level, but is not precise.
> +
> +2. Tx path buffer control
> +-------------------------
> +
> +Following files are provided to tune the buffer of tx path of the PCIe core.
> +
> +- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
> +- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
> +
> +These events influence the watermark of the buffer allocated for each
> +type. Rx means the inbound while Tx means outbound. The packets will
> +be stored in the buffer first and then posted either when the watermark
Change "posted" to "transmitted" as posted has a special meaning in PCI
and I don't think that is what you mean here... (I could be wrong!)
> +reached or when timed out. For a busy direction, you should increase
> +the related buffer watermark to avoid frequently posting and thus
> +enhance the performance. In most cases just keep the default value.
> +
> +The available tune data of above events is [0, 1, 2].
> +Writing a negative value will return an error, and out of range
> +values will be converted to 2. Note that the event value just
> +indicates a probable level, but is not precise.
> +
> +Trace
> +=====
> +
> +PTT trace is designed for dumping the TLP headers to the memory, which
> +can be used to analyze the transactions and usage condition of the PCIe
> +Link. You can choose to filter the traced headers by either requester ID,
> +or those downstream of a set of Root Ports on the same core of the PTT
> +device. It's also supported to trace the headers of certain type and of
> +certain direction.
> +
> +You can use the perf command `perf record` to set the parameters, start
> +trace and get the data. It's also supported to decode the trace
> +data with `perf report`. The control parameters for trace is inputted
> +as event code for each events, which will be further illustracted later.
illustrated
> +An example usage is like
> +::
> + $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
> + format=1/ -- sleep 5
> +
> +This will trace the TLP headers downstream root port 0000:00:10.1 (event
> +code for event 'filter' is 0x80001) with type of posted TLP requests,
> +direction of inbound and traced data format of 8DW.
> +
> +1. filter
> +---------
> +
> +The TLP headers to trace can be filtered by the Root Ports or the requester
> +ID of the endpoints, which are locates on the same core of the PTT device.
located
> +You can set the filter by spedifying the `filter` parameter which is required
> +to start the trace. The parameter value is 20 bit. The supported filters and
> +related values is outputted through `available_filters` sysfs attribute
> +under related PTT PMU directory, classified as Root Ports and Requesters
> +respectively.
> +::
> + $ cat available_filters
> + #### Root Ports ####
> + 0000:00:10.0 0x80001
> + 0000:00:11.0 0x80004
> + #### Requesters ####
> + 0000:01:00.0 0x00100
> + 0000:01:00.1 0x00101
> +
> +Note that multiple Root Ports can be specified at one time, but only
> +one Endpoint function can be specified in one trace. Specifying both
> +Root Port and function at the same time is not supported.
> +
> +If no filter is available, reading the available_filters will get the hint.
> +::
> + $ cat available_filters
> + #### No available filter ####
If you take not of my earlier feedback this bit may change slightly.
> +
> +The available_filters can be dynamically updated, which means you can always
> +get correct filter information when hotplug events happen, or when you manually
> +remove/rescan the devices.
> +
> +2. type
> +-------
> +
> +You can trace the TLP headers of certain types by specifying the `type`
> +parameter, which is required to start the trace. The parameter value is
> +8 bit. Current supported types and related values are shown below:
> +
> +8'b00000001: posted requests (P)
> +8'b00000010: non-posted requests (NP)
> +8'b00000100: completions (CPL)
> +
> +You can specify multiple types when tracing inbound TLP headers, but can only
> +specify one when tracing outbound TLP headers.
> +
> +3. direction
> +------------
> +
> +You can trace the TLP headers from certain direction, which is relative
> +to the Root Port or the PCIe core, by specifying the `direction` parameter.
> +This is optional and the default parameter is inbound. The parameter value
> +is 4 bit. When the desired format is 4DW, directions and related values
> +supported are shown below:
> +
> +4'b0000: inbound TLPs (P, NP, CPL)
> +4'b0001: outbound TLPs (P, NP, CPL)
> +4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
> +4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
> +
> +When the desired format is 8DW, directions and related values supported are
> +shown below:
> +
> +4'b0000: reserved
> +4'b0001: outbound TLPs (P, NP, CPL)
> +4'b0010: inbound TLPs (P, NP, CPL B)
> +4'b0011: inbound TLPs (CPL A)
> +
> +Inbound completions are classifed into two types:
classified
> +
> +completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
> +completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
> +
> +4. format
> +--------------
> +
> +You can change the format of the traced TLP headers by specifying the
> +`format` parameter. This is optional and the default format is 4DW.
As there is a default, there is no need to also say it is optional.
`format parameter. The default format is 4DW.
> +The parameter value is 4 bit. Current supported formats and related
> +values are shown below:
> +
> +4'b0000: 4DW length per TLP header
> +4'b0001: 8DW length per TLP header
> +
> +The traced TLP header format is different from the PCIe standard.
> +
> +When using the 8DW data format, the entire TLP header is logged
> +(Header DW0-3 shown below). For example, the TLP header for Memory
> +Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
> +the header for Configuration Requests is shown in Figure 2.20, etc.
> +
> +In addition, 8DW trace buffer entries contain a timestamp and
> +possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
> +Otherwise this field will be all 0.
> +
> +The bit[31:11] of DW0 is always 0x1fffff, which can be
> +used to distinguish the data format. 8DW format is like
> +::
> + bits [ 31:11 ][ 10:0 ]
> + |---------------------------------------|-------------------|
> + DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
> + DW1 [ Prefix ]
> + DW2 [ Header DW0 ]
> + DW3 [ Header DW1 ]
> + DW4 [ Header DW2 ]
> + DW5 [ Header DW3 ]
> + DW6 [ Reserved (0x0) ]
> + DW7 [ Time ]
> +
> +When using the 4DW data format, DW0 of the trace buffer entry
> +contains selected fields of DW0 of the TLP, together with a
> +timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
> +directly from the TLP header.
> +
> +4DW format is like
> +::
> + bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
> + |-----|---------|---|---|---|---|-------------|-------------|
> + DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
> + DW1 [ Header DW1 ]
> + DW2 [ Header DW2 ]
> + DW3 [ Header DW3 ]
> +
> +5. memory management
> +--------------------
> +
> +The traced TLP headers will be written to the memory allocated
> +by the driver. The hardware accepts 4 DMA address with same size,
> +and writes the buffer sequentially like below. If DMA addr 3 is
> +finished and the trace is still on, it will return to addr 0.
> +::
> + +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
> + +---------------------------------------------------------+
> +
> +Driver will allocate each DMA buffer of 4MiB. The finished buffer
> +will be copied to the perf AUX buffer allocated by the perf core.
> +Once the AUX buffer is full while the trace is still on, driver
> +will commit the AUX buffer first and then apply for a new one with
> +the same size. The size of AUX buffer is default to 16MiB. User can
> +adjust the size by specifying the `-m` parameter of the perf command.
> +
> +Note that there is a gap between committing the old AUX buffer and
> +applying a new one, which means the trace is stopped during the
> +moment and TLPs transferred in the moment cannot be traced. To avoid
> +this situation, you should begin the trace with large AUX buffer
> +enough to avoid this gap.
> +
> +6. decoding
> +-----------
> +
> +You can decode the traced data with `perf report -D` command (currently
> +only support to dump the raw trace data). The traced data will be decoded
> +according to the format described previously (take 8DW as an example):
> +::
> + [...perf headers and other information]
> + . ... HISI PTT data: size 4194304 bytes
> + . 00000000: 00 00 00 00 Prefix
> + . 00000004: 01 00 00 60 Header DW0
> + . 00000008: 0f 1e 00 01 Header DW1
> + . 0000000c: 04 00 00 00 Header DW2
> + . 00000010: 40 00 81 02 Header DW3
> + . 00000014: 33 c0 04 00 Time
> + . 00000020: 00 00 00 00 Prefix
> + . 00000024: 01 00 00 60 Header DW0
> + . 00000028: 0f 1e 00 01 Header DW1
> + . 0000002c: 04 00 00 00 Header DW2
> + . 00000030: 40 00 81 02 Header DW3
> + . 00000034: 02 00 00 00 Time
> + . 00000040: 00 00 00 00 Prefix
> + . 00000044: 01 00 00 60 Header DW0
> + . 00000048: 0f 1e 00 01 Header DW1
> + . 0000004c: 04 00 00 00 Header DW2
> + . 00000050: 40 00 81 02 Header DW3
> + [...]
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 1/8] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
2022-01-24 13:11 ` [PATCH v3 1/8] hwtracing: Add trace function " Yicong Yang
2022-02-07 11:42 ` Jonathan Cameron
@ 2022-02-07 18:11 ` John Garry
2022-02-08 8:57 ` Yicong Yang
1 sibling, 1 reply; 32+ messages in thread
From: John Garry @ 2022-02-07 18:11 UTC (permalink / raw)
To: Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
linux-pci, linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, song.bao.hua
On 24/01/2022 13:11, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> integrated Endpoint(RCiEP) device, providing the capability
> to dynamically monitor and tune the PCIe traffic, and trace
> the TLP headers.
>
> Add the driver for the device to enable the trace function.
> This patch adds basic function of trace, including the device's
> probe and initialization, functions for trace buffer allocation
> and trace enable/disable, register an interrupt handler to
> simply response to the DMA events. The user interface of trace
> will be added in the following patch.
>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
> drivers/Makefile | 1 +
> drivers/hwtracing/Kconfig | 2 +
> drivers/hwtracing/ptt/Kconfig | 11 +
> drivers/hwtracing/ptt/Makefile | 2 +
> drivers/hwtracing/ptt/hisi_ptt.c | 398 +++++++++++++++++++++++++++++++
> drivers/hwtracing/ptt/hisi_ptt.h | 159 ++++++++++++
> 6 files changed, 573 insertions(+)
> create mode 100644 drivers/hwtracing/ptt/Kconfig
> create mode 100644 drivers/hwtracing/ptt/Makefile
> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>
> diff --git a/drivers/Makefile b/drivers/Makefile
> index a110338c860c..ab3411e4eba5 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4) += thunderbolt/
> obj-$(CONFIG_CORESIGHT) += hwtracing/coresight/
> obj-y += hwtracing/intel_th/
> obj-$(CONFIG_STM) += hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT) += hwtracing/ptt/
> obj-$(CONFIG_ANDROID) += android/
> obj-$(CONFIG_NVMEM) += nvmem/
> obj-$(CONFIG_FPGA) += fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..911ee977103c 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>
> source "drivers/hwtracing/intel_th/Kconfig"
>
> +source "drivers/hwtracing/ptt/Kconfig"
> +
> endmenu
> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
> new file mode 100644
> index 000000000000..4f4f2459ac47
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Kconfig
> @@ -0,0 +1,11 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> + tristate "HiSilicon PCIe Tune and Trace Device"
> + depends on ARM64 && PCI && HAS_DMA && HAS_IOMEM
> + help
> + HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
exists
> + device, provides support for PCIe traffic tuning and
and it provides support...
> + tracing TLP headers to the memory.
> +
> + This driver can also be built as a module. If so, the module
> + will be called hisi_ptt.
> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
> new file mode 100644
> index 000000000000..6d0a0ca5c0a9
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
> @@ -0,0 +1,398 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/module.h>
> +#include <linux/sysfs.h>
> +
> +#include "hisi_ptt.h"
> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> + return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> + return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> + u32 val;
> +
> + return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> + val & HISI_PTT_TRACE_IDLE,
> + HISI_PTT_WAIT_POLL_INTERVAL_US,
> + HISI_PTT_WAIT_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> + struct device *dev = &hisi_ptt->pdev->dev;
> + struct hisi_ptt_dma_buffer *buffer, *tbuffer;
> +
> + list_for_each_entry_safe(buffer, tbuffer, &ctrl->trace_buf, list) {
> + list_del(&buffer->list);
> + dma_free_coherent(dev, buffer->size, buffer->addr,
> + buffer->dma);
> + kfree(buffer);
> + }
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> + struct device *dev = &hisi_ptt->pdev->dev;
> + struct hisi_ptt_dma_buffer *buffer;
> + int i, ret;
> +
> + hisi_ptt->trace_ctrl.buf_index = 0;
> +
> + /* Make sure the trace buffer is empty before allocating */
> + if (!list_empty(&ctrl->trace_buf)) {
> + list_for_each_entry(buffer, &ctrl->trace_buf, list)
> + memset(buffer->addr, 0, buffer->size);
> + return 0;
> + }
> +
> + for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
> + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
I may have asked this before: why no devm usage?
> + if (!buffer) {
> + ret = -ENOMEM;
> + goto err;
> + }
> +
> + buffer->addr = dma_alloc_coherent(dev, ctrl->buffer_size,
> + &buffer->dma, GFP_KERNEL);
> + if (!buffer->addr) {
> + kfree(buffer);
> + ret = -ENOMEM;
> + goto err;
> + }
> +
> + memset(buffer->addr, 0, buffer->size);
> +
> + buffer->index = i;
> + buffer->size = ctrl->buffer_size;
please double check if we really need to store this info separately,
i.e. is it const and same for all?
> + list_add_tail(&buffer->list, &ctrl->trace_buf);
> + }
> +
> + return 0;
> +err:
> + hisi_ptt_free_trace_buf(hisi_ptt);
> + return ret;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> + hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> + struct hisi_ptt_dma_buffer *cur;
> + u32 val;
> +
> + /* Check device idle before start trace */
> + if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> + pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> + return -EBUSY;
> + }
> +
> + /* Reset the DMA before start tracing */
> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> + val |= HISI_PTT_TRACE_CTRL_RST;
> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> + /*
> + * We'll be in the perf context where preemption is disabled,
> + * so use busy loop here.
what has preemption is disabled got to do with "busy loop"?
> + */
> + mdelay(HISI_PTT_RESET_WAIT_MS);
> +
> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> + val &= ~HISI_PTT_TRACE_CTRL_RST;
> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> + /* Clear the interrupt status */
> + writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> + /* Configure the trace DMA buffer */
> + list_for_each_entry(cur, &ctrl->trace_buf, list) {
> + writel(lower_32_bits(cur->dma),
> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> + writel(upper_32_bits(cur->dma),
> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> + }
> + writel(ctrl->buffer_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> + /* Set the trace control register */
> + val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> + if (!hisi_ptt->trace_ctrl.is_port)
> + val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> + /* Start the Trace */
> + val |= HISI_PTT_TRACE_CTRL_EN;
> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> + ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> +
> + return 0;
> +}
> +
> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> +{
> + struct hisi_ptt *hisi_ptt = context;
> + u32 status;
> +
> + status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> + /* Clear the interrupt status of buffer @buf_idx */
> + writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> + return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> +{
> + struct hisi_ptt *hisi_ptt = context;
> + u32 status;
> +
> + status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> + if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> + return IRQ_NONE;
> +
> + return IRQ_WAKE_THREAD;
Adding empty handler like this is not helpful. And from checking the
later code, the threaded part does nothing special, i.e nothing time
consuming, so I don't know why everything cannot be done in the hard
part for simplicity
> +}
> +
> +static void hisi_ptt_irq_free_vectors(void *pdev)
> +{
> + pci_free_irq_vectors(pdev);
> +}
> +
> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> +{
> + struct pci_dev *pdev = hisi_ptt->pdev;
> + int ret;
> +
> + ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> + if (ret < 0) {
> + pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> + return ret;
> + }
> +
> + ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> + if (ret < 0)
> + return ret;
> +
> + ret = devm_request_threaded_irq(&pdev->dev,
> + pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> + hisi_ptt_irq, hisi_ptt_isr, 0,
> + "hisi-ptt", hisi_ptt);
> + if (ret) {
> + pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> + pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> +{
> + struct hisi_ptt_filter_desc *filter;
> + struct hisi_ptt *hisi_ptt = data;
> + struct list_head *target_list;
> +
> + target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
> + &hisi_ptt->port_filters : &hisi_ptt->req_filters;
> +
> + filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> + if (!filter)
> + return -ENOMEM;
> +
> + filter->pdev = pdev;
> + filter->val = hisi_ptt_get_filter_val(pdev);
why do you need to store this also? if you're storing pdev, you seem to
be able to directly get hisi_ptt_get_filter_val() for it
> + list_add_tail(&filter->list, target_list);
> +
> + /* Update the available port mask */
> + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> + hisi_ptt->port_mask |= filter->val;
> +
> + return 0;
> +}
> +
> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> +{
> + struct hisi_ptt_filter_desc *filter, *tfilter;
> +
> + list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> + list_del(&filter->list);
> + kfree(filter);
> + }
> +
> + list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> + list_del(&filter->list);
> + kfree(filter);
> + }
> +}
> +
> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> + struct pci_dev *pdev = hisi_ptt->pdev;
> + struct pci_bus *bus;
> + u32 reg;
> +
> + INIT_LIST_HEAD(&hisi_ptt->port_filters);
> + INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> + /*
> + * The device range register provides the information about the
> + * root ports which the RCiEP can control and trace. The RCiEP
> + * and the root ports it support are on the same PCIe core, with
> + * same domain number but maybe different bus number. The device
> + * range register will tell us which root ports we can support,
> + * Bit[31:16] indicates the upper BDF numbers of the root port,
> + * while Bit[15:0] indicates the lower.
> + */
> + reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> + hisi_ptt->upper = reg >> 16;
> + hisi_ptt->lower = reg & 0xffff;
> +
> + reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> + hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> + hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> + bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
> + if (bus)
> + pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> +
> + /* Initialize trace controls */
> + INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> + hisi_ptt->trace_ctrl.buffer_size = HISI_PTT_TRACE_BUF_SIZE;
> + hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +}
> +
> +/*
> + * The DMA of PTT trace can only use direct mapping, due to some
> + * hardware restriction. Check whether there is an iommu or the
> + * policy of the iommu domain is passthrough, otherwise the trace
> + * cannot work.
> + */
> +static int hisi_ptt_check_iommu_mapping(struct hisi_ptt *hisi_ptt)
> +{
> + struct pci_dev *pdev = hisi_ptt->pdev;
> + struct iommu_domain *iommu_domain;
> +
> + iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> + if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> + return 0;
so what stops us changing the domain type later?
> +
> + return -EOPNOTSUPP;
> +}
> +
> +static int hisi_ptt_probe(struct pci_dev *pdev,
> + const struct pci_device_id *id)
> +{
> + struct hisi_ptt *hisi_ptt;
> + int ret;
> +
> + hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> + if (!hisi_ptt)
> + return -ENOMEM;
> +
> + mutex_init(&hisi_ptt->mutex);
> + hisi_ptt->pdev = pdev;
> +
> + /*
> + * Lifetime of pci_dev is longer than hisi_ptt,
> + * so directly reference to the pci name string.
> + */
> + hisi_ptt->name = pci_name(hisi_ptt->pdev);
> + pci_set_drvdata(pdev, hisi_ptt);
> +
> + ret = pcim_enable_device(pdev);
> + if (ret) {
> + pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> + return ret;
> + }
> +
> + ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> + if (ret) {
> + pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> + return ret;
> + }
> +
> + hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> +
> + ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> + if (ret) {
> + pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> + return ret;
> + }
> + pci_set_master(pdev);
> +
> + ret = hisi_ptt_register_irq(hisi_ptt);
> + if (ret)
> + return ret;
> +
> + hisi_ptt_init_ctrls(hisi_ptt);
> +
> + ret = hisi_ptt_check_iommu_mapping(hisi_ptt);
surely this should be done earlier in the probe
> + if (ret) {
> + pci_err(pdev, "cannot work with non-direct DMA mapping.\n");
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +void hisi_ptt_remove(struct pci_dev *pdev)
> +{
> + struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> +
> + if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> + hisi_ptt_trace_end(hisi_ptt);
> +
> + hisi_ptt_free_trace_buf(hisi_ptt);
> + hisi_ptt_release_filters(hisi_ptt);
> +}
> +
> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> + { PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> + { }
> +};
> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> +
> +static struct pci_driver hisi_ptt_driver = {
> + .name = "hisi_ptt",
> + .id_table = hisi_ptt_id_tbl,
> + .probe = hisi_ptt_probe,
> + .remove = hisi_ptt_remove,
> +};
> +module_pci_driver(hisi_ptt_driver);
> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> diff --git a/drivers/hwtracing/ptt/hisi_ptt.h b/drivers/hwtracing/ptt/hisi_ptt.h
> new file mode 100644
> index 000000000000..8aa5d6b78cc5
> --- /dev/null
> +++ b/drivers/hwtracing/ptt/hisi_ptt.h
> @@ -0,0 +1,159 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
> + * Author: Yicong Yang <yangyicong@hisilicon.com>
I saw "spinoff the declarations to a separate header" in the revision
notes. I don't understand the reasoning for that - if the declarations
are only used in the .c file then they can be declared there. They are
not an interface nor shared declarations.
Anyway it's the maintainers choice.
> + */
> +
> +#ifndef _HISI_PTT_H
> +#define _HISI_PTT_H
> +
> +#include <linux/bits.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/types.h>
> +
> +/*
> + * The definition of the device registers and register fields.
> + */
> +#define HISI_PTT_TRACE_ADDR_SIZE 0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0 0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0 0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE 0x8
> +#define HISI_PTT_TRACE_CTRL 0x0850
> +#define HISI_PTT_TRACE_CTRL_EN BIT(0)
> +#define HISI_PTT_TRACE_CTRL_RST BIT(1)
> +#define HISI_PTT_TRACE_CTRL_RXTX_SEL GENMASK(3, 2)
> +#define HISI_PTT_TRACE_CTRL_TYPE_SEL GENMASK(7, 4)
> +#define HISI_PTT_TRACE_CTRL_DATA_FORMAT BIT(14)
> +#define HISI_PTT_TRACE_CTRL_FILTER_MODE BIT(15)
> +#define HISI_PTT_TRACE_CTRL_TARGET_SEL GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT 0x0890
> +#define HISI_PTT_TRACE_INT_STAT_MASK GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK 0x0894
> +#define HISI_PTT_TRACE_WR_STS 0x08a0
> +#define HISI_PTT_TRACE_WR_STS_WRITE GENMASK(27, 0)
> +#define HISI_PTT_TRACE_WR_STS_BUFFER GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS 0x08b0
> +#define HISI_PTT_TRACE_IDLE BIT(0)
> +#define HISI_PTT_DEVICE_RANGE 0x0fe0
> +#define HISI_PTT_LOCATION 0x0fe8
> +#define HISI_PTT_CORE_ID GENMASK(15, 0)
> +#define HISI_PTT_SICL_ID GENMASK(31, 16)
> +
> +/* Parameters of PTT trace DMA part. */
> +#define HISI_PTT_TRACE_DMA_IRQ 0
> +#define HISI_PTT_TRACE_BUF_CNT 4
> +#define HISI_PTT_TRACE_BUF_SIZE SZ_4M
> +#define HISI_PTT_TRACE_TOTAL_BUF_SIZE (HISI_PTT_TRACE_BUF_SIZE * \
> + HISI_PTT_TRACE_BUF_CNT)
> +/* Wait time for DMA hardware to reset */
> +#define HISI_PTT_RESET_WAIT_MS 1000UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TIMEOUT_US 1000000UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US 100UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn) (PCI_FUNC(devfn) << 1)
> +
> +enum hisi_ptt_trace_status {
> + HISI_PTT_TRACE_STATUS_OFF = 0,
> + HISI_PTT_TRACE_STATUS_ON,
> +};
> +
> +/**
> + * struct hisi_ptt_dma_buffer - describe a single trace buffer of PTT trace.
> + * The detail of the data format is described
> + * in the documentation of PTT device.
> + * @list: entry of this buffer in the buffer list
> + * @size: size of this buffer
> + * @dma: DMA address of this buffer visible to the device
> + * @addr: virtual address of this buffer visible to the cpu
> + * @index: index of this buffer in the buffer list
> + */
> +struct hisi_ptt_dma_buffer {
> + struct list_head list;
> + unsigned int size;
> + dma_addr_t dma;
> + void *addr;
> + int index;
> +};
> +
> +/**
> + * struct hisi_ptt_trace_ctrl - control and status of PTT trace
> + * @status: current trace status
> + * @trace_buf: the trace buffers for holding the trace data
> + * @default_cpu: default cpu to start the trace session
> + * @buffer_size: the size of each trace buffer in the list
> + * @buf_index: the index of current using trace buffer
> + * @is_port: whether we're tracing root port or not
> + * @direction: direction of the TLP headers to trace
> + * @filter: filter value for tracing the TLP headers
> + * @format: format of the TLP headers to trace
> + * @type: type of the TLP headers to trace
> + */
> +struct hisi_ptt_trace_ctrl {
> + enum hisi_ptt_trace_status status;
> + struct list_head trace_buf;
> + int default_cpu;
> + u32 buffer_size;
> + u32 buf_index;
> + bool is_port;
> + u32 direction:2;
> + u32 filter:16;
> + u32 format:1;
> + u32 type:4;
> +};
> +
> +/**
> + * struct hisi_ptt_filter_desc - descriptor of the PTT trace filter
> + * @list: entry of this descriptor in the filter list
> + * @pdev: pci_dev related to this filter
> + * @val: filter value for the device
> + */
> +struct hisi_ptt_filter_desc {
> + struct list_head list;
> + struct pci_dev *pdev;
> + u16 val;
> +};
> +
> +/**
> + * struct hisi_ptt - per PTT device data
> + * @trace_ctrl: the control information of PTT trace
> + * @iobase: base IO address of the device
> + * @pdev: pci_dev of this PTT device
> + * @mutex: mutex to protect the filter list and serialize the perf process.
> + * @name: name of the PTT device
> + * @core_id: PCIe core ID this PTT device locates
please don't put stuff in the common control struct which can be worked
out on the fly. This is set in one function and then read in a callee.
> + * @sicl_id: SICL ID this PTT device locates
> + * @upper: the upper BDF range of the PCI devices managed by this PTT device
> + * @lower: the lower BDF range of the PCI devices managed by this PTT device
> + * @port_filters: the filter list of root ports
> + * @req_filters: the filter list of requester ID
> + * @port_mask: port mask of the managed root ports
> + */
> +struct hisi_ptt {
> + struct hisi_ptt_trace_ctrl trace_ctrl;
> + void __iomem *iobase;
> + struct pci_dev *pdev;
> + struct mutex mutex;
> + const char *name;
> + u16 core_id;
> + u16 sicl_id;
> + u32 upper;
> + u32 lower;
> +
> + /*
> + * The trace TLP headers can either be filtered by certain
> + * root port, or by the requester ID. Organize the filters
> + * by @port_filters and @req_filters here. The mask of all
> + * the valid ports is also cached for doing sanity check
> + * of user input.
> + */
> + struct list_head port_filters;
> + struct list_head req_filters;
> + u16 port_mask;
> +};
> +
> +#endif /* _HISI_PTT_H */
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 4/8] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device
2022-02-07 11:49 ` Jonathan Cameron
@ 2022-02-08 7:08 ` Yicong Yang
0 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-08 7:08 UTC (permalink / raw)
To: Jonathan Cameron, Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On 2022/2/7 19:49, Jonathan Cameron wrote:
> On Mon, 24 Jan 2022 21:11:14 +0800
> Yicong Yang <yangyicong@hisilicon.com> wrote:
>
>> Add tune function for the HiSilicon Tune and Trace device. The interface
>> of tune is exposed through sysfs attributes of PTT PMU device.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>
> A few trivial things inline, but looks good in general to me.
> With those tidied up
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>
Thanks for the comments.
>
>> ---
>> drivers/hwtracing/ptt/hisi_ptt.c | 154 +++++++++++++++++++++++++++++++
>> drivers/hwtracing/ptt/hisi_ptt.h | 19 ++++
>> 2 files changed, 173 insertions(+)
>>
>> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
>> index 2994354e690b..b11e702eb506 100644
>> --- a/drivers/hwtracing/ptt/hisi_ptt.c
>> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
>> @@ -21,6 +21,159 @@
>>
>> #include "hisi_ptt.h"
>>
>> +static int hisi_ptt_wait_tuning_finish(struct hisi_ptt *hisi_ptt)
>> +{
>> + u32 val;
>> +
>> + return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TUNING_INT_STAT,
>> + val, !(val & HISI_PTT_TUNING_INT_STAT_MASK),
>> + HISI_PTT_WAIT_POLL_INTERVAL_US,
>> + HISI_PTT_WAIT_TIMEOUT_US);
>> +}
>> +
>> +static int hisi_ptt_tune_data_get(struct hisi_ptt *hisi_ptt,
>> + u32 event, u16 *data)
>> +{
>> + u32 reg;
>> +
>> + reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
>> + reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
>> + reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
>> + event);
>> + writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
>> +
>> + /* Write all 1 to indicates it's the read process */
>> + writel(~0UL, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
>
> Just to check, this is includes the bits above the DATA_VAL_MASK?
> Fine if so, just seems odd to define a field but then write
> parts of the register that aren't part of that field.
>
yes. The valid data field is [0,15]. But
all 1 is used here to indicate that it's a
read process rather than a write process.
>> +
>> + if (hisi_ptt_wait_tuning_finish(hisi_ptt))
>> + return -ETIMEDOUT;
>> +
>> + reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
>> + reg &= HISI_PTT_TUNING_DATA_VAL_MASK;
>> + *data = (u16)reg;
>
> As below, prefer a FIELD_GET() for this.
>
sure. will use field ops here and below.
Thanks.
>> +
>> + return 0;
>> +}
>> +
>> +static int hisi_ptt_tune_data_set(struct hisi_ptt *hisi_ptt,
>> + u32 event, u16 data)
>> +{
>> + u32 reg;
>> +
>> + reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
>> + reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
>> + reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
>> + event);
>> + writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
>> +
>> + reg = data;
> Given you defined HISI_PTT_TUNING_DATA_VAL_MASK why not use it here
>
> writel(FIELD_PREP(..), ...)?
>
>> + writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
>> +
>> + if (hisi_ptt_wait_tuning_finish(hisi_ptt))
>> + return -ETIMEDOUT;
>> +
>> + return 0;
>> +}
>> +
>
>
> .
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 2/8] hisi_ptt: Register PMU device for PTT trace
2022-02-07 11:42 ` Jonathan Cameron
@ 2022-02-08 7:41 ` Yicong Yang
0 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-08 7:41 UTC (permalink / raw)
To: Jonathan Cameron, Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On 2022/2/7 19:42, Jonathan Cameron wrote:
> On Mon, 24 Jan 2022 21:11:12 +0800
> Yicong Yang <yangyicong@hisilicon.com> wrote:
>
>> Register PMU device of PTT trace, then users can use
>> trace through perf command. The driver makes use of perf
>> AUX trace and support following events to configure the
>> trace:
>>
>> - filter: select Root port or Endpoint to trace
>> - type: select the type of traced TLP headers
>> - direction: select the direction of traced TLP headers
>> - format: select the data format of the traced TLP headers
>>
>> This patch adds the PMU driver part of PTT trace. The perf
>> command support of PTT trace is added in the following
>> patch.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>
>
>> @@ -294,6 +346,405 @@ static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>> hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>> }
>>
>> +#define HISI_PTT_PMU_FILTER_IS_PORT BIT(19)
>> +#define HISI_PTT_PMU_FILTER_VAL_MASK GENMASK(15, 0)
>> +#define HISI_PTT_PMU_DIRECTION_MASK GENMASK(23, 20)
>> +#define HISI_PTT_PMU_TYPE_MASK GENMASK(31, 24)
>> +#define HISI_PTT_PMU_FORMAT_MASK GENMASK(35, 32)
>> +
>> +static ssize_t available_filters_show(struct device *dev,
>> + struct device_attribute *attr,
>> + char *buf)
>> +{
>> + struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>> + struct hisi_ptt_filter_desc *filter;
>> + int pos = 0;
>> +
>> + if (list_empty(&hisi_ptt->port_filters))
>> + return sysfs_emit(buf, "#### No available filter ####\n");
>> +
>
> This is a very unusual sysfs attribute.
> They are supposed to be one "thing" per file, so I'd have expected this to
> be at least two files
>
> root_ports_available_filters
> request_available_filters
> and no available filter is indicated by these attribute returning an empty
> string.
>
Split it makes sense to me as indeed we're maintaining two list for root ports
and requester each. but perhaps below names are better?
available_root_port_filters
available_requester_filters
And feels we don't need the titles if we split it into two files, so it'll be like
$ cat available_root_port_filters
0000:00:10.0 0x80001
0000:00:11.0 0x80004
$ cat available_requester_filters
0000:01:00.0 0x00100
0000:01:00.1 0x00101
It's also better for script I think.
> However you need to match convention for hwtracing drivers so if
> this is common approach perhaps you could point me to a similar
> example? My grep skills didn't find me one.
>
>> + mutex_lock(&hisi_ptt->mutex);
>> + pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>> + list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>> + pos += sysfs_emit_at(buf, pos, "%s 0x%05lx\n",
>> + pci_name(filter->pdev),
>> + hisi_ptt_get_filter_val(filter->pdev) |
>> + HISI_PTT_PMU_FILTER_IS_PORT);
>> +
>> + pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>> + list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> + pos += sysfs_emit_at(buf, pos, "%s 0x%05x\n",
>> + pci_name(filter->pdev),
>> + hisi_ptt_get_filter_val(filter->pdev));
>> +
>> + mutex_unlock(&hisi_ptt->mutex);
>> + return pos;
>> +}
>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>> +
>
> ...
>
>
>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>> +{
>> + int i, ret = -EINVAL;
>> +
>> + for (i = 0; i < list_size; i++)
>> + if (val == available_list[i]) {
>> + ret = 0;
>
> return 0;
>
ok.
>> + break;
>> + }
>> +
>> + return ret;
>
> return -EINVAL;
ok.
>
>> +}
>> +
>
>> +
>> +static void hisi_ptt_pmu_free_aux(void *aux)
>> +{
>> + struct hisi_ptt_pmu_buf *buf = aux;
>> +
>> + vunmap(buf->base);
>> + kfree(buf);
>> +}
>> +
>
>
> ...
>
>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>> +{
>> + struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> + struct hw_perf_event *hwc = &event->hw;
>> + int cpu = event->cpu;
>> +
>> + if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>
> This check is not entirely obvious to me. Perhaps a comment would help
> readers understand why this condition is successful, but doesn't involve
> actually starting the pmu?
>
Not sure I describe it correct and accurate. A perf session will add and start event on
each cpu, or only a range of cpus if user specified it by -C parameter of perf.
This information is passed to the PMU driver by the event->cpu, -1 indicates that user
didn't specify the cpu. This function will be called on every cpu or cpus specified by
the user.
Since we're not tracing CPUs and we don't need every cpu to start the trace, we add the
check here to only allow the event on the default cpu to start the trace. Other cpus
will just return. The default cpu is the 1st cpu of the NUMA nodes the PTT device locates.
>> + return 0;
>> +
>> + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>> +
>> + if (flags & PERF_EF_START) {
>> + hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>> + if (hwc->state & PERF_HES_STOPPED)
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>
> ...
>
>> /*
>> * The DMA of PTT trace can only use direct mapping, due to some
>> * hardware restriction. Check whether there is an iommu or the
>> @@ -359,6 +810,12 @@ static int hisi_ptt_probe(struct pci_dev *pdev,
>>
>> hisi_ptt_init_ctrls(hisi_ptt);
>>
>> + ret = hisi_ptt_register_pmu(hisi_ptt);
>> + if (ret) {
>> + pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>
> Given I think this exposes userspace interfaces, it should be the very
> last thing done in probe(). Otherwise we have a race condition (at least in
> theory) where someone starts using it before we then fail the iommu mapping check.
>
thanks for catching this. I think it'll be a problem and I'll get the iommu mapping check
in advance.
Thanks.
>
>> + return ret;
>> + }
>> +
>> ret = hisi_ptt_check_iommu_mapping(hisi_ptt);
>> if (ret) {
>> pci_err(pdev, "cannot work with non-direct DMA mapping.\n");
>
> Thanks,
>
> Jonathan
>
> .
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-01-24 13:11 ` [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
@ 2022-02-08 8:05 ` John Garry
2022-02-08 11:21 ` Yicong Yang
2022-02-14 12:55 ` Yicong Yang
1 sibling, 1 reply; 32+ messages in thread
From: John Garry @ 2022-02-08 8:05 UTC (permalink / raw)
To: Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, Jonathan Cameron,
daniel.thompson, joro, Shameerali Kolothum Thodi, robin.murphy,
peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
linux-pci, linux-perf-users, iommu
Cc: Zengtao (B), liuqi (BA),
Zhangshaokun, Linuxarm, yangyicong, Song Bao Hua (Barry Song)
On 24/01/2022 13:11, Yicong Yang wrote:
> The DMA of HiSilicon PTT device can only work with identical
> mapping. So add a quirk for the device to force the domain
> passthrough.
This patch should be earlier in the series, before the PTT driver, and
the comment on hisi_ptt_check_iommu_mapping() should mention what is
going on here.
>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
> 1 file changed, 16 insertions(+)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 6dc6d8b6b368..6f67a2b1dd27 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
> }
> }
>
> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
> + (pdev)->device == 0xa12e)
I assume that not all revisions will require this check, right?
> +
> +static int arm_smmu_def_domain_type(struct device *dev)
> +{
> + if (dev_is_pci(dev)) {
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (IS_HISI_PTT_DEVICE(pdev))
> + return IOMMU_DOMAIN_IDENTITY;
> + }
> +
> + return 0;
> +}
> +
> static struct iommu_ops arm_smmu_ops = {
> .capable = arm_smmu_capable,
> .domain_alloc = arm_smmu_domain_alloc,
> @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
> .sva_unbind = arm_smmu_sva_unbind,
> .sva_get_pasid = arm_smmu_sva_get_pasid,
> .page_response = arm_smmu_page_response,
> + .def_domain_type = arm_smmu_def_domain_type,
> .pgsize_bitmap = -1UL, /* Restricted during device attach */
> .owner = THIS_MODULE,
> };
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 1/8] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
2022-02-07 18:11 ` John Garry
@ 2022-02-08 8:57 ` Yicong Yang
0 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-08 8:57 UTC (permalink / raw)
To: John Garry, Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
daniel.thompson, joro, shameerali.kolothum.thodi, robin.murphy,
peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
linux-pci, linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, song.bao.hua
Hi John,
Thanks for the comments. some replies inline.
On 2022/2/8 2:11, John Garry wrote:
> On 24/01/2022 13:11, Yicong Yang wrote:
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>> integrated Endpoint(RCiEP) device, providing the capability
>> to dynamically monitor and tune the PCIe traffic, and trace
>> the TLP headers.
>>
>> Add the driver for the device to enable the trace function.
>> This patch adds basic function of trace, including the device's
>> probe and initialization, functions for trace buffer allocation
>> and trace enable/disable, register an interrupt handler to
>> simply response to the DMA events. The user interface of trace
>> will be added in the following patch.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>> drivers/Makefile | 1 +
>> drivers/hwtracing/Kconfig | 2 +
>> drivers/hwtracing/ptt/Kconfig | 11 +
>> drivers/hwtracing/ptt/Makefile | 2 +
>> drivers/hwtracing/ptt/hisi_ptt.c | 398 +++++++++++++++++++++++++++++++
>> drivers/hwtracing/ptt/hisi_ptt.h | 159 ++++++++++++
>> 6 files changed, 573 insertions(+)
>> create mode 100644 drivers/hwtracing/ptt/Kconfig
>> create mode 100644 drivers/hwtracing/ptt/Makefile
>> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>>
>> diff --git a/drivers/Makefile b/drivers/Makefile
>> index a110338c860c..ab3411e4eba5 100644
>> --- a/drivers/Makefile
>> +++ b/drivers/Makefile
>> @@ -175,6 +175,7 @@ obj-$(CONFIG_USB4) += thunderbolt/
>> obj-$(CONFIG_CORESIGHT) += hwtracing/coresight/
>> obj-y += hwtracing/intel_th/
>> obj-$(CONFIG_STM) += hwtracing/stm/
>> +obj-$(CONFIG_HISI_PTT) += hwtracing/ptt/
>> obj-$(CONFIG_ANDROID) += android/
>> obj-$(CONFIG_NVMEM) += nvmem/
>> obj-$(CONFIG_FPGA) += fpga/
>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>> index 13085835a636..911ee977103c 100644
>> --- a/drivers/hwtracing/Kconfig
>> +++ b/drivers/hwtracing/Kconfig
>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>> source "drivers/hwtracing/intel_th/Kconfig"
>> +source "drivers/hwtracing/ptt/Kconfig"
>> +
>> endmenu
>> diff --git a/drivers/hwtracing/ptt/Kconfig b/drivers/hwtracing/ptt/Kconfig
>> new file mode 100644
>> index 000000000000..4f4f2459ac47
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/Kconfig
>> @@ -0,0 +1,11 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config HISI_PTT
>> + tristate "HiSilicon PCIe Tune and Trace Device"
>> + depends on ARM64 && PCI && HAS_DMA && HAS_IOMEM
>> + help
>> + HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
>
> exists
>
>> + device, provides support for PCIe traffic tuning and
>
> and it provides support...
>
will fix, thanks.
>> + tracing TLP headers to the memory.
>> +
>> + This driver can also be built as a module. If so, the module
>> + will be called hisi_ptt.
>> diff --git a/drivers/hwtracing/ptt/Makefile b/drivers/hwtracing/ptt/Makefile
>> new file mode 100644
>> index 000000000000..908c09a98161
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>> diff --git a/drivers/hwtracing/ptt/hisi_ptt.c b/drivers/hwtracing/ptt/hisi_ptt.c
>> new file mode 100644
>> index 000000000000..6d0a0ca5c0a9
>> --- /dev/null
>> +++ b/drivers/hwtracing/ptt/hisi_ptt.c
>> @@ -0,0 +1,398 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for HiSilicon PCIe tune and trace device
>> + *
>> + * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
>> + * Author: Yicong Yang <yangyicong@hisilicon.com>
>> + */
>> +
[...]
>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> + struct device *dev = &hisi_ptt->pdev->dev;
>> + struct hisi_ptt_dma_buffer *buffer;
>> + int i, ret;
>> +
>> + hisi_ptt->trace_ctrl.buf_index = 0;
>> +
>> + /* Make sure the trace buffer is empty before allocating */
>> + if (!list_empty(&ctrl->trace_buf)) {
>> + list_for_each_entry(buffer, &ctrl->trace_buf, list)
>> + memset(buffer->addr, 0, buffer->size);
>> + return 0;
>> + }
>> +
>> + for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>> + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
>
> I may have asked this before: why no devm usage?
>
I remembered I was suggested for not using devm where we may need to manually
free it as it intends to be freed automically after the driver detachment.
>> + if (!buffer) {
>> + ret = -ENOMEM;
>> + goto err;
>> + }
>> +
>> + buffer->addr = dma_alloc_coherent(dev, ctrl->buffer_size,
>> + &buffer->dma, GFP_KERNEL);
>> + if (!buffer->addr) {
>> + kfree(buffer);
>> + ret = -ENOMEM;
>> + goto err;
>> + }
>> +
>> + memset(buffer->addr, 0, buffer->size);
>> +
>> + buffer->index = i;
>> + buffer->size = ctrl->buffer_size;
>
> please double check if we really need to store this info separately, i.e. is it const and same for all?
>
yes. I stored it for convenience but seems unnecessary now, I'll remove it.
>> + list_add_tail(&buffer->list, &ctrl->trace_buf);
>> + }
>> +
>> + return 0;
>> +err:
>> + hisi_ptt_free_trace_buf(hisi_ptt);
>> + return ret;
>> +}
>> +
>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>> +{
>> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> + hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>> +}
>> +
>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>> +{
>> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> + struct hisi_ptt_dma_buffer *cur;
>> + u32 val;
>> +
>> + /* Check device idle before start trace */
>> + if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>> + pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>> + return -EBUSY;
>> + }
>> +
>> + /* Reset the DMA before start tracing */
>> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> + val |= HISI_PTT_TRACE_CTRL_RST;
>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> + /*
>> + * We'll be in the perf context where preemption is disabled,
>> + * so use busy loop here.
>
> what has preemption is disabled got to do with "busy loop"?
>
The comment here to notice why we don't use a msleep() or similiar here as
we're in atomic context. Before we change to use perf, it's msleep() here.
>> + */
>> + mdelay(HISI_PTT_RESET_WAIT_MS);
>> +
>> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> + val &= ~HISI_PTT_TRACE_CTRL_RST;
>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> + /* Clear the interrupt status */
>> + writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>> +
>> + /* Configure the trace DMA buffer */
>> + list_for_each_entry(cur, &ctrl->trace_buf, list) {
>> + writel(lower_32_bits(cur->dma),
>> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> + writel(upper_32_bits(cur->dma),
>> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> + }
>> + writel(ctrl->buffer_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>> +
>> + /* Set the trace control register */
>> + val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>> + if (!hisi_ptt->trace_ctrl.is_port)
>> + val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>> +
>> + /* Start the Trace */
>> + val |= HISI_PTT_TRACE_CTRL_EN;
>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> + ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>> +
>> + return 0;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>> +{
>> + struct hisi_ptt *hisi_ptt = context;
>> + u32 status;
>> +
>> + status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +
>> + /* Clear the interrupt status of buffer @buf_idx */
>> + writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +
>> + return IRQ_HANDLED;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>> +{
>> + struct hisi_ptt *hisi_ptt = context;
>> + u32 status;
>> +
>> + status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> + if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>> + return IRQ_NONE;
>> +
>> + return IRQ_WAKE_THREAD;
>
> Adding empty handler like this is not helpful. And from checking the later code, the threaded part does nothing special, i.e nothing time consuming, so I don't know why everything cannot be done in the hard part for simplicity
>
In the following patch we're copying and committing data to the AUX buffer so we need a threaded part here. For this
patch just adding the stub here. Maybe I can add some comments to mention it ?
>> +}
>> +
>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>> +{
>> + pci_free_irq_vectors(pdev);
>> +}
>> +
>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>> +{
>> + struct pci_dev *pdev = hisi_ptt->pdev;
>> + int ret;
>> +
>> + ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>> + if (ret < 0) {
>> + pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>> + if (ret < 0)
>> + return ret;
>> +
>> + ret = devm_request_threaded_irq(&pdev->dev,
>> + pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>> + hisi_ptt_irq, hisi_ptt_isr, 0,
>> + "hisi-ptt", hisi_ptt);
>> + if (ret) {
>> + pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>> + pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>> +{
>> + struct hisi_ptt_filter_desc *filter;
>> + struct hisi_ptt *hisi_ptt = data;
>> + struct list_head *target_list;
>> +
>> + target_list = pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT ?
>> + &hisi_ptt->port_filters : &hisi_ptt->req_filters;
>> +
>> + filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>> + if (!filter)
>> + return -ENOMEM;
>> +
>> + filter->pdev = pdev;
>> + filter->val = hisi_ptt_get_filter_val(pdev);
>
> why do you need to store this also? if you're storing pdev, you seem to be able to directly get hisi_ptt_get_filter_val() for it
>
checked the used places and I think it canbe dropped. thanks.
>> + list_add_tail(&filter->list, target_list);
>> +
>> + /* Update the available port mask */
>> + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>> + hisi_ptt->port_mask |= filter->val;
>> +
>> + return 0;
>> +}
>> +
[...]
>> +/*
>> + * The DMA of PTT trace can only use direct mapping, due to some
>> + * hardware restriction. Check whether there is an iommu or the
>> + * policy of the iommu domain is passthrough, otherwise the trace
>> + * cannot work.
>> + */
>> +static int hisi_ptt_check_iommu_mapping(struct hisi_ptt *hisi_ptt)
>> +{
>> + struct pci_dev *pdev = hisi_ptt->pdev;
>> + struct iommu_domain *iommu_domain;
>> +
>> + iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> + if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> + return 0;
>
> so what stops us changing the domain type later?
>
sorry but I don't think I got the point.
>> +
>> + return -EOPNOTSUPP;
>> +}
>> +
>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>> + const struct pci_device_id *id)
>> +{
>> + struct hisi_ptt *hisi_ptt;
>> + int ret;
>> +
>> + hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>> + if (!hisi_ptt)
>> + return -ENOMEM;
>> +
>> + mutex_init(&hisi_ptt->mutex);
>> + hisi_ptt->pdev = pdev;
>> +
>> + /*
>> + * Lifetime of pci_dev is longer than hisi_ptt,
>> + * so directly reference to the pci name string.
>> + */
>> + hisi_ptt->name = pci_name(hisi_ptt->pdev);
>> + pci_set_drvdata(pdev, hisi_ptt);
>> +
>> + ret = pcim_enable_device(pdev);
>> + if (ret) {
>> + pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>> + return ret;
>> + }
>> +
>> + ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>> + if (ret) {
>> + pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>> + return ret;
>> + }
>> +
>> + hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>> +
>> + ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>> + if (ret) {
>> + pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>> + return ret;
>> + }
>> + pci_set_master(pdev);
>> +
>> + ret = hisi_ptt_register_irq(hisi_ptt);
>> + if (ret)
>> + return ret;
>> +
>> + hisi_ptt_init_ctrls(hisi_ptt);
>> +
>> + ret = hisi_ptt_check_iommu_mapping(hisi_ptt);
>
> surely this should be done earlier in the probe
>
yes it's a good point. will make it earlier.
>> + if (ret) {
>> + pci_err(pdev, "cannot work with non-direct DMA mapping.\n");
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
[...]
>> +/**
>> + * struct hisi_ptt - per PTT device data
>> + * @trace_ctrl: the control information of PTT trace
>> + * @iobase: base IO address of the device
>> + * @pdev: pci_dev of this PTT device
>> + * @mutex: mutex to protect the filter list and serialize the perf process.
>> + * @name: name of the PTT device
>> + * @core_id: PCIe core ID this PTT device locates
>
> please don't put stuff in the common control struct which can be worked out on the fly. This is set in one function and then read in a callee.
>
ok. will remove this.
Thanks,
Yicong
>> + * @sicl_id: SICL ID this PTT device locates
>> + * @upper: the upper BDF range of the PCI devices managed by this PTT device
>> + * @lower: the lower BDF range of the PCI devices managed by this PTT device
>> + * @port_filters: the filter list of root ports
>> + * @req_filters: the filter list of requester ID
>> + * @port_mask: port mask of the managed root ports
>> + */
>> +struct hisi_ptt {
>> + struct hisi_ptt_trace_ctrl trace_ctrl;
>> + void __iomem *iobase;
>> + struct pci_dev *pdev;
>> + struct mutex mutex;
>> + const char *name;
>> + u16 core_id;
>> + u16 sicl_id;
>> + u32 upper;
>> + u32 lower;
>> +
>> + /*
>> + * The trace TLP headers can either be filtered by certain
>> + * root port, or by the requester ID. Organize the filters
>> + * by @port_filters and @req_filters here. The mask of all
>> + * the valid ports is also cached for doing sanity check
>> + * of user input.
>> + */
>> + struct list_head port_filters;
>> + struct list_head req_filters;
>> + u16 port_mask;
>> +};
>> +
>> +#endif /* _HISI_PTT_H */
>
> .
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 1/8] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
2022-02-07 11:42 ` Jonathan Cameron
@ 2022-02-08 11:07 ` Yicong Yang
2022-02-14 12:51 ` Yicong Yang
0 siblings, 1 reply; 32+ messages in thread
From: Yicong Yang @ 2022-02-08 11:07 UTC (permalink / raw)
To: Jonathan Cameron, Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On 2022/2/7 19:42, Jonathan Cameron wrote:
> On Mon, 24 Jan 2022 21:11:11 +0800
> Yicong Yang <yangyicong@hisilicon.com> wrote:
>
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>> integrated Endpoint(RCiEP) device, providing the capability
>> to dynamically monitor and tune the PCIe traffic, and trace
>> the TLP headers.
>>
>> Add the driver for the device to enable the trace function.
>> This patch adds basic function of trace, including the device's
>> probe and initialization, functions for trace buffer allocation
>> and trace enable/disable, register an interrupt handler to
>> simply response to the DMA events. The user interface of trace
>> will be added in the following patch.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> Hi Yicong,
>
> I've not been following all the earlier discussion on this driver closely
> so I may well raise something that has already been addressed. If so
> just ignore the comment.
Thanks for the comments. It's ok for me to clarify it :).
Part replies inline and I need to do some test on the others.
>
> Thanks,
>
> Jonathan
>
>> ---
>> drivers/Makefile | 1 +
>> drivers/hwtracing/Kconfig | 2 +
>> drivers/hwtracing/ptt/Kconfig | 11 +
>> drivers/hwtracing/ptt/Makefile | 2 +
>> drivers/hwtracing/ptt/hisi_ptt.c | 398 +++++++++++++++++++++++++++++++
>> drivers/hwtracing/ptt/hisi_ptt.h | 159 ++++++++++++
>> 6 files changed, 573 insertions(+)
>> create mode 100644 drivers/hwtracing/ptt/Kconfig
>> create mode 100644 drivers/hwtracing/ptt/Makefile
>> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>>
[...]
>> +
>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> + struct device *dev = &hisi_ptt->pdev->dev;
>> + struct hisi_ptt_dma_buffer *buffer;
>> + int i, ret;
>> +
>> + hisi_ptt->trace_ctrl.buf_index = 0;
>> +
>> + /* Make sure the trace buffer is empty before allocating */
>
> This comment is misleading as it suggests it not being empty is
> a bad thing but the code handles it as an acceptable path.
> Perhaps:
> /*
> * If the trace buffer has already been allocated, zero the
> * memory.
> */
>
will make it less misleading. thanks.
>> + if (!list_empty(&ctrl->trace_buf)) {
>> + list_for_each_entry(buffer, &ctrl->trace_buf, list)
>> + memset(buffer->addr, 0, buffer->size);
>> + return 0;
>> + }
>> +
>> + for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>> + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
>> + if (!buffer) {
>> + ret = -ENOMEM;
>> + goto err;
>> + }
>> +
>> + buffer->addr = dma_alloc_coherent(dev, ctrl->buffer_size,
>> + &buffer->dma, GFP_KERNEL);
>> + if (!buffer->addr) {
>> + kfree(buffer);
>> + ret = -ENOMEM;
>> + goto err;
>> + }
>> +
>> + memset(buffer->addr, 0, buffer->size);
> See:
> https://lore.kernel.org/lkml/20190108130701.14161-4-hch@lst.de/
> dma_alloc_coherent() always zeros the memory for us hence there
> is no longer a dma_kzalloc_coherent()
>
thanks for the information. Then the memset here is redundant and will drop it.
>> +
>> + buffer->index = i;
>
> Carrying an index inside a list which corresponds directly
> to the position in the list is not particularly nice.
> Why can't we compute this index on the fly where the list
> is walked? Or am I misunderstanding and the order of the buffers
> is changed in a later patch?
>
The index is fixed once allocated and I stored it to avoid later
computing. But seems it's highly recommended to compute these sort
of things on the fly when necessary. John recommends the same things
on some other places so I think I can get these addressed.
> As a side note, is a list actually appropriate when we always
> have 4 of these buffers? Feels like an array of buffer
> structures might be cheaper.
>
>> + buffer->size = ctrl->buffer_size;
>> + list_add_tail(&buffer->list, &ctrl->trace_buf);
>> + }
>> +
>> + return 0;
>> +err:
>> + hisi_ptt_free_trace_buf(hisi_ptt);
>> + return ret;
>> +}
>> +
>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>> +{
>> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> + hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>> +}
>> +
>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>> +{
>> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> + struct hisi_ptt_dma_buffer *cur;
>> + u32 val;
>> +
>> + /* Check device idle before start trace */
>> + if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>> + pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>> + return -EBUSY;
>> + }
>> +
>> + /* Reset the DMA before start tracing */
>> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> + val |= HISI_PTT_TRACE_CTRL_RST;
>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> + /*
>> + * We'll be in the perf context where preemption is disabled,
>> + * so use busy loop here.
>> + */
>> + mdelay(HISI_PTT_RESET_WAIT_MS);
>
> Busy look for 1 second? Ouch. If we can reduce this in any way
> that would be great or if there is a means to do it before
> we disable preemption.
>
It's inherited from the previous version that was using msleep() and it's
somehow unacceptable in an atomic context I think. The reset here is
going to reset the write pointer of the hardware DMA so we can check the
whether the pointer before dereset it. I confirmed with our hardware
teams that it can be reduced to 10us. So I'll poll the write pointer register
for about 10us before continue here.
thanks for catching this!
>> +
>> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> + val &= ~HISI_PTT_TRACE_CTRL_RST;
>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> + /* Clear the interrupt status */
>> + writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>> +
>> + /* Configure the trace DMA buffer */
>> + list_for_each_entry(cur, &ctrl->trace_buf, list) {
>
> I comment on the use of cur->index above. Here it would be easy to compute
> the index as we go for example assuming we never end up with holes
> in the list.
>
ok.
>> + writel(lower_32_bits(cur->dma),
>> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> + writel(upper_32_bits(cur->dma),
>> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> + }
>> + writel(ctrl->buffer_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>> +
>> + /* Set the trace control register */
>> + val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>> + if (!hisi_ptt->trace_ctrl.is_port)
>> + val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>> +
>> + /* Start the Trace */
>> + val |= HISI_PTT_TRACE_CTRL_EN;
>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> + ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>> +
>> + return 0;
>> +}
>> +
>
> ...
>
>> +
>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>> +{
>> + struct pci_dev *pdev = hisi_ptt->pdev;
>> + struct pci_bus *bus;
>> + u32 reg;
>> +
>> + INIT_LIST_HEAD(&hisi_ptt->port_filters);
>> + INIT_LIST_HEAD(&hisi_ptt->req_filters);
>> +
>> + /*
>> + * The device range register provides the information about the
>> + * root ports which the RCiEP can control and trace. The RCiEP
>> + * and the root ports it support are on the same PCIe core, with
>> + * same domain number but maybe different bus number. The device
>> + * range register will tell us which root ports we can support,
>> + * Bit[31:16] indicates the upper BDF numbers of the root port,
>> + * while Bit[15:0] indicates the lower.
>> + */
>> + reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>> + hisi_ptt->upper = reg >> 16;
>> + hisi_ptt->lower = reg & 0xffff;
> Trivial:
> Perhaps worthing define HISI_PTT_DEVICE_RANGE_UPPER_MASK etc adn using
> FIELD_GET?
>
sure.
>> +
>> + reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>> + hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>> + hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>> +
>> + bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
>> + if (bus)
>> + pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>> +
>> + /* Initialize trace controls */
>> + INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>> + hisi_ptt->trace_ctrl.buffer_size = HISI_PTT_TRACE_BUF_SIZE;
>> + hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>> +}
>> +
[...]
>> +
>> +#define HISI_PCIE_CORE_PORT_ID(devfn) (PCI_FUNC(devfn) << 1)
>> +
>> +enum hisi_ptt_trace_status {
>> + HISI_PTT_TRACE_STATUS_OFF = 0,
>> + HISI_PTT_TRACE_STATUS_ON,
>> +};
>
> Why not just use a boolean given we only have off and on states?
>
An enum may make the code more readable I think.
Thanks,
Yicong
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 6/8] docs: Add HiSilicon PTT device driver documentation
2022-02-07 12:12 ` Jonathan Cameron
@ 2022-02-08 11:09 ` Yicong Yang
0 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-08 11:09 UTC (permalink / raw)
To: Jonathan Cameron, Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On 2022/2/7 20:12, Jonathan Cameron wrote:
> On Mon, 24 Jan 2022 21:11:16 +0800
> Yicong Yang <yangyicong@hisilicon.com> wrote:
>
>> Document the introduction and usage of HiSilicon PTT device driver.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> Nice document. A few trivial typos inline.
> I would give a RB except I've suggested you change a part of the
> sysfs interface which will affect the relevant documentation.
>
Thanks. I'll get these fixed and update the documentation with sysfs
interface updated.
> Thanks,
>
> Jonathan
>
>> ---
>> Documentation/trace/hisi-ptt.rst | 304 +++++++++++++++++++++++++++++++
>> 1 file changed, 304 insertions(+)
>> create mode 100644 Documentation/trace/hisi-ptt.rst
>>
>> diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
>> new file mode 100644
>> index 000000000000..f3269b11a2f6
>> --- /dev/null
>> +++ b/Documentation/trace/hisi-ptt.rst
>> @@ -0,0 +1,304 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +======================================
>> +HiSilicon PCIe Tune and Trace device
>> +======================================
>> +
>> +Introduction
>> +============
>> +
>> +HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
>> +integrated Endpoint (RCiEP) device, providing the capability
>> +to dynamically monitor and tune the PCIe link's events (tune),
>> +and trace the TLP headers (trace). The two functions are independent,
>> +but is recommended to use them together to analyze and enhance the
>> +PCIe link's performance.
>> +
>> +On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
>> +PCIe cores. Each PCIe core includes several Root Ports and a PTT
>> +RCiEP, like below. The PTT device is capable of tuning and
>> +tracing the link of the PCIe core.
>
> links
>
>> +::
>> + +--------------Core 0-------+
>> + | | [ PTT ] |
>> + | | [Root Port]---[Endpoint]
>> + | | [Root Port]---[Endpoint]
>> + | | [Root Port]---[Endpoint]
>> + Root Complex |------Core 1-------+
>> + | | [ PTT ] |
>> + | | [Root Port]---[ Switch ]---[Endpoint]
>> + | | [Root Port]---[Endpoint] `-[Endpoint]
>> + | | [Root Port]---[Endpoint]
>> + +---------------------------+
>> +
>> +The PTT device driver registers PMU device for each PTT device.
>
> registers one PMU device ..
>
>> +The name of each PTT device is composed of 'hisi_ptt' prefix with
>> +the id of the SICL and the Core where it locates. The Kunpeng 930
>> +SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
>> +IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
>> +Complex for each SICL.
>> +::
>> + /sys/devices/hisi_ptt<sicl_id>_<core_id>
>> +
>> +Tune
>> +====
>> +
>> +PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
>> +Currently we support events in 4 classes. The scope of the events
>> +covers the PCIe core to which the PTT device belongs.
>> +
>> +Each event is presented as a file under $(PTT PMU dir)/tune, and
>> +mostly a simple open/read/write/close cycle will be used to tune
>
> drop "mostly" as it doesn't add anything other than potential confusion.
>
>> +the event.
>> +::
>> + $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
>> + $ ls
>> + qos_tx_cpl qos_tx_np qos_tx_p
>> + tx_path_rx_req_alloc_buf_level
>> + tx_path_tx_req_alloc_buf_level
>> + $ cat qos_tx_dp
>> + 1
>> + $ echo 2 > qos_tx_dp
>> + $ cat qos_tx_dp
>> + 2
>> +
>> +Current value (numerical value) of the event can be simply read
>> +from the file, and the desired value written to the file to tune.
>> +
>> +1. Tx path QoS control
>> +------------------------
>> +
>> +The following files are provided to tune the QoS of the tx path of
>> +the PCIe core.
>> +
>> +- qos_tx_cpl: weight of Tx completion TLPs
>> +- qos_tx_np: weight of Tx non-posted TLPs
>> +- qos_tx_p: weight of Tx posted TLPs
>> +
>> +The weight influences the proportion of certain packets on the PCIe link.
>> +For example, for the storage scenario, increase the proportion
>> +of the completion packets on the link to enhance the performance as
>> +more completions are consumed.
>> +
>> +The available tune data of these events is [0, 1, 2].
>> +Writing a negative value will return an error, and out of range
>> +values will be converted to 2. Note that the event value just
>> +indicates a probable level, but is not precise.
>> +
>> +2. Tx path buffer control
>> +-------------------------
>> +
>> +Following files are provided to tune the buffer of tx path of the PCIe core.
>> +
>> +- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
>> +- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
>> +
>> +These events influence the watermark of the buffer allocated for each
>> +type. Rx means the inbound while Tx means outbound. The packets will
>> +be stored in the buffer first and then posted either when the watermark
>
> Change "posted" to "transmitted" as posted has a special meaning in PCI
> and I don't think that is what you mean here... (I could be wrong!)
>
>> +reached or when timed out. For a busy direction, you should increase
>> +the related buffer watermark to avoid frequently posting and thus
>> +enhance the performance. In most cases just keep the default value.
>> +
>> +The available tune data of above events is [0, 1, 2].
>> +Writing a negative value will return an error, and out of range
>> +values will be converted to 2. Note that the event value just
>> +indicates a probable level, but is not precise.
>> +
>> +Trace
>> +=====
>> +
>> +PTT trace is designed for dumping the TLP headers to the memory, which
>> +can be used to analyze the transactions and usage condition of the PCIe
>> +Link. You can choose to filter the traced headers by either requester ID,
>> +or those downstream of a set of Root Ports on the same core of the PTT
>> +device. It's also supported to trace the headers of certain type and of
>> +certain direction.
>> +
>> +You can use the perf command `perf record` to set the parameters, start
>> +trace and get the data. It's also supported to decode the trace
>> +data with `perf report`. The control parameters for trace is inputted
>> +as event code for each events, which will be further illustracted later.
>
> illustrated
>
>> +An example usage is like
>> +::
>> + $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
>> + format=1/ -- sleep 5
>> +
>> +This will trace the TLP headers downstream root port 0000:00:10.1 (event
>> +code for event 'filter' is 0x80001) with type of posted TLP requests,
>> +direction of inbound and traced data format of 8DW.
>> +
>> +1. filter
>> +---------
>> +
>> +The TLP headers to trace can be filtered by the Root Ports or the requester
>> +ID of the endpoints, which are locates on the same core of the PTT device.
>
> located
>
>> +You can set the filter by spedifying the `filter` parameter which is required
>> +to start the trace. The parameter value is 20 bit. The supported filters and
>> +related values is outputted through `available_filters` sysfs attribute
>> +under related PTT PMU directory, classified as Root Ports and Requesters
>> +respectively.
>> +::
>> + $ cat available_filters
>> + #### Root Ports ####
>> + 0000:00:10.0 0x80001
>> + 0000:00:11.0 0x80004
>> + #### Requesters ####
>> + 0000:01:00.0 0x00100
>> + 0000:01:00.1 0x00101
>> +
>> +Note that multiple Root Ports can be specified at one time, but only
>> +one Endpoint function can be specified in one trace. Specifying both
>> +Root Port and function at the same time is not supported.
>> +
>> +If no filter is available, reading the available_filters will get the hint.
>> +::
>> + $ cat available_filters
>> + #### No available filter ####
>
> If you take not of my earlier feedback this bit may change slightly.
>> +
>> +The available_filters can be dynamically updated, which means you can always
>> +get correct filter information when hotplug events happen, or when you manually
>> +remove/rescan the devices.
>> +
>> +2. type
>> +-------
>> +
>> +You can trace the TLP headers of certain types by specifying the `type`
>> +parameter, which is required to start the trace. The parameter value is
>> +8 bit. Current supported types and related values are shown below:
>> +
>> +8'b00000001: posted requests (P)
>> +8'b00000010: non-posted requests (NP)
>> +8'b00000100: completions (CPL)
>> +
>> +You can specify multiple types when tracing inbound TLP headers, but can only
>> +specify one when tracing outbound TLP headers.
>> +
>> +3. direction
>> +------------
>> +
>> +You can trace the TLP headers from certain direction, which is relative
>> +to the Root Port or the PCIe core, by specifying the `direction` parameter.
>> +This is optional and the default parameter is inbound. The parameter value
>> +is 4 bit. When the desired format is 4DW, directions and related values
>> +supported are shown below:
>> +
>> +4'b0000: inbound TLPs (P, NP, CPL)
>> +4'b0001: outbound TLPs (P, NP, CPL)
>> +4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
>> +4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
>> +
>> +When the desired format is 8DW, directions and related values supported are
>> +shown below:
>> +
>> +4'b0000: reserved
>> +4'b0001: outbound TLPs (P, NP, CPL)
>> +4'b0010: inbound TLPs (P, NP, CPL B)
>> +4'b0011: inbound TLPs (CPL A)
>> +
>> +Inbound completions are classifed into two types:
>
> classified
>
>> +
>> +completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
>> +completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
>> +
>> +4. format
>> +--------------
>> +
>> +You can change the format of the traced TLP headers by specifying the
>> +`format` parameter. This is optional and the default format is 4DW.
>
> As there is a default, there is no need to also say it is optional.
> `format parameter. The default format is 4DW.
>
>> +The parameter value is 4 bit. Current supported formats and related
>> +values are shown below:
>> +
>> +4'b0000: 4DW length per TLP header
>> +4'b0001: 8DW length per TLP header
>> +
>> +The traced TLP header format is different from the PCIe standard.
>> +
>> +When using the 8DW data format, the entire TLP header is logged
>> +(Header DW0-3 shown below). For example, the TLP header for Memory
>> +Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
>> +the header for Configuration Requests is shown in Figure 2.20, etc.
>> +
>> +In addition, 8DW trace buffer entries contain a timestamp and
>> +possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
>> +Otherwise this field will be all 0.
>> +
>> +The bit[31:11] of DW0 is always 0x1fffff, which can be
>> +used to distinguish the data format. 8DW format is like
>> +::
>> + bits [ 31:11 ][ 10:0 ]
>> + |---------------------------------------|-------------------|
>> + DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
>> + DW1 [ Prefix ]
>> + DW2 [ Header DW0 ]
>> + DW3 [ Header DW1 ]
>> + DW4 [ Header DW2 ]
>> + DW5 [ Header DW3 ]
>> + DW6 [ Reserved (0x0) ]
>> + DW7 [ Time ]
>> +
>> +When using the 4DW data format, DW0 of the trace buffer entry
>> +contains selected fields of DW0 of the TLP, together with a
>> +timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
>> +directly from the TLP header.
>> +
>> +4DW format is like
>> +::
>> + bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
>> + |-----|---------|---|---|---|---|-------------|-------------|
>> + DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
>> + DW1 [ Header DW1 ]
>> + DW2 [ Header DW2 ]
>> + DW3 [ Header DW3 ]
>> +
>> +5. memory management
>> +--------------------
>> +
>> +The traced TLP headers will be written to the memory allocated
>> +by the driver. The hardware accepts 4 DMA address with same size,
>> +and writes the buffer sequentially like below. If DMA addr 3 is
>> +finished and the trace is still on, it will return to addr 0.
>> +::
>> + +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
>> + +---------------------------------------------------------+
>> +
>> +Driver will allocate each DMA buffer of 4MiB. The finished buffer
>> +will be copied to the perf AUX buffer allocated by the perf core.
>> +Once the AUX buffer is full while the trace is still on, driver
>> +will commit the AUX buffer first and then apply for a new one with
>> +the same size. The size of AUX buffer is default to 16MiB. User can
>> +adjust the size by specifying the `-m` parameter of the perf command.
>> +
>> +Note that there is a gap between committing the old AUX buffer and
>> +applying a new one, which means the trace is stopped during the
>> +moment and TLPs transferred in the moment cannot be traced. To avoid
>> +this situation, you should begin the trace with large AUX buffer
>> +enough to avoid this gap.
>> +
>> +6. decoding
>> +-----------
>> +
>> +You can decode the traced data with `perf report -D` command (currently
>> +only support to dump the raw trace data). The traced data will be decoded
>> +according to the format described previously (take 8DW as an example):
>> +::
>> + [...perf headers and other information]
>> + . ... HISI PTT data: size 4194304 bytes
>> + . 00000000: 00 00 00 00 Prefix
>> + . 00000004: 01 00 00 60 Header DW0
>> + . 00000008: 0f 1e 00 01 Header DW1
>> + . 0000000c: 04 00 00 00 Header DW2
>> + . 00000010: 40 00 81 02 Header DW3
>> + . 00000014: 33 c0 04 00 Time
>> + . 00000020: 00 00 00 00 Prefix
>> + . 00000024: 01 00 00 60 Header DW0
>> + . 00000028: 0f 1e 00 01 Header DW1
>> + . 0000002c: 04 00 00 00 Header DW2
>> + . 00000030: 40 00 81 02 Header DW3
>> + . 00000034: 02 00 00 00 Time
>> + . 00000040: 00 00 00 00 Prefix
>> + . 00000044: 01 00 00 60 Header DW0
>> + . 00000048: 0f 1e 00 01 Header DW1
>> + . 0000004c: 04 00 00 00 Header DW2
>> + . 00000050: 40 00 81 02 Header DW3
>> + [...]
>
> .
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-08 8:05 ` John Garry
@ 2022-02-08 11:21 ` Yicong Yang
2022-02-08 11:56 ` John Garry
0 siblings, 1 reply; 32+ messages in thread
From: Yicong Yang @ 2022-02-08 11:21 UTC (permalink / raw)
To: John Garry, Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, Jonathan Cameron,
daniel.thompson, joro, Shameerali Kolothum Thodi, robin.murphy,
peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
linux-pci, linux-perf-users, iommu
Cc: Zengtao (B), liuqi (BA),
Zhangshaokun, Linuxarm, Song Bao Hua (Barry Song)
On 2022/2/8 16:05, John Garry wrote:
> On 24/01/2022 13:11, Yicong Yang wrote:
>> The DMA of HiSilicon PTT device can only work with identical
>> mapping. So add a quirk for the device to force the domain
>> passthrough.
>
> This patch should be earlier in the series, before the PTT driver, and the comment on hisi_ptt_check_iommu_mapping() should mention what is going on here.
>
ok I'll reorder the serives and modify the comments of hisi_ptt_check_iommu_mapping() like:
/*
* The DMA of PTT trace can only use direct mapping, due to some
* hardware restriction. Check whether there is an iommu or the
* policy of the iommu domain is passthrough, otherwise the trace
* cannot work.
*
* The PTT device is supposed to behind the arm smmu v3, which
* should have passthrough the device by a quirk. Otherwise user
* should manually set the iommu domain type to identity through
* sysfs.
*/
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>> 1 file changed, 16 insertions(+)
>>
>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>> index 6dc6d8b6b368..6f67a2b1dd27 100644
>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>> }
>> }
>> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
>> + (pdev)->device == 0xa12e)
>
> I assume that not all revisions will require this check, right?
>
For current revisions it's necessary.
>> +
>> +static int arm_smmu_def_domain_type(struct device *dev)
>> +{
>> + if (dev_is_pci(dev)) {
>> + struct pci_dev *pdev = to_pci_dev(dev);
>> +
>> + if (IS_HISI_PTT_DEVICE(pdev))
>> + return IOMMU_DOMAIN_IDENTITY;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> static struct iommu_ops arm_smmu_ops = {
>> .capable = arm_smmu_capable,
>> .domain_alloc = arm_smmu_domain_alloc,
>> @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
>> .sva_unbind = arm_smmu_sva_unbind,
>> .sva_get_pasid = arm_smmu_sva_get_pasid,
>> .page_response = arm_smmu_page_response,
>> + .def_domain_type = arm_smmu_def_domain_type,
>> .pgsize_bitmap = -1UL, /* Restricted during device attach */
>> .owner = THIS_MODULE,
>> };
>
> .
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-08 11:21 ` Yicong Yang
@ 2022-02-08 11:56 ` John Garry
2022-02-08 12:20 ` Yicong Yang
0 siblings, 1 reply; 32+ messages in thread
From: John Garry @ 2022-02-08 11:56 UTC (permalink / raw)
To: Yicong Yang, Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, Jonathan Cameron,
daniel.thompson, joro, Shameerali Kolothum Thodi, robin.murphy,
peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
linux-pci, linux-perf-users, iommu
Cc: Zengtao (B), liuqi (BA),
Zhangshaokun, Linuxarm, Song Bao Hua (Barry Song)
On 08/02/2022 11:21, Yicong Yang wrote:
>> This patch should be earlier in the series, before the PTT driver, and the comment on hisi_ptt_check_iommu_mapping() should mention what is going on here.
>>
> ok I'll reorder the serives and modify the comments of hisi_ptt_check_iommu_mapping() like:
>
> /*
> * The DMA of PTT trace can only use direct mapping, due to some
> * hardware restriction. Check whether there is an iommu or the
> * policy of the iommu domain is passthrough, otherwise the trace
> * cannot work.
IOMMU, capitalize acronyms
> *
> * The PTT device is supposed to behind the arm smmu v3, which
> * should have passthrough the device by a quirk. Otherwise user
> * should manually set the iommu domain type to identity through
> * sysfs.
Sorry, but I don't really understand your meaning here.
I did not think that if we have a default domain then we can change via
sysfs to anything else.
> */
>
>>> Signed-off-by: Yicong Yang<yangyicong@hisilicon.com>
>>> ---
>>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>>> 1 file changed, 16 insertions(+)
>>>
>>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>> index 6dc6d8b6b368..6f67a2b1dd27 100644
>>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>>> }
>>> }
>>> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
>>> + (pdev)->device == 0xa12e)
>> I assume that not all revisions will require this check, right?
So if you are very confident that the next revision will be fixed then I
would add a check for this current broken revision.
>>
> For current revisions it's necessary.
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-08 11:56 ` John Garry
@ 2022-02-08 12:20 ` Yicong Yang
0 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-08 12:20 UTC (permalink / raw)
To: John Garry, Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, Jonathan Cameron,
daniel.thompson, joro, Shameerali Kolothum Thodi, robin.murphy,
peterz, mingo, acme, linux-kernel, linux-arm-kernel, coresight,
linux-pci, linux-perf-users, iommu
Cc: Zengtao (B), liuqi (BA),
Zhangshaokun, Linuxarm, Song Bao Hua (Barry Song)
On 2022/2/8 19:56, John Garry wrote:
> On 08/02/2022 11:21, Yicong Yang wrote:
>>> This patch should be earlier in the series, before the PTT driver, and the comment on hisi_ptt_check_iommu_mapping() should mention what is going on here.
>>>
>> ok I'll reorder the serives and modify the comments of hisi_ptt_check_iommu_mapping() like:
>>
>> /*
>> * The DMA of PTT trace can only use direct mapping, due to some
>> * hardware restriction. Check whether there is an iommu or the
>> * policy of the iommu domain is passthrough, otherwise the trace
>> * cannot work.
>
> IOMMU, capitalize acronyms
>
ok.
>> *
>> * The PTT device is supposed to behind the arm smmu v3, which
>> * should have passthrough the device by a quirk. Otherwise user
>> * should manually set the iommu domain type to identity through
>> * sysfs.
>
> Sorry, but I don't really understand your meaning here.
>
> I did not think that if we have a default domain then we can change via sysfs to anything else.
>
ok I think the last sentence maybe misleading and better drop it.
>> */
>>
>>>> Signed-off-by: Yicong Yang<yangyicong@hisilicon.com>
>>>> ---
>>>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>>>> 1 file changed, 16 insertions(+)
>>>>
>>>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>> index 6dc6d8b6b368..6f67a2b1dd27 100644
>>>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>>>> }
>>>> }
>>>> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
>>>> + (pdev)->device == 0xa12e)
>>> I assume that not all revisions will require this check, right?
>
> So if you are very confident that the next revision will be fixed then I would add a check for this current broken revision.
>
>>>
>> For current revisions it's necessary.
>>
>
> .
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 1/8] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device
2022-02-08 11:07 ` Yicong Yang
@ 2022-02-14 12:51 ` Yicong Yang
0 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-14 12:51 UTC (permalink / raw)
To: Jonathan Cameron, Yicong Yang
Cc: gregkh, helgaas, alexander.shishkin, lorenzo.pieralisi, will,
mark.rutland, mathieu.poirier, suzuki.poulose, mike.leach,
leo.yan, daniel.thompson, joro, john.garry,
shameerali.kolothum.thodi, robin.murphy, peterz, mingo, acme,
linux-kernel, linux-arm-kernel, coresight, linux-pci,
linux-perf-users, iommu, prime.zeng, liuqi115, zhangshaokun,
linuxarm, song.bao.hua
On 2022/2/8 19:07, Yicong Yang wrote:
> On 2022/2/7 19:42, Jonathan Cameron wrote:
>> On Mon, 24 Jan 2022 21:11:11 +0800
>> Yicong Yang <yangyicong@hisilicon.com> wrote:
>>
>>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>>> integrated Endpoint(RCiEP) device, providing the capability
>>> to dynamically monitor and tune the PCIe traffic, and trace
>>> the TLP headers.
>>>
>>> Add the driver for the device to enable the trace function.
>>> This patch adds basic function of trace, including the device's
>>> probe and initialization, functions for trace buffer allocation
>>> and trace enable/disable, register an interrupt handler to
>>> simply response to the DMA events. The user interface of trace
>>> will be added in the following patch.
>>>
>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> Hi Yicong,
>>
>> I've not been following all the earlier discussion on this driver closely
>> so I may well raise something that has already been addressed. If so
>> just ignore the comment.
>
> Thanks for the comments. It's ok for me to clarify it :).
> Part replies inline and I need to do some test on the others.
>
>>
>> Thanks,
>>
>> Jonathan
>>
>>> ---
>>> drivers/Makefile | 1 +
>>> drivers/hwtracing/Kconfig | 2 +
>>> drivers/hwtracing/ptt/Kconfig | 11 +
>>> drivers/hwtracing/ptt/Makefile | 2 +
>>> drivers/hwtracing/ptt/hisi_ptt.c | 398 +++++++++++++++++++++++++++++++
>>> drivers/hwtracing/ptt/hisi_ptt.h | 159 ++++++++++++
>>> 6 files changed, 573 insertions(+)
>>> create mode 100644 drivers/hwtracing/ptt/Kconfig
>>> create mode 100644 drivers/hwtracing/ptt/Makefile
>>> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
>>> create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
>>>
> [...]
>>> +
>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>> +{
>>> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> + struct device *dev = &hisi_ptt->pdev->dev;
>>> + struct hisi_ptt_dma_buffer *buffer;
>>> + int i, ret;
>>> +
>>> + hisi_ptt->trace_ctrl.buf_index = 0;
>>> +
>>> + /* Make sure the trace buffer is empty before allocating */
>>
>> This comment is misleading as it suggests it not being empty is
>> a bad thing but the code handles it as an acceptable path.
>> Perhaps:
>> /*
>> * If the trace buffer has already been allocated, zero the
>> * memory.
>> */
>>
>
> will make it less misleading. thanks.
>
>>> + if (!list_empty(&ctrl->trace_buf)) {
>>> + list_for_each_entry(buffer, &ctrl->trace_buf, list)
>>> + memset(buffer->addr, 0, buffer->size);
>>> + return 0;
>>> + }
>>> +
>>> + for (i = 0; i < HISI_PTT_TRACE_BUF_CNT; ++i) {
>>> + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
>>> + if (!buffer) {
>>> + ret = -ENOMEM;
>>> + goto err;
>>> + }
>>> +
>>> + buffer->addr = dma_alloc_coherent(dev, ctrl->buffer_size,
>>> + &buffer->dma, GFP_KERNEL);
>>> + if (!buffer->addr) {
>>> + kfree(buffer);
>>> + ret = -ENOMEM;
>>> + goto err;
>>> + }
>>> +
>>> + memset(buffer->addr, 0, buffer->size);
>> See:
>> https://lore.kernel.org/lkml/20190108130701.14161-4-hch@lst.de/
>> dma_alloc_coherent() always zeros the memory for us hence there
>> is no longer a dma_kzalloc_coherent()
>>
>
> thanks for the information. Then the memset here is redundant and will drop it.
>
>>> +
>>> + buffer->index = i;
>>
>> Carrying an index inside a list which corresponds directly
>> to the position in the list is not particularly nice.
>> Why can't we compute this index on the fly where the list
>> is walked? Or am I misunderstanding and the order of the buffers
>> is changed in a later patch?
>>
>
> The index is fixed once allocated and I stored it to avoid later
> computing. But seems it's highly recommended to compute these sort
> of things on the fly when necessary. John recommends the same things
> on some other places so I think I can get these addressed.
>
>> As a side note, is a list actually appropriate when we always
>> have 4 of these buffers? Feels like an array of buffer
>> structures might be cheaper.
>>
As suggested here and below, I tried to maintianed the buffers with
an array instead of a list and it looks more straightforward and some
fields of buffer structure can also be dropped. So I think I can change
to use an array.
Thanks for the suggestion!
Yicong
>>> + buffer->size = ctrl->buffer_size;
>>> + list_add_tail(&buffer->list, &ctrl->trace_buf);
>>> + }
>>> +
>>> + return 0;
>>> +err:
>>> + hisi_ptt_free_trace_buf(hisi_ptt);
>>> + return ret;
>>> +}
>>> +
>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>> +{
>>> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> + hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>>> +}
>>> +
>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>> +{
>>> + struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>> + struct hisi_ptt_dma_buffer *cur;
>>> + u32 val;
>>> +
>>> + /* Check device idle before start trace */
>>> + if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>> + pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>>> + return -EBUSY;
>>> + }
>>> +
>>> + /* Reset the DMA before start tracing */
>>> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> + val |= HISI_PTT_TRACE_CTRL_RST;
>>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> + /*
>>> + * We'll be in the perf context where preemption is disabled,
>>> + * so use busy loop here.
>>> + */
>>> + mdelay(HISI_PTT_RESET_WAIT_MS);
>>
>> Busy look for 1 second? Ouch. If we can reduce this in any way
>> that would be great or if there is a means to do it before
>> we disable preemption.
>>
>
> It's inherited from the previous version that was using msleep() and it's
> somehow unacceptable in an atomic context I think. The reset here is
> going to reset the write pointer of the hardware DMA so we can check the
> whether the pointer before dereset it. I confirmed with our hardware
> teams that it can be reduced to 10us. So I'll poll the write pointer register
> for about 10us before continue here.
>
> thanks for catching this!
>
>>> +
>>> + val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> + val &= ~HISI_PTT_TRACE_CTRL_RST;
>>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> + /* Clear the interrupt status */
>>> + writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>> + writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>> +
>>> + /* Configure the trace DMA buffer */
>>> + list_for_each_entry(cur, &ctrl->trace_buf, list) {
>>
>> I comment on the use of cur->index above. Here it would be easy to compute
>> the index as we go for example assuming we never end up with holes
>> in the list.
>>
>
> ok.
>
>>> + writel(lower_32_bits(cur->dma),
>>> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>> + writel(upper_32_bits(cur->dma),
>>> + hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>> + cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>> + }
>>> + writel(ctrl->buffer_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>> +
>>> + /* Set the trace control register */
>>> + val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>> + val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>> + if (!hisi_ptt->trace_ctrl.is_port)
>>> + val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>> +
>>> + /* Start the Trace */
>>> + val |= HISI_PTT_TRACE_CTRL_EN;
>>> + writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>> +
>>> + ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>>> +
>>> + return 0;
>>> +}
>>> +
>>
>> ...
>>
>>> +
>>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>>> +{
>>> + struct pci_dev *pdev = hisi_ptt->pdev;
>>> + struct pci_bus *bus;
>>> + u32 reg;
>>> +
>>> + INIT_LIST_HEAD(&hisi_ptt->port_filters);
>>> + INIT_LIST_HEAD(&hisi_ptt->req_filters);
>>> +
>>> + /*
>>> + * The device range register provides the information about the
>>> + * root ports which the RCiEP can control and trace. The RCiEP
>>> + * and the root ports it support are on the same PCIe core, with
>>> + * same domain number but maybe different bus number. The device
>>> + * range register will tell us which root ports we can support,
>>> + * Bit[31:16] indicates the upper BDF numbers of the root port,
>>> + * while Bit[15:0] indicates the lower.
>>> + */
>>> + reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>>> + hisi_ptt->upper = reg >> 16;
>>> + hisi_ptt->lower = reg & 0xffff;
>> Trivial:
>> Perhaps worthing define HISI_PTT_DEVICE_RANGE_UPPER_MASK etc adn using
>> FIELD_GET?
>>
>
> sure.
>
>>> +
>>> + reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>> + hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>> + hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>> +
>>> + bus = pci_find_bus(pci_domain_nr(pdev->bus), PCI_BUS_NUM(hisi_ptt->upper));
>>> + if (bus)
>>> + pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>>> +
>>> + /* Initialize trace controls */
>>> + INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>>> + hisi_ptt->trace_ctrl.buffer_size = HISI_PTT_TRACE_BUF_SIZE;
>>> + hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>>> +}
>>> +
> [...]
>>> +
>>> +#define HISI_PCIE_CORE_PORT_ID(devfn) (PCI_FUNC(devfn) << 1)
>>> +
>>> +enum hisi_ptt_trace_status {
>>> + HISI_PTT_TRACE_STATUS_OFF = 0,
>>> + HISI_PTT_TRACE_STATUS_ON,
>>> +};
>>
>> Why not just use a boolean given we only have off and on states?
>>
>
> An enum may make the code more readable I think.
>
> Thanks,
> Yicong
>
> .
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-01-24 13:11 ` [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
2022-02-08 8:05 ` John Garry
@ 2022-02-14 12:55 ` Yicong Yang
2022-02-15 13:00 ` Will Deacon
1 sibling, 1 reply; 32+ messages in thread
From: Yicong Yang @ 2022-02-14 12:55 UTC (permalink / raw)
To: Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, will, mark.rutland, mathieu.poirier,
suzuki.poulose, mike.leach, leo.yan, jonathan.cameron,
daniel.thompson, joro, john.garry, shameerali.kolothum.thodi,
robin.murphy, peterz, mingo, acme, linux-kernel,
linux-arm-kernel, coresight, linux-pci, linux-perf-users, iommu
Cc: prime.zeng, liuqi115, zhangshaokun, linuxarm, song.bao.hua
Hi Robin,
Is this quirk ok with the SMMU v3 driver? Just want to confirm that I'm on the
right way to dealing with the issue of our device.
Thanks.
On 2022/1/24 21:11, Yicong Yang wrote:
> The DMA of HiSilicon PTT device can only work with identical
> mapping. So add a quirk for the device to force the domain
> passthrough.
>
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
> 1 file changed, 16 insertions(+)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 6dc6d8b6b368..6f67a2b1dd27 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
> }
> }
>
> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
> + (pdev)->device == 0xa12e)
> +
> +static int arm_smmu_def_domain_type(struct device *dev)
> +{
> + if (dev_is_pci(dev)) {
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (IS_HISI_PTT_DEVICE(pdev))
> + return IOMMU_DOMAIN_IDENTITY;
> + }
> +
> + return 0;
> +}
> +
> static struct iommu_ops arm_smmu_ops = {
> .capable = arm_smmu_capable,
> .domain_alloc = arm_smmu_domain_alloc,
> @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
> .sva_unbind = arm_smmu_sva_unbind,
> .sva_get_pasid = arm_smmu_sva_get_pasid,
> .page_response = arm_smmu_page_response,
> + .def_domain_type = arm_smmu_def_domain_type,
> .pgsize_bitmap = -1UL, /* Restricted during device attach */
> .owner = THIS_MODULE,
> };
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-14 12:55 ` Yicong Yang
@ 2022-02-15 13:00 ` Will Deacon
2022-02-15 13:30 ` Robin Murphy
0 siblings, 1 reply; 32+ messages in thread
From: Will Deacon @ 2022-02-15 13:00 UTC (permalink / raw)
To: Yicong Yang
Cc: Yicong Yang, gregkh, helgaas, alexander.shishkin,
lorenzo.pieralisi, mark.rutland, mathieu.poirier, suzuki.poulose,
mike.leach, leo.yan, jonathan.cameron, daniel.thompson, joro,
john.garry, shameerali.kolothum.thodi, robin.murphy, peterz,
mingo, acme, linux-kernel, linux-arm-kernel, coresight,
linux-pci, linux-perf-users, iommu, prime.zeng, liuqi115,
zhangshaokun, linuxarm, song.bao.hua
On Mon, Feb 14, 2022 at 08:55:20PM +0800, Yicong Yang wrote:
> On 2022/1/24 21:11, Yicong Yang wrote:
> > The DMA of HiSilicon PTT device can only work with identical
> > mapping. So add a quirk for the device to force the domain
> > passthrough.
> >
> > Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
> > 1 file changed, 16 insertions(+)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 6dc6d8b6b368..6f67a2b1dd27 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
> > }
> > }
> >
> > +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
> > + (pdev)->device == 0xa12e)
> > +
> > +static int arm_smmu_def_domain_type(struct device *dev)
> > +{
> > + if (dev_is_pci(dev)) {
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > +
> > + if (IS_HISI_PTT_DEVICE(pdev))
> > + return IOMMU_DOMAIN_IDENTITY;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > static struct iommu_ops arm_smmu_ops = {
> > .capable = arm_smmu_capable,
> > .domain_alloc = arm_smmu_domain_alloc,
> > @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
> > .sva_unbind = arm_smmu_sva_unbind,
> > .sva_get_pasid = arm_smmu_sva_get_pasid,
> > .page_response = arm_smmu_page_response,
> > + .def_domain_type = arm_smmu_def_domain_type,
> > .pgsize_bitmap = -1UL, /* Restricted during device attach */
> > .owner = THIS_MODULE,
> > };
> >
>
> Is this quirk ok with the SMMU v3 driver? Just want to confirm that I'm on the
> right way to dealing with the issue of our device.
I don't think the quirk should be in the SMMUv3 driver. Assumedly, you would
have the exact same problem if you stuck the PTT device behind a different
type of IOMMU, and so the quirk should be handled by a higher level of the
stack.
Will
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-15 13:00 ` Will Deacon
@ 2022-02-15 13:30 ` Robin Murphy
2022-02-15 13:42 ` Will Deacon
0 siblings, 1 reply; 32+ messages in thread
From: Robin Murphy @ 2022-02-15 13:30 UTC (permalink / raw)
To: Will Deacon, Yicong Yang
Cc: mark.rutland, prime.zeng, alexander.shishkin, linux-pci,
linuxarm, Yicong Yang, daniel.thompson, peterz, mingo, helgaas,
liuqi115, mike.leach, suzuki.poulose, coresight, acme,
zhangshaokun, linux-arm-kernel, mathieu.poirier, gregkh,
linux-kernel, linux-perf-users, iommu, leo.yan
On 2022-02-15 13:00, Will Deacon wrote:
> On Mon, Feb 14, 2022 at 08:55:20PM +0800, Yicong Yang wrote:
>> On 2022/1/24 21:11, Yicong Yang wrote:
>>> The DMA of HiSilicon PTT device can only work with identical
>>> mapping. So add a quirk for the device to force the domain
>>> passthrough.
>>>
>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>> ---
>>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>>> 1 file changed, 16 insertions(+)
>>>
>>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>> index 6dc6d8b6b368..6f67a2b1dd27 100644
>>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>>> }
>>> }
>>>
>>> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
>>> + (pdev)->device == 0xa12e)
>>> +
>>> +static int arm_smmu_def_domain_type(struct device *dev)
>>> +{
>>> + if (dev_is_pci(dev)) {
>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>> +
>>> + if (IS_HISI_PTT_DEVICE(pdev))
>>> + return IOMMU_DOMAIN_IDENTITY;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> static struct iommu_ops arm_smmu_ops = {
>>> .capable = arm_smmu_capable,
>>> .domain_alloc = arm_smmu_domain_alloc,
>>> @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
>>> .sva_unbind = arm_smmu_sva_unbind,
>>> .sva_get_pasid = arm_smmu_sva_get_pasid,
>>> .page_response = arm_smmu_page_response,
>>> + .def_domain_type = arm_smmu_def_domain_type,
>>> .pgsize_bitmap = -1UL, /* Restricted during device attach */
>>> .owner = THIS_MODULE,
>>> };
>>>
>>
>> Is this quirk ok with the SMMU v3 driver? Just want to confirm that I'm on the
>> right way to dealing with the issue of our device.
>
> I don't think the quirk should be in the SMMUv3 driver. Assumedly, you would
> have the exact same problem if you stuck the PTT device behind a different
> type of IOMMU, and so the quirk should be handled by a higher level of the
> stack.
Conceptually, yes, but I'm inclined to be pragmatic here. Default domain
quirks could only move out as far as the other end of the call from
iommu_get_def_domain_type() - it's not like we could rely on some flag
in a driver which may not even be loaded yet, let alone matched to the
device. And even then there's an equal and opposite argument for why the
core code should have to maintain a list of platform-specific quirks
rather than code specific to the relevant platforms. The fact is that a
HiSilicon RCiEP is not going to end up behind anything other than a
HiSilicon IOMMU, and if those ever stop being SMMUv3 *and* such a quirk
still exists we can worry about it then.
Ugly as it is, this is the status quo. I don't recall anyone ever
arguing that the equivalent quirks for Intel integrated graphics should
be made generic ;)
Cheers,
Robin.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-15 13:30 ` Robin Murphy
@ 2022-02-15 13:42 ` Will Deacon
2022-02-15 14:29 ` Robin Murphy
0 siblings, 1 reply; 32+ messages in thread
From: Will Deacon @ 2022-02-15 13:42 UTC (permalink / raw)
To: Robin Murphy
Cc: Yicong Yang, mark.rutland, prime.zeng, alexander.shishkin,
linux-pci, linuxarm, Yicong Yang, daniel.thompson, peterz, mingo,
helgaas, liuqi115, mike.leach, suzuki.poulose, coresight, acme,
zhangshaokun, linux-arm-kernel, mathieu.poirier, gregkh,
linux-kernel, linux-perf-users, iommu, leo.yan
On Tue, Feb 15, 2022 at 01:30:26PM +0000, Robin Murphy wrote:
> On 2022-02-15 13:00, Will Deacon wrote:
> > On Mon, Feb 14, 2022 at 08:55:20PM +0800, Yicong Yang wrote:
> > > On 2022/1/24 21:11, Yicong Yang wrote:
> > > > The DMA of HiSilicon PTT device can only work with identical
> > > > mapping. So add a quirk for the device to force the domain
> > > > passthrough.
> > > >
> > > > Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> > > > ---
> > > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
> > > > 1 file changed, 16 insertions(+)
> > > >
> > > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > > index 6dc6d8b6b368..6f67a2b1dd27 100644
> > > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > > > @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
> > > > }
> > > > }
> > > > +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
> > > > + (pdev)->device == 0xa12e)
> > > > +
> > > > +static int arm_smmu_def_domain_type(struct device *dev)
> > > > +{
> > > > + if (dev_is_pci(dev)) {
> > > > + struct pci_dev *pdev = to_pci_dev(dev);
> > > > +
> > > > + if (IS_HISI_PTT_DEVICE(pdev))
> > > > + return IOMMU_DOMAIN_IDENTITY;
> > > > + }
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > static struct iommu_ops arm_smmu_ops = {
> > > > .capable = arm_smmu_capable,
> > > > .domain_alloc = arm_smmu_domain_alloc,
> > > > @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
> > > > .sva_unbind = arm_smmu_sva_unbind,
> > > > .sva_get_pasid = arm_smmu_sva_get_pasid,
> > > > .page_response = arm_smmu_page_response,
> > > > + .def_domain_type = arm_smmu_def_domain_type,
> > > > .pgsize_bitmap = -1UL, /* Restricted during device attach */
> > > > .owner = THIS_MODULE,
> > > > };
> > > >
> > >
> > > Is this quirk ok with the SMMU v3 driver? Just want to confirm that I'm on the
> > > right way to dealing with the issue of our device.
> >
> > I don't think the quirk should be in the SMMUv3 driver. Assumedly, you would
> > have the exact same problem if you stuck the PTT device behind a different
> > type of IOMMU, and so the quirk should be handled by a higher level of the
> > stack.
>
> Conceptually, yes, but I'm inclined to be pragmatic here. Default domain
> quirks could only move out as far as the other end of the call from
> iommu_get_def_domain_type() - it's not like we could rely on some flag in a
> driver which may not even be loaded yet, let alone matched to the device.
> And even then there's an equal and opposite argument for why the core code
> should have to maintain a list of platform-specific quirks rather than code
> specific to the relevant platforms. The fact is that a HiSilicon RCiEP is
> not going to end up behind anything other than a HiSilicon IOMMU, and if
> those ever stop being SMMUv3 *and* such a quirk still exists we can worry
> about it then.
Perhaps, but you know that by adding this hook it's only a matter of time
before we get random compatible string matches in there, so I'd rather keep
the flood gates closed as long as we can.
Given that this is a PCI device, why can't we have a PCI quirk for devices
which require an identity mapping and then handle that in the IOMMU core?
> Ugly as it is, this is the status quo. I don't recall anyone ever arguing
> that the equivalent quirks for Intel integrated graphics should be made
> generic ;)
I don't know anything about Intel integrated graphics. Have they solved this
problem in a better way, or could they equally make use of a generic quirk?
Will
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-15 13:42 ` Will Deacon
@ 2022-02-15 14:29 ` Robin Murphy
2022-02-16 9:35 ` Yicong Yang
0 siblings, 1 reply; 32+ messages in thread
From: Robin Murphy @ 2022-02-15 14:29 UTC (permalink / raw)
To: Will Deacon
Cc: Yicong Yang, mark.rutland, prime.zeng, alexander.shishkin,
linux-pci, linuxarm, Yicong Yang, daniel.thompson, peterz, mingo,
helgaas, liuqi115, mike.leach, suzuki.poulose, coresight, acme,
zhangshaokun, linux-arm-kernel, mathieu.poirier, gregkh,
linux-kernel, linux-perf-users, iommu, leo.yan
On 2022-02-15 13:42, Will Deacon wrote:
> On Tue, Feb 15, 2022 at 01:30:26PM +0000, Robin Murphy wrote:
>> On 2022-02-15 13:00, Will Deacon wrote:
>>> On Mon, Feb 14, 2022 at 08:55:20PM +0800, Yicong Yang wrote:
>>>> On 2022/1/24 21:11, Yicong Yang wrote:
>>>>> The DMA of HiSilicon PTT device can only work with identical
>>>>> mapping. So add a quirk for the device to force the domain
>>>>> passthrough.
>>>>>
>>>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>>>> ---
>>>>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>>>>> 1 file changed, 16 insertions(+)
>>>>>
>>>>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>>> index 6dc6d8b6b368..6f67a2b1dd27 100644
>>>>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>>> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>>>>> }
>>>>> }
>>>>> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
>>>>> + (pdev)->device == 0xa12e)
>>>>> +
>>>>> +static int arm_smmu_def_domain_type(struct device *dev)
>>>>> +{
>>>>> + if (dev_is_pci(dev)) {
>>>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>>>> +
>>>>> + if (IS_HISI_PTT_DEVICE(pdev))
>>>>> + return IOMMU_DOMAIN_IDENTITY;
>>>>> + }
>>>>> +
>>>>> + return 0;
>>>>> +}
>>>>> +
>>>>> static struct iommu_ops arm_smmu_ops = {
>>>>> .capable = arm_smmu_capable,
>>>>> .domain_alloc = arm_smmu_domain_alloc,
>>>>> @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
>>>>> .sva_unbind = arm_smmu_sva_unbind,
>>>>> .sva_get_pasid = arm_smmu_sva_get_pasid,
>>>>> .page_response = arm_smmu_page_response,
>>>>> + .def_domain_type = arm_smmu_def_domain_type,
>>>>> .pgsize_bitmap = -1UL, /* Restricted during device attach */
>>>>> .owner = THIS_MODULE,
>>>>> };
>>>>>
>>>>
>>>> Is this quirk ok with the SMMU v3 driver? Just want to confirm that I'm on the
>>>> right way to dealing with the issue of our device.
>>>
>>> I don't think the quirk should be in the SMMUv3 driver. Assumedly, you would
>>> have the exact same problem if you stuck the PTT device behind a different
>>> type of IOMMU, and so the quirk should be handled by a higher level of the
>>> stack.
>>
>> Conceptually, yes, but I'm inclined to be pragmatic here. Default domain
>> quirks could only move out as far as the other end of the call from
>> iommu_get_def_domain_type() - it's not like we could rely on some flag in a
>> driver which may not even be loaded yet, let alone matched to the device.
>> And even then there's an equal and opposite argument for why the core code
>> should have to maintain a list of platform-specific quirks rather than code
>> specific to the relevant platforms. The fact is that a HiSilicon RCiEP is
>> not going to end up behind anything other than a HiSilicon IOMMU, and if
>> those ever stop being SMMUv3 *and* such a quirk still exists we can worry
>> about it then.
>
> Perhaps, but you know that by adding this hook it's only a matter of time
> before we get random compatible string matches in there, so I'd rather keep
> the flood gates closed as long as we can.
>
> Given that this is a PCI device, why can't we have a PCI quirk for devices
> which require an identity mapping and then handle that in the IOMMU core?
Oh, don't think I *like* having quirks in the driver, it just seems like
the least-worst choice from a bad bunch. All of the default domain
quirks so far (including this one) exist for integrated devices and/or
dodgy firmware setups such that they are platform-specific, so there is
no technical reason for trying to split *some* of them off into a
generic mechanism when the driver-based platform-specific mechanism
still needs to exist anyway (some of them do depend on driver state as
well).
Feel free to test the waters with a patch punting
qcom_smmu_def_domain_type() to core code, but I think you'll struggle to
find a reason to give in the commit message other than "I don't like it".
>> Ugly as it is, this is the status quo. I don't recall anyone ever arguing
>> that the equivalent quirks for Intel integrated graphics should be made
>> generic ;)
>
> I don't know anything about Intel integrated graphics. Have they solved this
> problem in a better way, or could they equally make use of a generic quirk?
See intel-iommu's device_def_domain_type() implementation. The shape of
it may seem quite familiar...
Robin.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity
2022-02-15 14:29 ` Robin Murphy
@ 2022-02-16 9:35 ` Yicong Yang
0 siblings, 0 replies; 32+ messages in thread
From: Yicong Yang @ 2022-02-16 9:35 UTC (permalink / raw)
To: Robin Murphy, Will Deacon
Cc: yangyicong, mark.rutland, prime.zeng, alexander.shishkin,
linux-pci, linuxarm, daniel.thompson, peterz, mingo, helgaas,
liuqi115, mike.leach, suzuki.poulose, coresight, acme,
zhangshaokun, linux-arm-kernel, mathieu.poirier, gregkh,
linux-kernel, linux-perf-users, iommu, leo.yan
On 2022/2/15 22:29, Robin Murphy wrote:
> On 2022-02-15 13:42, Will Deacon wrote:
>> On Tue, Feb 15, 2022 at 01:30:26PM +0000, Robin Murphy wrote:
>>> On 2022-02-15 13:00, Will Deacon wrote:
>>>> On Mon, Feb 14, 2022 at 08:55:20PM +0800, Yicong Yang wrote:
>>>>> On 2022/1/24 21:11, Yicong Yang wrote:
>>>>>> The DMA of HiSilicon PTT device can only work with identical
>>>>>> mapping. So add a quirk for the device to force the domain
>>>>>> passthrough.
>>>>>>
>>>>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>>>>> ---
>>>>>> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 ++++++++++++++++
>>>>>> 1 file changed, 16 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>>>> index 6dc6d8b6b368..6f67a2b1dd27 100644
>>>>>> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>>>> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
>>>>>> @@ -2838,6 +2838,21 @@ static int arm_smmu_dev_disable_feature(struct device *dev,
>>>>>> }
>>>>>> }
>>>>>> +#define IS_HISI_PTT_DEVICE(pdev) ((pdev)->vendor == PCI_VENDOR_ID_HUAWEI && \
>>>>>> + (pdev)->device == 0xa12e)
>>>>>> +
>>>>>> +static int arm_smmu_def_domain_type(struct device *dev)
>>>>>> +{
>>>>>> + if (dev_is_pci(dev)) {
>>>>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>>>>> +
>>>>>> + if (IS_HISI_PTT_DEVICE(pdev))
>>>>>> + return IOMMU_DOMAIN_IDENTITY;
>>>>>> + }
>>>>>> +
>>>>>> + return 0;
>>>>>> +}
>>>>>> +
>>>>>> static struct iommu_ops arm_smmu_ops = {
>>>>>> .capable = arm_smmu_capable,
>>>>>> .domain_alloc = arm_smmu_domain_alloc,
>>>>>> @@ -2863,6 +2878,7 @@ static struct iommu_ops arm_smmu_ops = {
>>>>>> .sva_unbind = arm_smmu_sva_unbind,
>>>>>> .sva_get_pasid = arm_smmu_sva_get_pasid,
>>>>>> .page_response = arm_smmu_page_response,
>>>>>> + .def_domain_type = arm_smmu_def_domain_type,
>>>>>> .pgsize_bitmap = -1UL, /* Restricted during device attach */
>>>>>> .owner = THIS_MODULE,
>>>>>> };
>>>>>>
>>>>>
>>>>> Is this quirk ok with the SMMU v3 driver? Just want to confirm that I'm on the
>>>>> right way to dealing with the issue of our device.
>>>>
>>>> I don't think the quirk should be in the SMMUv3 driver. Assumedly, you would
>>>> have the exact same problem if you stuck the PTT device behind a different
>>>> type of IOMMU, and so the quirk should be handled by a higher level of the
>>>> stack.
>>>
>>> Conceptually, yes, but I'm inclined to be pragmatic here. Default domain
>>> quirks could only move out as far as the other end of the call from
>>> iommu_get_def_domain_type() - it's not like we could rely on some flag in a
>>> driver which may not even be loaded yet, let alone matched to the device.
>>> And even then there's an equal and opposite argument for why the core code
>>> should have to maintain a list of platform-specific quirks rather than code
>>> specific to the relevant platforms. The fact is that a HiSilicon RCiEP is
>>> not going to end up behind anything other than a HiSilicon IOMMU, and if
>>> those ever stop being SMMUv3 *and* such a quirk still exists we can worry
>>> about it then.
>>
That's true that this RCiEP only appears behind the HiSilicon's IOMMU which using
SMMU v3 driver.
>> Perhaps, but you know that by adding this hook it's only a matter of time
>> before we get random compatible string matches in there, so I'd rather keep
>> the flood gates closed as long as we can.
>>
>> Given that this is a PCI device, why can't we have a PCI quirk for devices
>> which require an identity mapping and then handle that in the IOMMU core?
>
As Robin mentioned below, not only PCI devices but some platform devices also want
to passthrough the IOMMU. I noticed there're already some fields describe the device's
DMA information in struct device, so follow your point can it go there if we're going
to make it more generic?
Anyway if we're going to make all these quirks in a more generic place, I'll willing
to add this device there and have a test.
> Oh, don't think I *like* having quirks in the driver, it just seems like the least-worst choice from a bad bunch. All of the default domain quirks so far (including this one) exist for integrated devices and/or dodgy firmware setups such that they are platform-specific, so there is no technical reason for trying to split *some* of them off into a generic mechanism when the driver-based platform-specific mechanism still needs to exist anyway (some of them do depend on driver state as well).
>
> Feel free to test the waters with a patch punting qcom_smmu_def_domain_type() to core code, but I think you'll struggle to find a reason to give in the commit message other than "I don't like it".
>
>>> Ugly as it is, this is the status quo. I don't recall anyone ever arguing
>>> that the equivalent quirks for Intel integrated graphics should be made
>>> generic ;)
>>
>> I don't know anything about Intel integrated graphics. Have they solved this
>> problem in a better way, or could they equally make use of a generic quirk?
>
> See intel-iommu's device_def_domain_type() implementation. The shape of it may seem quite familiar...
>
Yes Intel's IOMMU passthrough some PCI devices in this way and this patch imitates that.
https://github.com/torvalds/linux/blob/master/drivers/iommu/intel/iommu.c#L2959
btw Will, much appreciated if you could have a look at the perf and PMU part of this driver.:)
Thanks,
Yicong
^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2022-02-16 9:35 UTC | newest]
Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-24 13:11 [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
2022-01-24 13:11 ` [PATCH v3 1/8] hwtracing: Add trace function " Yicong Yang
2022-02-07 11:42 ` Jonathan Cameron
2022-02-08 11:07 ` Yicong Yang
2022-02-14 12:51 ` Yicong Yang
2022-02-07 18:11 ` John Garry
2022-02-08 8:57 ` Yicong Yang
2022-01-24 13:11 ` [PATCH v3 2/8] hisi_ptt: Register PMU device for PTT trace Yicong Yang
2022-02-07 11:42 ` Jonathan Cameron
2022-02-08 7:41 ` Yicong Yang
2022-01-24 13:11 ` [PATCH v3 3/8] hisi_ptt: Add support for dynamically updating the filter list Yicong Yang
2022-01-24 13:11 ` [PATCH v3 4/8] hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace device Yicong Yang
2022-02-07 11:49 ` Jonathan Cameron
2022-02-08 7:08 ` Yicong Yang
2022-01-24 13:11 ` [PATCH v3 5/8] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver Yicong Yang
2022-02-07 11:55 ` Jonathan Cameron
2022-01-24 13:11 ` [PATCH v3 6/8] docs: Add HiSilicon PTT device driver documentation Yicong Yang
2022-02-07 12:12 ` Jonathan Cameron
2022-02-08 11:09 ` Yicong Yang
2022-01-24 13:11 ` [PATCH v3 7/8] MAINTAINERS: Add maintainer for HiSilicon PTT driver Yicong Yang
2022-01-24 13:11 ` [PATCH v3 8/8] iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to identity Yicong Yang
2022-02-08 8:05 ` John Garry
2022-02-08 11:21 ` Yicong Yang
2022-02-08 11:56 ` John Garry
2022-02-08 12:20 ` Yicong Yang
2022-02-14 12:55 ` Yicong Yang
2022-02-15 13:00 ` Will Deacon
2022-02-15 13:30 ` Robin Murphy
2022-02-15 13:42 ` Will Deacon
2022-02-15 14:29 ` Robin Murphy
2022-02-16 9:35 ` Yicong Yang
2022-02-07 9:40 ` [PATCH v3 0/8] Add support for HiSilicon PCIe Tune and Trace device Yicong Yang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).