[PATCH v2 0/6] Add support for HiSilicon PCIe Tune and Trace device

[PATCH v2 0/6] Add support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
integrated Endpoint (RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic (tune),
and trace the TLP headers (trace).

PTT tune is designed for monitoring and adjusting PCIe link parameters.
We provide several parameters of the PCIe link. Through the driver,
user can adjust the value of certain parameter to affect the PCIe link
for the purpose of enhancing the performance in certian situation.

PTT trace is designed for dumping the TLP headers to the memory, which
can be used to analyze the transactions and usage condition of the PCIe
Link. Users can choose filters to trace headers, by either requester
ID, or those downstream of a set of Root Ports on the same core of the
PTT device. It's also supported to trace the headers of certain type and
of certain direction.

The driver registers a PMU device for each PTT device. The trace can
be used through `perf record` and the traced headers can be decoded
by `perf report`. The perf command support for the device is also
added in this patchset. The tune can be used through the sysfs
attributes of related PMU device. See the documentation for the
detailed usage.

Change since v1:
- switch the user interface of trace to perf from debugfs
- switch the user interface of tune to sysfs from debugfs
- add perf tool support to start trace and decode the trace data
- address the comments of documentation from Bjorn
- add RMR[1] support of the device as trace works in RMR mode or
  direct DMA mode. RMR support is achieved by common APIs rather
  than the APIs implemented in [1].
Link: https://lore.kernel.org/lkml/1618654631-42454-1-git-send-email-yangyicong@hisilicon.com/

[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/

Qi Liu (1):
  perf tool: Add support for HiSilicon PCIe Tune and Trace device driver

Yicong Yang (5):
  iommu: Export iommu_{get,put}_resv_regions()
  hwtracing: Add trace function support for HiSilicon PCIe Tune and
    Trace device
  hwtracing: Add tune function support for HiSilicon PCIe Tune and Trace
    device
  docs: Add HiSilicon PTT device driver documentation
  MAINTAINERS: Add maintainer for HiSilicon PTT driver

 Documentation/trace/hisi-ptt.rst              |  305 ++++
 MAINTAINERS                                   |    7 +
 drivers/Makefile                              |    1 +
 drivers/hwtracing/Kconfig                     |    2 +
 drivers/hwtracing/hisilicon/Kconfig           |    8 +
 drivers/hwtracing/hisilicon/Makefile          |    2 +
 drivers/hwtracing/hisilicon/hisi_ptt.c        | 1313 +++++++++++++++++
 drivers/iommu/iommu.c                         |    2 +
 include/linux/iommu.h                         |    4 +-
 tools/perf/arch/arm/util/auxtrace.c           |   56 +-
 tools/perf/arch/arm/util/pmu.c                |    3 +
 tools/perf/arch/arm64/util/Build              |    2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         |  195 +++
 tools/perf/util/Build                         |    2 +
 tools/perf/util/auxtrace.c                    |    4 +
 tools/perf/util/auxtrace.h                    |    1 +
 tools/perf/util/hisi-ptt-decoder/Build        |    1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   |  170 +++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |   28 +
 tools/perf/util/hisi_ptt.c                    |  228 +++
 tools/perf/util/hisi_ptt.h                    |   28 +
 21 files changed, 2356 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/trace/hisi-ptt.rst
 create mode 100644 drivers/hwtracing/hisilicon/Kconfig
 create mode 100644 drivers/hwtracing/hisilicon/Makefile
 create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

-- 
2.33.0

[PATCH v2 0/6] Add support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
integrated Endpoint (RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic (tune),
and trace the TLP headers (trace).

PTT tune is designed for monitoring and adjusting PCIe link parameters.
We provide several parameters of the PCIe link. Through the driver,
user can adjust the value of certain parameter to affect the PCIe link
for the purpose of enhancing the performance in certian situation.

PTT trace is designed for dumping the TLP headers to the memory, which
can be used to analyze the transactions and usage condition of the PCIe
Link. Users can choose filters to trace headers, by either requester
ID, or those downstream of a set of Root Ports on the same core of the
PTT device. It's also supported to trace the headers of certain type and
of certain direction.

The driver registers a PMU device for each PTT device. The trace can
be used through `perf record` and the traced headers can be decoded
by `perf report`. The perf command support for the device is also
added in this patchset. The tune can be used through the sysfs
attributes of related PMU device. See the documentation for the
detailed usage.

Change since v1:
- switch the user interface of trace to perf from debugfs
- switch the user interface of tune to sysfs from debugfs
- add perf tool support to start trace and decode the trace data
- address the comments of documentation from Bjorn
- add RMR[1] support of the device as trace works in RMR mode or
  direct DMA mode. RMR support is achieved by common APIs rather
  than the APIs implemented in [1].
Link: https://lore.kernel.org/lkml/1618654631-42454-1-git-send-email-yangyicong@hisilicon.com/

[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/

Qi Liu (1):
  perf tool: Add support for HiSilicon PCIe Tune and Trace device driver

Yicong Yang (5):
  iommu: Export iommu_{get,put}_resv_regions()
  hwtracing: Add trace function support for HiSilicon PCIe Tune and
    Trace device
  hwtracing: Add tune function support for HiSilicon PCIe Tune and Trace
    device
  docs: Add HiSilicon PTT device driver documentation
  MAINTAINERS: Add maintainer for HiSilicon PTT driver

 Documentation/trace/hisi-ptt.rst              |  305 ++++
 MAINTAINERS                                   |    7 +
 drivers/Makefile                              |    1 +
 drivers/hwtracing/Kconfig                     |    2 +
 drivers/hwtracing/hisilicon/Kconfig           |    8 +
 drivers/hwtracing/hisilicon/Makefile          |    2 +
 drivers/hwtracing/hisilicon/hisi_ptt.c        | 1313 +++++++++++++++++
 drivers/iommu/iommu.c                         |    2 +
 include/linux/iommu.h                         |    4 +-
 tools/perf/arch/arm/util/auxtrace.c           |   56 +-
 tools/perf/arch/arm/util/pmu.c                |    3 +
 tools/perf/arch/arm64/util/Build              |    2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         |  195 +++
 tools/perf/util/Build                         |    2 +
 tools/perf/util/auxtrace.c                    |    4 +
 tools/perf/util/auxtrace.h                    |    1 +
 tools/perf/util/hisi-ptt-decoder/Build        |    1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   |  170 +++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |   28 +
 tools/perf/util/hisi_ptt.c                    |  228 +++
 tools/perf/util/hisi_ptt.h                    |   28 +
 21 files changed, 2356 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/trace/hisi-ptt.rst
 create mode 100644 drivers/hwtracing/hisilicon/Kconfig
 create mode 100644 drivers/hwtracing/hisilicon/Makefile
 create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

-- 
2.33.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 0/6] Add support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
integrated Endpoint (RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic (tune),
and trace the TLP headers (trace).

PTT tune is designed for monitoring and adjusting PCIe link parameters.
We provide several parameters of the PCIe link. Through the driver,
user can adjust the value of certain parameter to affect the PCIe link
for the purpose of enhancing the performance in certian situation.

PTT trace is designed for dumping the TLP headers to the memory, which
can be used to analyze the transactions and usage condition of the PCIe
Link. Users can choose filters to trace headers, by either requester
ID, or those downstream of a set of Root Ports on the same core of the
PTT device. It's also supported to trace the headers of certain type and
of certain direction.

The driver registers a PMU device for each PTT device. The trace can
be used through `perf record` and the traced headers can be decoded
by `perf report`. The perf command support for the device is also
added in this patchset. The tune can be used through the sysfs
attributes of related PMU device. See the documentation for the
detailed usage.

Change since v1:
- switch the user interface of trace to perf from debugfs
- switch the user interface of tune to sysfs from debugfs
- add perf tool support to start trace and decode the trace data
- address the comments of documentation from Bjorn
- add RMR[1] support of the device as trace works in RMR mode or
  direct DMA mode. RMR support is achieved by common APIs rather
  than the APIs implemented in [1].
Link: https://lore.kernel.org/lkml/1618654631-42454-1-git-send-email-yangyicong@hisilicon.com/

[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/

Qi Liu (1):
  perf tool: Add support for HiSilicon PCIe Tune and Trace device driver

Yicong Yang (5):
  iommu: Export iommu_{get,put}_resv_regions()
  hwtracing: Add trace function support for HiSilicon PCIe Tune and
    Trace device
  hwtracing: Add tune function support for HiSilicon PCIe Tune and Trace
    device
  docs: Add HiSilicon PTT device driver documentation
  MAINTAINERS: Add maintainer for HiSilicon PTT driver

 Documentation/trace/hisi-ptt.rst              |  305 ++++
 MAINTAINERS                                   |    7 +
 drivers/Makefile                              |    1 +
 drivers/hwtracing/Kconfig                     |    2 +
 drivers/hwtracing/hisilicon/Kconfig           |    8 +
 drivers/hwtracing/hisilicon/Makefile          |    2 +
 drivers/hwtracing/hisilicon/hisi_ptt.c        | 1313 +++++++++++++++++
 drivers/iommu/iommu.c                         |    2 +
 include/linux/iommu.h                         |    4 +-
 tools/perf/arch/arm/util/auxtrace.c           |   56 +-
 tools/perf/arch/arm/util/pmu.c                |    3 +
 tools/perf/arch/arm64/util/Build              |    2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         |  195 +++
 tools/perf/util/Build                         |    2 +
 tools/perf/util/auxtrace.c                    |    4 +
 tools/perf/util/auxtrace.h                    |    1 +
 tools/perf/util/hisi-ptt-decoder/Build        |    1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   |  170 +++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |   28 +
 tools/perf/util/hisi_ptt.c                    |  228 +++
 tools/perf/util/hisi_ptt.h                    |   28 +
 21 files changed, 2356 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/trace/hisi-ptt.rst
 create mode 100644 drivers/hwtracing/hisilicon/Kconfig
 create mode 100644 drivers/hwtracing/hisilicon/Makefile
 create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

-- 
2.33.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Yicong Yang]

Export iommu_{get,put}_resv_regions() to the modules so that the driver
can retrieve and use the reserved regions of the device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/iommu/iommu.c | 2 ++
 include/linux/iommu.h | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index dd7863e453a5..e96711eee965 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2792,6 +2792,7 @@ void iommu_get_resv_regions(struct device *dev, struct list_head *list)
 	if (ops && ops->get_resv_regions)
 		ops->get_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_get_resv_regions);
 
 void iommu_put_resv_regions(struct device *dev, struct list_head *list)
 {
@@ -2800,6 +2801,7 @@ void iommu_put_resv_regions(struct device *dev, struct list_head *list)
 	if (ops && ops->put_resv_regions)
 		ops->put_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_put_resv_regions);
 
 /**
  * generic_iommu_put_resv_regions - Reserved region driver helper
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index d2f3435e7d17..1b7b0f370e28 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -450,8 +450,8 @@ extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t io
 extern void iommu_set_fault_handler(struct iommu_domain *domain,
 			iommu_fault_handler_t handler, void *token);
 
-extern void iommu_get_resv_regions(struct device *dev, struct list_head *list);
-extern void iommu_put_resv_regions(struct device *dev, struct list_head *list);
+void iommu_get_resv_regions(struct device *dev, struct list_head *list);
+void iommu_put_resv_regions(struct device *dev, struct list_head *list);
 extern void generic_iommu_put_resv_regions(struct device *dev,
 					   struct list_head *list);
 extern void iommu_set_default_passthrough(bool cmd_line);
-- 
2.33.0

[PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Yicong Yang via iommu]

Export iommu_{get,put}_resv_regions() to the modules so that the driver
can retrieve and use the reserved regions of the device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/iommu/iommu.c | 2 ++
 include/linux/iommu.h | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index dd7863e453a5..e96711eee965 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2792,6 +2792,7 @@ void iommu_get_resv_regions(struct device *dev, struct list_head *list)
 	if (ops && ops->get_resv_regions)
 		ops->get_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_get_resv_regions);
 
 void iommu_put_resv_regions(struct device *dev, struct list_head *list)
 {
@@ -2800,6 +2801,7 @@ void iommu_put_resv_regions(struct device *dev, struct list_head *list)
 	if (ops && ops->put_resv_regions)
 		ops->put_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_put_resv_regions);
 
 /**
  * generic_iommu_put_resv_regions - Reserved region driver helper
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index d2f3435e7d17..1b7b0f370e28 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -450,8 +450,8 @@ extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t io
 extern void iommu_set_fault_handler(struct iommu_domain *domain,
 			iommu_fault_handler_t handler, void *token);
 
-extern void iommu_get_resv_regions(struct device *dev, struct list_head *list);
-extern void iommu_put_resv_regions(struct device *dev, struct list_head *list);
+void iommu_get_resv_regions(struct device *dev, struct list_head *list);
+void iommu_put_resv_regions(struct device *dev, struct list_head *list);
 extern void generic_iommu_put_resv_regions(struct device *dev,
 					   struct list_head *list);
 extern void iommu_set_default_passthrough(bool cmd_line);
-- 
2.33.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Yicong Yang]

Export iommu_{get,put}_resv_regions() to the modules so that the driver
can retrieve and use the reserved regions of the device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/iommu/iommu.c | 2 ++
 include/linux/iommu.h | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index dd7863e453a5..e96711eee965 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2792,6 +2792,7 @@ void iommu_get_resv_regions(struct device *dev, struct list_head *list)
 	if (ops && ops->get_resv_regions)
 		ops->get_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_get_resv_regions);
 
 void iommu_put_resv_regions(struct device *dev, struct list_head *list)
 {
@@ -2800,6 +2801,7 @@ void iommu_put_resv_regions(struct device *dev, struct list_head *list)
 	if (ops && ops->put_resv_regions)
 		ops->put_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_put_resv_regions);
 
 /**
  * generic_iommu_put_resv_regions - Reserved region driver helper
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index d2f3435e7d17..1b7b0f370e28 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -450,8 +450,8 @@ extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t io
 extern void iommu_set_fault_handler(struct iommu_domain *domain,
 			iommu_fault_handler_t handler, void *token);
 
-extern void iommu_get_resv_regions(struct device *dev, struct list_head *list);
-extern void iommu_put_resv_regions(struct device *dev, struct list_head *list);
+void iommu_get_resv_regions(struct device *dev, struct list_head *list);
+void iommu_put_resv_regions(struct device *dev, struct list_head *list);
 extern void generic_iommu_put_resv_regions(struct device *dev,
 					   struct list_head *list);
 extern void iommu_set_default_passthrough(bool cmd_line);
-- 
2.33.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Mathieu Poirier]

On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
> Export iommu_{get,put}_resv_regions() to the modules so that the driver
> can retrieve and use the reserved regions of the device.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  drivers/iommu/iommu.c | 2 ++
>  include/linux/iommu.h | 4 ++--
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index dd7863e453a5..e96711eee965 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -2792,6 +2792,7 @@ void iommu_get_resv_regions(struct device *dev, struct list_head *list)
>  	if (ops && ops->get_resv_regions)
>  		ops->get_resv_regions(dev, list);
>  }
> +EXPORT_SYMBOL_GPL(iommu_get_resv_regions);
>  
>  void iommu_put_resv_regions(struct device *dev, struct list_head *list)
>  {
> @@ -2800,6 +2801,7 @@ void iommu_put_resv_regions(struct device *dev, struct list_head *list)
>  	if (ops && ops->put_resv_regions)
>  		ops->put_resv_regions(dev, list);
>  }
> +EXPORT_SYMBOL_GPL(iommu_put_resv_regions);
>  
>  /**
>   * generic_iommu_put_resv_regions - Reserved region driver helper
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index d2f3435e7d17..1b7b0f370e28 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -450,8 +450,8 @@ extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t io
>  extern void iommu_set_fault_handler(struct iommu_domain *domain,
>  			iommu_fault_handler_t handler, void *token);
>  
> -extern void iommu_get_resv_regions(struct device *dev, struct list_head *list);
> -extern void iommu_put_resv_regions(struct device *dev, struct list_head *list);
> +void iommu_get_resv_regions(struct device *dev, struct list_head *list);
> +void iommu_put_resv_regions(struct device *dev, struct list_head *list);

Acked-by: Mathieu Poirier <mathieu.poirier@linaro.org>

>  extern void generic_iommu_put_resv_regions(struct device *dev,
>  					   struct list_head *list);
>  extern void iommu_set_default_passthrough(bool cmd_line);
> -- 
> 2.33.0
> 

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Mathieu Poirier]

On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
> Export iommu_{get,put}_resv_regions() to the modules so that the driver
> can retrieve and use the reserved regions of the device.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  drivers/iommu/iommu.c | 2 ++
>  include/linux/iommu.h | 4 ++--
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index dd7863e453a5..e96711eee965 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -2792,6 +2792,7 @@ void iommu_get_resv_regions(struct device *dev, struct list_head *list)
>  	if (ops && ops->get_resv_regions)
>  		ops->get_resv_regions(dev, list);
>  }
> +EXPORT_SYMBOL_GPL(iommu_get_resv_regions);
>  
>  void iommu_put_resv_regions(struct device *dev, struct list_head *list)
>  {
> @@ -2800,6 +2801,7 @@ void iommu_put_resv_regions(struct device *dev, struct list_head *list)
>  	if (ops && ops->put_resv_regions)
>  		ops->put_resv_regions(dev, list);
>  }
> +EXPORT_SYMBOL_GPL(iommu_put_resv_regions);
>  
>  /**
>   * generic_iommu_put_resv_regions - Reserved region driver helper
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index d2f3435e7d17..1b7b0f370e28 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -450,8 +450,8 @@ extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t io
>  extern void iommu_set_fault_handler(struct iommu_domain *domain,
>  			iommu_fault_handler_t handler, void *token);
>  
> -extern void iommu_get_resv_regions(struct device *dev, struct list_head *list);
> -extern void iommu_put_resv_regions(struct device *dev, struct list_head *list);
> +void iommu_get_resv_regions(struct device *dev, struct list_head *list);
> +void iommu_put_resv_regions(struct device *dev, struct list_head *list);

Acked-by: Mathieu Poirier <mathieu.poirier@linaro.org>

>  extern void generic_iommu_put_resv_regions(struct device *dev,
>  					   struct list_head *list);
>  extern void iommu_set_default_passthrough(bool cmd_line);
> -- 
> 2.33.0
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Mathieu Poirier]

On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
> Export iommu_{get,put}_resv_regions() to the modules so that the driver
> can retrieve and use the reserved regions of the device.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  drivers/iommu/iommu.c | 2 ++
>  include/linux/iommu.h | 4 ++--
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index dd7863e453a5..e96711eee965 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -2792,6 +2792,7 @@ void iommu_get_resv_regions(struct device *dev, struct list_head *list)
>  	if (ops && ops->get_resv_regions)
>  		ops->get_resv_regions(dev, list);
>  }
> +EXPORT_SYMBOL_GPL(iommu_get_resv_regions);
>  
>  void iommu_put_resv_regions(struct device *dev, struct list_head *list)
>  {
> @@ -2800,6 +2801,7 @@ void iommu_put_resv_regions(struct device *dev, struct list_head *list)
>  	if (ops && ops->put_resv_regions)
>  		ops->put_resv_regions(dev, list);
>  }
> +EXPORT_SYMBOL_GPL(iommu_put_resv_regions);
>  
>  /**
>   * generic_iommu_put_resv_regions - Reserved region driver helper
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index d2f3435e7d17..1b7b0f370e28 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -450,8 +450,8 @@ extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t io
>  extern void iommu_set_fault_handler(struct iommu_domain *domain,
>  			iommu_fault_handler_t handler, void *token);
>  
> -extern void iommu_get_resv_regions(struct device *dev, struct list_head *list);
> -extern void iommu_put_resv_regions(struct device *dev, struct list_head *list);
> +void iommu_get_resv_regions(struct device *dev, struct list_head *list);
> +void iommu_put_resv_regions(struct device *dev, struct list_head *list);

Acked-by: Mathieu Poirier <mathieu.poirier@linaro.org>

>  extern void generic_iommu_put_resv_regions(struct device *dev,
>  					   struct list_head *list);
>  extern void iommu_set_default_passthrough(bool cmd_line);
> -- 
> 2.33.0
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Joerg Roedel]

On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
> Export iommu_{get,put}_resv_regions() to the modules so that the driver
> can retrieve and use the reserved regions of the device.

Why should any driver bother? These functions are only used by the iommu
core to call into iommu drivers to get information about needed direct
mappings. Why drivers need this information belongs into this commit
message.

Regards,

	Joerg

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Joerg Roedel]

On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
> Export iommu_{get,put}_resv_regions() to the modules so that the driver
> can retrieve and use the reserved regions of the device.

Why should any driver bother? These functions are only used by the iommu
core to call into iommu drivers to get information about needed direct
mappings. Why drivers need this information belongs into this commit
message.

Regards,

	Joerg


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Joerg Roedel]

On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
> Export iommu_{get,put}_resv_regions() to the modules so that the driver
> can retrieve and use the reserved regions of the device.

Why should any driver bother? These functions are only used by the iommu
core to call into iommu drivers to get information about needed direct
mappings. Why drivers need this information belongs into this commit
message.

Regards,

	Joerg

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Yicong Yang]

On 2021/12/6 19:56, Joerg Roedel wrote:
> On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
>> Export iommu_{get,put}_resv_regions() to the modules so that the driver
>> can retrieve and use the reserved regions of the device.
> 
> Why should any driver bother? These functions are only used by the iommu
> core to call into iommu drivers to get information about needed direct
> mappings. Why drivers need this information belongs into this commit
> message.
> 

Our driver makes use of RMR[1], so we need these APIs to retrieve the reserved
memory region in the driver. The device will store the traced data to
the rmr memory and the drivers needs to know the memory address for reading
the data. Due to some hardware limitation, the device can only use direct
mapping for DMA so we need RMR when the iommu is presented and the policy
of the iommu domain is not passthrough.

[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/

Thanks,
Yicong

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Yicong Yang]

On 2021/12/6 19:56, Joerg Roedel wrote:
> On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
>> Export iommu_{get,put}_resv_regions() to the modules so that the driver
>> can retrieve and use the reserved regions of the device.
> 
> Why should any driver bother? These functions are only used by the iommu
> core to call into iommu drivers to get information about needed direct
> mappings. Why drivers need this information belongs into this commit
> message.
> 

Our driver makes use of RMR[1], so we need these APIs to retrieve the reserved
memory region in the driver. The device will store the traced data to
the rmr memory and the drivers needs to know the memory address for reading
the data. Due to some hardware limitation, the device can only use direct
mapping for DMA so we need RMR when the iommu is presented and the policy
of the iommu domain is not passthrough.

[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/

Thanks,
Yicong

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 1/6] iommu: Export iommu_{get,put}_resv_regions()

[Yicong Yang via iommu]

On 2021/12/6 19:56, Joerg Roedel wrote:
> On Tue, Nov 16, 2021 at 05:06:20PM +0800, Yicong Yang wrote:
>> Export iommu_{get,put}_resv_regions() to the modules so that the driver
>> can retrieve and use the reserved regions of the device.
> 
> Why should any driver bother? These functions are only used by the iommu
> core to call into iommu drivers to get information about needed direct
> mappings. Why drivers need this information belongs into this commit
> message.
> 

Our driver makes use of RMR[1], so we need these APIs to retrieve the reserved
memory region in the driver. The device will store the traced data to
the rmr memory and the drivers needs to know the memory address for reading
the data. Due to some hardware limitation, the device can only use direct
mapping for DMA so we need RMR when the iommu is presented and the policy
of the iommu domain is not passthrough.

[1] https://lore.kernel.org/linux-acpi/20210805080724.480-1-shameerali.kolothum.thodi@huawei.com/

Thanks,
Yicong
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
integrated Endpoint(RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic(tune),
and trace the TLP headers(trace).

Add the driver for the device to enable the trace function. The driver
will create PMU device for each PTT device, and users can start trace
through perf command.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/Makefile                       |    1 +
 drivers/hwtracing/Kconfig              |    2 +
 drivers/hwtracing/hisilicon/Kconfig    |    8 +
 drivers/hwtracing/hisilicon/Makefile   |    2 +
 drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
 5 files changed, 1159 insertions(+)
 create mode 100644 drivers/hwtracing/hisilicon/Kconfig
 create mode 100644 drivers/hwtracing/hisilicon/Makefile
 create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c

diff --git a/drivers/Makefile b/drivers/Makefile
index be5d40ae1488..bb0dc9b55ea2 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
 obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
 obj-y				+= hwtracing/intel_th/
 obj-$(CONFIG_STM)		+= hwtracing/stm/
+obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
 obj-$(CONFIG_ANDROID)		+= android/
 obj-$(CONFIG_NVMEM)		+= nvmem/
 obj-$(CONFIG_FPGA)		+= fpga/
diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
index 13085835a636..e3796b17541a 100644
--- a/drivers/hwtracing/Kconfig
+++ b/drivers/hwtracing/Kconfig
@@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
 
 source "drivers/hwtracing/intel_th/Kconfig"
 
+source "drivers/hwtracing/hisilicon/Kconfig"
+
 endmenu
diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
new file mode 100644
index 000000000000..9c3ab80d99fe
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config HISI_PTT
+	tristate "HiSilicon PCIe Tune and Trace Device"
+	depends on PCI && HAS_DMA && HAS_IOMEM
+	help
+	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
+	  device, provides support for PCIe traffic tuning and
+	  tracing TLP headers to the memory.
diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
new file mode 100644
index 000000000000..908c09a98161
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
new file mode 100644
index 000000000000..e11e9b6cc2a8
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
@@ -0,0 +1,1146 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for HiSilicon PCIe tune and trace device
+ *
+ * Copyright (c) 2021 HiSilicon Limited.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/dma-iommu.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iommu.h>
+#include <linux/iopoll.h>
+#include <linux/kfifo.h>
+#include <linux/module.h>
+#include <linux/notifier.h>
+#include <linux/pci.h>
+#include <linux/perf_event.h>
+#include <linux/sizes.h>
+#include <linux/sysfs.h>
+#include <linux/vmalloc.h>
+#include <linux/workqueue.h>
+
+#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
+#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
+#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
+#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
+#define HISI_PTT_TRACE_CTRL		0x0850
+#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
+#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
+#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
+#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
+#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
+#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
+#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
+#define HISI_PTT_TRACE_INT_STAT		0x0890
+#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
+#define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TRACE_WR_STS		0x08a0
+#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
+#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
+#define HISI_PTT_TRACE_STS		0x08b0
+#define   HISI_PTT_TRACE_IDLE		BIT(0)
+#define HISI_PTT_DEVICE_RANGE		0x0fe0
+#define HISI_PTT_LOCATION		0x0fe8
+#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
+#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
+
+#define HISI_PTT_TRACE_DMA_IRQ			0
+#define HISI_PTT_TRACE_BUFLETS_CNT		4
+#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
+#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
+						 HISI_PTT_TRACE_BUFLETS_CNT)
+#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
+
+/* Delay time for filter updating work */
+#define HISI_PTT_WORK_DELAY_MS		100UL
+/* Wait time for DMA hardware to reset */
+#define HISI_PTT_RESET_WAIT_MS		1000UL
+/* Poll timeout and interval for waiting hardware work to finish */
+#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
+#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
+
+#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
+
+enum hisi_ptt_trace_status {
+	HISI_PTT_TRACE_STATUS_OFF = 0,
+	HISI_PTT_TRACE_STATUS_ON,
+};
+
+struct hisi_ptt_dma_buflet {
+	struct list_head list;
+	unsigned int size;
+	dma_addr_t dma;
+
+	/*
+	 * The address of the buflet holding the trace data.
+	 * See Documentation/trace/hisi-ptt.rst for the details
+	 * of the data format.
+	 */
+	void *addr;
+	int index;
+};
+
+struct hisi_ptt_trace_ctrl {
+	enum hisi_ptt_trace_status status;
+	struct perf_output_handle handle;
+	struct list_head trace_buf;
+	/*
+	 * The index of the buffer which trace data
+	 * currently is writing to.
+	 */
+	u32 buf_index;
+
+	int default_cpu;
+	u32 buflet_size;
+	bool is_port;
+	u32 format:1;		/* Format of the traced TLP headers */
+	u32 type:4;		/* Type of the TLP headers to trace */
+	u32 direction:2;	/* Direction of the TLP headers to trace */
+	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
+
+	phys_addr_t rmr_addr;
+	size_t rmr_length;
+	bool has_rmr;
+};
+
+struct hisi_ptt_filter_desc {
+	struct list_head list;
+	struct pci_dev *pdev;
+	u16 val;
+};
+
+struct hisi_ptt_pmu_buf {
+	size_t length;
+	int nr_pages;
+	void *base;
+	long pos;
+};
+
+/* Structure containing the information for filter updating */
+struct hisi_ptt_filter_update_info {
+	struct pci_dev *pdev;
+	u32 port_devid;
+	bool is_port;
+	bool is_add;
+	u16 val;
+};
+
+struct hisi_ptt {
+	struct hisi_ptt_trace_ctrl trace_ctrl;
+	struct notifier_block hisi_ptt_nb;
+	struct pmu hisi_ptt_pmu;
+	void __iomem *iobase;
+	struct pci_dev *pdev;
+	/*
+	 * Use the mutex to protect the filter list and
+	 * serialize the perf process.
+	 */
+	struct mutex mutex;
+	const char *name;
+	u16 core_id;
+	u16 sicl_id;
+	/* PCI device range managed by the PTT device */
+	u32 upper;
+	u32 lower;
+	u8 busnr;
+
+	/*
+	 * The trace TLP headers can either be filtered by certain
+	 * root port, or by the requester ID. Organize the filters
+	 * by @port_filters and @req_filters here. The mask of all
+	 * the valid ports is also cached for doing sanity check
+	 * of user input.
+	 */
+	struct list_head port_filters;
+	struct list_head req_filters;
+	u16 port_mask;
+
+	/*
+	 * We use a delayed work here to avoid indefinitely waiting for
+	 * the hisi_ptt->mutex which protecting the filter list. The
+	 * work will be delayed only if the mutex can not be held,
+	 * otherwise no delay will be applied.
+	 */
+	struct delayed_work work;
+	spinlock_t filter_update_lock;
+	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
+		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
+};
+
+static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
+{
+	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
+}
+
+static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
+{
+	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
+		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
+
+	return PCI_DEVID(pdev->bus->number, pdev->devfn);
+}
+
+static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
+				  val & HISI_PTT_TRACE_IDLE,
+				  HISI_PTT_WAIT_POLL_INTERVAL_US,
+				  HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
+
+	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
+		list_del(&buflet->list);
+
+		if (ctrl->has_rmr)
+			memunmap(buflet->addr);
+		else
+			dma_free_coherent(dev, buflet->size, buflet->addr,
+					  buflet->dma);
+
+		kfree(buflet);
+	}
+}
+
+static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	struct hisi_ptt_dma_buflet *buflet;
+	int i, ret;
+
+	hisi_ptt->trace_ctrl.buf_index = 0;
+
+	/* Make sure the trace buffer is empty before allocating */
+	if (!list_empty(&ctrl->trace_buf)) {
+		list_for_each_entry(buflet, &ctrl->trace_buf, list)
+			memset(buflet->addr, 0, buflet->size);
+		return 0;
+	}
+
+	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
+		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
+		if (!buflet) {
+			ret = -ENOMEM;
+			goto err;
+		}
+
+		if (ctrl->has_rmr) {
+			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
+
+			buflet->dma = base;
+			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
+		} else {
+			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
+							  &buflet->dma, GFP_KERNEL);
+		}
+
+		if (!buflet->addr) {
+			kfree(buflet);
+			ret = -ENOMEM;
+			goto err;
+		}
+
+		memset(buflet->addr, 0, buflet->size);
+
+		buflet->index = i;
+		buflet->size = ctrl->buflet_size;
+		list_add_tail(&buflet->list, &ctrl->trace_buf);
+	}
+
+	return 0;
+err:
+	hisi_ptt_free_trace_buf(hisi_ptt);
+	return ret;
+}
+
+static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
+{
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
+}
+
+static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct hisi_ptt_dma_buflet *cur;
+	u32 val;
+
+	/* Check device idle before start trace */
+	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
+		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
+		return -EBUSY;
+	}
+
+	/* Reset the DMA before start tracing */
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val |= HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	/*
+	 * We'll be in the perf context where preemption is disabled,
+	 * so use busy loop here.
+	 */
+	mdelay(HISI_PTT_RESET_WAIT_MS);
+
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val &= ~HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	/* clear the interrupt status */
+	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
+
+	list_for_each_entry(cur, &ctrl->trace_buf, list) {
+		writel(lower_32_bits(cur->dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
+		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+		writel(upper_32_bits(cur->dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
+		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+	}
+	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
+
+	/* set the trace control register */
+	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
+	if (!hisi_ptt->trace_ctrl.is_port)
+		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
+
+	val |= HISI_PTT_TRACE_CTRL_EN;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
+
+	return 0;
+}
+
+static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
+{
+	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+	struct perf_event *event = handle->event;
+	struct hisi_ptt_dma_buflet *cur;
+	struct hisi_ptt_pmu_buf *buf;
+
+	buf = perf_get_aux(handle);
+	if (!buf || !handle->size)
+		return -EINVAL;
+
+	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
+		if (cur->index == index)
+			break;
+
+	memcpy(buf->base + buf->pos, cur->addr, cur->size);
+	memset(cur->addr, 0, cur->size);
+	buf->pos += cur->size;
+
+	if (stop) {
+		perf_aux_output_end(handle, buf->pos);
+	} else if (buf->length - buf->pos < cur->size) {
+		perf_aux_output_skip(handle, buf->length - buf->pos);
+		perf_aux_output_end(handle, buf->pos);
+
+		buf = perf_aux_output_begin(handle, event);
+		if (!buf)
+			return -EINVAL;
+
+		buf->pos = handle->head % buf->length;
+		if (buf->length - buf->pos < cur->size) {
+			perf_aux_output_end(handle, 0);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static irqreturn_t hisi_ptt_isr(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status, buf_idx;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	buf_idx = ffs(status) - 1;
+
+	/* Clear the interrupt status of buflet @buf_idx */
+	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+
+	/*
+	 * Update the AUX buffer and cache the current buflet index,
+	 * as we need to know this and save the data when the trace
+	 * is ended out of the interrupt handler. End the trace
+	 * if the updating fails.
+	 */
+	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
+		hisi_ptt_trace_end(hisi_ptt);
+	else
+		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
+
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t hisi_ptt_irq(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
+		return IRQ_NONE;
+
+	return IRQ_WAKE_THREAD;
+}
+
+static void hisi_ptt_irq_free_vectors(void *pdev)
+{
+	pci_free_irq_vectors(pdev);
+}
+
+static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	int ret;
+
+	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
+	if (ret < 0) {
+		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
+		return ret;
+	}
+
+	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
+	if (ret < 0)
+		return ret;
+
+	ret = devm_request_threaded_irq(&pdev->dev,
+					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
+					hisi_ptt_irq, hisi_ptt_isr, 0,
+					"hisi-ptt", hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
+			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_update_filters(struct work_struct *work)
+{
+	struct delayed_work *delayed_work = to_delayed_work(work);
+	struct hisi_ptt_filter_update_info info;
+	struct hisi_ptt_filter_desc *filter;
+	struct list_head *target_list;
+	struct hisi_ptt *hisi_ptt;
+
+	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
+
+	if (!mutex_trylock(&hisi_ptt->mutex)) {
+		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
+		return;
+	}
+
+	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
+		target_list = info.is_port ? &hisi_ptt->port_filters :
+			      &hisi_ptt->req_filters;
+
+		if (info.is_add) {
+			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
+			if (!filter)
+				continue;
+
+			filter->pdev = info.pdev;
+			filter->val = info.val;
+
+			list_add_tail(&filter->list, target_list);
+		} else {
+			list_for_each_entry(filter, target_list, list)
+				if (filter->val == info.val) {
+					list_del(&filter->list);
+					kfree(filter);
+					break;
+				}
+		}
+
+		/* Update the available port mask */
+		if (!info.is_port)
+			continue;
+
+		if (info.is_add)
+			hisi_ptt->port_mask |= info.val;
+		else
+			hisi_ptt->port_mask &= ~info.val;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
+				    struct hisi_ptt_filter_update_info *info)
+{
+	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
+
+	if (!root_port)
+		return;
+
+	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
+	if (info->port_devid < hisi_ptt->lower ||
+	    info->port_devid > hisi_ptt->upper)
+		return;
+
+	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
+	info->val = hisi_ptt_get_filter_val(info->pdev);
+
+	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
+				&hisi_ptt->filter_update_lock))
+		schedule_delayed_work(&hisi_ptt->work, 0);
+	else
+		pci_warn(hisi_ptt->pdev,
+			 "filter update fifo overflow for target %s\n",
+			 pci_name(info->pdev));
+}
+
+/*
+ * A PCI bus notifier is used here for dynamically updating the filter
+ * list.
+ */
+static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
+				  void *data)
+{
+	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
+	struct hisi_ptt_filter_update_info info;
+	struct device *dev = data;
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	info.pdev = pdev;
+
+	switch (action) {
+	case BUS_NOTIFY_ADD_DEVICE:
+		info.is_add = true;
+		break;
+	case BUS_NOTIFY_DEL_DEVICE:
+		info.is_add = false;
+		break;
+	default:
+		return 0;
+	}
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+	return 0;
+}
+
+static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+{
+	struct hisi_ptt_filter_update_info info = {
+		.pdev = pdev,
+		.is_add = true,
+	};
+	struct hisi_ptt *hisi_ptt = data;
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+	return 0;
+}
+
+static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_filter_desc *filter, *tfilter;
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+}
+
+static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	struct pci_bus *bus;
+	u32 reg;
+
+	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
+	spin_lock_init(&hisi_ptt->filter_update_lock);
+	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
+	INIT_LIST_HEAD(&hisi_ptt->port_filters);
+	INIT_LIST_HEAD(&hisi_ptt->req_filters);
+
+	/*
+	 * The device range register provides the information about the
+	 * root ports which the RCiEP can control and trace. The RCiEP
+	 * and the root ports it support are on the same PCIe core, with
+	 * same domain number but maybe different bus number. The device
+	 * range register will tell us which root ports we can support,
+	 * Bit[31:16] indicates the upper BDF numbers of the root port,
+	 * while Bit[15:0] indicates the lower.
+	 */
+	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
+	hisi_ptt->upper = reg >> 16;
+	hisi_ptt->lower = reg & 0xffff;
+	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
+	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
+	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
+
+	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
+	if (bus)
+		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
+
+	/* Initialize trace controls */
+	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
+	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
+	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
+}
+
+#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
+#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
+#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
+#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
+#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
+
+static ssize_t available_filters_show(struct device *dev,
+				      struct device_attribute *attr,
+				      char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct hisi_ptt_filter_desc *filter;
+	int pos = 0;
+
+	if (list_empty(&hisi_ptt->port_filters))
+		return sysfs_emit(buf, "#### No available filter ####\n");
+
+	mutex_lock(&hisi_ptt->mutex);
+	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
+	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev) |
+				     HISI_PTT_PMU_FILTER_IS_PORT);
+
+	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
+	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev));
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return pos;
+}
+static DEVICE_ATTR_ADMIN_RO(available_filters);
+
+PMU_FORMAT_ATTR(filter,		"config:0-19");
+PMU_FORMAT_ATTR(direction,	"config:20-23");
+PMU_FORMAT_ATTR(type,		"config:24-31");
+PMU_FORMAT_ATTR(format,		"config:32-35");
+
+static struct attribute *hisi_ptt_pmu_format_attrs[] = {
+	&format_attr_filter.attr,
+	&format_attr_direction.attr,
+	&format_attr_type.attr,
+	&format_attr_format.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_format_group = {
+	.name = "format",
+	.attrs = hisi_ptt_pmu_format_attrs,
+};
+
+static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
+	&dev_attr_available_filters.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_filter_group = {
+	.attrs = hisi_ptt_pmu_filter_attrs,
+};
+
+static const struct attribute_group *hisi_ptt_pmu_groups[] = {
+	&hisi_ptt_pmu_format_group,
+	&hisi_ptt_pmu_filter_group,
+	NULL
+};
+
+/*
+ * The supported value of the direction parameter. See hisi_ptt.rst
+ * documentation for more details.
+ */
+static u32 hisi_ptt_trace_available_direction[] = {
+	0,
+	1,
+	2,
+	3,
+};
+
+/* Different types can be set simultaneously */
+static u32 hisi_ptt_trace_available_type[] = {
+	1,	/* posted_request */
+	2,	/* non-posted_request */
+	4,	/* completion */
+};
+
+static u32 hisi_ptt_trace_availble_format[] = {
+	0,	/* 4DW */
+	1,	/* 8DW */
+};
+
+/*
+ * Check whether the config is valid or not. Some configs are multi-selectable
+ * and can be set simultaneously, while some are single selectable (onehot).
+ * Use this function to check the non-onehot configs while
+ * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
+ */
+static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
+{
+	int i;
+
+	/*
+	 * The non-onehot configs cannot be 0. Walk the available
+	 * list and clear the valid bits of the config. If there
+	 * is any resident bit after the walk then the config is
+	 * invalid.
+	 */
+	for (i = 0; i < list_size; i++)
+		val &= ~available_list[i];
+
+	return val ? -EINVAL : 0;
+}
+
+static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
+{
+	int i, ret = -EINVAL;
+
+	for (i = 0; i < list_size; i++)
+		if (val == available_list[i]) {
+			ret = 0;
+			break;
+		}
+
+	return ret;
+}
+
+static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
+{
+	unsigned long val, port_mask = hisi_ptt->port_mask;
+	struct hisi_ptt_filter_desc *filter;
+	int ret = -EINVAL;
+
+	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
+	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
+
+	/*
+	 * Port filters are defined as bit mask. For port filters, check
+	 * the bits in the @val are within the range of hisi_ptt->port_mask
+	 * and whether it's empty or not, otherwise user has specified
+	 * some unsupported root ports.
+	 *
+	 * For Requester ID filters, walk the available filter list to see
+	 * whether we have one matched.
+	 */
+	if (hisi_ptt->trace_ctrl.is_port &&
+	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
+		ret = 0;
+	} else {
+		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+			if (filter->val == val) {
+				ret = 0;
+				break;
+			}
+	}
+
+	if (ret)
+		return ret;
+
+	hisi_ptt->trace_ctrl.filter = val;
+	return 0;
+}
+
+static int hisi_ptt_pmu_event_init(struct perf_event *event)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	int ret;
+	u32 val;
+
+	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
+		return -ENOENT;
+
+	mutex_lock(&hisi_ptt->mutex);
+
+	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
+	if (ret < 0)
+		goto out;
+
+	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
+						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
+	if (ret < 0)
+		goto out;
+	ctrl->direction = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
+
+	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
+					  ARRAY_SIZE(hisi_ptt_trace_available_type));
+	if (ret < 0)
+		goto out;
+	ctrl->type = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
+						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
+	if (ret < 0)
+		goto out;
+	ctrl->format = val;
+
+out:
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret;
+}
+
+static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
+				    int nr_pages, bool overwrite)
+{
+	struct hisi_ptt_pmu_buf *buf;
+	struct page **pagelist;
+	int i;
+
+	if (overwrite) {
+		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
+		return NULL;
+	}
+
+	/* If the pages size less than buflets, we cannot start trace */
+	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
+		return NULL;
+
+	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
+	if (!buf)
+		return NULL;
+
+	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
+	if (!pagelist) {
+		kfree(buf);
+		return NULL;
+	}
+
+	for (i = 0; i < nr_pages; i++)
+		pagelist[i] = virt_to_page(pages[i]);
+
+	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
+	if (!buf->base) {
+		kfree(pagelist);
+		kfree(buf);
+		return NULL;
+	}
+
+	buf->nr_pages = nr_pages;
+	buf->length = nr_pages * PAGE_SIZE;
+	buf->pos = 0;
+
+	kfree(pagelist);
+	return buf;
+}
+
+static void hisi_ptt_pmu_free_aux(void *aux)
+{
+	struct hisi_ptt_pmu_buf *buf = aux;
+
+	vunmap(buf->base);
+	kfree(buf);
+}
+
+static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+	struct hw_perf_event *hwc = &event->hw;
+	struct hisi_ptt_pmu_buf *buf;
+	int cpu = event->cpu;
+	int ret;
+
+	hwc->state = 0;
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
+		goto stop;
+	}
+
+	if (cpu == -1)
+		cpu = hisi_ptt->trace_ctrl.default_cpu;
+
+	/*
+	 * Handle the interrupt on the same cpu which starts the trace to avoid
+	 * context mismatch. Otherwise we'll trigger the WARN from the perf
+	 * core in event_function_local().
+	 */
+	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
+				 cpumask_of(cpu)));
+
+	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
+		goto stop;
+	}
+
+	buf = perf_aux_output_begin(handle, event);
+	if (!buf) {
+		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
+		goto stop;
+	}
+
+	buf->pos = handle->head % buf->length;
+
+	ret = hisi_ptt_trace_start(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
+		perf_aux_output_end(handle, 0);
+		goto stop;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return;
+stop:
+	event->hw.state |= PERF_HES_STOPPED;
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+
+	if (hwc->state & PERF_HES_STOPPED)
+		return;
+
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+		hisi_ptt_trace_end(hisi_ptt);
+		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
+		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
+	}
+	mutex_unlock(&hisi_ptt->mutex);
+
+	hwc->state |= PERF_HES_STOPPED;
+	perf_event_update_userpage(event);
+	hwc->state |= PERF_HES_UPTODATE;
+}
+
+static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+	int cpu = event->cpu;
+
+	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
+		return 0;
+
+	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+
+	if (flags & PERF_EF_START) {
+		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
+		if (hwc->state & PERF_HES_STOPPED)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
+{
+	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
+}
+
+static void hisi_ptt_unregister_pmu(void *priv)
+{
+	perf_pmu_unregister(priv);
+}
+
+static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
+{
+	char *pmu_name;
+	int ret;
+
+	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
+		.module		= THIS_MODULE,
+		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
+		.task_ctx_nr	= perf_sw_context,
+		.attr_groups	= hisi_ptt_pmu_groups,
+		.event_init	= hisi_ptt_pmu_event_init,
+		.setup_aux	= hisi_ptt_pmu_setup_aux,
+		.free_aux	= hisi_ptt_pmu_free_aux,
+		.start		= hisi_ptt_pmu_start,
+		.stop		= hisi_ptt_pmu_stop,
+		.add		= hisi_ptt_pmu_add,
+		.del		= hisi_ptt_pmu_del,
+	};
+
+	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
+				  hisi_ptt->sicl_id, hisi_ptt->core_id);
+	if (!pmu_name)
+		return -ENOMEM;
+
+	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
+	if (ret)
+		return ret;
+
+	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
+					hisi_ptt_unregister_pmu,
+					&hisi_ptt->hisi_ptt_pmu);
+}
+
+/*
+ * Get RMR address if provided by the firmware.
+ * Return 0 if the IOMMU doesn't present or the policy of the
+ * IOMMU domain is passthrough or we get a usable RMR region.
+ * Otherwise a negative value is returned.
+ */
+static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	struct iommu_domain *iommu_domain;
+	struct iommu_resv_region *region;
+	LIST_HEAD(list);
+
+	/*
+	 * Use direct DMA if IOMMU does not present or the policy of the
+	 * IOMMU domain is passthrough.
+	 */
+	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
+	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
+		return 0;
+
+	iommu_get_resv_regions(&pdev->dev, &list);
+	list_for_each_entry(region, &list, list)
+		if (region->type == IOMMU_RESV_DIRECT &&
+		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
+			hisi_ptt->trace_ctrl.has_rmr = true;
+			hisi_ptt->trace_ctrl.rmr_addr = region->start;
+			hisi_ptt->trace_ctrl.rmr_length = region->length;
+			break;
+		}
+
+	iommu_put_resv_regions(&pdev->dev, &list);
+	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
+}
+
+static int hisi_ptt_probe(struct pci_dev *pdev,
+			  const struct pci_device_id *id)
+{
+	struct hisi_ptt *hisi_ptt;
+	int ret;
+
+	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
+	if (!hisi_ptt)
+		return -ENOMEM;
+
+	mutex_init(&hisi_ptt->mutex);
+	hisi_ptt->pdev = pdev;
+
+	/*
+	 * Lifetime of pci_dev is longer than hisi_ptt,
+	 * so directly reference to the pci name string.
+	 */
+	hisi_ptt->name = pci_name(hisi_ptt->pdev);
+	pci_set_drvdata(pdev, hisi_ptt);
+
+	ret = pcim_enable_device(pdev);
+	if (ret) {
+		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
+		return ret;
+	}
+
+	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
+	if (ret) {
+		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
+		return ret;
+	}
+
+	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
+
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret) {
+		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
+		return ret;
+	}
+	pci_set_master(pdev);
+
+	ret = hisi_ptt_register_irq(hisi_ptt);
+	if (ret)
+		return ret;
+
+	hisi_ptt_init_ctrls(hisi_ptt);
+
+	ret = hisi_ptt_register_pmu(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
+		return ret;
+	}
+
+	ret = hisi_ptt_get_rmr(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
+		return ret;
+	}
+
+	/* Register the bus notifier for dynamically updating the filter list */
+	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
+	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+	if (ret)
+		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
+
+	return 0;
+}
+
+void hisi_ptt_remove(struct pci_dev *pdev)
+{
+	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
+
+	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+
+	/* Cancel any work that has been queued */
+	cancel_delayed_work_sync(&hisi_ptt->work);
+
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
+		hisi_ptt_trace_end(hisi_ptt);
+
+	hisi_ptt_free_trace_buf(hisi_ptt);
+	hisi_ptt_release_filters(hisi_ptt);
+}
+
+static const struct pci_device_id hisi_ptt_id_tbl[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
+	{ }
+};
+MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
+
+static struct pci_driver hisi_ptt_driver = {
+	.name = "hisi_ptt",
+	.id_table = hisi_ptt_id_tbl,
+	.probe = hisi_ptt_probe,
+	.remove = hisi_ptt_remove,
+};
+module_pci_driver(hisi_ptt_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
+MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
-- 
2.33.0

[PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
integrated Endpoint(RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic(tune),
and trace the TLP headers(trace).

Add the driver for the device to enable the trace function. The driver
will create PMU device for each PTT device, and users can start trace
through perf command.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/Makefile                       |    1 +
 drivers/hwtracing/Kconfig              |    2 +
 drivers/hwtracing/hisilicon/Kconfig    |    8 +
 drivers/hwtracing/hisilicon/Makefile   |    2 +
 drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
 5 files changed, 1159 insertions(+)
 create mode 100644 drivers/hwtracing/hisilicon/Kconfig
 create mode 100644 drivers/hwtracing/hisilicon/Makefile
 create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c

diff --git a/drivers/Makefile b/drivers/Makefile
index be5d40ae1488..bb0dc9b55ea2 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
 obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
 obj-y				+= hwtracing/intel_th/
 obj-$(CONFIG_STM)		+= hwtracing/stm/
+obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
 obj-$(CONFIG_ANDROID)		+= android/
 obj-$(CONFIG_NVMEM)		+= nvmem/
 obj-$(CONFIG_FPGA)		+= fpga/
diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
index 13085835a636..e3796b17541a 100644
--- a/drivers/hwtracing/Kconfig
+++ b/drivers/hwtracing/Kconfig
@@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
 
 source "drivers/hwtracing/intel_th/Kconfig"
 
+source "drivers/hwtracing/hisilicon/Kconfig"
+
 endmenu
diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
new file mode 100644
index 000000000000..9c3ab80d99fe
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config HISI_PTT
+	tristate "HiSilicon PCIe Tune and Trace Device"
+	depends on PCI && HAS_DMA && HAS_IOMEM
+	help
+	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
+	  device, provides support for PCIe traffic tuning and
+	  tracing TLP headers to the memory.
diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
new file mode 100644
index 000000000000..908c09a98161
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
new file mode 100644
index 000000000000..e11e9b6cc2a8
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
@@ -0,0 +1,1146 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for HiSilicon PCIe tune and trace device
+ *
+ * Copyright (c) 2021 HiSilicon Limited.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/dma-iommu.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iommu.h>
+#include <linux/iopoll.h>
+#include <linux/kfifo.h>
+#include <linux/module.h>
+#include <linux/notifier.h>
+#include <linux/pci.h>
+#include <linux/perf_event.h>
+#include <linux/sizes.h>
+#include <linux/sysfs.h>
+#include <linux/vmalloc.h>
+#include <linux/workqueue.h>
+
+#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
+#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
+#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
+#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
+#define HISI_PTT_TRACE_CTRL		0x0850
+#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
+#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
+#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
+#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
+#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
+#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
+#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
+#define HISI_PTT_TRACE_INT_STAT		0x0890
+#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
+#define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TRACE_WR_STS		0x08a0
+#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
+#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
+#define HISI_PTT_TRACE_STS		0x08b0
+#define   HISI_PTT_TRACE_IDLE		BIT(0)
+#define HISI_PTT_DEVICE_RANGE		0x0fe0
+#define HISI_PTT_LOCATION		0x0fe8
+#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
+#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
+
+#define HISI_PTT_TRACE_DMA_IRQ			0
+#define HISI_PTT_TRACE_BUFLETS_CNT		4
+#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
+#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
+						 HISI_PTT_TRACE_BUFLETS_CNT)
+#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
+
+/* Delay time for filter updating work */
+#define HISI_PTT_WORK_DELAY_MS		100UL
+/* Wait time for DMA hardware to reset */
+#define HISI_PTT_RESET_WAIT_MS		1000UL
+/* Poll timeout and interval for waiting hardware work to finish */
+#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
+#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
+
+#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
+
+enum hisi_ptt_trace_status {
+	HISI_PTT_TRACE_STATUS_OFF = 0,
+	HISI_PTT_TRACE_STATUS_ON,
+};
+
+struct hisi_ptt_dma_buflet {
+	struct list_head list;
+	unsigned int size;
+	dma_addr_t dma;
+
+	/*
+	 * The address of the buflet holding the trace data.
+	 * See Documentation/trace/hisi-ptt.rst for the details
+	 * of the data format.
+	 */
+	void *addr;
+	int index;
+};
+
+struct hisi_ptt_trace_ctrl {
+	enum hisi_ptt_trace_status status;
+	struct perf_output_handle handle;
+	struct list_head trace_buf;
+	/*
+	 * The index of the buffer which trace data
+	 * currently is writing to.
+	 */
+	u32 buf_index;
+
+	int default_cpu;
+	u32 buflet_size;
+	bool is_port;
+	u32 format:1;		/* Format of the traced TLP headers */
+	u32 type:4;		/* Type of the TLP headers to trace */
+	u32 direction:2;	/* Direction of the TLP headers to trace */
+	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
+
+	phys_addr_t rmr_addr;
+	size_t rmr_length;
+	bool has_rmr;
+};
+
+struct hisi_ptt_filter_desc {
+	struct list_head list;
+	struct pci_dev *pdev;
+	u16 val;
+};
+
+struct hisi_ptt_pmu_buf {
+	size_t length;
+	int nr_pages;
+	void *base;
+	long pos;
+};
+
+/* Structure containing the information for filter updating */
+struct hisi_ptt_filter_update_info {
+	struct pci_dev *pdev;
+	u32 port_devid;
+	bool is_port;
+	bool is_add;
+	u16 val;
+};
+
+struct hisi_ptt {
+	struct hisi_ptt_trace_ctrl trace_ctrl;
+	struct notifier_block hisi_ptt_nb;
+	struct pmu hisi_ptt_pmu;
+	void __iomem *iobase;
+	struct pci_dev *pdev;
+	/*
+	 * Use the mutex to protect the filter list and
+	 * serialize the perf process.
+	 */
+	struct mutex mutex;
+	const char *name;
+	u16 core_id;
+	u16 sicl_id;
+	/* PCI device range managed by the PTT device */
+	u32 upper;
+	u32 lower;
+	u8 busnr;
+
+	/*
+	 * The trace TLP headers can either be filtered by certain
+	 * root port, or by the requester ID. Organize the filters
+	 * by @port_filters and @req_filters here. The mask of all
+	 * the valid ports is also cached for doing sanity check
+	 * of user input.
+	 */
+	struct list_head port_filters;
+	struct list_head req_filters;
+	u16 port_mask;
+
+	/*
+	 * We use a delayed work here to avoid indefinitely waiting for
+	 * the hisi_ptt->mutex which protecting the filter list. The
+	 * work will be delayed only if the mutex can not be held,
+	 * otherwise no delay will be applied.
+	 */
+	struct delayed_work work;
+	spinlock_t filter_update_lock;
+	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
+		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
+};
+
+static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
+{
+	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
+}
+
+static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
+{
+	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
+		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
+
+	return PCI_DEVID(pdev->bus->number, pdev->devfn);
+}
+
+static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
+				  val & HISI_PTT_TRACE_IDLE,
+				  HISI_PTT_WAIT_POLL_INTERVAL_US,
+				  HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
+
+	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
+		list_del(&buflet->list);
+
+		if (ctrl->has_rmr)
+			memunmap(buflet->addr);
+		else
+			dma_free_coherent(dev, buflet->size, buflet->addr,
+					  buflet->dma);
+
+		kfree(buflet);
+	}
+}
+
+static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	struct hisi_ptt_dma_buflet *buflet;
+	int i, ret;
+
+	hisi_ptt->trace_ctrl.buf_index = 0;
+
+	/* Make sure the trace buffer is empty before allocating */
+	if (!list_empty(&ctrl->trace_buf)) {
+		list_for_each_entry(buflet, &ctrl->trace_buf, list)
+			memset(buflet->addr, 0, buflet->size);
+		return 0;
+	}
+
+	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
+		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
+		if (!buflet) {
+			ret = -ENOMEM;
+			goto err;
+		}
+
+		if (ctrl->has_rmr) {
+			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
+
+			buflet->dma = base;
+			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
+		} else {
+			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
+							  &buflet->dma, GFP_KERNEL);
+		}
+
+		if (!buflet->addr) {
+			kfree(buflet);
+			ret = -ENOMEM;
+			goto err;
+		}
+
+		memset(buflet->addr, 0, buflet->size);
+
+		buflet->index = i;
+		buflet->size = ctrl->buflet_size;
+		list_add_tail(&buflet->list, &ctrl->trace_buf);
+	}
+
+	return 0;
+err:
+	hisi_ptt_free_trace_buf(hisi_ptt);
+	return ret;
+}
+
+static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
+{
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
+}
+
+static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct hisi_ptt_dma_buflet *cur;
+	u32 val;
+
+	/* Check device idle before start trace */
+	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
+		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
+		return -EBUSY;
+	}
+
+	/* Reset the DMA before start tracing */
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val |= HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	/*
+	 * We'll be in the perf context where preemption is disabled,
+	 * so use busy loop here.
+	 */
+	mdelay(HISI_PTT_RESET_WAIT_MS);
+
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val &= ~HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	/* clear the interrupt status */
+	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
+
+	list_for_each_entry(cur, &ctrl->trace_buf, list) {
+		writel(lower_32_bits(cur->dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
+		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+		writel(upper_32_bits(cur->dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
+		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+	}
+	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
+
+	/* set the trace control register */
+	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
+	if (!hisi_ptt->trace_ctrl.is_port)
+		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
+
+	val |= HISI_PTT_TRACE_CTRL_EN;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
+
+	return 0;
+}
+
+static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
+{
+	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+	struct perf_event *event = handle->event;
+	struct hisi_ptt_dma_buflet *cur;
+	struct hisi_ptt_pmu_buf *buf;
+
+	buf = perf_get_aux(handle);
+	if (!buf || !handle->size)
+		return -EINVAL;
+
+	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
+		if (cur->index == index)
+			break;
+
+	memcpy(buf->base + buf->pos, cur->addr, cur->size);
+	memset(cur->addr, 0, cur->size);
+	buf->pos += cur->size;
+
+	if (stop) {
+		perf_aux_output_end(handle, buf->pos);
+	} else if (buf->length - buf->pos < cur->size) {
+		perf_aux_output_skip(handle, buf->length - buf->pos);
+		perf_aux_output_end(handle, buf->pos);
+
+		buf = perf_aux_output_begin(handle, event);
+		if (!buf)
+			return -EINVAL;
+
+		buf->pos = handle->head % buf->length;
+		if (buf->length - buf->pos < cur->size) {
+			perf_aux_output_end(handle, 0);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static irqreturn_t hisi_ptt_isr(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status, buf_idx;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	buf_idx = ffs(status) - 1;
+
+	/* Clear the interrupt status of buflet @buf_idx */
+	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+
+	/*
+	 * Update the AUX buffer and cache the current buflet index,
+	 * as we need to know this and save the data when the trace
+	 * is ended out of the interrupt handler. End the trace
+	 * if the updating fails.
+	 */
+	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
+		hisi_ptt_trace_end(hisi_ptt);
+	else
+		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
+
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t hisi_ptt_irq(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
+		return IRQ_NONE;
+
+	return IRQ_WAKE_THREAD;
+}
+
+static void hisi_ptt_irq_free_vectors(void *pdev)
+{
+	pci_free_irq_vectors(pdev);
+}
+
+static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	int ret;
+
+	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
+	if (ret < 0) {
+		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
+		return ret;
+	}
+
+	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
+	if (ret < 0)
+		return ret;
+
+	ret = devm_request_threaded_irq(&pdev->dev,
+					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
+					hisi_ptt_irq, hisi_ptt_isr, 0,
+					"hisi-ptt", hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
+			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_update_filters(struct work_struct *work)
+{
+	struct delayed_work *delayed_work = to_delayed_work(work);
+	struct hisi_ptt_filter_update_info info;
+	struct hisi_ptt_filter_desc *filter;
+	struct list_head *target_list;
+	struct hisi_ptt *hisi_ptt;
+
+	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
+
+	if (!mutex_trylock(&hisi_ptt->mutex)) {
+		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
+		return;
+	}
+
+	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
+		target_list = info.is_port ? &hisi_ptt->port_filters :
+			      &hisi_ptt->req_filters;
+
+		if (info.is_add) {
+			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
+			if (!filter)
+				continue;
+
+			filter->pdev = info.pdev;
+			filter->val = info.val;
+
+			list_add_tail(&filter->list, target_list);
+		} else {
+			list_for_each_entry(filter, target_list, list)
+				if (filter->val == info.val) {
+					list_del(&filter->list);
+					kfree(filter);
+					break;
+				}
+		}
+
+		/* Update the available port mask */
+		if (!info.is_port)
+			continue;
+
+		if (info.is_add)
+			hisi_ptt->port_mask |= info.val;
+		else
+			hisi_ptt->port_mask &= ~info.val;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
+				    struct hisi_ptt_filter_update_info *info)
+{
+	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
+
+	if (!root_port)
+		return;
+
+	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
+	if (info->port_devid < hisi_ptt->lower ||
+	    info->port_devid > hisi_ptt->upper)
+		return;
+
+	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
+	info->val = hisi_ptt_get_filter_val(info->pdev);
+
+	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
+				&hisi_ptt->filter_update_lock))
+		schedule_delayed_work(&hisi_ptt->work, 0);
+	else
+		pci_warn(hisi_ptt->pdev,
+			 "filter update fifo overflow for target %s\n",
+			 pci_name(info->pdev));
+}
+
+/*
+ * A PCI bus notifier is used here for dynamically updating the filter
+ * list.
+ */
+static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
+				  void *data)
+{
+	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
+	struct hisi_ptt_filter_update_info info;
+	struct device *dev = data;
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	info.pdev = pdev;
+
+	switch (action) {
+	case BUS_NOTIFY_ADD_DEVICE:
+		info.is_add = true;
+		break;
+	case BUS_NOTIFY_DEL_DEVICE:
+		info.is_add = false;
+		break;
+	default:
+		return 0;
+	}
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+	return 0;
+}
+
+static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+{
+	struct hisi_ptt_filter_update_info info = {
+		.pdev = pdev,
+		.is_add = true,
+	};
+	struct hisi_ptt *hisi_ptt = data;
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+	return 0;
+}
+
+static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_filter_desc *filter, *tfilter;
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+}
+
+static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	struct pci_bus *bus;
+	u32 reg;
+
+	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
+	spin_lock_init(&hisi_ptt->filter_update_lock);
+	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
+	INIT_LIST_HEAD(&hisi_ptt->port_filters);
+	INIT_LIST_HEAD(&hisi_ptt->req_filters);
+
+	/*
+	 * The device range register provides the information about the
+	 * root ports which the RCiEP can control and trace. The RCiEP
+	 * and the root ports it support are on the same PCIe core, with
+	 * same domain number but maybe different bus number. The device
+	 * range register will tell us which root ports we can support,
+	 * Bit[31:16] indicates the upper BDF numbers of the root port,
+	 * while Bit[15:0] indicates the lower.
+	 */
+	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
+	hisi_ptt->upper = reg >> 16;
+	hisi_ptt->lower = reg & 0xffff;
+	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
+	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
+	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
+
+	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
+	if (bus)
+		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
+
+	/* Initialize trace controls */
+	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
+	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
+	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
+}
+
+#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
+#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
+#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
+#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
+#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
+
+static ssize_t available_filters_show(struct device *dev,
+				      struct device_attribute *attr,
+				      char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct hisi_ptt_filter_desc *filter;
+	int pos = 0;
+
+	if (list_empty(&hisi_ptt->port_filters))
+		return sysfs_emit(buf, "#### No available filter ####\n");
+
+	mutex_lock(&hisi_ptt->mutex);
+	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
+	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev) |
+				     HISI_PTT_PMU_FILTER_IS_PORT);
+
+	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
+	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev));
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return pos;
+}
+static DEVICE_ATTR_ADMIN_RO(available_filters);
+
+PMU_FORMAT_ATTR(filter,		"config:0-19");
+PMU_FORMAT_ATTR(direction,	"config:20-23");
+PMU_FORMAT_ATTR(type,		"config:24-31");
+PMU_FORMAT_ATTR(format,		"config:32-35");
+
+static struct attribute *hisi_ptt_pmu_format_attrs[] = {
+	&format_attr_filter.attr,
+	&format_attr_direction.attr,
+	&format_attr_type.attr,
+	&format_attr_format.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_format_group = {
+	.name = "format",
+	.attrs = hisi_ptt_pmu_format_attrs,
+};
+
+static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
+	&dev_attr_available_filters.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_filter_group = {
+	.attrs = hisi_ptt_pmu_filter_attrs,
+};
+
+static const struct attribute_group *hisi_ptt_pmu_groups[] = {
+	&hisi_ptt_pmu_format_group,
+	&hisi_ptt_pmu_filter_group,
+	NULL
+};
+
+/*
+ * The supported value of the direction parameter. See hisi_ptt.rst
+ * documentation for more details.
+ */
+static u32 hisi_ptt_trace_available_direction[] = {
+	0,
+	1,
+	2,
+	3,
+};
+
+/* Different types can be set simultaneously */
+static u32 hisi_ptt_trace_available_type[] = {
+	1,	/* posted_request */
+	2,	/* non-posted_request */
+	4,	/* completion */
+};
+
+static u32 hisi_ptt_trace_availble_format[] = {
+	0,	/* 4DW */
+	1,	/* 8DW */
+};
+
+/*
+ * Check whether the config is valid or not. Some configs are multi-selectable
+ * and can be set simultaneously, while some are single selectable (onehot).
+ * Use this function to check the non-onehot configs while
+ * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
+ */
+static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
+{
+	int i;
+
+	/*
+	 * The non-onehot configs cannot be 0. Walk the available
+	 * list and clear the valid bits of the config. If there
+	 * is any resident bit after the walk then the config is
+	 * invalid.
+	 */
+	for (i = 0; i < list_size; i++)
+		val &= ~available_list[i];
+
+	return val ? -EINVAL : 0;
+}
+
+static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
+{
+	int i, ret = -EINVAL;
+
+	for (i = 0; i < list_size; i++)
+		if (val == available_list[i]) {
+			ret = 0;
+			break;
+		}
+
+	return ret;
+}
+
+static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
+{
+	unsigned long val, port_mask = hisi_ptt->port_mask;
+	struct hisi_ptt_filter_desc *filter;
+	int ret = -EINVAL;
+
+	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
+	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
+
+	/*
+	 * Port filters are defined as bit mask. For port filters, check
+	 * the bits in the @val are within the range of hisi_ptt->port_mask
+	 * and whether it's empty or not, otherwise user has specified
+	 * some unsupported root ports.
+	 *
+	 * For Requester ID filters, walk the available filter list to see
+	 * whether we have one matched.
+	 */
+	if (hisi_ptt->trace_ctrl.is_port &&
+	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
+		ret = 0;
+	} else {
+		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+			if (filter->val == val) {
+				ret = 0;
+				break;
+			}
+	}
+
+	if (ret)
+		return ret;
+
+	hisi_ptt->trace_ctrl.filter = val;
+	return 0;
+}
+
+static int hisi_ptt_pmu_event_init(struct perf_event *event)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	int ret;
+	u32 val;
+
+	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
+		return -ENOENT;
+
+	mutex_lock(&hisi_ptt->mutex);
+
+	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
+	if (ret < 0)
+		goto out;
+
+	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
+						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
+	if (ret < 0)
+		goto out;
+	ctrl->direction = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
+
+	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
+					  ARRAY_SIZE(hisi_ptt_trace_available_type));
+	if (ret < 0)
+		goto out;
+	ctrl->type = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
+						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
+	if (ret < 0)
+		goto out;
+	ctrl->format = val;
+
+out:
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret;
+}
+
+static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
+				    int nr_pages, bool overwrite)
+{
+	struct hisi_ptt_pmu_buf *buf;
+	struct page **pagelist;
+	int i;
+
+	if (overwrite) {
+		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
+		return NULL;
+	}
+
+	/* If the pages size less than buflets, we cannot start trace */
+	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
+		return NULL;
+
+	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
+	if (!buf)
+		return NULL;
+
+	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
+	if (!pagelist) {
+		kfree(buf);
+		return NULL;
+	}
+
+	for (i = 0; i < nr_pages; i++)
+		pagelist[i] = virt_to_page(pages[i]);
+
+	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
+	if (!buf->base) {
+		kfree(pagelist);
+		kfree(buf);
+		return NULL;
+	}
+
+	buf->nr_pages = nr_pages;
+	buf->length = nr_pages * PAGE_SIZE;
+	buf->pos = 0;
+
+	kfree(pagelist);
+	return buf;
+}
+
+static void hisi_ptt_pmu_free_aux(void *aux)
+{
+	struct hisi_ptt_pmu_buf *buf = aux;
+
+	vunmap(buf->base);
+	kfree(buf);
+}
+
+static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+	struct hw_perf_event *hwc = &event->hw;
+	struct hisi_ptt_pmu_buf *buf;
+	int cpu = event->cpu;
+	int ret;
+
+	hwc->state = 0;
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
+		goto stop;
+	}
+
+	if (cpu == -1)
+		cpu = hisi_ptt->trace_ctrl.default_cpu;
+
+	/*
+	 * Handle the interrupt on the same cpu which starts the trace to avoid
+	 * context mismatch. Otherwise we'll trigger the WARN from the perf
+	 * core in event_function_local().
+	 */
+	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
+				 cpumask_of(cpu)));
+
+	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
+		goto stop;
+	}
+
+	buf = perf_aux_output_begin(handle, event);
+	if (!buf) {
+		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
+		goto stop;
+	}
+
+	buf->pos = handle->head % buf->length;
+
+	ret = hisi_ptt_trace_start(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
+		perf_aux_output_end(handle, 0);
+		goto stop;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return;
+stop:
+	event->hw.state |= PERF_HES_STOPPED;
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+
+	if (hwc->state & PERF_HES_STOPPED)
+		return;
+
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+		hisi_ptt_trace_end(hisi_ptt);
+		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
+		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
+	}
+	mutex_unlock(&hisi_ptt->mutex);
+
+	hwc->state |= PERF_HES_STOPPED;
+	perf_event_update_userpage(event);
+	hwc->state |= PERF_HES_UPTODATE;
+}
+
+static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+	int cpu = event->cpu;
+
+	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
+		return 0;
+
+	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+
+	if (flags & PERF_EF_START) {
+		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
+		if (hwc->state & PERF_HES_STOPPED)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
+{
+	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
+}
+
+static void hisi_ptt_unregister_pmu(void *priv)
+{
+	perf_pmu_unregister(priv);
+}
+
+static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
+{
+	char *pmu_name;
+	int ret;
+
+	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
+		.module		= THIS_MODULE,
+		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
+		.task_ctx_nr	= perf_sw_context,
+		.attr_groups	= hisi_ptt_pmu_groups,
+		.event_init	= hisi_ptt_pmu_event_init,
+		.setup_aux	= hisi_ptt_pmu_setup_aux,
+		.free_aux	= hisi_ptt_pmu_free_aux,
+		.start		= hisi_ptt_pmu_start,
+		.stop		= hisi_ptt_pmu_stop,
+		.add		= hisi_ptt_pmu_add,
+		.del		= hisi_ptt_pmu_del,
+	};
+
+	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
+				  hisi_ptt->sicl_id, hisi_ptt->core_id);
+	if (!pmu_name)
+		return -ENOMEM;
+
+	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
+	if (ret)
+		return ret;
+
+	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
+					hisi_ptt_unregister_pmu,
+					&hisi_ptt->hisi_ptt_pmu);
+}
+
+/*
+ * Get RMR address if provided by the firmware.
+ * Return 0 if the IOMMU doesn't present or the policy of the
+ * IOMMU domain is passthrough or we get a usable RMR region.
+ * Otherwise a negative value is returned.
+ */
+static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	struct iommu_domain *iommu_domain;
+	struct iommu_resv_region *region;
+	LIST_HEAD(list);
+
+	/*
+	 * Use direct DMA if IOMMU does not present or the policy of the
+	 * IOMMU domain is passthrough.
+	 */
+	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
+	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
+		return 0;
+
+	iommu_get_resv_regions(&pdev->dev, &list);
+	list_for_each_entry(region, &list, list)
+		if (region->type == IOMMU_RESV_DIRECT &&
+		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
+			hisi_ptt->trace_ctrl.has_rmr = true;
+			hisi_ptt->trace_ctrl.rmr_addr = region->start;
+			hisi_ptt->trace_ctrl.rmr_length = region->length;
+			break;
+		}
+
+	iommu_put_resv_regions(&pdev->dev, &list);
+	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
+}
+
+static int hisi_ptt_probe(struct pci_dev *pdev,
+			  const struct pci_device_id *id)
+{
+	struct hisi_ptt *hisi_ptt;
+	int ret;
+
+	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
+	if (!hisi_ptt)
+		return -ENOMEM;
+
+	mutex_init(&hisi_ptt->mutex);
+	hisi_ptt->pdev = pdev;
+
+	/*
+	 * Lifetime of pci_dev is longer than hisi_ptt,
+	 * so directly reference to the pci name string.
+	 */
+	hisi_ptt->name = pci_name(hisi_ptt->pdev);
+	pci_set_drvdata(pdev, hisi_ptt);
+
+	ret = pcim_enable_device(pdev);
+	if (ret) {
+		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
+		return ret;
+	}
+
+	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
+	if (ret) {
+		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
+		return ret;
+	}
+
+	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
+
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret) {
+		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
+		return ret;
+	}
+	pci_set_master(pdev);
+
+	ret = hisi_ptt_register_irq(hisi_ptt);
+	if (ret)
+		return ret;
+
+	hisi_ptt_init_ctrls(hisi_ptt);
+
+	ret = hisi_ptt_register_pmu(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
+		return ret;
+	}
+
+	ret = hisi_ptt_get_rmr(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
+		return ret;
+	}
+
+	/* Register the bus notifier for dynamically updating the filter list */
+	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
+	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+	if (ret)
+		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
+
+	return 0;
+}
+
+void hisi_ptt_remove(struct pci_dev *pdev)
+{
+	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
+
+	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+
+	/* Cancel any work that has been queued */
+	cancel_delayed_work_sync(&hisi_ptt->work);
+
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
+		hisi_ptt_trace_end(hisi_ptt);
+
+	hisi_ptt_free_trace_buf(hisi_ptt);
+	hisi_ptt_release_filters(hisi_ptt);
+}
+
+static const struct pci_device_id hisi_ptt_id_tbl[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
+	{ }
+};
+MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
+
+static struct pci_driver hisi_ptt_driver = {
+	.name = "hisi_ptt",
+	.id_table = hisi_ptt_id_tbl,
+	.probe = hisi_ptt_probe,
+	.remove = hisi_ptt_remove,
+};
+module_pci_driver(hisi_ptt_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
+MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
-- 
2.33.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
integrated Endpoint(RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic(tune),
and trace the TLP headers(trace).

Add the driver for the device to enable the trace function. The driver
will create PMU device for each PTT device, and users can start trace
through perf command.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/Makefile                       |    1 +
 drivers/hwtracing/Kconfig              |    2 +
 drivers/hwtracing/hisilicon/Kconfig    |    8 +
 drivers/hwtracing/hisilicon/Makefile   |    2 +
 drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
 5 files changed, 1159 insertions(+)
 create mode 100644 drivers/hwtracing/hisilicon/Kconfig
 create mode 100644 drivers/hwtracing/hisilicon/Makefile
 create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c

diff --git a/drivers/Makefile b/drivers/Makefile
index be5d40ae1488..bb0dc9b55ea2 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
 obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
 obj-y				+= hwtracing/intel_th/
 obj-$(CONFIG_STM)		+= hwtracing/stm/
+obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
 obj-$(CONFIG_ANDROID)		+= android/
 obj-$(CONFIG_NVMEM)		+= nvmem/
 obj-$(CONFIG_FPGA)		+= fpga/
diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
index 13085835a636..e3796b17541a 100644
--- a/drivers/hwtracing/Kconfig
+++ b/drivers/hwtracing/Kconfig
@@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
 
 source "drivers/hwtracing/intel_th/Kconfig"
 
+source "drivers/hwtracing/hisilicon/Kconfig"
+
 endmenu
diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
new file mode 100644
index 000000000000..9c3ab80d99fe
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config HISI_PTT
+	tristate "HiSilicon PCIe Tune and Trace Device"
+	depends on PCI && HAS_DMA && HAS_IOMEM
+	help
+	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
+	  device, provides support for PCIe traffic tuning and
+	  tracing TLP headers to the memory.
diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
new file mode 100644
index 000000000000..908c09a98161
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
new file mode 100644
index 000000000000..e11e9b6cc2a8
--- /dev/null
+++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
@@ -0,0 +1,1146 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for HiSilicon PCIe tune and trace device
+ *
+ * Copyright (c) 2021 HiSilicon Limited.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/dma-iommu.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iommu.h>
+#include <linux/iopoll.h>
+#include <linux/kfifo.h>
+#include <linux/module.h>
+#include <linux/notifier.h>
+#include <linux/pci.h>
+#include <linux/perf_event.h>
+#include <linux/sizes.h>
+#include <linux/sysfs.h>
+#include <linux/vmalloc.h>
+#include <linux/workqueue.h>
+
+#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
+#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
+#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
+#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
+#define HISI_PTT_TRACE_CTRL		0x0850
+#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
+#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
+#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
+#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
+#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
+#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
+#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
+#define HISI_PTT_TRACE_INT_STAT		0x0890
+#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
+#define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TRACE_WR_STS		0x08a0
+#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
+#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
+#define HISI_PTT_TRACE_STS		0x08b0
+#define   HISI_PTT_TRACE_IDLE		BIT(0)
+#define HISI_PTT_DEVICE_RANGE		0x0fe0
+#define HISI_PTT_LOCATION		0x0fe8
+#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
+#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
+
+#define HISI_PTT_TRACE_DMA_IRQ			0
+#define HISI_PTT_TRACE_BUFLETS_CNT		4
+#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
+#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
+						 HISI_PTT_TRACE_BUFLETS_CNT)
+#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
+
+/* Delay time for filter updating work */
+#define HISI_PTT_WORK_DELAY_MS		100UL
+/* Wait time for DMA hardware to reset */
+#define HISI_PTT_RESET_WAIT_MS		1000UL
+/* Poll timeout and interval for waiting hardware work to finish */
+#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
+#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
+
+#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
+
+enum hisi_ptt_trace_status {
+	HISI_PTT_TRACE_STATUS_OFF = 0,
+	HISI_PTT_TRACE_STATUS_ON,
+};
+
+struct hisi_ptt_dma_buflet {
+	struct list_head list;
+	unsigned int size;
+	dma_addr_t dma;
+
+	/*
+	 * The address of the buflet holding the trace data.
+	 * See Documentation/trace/hisi-ptt.rst for the details
+	 * of the data format.
+	 */
+	void *addr;
+	int index;
+};
+
+struct hisi_ptt_trace_ctrl {
+	enum hisi_ptt_trace_status status;
+	struct perf_output_handle handle;
+	struct list_head trace_buf;
+	/*
+	 * The index of the buffer which trace data
+	 * currently is writing to.
+	 */
+	u32 buf_index;
+
+	int default_cpu;
+	u32 buflet_size;
+	bool is_port;
+	u32 format:1;		/* Format of the traced TLP headers */
+	u32 type:4;		/* Type of the TLP headers to trace */
+	u32 direction:2;	/* Direction of the TLP headers to trace */
+	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
+
+	phys_addr_t rmr_addr;
+	size_t rmr_length;
+	bool has_rmr;
+};
+
+struct hisi_ptt_filter_desc {
+	struct list_head list;
+	struct pci_dev *pdev;
+	u16 val;
+};
+
+struct hisi_ptt_pmu_buf {
+	size_t length;
+	int nr_pages;
+	void *base;
+	long pos;
+};
+
+/* Structure containing the information for filter updating */
+struct hisi_ptt_filter_update_info {
+	struct pci_dev *pdev;
+	u32 port_devid;
+	bool is_port;
+	bool is_add;
+	u16 val;
+};
+
+struct hisi_ptt {
+	struct hisi_ptt_trace_ctrl trace_ctrl;
+	struct notifier_block hisi_ptt_nb;
+	struct pmu hisi_ptt_pmu;
+	void __iomem *iobase;
+	struct pci_dev *pdev;
+	/*
+	 * Use the mutex to protect the filter list and
+	 * serialize the perf process.
+	 */
+	struct mutex mutex;
+	const char *name;
+	u16 core_id;
+	u16 sicl_id;
+	/* PCI device range managed by the PTT device */
+	u32 upper;
+	u32 lower;
+	u8 busnr;
+
+	/*
+	 * The trace TLP headers can either be filtered by certain
+	 * root port, or by the requester ID. Organize the filters
+	 * by @port_filters and @req_filters here. The mask of all
+	 * the valid ports is also cached for doing sanity check
+	 * of user input.
+	 */
+	struct list_head port_filters;
+	struct list_head req_filters;
+	u16 port_mask;
+
+	/*
+	 * We use a delayed work here to avoid indefinitely waiting for
+	 * the hisi_ptt->mutex which protecting the filter list. The
+	 * work will be delayed only if the mutex can not be held,
+	 * otherwise no delay will be applied.
+	 */
+	struct delayed_work work;
+	spinlock_t filter_update_lock;
+	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
+		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
+};
+
+static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
+{
+	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
+}
+
+static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
+{
+	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
+		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
+
+	return PCI_DEVID(pdev->bus->number, pdev->devfn);
+}
+
+static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
+				  val & HISI_PTT_TRACE_IDLE,
+				  HISI_PTT_WAIT_POLL_INTERVAL_US,
+				  HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
+
+	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
+		list_del(&buflet->list);
+
+		if (ctrl->has_rmr)
+			memunmap(buflet->addr);
+		else
+			dma_free_coherent(dev, buflet->size, buflet->addr,
+					  buflet->dma);
+
+		kfree(buflet);
+	}
+}
+
+static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct device *dev = &hisi_ptt->pdev->dev;
+	struct hisi_ptt_dma_buflet *buflet;
+	int i, ret;
+
+	hisi_ptt->trace_ctrl.buf_index = 0;
+
+	/* Make sure the trace buffer is empty before allocating */
+	if (!list_empty(&ctrl->trace_buf)) {
+		list_for_each_entry(buflet, &ctrl->trace_buf, list)
+			memset(buflet->addr, 0, buflet->size);
+		return 0;
+	}
+
+	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
+		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
+		if (!buflet) {
+			ret = -ENOMEM;
+			goto err;
+		}
+
+		if (ctrl->has_rmr) {
+			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
+
+			buflet->dma = base;
+			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
+		} else {
+			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
+							  &buflet->dma, GFP_KERNEL);
+		}
+
+		if (!buflet->addr) {
+			kfree(buflet);
+			ret = -ENOMEM;
+			goto err;
+		}
+
+		memset(buflet->addr, 0, buflet->size);
+
+		buflet->index = i;
+		buflet->size = ctrl->buflet_size;
+		list_add_tail(&buflet->list, &ctrl->trace_buf);
+	}
+
+	return 0;
+err:
+	hisi_ptt_free_trace_buf(hisi_ptt);
+	return ret;
+}
+
+static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
+{
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
+}
+
+static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	struct hisi_ptt_dma_buflet *cur;
+	u32 val;
+
+	/* Check device idle before start trace */
+	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
+		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
+		return -EBUSY;
+	}
+
+	/* Reset the DMA before start tracing */
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val |= HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	/*
+	 * We'll be in the perf context where preemption is disabled,
+	 * so use busy loop here.
+	 */
+	mdelay(HISI_PTT_RESET_WAIT_MS);
+
+	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+	val &= ~HISI_PTT_TRACE_CTRL_RST;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	/* clear the interrupt status */
+	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
+
+	list_for_each_entry(cur, &ctrl->trace_buf, list) {
+		writel(lower_32_bits(cur->dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
+		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+		writel(upper_32_bits(cur->dma),
+		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
+		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
+	}
+	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
+
+	/* set the trace control register */
+	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
+	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
+	if (!hisi_ptt->trace_ctrl.is_port)
+		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
+
+	val |= HISI_PTT_TRACE_CTRL_EN;
+	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
+
+	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
+
+	return 0;
+}
+
+static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
+{
+	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+	struct perf_event *event = handle->event;
+	struct hisi_ptt_dma_buflet *cur;
+	struct hisi_ptt_pmu_buf *buf;
+
+	buf = perf_get_aux(handle);
+	if (!buf || !handle->size)
+		return -EINVAL;
+
+	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
+		if (cur->index == index)
+			break;
+
+	memcpy(buf->base + buf->pos, cur->addr, cur->size);
+	memset(cur->addr, 0, cur->size);
+	buf->pos += cur->size;
+
+	if (stop) {
+		perf_aux_output_end(handle, buf->pos);
+	} else if (buf->length - buf->pos < cur->size) {
+		perf_aux_output_skip(handle, buf->length - buf->pos);
+		perf_aux_output_end(handle, buf->pos);
+
+		buf = perf_aux_output_begin(handle, event);
+		if (!buf)
+			return -EINVAL;
+
+		buf->pos = handle->head % buf->length;
+		if (buf->length - buf->pos < cur->size) {
+			perf_aux_output_end(handle, 0);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static irqreturn_t hisi_ptt_isr(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status, buf_idx;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	buf_idx = ffs(status) - 1;
+
+	/* Clear the interrupt status of buflet @buf_idx */
+	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+
+	/*
+	 * Update the AUX buffer and cache the current buflet index,
+	 * as we need to know this and save the data when the trace
+	 * is ended out of the interrupt handler. End the trace
+	 * if the updating fails.
+	 */
+	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
+		hisi_ptt_trace_end(hisi_ptt);
+	else
+		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
+
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t hisi_ptt_irq(int irq, void *context)
+{
+	struct hisi_ptt *hisi_ptt = context;
+	u32 status;
+
+	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
+	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
+		return IRQ_NONE;
+
+	return IRQ_WAKE_THREAD;
+}
+
+static void hisi_ptt_irq_free_vectors(void *pdev)
+{
+	pci_free_irq_vectors(pdev);
+}
+
+static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	int ret;
+
+	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
+	if (ret < 0) {
+		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
+		return ret;
+	}
+
+	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
+	if (ret < 0)
+		return ret;
+
+	ret = devm_request_threaded_irq(&pdev->dev,
+					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
+					hisi_ptt_irq, hisi_ptt_isr, 0,
+					"hisi-ptt", hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
+			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_update_filters(struct work_struct *work)
+{
+	struct delayed_work *delayed_work = to_delayed_work(work);
+	struct hisi_ptt_filter_update_info info;
+	struct hisi_ptt_filter_desc *filter;
+	struct list_head *target_list;
+	struct hisi_ptt *hisi_ptt;
+
+	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
+
+	if (!mutex_trylock(&hisi_ptt->mutex)) {
+		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
+		return;
+	}
+
+	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
+		target_list = info.is_port ? &hisi_ptt->port_filters :
+			      &hisi_ptt->req_filters;
+
+		if (info.is_add) {
+			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
+			if (!filter)
+				continue;
+
+			filter->pdev = info.pdev;
+			filter->val = info.val;
+
+			list_add_tail(&filter->list, target_list);
+		} else {
+			list_for_each_entry(filter, target_list, list)
+				if (filter->val == info.val) {
+					list_del(&filter->list);
+					kfree(filter);
+					break;
+				}
+		}
+
+		/* Update the available port mask */
+		if (!info.is_port)
+			continue;
+
+		if (info.is_add)
+			hisi_ptt->port_mask |= info.val;
+		else
+			hisi_ptt->port_mask &= ~info.val;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
+				    struct hisi_ptt_filter_update_info *info)
+{
+	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
+
+	if (!root_port)
+		return;
+
+	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
+	if (info->port_devid < hisi_ptt->lower ||
+	    info->port_devid > hisi_ptt->upper)
+		return;
+
+	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
+	info->val = hisi_ptt_get_filter_val(info->pdev);
+
+	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
+				&hisi_ptt->filter_update_lock))
+		schedule_delayed_work(&hisi_ptt->work, 0);
+	else
+		pci_warn(hisi_ptt->pdev,
+			 "filter update fifo overflow for target %s\n",
+			 pci_name(info->pdev));
+}
+
+/*
+ * A PCI bus notifier is used here for dynamically updating the filter
+ * list.
+ */
+static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
+				  void *data)
+{
+	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
+	struct hisi_ptt_filter_update_info info;
+	struct device *dev = data;
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	info.pdev = pdev;
+
+	switch (action) {
+	case BUS_NOTIFY_ADD_DEVICE:
+		info.is_add = true;
+		break;
+	case BUS_NOTIFY_DEL_DEVICE:
+		info.is_add = false;
+		break;
+	default:
+		return 0;
+	}
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+	return 0;
+}
+
+static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
+{
+	struct hisi_ptt_filter_update_info info = {
+		.pdev = pdev,
+		.is_add = true,
+	};
+	struct hisi_ptt *hisi_ptt = data;
+
+	hisi_ptt_update_fifo_in(hisi_ptt, &info);
+
+	return 0;
+}
+
+static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
+{
+	struct hisi_ptt_filter_desc *filter, *tfilter;
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+
+	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
+		list_del(&filter->list);
+		kfree(filter);
+	}
+}
+
+static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	struct pci_bus *bus;
+	u32 reg;
+
+	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
+	spin_lock_init(&hisi_ptt->filter_update_lock);
+	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
+	INIT_LIST_HEAD(&hisi_ptt->port_filters);
+	INIT_LIST_HEAD(&hisi_ptt->req_filters);
+
+	/*
+	 * The device range register provides the information about the
+	 * root ports which the RCiEP can control and trace. The RCiEP
+	 * and the root ports it support are on the same PCIe core, with
+	 * same domain number but maybe different bus number. The device
+	 * range register will tell us which root ports we can support,
+	 * Bit[31:16] indicates the upper BDF numbers of the root port,
+	 * while Bit[15:0] indicates the lower.
+	 */
+	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
+	hisi_ptt->upper = reg >> 16;
+	hisi_ptt->lower = reg & 0xffff;
+	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
+	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
+	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
+
+	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
+	if (bus)
+		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
+
+	/* Initialize trace controls */
+	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
+	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
+	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
+}
+
+#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
+#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
+#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
+#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
+#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
+
+static ssize_t available_filters_show(struct device *dev,
+				      struct device_attribute *attr,
+				      char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct hisi_ptt_filter_desc *filter;
+	int pos = 0;
+
+	if (list_empty(&hisi_ptt->port_filters))
+		return sysfs_emit(buf, "#### No available filter ####\n");
+
+	mutex_lock(&hisi_ptt->mutex);
+	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
+	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev) |
+				     HISI_PTT_PMU_FILTER_IS_PORT);
+
+	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
+	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
+				     pci_name(filter->pdev),
+				     hisi_ptt_get_filter_val(filter->pdev));
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return pos;
+}
+static DEVICE_ATTR_ADMIN_RO(available_filters);
+
+PMU_FORMAT_ATTR(filter,		"config:0-19");
+PMU_FORMAT_ATTR(direction,	"config:20-23");
+PMU_FORMAT_ATTR(type,		"config:24-31");
+PMU_FORMAT_ATTR(format,		"config:32-35");
+
+static struct attribute *hisi_ptt_pmu_format_attrs[] = {
+	&format_attr_filter.attr,
+	&format_attr_direction.attr,
+	&format_attr_type.attr,
+	&format_attr_format.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_format_group = {
+	.name = "format",
+	.attrs = hisi_ptt_pmu_format_attrs,
+};
+
+static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
+	&dev_attr_available_filters.attr,
+	NULL
+};
+
+static struct attribute_group hisi_ptt_pmu_filter_group = {
+	.attrs = hisi_ptt_pmu_filter_attrs,
+};
+
+static const struct attribute_group *hisi_ptt_pmu_groups[] = {
+	&hisi_ptt_pmu_format_group,
+	&hisi_ptt_pmu_filter_group,
+	NULL
+};
+
+/*
+ * The supported value of the direction parameter. See hisi_ptt.rst
+ * documentation for more details.
+ */
+static u32 hisi_ptt_trace_available_direction[] = {
+	0,
+	1,
+	2,
+	3,
+};
+
+/* Different types can be set simultaneously */
+static u32 hisi_ptt_trace_available_type[] = {
+	1,	/* posted_request */
+	2,	/* non-posted_request */
+	4,	/* completion */
+};
+
+static u32 hisi_ptt_trace_availble_format[] = {
+	0,	/* 4DW */
+	1,	/* 8DW */
+};
+
+/*
+ * Check whether the config is valid or not. Some configs are multi-selectable
+ * and can be set simultaneously, while some are single selectable (onehot).
+ * Use this function to check the non-onehot configs while
+ * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
+ */
+static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
+{
+	int i;
+
+	/*
+	 * The non-onehot configs cannot be 0. Walk the available
+	 * list and clear the valid bits of the config. If there
+	 * is any resident bit after the walk then the config is
+	 * invalid.
+	 */
+	for (i = 0; i < list_size; i++)
+		val &= ~available_list[i];
+
+	return val ? -EINVAL : 0;
+}
+
+static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
+{
+	int i, ret = -EINVAL;
+
+	for (i = 0; i < list_size; i++)
+		if (val == available_list[i]) {
+			ret = 0;
+			break;
+		}
+
+	return ret;
+}
+
+static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
+{
+	unsigned long val, port_mask = hisi_ptt->port_mask;
+	struct hisi_ptt_filter_desc *filter;
+	int ret = -EINVAL;
+
+	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
+	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
+
+	/*
+	 * Port filters are defined as bit mask. For port filters, check
+	 * the bits in the @val are within the range of hisi_ptt->port_mask
+	 * and whether it's empty or not, otherwise user has specified
+	 * some unsupported root ports.
+	 *
+	 * For Requester ID filters, walk the available filter list to see
+	 * whether we have one matched.
+	 */
+	if (hisi_ptt->trace_ctrl.is_port &&
+	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
+		ret = 0;
+	} else {
+		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
+			if (filter->val == val) {
+				ret = 0;
+				break;
+			}
+	}
+
+	if (ret)
+		return ret;
+
+	hisi_ptt->trace_ctrl.filter = val;
+	return 0;
+}
+
+static int hisi_ptt_pmu_event_init(struct perf_event *event)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
+	int ret;
+	u32 val;
+
+	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
+		return -ENOENT;
+
+	mutex_lock(&hisi_ptt->mutex);
+
+	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
+	if (ret < 0)
+		goto out;
+
+	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
+						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
+	if (ret < 0)
+		goto out;
+	ctrl->direction = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
+
+	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
+					  ARRAY_SIZE(hisi_ptt_trace_available_type));
+	if (ret < 0)
+		goto out;
+	ctrl->type = val;
+
+	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
+	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
+						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
+	if (ret < 0)
+		goto out;
+	ctrl->format = val;
+
+out:
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret;
+}
+
+static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
+				    int nr_pages, bool overwrite)
+{
+	struct hisi_ptt_pmu_buf *buf;
+	struct page **pagelist;
+	int i;
+
+	if (overwrite) {
+		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
+		return NULL;
+	}
+
+	/* If the pages size less than buflets, we cannot start trace */
+	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
+		return NULL;
+
+	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
+	if (!buf)
+		return NULL;
+
+	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
+	if (!pagelist) {
+		kfree(buf);
+		return NULL;
+	}
+
+	for (i = 0; i < nr_pages; i++)
+		pagelist[i] = virt_to_page(pages[i]);
+
+	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
+	if (!buf->base) {
+		kfree(pagelist);
+		kfree(buf);
+		return NULL;
+	}
+
+	buf->nr_pages = nr_pages;
+	buf->length = nr_pages * PAGE_SIZE;
+	buf->pos = 0;
+
+	kfree(pagelist);
+	return buf;
+}
+
+static void hisi_ptt_pmu_free_aux(void *aux)
+{
+	struct hisi_ptt_pmu_buf *buf = aux;
+
+	vunmap(buf->base);
+	kfree(buf);
+}
+
+static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
+	struct hw_perf_event *hwc = &event->hw;
+	struct hisi_ptt_pmu_buf *buf;
+	int cpu = event->cpu;
+	int ret;
+
+	hwc->state = 0;
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
+		goto stop;
+	}
+
+	if (cpu == -1)
+		cpu = hisi_ptt->trace_ctrl.default_cpu;
+
+	/*
+	 * Handle the interrupt on the same cpu which starts the trace to avoid
+	 * context mismatch. Otherwise we'll trigger the WARN from the perf
+	 * core in event_function_local().
+	 */
+	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
+				 cpumask_of(cpu)));
+
+	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
+		goto stop;
+	}
+
+	buf = perf_aux_output_begin(handle, event);
+	if (!buf) {
+		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
+		goto stop;
+	}
+
+	buf->pos = handle->head % buf->length;
+
+	ret = hisi_ptt_trace_start(hisi_ptt);
+	if (ret) {
+		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
+		perf_aux_output_end(handle, 0);
+		goto stop;
+	}
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return;
+stop:
+	event->hw.state |= PERF_HES_STOPPED;
+	mutex_unlock(&hisi_ptt->mutex);
+}
+
+static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+
+	if (hwc->state & PERF_HES_STOPPED)
+		return;
+
+	mutex_lock(&hisi_ptt->mutex);
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
+		hisi_ptt_trace_end(hisi_ptt);
+		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
+		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
+	}
+	mutex_unlock(&hisi_ptt->mutex);
+
+	hwc->state |= PERF_HES_STOPPED;
+	perf_event_update_userpage(event);
+	hwc->state |= PERF_HES_UPTODATE;
+}
+
+static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+	int cpu = event->cpu;
+
+	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
+		return 0;
+
+	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+
+	if (flags & PERF_EF_START) {
+		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
+		if (hwc->state & PERF_HES_STOPPED)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
+{
+	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
+}
+
+static void hisi_ptt_unregister_pmu(void *priv)
+{
+	perf_pmu_unregister(priv);
+}
+
+static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
+{
+	char *pmu_name;
+	int ret;
+
+	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
+		.module		= THIS_MODULE,
+		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
+		.task_ctx_nr	= perf_sw_context,
+		.attr_groups	= hisi_ptt_pmu_groups,
+		.event_init	= hisi_ptt_pmu_event_init,
+		.setup_aux	= hisi_ptt_pmu_setup_aux,
+		.free_aux	= hisi_ptt_pmu_free_aux,
+		.start		= hisi_ptt_pmu_start,
+		.stop		= hisi_ptt_pmu_stop,
+		.add		= hisi_ptt_pmu_add,
+		.del		= hisi_ptt_pmu_del,
+	};
+
+	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
+				  hisi_ptt->sicl_id, hisi_ptt->core_id);
+	if (!pmu_name)
+		return -ENOMEM;
+
+	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
+	if (ret)
+		return ret;
+
+	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
+					hisi_ptt_unregister_pmu,
+					&hisi_ptt->hisi_ptt_pmu);
+}
+
+/*
+ * Get RMR address if provided by the firmware.
+ * Return 0 if the IOMMU doesn't present or the policy of the
+ * IOMMU domain is passthrough or we get a usable RMR region.
+ * Otherwise a negative value is returned.
+ */
+static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
+{
+	struct pci_dev *pdev = hisi_ptt->pdev;
+	struct iommu_domain *iommu_domain;
+	struct iommu_resv_region *region;
+	LIST_HEAD(list);
+
+	/*
+	 * Use direct DMA if IOMMU does not present or the policy of the
+	 * IOMMU domain is passthrough.
+	 */
+	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
+	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
+		return 0;
+
+	iommu_get_resv_regions(&pdev->dev, &list);
+	list_for_each_entry(region, &list, list)
+		if (region->type == IOMMU_RESV_DIRECT &&
+		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
+			hisi_ptt->trace_ctrl.has_rmr = true;
+			hisi_ptt->trace_ctrl.rmr_addr = region->start;
+			hisi_ptt->trace_ctrl.rmr_length = region->length;
+			break;
+		}
+
+	iommu_put_resv_regions(&pdev->dev, &list);
+	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
+}
+
+static int hisi_ptt_probe(struct pci_dev *pdev,
+			  const struct pci_device_id *id)
+{
+	struct hisi_ptt *hisi_ptt;
+	int ret;
+
+	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
+	if (!hisi_ptt)
+		return -ENOMEM;
+
+	mutex_init(&hisi_ptt->mutex);
+	hisi_ptt->pdev = pdev;
+
+	/*
+	 * Lifetime of pci_dev is longer than hisi_ptt,
+	 * so directly reference to the pci name string.
+	 */
+	hisi_ptt->name = pci_name(hisi_ptt->pdev);
+	pci_set_drvdata(pdev, hisi_ptt);
+
+	ret = pcim_enable_device(pdev);
+	if (ret) {
+		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
+		return ret;
+	}
+
+	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
+	if (ret) {
+		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
+		return ret;
+	}
+
+	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
+
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret) {
+		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
+		return ret;
+	}
+	pci_set_master(pdev);
+
+	ret = hisi_ptt_register_irq(hisi_ptt);
+	if (ret)
+		return ret;
+
+	hisi_ptt_init_ctrls(hisi_ptt);
+
+	ret = hisi_ptt_register_pmu(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
+		return ret;
+	}
+
+	ret = hisi_ptt_get_rmr(hisi_ptt);
+	if (ret) {
+		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
+		return ret;
+	}
+
+	/* Register the bus notifier for dynamically updating the filter list */
+	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
+	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+	if (ret)
+		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
+
+	return 0;
+}
+
+void hisi_ptt_remove(struct pci_dev *pdev)
+{
+	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
+
+	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
+
+	/* Cancel any work that has been queued */
+	cancel_delayed_work_sync(&hisi_ptt->work);
+
+	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
+		hisi_ptt_trace_end(hisi_ptt);
+
+	hisi_ptt_free_trace_buf(hisi_ptt);
+	hisi_ptt_release_filters(hisi_ptt);
+}
+
+static const struct pci_device_id hisi_ptt_id_tbl[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
+	{ }
+};
+MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
+
+static struct pci_driver hisi_ptt_driver = {
+	.name = "hisi_ptt",
+	.id_table = hisi_ptt_id_tbl,
+	.probe = hisi_ptt_probe,
+	.remove = hisi_ptt_remove,
+};
+module_pci_driver(hisi_ptt_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
+MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
-- 
2.33.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-16 09:06, Yicong Yang via iommu wrote:
[...]
> +/*
> + * Get RMR address if provided by the firmware.
> + * Return 0 if the IOMMU doesn't present or the policy of the
> + * IOMMU domain is passthrough or we get a usable RMR region.
> + * Otherwise a negative value is returned.
> + */
> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct iommu_domain *iommu_domain;
> +	struct iommu_resv_region *region;
> +	LIST_HEAD(list);
> +
> +	/*
> +	 * Use direct DMA if IOMMU does not present or the policy of the
> +	 * IOMMU domain is passthrough.
> +	 */
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	iommu_get_resv_regions(&pdev->dev, &list);
> +	list_for_each_entry(region, &list, list)
> +		if (region->type == IOMMU_RESV_DIRECT &&
> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> +			hisi_ptt->trace_ctrl.has_rmr = true;
> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> +			break;
> +		}
> +
> +	iommu_put_resv_regions(&pdev->dev, &list);
> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> +}

No.

The whole point of RMRs is for devices that are already configured to 
access the given address range in a manner beyond the kernel's control. 
If you can do this, it proves that you should not have an RMR in the 
first place.

The notion of a kernel driver explicitly configuring its device to DMA 
into any random RMR that looks big enough is so egregiously wrong that 
I'm almost lost for words...

Robin.

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-16 09:06, Yicong Yang via iommu wrote:
[...]
> +/*
> + * Get RMR address if provided by the firmware.
> + * Return 0 if the IOMMU doesn't present or the policy of the
> + * IOMMU domain is passthrough or we get a usable RMR region.
> + * Otherwise a negative value is returned.
> + */
> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct iommu_domain *iommu_domain;
> +	struct iommu_resv_region *region;
> +	LIST_HEAD(list);
> +
> +	/*
> +	 * Use direct DMA if IOMMU does not present or the policy of the
> +	 * IOMMU domain is passthrough.
> +	 */
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	iommu_get_resv_regions(&pdev->dev, &list);
> +	list_for_each_entry(region, &list, list)
> +		if (region->type == IOMMU_RESV_DIRECT &&
> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> +			hisi_ptt->trace_ctrl.has_rmr = true;
> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> +			break;
> +		}
> +
> +	iommu_put_resv_regions(&pdev->dev, &list);
> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> +}

No.

The whole point of RMRs is for devices that are already configured to 
access the given address range in a manner beyond the kernel's control. 
If you can do this, it proves that you should not have an RMR in the 
first place.

The notion of a kernel driver explicitly configuring its device to DMA 
into any random RMR that looks big enough is so egregiously wrong that 
I'm almost lost for words...

Robin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-16 09:06, Yicong Yang via iommu wrote:
[...]
> +/*
> + * Get RMR address if provided by the firmware.
> + * Return 0 if the IOMMU doesn't present or the policy of the
> + * IOMMU domain is passthrough or we get a usable RMR region.
> + * Otherwise a negative value is returned.
> + */
> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct iommu_domain *iommu_domain;
> +	struct iommu_resv_region *region;
> +	LIST_HEAD(list);
> +
> +	/*
> +	 * Use direct DMA if IOMMU does not present or the policy of the
> +	 * IOMMU domain is passthrough.
> +	 */
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	iommu_get_resv_regions(&pdev->dev, &list);
> +	list_for_each_entry(region, &list, list)
> +		if (region->type == IOMMU_RESV_DIRECT &&
> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> +			hisi_ptt->trace_ctrl.has_rmr = true;
> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> +			break;
> +		}
> +
> +	iommu_put_resv_regions(&pdev->dev, &list);
> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> +}

No.

The whole point of RMRs is for devices that are already configured to 
access the given address range in a manner beyond the kernel's control. 
If you can do this, it proves that you should not have an RMR in the 
first place.

The notion of a kernel driver explicitly configuring its device to DMA 
into any random RMR that looks big enough is so egregiously wrong that 
I'm almost lost for words...

Robin.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/16 18:56, Robin Murphy wrote:
> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
> [...]
>> +/*
>> + * Get RMR address if provided by the firmware.
>> + * Return 0 if the IOMMU doesn't present or the policy of the
>> + * IOMMU domain is passthrough or we get a usable RMR region.
>> + * Otherwise a negative value is returned.
>> + */
>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>> +{
>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>> +    struct iommu_domain *iommu_domain;
>> +    struct iommu_resv_region *region;
>> +    LIST_HEAD(list);
>> +
>> +    /*
>> +     * Use direct DMA if IOMMU does not present or the policy of the
>> +     * IOMMU domain is passthrough.
>> +     */
>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> +        return 0;
>> +
>> +    iommu_get_resv_regions(&pdev->dev, &list);
>> +    list_for_each_entry(region, &list, list)
>> +        if (region->type == IOMMU_RESV_DIRECT &&
>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>> +            break;
>> +        }
>> +
>> +    iommu_put_resv_regions(&pdev->dev, &list);
>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>> +}
> 
> No.
> 
> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
> 
> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
> 

our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.

is it ok to let bios configure the address to the device and from CPU side we just read it?

Thanks,
Yicong

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

On 2021/11/16 18:56, Robin Murphy wrote:
> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
> [...]
>> +/*
>> + * Get RMR address if provided by the firmware.
>> + * Return 0 if the IOMMU doesn't present or the policy of the
>> + * IOMMU domain is passthrough or we get a usable RMR region.
>> + * Otherwise a negative value is returned.
>> + */
>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>> +{
>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>> +    struct iommu_domain *iommu_domain;
>> +    struct iommu_resv_region *region;
>> +    LIST_HEAD(list);
>> +
>> +    /*
>> +     * Use direct DMA if IOMMU does not present or the policy of the
>> +     * IOMMU domain is passthrough.
>> +     */
>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> +        return 0;
>> +
>> +    iommu_get_resv_regions(&pdev->dev, &list);
>> +    list_for_each_entry(region, &list, list)
>> +        if (region->type == IOMMU_RESV_DIRECT &&
>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>> +            break;
>> +        }
>> +
>> +    iommu_put_resv_regions(&pdev->dev, &list);
>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>> +}
> 
> No.
> 
> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
> 
> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
> 

our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.

is it ok to let bios configure the address to the device and from CPU side we just read it?

Thanks,
Yicong


_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/16 18:56, Robin Murphy wrote:
> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
> [...]
>> +/*
>> + * Get RMR address if provided by the firmware.
>> + * Return 0 if the IOMMU doesn't present or the policy of the
>> + * IOMMU domain is passthrough or we get a usable RMR region.
>> + * Otherwise a negative value is returned.
>> + */
>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>> +{
>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>> +    struct iommu_domain *iommu_domain;
>> +    struct iommu_resv_region *region;
>> +    LIST_HEAD(list);
>> +
>> +    /*
>> +     * Use direct DMA if IOMMU does not present or the policy of the
>> +     * IOMMU domain is passthrough.
>> +     */
>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> +        return 0;
>> +
>> +    iommu_get_resv_regions(&pdev->dev, &list);
>> +    list_for_each_entry(region, &list, list)
>> +        if (region->type == IOMMU_RESV_DIRECT &&
>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>> +            break;
>> +        }
>> +
>> +    iommu_put_resv_regions(&pdev->dev, &list);
>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>> +}
> 
> No.
> 
> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
> 
> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
> 

our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.

is it ok to let bios configure the address to the device and from CPU side we just read it?

Thanks,
Yicong



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

Hi Robin,

On 2021/11/16 19:37, Yicong Yang wrote:
> On 2021/11/16 18:56, Robin Murphy wrote:
>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>> [...]
>>> +/*
>>> + * Get RMR address if provided by the firmware.
>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>> + * Otherwise a negative value is returned.
>>> + */
>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>> +    struct iommu_domain *iommu_domain;
>>> +    struct iommu_resv_region *region;
>>> +    LIST_HEAD(list);
>>> +
>>> +    /*
>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>> +     * IOMMU domain is passthrough.
>>> +     */
>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>> +        return 0;
>>> +
>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>> +    list_for_each_entry(region, &list, list)
>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>> +            break;
>>> +        }
>>> +
>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>> +}
>>
>> No.
>>
>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>
>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>
> 
> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
> 
> is it ok to let bios configure the address to the device and from CPU side we just read it?
> 

Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?

> 
> 
> .
> 

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

Hi Robin,

On 2021/11/16 19:37, Yicong Yang wrote:
> On 2021/11/16 18:56, Robin Murphy wrote:
>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>> [...]
>>> +/*
>>> + * Get RMR address if provided by the firmware.
>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>> + * Otherwise a negative value is returned.
>>> + */
>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>> +    struct iommu_domain *iommu_domain;
>>> +    struct iommu_resv_region *region;
>>> +    LIST_HEAD(list);
>>> +
>>> +    /*
>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>> +     * IOMMU domain is passthrough.
>>> +     */
>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>> +        return 0;
>>> +
>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>> +    list_for_each_entry(region, &list, list)
>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>> +            break;
>>> +        }
>>> +
>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>> +}
>>
>> No.
>>
>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>
>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>
> 
> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
> 
> is it ok to let bios configure the address to the device and from CPU side we just read it?
> 

Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?

> 
> 
> .
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

Hi Robin,

On 2021/11/16 19:37, Yicong Yang wrote:
> On 2021/11/16 18:56, Robin Murphy wrote:
>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>> [...]
>>> +/*
>>> + * Get RMR address if provided by the firmware.
>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>> + * Otherwise a negative value is returned.
>>> + */
>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>> +{
>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>> +    struct iommu_domain *iommu_domain;
>>> +    struct iommu_resv_region *region;
>>> +    LIST_HEAD(list);
>>> +
>>> +    /*
>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>> +     * IOMMU domain is passthrough.
>>> +     */
>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>> +        return 0;
>>> +
>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>> +    list_for_each_entry(region, &list, list)
>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>> +            break;
>>> +        }
>>> +
>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>> +}
>>
>> No.
>>
>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>
>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>
> 
> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
> 
> is it ok to let bios configure the address to the device and from CPU side we just read it?
> 

Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?

> 
> 
> .
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-18 09:01, Yicong Yang via iommu wrote:
> Hi Robin,
> 
> On 2021/11/16 19:37, Yicong Yang wrote:
>> On 2021/11/16 18:56, Robin Murphy wrote:
>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>> [...]
>>>> +/*
>>>> + * Get RMR address if provided by the firmware.
>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>> + * Otherwise a negative value is returned.
>>>> + */
>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +    struct iommu_domain *iommu_domain;
>>>> +    struct iommu_resv_region *region;
>>>> +    LIST_HEAD(list);
>>>> +
>>>> +    /*
>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>> +     * IOMMU domain is passthrough.
>>>> +     */
>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>> +        return 0;
>>>> +
>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>> +    list_for_each_entry(region, &list, list)
>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>> +            break;
>>>> +        }
>>>> +
>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>> +}
>>>
>>> No.
>>>
>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>
>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>
>>
>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>
>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>
> 
> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?

If the firmware configures the device so that it's actively tracing and 
writing out to memory while the kernel boots, then that is a valid 
reason to have an RMR. However what you're doing in the driver is still 
complete nonsense. As far as I can follow, the way it's working is this:

- At probe time, the initial state of the hardware is entirely ignored. 
If it *is* already active, there appears to be a fun chance of crashing 
if TRACE_INT_MASK is clear and an interrupt happens to fire before 
anyone has got round to calling perf_aux_output_begin() to make 
trace_ctrl.handle.rb non-NULL.

- Later, once the user starts a tracing session, a buffer is set up 
*either* as a completely normal DMA allocation, or by memremap()ing some 
random IOVA carveout which may or may not be whatever memory the 
firmware was tracing to.

- The hardware is then reset and completely reprogrammed to use the new 
buffer, again without any consideration of its previous state (other 
than possibly timing out and failing if it's already running and that 
means it never goes idle).

Therefore the driver does not seem to respect any prior configuration of 
the device by firmware, does not seem to expect it to be running at boot 
time, does not seem to have any way to preserve and export any trace 
data captured in an RMR if it *was* running at boot time, and thus 
without loss of generality could simply use the dma_alloc_coherent() 
path all the time. Am I missing anything?

As things stand, RMRs are not yet supported upstream (FYI we're still 
working on fixing the spec...), so the code above is at best dead, and 
at worst actively wrong. Furthermore, if the expected usage model *is* 
that the kernel driver completely resets and reprograms the hardware, 
then even if there is an RMR for boot-time tracing I would rather expect 
it to be flagged as remappable, and thus potentially end up as an 
IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.

And after all that, if you really do have a genuine need to respect and 
preserve prior firmware configuration of the device, then I would surely 
expect to see the driver actually doing exactly that. Presumably: at 
probe time, look at TRACE_CTRL; if the device is already configured, 
read out that configuration - especially including TRACE_ADDR_* - and 
make sure to reuse it. Not go off on a tangent blindly poking into 
internal IOMMU API abstractions in the vain hope that the first thing 
you find happens to be sort-of-related to the information that you 
actually care about.

Thanks,
Robin.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-18 09:01, Yicong Yang via iommu wrote:
> Hi Robin,
> 
> On 2021/11/16 19:37, Yicong Yang wrote:
>> On 2021/11/16 18:56, Robin Murphy wrote:
>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>> [...]
>>>> +/*
>>>> + * Get RMR address if provided by the firmware.
>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>> + * Otherwise a negative value is returned.
>>>> + */
>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +    struct iommu_domain *iommu_domain;
>>>> +    struct iommu_resv_region *region;
>>>> +    LIST_HEAD(list);
>>>> +
>>>> +    /*
>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>> +     * IOMMU domain is passthrough.
>>>> +     */
>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>> +        return 0;
>>>> +
>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>> +    list_for_each_entry(region, &list, list)
>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>> +            break;
>>>> +        }
>>>> +
>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>> +}
>>>
>>> No.
>>>
>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>
>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>
>>
>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>
>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>
> 
> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?

If the firmware configures the device so that it's actively tracing and 
writing out to memory while the kernel boots, then that is a valid 
reason to have an RMR. However what you're doing in the driver is still 
complete nonsense. As far as I can follow, the way it's working is this:

- At probe time, the initial state of the hardware is entirely ignored. 
If it *is* already active, there appears to be a fun chance of crashing 
if TRACE_INT_MASK is clear and an interrupt happens to fire before 
anyone has got round to calling perf_aux_output_begin() to make 
trace_ctrl.handle.rb non-NULL.

- Later, once the user starts a tracing session, a buffer is set up 
*either* as a completely normal DMA allocation, or by memremap()ing some 
random IOVA carveout which may or may not be whatever memory the 
firmware was tracing to.

- The hardware is then reset and completely reprogrammed to use the new 
buffer, again without any consideration of its previous state (other 
than possibly timing out and failing if it's already running and that 
means it never goes idle).

Therefore the driver does not seem to respect any prior configuration of 
the device by firmware, does not seem to expect it to be running at boot 
time, does not seem to have any way to preserve and export any trace 
data captured in an RMR if it *was* running at boot time, and thus 
without loss of generality could simply use the dma_alloc_coherent() 
path all the time. Am I missing anything?

As things stand, RMRs are not yet supported upstream (FYI we're still 
working on fixing the spec...), so the code above is at best dead, and 
at worst actively wrong. Furthermore, if the expected usage model *is* 
that the kernel driver completely resets and reprograms the hardware, 
then even if there is an RMR for boot-time tracing I would rather expect 
it to be flagged as remappable, and thus potentially end up as an 
IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.

And after all that, if you really do have a genuine need to respect and 
preserve prior firmware configuration of the device, then I would surely 
expect to see the driver actually doing exactly that. Presumably: at 
probe time, look at TRACE_CTRL; if the device is already configured, 
read out that configuration - especially including TRACE_ADDR_* - and 
make sure to reuse it. Not go off on a tangent blindly poking into 
internal IOMMU API abstractions in the vain hope that the first thing 
you find happens to be sort-of-related to the information that you 
actually care about.

Thanks,
Robin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-18 09:01, Yicong Yang via iommu wrote:
> Hi Robin,
> 
> On 2021/11/16 19:37, Yicong Yang wrote:
>> On 2021/11/16 18:56, Robin Murphy wrote:
>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>> [...]
>>>> +/*
>>>> + * Get RMR address if provided by the firmware.
>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>> + * Otherwise a negative value is returned.
>>>> + */
>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +    struct iommu_domain *iommu_domain;
>>>> +    struct iommu_resv_region *region;
>>>> +    LIST_HEAD(list);
>>>> +
>>>> +    /*
>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>> +     * IOMMU domain is passthrough.
>>>> +     */
>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>> +        return 0;
>>>> +
>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>> +    list_for_each_entry(region, &list, list)
>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>> +            break;
>>>> +        }
>>>> +
>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>> +}
>>>
>>> No.
>>>
>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>
>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>
>>
>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>
>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>
> 
> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?

If the firmware configures the device so that it's actively tracing and 
writing out to memory while the kernel boots, then that is a valid 
reason to have an RMR. However what you're doing in the driver is still 
complete nonsense. As far as I can follow, the way it's working is this:

- At probe time, the initial state of the hardware is entirely ignored. 
If it *is* already active, there appears to be a fun chance of crashing 
if TRACE_INT_MASK is clear and an interrupt happens to fire before 
anyone has got round to calling perf_aux_output_begin() to make 
trace_ctrl.handle.rb non-NULL.

- Later, once the user starts a tracing session, a buffer is set up 
*either* as a completely normal DMA allocation, or by memremap()ing some 
random IOVA carveout which may or may not be whatever memory the 
firmware was tracing to.

- The hardware is then reset and completely reprogrammed to use the new 
buffer, again without any consideration of its previous state (other 
than possibly timing out and failing if it's already running and that 
means it never goes idle).

Therefore the driver does not seem to respect any prior configuration of 
the device by firmware, does not seem to expect it to be running at boot 
time, does not seem to have any way to preserve and export any trace 
data captured in an RMR if it *was* running at boot time, and thus 
without loss of generality could simply use the dma_alloc_coherent() 
path all the time. Am I missing anything?

As things stand, RMRs are not yet supported upstream (FYI we're still 
working on fixing the spec...), so the code above is at best dead, and 
at worst actively wrong. Furthermore, if the expected usage model *is* 
that the kernel driver completely resets and reprograms the hardware, 
then even if there is an RMR for boot-time tracing I would rather expect 
it to be flagged as remappable, and thus potentially end up as an 
IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.

And after all that, if you really do have a genuine need to respect and 
preserve prior firmware configuration of the device, then I would surely 
expect to see the driver actually doing exactly that. Presumably: at 
probe time, look at TRACE_CTRL; if the device is already configured, 
read out that configuration - especially including TRACE_ADDR_* - and 
make sure to reuse it. Not go off on a tangent blindly poking into 
internal IOMMU API abstractions in the vain hope that the first thing 
you find happens to be sort-of-related to the information that you 
actually care about.

Thanks,
Robin.

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

On 2021/11/25 23:49, Robin Murphy wrote:
> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>> Hi Robin,
>>
>> On 2021/11/16 19:37, Yicong Yang wrote:
>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>> [...]
>>>>> +/*
>>>>> + * Get RMR address if provided by the firmware.
>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>> + * Otherwise a negative value is returned.
>>>>> + */
>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +    struct iommu_domain *iommu_domain;
>>>>> +    struct iommu_resv_region *region;
>>>>> +    LIST_HEAD(list);
>>>>> +
>>>>> +    /*
>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>> +     * IOMMU domain is passthrough.
>>>>> +     */
>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>> +        return 0;
>>>>> +
>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>> +    list_for_each_entry(region, &list, list)
>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>> +            break;
>>>>> +        }
>>>>> +
>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>> +}
>>>>
>>>> No.
>>>>
>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>
>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>
>>>
>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>
>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>
>>
>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
> 
> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
> 
> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
> 
> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
> 
> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
> 
> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
> 

Thanks for the further explanation and I think I understand your concerns more clearer.

The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
the trace and do boot time tracing.

> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
> 

Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.

> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
> 

Yes, we do need RMR to make the device work at situation where the mapping is non-identical.

We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
firmware configure the TRACE_ADDR_* to the device?

As suggested, I think I'll need to modify the RMR codes like

- check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
- if smmu is not presented, use direct DMA
- try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
- check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
- if no rmr but smmu works in passthrough mode, use direct DMA
- otherwise fails the probe

If I miss something please point it out.

Thanks,
Yicong






_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/25 23:49, Robin Murphy wrote:
> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>> Hi Robin,
>>
>> On 2021/11/16 19:37, Yicong Yang wrote:
>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>> [...]
>>>>> +/*
>>>>> + * Get RMR address if provided by the firmware.
>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>> + * Otherwise a negative value is returned.
>>>>> + */
>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +    struct iommu_domain *iommu_domain;
>>>>> +    struct iommu_resv_region *region;
>>>>> +    LIST_HEAD(list);
>>>>> +
>>>>> +    /*
>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>> +     * IOMMU domain is passthrough.
>>>>> +     */
>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>> +        return 0;
>>>>> +
>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>> +    list_for_each_entry(region, &list, list)
>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>> +            break;
>>>>> +        }
>>>>> +
>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>> +}
>>>>
>>>> No.
>>>>
>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>
>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>
>>>
>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>
>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>
>>
>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
> 
> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
> 
> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
> 
> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
> 
> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
> 
> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
> 

Thanks for the further explanation and I think I understand your concerns more clearer.

The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
the trace and do boot time tracing.

> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
> 

Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.

> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
> 

Yes, we do need RMR to make the device work at situation where the mapping is non-identical.

We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
firmware configure the TRACE_ADDR_* to the device?

As suggested, I think I'll need to modify the RMR codes like

- check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
- if smmu is not presented, use direct DMA
- try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
- check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
- if no rmr but smmu works in passthrough mode, use direct DMA
- otherwise fails the probe

If I miss something please point it out.

Thanks,
Yicong







_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/25 23:49, Robin Murphy wrote:
> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>> Hi Robin,
>>
>> On 2021/11/16 19:37, Yicong Yang wrote:
>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>> [...]
>>>>> +/*
>>>>> + * Get RMR address if provided by the firmware.
>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>> + * Otherwise a negative value is returned.
>>>>> + */
>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +    struct iommu_domain *iommu_domain;
>>>>> +    struct iommu_resv_region *region;
>>>>> +    LIST_HEAD(list);
>>>>> +
>>>>> +    /*
>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>> +     * IOMMU domain is passthrough.
>>>>> +     */
>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>> +        return 0;
>>>>> +
>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>> +    list_for_each_entry(region, &list, list)
>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>> +            break;
>>>>> +        }
>>>>> +
>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>> +}
>>>>
>>>> No.
>>>>
>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>
>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>
>>>
>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>
>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>
>>
>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
> 
> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
> 
> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
> 
> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
> 
> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
> 
> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
> 

Thanks for the further explanation and I think I understand your concerns more clearer.

The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
the trace and do boot time tracing.

> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
> 

Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.

> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
> 

Yes, we do need RMR to make the device work at situation where the mapping is non-identical.

We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
firmware configure the TRACE_ADDR_* to the device?

As suggested, I think I'll need to modify the RMR codes like

- check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
- if smmu is not presented, use direct DMA
- try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
- check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
- if no rmr but smmu works in passthrough mode, use direct DMA
- otherwise fails the probe

If I miss something please point it out.

Thanks,
Yicong

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-29 08:22, Yicong Yang via iommu wrote:
> On 2021/11/25 23:49, Robin Murphy wrote:
>> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>>> Hi Robin,
>>>
>>> On 2021/11/16 19:37, Yicong Yang wrote:
>>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>>> [...]
>>>>>> +/*
>>>>>> + * Get RMR address if provided by the firmware.
>>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>>> + * Otherwise a negative value is returned.
>>>>>> + */
>>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>>> +{
>>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>>> +    struct iommu_domain *iommu_domain;
>>>>>> +    struct iommu_resv_region *region;
>>>>>> +    LIST_HEAD(list);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>>> +     * IOMMU domain is passthrough.
>>>>>> +     */
>>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>>> +        return 0;
>>>>>> +
>>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>>> +    list_for_each_entry(region, &list, list)
>>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>>> +            break;
>>>>>> +        }
>>>>>> +
>>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>>> +}
>>>>>
>>>>> No.
>>>>>
>>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>>
>>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>>
>>>>
>>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>>
>>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>>
>>>
>>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
>>
>> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
>>
>> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
>>
>> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
>>
>> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
>>
>> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
>>
> 
> Thanks for the further explanation and I think I understand your concerns more clearer.
> 
> The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
> So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
> is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
> to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
> the trace and do boot time tracing.
> 
>> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
>>
> 
> Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.
> 
>> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
>>
> 
> Yes, we do need RMR to make the device work at situation where the mapping is non-identical.
> 
> We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
> firmware configure the TRACE_ADDR_* to the device?
> 
> As suggested, I think I'll need to modify the RMR codes like
> 
> - check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
> - if smmu is not presented, use direct DMA
> - try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
> - check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
> - if no rmr but smmu works in passthrough mode, use direct DMA
> - otherwise fails the probe
> 
> If I miss something please point it out.

Thanks for clarifying. Unfortunately it also confirms my suspicion that 
this is exactly the kind of misuse of RMRs that we don't want to 
support. You can ignore most of what I said above which applies to the 
genuine RMR use-case of the device already being configured.

If the device really can't handle SMMU translation then that can be 
dealt with entirely within Linux. Give it a iommu_def_domain_type quirk 
to force passthrough; or maybe fail probe if a DMA domain is present and 
tell the user to change the domain type via sysfs manually; or maybe set 
up your own IOMMU domain and manually map things 1:1 if you really want 
to; there are plenty of possible options for implementing that kind of 
internal software policy. Abusing external firmware mechanisms is not a 
reasonable one, however.

I also can't help be curious as to exactly *why* the device doesn't work 
with translation. If it's an RCiEP with some different path to memory 
that physically bypasses the SMMU compared to "normal" PCIe traffic, 
then that should be fixed by having the IORT mappings describe the 
underlying topology correctly in the first place. If it turns out just 
to be the case that the device only actually drives enough address bits 
to cover the physical memory map, and using translation happens to 
result in IOVAs larger than that which then get truncated and go wrong, 
that's fixed by simply setting the right DMA mask in the driver. If it's 
some complicated interconnect latency/deadlock thing related to 
translation delays as traffic flows through the SMMU, I'd expect that to 
matter regardless of whether the input address happens to match the 
output address or not. At this point I'm starting to get slightly 
suspicious of whether we're even trying to solve the right problem at all.

Thanks,
Robin.

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-29 08:22, Yicong Yang via iommu wrote:
> On 2021/11/25 23:49, Robin Murphy wrote:
>> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>>> Hi Robin,
>>>
>>> On 2021/11/16 19:37, Yicong Yang wrote:
>>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>>> [...]
>>>>>> +/*
>>>>>> + * Get RMR address if provided by the firmware.
>>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>>> + * Otherwise a negative value is returned.
>>>>>> + */
>>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>>> +{
>>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>>> +    struct iommu_domain *iommu_domain;
>>>>>> +    struct iommu_resv_region *region;
>>>>>> +    LIST_HEAD(list);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>>> +     * IOMMU domain is passthrough.
>>>>>> +     */
>>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>>> +        return 0;
>>>>>> +
>>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>>> +    list_for_each_entry(region, &list, list)
>>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>>> +            break;
>>>>>> +        }
>>>>>> +
>>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>>> +}
>>>>>
>>>>> No.
>>>>>
>>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>>
>>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>>
>>>>
>>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>>
>>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>>
>>>
>>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
>>
>> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
>>
>> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
>>
>> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
>>
>> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
>>
>> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
>>
> 
> Thanks for the further explanation and I think I understand your concerns more clearer.
> 
> The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
> So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
> is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
> to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
> the trace and do boot time tracing.
> 
>> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
>>
> 
> Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.
> 
>> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
>>
> 
> Yes, we do need RMR to make the device work at situation where the mapping is non-identical.
> 
> We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
> firmware configure the TRACE_ADDR_* to the device?
> 
> As suggested, I think I'll need to modify the RMR codes like
> 
> - check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
> - if smmu is not presented, use direct DMA
> - try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
> - check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
> - if no rmr but smmu works in passthrough mode, use direct DMA
> - otherwise fails the probe
> 
> If I miss something please point it out.

Thanks for clarifying. Unfortunately it also confirms my suspicion that 
this is exactly the kind of misuse of RMRs that we don't want to 
support. You can ignore most of what I said above which applies to the 
genuine RMR use-case of the device already being configured.

If the device really can't handle SMMU translation then that can be 
dealt with entirely within Linux. Give it a iommu_def_domain_type quirk 
to force passthrough; or maybe fail probe if a DMA domain is present and 
tell the user to change the domain type via sysfs manually; or maybe set 
up your own IOMMU domain and manually map things 1:1 if you really want 
to; there are plenty of possible options for implementing that kind of 
internal software policy. Abusing external firmware mechanisms is not a 
reasonable one, however.

I also can't help be curious as to exactly *why* the device doesn't work 
with translation. If it's an RCiEP with some different path to memory 
that physically bypasses the SMMU compared to "normal" PCIe traffic, 
then that should be fixed by having the IORT mappings describe the 
underlying topology correctly in the first place. If it turns out just 
to be the case that the device only actually drives enough address bits 
to cover the physical memory map, and using translation happens to 
result in IOVAs larger than that which then get truncated and go wrong, 
that's fixed by simply setting the right DMA mask in the driver. If it's 
some complicated interconnect latency/deadlock thing related to 
translation delays as traffic flows through the SMMU, I'd expect that to 
matter regardless of whether the input address happens to match the 
output address or not. At this point I'm starting to get slightly 
suspicious of whether we're even trying to solve the right problem at all.

Thanks,
Robin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Robin Murphy]

On 2021-11-29 08:22, Yicong Yang via iommu wrote:
> On 2021/11/25 23:49, Robin Murphy wrote:
>> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>>> Hi Robin,
>>>
>>> On 2021/11/16 19:37, Yicong Yang wrote:
>>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>>> [...]
>>>>>> +/*
>>>>>> + * Get RMR address if provided by the firmware.
>>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>>> + * Otherwise a negative value is returned.
>>>>>> + */
>>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>>> +{
>>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>>> +    struct iommu_domain *iommu_domain;
>>>>>> +    struct iommu_resv_region *region;
>>>>>> +    LIST_HEAD(list);
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>>> +     * IOMMU domain is passthrough.
>>>>>> +     */
>>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>>> +        return 0;
>>>>>> +
>>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>>> +    list_for_each_entry(region, &list, list)
>>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>>> +            break;
>>>>>> +        }
>>>>>> +
>>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>>> +}
>>>>>
>>>>> No.
>>>>>
>>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>>
>>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>>
>>>>
>>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>>
>>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>>
>>>
>>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
>>
>> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
>>
>> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
>>
>> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
>>
>> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
>>
>> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
>>
> 
> Thanks for the further explanation and I think I understand your concerns more clearer.
> 
> The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
> So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
> is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
> to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
> the trace and do boot time tracing.
> 
>> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
>>
> 
> Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.
> 
>> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
>>
> 
> Yes, we do need RMR to make the device work at situation where the mapping is non-identical.
> 
> We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
> firmware configure the TRACE_ADDR_* to the device?
> 
> As suggested, I think I'll need to modify the RMR codes like
> 
> - check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
> - if smmu is not presented, use direct DMA
> - try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
> - check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
> - if no rmr but smmu works in passthrough mode, use direct DMA
> - otherwise fails the probe
> 
> If I miss something please point it out.

Thanks for clarifying. Unfortunately it also confirms my suspicion that 
this is exactly the kind of misuse of RMRs that we don't want to 
support. You can ignore most of what I said above which applies to the 
genuine RMR use-case of the device already being configured.

If the device really can't handle SMMU translation then that can be 
dealt with entirely within Linux. Give it a iommu_def_domain_type quirk 
to force passthrough; or maybe fail probe if a DMA domain is present and 
tell the user to change the domain type via sysfs manually; or maybe set 
up your own IOMMU domain and manually map things 1:1 if you really want 
to; there are plenty of possible options for implementing that kind of 
internal software policy. Abusing external firmware mechanisms is not a 
reasonable one, however.

I also can't help be curious as to exactly *why* the device doesn't work 
with translation. If it's an RCiEP with some different path to memory 
that physically bypasses the SMMU compared to "normal" PCIe traffic, 
then that should be fixed by having the IORT mappings describe the 
underlying topology correctly in the first place. If it turns out just 
to be the case that the device only actually drives enough address bits 
to cover the physical memory map, and using translation happens to 
result in IOVAs larger than that which then get truncated and go wrong, 
that's fixed by simply setting the right DMA mask in the driver. If it's 
some complicated interconnect latency/deadlock thing related to 
translation delays as traffic flows through the SMMU, I'd expect that to 
matter regardless of whether the input address happens to match the 
output address or not. At this point I'm starting to get slightly 
suspicious of whether we're even trying to solve the right problem at all.

Thanks,
Robin.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/12/8 0:31, Robin Murphy wrote:
> On 2021-11-29 08:22, Yicong Yang via iommu wrote:
>> On 2021/11/25 23:49, Robin Murphy wrote:
>>> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>>>> Hi Robin,
>>>>
>>>> On 2021/11/16 19:37, Yicong Yang wrote:
>>>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>>>> [...]
>>>>>>> +/*
>>>>>>> + * Get RMR address if provided by the firmware.
>>>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>>>> + * Otherwise a negative value is returned.
>>>>>>> + */
>>>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>>>> +{
>>>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>>>> +    struct iommu_domain *iommu_domain;
>>>>>>> +    struct iommu_resv_region *region;
>>>>>>> +    LIST_HEAD(list);
>>>>>>> +
>>>>>>> +    /*
>>>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>>>> +     * IOMMU domain is passthrough.
>>>>>>> +     */
>>>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>>>> +        return 0;
>>>>>>> +
>>>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>>>> +    list_for_each_entry(region, &list, list)
>>>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>>>> +            break;
>>>>>>> +        }
>>>>>>> +
>>>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>>>> +}
>>>>>>
>>>>>> No.
>>>>>>
>>>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>>>
>>>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>>>
>>>>>
>>>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>>>
>>>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>>>
>>>>
>>>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
>>>
>>> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
>>>
>>> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
>>>
>>> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
>>>
>>> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
>>>
>>> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
>>>
>>
>> Thanks for the further explanation and I think I understand your concerns more clearer.
>>
>> The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
>> So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
>> is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
>> to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
>> the trace and do boot time tracing.
>>
>>> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
>>>
>>
>> Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.
>>
>>> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
>>>
>>
>> Yes, we do need RMR to make the device work at situation where the mapping is non-identical.
>>
>> We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
>> firmware configure the TRACE_ADDR_* to the device?
>>
>> As suggested, I think I'll need to modify the RMR codes like
>>
>> - check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
>> - if smmu is not presented, use direct DMA
>> - try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
>> - check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
>> - if no rmr but smmu works in passthrough mode, use direct DMA
>> - otherwise fails the probe
>>
>> If I miss something please point it out.
> 
> Thanks for clarifying. Unfortunately it also confirms my suspicion that this is exactly the kind of misuse of RMRs that we don't want to support. You can ignore most of what I said above which applies to the genuine RMR use-case of the device already being configured.
> 
> If the device really can't handle SMMU translation then that can be dealt with entirely within Linux. Give it a iommu_def_domain_type quirk to force passthrough; or maybe fail probe if a DMA domain is present and tell the user to change the domain type via sysfs manually; or maybe set up your own IOMMU domain and manually map things 1:1 if you really want to; there are plenty of possible options for implementing that kind of internal software policy. Abusing external firmware mechanisms is not a reasonable one, however.
> > I also can't help be curious as to exactly *why* the device doesn't work with translation. If it's an RCiEP with some different path to memory that physically bypasses the SMMU compared to "normal" PCIe traffic, then that should be fixed by having the IORT mappings describe the underlying topology correctly in the first place. If it turns out just to be the case that the device only actually drives enough address bits to cover the physical memory map, and using translation happens to result in IOVAs larger than that which then get truncated and go wrong, that's fixed by simply setting the right DMA mask in the driver. If it's some complicated interconnect latency/deadlock thing related to translation delays as traffic flows through the SMMU, I'd expect that to matter regardless of whether the input address happens to match the output address or not. At this point I'm starting to get slightly suspicious of whether we're even trying to solve the right problem at all.
> 

Thanks a lot for the kind suggestions. Seems we're not using RMR properly and it's ok for us to solve this problem in the kernel side, thanks for point
it out.

The implementation of PTT device is like what you mentioned that the DMA from PTT is bypassed by the SMMU. I need more knowledge of IORT to see whether
we can get this fixed from the IORT side and have a test if it's a possible solution before I can reply. And I'll try and test if we can have a
iommu_def_domain_type quirk to deal with this hardware limitation.

Regards,
Yicong

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/12/8 0:31, Robin Murphy wrote:
> On 2021-11-29 08:22, Yicong Yang via iommu wrote:
>> On 2021/11/25 23:49, Robin Murphy wrote:
>>> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>>>> Hi Robin,
>>>>
>>>> On 2021/11/16 19:37, Yicong Yang wrote:
>>>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>>>> [...]
>>>>>>> +/*
>>>>>>> + * Get RMR address if provided by the firmware.
>>>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>>>> + * Otherwise a negative value is returned.
>>>>>>> + */
>>>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>>>> +{
>>>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>>>> +    struct iommu_domain *iommu_domain;
>>>>>>> +    struct iommu_resv_region *region;
>>>>>>> +    LIST_HEAD(list);
>>>>>>> +
>>>>>>> +    /*
>>>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>>>> +     * IOMMU domain is passthrough.
>>>>>>> +     */
>>>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>>>> +        return 0;
>>>>>>> +
>>>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>>>> +    list_for_each_entry(region, &list, list)
>>>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>>>> +            break;
>>>>>>> +        }
>>>>>>> +
>>>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>>>> +}
>>>>>>
>>>>>> No.
>>>>>>
>>>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>>>
>>>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>>>
>>>>>
>>>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>>>
>>>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>>>
>>>>
>>>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
>>>
>>> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
>>>
>>> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
>>>
>>> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
>>>
>>> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
>>>
>>> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
>>>
>>
>> Thanks for the further explanation and I think I understand your concerns more clearer.
>>
>> The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
>> So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
>> is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
>> to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
>> the trace and do boot time tracing.
>>
>>> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
>>>
>>
>> Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.
>>
>>> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
>>>
>>
>> Yes, we do need RMR to make the device work at situation where the mapping is non-identical.
>>
>> We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
>> firmware configure the TRACE_ADDR_* to the device?
>>
>> As suggested, I think I'll need to modify the RMR codes like
>>
>> - check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
>> - if smmu is not presented, use direct DMA
>> - try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
>> - check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
>> - if no rmr but smmu works in passthrough mode, use direct DMA
>> - otherwise fails the probe
>>
>> If I miss something please point it out.
> 
> Thanks for clarifying. Unfortunately it also confirms my suspicion that this is exactly the kind of misuse of RMRs that we don't want to support. You can ignore most of what I said above which applies to the genuine RMR use-case of the device already being configured.
> 
> If the device really can't handle SMMU translation then that can be dealt with entirely within Linux. Give it a iommu_def_domain_type quirk to force passthrough; or maybe fail probe if a DMA domain is present and tell the user to change the domain type via sysfs manually; or maybe set up your own IOMMU domain and manually map things 1:1 if you really want to; there are plenty of possible options for implementing that kind of internal software policy. Abusing external firmware mechanisms is not a reasonable one, however.
> > I also can't help be curious as to exactly *why* the device doesn't work with translation. If it's an RCiEP with some different path to memory that physically bypasses the SMMU compared to "normal" PCIe traffic, then that should be fixed by having the IORT mappings describe the underlying topology correctly in the first place. If it turns out just to be the case that the device only actually drives enough address bits to cover the physical memory map, and using translation happens to result in IOVAs larger than that which then get truncated and go wrong, that's fixed by simply setting the right DMA mask in the driver. If it's some complicated interconnect latency/deadlock thing related to translation delays as traffic flows through the SMMU, I'd expect that to matter regardless of whether the input address happens to match the output address or not. At this point I'm starting to get slightly suspicious of whether we're even trying to solve the right problem at all.
> 

Thanks a lot for the kind suggestions. Seems we're not using RMR properly and it's ok for us to solve this problem in the kernel side, thanks for point
it out.

The implementation of PTT device is like what you mentioned that the DMA from PTT is bypassed by the SMMU. I need more knowledge of IORT to see whether
we can get this fixed from the IORT side and have a test if it's a possible solution before I can reply. And I'll try and test if we can have a
iommu_def_domain_type quirk to deal with this hardware limitation.

Regards,
Yicong






_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

On 2021/12/8 0:31, Robin Murphy wrote:
> On 2021-11-29 08:22, Yicong Yang via iommu wrote:
>> On 2021/11/25 23:49, Robin Murphy wrote:
>>> On 2021-11-18 09:01, Yicong Yang via iommu wrote:
>>>> Hi Robin,
>>>>
>>>> On 2021/11/16 19:37, Yicong Yang wrote:
>>>>> On 2021/11/16 18:56, Robin Murphy wrote:
>>>>>> On 2021-11-16 09:06, Yicong Yang via iommu wrote:
>>>>>> [...]
>>>>>>> +/*
>>>>>>> + * Get RMR address if provided by the firmware.
>>>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>>>> + * Otherwise a negative value is returned.
>>>>>>> + */
>>>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>>>> +{
>>>>>>> +    struct pci_dev *pdev = hisi_ptt->pdev;
>>>>>>> +    struct iommu_domain *iommu_domain;
>>>>>>> +    struct iommu_resv_region *region;
>>>>>>> +    LIST_HEAD(list);
>>>>>>> +
>>>>>>> +    /*
>>>>>>> +     * Use direct DMA if IOMMU does not present or the policy of the
>>>>>>> +     * IOMMU domain is passthrough.
>>>>>>> +     */
>>>>>>> +    iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>>>> +    if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>>>> +        return 0;
>>>>>>> +
>>>>>>> +    iommu_get_resv_regions(&pdev->dev, &list);
>>>>>>> +    list_for_each_entry(region, &list, list)
>>>>>>> +        if (region->type == IOMMU_RESV_DIRECT &&
>>>>>>> +            region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>>>> +            hisi_ptt->trace_ctrl.has_rmr = true;
>>>>>>> +            hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>>>> +            hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>>>> +            break;
>>>>>>> +        }
>>>>>>> +
>>>>>>> +    iommu_put_resv_regions(&pdev->dev, &list);
>>>>>>> +    return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>>>> +}
>>>>>>
>>>>>> No.
>>>>>>
>>>>>> The whole point of RMRs is for devices that are already configured to access the given address range in a manner beyond the kernel's control. If you can do this, it proves that you should not have an RMR in the first place.
>>>>>>
>>>>>> The notion of a kernel driver explicitly configuring its device to DMA into any random RMR that looks big enough is so egregiously wrong that I'm almost lost for words...
>>>>>>
>>>>>
>>>>> our bios will reserve such a region and reported it through iort. the device will write to the region and in the driver we need to access the region
>>>>> to get the traced data. the region is reserved exclusively and will not be accessed by kernel or other devices.
>>>>>
>>>>> is it ok to let bios configure the address to the device and from CPU side we just read it?
>>>>>
>>>>
>>>> Any suggestion?  Is this still an issue you concern if we move the configuration of the device address to BIOS and just read from the CPU side?
>>>
>>> If the firmware configures the device so that it's actively tracing and writing out to memory while the kernel boots, then that is a valid reason to have an RMR. However what you're doing in the driver is still complete nonsense. As far as I can follow, the way it's working is this:
>>>
>>> - At probe time, the initial state of the hardware is entirely ignored. If it *is* already active, there appears to be a fun chance of crashing if TRACE_INT_MASK is clear and an interrupt happens to fire before anyone has got round to calling perf_aux_output_begin() to make trace_ctrl.handle.rb non-NULL.
>>>
>>> - Later, once the user starts a tracing session, a buffer is set up *either* as a completely normal DMA allocation, or by memremap()ing some random IOVA carveout which may or may not be whatever memory the firmware was tracing to.
>>>
>>> - The hardware is then reset and completely reprogrammed to use the new buffer, again without any consideration of its previous state (other than possibly timing out and failing if it's already running and that means it never goes idle).
>>>
>>> Therefore the driver does not seem to respect any prior configuration of the device by firmware, does not seem to expect it to be running at boot time, does not seem to have any way to preserve and export any trace data captured in an RMR if it *was* running at boot time, and thus without loss of generality could simply use the dma_alloc_coherent() path all the time. Am I missing anything?
>>>
>>
>> Thanks for the further explanation and I think I understand your concerns more clearer.
>>
>> The trace is not supposed to begin by the firmware at boot time. Due to some hardware restriction, the device cannot trace with non-identical mapping.
>> So we'd like to use RMR to make the device work when the dma mapping is non-identical. Thus we check here to decide whether to use RMR or not: if the iommu
>> is not presented or in the passthrough mode, we can use direct DMA by dma_alloc_coherent(); if the iommu is present and the mode is not passthrough, we try
>> to retrieve RMR or we fail the probe. The firmware is expected to reserve a range of memory and reports it to the driver and is not expected to configure
>> the trace and do boot time tracing.
>>
>>> As things stand, RMRs are not yet supported upstream (FYI we're still working on fixing the spec...), so the code above is at best dead, and at worst actively wrong. Furthermore, if the expected usage model *is* that the kernel driver completely resets and reprograms the hardware, then even if there is an RMR for boot-time tracing I would rather expect it to be flagged as remappable, and thus potentially end up as an IOMMU_RESV_DIRECT_RELAXABLE reservation which you wouldn't match anyway.
>>>
>>
>> Yes the firmware is not expected to start the trace. Will change the desired flag to IOMMU_RESV_DIRECT_RELAXABLE and have a test.
>>
>>> And after all that, if you really do have a genuine need to respect and preserve prior firmware configuration of the device, then I would surely expect to see the driver actually doing exactly that. Presumably: at probe time, look at TRACE_CTRL; if the device is already configured, read out that configuration - especially including TRACE_ADDR_* - and make sure to reuse it. Not go off on a tangent blindly poking into internal IOMMU API abstractions in the vain hope that the first thing you find happens to be sort-of-related to the information that you actually care about.
>>>
>>
>> Yes, we do need RMR to make the device work at situation where the mapping is non-identical.
>>
>> We're certain that the bios won't start and configure the trace in this device's usage, is it still necessary to make
>> firmware configure the TRACE_ADDR_* to the device?
>>
>> As suggested, I think I'll need to modify the RMR codes like
>>
>> - check TRACE_CTRL, and stop it if it's started. (won't happen but check for sanity)
>> - if smmu is not presented, use direct DMA
>> - try to retrieve RMR address with flag IOMMU_RESV_DIRECT_RELAXABLE , if presented set hisi_ptt->has_rmr. in this case we won't use direct DMA
>> - check if the TRACE_ADDR_* has been configured. if so don't reconfigure it when trace
>> - if no rmr but smmu works in passthrough mode, use direct DMA
>> - otherwise fails the probe
>>
>> If I miss something please point it out.
> 
> Thanks for clarifying. Unfortunately it also confirms my suspicion that this is exactly the kind of misuse of RMRs that we don't want to support. You can ignore most of what I said above which applies to the genuine RMR use-case of the device already being configured.
> 
> If the device really can't handle SMMU translation then that can be dealt with entirely within Linux. Give it a iommu_def_domain_type quirk to force passthrough; or maybe fail probe if a DMA domain is present and tell the user to change the domain type via sysfs manually; or maybe set up your own IOMMU domain and manually map things 1:1 if you really want to; there are plenty of possible options for implementing that kind of internal software policy. Abusing external firmware mechanisms is not a reasonable one, however.
> > I also can't help be curious as to exactly *why* the device doesn't work with translation. If it's an RCiEP with some different path to memory that physically bypasses the SMMU compared to "normal" PCIe traffic, then that should be fixed by having the IORT mappings describe the underlying topology correctly in the first place. If it turns out just to be the case that the device only actually drives enough address bits to cover the physical memory map, and using translation happens to result in IOVAs larger than that which then get truncated and go wrong, that's fixed by simply setting the right DMA mask in the driver. If it's some complicated interconnect latency/deadlock thing related to translation delays as traffic flows through the SMMU, I'd expect that to matter regardless of whether the input address happens to match the output address or not. At this point I'm starting to get slightly suspicious of whether we're even trying to solve the right problem at all.
> 

Thanks a lot for the kind suggestions. Seems we're not using RMR properly and it's ok for us to solve this problem in the kernel side, thanks for point
it out.

The implementation of PTT device is like what you mentioned that the DMA from PTT is bypassed by the SMMU. I need more knowledge of IORT to see whether
we can get this fixed from the IORT side and have a test if it's a possible solution before I can reply. And I'll try and test if we can have a
iommu_def_domain_type quirk to deal with this hardware limitation.

Regards,
Yicong





_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

Hi Yicong,

On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> integrated Endpoint(RCiEP) device, providing the capability
> to dynamically monitor and tune the PCIe traffic(tune),
> and trace the TLP headers(trace).
>

Is there a reason to put "tune" and "trace" are whithin parentheses?  

> Add the driver for the device to enable the trace function. The driver
> will create PMU device for each PTT device, and users can start trace
> through perf command.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  drivers/Makefile                       |    1 +
>  drivers/hwtracing/Kconfig              |    2 +
>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++

Not sure about the "hisilicon" directory.  Right now we have "coresight",
"intel_th" and "stm" - they concentrate on the technology rather than the
vendor.  I think "ptt" would be just fine: 

drivers/hwtracing/ptt/Kconfig 
drivers/hwtracing/ptt/Makefile
drivers/hwtracing/ptt/hisi_ptt.c

That way if other vendors want to introduce support for the same kind of
technology we can simply put them under "drivers/hwtracing/ptt/" and things are
still accurate.


>  5 files changed, 1159 insertions(+)
>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
> 
> diff --git a/drivers/Makefile b/drivers/Makefile
> index be5d40ae1488..bb0dc9b55ea2 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>  obj-y				+= hwtracing/intel_th/
>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>  obj-$(CONFIG_ANDROID)		+= android/
>  obj-$(CONFIG_NVMEM)		+= nvmem/
>  obj-$(CONFIG_FPGA)		+= fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..e3796b17541a 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>  
>  source "drivers/hwtracing/intel_th/Kconfig"
>  
> +source "drivers/hwtracing/hisilicon/Kconfig"
> +
>  endmenu
> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
> new file mode 100644
> index 000000000000..9c3ab80d99fe
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/Kconfig
> @@ -0,0 +1,8 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> +	tristate "HiSilicon PCIe Tune and Trace Device"
> +	depends on PCI && HAS_DMA && HAS_IOMEM

What Arm architecture are you aiming this for?  Can it be found on both armv7
and armv8?

> +	help
> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
> +	  device, provides support for PCIe traffic tuning and
> +	  tracing TLP headers to the memory.

Please indicate what the name of the driver will be when compiled as a module.

> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
> new file mode 100644
> index 000000000000..e11e9b6cc2a8
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
> @@ -0,0 +1,1146 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2021 HiSilicon Limited.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/kfifo.h>
> +#include <linux/module.h>
> +#include <linux/notifier.h>
> +#include <linux/pci.h>
> +#include <linux/perf_event.h>
> +#include <linux/sizes.h>
> +#include <linux/sysfs.h>
> +#include <linux/vmalloc.h>
> +#include <linux/workqueue.h>
> +
> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> +#define HISI_PTT_TRACE_CTRL		0x0850
> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS		0x08b0
> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> +#define HISI_PTT_LOCATION		0x0fe8
> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)

Extra spaces between #define and the literals.  Please align the values with
that of HISI_PTT_TRACE_DMA_IRQ below.

> +
> +#define HISI_PTT_TRACE_DMA_IRQ			0
> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> +
> +/* Delay time for filter updating work */
> +#define HISI_PTT_WORK_DELAY_MS		100UL
> +/* Wait time for DMA hardware to reset */
> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> +
> +enum hisi_ptt_trace_status {
> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> +	HISI_PTT_TRACE_STATUS_ON,
> +};
> +
> +struct hisi_ptt_dma_buflet {

I'm not sure what a "buflet" is...  Probably best to just call this
hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
know due to the lack of proper structure documentation.  Please have a look at
how "coresight_device" is documented.  The same applies to the rest of the
structures declared below.

> +	struct list_head list;
> +	unsigned int size;
> +	dma_addr_t dma;
> +
> +	/*
> +	 * The address of the buflet holding the trace data.
> +	 * See Documentation/trace/hisi-ptt.rst for the details

As of this writing, hisi-ptt.rst doesn't exist.

> +	 * of the data format.
> +	 */
> +	void *addr;
> +	int index;
> +};
> +
> +struct hisi_ptt_trace_ctrl {
> +	enum hisi_ptt_trace_status status;
> +	struct perf_output_handle handle;
> +	struct list_head trace_buf;
> +	/*
> +	 * The index of the buffer which trace data
> +	 * currently is writing to.
> +	 */
> +	u32 buf_index;
> +
> +	int default_cpu;
> +	u32 buflet_size;
> +	bool is_port;
> +	u32 format:1;		/* Format of the traced TLP headers */
> +	u32 type:4;		/* Type of the TLP headers to trace */
> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> +
> +	phys_addr_t rmr_addr;
> +	size_t rmr_length;
> +	bool has_rmr;
> +};
> +
> +struct hisi_ptt_filter_desc {
> +	struct list_head list;
> +	struct pci_dev *pdev;
> +	u16 val;
> +};
> +
> +struct hisi_ptt_pmu_buf {
> +	size_t length;
> +	int nr_pages;
> +	void *base;
> +	long pos;
> +};
> +
> +/* Structure containing the information for filter updating */
> +struct hisi_ptt_filter_update_info {
> +	struct pci_dev *pdev;
> +	u32 port_devid;
> +	bool is_port;
> +	bool is_add;
> +	u16 val;
> +};
> +
> +struct hisi_ptt {
> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> +	struct notifier_block hisi_ptt_nb;
> +	struct pmu hisi_ptt_pmu;
> +	void __iomem *iobase;
> +	struct pci_dev *pdev;
> +	/*
> +	 * Use the mutex to protect the filter list and
> +	 * serialize the perf process.
> +	 */
> +	struct mutex mutex;
> +	const char *name;
> +	u16 core_id;
> +	u16 sicl_id;
> +	/* PCI device range managed by the PTT device */
> +	u32 upper;
> +	u32 lower;
> +	u8 busnr;
> +
> +	/*
> +	 * The trace TLP headers can either be filtered by certain
> +	 * root port, or by the requester ID. Organize the filters
> +	 * by @port_filters and @req_filters here. The mask of all
> +	 * the valid ports is also cached for doing sanity check
> +	 * of user input.
> +	 */
> +	struct list_head port_filters;
> +	struct list_head req_filters;
> +	u16 port_mask;
> +
> +	/*
> +	 * We use a delayed work here to avoid indefinitely waiting for
> +	 * the hisi_ptt->mutex which protecting the filter list. The
> +	 * work will be delayed only if the mutex can not be held,
> +	 * otherwise no delay will be applied.
> +	 */
> +	struct delayed_work work;
> +	spinlock_t filter_update_lock;
> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> +};

It might be a good idea to spinoff a hisi_ptt.h for the above declarations.

> +
> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> +{
> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> +}

I'm not sure there is any value to this inline function.  I suggest to simply
call container_of() whenever it is needed.

> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> +				  val & HISI_PTT_TRACE_IDLE,
> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> +				  HISI_PTT_WAIT_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> +
> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> +		list_del(&buflet->list);
> +
> +		if (ctrl->has_rmr)
> +			memunmap(buflet->addr);
> +		else
> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> +					  buflet->dma);
> +
> +		kfree(buflet);
> +	}
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	struct hisi_ptt_dma_buflet *buflet;
> +	int i, ret;
> +
> +	hisi_ptt->trace_ctrl.buf_index = 0;
> +
> +	/* Make sure the trace buffer is empty before allocating */
> +	if (!list_empty(&ctrl->trace_buf)) {
> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> +			memset(buflet->addr, 0, buflet->size);
> +		return 0;
> +	}
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> +		if (!buflet) {
> +			ret = -ENOMEM;
> +			goto err;
> +		}
> +
> +		if (ctrl->has_rmr) {
> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> +
> +			buflet->dma = base;
> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> +		} else {
> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> +							  &buflet->dma, GFP_KERNEL);
> +		}
> +
> +		if (!buflet->addr) {
> +			kfree(buflet);
> +			ret = -ENOMEM;
> +			goto err;
> +		}
> +
> +		memset(buflet->addr, 0, buflet->size);
> +
> +		buflet->index = i;
> +		buflet->size = ctrl->buflet_size;
> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> +	}
> +
> +	return 0;
> +err:
> +	hisi_ptt_free_trace_buf(hisi_ptt);
> +	return ret;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct hisi_ptt_dma_buflet *cur;
> +	u32 val;
> +
> +	/* Check device idle before start trace */
> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> +		return -EBUSY;
> +	}
> +
> +	/* Reset the DMA before start tracing */
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val |= HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/*
> +	 * We'll be in the perf context where preemption is disabled,
> +	 * so use busy loop here.
> +	 */
> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> +
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/* clear the interrupt status */
> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> +		writel(lower_32_bits(cur->dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> +		writel(upper_32_bits(cur->dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> +	}
> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> +	/* set the trace control register */
> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> +	if (!hisi_ptt->trace_ctrl.is_port)
> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> +	val |= HISI_PTT_TRACE_CTRL_EN;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> +{
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct perf_event *event = handle->event;
> +	struct hisi_ptt_dma_buflet *cur;
> +	struct hisi_ptt_pmu_buf *buf;
> +
> +	buf = perf_get_aux(handle);
> +	if (!buf || !handle->size)
> +		return -EINVAL;
> +
> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> +		if (cur->index == index)
> +			break;
> +
> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> +	memset(cur->addr, 0, cur->size);
> +	buf->pos += cur->size;
> +
> +	if (stop) {
> +		perf_aux_output_end(handle, buf->pos);
> +	} else if (buf->length - buf->pos < cur->size) {
> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> +		perf_aux_output_end(handle, buf->pos);
> +
> +		buf = perf_aux_output_begin(handle, event);
> +		if (!buf)
> +			return -EINVAL;
> +
> +		buf->pos = handle->head % buf->length;
> +		if (buf->length - buf->pos < cur->size) {
> +			perf_aux_output_end(handle, 0);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status, buf_idx;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	buf_idx = ffs(status) - 1;
> +
> +	/* Clear the interrupt status of buflet @buf_idx */
> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> +	/*
> +	 * Update the AUX buffer and cache the current buflet index,
> +	 * as we need to know this and save the data when the trace
> +	 * is ended out of the interrupt handler. End the trace
> +	 * if the updating fails.
> +	 */
> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> +		hisi_ptt_trace_end(hisi_ptt);
> +	else
> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> +		return IRQ_NONE;
> +
> +	return IRQ_WAKE_THREAD;
> +}
> +
> +static void hisi_ptt_irq_free_vectors(void *pdev)
> +{
> +	pci_free_irq_vectors(pdev);
> +}
> +
> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	int ret;
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> +	if (ret < 0) {
> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = devm_request_threaded_irq(&pdev->dev,
> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> +					"hisi-ptt", hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_update_filters(struct work_struct *work)
> +{
> +	struct delayed_work *delayed_work = to_delayed_work(work);
> +	struct hisi_ptt_filter_update_info info;
> +	struct hisi_ptt_filter_desc *filter;
> +	struct list_head *target_list;
> +	struct hisi_ptt *hisi_ptt;
> +
> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> +
> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> +		return;
> +	}
> +
> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> +			      &hisi_ptt->req_filters;
> +
> +		if (info.is_add) {
> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> +			if (!filter)
> +				continue;
> +
> +			filter->pdev = info.pdev;
> +			filter->val = info.val;
> +
> +			list_add_tail(&filter->list, target_list);
> +		} else {
> +			list_for_each_entry(filter, target_list, list)
> +				if (filter->val == info.val) {
> +					list_del(&filter->list);
> +					kfree(filter);
> +					break;
> +				}
> +		}
> +
> +		/* Update the available port mask */
> +		if (!info.is_port)
> +			continue;
> +
> +		if (info.is_add)
> +			hisi_ptt->port_mask |= info.val;
> +		else
> +			hisi_ptt->port_mask &= ~info.val;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> +				    struct hisi_ptt_filter_update_info *info)
> +{
> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> +
> +	if (!root_port)
> +		return;
> +
> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> +	if (info->port_devid < hisi_ptt->lower ||
> +	    info->port_devid > hisi_ptt->upper)
> +		return;
> +
> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> +
> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> +				&hisi_ptt->filter_update_lock))
> +		schedule_delayed_work(&hisi_ptt->work, 0);
> +	else
> +		pci_warn(hisi_ptt->pdev,
> +			 "filter update fifo overflow for target %s\n",
> +			 pci_name(info->pdev));
> +}
> +
> +/*
> + * A PCI bus notifier is used here for dynamically updating the filter
> + * list.
> + */
> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> +				  void *data)
> +{
> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> +	struct hisi_ptt_filter_update_info info;
> +	struct device *dev = data;
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +
> +	info.pdev = pdev;
> +
> +	switch (action) {
> +	case BUS_NOTIFY_ADD_DEVICE:
> +		info.is_add = true;
> +		break;
> +	case BUS_NOTIFY_DEL_DEVICE:
> +		info.is_add = false;
> +		break;
> +	default:
> +		return 0;
> +	}
> +
> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> +{
> +	struct hisi_ptt_filter_update_info info = {
> +		.pdev = pdev,
> +		.is_add = true,
> +	};
> +	struct hisi_ptt *hisi_ptt = data;
> +
> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +}
> +
> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct pci_bus *bus;
> +	u32 reg;
> +
> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> +	/*
> +	 * The device range register provides the information about the
> +	 * root ports which the RCiEP can control and trace. The RCiEP
> +	 * and the root ports it support are on the same PCIe core, with
> +	 * same domain number but maybe different bus number. The device
> +	 * range register will tell us which root ports we can support,
> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> +	 * while Bit[15:0] indicates the lower.
> +	 */
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> +	hisi_ptt->upper = reg >> 16;
> +	hisi_ptt->lower = reg & 0xffff;
> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> +
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> +	if (bus)
> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);

What happens if bus is NULL?  Should this be reported to the caller and cause
hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.

> +
> +	/* Initialize trace controls */
> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +}
> +
> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> +
> +static ssize_t available_filters_show(struct device *dev,
> +				      struct device_attribute *attr,
> +				      char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	if (list_empty(&hisi_ptt->port_filters))
> +		return sysfs_emit(buf, "#### No available filter ####\n");
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev) |
> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> +
> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev));
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> +
> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> +PMU_FORMAT_ATTR(type,		"config:24-31");
> +PMU_FORMAT_ATTR(format,		"config:32-35");
> +
> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> +	&format_attr_filter.attr,
> +	&format_attr_direction.attr,
> +	&format_attr_type.attr,
> +	&format_attr_format.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_format_group = {
> +	.name = "format",
> +	.attrs = hisi_ptt_pmu_format_attrs,
> +};
> +
> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> +	&dev_attr_available_filters.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> +	.attrs = hisi_ptt_pmu_filter_attrs,
> +};
> +
> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> +	&hisi_ptt_pmu_format_group,
> +	&hisi_ptt_pmu_filter_group,
> +	NULL
> +};
> +
> +/*
> + * The supported value of the direction parameter. See hisi_ptt.rst
> + * documentation for more details.
> + */
> +static u32 hisi_ptt_trace_available_direction[] = {
> +	0,
> +	1,
> +	2,
> +	3,
> +};
> +
> +/* Different types can be set simultaneously */
> +static u32 hisi_ptt_trace_available_type[] = {
> +	1,	/* posted_request */
> +	2,	/* non-posted_request */
> +	4,	/* completion */
> +};
> +
> +static u32 hisi_ptt_trace_availble_format[] = {
> +	0,	/* 4DW */
> +	1,	/* 8DW */
> +};
> +
> +/*
> + * Check whether the config is valid or not. Some configs are multi-selectable
> + * and can be set simultaneously, while some are single selectable (onehot).
> + * Use this function to check the non-onehot configs while
> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> + */
> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	/*
> +	 * The non-onehot configs cannot be 0. Walk the available
> +	 * list and clear the valid bits of the config. If there
> +	 * is any resident bit after the walk then the config is
> +	 * invalid.
> +	 */
> +	for (i = 0; i < list_size; i++)
> +		val &= ~available_list[i];
> +
> +	return val ? -EINVAL : 0;
> +}
> +
> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> +{
> +	int i, ret = -EINVAL;
> +
> +	for (i = 0; i < list_size; i++)
> +		if (val == available_list[i]) {
> +			ret = 0;
> +			break;
> +		}
> +
> +	return ret;
> +}
> +
> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> +{
> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> +	struct hisi_ptt_filter_desc *filter;
> +	int ret = -EINVAL;
> +
> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> +
> +	/*
> +	 * Port filters are defined as bit mask. For port filters, check
> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> +	 * and whether it's empty or not, otherwise user has specified
> +	 * some unsupported root ports.
> +	 *
> +	 * For Requester ID filters, walk the available filter list to see
> +	 * whether we have one matched.
> +	 */
> +	if (hisi_ptt->trace_ctrl.is_port &&
> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> +		ret = 0;
> +	} else {
> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +			if (filter->val == val) {
> +				ret = 0;
> +				break;
> +			}
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.filter = val;
> +	return 0;
> +}
> +
> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	int ret;
> +	u32 val;
> +
> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> +		return -ENOENT;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +
> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> +	if (ret < 0)
> +		goto out;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->direction = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> +
> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->type = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->format = val;
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return ret;
> +}
> +
> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> +				    int nr_pages, bool overwrite)
> +{
> +	struct hisi_ptt_pmu_buf *buf;
> +	struct page **pagelist;
> +	int i;
> +
> +	if (overwrite) {
> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> +		return NULL;
> +	}
> +
> +	/* If the pages size less than buflets, we cannot start trace */
> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> +		return NULL;
> +
> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> +	if (!buf)
> +		return NULL;
> +
> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> +	if (!pagelist) {
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++)
> +		pagelist[i] = virt_to_page(pages[i]);
> +
> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> +	if (!buf->base) {
> +		kfree(pagelist);
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	buf->nr_pages = nr_pages;
> +	buf->length = nr_pages * PAGE_SIZE;
> +	buf->pos = 0;
> +
> +	kfree(pagelist);
> +	return buf;
> +}
> +
> +static void hisi_ptt_pmu_free_aux(void *aux)
> +{
> +	struct hisi_ptt_pmu_buf *buf = aux;
> +
> +	vunmap(buf->base);
> +	kfree(buf);
> +}
> +
> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct hw_perf_event *hwc = &event->hw;
> +	struct hisi_ptt_pmu_buf *buf;
> +	int cpu = event->cpu;
> +	int ret;
> +
> +	hwc->state = 0;
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> +		goto stop;
> +	}
> +
> +	if (cpu == -1)
> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> +
> +	/*
> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> +	 * core in event_function_local().
> +	 */
> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> +				 cpumask_of(cpu)));
> +
> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> +		goto stop;
> +	}
> +
> +	buf = perf_aux_output_begin(handle, event);
> +	if (!buf) {
> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> +		goto stop;
> +	}
> +
> +	buf->pos = handle->head % buf->length;
> +
> +	ret = hisi_ptt_trace_start(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> +		perf_aux_output_end(handle, 0);
> +		goto stop;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return;
> +stop:
> +	event->hw.state |= PERF_HES_STOPPED;
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +
> +	if (hwc->state & PERF_HES_STOPPED)
> +		return;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> +		hisi_ptt_trace_end(hisi_ptt);
> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> +	}
> +	mutex_unlock(&hisi_ptt->mutex);
> +
> +	hwc->state |= PERF_HES_STOPPED;
> +	perf_event_update_userpage(event);
> +	hwc->state |= PERF_HES_UPTODATE;
> +}
> +
> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +	int cpu = event->cpu;
> +
> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> +		return 0;
> +
> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +
> +	if (flags & PERF_EF_START) {
> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> +		if (hwc->state & PERF_HES_STOPPED)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> +{
> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> +}
> +
> +static void hisi_ptt_unregister_pmu(void *priv)
> +{
> +	perf_pmu_unregister(priv);
> +}
> +
> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> +{
> +	char *pmu_name;
> +	int ret;
> +
> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> +		.module		= THIS_MODULE,
> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> +		.task_ctx_nr	= perf_sw_context,
> +		.attr_groups	= hisi_ptt_pmu_groups,
> +		.event_init	= hisi_ptt_pmu_event_init,
> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> +		.free_aux	= hisi_ptt_pmu_free_aux,
> +		.start		= hisi_ptt_pmu_start,
> +		.stop		= hisi_ptt_pmu_stop,
> +		.add		= hisi_ptt_pmu_add,
> +		.del		= hisi_ptt_pmu_del,
> +	};
> +
> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> +	if (!pmu_name)
> +		return -ENOMEM;
> +
> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> +	if (ret)
> +		return ret;
> +
> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> +					hisi_ptt_unregister_pmu,
> +					&hisi_ptt->hisi_ptt_pmu);
> +}
> +
> +/*
> + * Get RMR address if provided by the firmware.
> + * Return 0 if the IOMMU doesn't present or the policy of the
> + * IOMMU domain is passthrough or we get a usable RMR region.
> + * Otherwise a negative value is returned.
> + */
> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct iommu_domain *iommu_domain;
> +	struct iommu_resv_region *region;
> +	LIST_HEAD(list);
> +
> +	/*
> +	 * Use direct DMA if IOMMU does not present or the policy of the
> +	 * IOMMU domain is passthrough.
> +	 */
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	iommu_get_resv_regions(&pdev->dev, &list);
> +	list_for_each_entry(region, &list, list)
> +		if (region->type == IOMMU_RESV_DIRECT &&
> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> +			hisi_ptt->trace_ctrl.has_rmr = true;
> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> +			break;
> +		}
> +
> +	iommu_put_resv_regions(&pdev->dev, &list);
> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> +}
> +
> +static int hisi_ptt_probe(struct pci_dev *pdev,
> +			  const struct pci_device_id *id)
> +{
> +	struct hisi_ptt *hisi_ptt;
> +	int ret;
> +
> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> +	if (!hisi_ptt)
> +		return -ENOMEM;
> +
> +	mutex_init(&hisi_ptt->mutex);
> +	hisi_ptt->pdev = pdev;
> +
> +	/*
> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> +	 * so directly reference to the pci name string.
> +	 */
> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> +	pci_set_drvdata(pdev, hisi_ptt);
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> +	if (ret) {
> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> +
> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> +	if (ret) {
> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> +		return ret;
> +	}
> +	pci_set_master(pdev);
> +
> +	ret = hisi_ptt_register_irq(hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt_init_ctrls(hisi_ptt);
> +
> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> +		return ret;
> +	}

If this fails the pci_free_irq_vectors() is not called.

I am out of time for today - I will continue tomorrow.

Thanks,
Mathieu

> +
> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> +		return ret;
> +	}
> +
> +	/* Register the bus notifier for dynamically updating the filter list */
> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> +	if (ret)
> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> +
> +	return 0;
> +}
> +
> +void hisi_ptt_remove(struct pci_dev *pdev)
> +{
> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> +
> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> +
> +	/* Cancel any work that has been queued */
> +	cancel_delayed_work_sync(&hisi_ptt->work);
> +
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> +		hisi_ptt_trace_end(hisi_ptt);
> +
> +	hisi_ptt_free_trace_buf(hisi_ptt);
> +	hisi_ptt_release_filters(hisi_ptt);
> +}
> +
> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> +	{ }
> +};
> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> +
> +static struct pci_driver hisi_ptt_driver = {
> +	.name = "hisi_ptt",
> +	.id_table = hisi_ptt_id_tbl,
> +	.probe = hisi_ptt_probe,
> +	.remove = hisi_ptt_remove,
> +};
> +module_pci_driver(hisi_ptt_driver);
> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> -- 
> 2.33.0
> 

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

Hi Yicong,

On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> integrated Endpoint(RCiEP) device, providing the capability
> to dynamically monitor and tune the PCIe traffic(tune),
> and trace the TLP headers(trace).
>

Is there a reason to put "tune" and "trace" are whithin parentheses?  

> Add the driver for the device to enable the trace function. The driver
> will create PMU device for each PTT device, and users can start trace
> through perf command.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  drivers/Makefile                       |    1 +
>  drivers/hwtracing/Kconfig              |    2 +
>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++

Not sure about the "hisilicon" directory.  Right now we have "coresight",
"intel_th" and "stm" - they concentrate on the technology rather than the
vendor.  I think "ptt" would be just fine: 

drivers/hwtracing/ptt/Kconfig 
drivers/hwtracing/ptt/Makefile
drivers/hwtracing/ptt/hisi_ptt.c

That way if other vendors want to introduce support for the same kind of
technology we can simply put them under "drivers/hwtracing/ptt/" and things are
still accurate.


>  5 files changed, 1159 insertions(+)
>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
> 
> diff --git a/drivers/Makefile b/drivers/Makefile
> index be5d40ae1488..bb0dc9b55ea2 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>  obj-y				+= hwtracing/intel_th/
>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>  obj-$(CONFIG_ANDROID)		+= android/
>  obj-$(CONFIG_NVMEM)		+= nvmem/
>  obj-$(CONFIG_FPGA)		+= fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..e3796b17541a 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>  
>  source "drivers/hwtracing/intel_th/Kconfig"
>  
> +source "drivers/hwtracing/hisilicon/Kconfig"
> +
>  endmenu
> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
> new file mode 100644
> index 000000000000..9c3ab80d99fe
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/Kconfig
> @@ -0,0 +1,8 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> +	tristate "HiSilicon PCIe Tune and Trace Device"
> +	depends on PCI && HAS_DMA && HAS_IOMEM

What Arm architecture are you aiming this for?  Can it be found on both armv7
and armv8?

> +	help
> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
> +	  device, provides support for PCIe traffic tuning and
> +	  tracing TLP headers to the memory.

Please indicate what the name of the driver will be when compiled as a module.

> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
> new file mode 100644
> index 000000000000..e11e9b6cc2a8
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
> @@ -0,0 +1,1146 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2021 HiSilicon Limited.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/kfifo.h>
> +#include <linux/module.h>
> +#include <linux/notifier.h>
> +#include <linux/pci.h>
> +#include <linux/perf_event.h>
> +#include <linux/sizes.h>
> +#include <linux/sysfs.h>
> +#include <linux/vmalloc.h>
> +#include <linux/workqueue.h>
> +
> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> +#define HISI_PTT_TRACE_CTRL		0x0850
> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS		0x08b0
> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> +#define HISI_PTT_LOCATION		0x0fe8
> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)

Extra spaces between #define and the literals.  Please align the values with
that of HISI_PTT_TRACE_DMA_IRQ below.

> +
> +#define HISI_PTT_TRACE_DMA_IRQ			0
> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> +
> +/* Delay time for filter updating work */
> +#define HISI_PTT_WORK_DELAY_MS		100UL
> +/* Wait time for DMA hardware to reset */
> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> +
> +enum hisi_ptt_trace_status {
> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> +	HISI_PTT_TRACE_STATUS_ON,
> +};
> +
> +struct hisi_ptt_dma_buflet {

I'm not sure what a "buflet" is...  Probably best to just call this
hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
know due to the lack of proper structure documentation.  Please have a look at
how "coresight_device" is documented.  The same applies to the rest of the
structures declared below.

> +	struct list_head list;
> +	unsigned int size;
> +	dma_addr_t dma;
> +
> +	/*
> +	 * The address of the buflet holding the trace data.
> +	 * See Documentation/trace/hisi-ptt.rst for the details

As of this writing, hisi-ptt.rst doesn't exist.

> +	 * of the data format.
> +	 */
> +	void *addr;
> +	int index;
> +};
> +
> +struct hisi_ptt_trace_ctrl {
> +	enum hisi_ptt_trace_status status;
> +	struct perf_output_handle handle;
> +	struct list_head trace_buf;
> +	/*
> +	 * The index of the buffer which trace data
> +	 * currently is writing to.
> +	 */
> +	u32 buf_index;
> +
> +	int default_cpu;
> +	u32 buflet_size;
> +	bool is_port;
> +	u32 format:1;		/* Format of the traced TLP headers */
> +	u32 type:4;		/* Type of the TLP headers to trace */
> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> +
> +	phys_addr_t rmr_addr;
> +	size_t rmr_length;
> +	bool has_rmr;
> +};
> +
> +struct hisi_ptt_filter_desc {
> +	struct list_head list;
> +	struct pci_dev *pdev;
> +	u16 val;
> +};
> +
> +struct hisi_ptt_pmu_buf {
> +	size_t length;
> +	int nr_pages;
> +	void *base;
> +	long pos;
> +};
> +
> +/* Structure containing the information for filter updating */
> +struct hisi_ptt_filter_update_info {
> +	struct pci_dev *pdev;
> +	u32 port_devid;
> +	bool is_port;
> +	bool is_add;
> +	u16 val;
> +};
> +
> +struct hisi_ptt {
> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> +	struct notifier_block hisi_ptt_nb;
> +	struct pmu hisi_ptt_pmu;
> +	void __iomem *iobase;
> +	struct pci_dev *pdev;
> +	/*
> +	 * Use the mutex to protect the filter list and
> +	 * serialize the perf process.
> +	 */
> +	struct mutex mutex;
> +	const char *name;
> +	u16 core_id;
> +	u16 sicl_id;
> +	/* PCI device range managed by the PTT device */
> +	u32 upper;
> +	u32 lower;
> +	u8 busnr;
> +
> +	/*
> +	 * The trace TLP headers can either be filtered by certain
> +	 * root port, or by the requester ID. Organize the filters
> +	 * by @port_filters and @req_filters here. The mask of all
> +	 * the valid ports is also cached for doing sanity check
> +	 * of user input.
> +	 */
> +	struct list_head port_filters;
> +	struct list_head req_filters;
> +	u16 port_mask;
> +
> +	/*
> +	 * We use a delayed work here to avoid indefinitely waiting for
> +	 * the hisi_ptt->mutex which protecting the filter list. The
> +	 * work will be delayed only if the mutex can not be held,
> +	 * otherwise no delay will be applied.
> +	 */
> +	struct delayed_work work;
> +	spinlock_t filter_update_lock;
> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> +};

It might be a good idea to spinoff a hisi_ptt.h for the above declarations.

> +
> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> +{
> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> +}

I'm not sure there is any value to this inline function.  I suggest to simply
call container_of() whenever it is needed.

> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> +				  val & HISI_PTT_TRACE_IDLE,
> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> +				  HISI_PTT_WAIT_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> +
> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> +		list_del(&buflet->list);
> +
> +		if (ctrl->has_rmr)
> +			memunmap(buflet->addr);
> +		else
> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> +					  buflet->dma);
> +
> +		kfree(buflet);
> +	}
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	struct hisi_ptt_dma_buflet *buflet;
> +	int i, ret;
> +
> +	hisi_ptt->trace_ctrl.buf_index = 0;
> +
> +	/* Make sure the trace buffer is empty before allocating */
> +	if (!list_empty(&ctrl->trace_buf)) {
> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> +			memset(buflet->addr, 0, buflet->size);
> +		return 0;
> +	}
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> +		if (!buflet) {
> +			ret = -ENOMEM;
> +			goto err;
> +		}
> +
> +		if (ctrl->has_rmr) {
> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> +
> +			buflet->dma = base;
> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> +		} else {
> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> +							  &buflet->dma, GFP_KERNEL);
> +		}
> +
> +		if (!buflet->addr) {
> +			kfree(buflet);
> +			ret = -ENOMEM;
> +			goto err;
> +		}
> +
> +		memset(buflet->addr, 0, buflet->size);
> +
> +		buflet->index = i;
> +		buflet->size = ctrl->buflet_size;
> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> +	}
> +
> +	return 0;
> +err:
> +	hisi_ptt_free_trace_buf(hisi_ptt);
> +	return ret;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct hisi_ptt_dma_buflet *cur;
> +	u32 val;
> +
> +	/* Check device idle before start trace */
> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> +		return -EBUSY;
> +	}
> +
> +	/* Reset the DMA before start tracing */
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val |= HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/*
> +	 * We'll be in the perf context where preemption is disabled,
> +	 * so use busy loop here.
> +	 */
> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> +
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/* clear the interrupt status */
> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> +		writel(lower_32_bits(cur->dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> +		writel(upper_32_bits(cur->dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> +	}
> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> +	/* set the trace control register */
> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> +	if (!hisi_ptt->trace_ctrl.is_port)
> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> +	val |= HISI_PTT_TRACE_CTRL_EN;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> +{
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct perf_event *event = handle->event;
> +	struct hisi_ptt_dma_buflet *cur;
> +	struct hisi_ptt_pmu_buf *buf;
> +
> +	buf = perf_get_aux(handle);
> +	if (!buf || !handle->size)
> +		return -EINVAL;
> +
> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> +		if (cur->index == index)
> +			break;
> +
> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> +	memset(cur->addr, 0, cur->size);
> +	buf->pos += cur->size;
> +
> +	if (stop) {
> +		perf_aux_output_end(handle, buf->pos);
> +	} else if (buf->length - buf->pos < cur->size) {
> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> +		perf_aux_output_end(handle, buf->pos);
> +
> +		buf = perf_aux_output_begin(handle, event);
> +		if (!buf)
> +			return -EINVAL;
> +
> +		buf->pos = handle->head % buf->length;
> +		if (buf->length - buf->pos < cur->size) {
> +			perf_aux_output_end(handle, 0);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status, buf_idx;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	buf_idx = ffs(status) - 1;
> +
> +	/* Clear the interrupt status of buflet @buf_idx */
> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> +	/*
> +	 * Update the AUX buffer and cache the current buflet index,
> +	 * as we need to know this and save the data when the trace
> +	 * is ended out of the interrupt handler. End the trace
> +	 * if the updating fails.
> +	 */
> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> +		hisi_ptt_trace_end(hisi_ptt);
> +	else
> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> +		return IRQ_NONE;
> +
> +	return IRQ_WAKE_THREAD;
> +}
> +
> +static void hisi_ptt_irq_free_vectors(void *pdev)
> +{
> +	pci_free_irq_vectors(pdev);
> +}
> +
> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	int ret;
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> +	if (ret < 0) {
> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = devm_request_threaded_irq(&pdev->dev,
> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> +					"hisi-ptt", hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_update_filters(struct work_struct *work)
> +{
> +	struct delayed_work *delayed_work = to_delayed_work(work);
> +	struct hisi_ptt_filter_update_info info;
> +	struct hisi_ptt_filter_desc *filter;
> +	struct list_head *target_list;
> +	struct hisi_ptt *hisi_ptt;
> +
> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> +
> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> +		return;
> +	}
> +
> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> +			      &hisi_ptt->req_filters;
> +
> +		if (info.is_add) {
> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> +			if (!filter)
> +				continue;
> +
> +			filter->pdev = info.pdev;
> +			filter->val = info.val;
> +
> +			list_add_tail(&filter->list, target_list);
> +		} else {
> +			list_for_each_entry(filter, target_list, list)
> +				if (filter->val == info.val) {
> +					list_del(&filter->list);
> +					kfree(filter);
> +					break;
> +				}
> +		}
> +
> +		/* Update the available port mask */
> +		if (!info.is_port)
> +			continue;
> +
> +		if (info.is_add)
> +			hisi_ptt->port_mask |= info.val;
> +		else
> +			hisi_ptt->port_mask &= ~info.val;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> +				    struct hisi_ptt_filter_update_info *info)
> +{
> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> +
> +	if (!root_port)
> +		return;
> +
> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> +	if (info->port_devid < hisi_ptt->lower ||
> +	    info->port_devid > hisi_ptt->upper)
> +		return;
> +
> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> +
> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> +				&hisi_ptt->filter_update_lock))
> +		schedule_delayed_work(&hisi_ptt->work, 0);
> +	else
> +		pci_warn(hisi_ptt->pdev,
> +			 "filter update fifo overflow for target %s\n",
> +			 pci_name(info->pdev));
> +}
> +
> +/*
> + * A PCI bus notifier is used here for dynamically updating the filter
> + * list.
> + */
> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> +				  void *data)
> +{
> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> +	struct hisi_ptt_filter_update_info info;
> +	struct device *dev = data;
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +
> +	info.pdev = pdev;
> +
> +	switch (action) {
> +	case BUS_NOTIFY_ADD_DEVICE:
> +		info.is_add = true;
> +		break;
> +	case BUS_NOTIFY_DEL_DEVICE:
> +		info.is_add = false;
> +		break;
> +	default:
> +		return 0;
> +	}
> +
> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> +{
> +	struct hisi_ptt_filter_update_info info = {
> +		.pdev = pdev,
> +		.is_add = true,
> +	};
> +	struct hisi_ptt *hisi_ptt = data;
> +
> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +}
> +
> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct pci_bus *bus;
> +	u32 reg;
> +
> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> +	/*
> +	 * The device range register provides the information about the
> +	 * root ports which the RCiEP can control and trace. The RCiEP
> +	 * and the root ports it support are on the same PCIe core, with
> +	 * same domain number but maybe different bus number. The device
> +	 * range register will tell us which root ports we can support,
> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> +	 * while Bit[15:0] indicates the lower.
> +	 */
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> +	hisi_ptt->upper = reg >> 16;
> +	hisi_ptt->lower = reg & 0xffff;
> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> +
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> +	if (bus)
> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);

What happens if bus is NULL?  Should this be reported to the caller and cause
hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.

> +
> +	/* Initialize trace controls */
> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +}
> +
> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> +
> +static ssize_t available_filters_show(struct device *dev,
> +				      struct device_attribute *attr,
> +				      char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	if (list_empty(&hisi_ptt->port_filters))
> +		return sysfs_emit(buf, "#### No available filter ####\n");
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev) |
> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> +
> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev));
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> +
> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> +PMU_FORMAT_ATTR(type,		"config:24-31");
> +PMU_FORMAT_ATTR(format,		"config:32-35");
> +
> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> +	&format_attr_filter.attr,
> +	&format_attr_direction.attr,
> +	&format_attr_type.attr,
> +	&format_attr_format.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_format_group = {
> +	.name = "format",
> +	.attrs = hisi_ptt_pmu_format_attrs,
> +};
> +
> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> +	&dev_attr_available_filters.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> +	.attrs = hisi_ptt_pmu_filter_attrs,
> +};
> +
> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> +	&hisi_ptt_pmu_format_group,
> +	&hisi_ptt_pmu_filter_group,
> +	NULL
> +};
> +
> +/*
> + * The supported value of the direction parameter. See hisi_ptt.rst
> + * documentation for more details.
> + */
> +static u32 hisi_ptt_trace_available_direction[] = {
> +	0,
> +	1,
> +	2,
> +	3,
> +};
> +
> +/* Different types can be set simultaneously */
> +static u32 hisi_ptt_trace_available_type[] = {
> +	1,	/* posted_request */
> +	2,	/* non-posted_request */
> +	4,	/* completion */
> +};
> +
> +static u32 hisi_ptt_trace_availble_format[] = {
> +	0,	/* 4DW */
> +	1,	/* 8DW */
> +};
> +
> +/*
> + * Check whether the config is valid or not. Some configs are multi-selectable
> + * and can be set simultaneously, while some are single selectable (onehot).
> + * Use this function to check the non-onehot configs while
> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> + */
> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	/*
> +	 * The non-onehot configs cannot be 0. Walk the available
> +	 * list and clear the valid bits of the config. If there
> +	 * is any resident bit after the walk then the config is
> +	 * invalid.
> +	 */
> +	for (i = 0; i < list_size; i++)
> +		val &= ~available_list[i];
> +
> +	return val ? -EINVAL : 0;
> +}
> +
> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> +{
> +	int i, ret = -EINVAL;
> +
> +	for (i = 0; i < list_size; i++)
> +		if (val == available_list[i]) {
> +			ret = 0;
> +			break;
> +		}
> +
> +	return ret;
> +}
> +
> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> +{
> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> +	struct hisi_ptt_filter_desc *filter;
> +	int ret = -EINVAL;
> +
> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> +
> +	/*
> +	 * Port filters are defined as bit mask. For port filters, check
> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> +	 * and whether it's empty or not, otherwise user has specified
> +	 * some unsupported root ports.
> +	 *
> +	 * For Requester ID filters, walk the available filter list to see
> +	 * whether we have one matched.
> +	 */
> +	if (hisi_ptt->trace_ctrl.is_port &&
> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> +		ret = 0;
> +	} else {
> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +			if (filter->val == val) {
> +				ret = 0;
> +				break;
> +			}
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.filter = val;
> +	return 0;
> +}
> +
> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	int ret;
> +	u32 val;
> +
> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> +		return -ENOENT;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +
> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> +	if (ret < 0)
> +		goto out;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->direction = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> +
> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->type = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->format = val;
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return ret;
> +}
> +
> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> +				    int nr_pages, bool overwrite)
> +{
> +	struct hisi_ptt_pmu_buf *buf;
> +	struct page **pagelist;
> +	int i;
> +
> +	if (overwrite) {
> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> +		return NULL;
> +	}
> +
> +	/* If the pages size less than buflets, we cannot start trace */
> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> +		return NULL;
> +
> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> +	if (!buf)
> +		return NULL;
> +
> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> +	if (!pagelist) {
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++)
> +		pagelist[i] = virt_to_page(pages[i]);
> +
> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> +	if (!buf->base) {
> +		kfree(pagelist);
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	buf->nr_pages = nr_pages;
> +	buf->length = nr_pages * PAGE_SIZE;
> +	buf->pos = 0;
> +
> +	kfree(pagelist);
> +	return buf;
> +}
> +
> +static void hisi_ptt_pmu_free_aux(void *aux)
> +{
> +	struct hisi_ptt_pmu_buf *buf = aux;
> +
> +	vunmap(buf->base);
> +	kfree(buf);
> +}
> +
> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct hw_perf_event *hwc = &event->hw;
> +	struct hisi_ptt_pmu_buf *buf;
> +	int cpu = event->cpu;
> +	int ret;
> +
> +	hwc->state = 0;
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> +		goto stop;
> +	}
> +
> +	if (cpu == -1)
> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> +
> +	/*
> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> +	 * core in event_function_local().
> +	 */
> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> +				 cpumask_of(cpu)));
> +
> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> +		goto stop;
> +	}
> +
> +	buf = perf_aux_output_begin(handle, event);
> +	if (!buf) {
> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> +		goto stop;
> +	}
> +
> +	buf->pos = handle->head % buf->length;
> +
> +	ret = hisi_ptt_trace_start(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> +		perf_aux_output_end(handle, 0);
> +		goto stop;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return;
> +stop:
> +	event->hw.state |= PERF_HES_STOPPED;
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +
> +	if (hwc->state & PERF_HES_STOPPED)
> +		return;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> +		hisi_ptt_trace_end(hisi_ptt);
> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> +	}
> +	mutex_unlock(&hisi_ptt->mutex);
> +
> +	hwc->state |= PERF_HES_STOPPED;
> +	perf_event_update_userpage(event);
> +	hwc->state |= PERF_HES_UPTODATE;
> +}
> +
> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +	int cpu = event->cpu;
> +
> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> +		return 0;
> +
> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +
> +	if (flags & PERF_EF_START) {
> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> +		if (hwc->state & PERF_HES_STOPPED)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> +{
> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> +}
> +
> +static void hisi_ptt_unregister_pmu(void *priv)
> +{
> +	perf_pmu_unregister(priv);
> +}
> +
> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> +{
> +	char *pmu_name;
> +	int ret;
> +
> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> +		.module		= THIS_MODULE,
> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> +		.task_ctx_nr	= perf_sw_context,
> +		.attr_groups	= hisi_ptt_pmu_groups,
> +		.event_init	= hisi_ptt_pmu_event_init,
> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> +		.free_aux	= hisi_ptt_pmu_free_aux,
> +		.start		= hisi_ptt_pmu_start,
> +		.stop		= hisi_ptt_pmu_stop,
> +		.add		= hisi_ptt_pmu_add,
> +		.del		= hisi_ptt_pmu_del,
> +	};
> +
> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> +	if (!pmu_name)
> +		return -ENOMEM;
> +
> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> +	if (ret)
> +		return ret;
> +
> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> +					hisi_ptt_unregister_pmu,
> +					&hisi_ptt->hisi_ptt_pmu);
> +}
> +
> +/*
> + * Get RMR address if provided by the firmware.
> + * Return 0 if the IOMMU doesn't present or the policy of the
> + * IOMMU domain is passthrough or we get a usable RMR region.
> + * Otherwise a negative value is returned.
> + */
> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct iommu_domain *iommu_domain;
> +	struct iommu_resv_region *region;
> +	LIST_HEAD(list);
> +
> +	/*
> +	 * Use direct DMA if IOMMU does not present or the policy of the
> +	 * IOMMU domain is passthrough.
> +	 */
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	iommu_get_resv_regions(&pdev->dev, &list);
> +	list_for_each_entry(region, &list, list)
> +		if (region->type == IOMMU_RESV_DIRECT &&
> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> +			hisi_ptt->trace_ctrl.has_rmr = true;
> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> +			break;
> +		}
> +
> +	iommu_put_resv_regions(&pdev->dev, &list);
> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> +}
> +
> +static int hisi_ptt_probe(struct pci_dev *pdev,
> +			  const struct pci_device_id *id)
> +{
> +	struct hisi_ptt *hisi_ptt;
> +	int ret;
> +
> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> +	if (!hisi_ptt)
> +		return -ENOMEM;
> +
> +	mutex_init(&hisi_ptt->mutex);
> +	hisi_ptt->pdev = pdev;
> +
> +	/*
> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> +	 * so directly reference to the pci name string.
> +	 */
> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> +	pci_set_drvdata(pdev, hisi_ptt);
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> +	if (ret) {
> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> +
> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> +	if (ret) {
> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> +		return ret;
> +	}
> +	pci_set_master(pdev);
> +
> +	ret = hisi_ptt_register_irq(hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt_init_ctrls(hisi_ptt);
> +
> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> +		return ret;
> +	}

If this fails the pci_free_irq_vectors() is not called.

I am out of time for today - I will continue tomorrow.

Thanks,
Mathieu

> +
> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> +		return ret;
> +	}
> +
> +	/* Register the bus notifier for dynamically updating the filter list */
> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> +	if (ret)
> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> +
> +	return 0;
> +}
> +
> +void hisi_ptt_remove(struct pci_dev *pdev)
> +{
> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> +
> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> +
> +	/* Cancel any work that has been queued */
> +	cancel_delayed_work_sync(&hisi_ptt->work);
> +
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> +		hisi_ptt_trace_end(hisi_ptt);
> +
> +	hisi_ptt_free_trace_buf(hisi_ptt);
> +	hisi_ptt_release_filters(hisi_ptt);
> +}
> +
> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> +	{ }
> +};
> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> +
> +static struct pci_driver hisi_ptt_driver = {
> +	.name = "hisi_ptt",
> +	.id_table = hisi_ptt_id_tbl,
> +	.probe = hisi_ptt_probe,
> +	.remove = hisi_ptt_remove,
> +};
> +module_pci_driver(hisi_ptt_driver);
> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> -- 
> 2.33.0
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

Hi Yicong,

On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> integrated Endpoint(RCiEP) device, providing the capability
> to dynamically monitor and tune the PCIe traffic(tune),
> and trace the TLP headers(trace).
>

Is there a reason to put "tune" and "trace" are whithin parentheses?  

> Add the driver for the device to enable the trace function. The driver
> will create PMU device for each PTT device, and users can start trace
> through perf command.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  drivers/Makefile                       |    1 +
>  drivers/hwtracing/Kconfig              |    2 +
>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++

Not sure about the "hisilicon" directory.  Right now we have "coresight",
"intel_th" and "stm" - they concentrate on the technology rather than the
vendor.  I think "ptt" would be just fine: 

drivers/hwtracing/ptt/Kconfig 
drivers/hwtracing/ptt/Makefile
drivers/hwtracing/ptt/hisi_ptt.c

That way if other vendors want to introduce support for the same kind of
technology we can simply put them under "drivers/hwtracing/ptt/" and things are
still accurate.


>  5 files changed, 1159 insertions(+)
>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
> 
> diff --git a/drivers/Makefile b/drivers/Makefile
> index be5d40ae1488..bb0dc9b55ea2 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>  obj-y				+= hwtracing/intel_th/
>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>  obj-$(CONFIG_ANDROID)		+= android/
>  obj-$(CONFIG_NVMEM)		+= nvmem/
>  obj-$(CONFIG_FPGA)		+= fpga/
> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> index 13085835a636..e3796b17541a 100644
> --- a/drivers/hwtracing/Kconfig
> +++ b/drivers/hwtracing/Kconfig
> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>  
>  source "drivers/hwtracing/intel_th/Kconfig"
>  
> +source "drivers/hwtracing/hisilicon/Kconfig"
> +
>  endmenu
> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
> new file mode 100644
> index 000000000000..9c3ab80d99fe
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/Kconfig
> @@ -0,0 +1,8 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config HISI_PTT
> +	tristate "HiSilicon PCIe Tune and Trace Device"
> +	depends on PCI && HAS_DMA && HAS_IOMEM

What Arm architecture are you aiming this for?  Can it be found on both armv7
and armv8?

> +	help
> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
> +	  device, provides support for PCIe traffic tuning and
> +	  tracing TLP headers to the memory.

Please indicate what the name of the driver will be when compiled as a module.

> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
> new file mode 100644
> index 000000000000..908c09a98161
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/Makefile
> @@ -0,0 +1,2 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
> new file mode 100644
> index 000000000000..e11e9b6cc2a8
> --- /dev/null
> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
> @@ -0,0 +1,1146 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for HiSilicon PCIe tune and trace device
> + *
> + * Copyright (c) 2021 HiSilicon Limited.
> + */
> +
> +#include <linux/bitfield.h>
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iommu.h>
> +#include <linux/iopoll.h>
> +#include <linux/kfifo.h>
> +#include <linux/module.h>
> +#include <linux/notifier.h>
> +#include <linux/pci.h>
> +#include <linux/perf_event.h>
> +#include <linux/sizes.h>
> +#include <linux/sysfs.h>
> +#include <linux/vmalloc.h>
> +#include <linux/workqueue.h>
> +
> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> +#define HISI_PTT_TRACE_CTRL		0x0850
> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> +#define HISI_PTT_TRACE_STS		0x08b0
> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> +#define HISI_PTT_LOCATION		0x0fe8
> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)

Extra spaces between #define and the literals.  Please align the values with
that of HISI_PTT_TRACE_DMA_IRQ below.

> +
> +#define HISI_PTT_TRACE_DMA_IRQ			0
> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> +
> +/* Delay time for filter updating work */
> +#define HISI_PTT_WORK_DELAY_MS		100UL
> +/* Wait time for DMA hardware to reset */
> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> +/* Poll timeout and interval for waiting hardware work to finish */
> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> +
> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> +
> +enum hisi_ptt_trace_status {
> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> +	HISI_PTT_TRACE_STATUS_ON,
> +};
> +
> +struct hisi_ptt_dma_buflet {

I'm not sure what a "buflet" is...  Probably best to just call this
hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
know due to the lack of proper structure documentation.  Please have a look at
how "coresight_device" is documented.  The same applies to the rest of the
structures declared below.

> +	struct list_head list;
> +	unsigned int size;
> +	dma_addr_t dma;
> +
> +	/*
> +	 * The address of the buflet holding the trace data.
> +	 * See Documentation/trace/hisi-ptt.rst for the details

As of this writing, hisi-ptt.rst doesn't exist.

> +	 * of the data format.
> +	 */
> +	void *addr;
> +	int index;
> +};
> +
> +struct hisi_ptt_trace_ctrl {
> +	enum hisi_ptt_trace_status status;
> +	struct perf_output_handle handle;
> +	struct list_head trace_buf;
> +	/*
> +	 * The index of the buffer which trace data
> +	 * currently is writing to.
> +	 */
> +	u32 buf_index;
> +
> +	int default_cpu;
> +	u32 buflet_size;
> +	bool is_port;
> +	u32 format:1;		/* Format of the traced TLP headers */
> +	u32 type:4;		/* Type of the TLP headers to trace */
> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> +
> +	phys_addr_t rmr_addr;
> +	size_t rmr_length;
> +	bool has_rmr;
> +};
> +
> +struct hisi_ptt_filter_desc {
> +	struct list_head list;
> +	struct pci_dev *pdev;
> +	u16 val;
> +};
> +
> +struct hisi_ptt_pmu_buf {
> +	size_t length;
> +	int nr_pages;
> +	void *base;
> +	long pos;
> +};
> +
> +/* Structure containing the information for filter updating */
> +struct hisi_ptt_filter_update_info {
> +	struct pci_dev *pdev;
> +	u32 port_devid;
> +	bool is_port;
> +	bool is_add;
> +	u16 val;
> +};
> +
> +struct hisi_ptt {
> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> +	struct notifier_block hisi_ptt_nb;
> +	struct pmu hisi_ptt_pmu;
> +	void __iomem *iobase;
> +	struct pci_dev *pdev;
> +	/*
> +	 * Use the mutex to protect the filter list and
> +	 * serialize the perf process.
> +	 */
> +	struct mutex mutex;
> +	const char *name;
> +	u16 core_id;
> +	u16 sicl_id;
> +	/* PCI device range managed by the PTT device */
> +	u32 upper;
> +	u32 lower;
> +	u8 busnr;
> +
> +	/*
> +	 * The trace TLP headers can either be filtered by certain
> +	 * root port, or by the requester ID. Organize the filters
> +	 * by @port_filters and @req_filters here. The mask of all
> +	 * the valid ports is also cached for doing sanity check
> +	 * of user input.
> +	 */
> +	struct list_head port_filters;
> +	struct list_head req_filters;
> +	u16 port_mask;
> +
> +	/*
> +	 * We use a delayed work here to avoid indefinitely waiting for
> +	 * the hisi_ptt->mutex which protecting the filter list. The
> +	 * work will be delayed only if the mutex can not be held,
> +	 * otherwise no delay will be applied.
> +	 */
> +	struct delayed_work work;
> +	spinlock_t filter_update_lock;
> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> +};

It might be a good idea to spinoff a hisi_ptt.h for the above declarations.

> +
> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> +{
> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> +}

I'm not sure there is any value to this inline function.  I suggest to simply
call container_of() whenever it is needed.

> +
> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> +{
> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> +
> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> +}
> +
> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> +{
> +	u32 val;
> +
> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> +				  val & HISI_PTT_TRACE_IDLE,
> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> +				  HISI_PTT_WAIT_TIMEOUT_US);
> +}
> +
> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> +
> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> +		list_del(&buflet->list);
> +
> +		if (ctrl->has_rmr)
> +			memunmap(buflet->addr);
> +		else
> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> +					  buflet->dma);
> +
> +		kfree(buflet);
> +	}
> +}
> +
> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct device *dev = &hisi_ptt->pdev->dev;
> +	struct hisi_ptt_dma_buflet *buflet;
> +	int i, ret;
> +
> +	hisi_ptt->trace_ctrl.buf_index = 0;
> +
> +	/* Make sure the trace buffer is empty before allocating */
> +	if (!list_empty(&ctrl->trace_buf)) {
> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> +			memset(buflet->addr, 0, buflet->size);
> +		return 0;
> +	}
> +
> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> +		if (!buflet) {
> +			ret = -ENOMEM;
> +			goto err;
> +		}
> +
> +		if (ctrl->has_rmr) {
> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> +
> +			buflet->dma = base;
> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> +		} else {
> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> +							  &buflet->dma, GFP_KERNEL);
> +		}
> +
> +		if (!buflet->addr) {
> +			kfree(buflet);
> +			ret = -ENOMEM;
> +			goto err;
> +		}
> +
> +		memset(buflet->addr, 0, buflet->size);
> +
> +		buflet->index = i;
> +		buflet->size = ctrl->buflet_size;
> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> +	}
> +
> +	return 0;
> +err:
> +	hisi_ptt_free_trace_buf(hisi_ptt);
> +	return ret;
> +}
> +
> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> +{
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> +}
> +
> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	struct hisi_ptt_dma_buflet *cur;
> +	u32 val;
> +
> +	/* Check device idle before start trace */
> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> +		return -EBUSY;
> +	}
> +
> +	/* Reset the DMA before start tracing */
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val |= HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/*
> +	 * We'll be in the perf context where preemption is disabled,
> +	 * so use busy loop here.
> +	 */
> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> +
> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	/* clear the interrupt status */
> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> +
> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> +		writel(lower_32_bits(cur->dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> +		writel(upper_32_bits(cur->dma),
> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> +	}
> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> +
> +	/* set the trace control register */
> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> +	if (!hisi_ptt->trace_ctrl.is_port)
> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> +
> +	val |= HISI_PTT_TRACE_CTRL_EN;
> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> +
> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> +{
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct perf_event *event = handle->event;
> +	struct hisi_ptt_dma_buflet *cur;
> +	struct hisi_ptt_pmu_buf *buf;
> +
> +	buf = perf_get_aux(handle);
> +	if (!buf || !handle->size)
> +		return -EINVAL;
> +
> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> +		if (cur->index == index)
> +			break;
> +
> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> +	memset(cur->addr, 0, cur->size);
> +	buf->pos += cur->size;
> +
> +	if (stop) {
> +		perf_aux_output_end(handle, buf->pos);
> +	} else if (buf->length - buf->pos < cur->size) {
> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> +		perf_aux_output_end(handle, buf->pos);
> +
> +		buf = perf_aux_output_begin(handle, event);
> +		if (!buf)
> +			return -EINVAL;
> +
> +		buf->pos = handle->head % buf->length;
> +		if (buf->length - buf->pos < cur->size) {
> +			perf_aux_output_end(handle, 0);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status, buf_idx;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	buf_idx = ffs(status) - 1;
> +
> +	/* Clear the interrupt status of buflet @buf_idx */
> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +
> +	/*
> +	 * Update the AUX buffer and cache the current buflet index,
> +	 * as we need to know this and save the data when the trace
> +	 * is ended out of the interrupt handler. End the trace
> +	 * if the updating fails.
> +	 */
> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> +		hisi_ptt_trace_end(hisi_ptt);
> +	else
> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> +{
> +	struct hisi_ptt *hisi_ptt = context;
> +	u32 status;
> +
> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> +		return IRQ_NONE;
> +
> +	return IRQ_WAKE_THREAD;
> +}
> +
> +static void hisi_ptt_irq_free_vectors(void *pdev)
> +{
> +	pci_free_irq_vectors(pdev);
> +}
> +
> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	int ret;
> +
> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> +	if (ret < 0) {
> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = devm_request_threaded_irq(&pdev->dev,
> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> +					"hisi-ptt", hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_update_filters(struct work_struct *work)
> +{
> +	struct delayed_work *delayed_work = to_delayed_work(work);
> +	struct hisi_ptt_filter_update_info info;
> +	struct hisi_ptt_filter_desc *filter;
> +	struct list_head *target_list;
> +	struct hisi_ptt *hisi_ptt;
> +
> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> +
> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> +		return;
> +	}
> +
> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> +			      &hisi_ptt->req_filters;
> +
> +		if (info.is_add) {
> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> +			if (!filter)
> +				continue;
> +
> +			filter->pdev = info.pdev;
> +			filter->val = info.val;
> +
> +			list_add_tail(&filter->list, target_list);
> +		} else {
> +			list_for_each_entry(filter, target_list, list)
> +				if (filter->val == info.val) {
> +					list_del(&filter->list);
> +					kfree(filter);
> +					break;
> +				}
> +		}
> +
> +		/* Update the available port mask */
> +		if (!info.is_port)
> +			continue;
> +
> +		if (info.is_add)
> +			hisi_ptt->port_mask |= info.val;
> +		else
> +			hisi_ptt->port_mask &= ~info.val;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> +				    struct hisi_ptt_filter_update_info *info)
> +{
> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> +
> +	if (!root_port)
> +		return;
> +
> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> +	if (info->port_devid < hisi_ptt->lower ||
> +	    info->port_devid > hisi_ptt->upper)
> +		return;
> +
> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> +
> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> +				&hisi_ptt->filter_update_lock))
> +		schedule_delayed_work(&hisi_ptt->work, 0);
> +	else
> +		pci_warn(hisi_ptt->pdev,
> +			 "filter update fifo overflow for target %s\n",
> +			 pci_name(info->pdev));
> +}
> +
> +/*
> + * A PCI bus notifier is used here for dynamically updating the filter
> + * list.
> + */
> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> +				  void *data)
> +{
> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> +	struct hisi_ptt_filter_update_info info;
> +	struct device *dev = data;
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +
> +	info.pdev = pdev;
> +
> +	switch (action) {
> +	case BUS_NOTIFY_ADD_DEVICE:
> +		info.is_add = true;
> +		break;
> +	case BUS_NOTIFY_DEL_DEVICE:
> +		info.is_add = false;
> +		break;
> +	default:
> +		return 0;
> +	}
> +
> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> +
> +	return 0;
> +}
> +
> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> +{
> +	struct hisi_ptt_filter_update_info info = {
> +		.pdev = pdev,
> +		.is_add = true,
> +	};
> +	struct hisi_ptt *hisi_ptt = data;
> +
> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> +{
> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +
> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> +		list_del(&filter->list);
> +		kfree(filter);
> +	}
> +}
> +
> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct pci_bus *bus;
> +	u32 reg;
> +
> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> +
> +	/*
> +	 * The device range register provides the information about the
> +	 * root ports which the RCiEP can control and trace. The RCiEP
> +	 * and the root ports it support are on the same PCIe core, with
> +	 * same domain number but maybe different bus number. The device
> +	 * range register will tell us which root ports we can support,
> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> +	 * while Bit[15:0] indicates the lower.
> +	 */
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> +	hisi_ptt->upper = reg >> 16;
> +	hisi_ptt->lower = reg & 0xffff;
> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> +
> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> +
> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> +	if (bus)
> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);

What happens if bus is NULL?  Should this be reported to the caller and cause
hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.

> +
> +	/* Initialize trace controls */
> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> +}
> +
> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> +
> +static ssize_t available_filters_show(struct device *dev,
> +				      struct device_attribute *attr,
> +				      char *buf)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> +	struct hisi_ptt_filter_desc *filter;
> +	int pos = 0;
> +
> +	if (list_empty(&hisi_ptt->port_filters))
> +		return sysfs_emit(buf, "#### No available filter ####\n");
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev) |
> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> +
> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> +				     pci_name(filter->pdev),
> +				     hisi_ptt_get_filter_val(filter->pdev));
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return pos;
> +}
> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> +
> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> +PMU_FORMAT_ATTR(type,		"config:24-31");
> +PMU_FORMAT_ATTR(format,		"config:32-35");
> +
> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> +	&format_attr_filter.attr,
> +	&format_attr_direction.attr,
> +	&format_attr_type.attr,
> +	&format_attr_format.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_format_group = {
> +	.name = "format",
> +	.attrs = hisi_ptt_pmu_format_attrs,
> +};
> +
> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> +	&dev_attr_available_filters.attr,
> +	NULL
> +};
> +
> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> +	.attrs = hisi_ptt_pmu_filter_attrs,
> +};
> +
> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> +	&hisi_ptt_pmu_format_group,
> +	&hisi_ptt_pmu_filter_group,
> +	NULL
> +};
> +
> +/*
> + * The supported value of the direction parameter. See hisi_ptt.rst
> + * documentation for more details.
> + */
> +static u32 hisi_ptt_trace_available_direction[] = {
> +	0,
> +	1,
> +	2,
> +	3,
> +};
> +
> +/* Different types can be set simultaneously */
> +static u32 hisi_ptt_trace_available_type[] = {
> +	1,	/* posted_request */
> +	2,	/* non-posted_request */
> +	4,	/* completion */
> +};
> +
> +static u32 hisi_ptt_trace_availble_format[] = {
> +	0,	/* 4DW */
> +	1,	/* 8DW */
> +};
> +
> +/*
> + * Check whether the config is valid or not. Some configs are multi-selectable
> + * and can be set simultaneously, while some are single selectable (onehot).
> + * Use this function to check the non-onehot configs while
> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> + */
> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> +{
> +	int i;
> +
> +	/*
> +	 * The non-onehot configs cannot be 0. Walk the available
> +	 * list and clear the valid bits of the config. If there
> +	 * is any resident bit after the walk then the config is
> +	 * invalid.
> +	 */
> +	for (i = 0; i < list_size; i++)
> +		val &= ~available_list[i];
> +
> +	return val ? -EINVAL : 0;
> +}
> +
> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> +{
> +	int i, ret = -EINVAL;
> +
> +	for (i = 0; i < list_size; i++)
> +		if (val == available_list[i]) {
> +			ret = 0;
> +			break;
> +		}
> +
> +	return ret;
> +}
> +
> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> +{
> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> +	struct hisi_ptt_filter_desc *filter;
> +	int ret = -EINVAL;
> +
> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> +
> +	/*
> +	 * Port filters are defined as bit mask. For port filters, check
> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> +	 * and whether it's empty or not, otherwise user has specified
> +	 * some unsupported root ports.
> +	 *
> +	 * For Requester ID filters, walk the available filter list to see
> +	 * whether we have one matched.
> +	 */
> +	if (hisi_ptt->trace_ctrl.is_port &&
> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> +		ret = 0;
> +	} else {
> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> +			if (filter->val == val) {
> +				ret = 0;
> +				break;
> +			}
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt->trace_ctrl.filter = val;
> +	return 0;
> +}
> +
> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> +	int ret;
> +	u32 val;
> +
> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> +		return -ENOENT;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +
> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> +	if (ret < 0)
> +		goto out;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->direction = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> +
> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->type = val;
> +
> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> +	if (ret < 0)
> +		goto out;
> +	ctrl->format = val;
> +
> +out:
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return ret;
> +}
> +
> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> +				    int nr_pages, bool overwrite)
> +{
> +	struct hisi_ptt_pmu_buf *buf;
> +	struct page **pagelist;
> +	int i;
> +
> +	if (overwrite) {
> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> +		return NULL;
> +	}
> +
> +	/* If the pages size less than buflets, we cannot start trace */
> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> +		return NULL;
> +
> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> +	if (!buf)
> +		return NULL;
> +
> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> +	if (!pagelist) {
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++)
> +		pagelist[i] = virt_to_page(pages[i]);
> +
> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> +	if (!buf->base) {
> +		kfree(pagelist);
> +		kfree(buf);
> +		return NULL;
> +	}
> +
> +	buf->nr_pages = nr_pages;
> +	buf->length = nr_pages * PAGE_SIZE;
> +	buf->pos = 0;
> +
> +	kfree(pagelist);
> +	return buf;
> +}
> +
> +static void hisi_ptt_pmu_free_aux(void *aux)
> +{
> +	struct hisi_ptt_pmu_buf *buf = aux;
> +
> +	vunmap(buf->base);
> +	kfree(buf);
> +}
> +
> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> +	struct hw_perf_event *hwc = &event->hw;
> +	struct hisi_ptt_pmu_buf *buf;
> +	int cpu = event->cpu;
> +	int ret;
> +
> +	hwc->state = 0;
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> +		goto stop;
> +	}
> +
> +	if (cpu == -1)
> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> +
> +	/*
> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> +	 * core in event_function_local().
> +	 */
> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> +				 cpumask_of(cpu)));
> +
> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> +		goto stop;
> +	}
> +
> +	buf = perf_aux_output_begin(handle, event);
> +	if (!buf) {
> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> +		goto stop;
> +	}
> +
> +	buf->pos = handle->head % buf->length;
> +
> +	ret = hisi_ptt_trace_start(hisi_ptt);
> +	if (ret) {
> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> +		perf_aux_output_end(handle, 0);
> +		goto stop;
> +	}
> +
> +	mutex_unlock(&hisi_ptt->mutex);
> +	return;
> +stop:
> +	event->hw.state |= PERF_HES_STOPPED;
> +	mutex_unlock(&hisi_ptt->mutex);
> +}
> +
> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +
> +	if (hwc->state & PERF_HES_STOPPED)
> +		return;
> +
> +	mutex_lock(&hisi_ptt->mutex);
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> +		hisi_ptt_trace_end(hisi_ptt);
> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> +	}
> +	mutex_unlock(&hisi_ptt->mutex);
> +
> +	hwc->state |= PERF_HES_STOPPED;
> +	perf_event_update_userpage(event);
> +	hwc->state |= PERF_HES_UPTODATE;
> +}
> +
> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> +{
> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> +	struct hw_perf_event *hwc = &event->hw;
> +	int cpu = event->cpu;
> +
> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> +		return 0;
> +
> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> +
> +	if (flags & PERF_EF_START) {
> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> +		if (hwc->state & PERF_HES_STOPPED)
> +			return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> +{
> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> +}
> +
> +static void hisi_ptt_unregister_pmu(void *priv)
> +{
> +	perf_pmu_unregister(priv);
> +}
> +
> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> +{
> +	char *pmu_name;
> +	int ret;
> +
> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> +		.module		= THIS_MODULE,
> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> +		.task_ctx_nr	= perf_sw_context,
> +		.attr_groups	= hisi_ptt_pmu_groups,
> +		.event_init	= hisi_ptt_pmu_event_init,
> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> +		.free_aux	= hisi_ptt_pmu_free_aux,
> +		.start		= hisi_ptt_pmu_start,
> +		.stop		= hisi_ptt_pmu_stop,
> +		.add		= hisi_ptt_pmu_add,
> +		.del		= hisi_ptt_pmu_del,
> +	};
> +
> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> +	if (!pmu_name)
> +		return -ENOMEM;
> +
> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> +	if (ret)
> +		return ret;
> +
> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> +					hisi_ptt_unregister_pmu,
> +					&hisi_ptt->hisi_ptt_pmu);
> +}
> +
> +/*
> + * Get RMR address if provided by the firmware.
> + * Return 0 if the IOMMU doesn't present or the policy of the
> + * IOMMU domain is passthrough or we get a usable RMR region.
> + * Otherwise a negative value is returned.
> + */
> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> +{
> +	struct pci_dev *pdev = hisi_ptt->pdev;
> +	struct iommu_domain *iommu_domain;
> +	struct iommu_resv_region *region;
> +	LIST_HEAD(list);
> +
> +	/*
> +	 * Use direct DMA if IOMMU does not present or the policy of the
> +	 * IOMMU domain is passthrough.
> +	 */
> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> +		return 0;
> +
> +	iommu_get_resv_regions(&pdev->dev, &list);
> +	list_for_each_entry(region, &list, list)
> +		if (region->type == IOMMU_RESV_DIRECT &&
> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> +			hisi_ptt->trace_ctrl.has_rmr = true;
> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> +			break;
> +		}
> +
> +	iommu_put_resv_regions(&pdev->dev, &list);
> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> +}
> +
> +static int hisi_ptt_probe(struct pci_dev *pdev,
> +			  const struct pci_device_id *id)
> +{
> +	struct hisi_ptt *hisi_ptt;
> +	int ret;
> +
> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> +	if (!hisi_ptt)
> +		return -ENOMEM;
> +
> +	mutex_init(&hisi_ptt->mutex);
> +	hisi_ptt->pdev = pdev;
> +
> +	/*
> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> +	 * so directly reference to the pci name string.
> +	 */
> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> +	pci_set_drvdata(pdev, hisi_ptt);
> +
> +	ret = pcim_enable_device(pdev);
> +	if (ret) {
> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> +	if (ret) {
> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> +		return ret;
> +	}
> +
> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> +
> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> +	if (ret) {
> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> +		return ret;
> +	}
> +	pci_set_master(pdev);
> +
> +	ret = hisi_ptt_register_irq(hisi_ptt);
> +	if (ret)
> +		return ret;
> +
> +	hisi_ptt_init_ctrls(hisi_ptt);
> +
> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> +		return ret;
> +	}

If this fails the pci_free_irq_vectors() is not called.

I am out of time for today - I will continue tomorrow.

Thanks,
Mathieu

> +
> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> +	if (ret) {
> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> +		return ret;
> +	}
> +
> +	/* Register the bus notifier for dynamically updating the filter list */
> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> +	if (ret)
> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> +
> +	return 0;
> +}
> +
> +void hisi_ptt_remove(struct pci_dev *pdev)
> +{
> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> +
> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> +
> +	/* Cancel any work that has been queued */
> +	cancel_delayed_work_sync(&hisi_ptt->work);
> +
> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> +		hisi_ptt_trace_end(hisi_ptt);
> +
> +	hisi_ptt_free_trace_buf(hisi_ptt);
> +	hisi_ptt_release_filters(hisi_ptt);
> +}
> +
> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> +	{ }
> +};
> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> +
> +static struct pci_driver hisi_ptt_driver = {
> +	.name = "hisi_ptt",
> +	.id_table = hisi_ptt_id_tbl,
> +	.probe = hisi_ptt_probe,
> +	.remove = hisi_ptt_remove,
> +};
> +module_pci_driver(hisi_ptt_driver);
> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> -- 
> 2.33.0
> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

Hi Mathieu,

Thanks for the comments! Replies inline.

On 2021/11/25 2:51, Mathieu Poirier wrote:
> Hi Yicong,
> 
> On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>> integrated Endpoint(RCiEP) device, providing the capability
>> to dynamically monitor and tune the PCIe traffic(tune),
>> and trace the TLP headers(trace).
>>
> 
> Is there a reason to put "tune" and "trace" are whithin parentheses?  
> 

yeah. I want to use these single word to denote the related function, and
place them in the parantheses near the function's description to make sure
readers can match them.

>> Add the driver for the device to enable the trace function. The driver
>> will create PMU device for each PTT device, and users can start trace
>> through perf command.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>>  drivers/Makefile                       |    1 +
>>  drivers/hwtracing/Kconfig              |    2 +
>>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
> 
> Not sure about the "hisilicon" directory.  Right now we have "coresight",
> "intel_th" and "stm" - they concentrate on the technology rather than the
> vendor.  I think "ptt" would be just fine: 
> 
> drivers/hwtracing/ptt/Kconfig 
> drivers/hwtracing/ptt/Makefile
> drivers/hwtracing/ptt/hisi_ptt.c
> 
> That way if other vendors want to introduce support for the same kind of
> technology we can simply put them under "drivers/hwtracing/ptt/" and things are
> still accurate.
> 

sure. sounds sensible. will rename the directory to "ptt" if no objection.

> 
>>  5 files changed, 1159 insertions(+)
>>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
>>
>> diff --git a/drivers/Makefile b/drivers/Makefile
>> index be5d40ae1488..bb0dc9b55ea2 100644
>> --- a/drivers/Makefile
>> +++ b/drivers/Makefile
>> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>  obj-y				+= hwtracing/intel_th/
>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>>  obj-$(CONFIG_ANDROID)		+= android/
>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>  obj-$(CONFIG_FPGA)		+= fpga/
>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>> index 13085835a636..e3796b17541a 100644
>> --- a/drivers/hwtracing/Kconfig
>> +++ b/drivers/hwtracing/Kconfig
>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>  
>>  source "drivers/hwtracing/intel_th/Kconfig"
>>  
>> +source "drivers/hwtracing/hisilicon/Kconfig"
>> +
>>  endmenu
>> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
>> new file mode 100644
>> index 000000000000..9c3ab80d99fe
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/Kconfig
>> @@ -0,0 +1,8 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config HISI_PTT
>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>> +	depends on PCI && HAS_DMA && HAS_IOMEM
> 
> What Arm architecture are you aiming this for?  Can it be found on both armv7
> and armv8?
> 

currently the device will appear on Kunpeng 930, which is armv8.

>> +	help
>> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
>> +	  device, provides support for PCIe traffic tuning and
>> +	  tracing TLP headers to the memory.
> 
> Please indicate what the name of the driver will be when compiled as a module.
> 

sure. will add the module's name here.

>> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
>> new file mode 100644
>> index 000000000000..908c09a98161
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
>> new file mode 100644
>> index 000000000000..e11e9b6cc2a8
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
>> @@ -0,0 +1,1146 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for HiSilicon PCIe tune and trace device
>> + *
>> + * Copyright (c) 2021 HiSilicon Limited.
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/bitops.h>
>> +#include <linux/delay.h>
>> +#include <linux/dma-iommu.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io.h>
>> +#include <linux/iommu.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/kfifo.h>
>> +#include <linux/module.h>
>> +#include <linux/notifier.h>
>> +#include <linux/pci.h>
>> +#include <linux/perf_event.h>
>> +#include <linux/sizes.h>
>> +#include <linux/sysfs.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/workqueue.h>
>> +
>> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
>> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
>> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
>> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
>> +#define HISI_PTT_TRACE_CTRL		0x0850
>> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
>> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
>> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
>> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
>> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
>> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
>> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
>> +#define HISI_PTT_TRACE_INT_STAT		0x0890
>> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
>> +#define HISI_PTT_TRACE_INT_MASK		0x0894
>> +#define HISI_PTT_TRACE_WR_STS		0x08a0
>> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
>> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
>> +#define HISI_PTT_TRACE_STS		0x08b0
>> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
>> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
>> +#define HISI_PTT_LOCATION		0x0fe8
>> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
>> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> 
> Extra spaces between #define and the literals.  Please align the values with
> that of HISI_PTT_TRACE_DMA_IRQ below.
> 

Spaces are intended to distinguish the definition of the register offset and
related register fields. also used in include/uapi/linux/pci_regs.h.

HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
in the mail text, maybe because the tab is not 8 characters when display.

>> +
>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>> +
>> +/* Delay time for filter updating work */
>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>> +/* Wait time for DMA hardware to reset */
>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>> +/* Poll timeout and interval for waiting hardware work to finish */
>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>> +
>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>> +
>> +enum hisi_ptt_trace_status {
>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>> +	HISI_PTT_TRACE_STATUS_ON,
>> +};
>> +
>> +struct hisi_ptt_dma_buflet {
> 
> I'm not sure what a "buflet" is...  Probably best to just call this
> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> know due to the lack of proper structure documentation.  Please have a look at
> how "coresight_device" is documented.  The same applies to the rest of the
> structures declared below.
> 

It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
(we call it buflet) of 4MiB ...".

The total DMA buffer of PTT device is divided into 4 parts, and each part is called
a 'buflet', which means a small buffer (there exists other words with similiar
format: droplet, wavelet, ...).

The device is designed like this to make sure we won't lose any data when we have to change
the buffer address, the device can continue to write to the next buflet and don't
neet to pause.

>> +	struct list_head list;
>> +	unsigned int size;
>> +	dma_addr_t dma;
>> +
>> +	/*
>> +	 * The address of the buflet holding the trace data.
>> +	 * See Documentation/trace/hisi-ptt.rst for the details
> 
> As of this writing, hisi-ptt.rst doesn't exist.
> 

The comment intends to direct the user to the documentation if the user
want to know details about the data format. The data format is decribed
in the section 4 of the doc.

>> +	 * of the data format.
>> +	 */
>> +	void *addr;
>> +	int index;
>> +};
>> +
>> +struct hisi_ptt_trace_ctrl {
>> +	enum hisi_ptt_trace_status status;
>> +	struct perf_output_handle handle;
>> +	struct list_head trace_buf;
>> +	/*
>> +	 * The index of the buffer which trace data
>> +	 * currently is writing to.
>> +	 */
>> +	u32 buf_index;
>> +
>> +	int default_cpu;
>> +	u32 buflet_size;
>> +	bool is_port;
>> +	u32 format:1;		/* Format of the traced TLP headers */
>> +	u32 type:4;		/* Type of the TLP headers to trace */
>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>> +
>> +	phys_addr_t rmr_addr;
>> +	size_t rmr_length;
>> +	bool has_rmr;
>> +};
>> +
>> +struct hisi_ptt_filter_desc {
>> +	struct list_head list;
>> +	struct pci_dev *pdev;
>> +	u16 val;
>> +};
>> +
>> +struct hisi_ptt_pmu_buf {
>> +	size_t length;
>> +	int nr_pages;
>> +	void *base;
>> +	long pos;
>> +};
>> +
>> +/* Structure containing the information for filter updating */
>> +struct hisi_ptt_filter_update_info {
>> +	struct pci_dev *pdev;
>> +	u32 port_devid;
>> +	bool is_port;
>> +	bool is_add;
>> +	u16 val;
>> +};
>> +
>> +struct hisi_ptt {
>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>> +	struct notifier_block hisi_ptt_nb;
>> +	struct pmu hisi_ptt_pmu;
>> +	void __iomem *iobase;
>> +	struct pci_dev *pdev;
>> +	/*
>> +	 * Use the mutex to protect the filter list and
>> +	 * serialize the perf process.
>> +	 */
>> +	struct mutex mutex;
>> +	const char *name;
>> +	u16 core_id;
>> +	u16 sicl_id;
>> +	/* PCI device range managed by the PTT device */
>> +	u32 upper;
>> +	u32 lower;
>> +	u8 busnr;
>> +
>> +	/*
>> +	 * The trace TLP headers can either be filtered by certain
>> +	 * root port, or by the requester ID. Organize the filters
>> +	 * by @port_filters and @req_filters here. The mask of all
>> +	 * the valid ports is also cached for doing sanity check
>> +	 * of user input.
>> +	 */
>> +	struct list_head port_filters;
>> +	struct list_head req_filters;
>> +	u16 port_mask;
>> +
>> +	/*
>> +	 * We use a delayed work here to avoid indefinitely waiting for
>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>> +	 * work will be delayed only if the mutex can not be held,
>> +	 * otherwise no delay will be applied.
>> +	 */
>> +	struct delayed_work work;
>> +	spinlock_t filter_update_lock;
>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>> +};
> 
> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> 

I'm not sure whether it is common or permitted to provide a header which is only used by
this driver with a single file.

>> +
>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>> +{
>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>> +}
> 
> I'm not sure there is any value to this inline function.  I suggest to simply
> call container_of() whenever it is needed.
> 

ok.

>> +
>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>> +{
>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>> +
>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>> +}
>> +
>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>> +{
>> +	u32 val;
>> +
>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>> +				  val & HISI_PTT_TRACE_IDLE,
>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>> +}
>> +
>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>> +
>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>> +		list_del(&buflet->list);
>> +
>> +		if (ctrl->has_rmr)
>> +			memunmap(buflet->addr);
>> +		else
>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>> +					  buflet->dma);
>> +
>> +		kfree(buflet);
>> +	}
>> +}
>> +
>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	struct hisi_ptt_dma_buflet *buflet;
>> +	int i, ret;
>> +
>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>> +
>> +	/* Make sure the trace buffer is empty before allocating */
>> +	if (!list_empty(&ctrl->trace_buf)) {
>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>> +			memset(buflet->addr, 0, buflet->size);
>> +		return 0;
>> +	}
>> +
>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>> +		if (!buflet) {
>> +			ret = -ENOMEM;
>> +			goto err;
>> +		}
>> +
>> +		if (ctrl->has_rmr) {
>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>> +
>> +			buflet->dma = base;
>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>> +		} else {
>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>> +							  &buflet->dma, GFP_KERNEL);
>> +		}
>> +
>> +		if (!buflet->addr) {
>> +			kfree(buflet);
>> +			ret = -ENOMEM;
>> +			goto err;
>> +		}
>> +
>> +		memset(buflet->addr, 0, buflet->size);
>> +
>> +		buflet->index = i;
>> +		buflet->size = ctrl->buflet_size;
>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>> +	}
>> +
>> +	return 0;
>> +err:
>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>> +	return ret;
>> +}
>> +
>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>> +{
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>> +}
>> +
>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct hisi_ptt_dma_buflet *cur;
>> +	u32 val;
>> +
>> +	/* Check device idle before start trace */
>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>> +		return -EBUSY;
>> +	}
>> +
>> +	/* Reset the DMA before start tracing */
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	/*
>> +	 * We'll be in the perf context where preemption is disabled,
>> +	 * so use busy loop here.
>> +	 */
>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>> +
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	/* clear the interrupt status */
>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>> +
>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>> +		writel(lower_32_bits(cur->dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> +		writel(upper_32_bits(cur->dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> +	}
>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>> +
>> +	/* set the trace control register */
>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>> +	if (!hisi_ptt->trace_ctrl.is_port)
>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>> +
>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>> +{
>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +	struct perf_event *event = handle->event;
>> +	struct hisi_ptt_dma_buflet *cur;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +
>> +	buf = perf_get_aux(handle);
>> +	if (!buf || !handle->size)
>> +		return -EINVAL;
>> +
>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>> +		if (cur->index == index)
>> +			break;
>> +
>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>> +	memset(cur->addr, 0, cur->size);
>> +	buf->pos += cur->size;
>> +
>> +	if (stop) {
>> +		perf_aux_output_end(handle, buf->pos);
>> +	} else if (buf->length - buf->pos < cur->size) {
>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>> +		perf_aux_output_end(handle, buf->pos);
>> +
>> +		buf = perf_aux_output_begin(handle, event);
>> +		if (!buf)
>> +			return -EINVAL;
>> +
>> +		buf->pos = handle->head % buf->length;
>> +		if (buf->length - buf->pos < cur->size) {
>> +			perf_aux_output_end(handle, 0);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>> +{
>> +	struct hisi_ptt *hisi_ptt = context;
>> +	u32 status, buf_idx;
>> +
>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	buf_idx = ffs(status) - 1;
>> +
>> +	/* Clear the interrupt status of buflet @buf_idx */
>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +
>> +	/*
>> +	 * Update the AUX buffer and cache the current buflet index,
>> +	 * as we need to know this and save the data when the trace
>> +	 * is ended out of the interrupt handler. End the trace
>> +	 * if the updating fails.
>> +	 */
>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +	else
>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>> +
>> +	return IRQ_HANDLED;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>> +{
>> +	struct hisi_ptt *hisi_ptt = context;
>> +	u32 status;
>> +
>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>> +		return IRQ_NONE;
>> +
>> +	return IRQ_WAKE_THREAD;
>> +}
>> +
>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>> +{
>> +	pci_free_irq_vectors(pdev);
>> +}
>> +
>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	int ret;
>> +
>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>> +	if (ret < 0) {
>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	ret = devm_request_threaded_irq(&pdev->dev,
>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>> +					"hisi-ptt", hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_update_filters(struct work_struct *work)
>> +{
>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>> +	struct hisi_ptt_filter_update_info info;
>> +	struct hisi_ptt_filter_desc *filter;
>> +	struct list_head *target_list;
>> +	struct hisi_ptt *hisi_ptt;
>> +
>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>> +
>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>> +		return;
>> +	}
>> +
>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>> +			      &hisi_ptt->req_filters;
>> +
>> +		if (info.is_add) {
>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>> +			if (!filter)
>> +				continue;
>> +
>> +			filter->pdev = info.pdev;
>> +			filter->val = info.val;
>> +
>> +			list_add_tail(&filter->list, target_list);
>> +		} else {
>> +			list_for_each_entry(filter, target_list, list)
>> +				if (filter->val == info.val) {
>> +					list_del(&filter->list);
>> +					kfree(filter);
>> +					break;
>> +				}
>> +		}
>> +
>> +		/* Update the available port mask */
>> +		if (!info.is_port)
>> +			continue;
>> +
>> +		if (info.is_add)
>> +			hisi_ptt->port_mask |= info.val;
>> +		else
>> +			hisi_ptt->port_mask &= ~info.val;
>> +	}
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +}
>> +
>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>> +				    struct hisi_ptt_filter_update_info *info)
>> +{
>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>> +
>> +	if (!root_port)
>> +		return;
>> +
>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>> +	if (info->port_devid < hisi_ptt->lower ||
>> +	    info->port_devid > hisi_ptt->upper)
>> +		return;
>> +
>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>> +
>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>> +				&hisi_ptt->filter_update_lock))
>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>> +	else
>> +		pci_warn(hisi_ptt->pdev,
>> +			 "filter update fifo overflow for target %s\n",
>> +			 pci_name(info->pdev));
>> +}
>> +
>> +/*
>> + * A PCI bus notifier is used here for dynamically updating the filter
>> + * list.
>> + */
>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>> +				  void *data)
>> +{
>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>> +	struct hisi_ptt_filter_update_info info;
>> +	struct device *dev = data;
>> +	struct pci_dev *pdev = to_pci_dev(dev);
>> +
>> +	info.pdev = pdev;
>> +
>> +	switch (action) {
>> +	case BUS_NOTIFY_ADD_DEVICE:
>> +		info.is_add = true;
>> +		break;
>> +	case BUS_NOTIFY_DEL_DEVICE:
>> +		info.is_add = false;
>> +		break;
>> +	default:
>> +		return 0;
>> +	}
>> +
>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>> +{
>> +	struct hisi_ptt_filter_update_info info = {
>> +		.pdev = pdev,
>> +		.is_add = true,
>> +	};
>> +	struct hisi_ptt *hisi_ptt = data;
>> +
>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>> +
>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>> +		list_del(&filter->list);
>> +		kfree(filter);
>> +	}
>> +
>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>> +		list_del(&filter->list);
>> +		kfree(filter);
>> +	}
>> +}
>> +
>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	struct pci_bus *bus;
>> +	u32 reg;
>> +
>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>> +
>> +	/*
>> +	 * The device range register provides the information about the
>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>> +	 * and the root ports it support are on the same PCIe core, with
>> +	 * same domain number but maybe different bus number. The device
>> +	 * range register will tell us which root ports we can support,
>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>> +	 * while Bit[15:0] indicates the lower.
>> +	 */
>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>> +	hisi_ptt->upper = reg >> 16;
>> +	hisi_ptt->lower = reg & 0xffff;
>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>> +
>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>> +
>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>> +	if (bus)
>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> 
> What happens if bus is NULL?  Should this be reported to the caller and cause
> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> 

It's ok if the list is NULL in the probe process,
as the device maybe hotplugged after the PTT driver probe, in which case
we can detect the event and update the list as we register a bus notifier.

>> +
>> +	/* Initialize trace controls */
>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>> +}
>> +
>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>> +
>> +static ssize_t available_filters_show(struct device *dev,
>> +				      struct device_attribute *attr,
>> +				      char *buf)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>> +	struct hisi_ptt_filter_desc *filter;
>> +	int pos = 0;
>> +
>> +	if (list_empty(&hisi_ptt->port_filters))
>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>> +				     pci_name(filter->pdev),
>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>> +
>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>> +				     pci_name(filter->pdev),
>> +				     hisi_ptt_get_filter_val(filter->pdev));
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return pos;
>> +}
>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>> +
>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>> +
>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>> +	&format_attr_filter.attr,
>> +	&format_attr_direction.attr,
>> +	&format_attr_type.attr,
>> +	&format_attr_format.attr,
>> +	NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>> +	.name = "format",
>> +	.attrs = hisi_ptt_pmu_format_attrs,
>> +};
>> +
>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>> +	&dev_attr_available_filters.attr,
>> +	NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>> +};
>> +
>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>> +	&hisi_ptt_pmu_format_group,
>> +	&hisi_ptt_pmu_filter_group,
>> +	NULL
>> +};
>> +
>> +/*
>> + * The supported value of the direction parameter. See hisi_ptt.rst
>> + * documentation for more details.
>> + */
>> +static u32 hisi_ptt_trace_available_direction[] = {
>> +	0,
>> +	1,
>> +	2,
>> +	3,
>> +};
>> +
>> +/* Different types can be set simultaneously */
>> +static u32 hisi_ptt_trace_available_type[] = {
>> +	1,	/* posted_request */
>> +	2,	/* non-posted_request */
>> +	4,	/* completion */
>> +};
>> +
>> +static u32 hisi_ptt_trace_availble_format[] = {
>> +	0,	/* 4DW */
>> +	1,	/* 8DW */
>> +};
>> +
>> +/*
>> + * Check whether the config is valid or not. Some configs are multi-selectable
>> + * and can be set simultaneously, while some are single selectable (onehot).
>> + * Use this function to check the non-onehot configs while
>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>> + */
>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>> +{
>> +	int i;
>> +
>> +	/*
>> +	 * The non-onehot configs cannot be 0. Walk the available
>> +	 * list and clear the valid bits of the config. If there
>> +	 * is any resident bit after the walk then the config is
>> +	 * invalid.
>> +	 */
>> +	for (i = 0; i < list_size; i++)
>> +		val &= ~available_list[i];
>> +
>> +	return val ? -EINVAL : 0;
>> +}
>> +
>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>> +{
>> +	int i, ret = -EINVAL;
>> +
>> +	for (i = 0; i < list_size; i++)
>> +		if (val == available_list[i]) {
>> +			ret = 0;
>> +			break;
>> +		}
>> +
>> +	return ret;
>> +}
>> +
>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>> +{
>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>> +	struct hisi_ptt_filter_desc *filter;
>> +	int ret = -EINVAL;
>> +
>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>> +
>> +	/*
>> +	 * Port filters are defined as bit mask. For port filters, check
>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>> +	 * and whether it's empty or not, otherwise user has specified
>> +	 * some unsupported root ports.
>> +	 *
>> +	 * For Requester ID filters, walk the available filter list to see
>> +	 * whether we have one matched.
>> +	 */
>> +	if (hisi_ptt->trace_ctrl.is_port &&
>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>> +		ret = 0;
>> +	} else {
>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> +			if (filter->val == val) {
>> +				ret = 0;
>> +				break;
>> +			}
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	hisi_ptt->trace_ctrl.filter = val;
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	int ret;
>> +	u32 val;
>> +
>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>> +		return -ENOENT;
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +
>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>> +	if (ret < 0)
>> +		goto out;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->direction = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>> +
>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->type = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->format = val;
>> +
>> +out:
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return ret;
>> +}
>> +
>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>> +				    int nr_pages, bool overwrite)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	struct page **pagelist;
>> +	int i;
>> +
>> +	if (overwrite) {
>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>> +		return NULL;
>> +	}
>> +
>> +	/* If the pages size less than buflets, we cannot start trace */
>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>> +		return NULL;
>> +
>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>> +	if (!buf)
>> +		return NULL;
>> +
>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>> +	if (!pagelist) {
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	for (i = 0; i < nr_pages; i++)
>> +		pagelist[i] = virt_to_page(pages[i]);
>> +
>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>> +	if (!buf->base) {
>> +		kfree(pagelist);
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	buf->nr_pages = nr_pages;
>> +	buf->length = nr_pages * PAGE_SIZE;
>> +	buf->pos = 0;
>> +
>> +	kfree(pagelist);
>> +	return buf;
>> +}
>> +
>> +static void hisi_ptt_pmu_free_aux(void *aux)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf = aux;
>> +
>> +	vunmap(buf->base);
>> +	kfree(buf);
>> +}
>> +
>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +	struct hw_perf_event *hwc = &event->hw;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	int cpu = event->cpu;
>> +	int ret;
>> +
>> +	hwc->state = 0;
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>> +		goto stop;
>> +	}
>> +
>> +	if (cpu == -1)
>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>> +
>> +	/*
>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>> +	 * core in event_function_local().
>> +	 */
>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +				 cpumask_of(cpu)));
>> +
>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>> +	if (ret) {
>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>> +		goto stop;
>> +	}
>> +
>> +	buf = perf_aux_output_begin(handle, event);
>> +	if (!buf) {
>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>> +		goto stop;
>> +	}
>> +
>> +	buf->pos = handle->head % buf->length;
>> +
>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>> +	if (ret) {
>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>> +		perf_aux_output_end(handle, 0);
>> +		goto stop;
>> +	}
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return;
>> +stop:
>> +	event->hw.state |= PERF_HES_STOPPED;
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +}
>> +
>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hw_perf_event *hwc = &event->hw;
>> +
>> +	if (hwc->state & PERF_HES_STOPPED)
>> +		return;
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>> +	}
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +
>> +	hwc->state |= PERF_HES_STOPPED;
>> +	perf_event_update_userpage(event);
>> +	hwc->state |= PERF_HES_UPTODATE;
>> +}
>> +
>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hw_perf_event *hwc = &event->hw;
>> +	int cpu = event->cpu;
>> +
>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>> +		return 0;
>> +
>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>> +
>> +	if (flags & PERF_EF_START) {
>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>> +		if (hwc->state & PERF_HES_STOPPED)
>> +			return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>> +{
>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>> +}
>> +
>> +static void hisi_ptt_unregister_pmu(void *priv)
>> +{
>> +	perf_pmu_unregister(priv);
>> +}
>> +
>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>> +{
>> +	char *pmu_name;
>> +	int ret;
>> +
>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>> +		.module		= THIS_MODULE,
>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>> +		.task_ctx_nr	= perf_sw_context,
>> +		.attr_groups	= hisi_ptt_pmu_groups,
>> +		.event_init	= hisi_ptt_pmu_event_init,
>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>> +		.start		= hisi_ptt_pmu_start,
>> +		.stop		= hisi_ptt_pmu_stop,
>> +		.add		= hisi_ptt_pmu_add,
>> +		.del		= hisi_ptt_pmu_del,
>> +	};
>> +
>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>> +	if (!pmu_name)
>> +		return -ENOMEM;
>> +
>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>> +					hisi_ptt_unregister_pmu,
>> +					&hisi_ptt->hisi_ptt_pmu);
>> +}
>> +
>> +/*
>> + * Get RMR address if provided by the firmware.
>> + * Return 0 if the IOMMU doesn't present or the policy of the
>> + * IOMMU domain is passthrough or we get a usable RMR region.
>> + * Otherwise a negative value is returned.
>> + */
>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	struct iommu_domain *iommu_domain;
>> +	struct iommu_resv_region *region;
>> +	LIST_HEAD(list);
>> +
>> +	/*
>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>> +	 * IOMMU domain is passthrough.
>> +	 */
>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> +		return 0;
>> +
>> +	iommu_get_resv_regions(&pdev->dev, &list);
>> +	list_for_each_entry(region, &list, list)
>> +		if (region->type == IOMMU_RESV_DIRECT &&
>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>> +			break;
>> +		}
>> +
>> +	iommu_put_resv_regions(&pdev->dev, &list);
>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>> +}
>> +
>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>> +			  const struct pci_device_id *id)
>> +{
>> +	struct hisi_ptt *hisi_ptt;
>> +	int ret;
>> +
>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>> +	if (!hisi_ptt)
>> +		return -ENOMEM;
>> +
>> +	mutex_init(&hisi_ptt->mutex);
>> +	hisi_ptt->pdev = pdev;
>> +
>> +	/*
>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>> +	 * so directly reference to the pci name string.
>> +	 */
>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>> +	pci_set_drvdata(pdev, hisi_ptt);
>> +
>> +	ret = pcim_enable_device(pdev);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>> +
>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>> +	if (ret) {
>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +	pci_set_master(pdev);
>> +
>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	hisi_ptt_init_ctrls(hisi_ptt);
>> +
>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>> +		return ret;
>> +	}
> 
> If this fails the pci_free_irq_vectors() is not called.
> 

It'll be called as we've registered a devm callback when
allocating the irq vector. See hisi_ptt_register_irq().

> I am out of time for today - I will continue tomorrow.
> 

Sure. Thanks!
Yicong

> Thanks,
> Mathieu
> 
>> +
>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>> +		return ret;
>> +	}
>> +
>> +	/* Register the bus notifier for dynamically updating the filter list */
>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>> +	if (ret)
>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>> +
>> +	return 0;
>> +}
>> +
>> +void hisi_ptt_remove(struct pci_dev *pdev)
>> +{
>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>> +
>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>> +
>> +	/* Cancel any work that has been queued */
>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>> +
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +
>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>> +	hisi_ptt_release_filters(hisi_ptt);
>> +}
>> +
>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>> +	{ }
>> +};
>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>> +
>> +static struct pci_driver hisi_ptt_driver = {
>> +	.name = "hisi_ptt",
>> +	.id_table = hisi_ptt_id_tbl,
>> +	.probe = hisi_ptt_probe,
>> +	.remove = hisi_ptt_remove,
>> +};
>> +module_pci_driver(hisi_ptt_driver);
>> +
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>> -- 
>> 2.33.0
>>
> .
> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

Hi Mathieu,

Thanks for the comments! Replies inline.

On 2021/11/25 2:51, Mathieu Poirier wrote:
> Hi Yicong,
> 
> On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>> integrated Endpoint(RCiEP) device, providing the capability
>> to dynamically monitor and tune the PCIe traffic(tune),
>> and trace the TLP headers(trace).
>>
> 
> Is there a reason to put "tune" and "trace" are whithin parentheses?  
> 

yeah. I want to use these single word to denote the related function, and
place them in the parantheses near the function's description to make sure
readers can match them.

>> Add the driver for the device to enable the trace function. The driver
>> will create PMU device for each PTT device, and users can start trace
>> through perf command.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>>  drivers/Makefile                       |    1 +
>>  drivers/hwtracing/Kconfig              |    2 +
>>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
> 
> Not sure about the "hisilicon" directory.  Right now we have "coresight",
> "intel_th" and "stm" - they concentrate on the technology rather than the
> vendor.  I think "ptt" would be just fine: 
> 
> drivers/hwtracing/ptt/Kconfig 
> drivers/hwtracing/ptt/Makefile
> drivers/hwtracing/ptt/hisi_ptt.c
> 
> That way if other vendors want to introduce support for the same kind of
> technology we can simply put them under "drivers/hwtracing/ptt/" and things are
> still accurate.
> 

sure. sounds sensible. will rename the directory to "ptt" if no objection.

> 
>>  5 files changed, 1159 insertions(+)
>>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
>>
>> diff --git a/drivers/Makefile b/drivers/Makefile
>> index be5d40ae1488..bb0dc9b55ea2 100644
>> --- a/drivers/Makefile
>> +++ b/drivers/Makefile
>> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>  obj-y				+= hwtracing/intel_th/
>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>>  obj-$(CONFIG_ANDROID)		+= android/
>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>  obj-$(CONFIG_FPGA)		+= fpga/
>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>> index 13085835a636..e3796b17541a 100644
>> --- a/drivers/hwtracing/Kconfig
>> +++ b/drivers/hwtracing/Kconfig
>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>  
>>  source "drivers/hwtracing/intel_th/Kconfig"
>>  
>> +source "drivers/hwtracing/hisilicon/Kconfig"
>> +
>>  endmenu
>> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
>> new file mode 100644
>> index 000000000000..9c3ab80d99fe
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/Kconfig
>> @@ -0,0 +1,8 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config HISI_PTT
>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>> +	depends on PCI && HAS_DMA && HAS_IOMEM
> 
> What Arm architecture are you aiming this for?  Can it be found on both armv7
> and armv8?
> 

currently the device will appear on Kunpeng 930, which is armv8.

>> +	help
>> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
>> +	  device, provides support for PCIe traffic tuning and
>> +	  tracing TLP headers to the memory.
> 
> Please indicate what the name of the driver will be when compiled as a module.
> 

sure. will add the module's name here.

>> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
>> new file mode 100644
>> index 000000000000..908c09a98161
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
>> new file mode 100644
>> index 000000000000..e11e9b6cc2a8
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
>> @@ -0,0 +1,1146 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for HiSilicon PCIe tune and trace device
>> + *
>> + * Copyright (c) 2021 HiSilicon Limited.
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/bitops.h>
>> +#include <linux/delay.h>
>> +#include <linux/dma-iommu.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io.h>
>> +#include <linux/iommu.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/kfifo.h>
>> +#include <linux/module.h>
>> +#include <linux/notifier.h>
>> +#include <linux/pci.h>
>> +#include <linux/perf_event.h>
>> +#include <linux/sizes.h>
>> +#include <linux/sysfs.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/workqueue.h>
>> +
>> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
>> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
>> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
>> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
>> +#define HISI_PTT_TRACE_CTRL		0x0850
>> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
>> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
>> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
>> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
>> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
>> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
>> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
>> +#define HISI_PTT_TRACE_INT_STAT		0x0890
>> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
>> +#define HISI_PTT_TRACE_INT_MASK		0x0894
>> +#define HISI_PTT_TRACE_WR_STS		0x08a0
>> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
>> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
>> +#define HISI_PTT_TRACE_STS		0x08b0
>> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
>> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
>> +#define HISI_PTT_LOCATION		0x0fe8
>> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
>> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> 
> Extra spaces between #define and the literals.  Please align the values with
> that of HISI_PTT_TRACE_DMA_IRQ below.
> 

Spaces are intended to distinguish the definition of the register offset and
related register fields. also used in include/uapi/linux/pci_regs.h.

HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
in the mail text, maybe because the tab is not 8 characters when display.

>> +
>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>> +
>> +/* Delay time for filter updating work */
>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>> +/* Wait time for DMA hardware to reset */
>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>> +/* Poll timeout and interval for waiting hardware work to finish */
>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>> +
>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>> +
>> +enum hisi_ptt_trace_status {
>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>> +	HISI_PTT_TRACE_STATUS_ON,
>> +};
>> +
>> +struct hisi_ptt_dma_buflet {
> 
> I'm not sure what a "buflet" is...  Probably best to just call this
> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> know due to the lack of proper structure documentation.  Please have a look at
> how "coresight_device" is documented.  The same applies to the rest of the
> structures declared below.
> 

It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
(we call it buflet) of 4MiB ...".

The total DMA buffer of PTT device is divided into 4 parts, and each part is called
a 'buflet', which means a small buffer (there exists other words with similiar
format: droplet, wavelet, ...).

The device is designed like this to make sure we won't lose any data when we have to change
the buffer address, the device can continue to write to the next buflet and don't
neet to pause.

>> +	struct list_head list;
>> +	unsigned int size;
>> +	dma_addr_t dma;
>> +
>> +	/*
>> +	 * The address of the buflet holding the trace data.
>> +	 * See Documentation/trace/hisi-ptt.rst for the details
> 
> As of this writing, hisi-ptt.rst doesn't exist.
> 

The comment intends to direct the user to the documentation if the user
want to know details about the data format. The data format is decribed
in the section 4 of the doc.

>> +	 * of the data format.
>> +	 */
>> +	void *addr;
>> +	int index;
>> +};
>> +
>> +struct hisi_ptt_trace_ctrl {
>> +	enum hisi_ptt_trace_status status;
>> +	struct perf_output_handle handle;
>> +	struct list_head trace_buf;
>> +	/*
>> +	 * The index of the buffer which trace data
>> +	 * currently is writing to.
>> +	 */
>> +	u32 buf_index;
>> +
>> +	int default_cpu;
>> +	u32 buflet_size;
>> +	bool is_port;
>> +	u32 format:1;		/* Format of the traced TLP headers */
>> +	u32 type:4;		/* Type of the TLP headers to trace */
>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>> +
>> +	phys_addr_t rmr_addr;
>> +	size_t rmr_length;
>> +	bool has_rmr;
>> +};
>> +
>> +struct hisi_ptt_filter_desc {
>> +	struct list_head list;
>> +	struct pci_dev *pdev;
>> +	u16 val;
>> +};
>> +
>> +struct hisi_ptt_pmu_buf {
>> +	size_t length;
>> +	int nr_pages;
>> +	void *base;
>> +	long pos;
>> +};
>> +
>> +/* Structure containing the information for filter updating */
>> +struct hisi_ptt_filter_update_info {
>> +	struct pci_dev *pdev;
>> +	u32 port_devid;
>> +	bool is_port;
>> +	bool is_add;
>> +	u16 val;
>> +};
>> +
>> +struct hisi_ptt {
>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>> +	struct notifier_block hisi_ptt_nb;
>> +	struct pmu hisi_ptt_pmu;
>> +	void __iomem *iobase;
>> +	struct pci_dev *pdev;
>> +	/*
>> +	 * Use the mutex to protect the filter list and
>> +	 * serialize the perf process.
>> +	 */
>> +	struct mutex mutex;
>> +	const char *name;
>> +	u16 core_id;
>> +	u16 sicl_id;
>> +	/* PCI device range managed by the PTT device */
>> +	u32 upper;
>> +	u32 lower;
>> +	u8 busnr;
>> +
>> +	/*
>> +	 * The trace TLP headers can either be filtered by certain
>> +	 * root port, or by the requester ID. Organize the filters
>> +	 * by @port_filters and @req_filters here. The mask of all
>> +	 * the valid ports is also cached for doing sanity check
>> +	 * of user input.
>> +	 */
>> +	struct list_head port_filters;
>> +	struct list_head req_filters;
>> +	u16 port_mask;
>> +
>> +	/*
>> +	 * We use a delayed work here to avoid indefinitely waiting for
>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>> +	 * work will be delayed only if the mutex can not be held,
>> +	 * otherwise no delay will be applied.
>> +	 */
>> +	struct delayed_work work;
>> +	spinlock_t filter_update_lock;
>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>> +};
> 
> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> 

I'm not sure whether it is common or permitted to provide a header which is only used by
this driver with a single file.

>> +
>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>> +{
>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>> +}
> 
> I'm not sure there is any value to this inline function.  I suggest to simply
> call container_of() whenever it is needed.
> 

ok.

>> +
>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>> +{
>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>> +
>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>> +}
>> +
>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>> +{
>> +	u32 val;
>> +
>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>> +				  val & HISI_PTT_TRACE_IDLE,
>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>> +}
>> +
>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>> +
>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>> +		list_del(&buflet->list);
>> +
>> +		if (ctrl->has_rmr)
>> +			memunmap(buflet->addr);
>> +		else
>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>> +					  buflet->dma);
>> +
>> +		kfree(buflet);
>> +	}
>> +}
>> +
>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	struct hisi_ptt_dma_buflet *buflet;
>> +	int i, ret;
>> +
>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>> +
>> +	/* Make sure the trace buffer is empty before allocating */
>> +	if (!list_empty(&ctrl->trace_buf)) {
>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>> +			memset(buflet->addr, 0, buflet->size);
>> +		return 0;
>> +	}
>> +
>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>> +		if (!buflet) {
>> +			ret = -ENOMEM;
>> +			goto err;
>> +		}
>> +
>> +		if (ctrl->has_rmr) {
>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>> +
>> +			buflet->dma = base;
>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>> +		} else {
>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>> +							  &buflet->dma, GFP_KERNEL);
>> +		}
>> +
>> +		if (!buflet->addr) {
>> +			kfree(buflet);
>> +			ret = -ENOMEM;
>> +			goto err;
>> +		}
>> +
>> +		memset(buflet->addr, 0, buflet->size);
>> +
>> +		buflet->index = i;
>> +		buflet->size = ctrl->buflet_size;
>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>> +	}
>> +
>> +	return 0;
>> +err:
>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>> +	return ret;
>> +}
>> +
>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>> +{
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>> +}
>> +
>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct hisi_ptt_dma_buflet *cur;
>> +	u32 val;
>> +
>> +	/* Check device idle before start trace */
>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>> +		return -EBUSY;
>> +	}
>> +
>> +	/* Reset the DMA before start tracing */
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	/*
>> +	 * We'll be in the perf context where preemption is disabled,
>> +	 * so use busy loop here.
>> +	 */
>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>> +
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	/* clear the interrupt status */
>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>> +
>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>> +		writel(lower_32_bits(cur->dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> +		writel(upper_32_bits(cur->dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> +	}
>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>> +
>> +	/* set the trace control register */
>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>> +	if (!hisi_ptt->trace_ctrl.is_port)
>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>> +
>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>> +{
>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +	struct perf_event *event = handle->event;
>> +	struct hisi_ptt_dma_buflet *cur;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +
>> +	buf = perf_get_aux(handle);
>> +	if (!buf || !handle->size)
>> +		return -EINVAL;
>> +
>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>> +		if (cur->index == index)
>> +			break;
>> +
>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>> +	memset(cur->addr, 0, cur->size);
>> +	buf->pos += cur->size;
>> +
>> +	if (stop) {
>> +		perf_aux_output_end(handle, buf->pos);
>> +	} else if (buf->length - buf->pos < cur->size) {
>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>> +		perf_aux_output_end(handle, buf->pos);
>> +
>> +		buf = perf_aux_output_begin(handle, event);
>> +		if (!buf)
>> +			return -EINVAL;
>> +
>> +		buf->pos = handle->head % buf->length;
>> +		if (buf->length - buf->pos < cur->size) {
>> +			perf_aux_output_end(handle, 0);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>> +{
>> +	struct hisi_ptt *hisi_ptt = context;
>> +	u32 status, buf_idx;
>> +
>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	buf_idx = ffs(status) - 1;
>> +
>> +	/* Clear the interrupt status of buflet @buf_idx */
>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +
>> +	/*
>> +	 * Update the AUX buffer and cache the current buflet index,
>> +	 * as we need to know this and save the data when the trace
>> +	 * is ended out of the interrupt handler. End the trace
>> +	 * if the updating fails.
>> +	 */
>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +	else
>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>> +
>> +	return IRQ_HANDLED;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>> +{
>> +	struct hisi_ptt *hisi_ptt = context;
>> +	u32 status;
>> +
>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>> +		return IRQ_NONE;
>> +
>> +	return IRQ_WAKE_THREAD;
>> +}
>> +
>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>> +{
>> +	pci_free_irq_vectors(pdev);
>> +}
>> +
>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	int ret;
>> +
>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>> +	if (ret < 0) {
>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	ret = devm_request_threaded_irq(&pdev->dev,
>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>> +					"hisi-ptt", hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_update_filters(struct work_struct *work)
>> +{
>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>> +	struct hisi_ptt_filter_update_info info;
>> +	struct hisi_ptt_filter_desc *filter;
>> +	struct list_head *target_list;
>> +	struct hisi_ptt *hisi_ptt;
>> +
>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>> +
>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>> +		return;
>> +	}
>> +
>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>> +			      &hisi_ptt->req_filters;
>> +
>> +		if (info.is_add) {
>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>> +			if (!filter)
>> +				continue;
>> +
>> +			filter->pdev = info.pdev;
>> +			filter->val = info.val;
>> +
>> +			list_add_tail(&filter->list, target_list);
>> +		} else {
>> +			list_for_each_entry(filter, target_list, list)
>> +				if (filter->val == info.val) {
>> +					list_del(&filter->list);
>> +					kfree(filter);
>> +					break;
>> +				}
>> +		}
>> +
>> +		/* Update the available port mask */
>> +		if (!info.is_port)
>> +			continue;
>> +
>> +		if (info.is_add)
>> +			hisi_ptt->port_mask |= info.val;
>> +		else
>> +			hisi_ptt->port_mask &= ~info.val;
>> +	}
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +}
>> +
>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>> +				    struct hisi_ptt_filter_update_info *info)
>> +{
>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>> +
>> +	if (!root_port)
>> +		return;
>> +
>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>> +	if (info->port_devid < hisi_ptt->lower ||
>> +	    info->port_devid > hisi_ptt->upper)
>> +		return;
>> +
>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>> +
>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>> +				&hisi_ptt->filter_update_lock))
>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>> +	else
>> +		pci_warn(hisi_ptt->pdev,
>> +			 "filter update fifo overflow for target %s\n",
>> +			 pci_name(info->pdev));
>> +}
>> +
>> +/*
>> + * A PCI bus notifier is used here for dynamically updating the filter
>> + * list.
>> + */
>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>> +				  void *data)
>> +{
>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>> +	struct hisi_ptt_filter_update_info info;
>> +	struct device *dev = data;
>> +	struct pci_dev *pdev = to_pci_dev(dev);
>> +
>> +	info.pdev = pdev;
>> +
>> +	switch (action) {
>> +	case BUS_NOTIFY_ADD_DEVICE:
>> +		info.is_add = true;
>> +		break;
>> +	case BUS_NOTIFY_DEL_DEVICE:
>> +		info.is_add = false;
>> +		break;
>> +	default:
>> +		return 0;
>> +	}
>> +
>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>> +{
>> +	struct hisi_ptt_filter_update_info info = {
>> +		.pdev = pdev,
>> +		.is_add = true,
>> +	};
>> +	struct hisi_ptt *hisi_ptt = data;
>> +
>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>> +
>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>> +		list_del(&filter->list);
>> +		kfree(filter);
>> +	}
>> +
>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>> +		list_del(&filter->list);
>> +		kfree(filter);
>> +	}
>> +}
>> +
>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	struct pci_bus *bus;
>> +	u32 reg;
>> +
>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>> +
>> +	/*
>> +	 * The device range register provides the information about the
>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>> +	 * and the root ports it support are on the same PCIe core, with
>> +	 * same domain number but maybe different bus number. The device
>> +	 * range register will tell us which root ports we can support,
>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>> +	 * while Bit[15:0] indicates the lower.
>> +	 */
>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>> +	hisi_ptt->upper = reg >> 16;
>> +	hisi_ptt->lower = reg & 0xffff;
>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>> +
>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>> +
>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>> +	if (bus)
>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> 
> What happens if bus is NULL?  Should this be reported to the caller and cause
> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> 

It's ok if the list is NULL in the probe process,
as the device maybe hotplugged after the PTT driver probe, in which case
we can detect the event and update the list as we register a bus notifier.

>> +
>> +	/* Initialize trace controls */
>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>> +}
>> +
>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>> +
>> +static ssize_t available_filters_show(struct device *dev,
>> +				      struct device_attribute *attr,
>> +				      char *buf)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>> +	struct hisi_ptt_filter_desc *filter;
>> +	int pos = 0;
>> +
>> +	if (list_empty(&hisi_ptt->port_filters))
>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>> +				     pci_name(filter->pdev),
>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>> +
>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>> +				     pci_name(filter->pdev),
>> +				     hisi_ptt_get_filter_val(filter->pdev));
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return pos;
>> +}
>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>> +
>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>> +
>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>> +	&format_attr_filter.attr,
>> +	&format_attr_direction.attr,
>> +	&format_attr_type.attr,
>> +	&format_attr_format.attr,
>> +	NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>> +	.name = "format",
>> +	.attrs = hisi_ptt_pmu_format_attrs,
>> +};
>> +
>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>> +	&dev_attr_available_filters.attr,
>> +	NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>> +};
>> +
>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>> +	&hisi_ptt_pmu_format_group,
>> +	&hisi_ptt_pmu_filter_group,
>> +	NULL
>> +};
>> +
>> +/*
>> + * The supported value of the direction parameter. See hisi_ptt.rst
>> + * documentation for more details.
>> + */
>> +static u32 hisi_ptt_trace_available_direction[] = {
>> +	0,
>> +	1,
>> +	2,
>> +	3,
>> +};
>> +
>> +/* Different types can be set simultaneously */
>> +static u32 hisi_ptt_trace_available_type[] = {
>> +	1,	/* posted_request */
>> +	2,	/* non-posted_request */
>> +	4,	/* completion */
>> +};
>> +
>> +static u32 hisi_ptt_trace_availble_format[] = {
>> +	0,	/* 4DW */
>> +	1,	/* 8DW */
>> +};
>> +
>> +/*
>> + * Check whether the config is valid or not. Some configs are multi-selectable
>> + * and can be set simultaneously, while some are single selectable (onehot).
>> + * Use this function to check the non-onehot configs while
>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>> + */
>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>> +{
>> +	int i;
>> +
>> +	/*
>> +	 * The non-onehot configs cannot be 0. Walk the available
>> +	 * list and clear the valid bits of the config. If there
>> +	 * is any resident bit after the walk then the config is
>> +	 * invalid.
>> +	 */
>> +	for (i = 0; i < list_size; i++)
>> +		val &= ~available_list[i];
>> +
>> +	return val ? -EINVAL : 0;
>> +}
>> +
>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>> +{
>> +	int i, ret = -EINVAL;
>> +
>> +	for (i = 0; i < list_size; i++)
>> +		if (val == available_list[i]) {
>> +			ret = 0;
>> +			break;
>> +		}
>> +
>> +	return ret;
>> +}
>> +
>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>> +{
>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>> +	struct hisi_ptt_filter_desc *filter;
>> +	int ret = -EINVAL;
>> +
>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>> +
>> +	/*
>> +	 * Port filters are defined as bit mask. For port filters, check
>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>> +	 * and whether it's empty or not, otherwise user has specified
>> +	 * some unsupported root ports.
>> +	 *
>> +	 * For Requester ID filters, walk the available filter list to see
>> +	 * whether we have one matched.
>> +	 */
>> +	if (hisi_ptt->trace_ctrl.is_port &&
>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>> +		ret = 0;
>> +	} else {
>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> +			if (filter->val == val) {
>> +				ret = 0;
>> +				break;
>> +			}
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	hisi_ptt->trace_ctrl.filter = val;
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	int ret;
>> +	u32 val;
>> +
>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>> +		return -ENOENT;
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +
>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>> +	if (ret < 0)
>> +		goto out;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->direction = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>> +
>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->type = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->format = val;
>> +
>> +out:
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return ret;
>> +}
>> +
>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>> +				    int nr_pages, bool overwrite)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	struct page **pagelist;
>> +	int i;
>> +
>> +	if (overwrite) {
>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>> +		return NULL;
>> +	}
>> +
>> +	/* If the pages size less than buflets, we cannot start trace */
>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>> +		return NULL;
>> +
>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>> +	if (!buf)
>> +		return NULL;
>> +
>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>> +	if (!pagelist) {
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	for (i = 0; i < nr_pages; i++)
>> +		pagelist[i] = virt_to_page(pages[i]);
>> +
>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>> +	if (!buf->base) {
>> +		kfree(pagelist);
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	buf->nr_pages = nr_pages;
>> +	buf->length = nr_pages * PAGE_SIZE;
>> +	buf->pos = 0;
>> +
>> +	kfree(pagelist);
>> +	return buf;
>> +}
>> +
>> +static void hisi_ptt_pmu_free_aux(void *aux)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf = aux;
>> +
>> +	vunmap(buf->base);
>> +	kfree(buf);
>> +}
>> +
>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +	struct hw_perf_event *hwc = &event->hw;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	int cpu = event->cpu;
>> +	int ret;
>> +
>> +	hwc->state = 0;
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>> +		goto stop;
>> +	}
>> +
>> +	if (cpu == -1)
>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>> +
>> +	/*
>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>> +	 * core in event_function_local().
>> +	 */
>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +				 cpumask_of(cpu)));
>> +
>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>> +	if (ret) {
>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>> +		goto stop;
>> +	}
>> +
>> +	buf = perf_aux_output_begin(handle, event);
>> +	if (!buf) {
>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>> +		goto stop;
>> +	}
>> +
>> +	buf->pos = handle->head % buf->length;
>> +
>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>> +	if (ret) {
>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>> +		perf_aux_output_end(handle, 0);
>> +		goto stop;
>> +	}
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return;
>> +stop:
>> +	event->hw.state |= PERF_HES_STOPPED;
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +}
>> +
>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hw_perf_event *hwc = &event->hw;
>> +
>> +	if (hwc->state & PERF_HES_STOPPED)
>> +		return;
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>> +	}
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +
>> +	hwc->state |= PERF_HES_STOPPED;
>> +	perf_event_update_userpage(event);
>> +	hwc->state |= PERF_HES_UPTODATE;
>> +}
>> +
>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hw_perf_event *hwc = &event->hw;
>> +	int cpu = event->cpu;
>> +
>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>> +		return 0;
>> +
>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>> +
>> +	if (flags & PERF_EF_START) {
>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>> +		if (hwc->state & PERF_HES_STOPPED)
>> +			return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>> +{
>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>> +}
>> +
>> +static void hisi_ptt_unregister_pmu(void *priv)
>> +{
>> +	perf_pmu_unregister(priv);
>> +}
>> +
>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>> +{
>> +	char *pmu_name;
>> +	int ret;
>> +
>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>> +		.module		= THIS_MODULE,
>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>> +		.task_ctx_nr	= perf_sw_context,
>> +		.attr_groups	= hisi_ptt_pmu_groups,
>> +		.event_init	= hisi_ptt_pmu_event_init,
>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>> +		.start		= hisi_ptt_pmu_start,
>> +		.stop		= hisi_ptt_pmu_stop,
>> +		.add		= hisi_ptt_pmu_add,
>> +		.del		= hisi_ptt_pmu_del,
>> +	};
>> +
>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>> +	if (!pmu_name)
>> +		return -ENOMEM;
>> +
>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>> +					hisi_ptt_unregister_pmu,
>> +					&hisi_ptt->hisi_ptt_pmu);
>> +}
>> +
>> +/*
>> + * Get RMR address if provided by the firmware.
>> + * Return 0 if the IOMMU doesn't present or the policy of the
>> + * IOMMU domain is passthrough or we get a usable RMR region.
>> + * Otherwise a negative value is returned.
>> + */
>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	struct iommu_domain *iommu_domain;
>> +	struct iommu_resv_region *region;
>> +	LIST_HEAD(list);
>> +
>> +	/*
>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>> +	 * IOMMU domain is passthrough.
>> +	 */
>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> +		return 0;
>> +
>> +	iommu_get_resv_regions(&pdev->dev, &list);
>> +	list_for_each_entry(region, &list, list)
>> +		if (region->type == IOMMU_RESV_DIRECT &&
>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>> +			break;
>> +		}
>> +
>> +	iommu_put_resv_regions(&pdev->dev, &list);
>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>> +}
>> +
>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>> +			  const struct pci_device_id *id)
>> +{
>> +	struct hisi_ptt *hisi_ptt;
>> +	int ret;
>> +
>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>> +	if (!hisi_ptt)
>> +		return -ENOMEM;
>> +
>> +	mutex_init(&hisi_ptt->mutex);
>> +	hisi_ptt->pdev = pdev;
>> +
>> +	/*
>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>> +	 * so directly reference to the pci name string.
>> +	 */
>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>> +	pci_set_drvdata(pdev, hisi_ptt);
>> +
>> +	ret = pcim_enable_device(pdev);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>> +
>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>> +	if (ret) {
>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +	pci_set_master(pdev);
>> +
>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	hisi_ptt_init_ctrls(hisi_ptt);
>> +
>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>> +		return ret;
>> +	}
> 
> If this fails the pci_free_irq_vectors() is not called.
> 

It'll be called as we've registered a devm callback when
allocating the irq vector. See hisi_ptt_register_irq().

> I am out of time for today - I will continue tomorrow.
> 

Sure. Thanks!
Yicong

> Thanks,
> Mathieu
> 
>> +
>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>> +		return ret;
>> +	}
>> +
>> +	/* Register the bus notifier for dynamically updating the filter list */
>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>> +	if (ret)
>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>> +
>> +	return 0;
>> +}
>> +
>> +void hisi_ptt_remove(struct pci_dev *pdev)
>> +{
>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>> +
>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>> +
>> +	/* Cancel any work that has been queued */
>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>> +
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +
>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>> +	hisi_ptt_release_filters(hisi_ptt);
>> +}
>> +
>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>> +	{ }
>> +};
>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>> +
>> +static struct pci_driver hisi_ptt_driver = {
>> +	.name = "hisi_ptt",
>> +	.id_table = hisi_ptt_id_tbl,
>> +	.probe = hisi_ptt_probe,
>> +	.remove = hisi_ptt_remove,
>> +};
>> +module_pci_driver(hisi_ptt_driver);
>> +
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>> -- 
>> 2.33.0
>>
> .
> 

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

Hi Mathieu,

Thanks for the comments! Replies inline.

On 2021/11/25 2:51, Mathieu Poirier wrote:
> Hi Yicong,
> 
> On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>> integrated Endpoint(RCiEP) device, providing the capability
>> to dynamically monitor and tune the PCIe traffic(tune),
>> and trace the TLP headers(trace).
>>
> 
> Is there a reason to put "tune" and "trace" are whithin parentheses?  
> 

yeah. I want to use these single word to denote the related function, and
place them in the parantheses near the function's description to make sure
readers can match them.

>> Add the driver for the device to enable the trace function. The driver
>> will create PMU device for each PTT device, and users can start trace
>> through perf command.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>>  drivers/Makefile                       |    1 +
>>  drivers/hwtracing/Kconfig              |    2 +
>>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
> 
> Not sure about the "hisilicon" directory.  Right now we have "coresight",
> "intel_th" and "stm" - they concentrate on the technology rather than the
> vendor.  I think "ptt" would be just fine: 
> 
> drivers/hwtracing/ptt/Kconfig 
> drivers/hwtracing/ptt/Makefile
> drivers/hwtracing/ptt/hisi_ptt.c
> 
> That way if other vendors want to introduce support for the same kind of
> technology we can simply put them under "drivers/hwtracing/ptt/" and things are
> still accurate.
> 

sure. sounds sensible. will rename the directory to "ptt" if no objection.

> 
>>  5 files changed, 1159 insertions(+)
>>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
>>
>> diff --git a/drivers/Makefile b/drivers/Makefile
>> index be5d40ae1488..bb0dc9b55ea2 100644
>> --- a/drivers/Makefile
>> +++ b/drivers/Makefile
>> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>  obj-y				+= hwtracing/intel_th/
>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>>  obj-$(CONFIG_ANDROID)		+= android/
>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>  obj-$(CONFIG_FPGA)		+= fpga/
>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>> index 13085835a636..e3796b17541a 100644
>> --- a/drivers/hwtracing/Kconfig
>> +++ b/drivers/hwtracing/Kconfig
>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>  
>>  source "drivers/hwtracing/intel_th/Kconfig"
>>  
>> +source "drivers/hwtracing/hisilicon/Kconfig"
>> +
>>  endmenu
>> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
>> new file mode 100644
>> index 000000000000..9c3ab80d99fe
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/Kconfig
>> @@ -0,0 +1,8 @@
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +config HISI_PTT
>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>> +	depends on PCI && HAS_DMA && HAS_IOMEM
> 
> What Arm architecture are you aiming this for?  Can it be found on both armv7
> and armv8?
> 

currently the device will appear on Kunpeng 930, which is armv8.

>> +	help
>> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
>> +	  device, provides support for PCIe traffic tuning and
>> +	  tracing TLP headers to the memory.
> 
> Please indicate what the name of the driver will be when compiled as a module.
> 

sure. will add the module's name here.

>> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
>> new file mode 100644
>> index 000000000000..908c09a98161
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/Makefile
>> @@ -0,0 +1,2 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
>> new file mode 100644
>> index 000000000000..e11e9b6cc2a8
>> --- /dev/null
>> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
>> @@ -0,0 +1,1146 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for HiSilicon PCIe tune and trace device
>> + *
>> + * Copyright (c) 2021 HiSilicon Limited.
>> + */
>> +
>> +#include <linux/bitfield.h>
>> +#include <linux/bitops.h>
>> +#include <linux/delay.h>
>> +#include <linux/dma-iommu.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io.h>
>> +#include <linux/iommu.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/kfifo.h>
>> +#include <linux/module.h>
>> +#include <linux/notifier.h>
>> +#include <linux/pci.h>
>> +#include <linux/perf_event.h>
>> +#include <linux/sizes.h>
>> +#include <linux/sysfs.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/workqueue.h>
>> +
>> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
>> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
>> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
>> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
>> +#define HISI_PTT_TRACE_CTRL		0x0850
>> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
>> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
>> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
>> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
>> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
>> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
>> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
>> +#define HISI_PTT_TRACE_INT_STAT		0x0890
>> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
>> +#define HISI_PTT_TRACE_INT_MASK		0x0894
>> +#define HISI_PTT_TRACE_WR_STS		0x08a0
>> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
>> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
>> +#define HISI_PTT_TRACE_STS		0x08b0
>> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
>> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
>> +#define HISI_PTT_LOCATION		0x0fe8
>> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
>> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> 
> Extra spaces between #define and the literals.  Please align the values with
> that of HISI_PTT_TRACE_DMA_IRQ below.
> 

Spaces are intended to distinguish the definition of the register offset and
related register fields. also used in include/uapi/linux/pci_regs.h.

HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
in the mail text, maybe because the tab is not 8 characters when display.

>> +
>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>> +
>> +/* Delay time for filter updating work */
>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>> +/* Wait time for DMA hardware to reset */
>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>> +/* Poll timeout and interval for waiting hardware work to finish */
>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>> +
>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>> +
>> +enum hisi_ptt_trace_status {
>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>> +	HISI_PTT_TRACE_STATUS_ON,
>> +};
>> +
>> +struct hisi_ptt_dma_buflet {
> 
> I'm not sure what a "buflet" is...  Probably best to just call this
> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> know due to the lack of proper structure documentation.  Please have a look at
> how "coresight_device" is documented.  The same applies to the rest of the
> structures declared below.
> 

It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
(we call it buflet) of 4MiB ...".

The total DMA buffer of PTT device is divided into 4 parts, and each part is called
a 'buflet', which means a small buffer (there exists other words with similiar
format: droplet, wavelet, ...).

The device is designed like this to make sure we won't lose any data when we have to change
the buffer address, the device can continue to write to the next buflet and don't
neet to pause.

>> +	struct list_head list;
>> +	unsigned int size;
>> +	dma_addr_t dma;
>> +
>> +	/*
>> +	 * The address of the buflet holding the trace data.
>> +	 * See Documentation/trace/hisi-ptt.rst for the details
> 
> As of this writing, hisi-ptt.rst doesn't exist.
> 

The comment intends to direct the user to the documentation if the user
want to know details about the data format. The data format is decribed
in the section 4 of the doc.

>> +	 * of the data format.
>> +	 */
>> +	void *addr;
>> +	int index;
>> +};
>> +
>> +struct hisi_ptt_trace_ctrl {
>> +	enum hisi_ptt_trace_status status;
>> +	struct perf_output_handle handle;
>> +	struct list_head trace_buf;
>> +	/*
>> +	 * The index of the buffer which trace data
>> +	 * currently is writing to.
>> +	 */
>> +	u32 buf_index;
>> +
>> +	int default_cpu;
>> +	u32 buflet_size;
>> +	bool is_port;
>> +	u32 format:1;		/* Format of the traced TLP headers */
>> +	u32 type:4;		/* Type of the TLP headers to trace */
>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>> +
>> +	phys_addr_t rmr_addr;
>> +	size_t rmr_length;
>> +	bool has_rmr;
>> +};
>> +
>> +struct hisi_ptt_filter_desc {
>> +	struct list_head list;
>> +	struct pci_dev *pdev;
>> +	u16 val;
>> +};
>> +
>> +struct hisi_ptt_pmu_buf {
>> +	size_t length;
>> +	int nr_pages;
>> +	void *base;
>> +	long pos;
>> +};
>> +
>> +/* Structure containing the information for filter updating */
>> +struct hisi_ptt_filter_update_info {
>> +	struct pci_dev *pdev;
>> +	u32 port_devid;
>> +	bool is_port;
>> +	bool is_add;
>> +	u16 val;
>> +};
>> +
>> +struct hisi_ptt {
>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>> +	struct notifier_block hisi_ptt_nb;
>> +	struct pmu hisi_ptt_pmu;
>> +	void __iomem *iobase;
>> +	struct pci_dev *pdev;
>> +	/*
>> +	 * Use the mutex to protect the filter list and
>> +	 * serialize the perf process.
>> +	 */
>> +	struct mutex mutex;
>> +	const char *name;
>> +	u16 core_id;
>> +	u16 sicl_id;
>> +	/* PCI device range managed by the PTT device */
>> +	u32 upper;
>> +	u32 lower;
>> +	u8 busnr;
>> +
>> +	/*
>> +	 * The trace TLP headers can either be filtered by certain
>> +	 * root port, or by the requester ID. Organize the filters
>> +	 * by @port_filters and @req_filters here. The mask of all
>> +	 * the valid ports is also cached for doing sanity check
>> +	 * of user input.
>> +	 */
>> +	struct list_head port_filters;
>> +	struct list_head req_filters;
>> +	u16 port_mask;
>> +
>> +	/*
>> +	 * We use a delayed work here to avoid indefinitely waiting for
>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>> +	 * work will be delayed only if the mutex can not be held,
>> +	 * otherwise no delay will be applied.
>> +	 */
>> +	struct delayed_work work;
>> +	spinlock_t filter_update_lock;
>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>> +};
> 
> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> 

I'm not sure whether it is common or permitted to provide a header which is only used by
this driver with a single file.

>> +
>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>> +{
>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>> +}
> 
> I'm not sure there is any value to this inline function.  I suggest to simply
> call container_of() whenever it is needed.
> 

ok.

>> +
>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>> +{
>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>> +
>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>> +}
>> +
>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>> +{
>> +	u32 val;
>> +
>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>> +				  val & HISI_PTT_TRACE_IDLE,
>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>> +}
>> +
>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>> +
>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>> +		list_del(&buflet->list);
>> +
>> +		if (ctrl->has_rmr)
>> +			memunmap(buflet->addr);
>> +		else
>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>> +					  buflet->dma);
>> +
>> +		kfree(buflet);
>> +	}
>> +}
>> +
>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct device *dev = &hisi_ptt->pdev->dev;
>> +	struct hisi_ptt_dma_buflet *buflet;
>> +	int i, ret;
>> +
>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>> +
>> +	/* Make sure the trace buffer is empty before allocating */
>> +	if (!list_empty(&ctrl->trace_buf)) {
>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>> +			memset(buflet->addr, 0, buflet->size);
>> +		return 0;
>> +	}
>> +
>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>> +		if (!buflet) {
>> +			ret = -ENOMEM;
>> +			goto err;
>> +		}
>> +
>> +		if (ctrl->has_rmr) {
>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>> +
>> +			buflet->dma = base;
>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>> +		} else {
>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>> +							  &buflet->dma, GFP_KERNEL);
>> +		}
>> +
>> +		if (!buflet->addr) {
>> +			kfree(buflet);
>> +			ret = -ENOMEM;
>> +			goto err;
>> +		}
>> +
>> +		memset(buflet->addr, 0, buflet->size);
>> +
>> +		buflet->index = i;
>> +		buflet->size = ctrl->buflet_size;
>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>> +	}
>> +
>> +	return 0;
>> +err:
>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>> +	return ret;
>> +}
>> +
>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>> +{
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>> +}
>> +
>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	struct hisi_ptt_dma_buflet *cur;
>> +	u32 val;
>> +
>> +	/* Check device idle before start trace */
>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>> +		return -EBUSY;
>> +	}
>> +
>> +	/* Reset the DMA before start tracing */
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	/*
>> +	 * We'll be in the perf context where preemption is disabled,
>> +	 * so use busy loop here.
>> +	 */
>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>> +
>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	/* clear the interrupt status */
>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>> +
>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>> +		writel(lower_32_bits(cur->dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> +		writel(upper_32_bits(cur->dma),
>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>> +	}
>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>> +
>> +	/* set the trace control register */
>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>> +	if (!hisi_ptt->trace_ctrl.is_port)
>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>> +
>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>> +
>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>> +{
>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +	struct perf_event *event = handle->event;
>> +	struct hisi_ptt_dma_buflet *cur;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +
>> +	buf = perf_get_aux(handle);
>> +	if (!buf || !handle->size)
>> +		return -EINVAL;
>> +
>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>> +		if (cur->index == index)
>> +			break;
>> +
>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>> +	memset(cur->addr, 0, cur->size);
>> +	buf->pos += cur->size;
>> +
>> +	if (stop) {
>> +		perf_aux_output_end(handle, buf->pos);
>> +	} else if (buf->length - buf->pos < cur->size) {
>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>> +		perf_aux_output_end(handle, buf->pos);
>> +
>> +		buf = perf_aux_output_begin(handle, event);
>> +		if (!buf)
>> +			return -EINVAL;
>> +
>> +		buf->pos = handle->head % buf->length;
>> +		if (buf->length - buf->pos < cur->size) {
>> +			perf_aux_output_end(handle, 0);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>> +{
>> +	struct hisi_ptt *hisi_ptt = context;
>> +	u32 status, buf_idx;
>> +
>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	buf_idx = ffs(status) - 1;
>> +
>> +	/* Clear the interrupt status of buflet @buf_idx */
>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +
>> +	/*
>> +	 * Update the AUX buffer and cache the current buflet index,
>> +	 * as we need to know this and save the data when the trace
>> +	 * is ended out of the interrupt handler. End the trace
>> +	 * if the updating fails.
>> +	 */
>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +	else
>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>> +
>> +	return IRQ_HANDLED;
>> +}
>> +
>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>> +{
>> +	struct hisi_ptt *hisi_ptt = context;
>> +	u32 status;
>> +
>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>> +		return IRQ_NONE;
>> +
>> +	return IRQ_WAKE_THREAD;
>> +}
>> +
>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>> +{
>> +	pci_free_irq_vectors(pdev);
>> +}
>> +
>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	int ret;
>> +
>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>> +	if (ret < 0) {
>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	ret = devm_request_threaded_irq(&pdev->dev,
>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>> +					"hisi-ptt", hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>> +		return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_update_filters(struct work_struct *work)
>> +{
>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>> +	struct hisi_ptt_filter_update_info info;
>> +	struct hisi_ptt_filter_desc *filter;
>> +	struct list_head *target_list;
>> +	struct hisi_ptt *hisi_ptt;
>> +
>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>> +
>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>> +		return;
>> +	}
>> +
>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>> +			      &hisi_ptt->req_filters;
>> +
>> +		if (info.is_add) {
>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>> +			if (!filter)
>> +				continue;
>> +
>> +			filter->pdev = info.pdev;
>> +			filter->val = info.val;
>> +
>> +			list_add_tail(&filter->list, target_list);
>> +		} else {
>> +			list_for_each_entry(filter, target_list, list)
>> +				if (filter->val == info.val) {
>> +					list_del(&filter->list);
>> +					kfree(filter);
>> +					break;
>> +				}
>> +		}
>> +
>> +		/* Update the available port mask */
>> +		if (!info.is_port)
>> +			continue;
>> +
>> +		if (info.is_add)
>> +			hisi_ptt->port_mask |= info.val;
>> +		else
>> +			hisi_ptt->port_mask &= ~info.val;
>> +	}
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +}
>> +
>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>> +				    struct hisi_ptt_filter_update_info *info)
>> +{
>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>> +
>> +	if (!root_port)
>> +		return;
>> +
>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>> +	if (info->port_devid < hisi_ptt->lower ||
>> +	    info->port_devid > hisi_ptt->upper)
>> +		return;
>> +
>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>> +
>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>> +				&hisi_ptt->filter_update_lock))
>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>> +	else
>> +		pci_warn(hisi_ptt->pdev,
>> +			 "filter update fifo overflow for target %s\n",
>> +			 pci_name(info->pdev));
>> +}
>> +
>> +/*
>> + * A PCI bus notifier is used here for dynamically updating the filter
>> + * list.
>> + */
>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>> +				  void *data)
>> +{
>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>> +	struct hisi_ptt_filter_update_info info;
>> +	struct device *dev = data;
>> +	struct pci_dev *pdev = to_pci_dev(dev);
>> +
>> +	info.pdev = pdev;
>> +
>> +	switch (action) {
>> +	case BUS_NOTIFY_ADD_DEVICE:
>> +		info.is_add = true;
>> +		break;
>> +	case BUS_NOTIFY_DEL_DEVICE:
>> +		info.is_add = false;
>> +		break;
>> +	default:
>> +		return 0;
>> +	}
>> +
>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>> +
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>> +{
>> +	struct hisi_ptt_filter_update_info info = {
>> +		.pdev = pdev,
>> +		.is_add = true,
>> +	};
>> +	struct hisi_ptt *hisi_ptt = data;
>> +
>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>> +
>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>> +		list_del(&filter->list);
>> +		kfree(filter);
>> +	}
>> +
>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>> +		list_del(&filter->list);
>> +		kfree(filter);
>> +	}
>> +}
>> +
>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	struct pci_bus *bus;
>> +	u32 reg;
>> +
>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>> +
>> +	/*
>> +	 * The device range register provides the information about the
>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>> +	 * and the root ports it support are on the same PCIe core, with
>> +	 * same domain number but maybe different bus number. The device
>> +	 * range register will tell us which root ports we can support,
>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>> +	 * while Bit[15:0] indicates the lower.
>> +	 */
>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>> +	hisi_ptt->upper = reg >> 16;
>> +	hisi_ptt->lower = reg & 0xffff;
>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>> +
>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>> +
>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>> +	if (bus)
>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> 
> What happens if bus is NULL?  Should this be reported to the caller and cause
> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> 

It's ok if the list is NULL in the probe process,
as the device maybe hotplugged after the PTT driver probe, in which case
we can detect the event and update the list as we register a bus notifier.

>> +
>> +	/* Initialize trace controls */
>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>> +}
>> +
>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>> +
>> +static ssize_t available_filters_show(struct device *dev,
>> +				      struct device_attribute *attr,
>> +				      char *buf)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>> +	struct hisi_ptt_filter_desc *filter;
>> +	int pos = 0;
>> +
>> +	if (list_empty(&hisi_ptt->port_filters))
>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>> +				     pci_name(filter->pdev),
>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>> +
>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>> +				     pci_name(filter->pdev),
>> +				     hisi_ptt_get_filter_val(filter->pdev));
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return pos;
>> +}
>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>> +
>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>> +
>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>> +	&format_attr_filter.attr,
>> +	&format_attr_direction.attr,
>> +	&format_attr_type.attr,
>> +	&format_attr_format.attr,
>> +	NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>> +	.name = "format",
>> +	.attrs = hisi_ptt_pmu_format_attrs,
>> +};
>> +
>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>> +	&dev_attr_available_filters.attr,
>> +	NULL
>> +};
>> +
>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>> +};
>> +
>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>> +	&hisi_ptt_pmu_format_group,
>> +	&hisi_ptt_pmu_filter_group,
>> +	NULL
>> +};
>> +
>> +/*
>> + * The supported value of the direction parameter. See hisi_ptt.rst
>> + * documentation for more details.
>> + */
>> +static u32 hisi_ptt_trace_available_direction[] = {
>> +	0,
>> +	1,
>> +	2,
>> +	3,
>> +};
>> +
>> +/* Different types can be set simultaneously */
>> +static u32 hisi_ptt_trace_available_type[] = {
>> +	1,	/* posted_request */
>> +	2,	/* non-posted_request */
>> +	4,	/* completion */
>> +};
>> +
>> +static u32 hisi_ptt_trace_availble_format[] = {
>> +	0,	/* 4DW */
>> +	1,	/* 8DW */
>> +};
>> +
>> +/*
>> + * Check whether the config is valid or not. Some configs are multi-selectable
>> + * and can be set simultaneously, while some are single selectable (onehot).
>> + * Use this function to check the non-onehot configs while
>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>> + */
>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>> +{
>> +	int i;
>> +
>> +	/*
>> +	 * The non-onehot configs cannot be 0. Walk the available
>> +	 * list and clear the valid bits of the config. If there
>> +	 * is any resident bit after the walk then the config is
>> +	 * invalid.
>> +	 */
>> +	for (i = 0; i < list_size; i++)
>> +		val &= ~available_list[i];
>> +
>> +	return val ? -EINVAL : 0;
>> +}
>> +
>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>> +{
>> +	int i, ret = -EINVAL;
>> +
>> +	for (i = 0; i < list_size; i++)
>> +		if (val == available_list[i]) {
>> +			ret = 0;
>> +			break;
>> +		}
>> +
>> +	return ret;
>> +}
>> +
>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>> +{
>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>> +	struct hisi_ptt_filter_desc *filter;
>> +	int ret = -EINVAL;
>> +
>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>> +
>> +	/*
>> +	 * Port filters are defined as bit mask. For port filters, check
>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>> +	 * and whether it's empty or not, otherwise user has specified
>> +	 * some unsupported root ports.
>> +	 *
>> +	 * For Requester ID filters, walk the available filter list to see
>> +	 * whether we have one matched.
>> +	 */
>> +	if (hisi_ptt->trace_ctrl.is_port &&
>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>> +		ret = 0;
>> +	} else {
>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>> +			if (filter->val == val) {
>> +				ret = 0;
>> +				break;
>> +			}
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	hisi_ptt->trace_ctrl.filter = val;
>> +	return 0;
>> +}
>> +
>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>> +	int ret;
>> +	u32 val;
>> +
>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>> +		return -ENOENT;
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +
>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>> +	if (ret < 0)
>> +		goto out;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->direction = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>> +
>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->type = val;
>> +
>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>> +	if (ret < 0)
>> +		goto out;
>> +	ctrl->format = val;
>> +
>> +out:
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return ret;
>> +}
>> +
>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>> +				    int nr_pages, bool overwrite)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	struct page **pagelist;
>> +	int i;
>> +
>> +	if (overwrite) {
>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>> +		return NULL;
>> +	}
>> +
>> +	/* If the pages size less than buflets, we cannot start trace */
>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>> +		return NULL;
>> +
>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>> +	if (!buf)
>> +		return NULL;
>> +
>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>> +	if (!pagelist) {
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	for (i = 0; i < nr_pages; i++)
>> +		pagelist[i] = virt_to_page(pages[i]);
>> +
>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>> +	if (!buf->base) {
>> +		kfree(pagelist);
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	buf->nr_pages = nr_pages;
>> +	buf->length = nr_pages * PAGE_SIZE;
>> +	buf->pos = 0;
>> +
>> +	kfree(pagelist);
>> +	return buf;
>> +}
>> +
>> +static void hisi_ptt_pmu_free_aux(void *aux)
>> +{
>> +	struct hisi_ptt_pmu_buf *buf = aux;
>> +
>> +	vunmap(buf->base);
>> +	kfree(buf);
>> +}
>> +
>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>> +	struct hw_perf_event *hwc = &event->hw;
>> +	struct hisi_ptt_pmu_buf *buf;
>> +	int cpu = event->cpu;
>> +	int ret;
>> +
>> +	hwc->state = 0;
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>> +		goto stop;
>> +	}
>> +
>> +	if (cpu == -1)
>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>> +
>> +	/*
>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>> +	 * core in event_function_local().
>> +	 */
>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>> +				 cpumask_of(cpu)));
>> +
>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>> +	if (ret) {
>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>> +		goto stop;
>> +	}
>> +
>> +	buf = perf_aux_output_begin(handle, event);
>> +	if (!buf) {
>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>> +		goto stop;
>> +	}
>> +
>> +	buf->pos = handle->head % buf->length;
>> +
>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>> +	if (ret) {
>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>> +		perf_aux_output_end(handle, 0);
>> +		goto stop;
>> +	}
>> +
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +	return;
>> +stop:
>> +	event->hw.state |= PERF_HES_STOPPED;
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +}
>> +
>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hw_perf_event *hwc = &event->hw;
>> +
>> +	if (hwc->state & PERF_HES_STOPPED)
>> +		return;
>> +
>> +	mutex_lock(&hisi_ptt->mutex);
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>> +	}
>> +	mutex_unlock(&hisi_ptt->mutex);
>> +
>> +	hwc->state |= PERF_HES_STOPPED;
>> +	perf_event_update_userpage(event);
>> +	hwc->state |= PERF_HES_UPTODATE;
>> +}
>> +
>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>> +{
>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>> +	struct hw_perf_event *hwc = &event->hw;
>> +	int cpu = event->cpu;
>> +
>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>> +		return 0;
>> +
>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>> +
>> +	if (flags & PERF_EF_START) {
>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>> +		if (hwc->state & PERF_HES_STOPPED)
>> +			return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>> +{
>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>> +}
>> +
>> +static void hisi_ptt_unregister_pmu(void *priv)
>> +{
>> +	perf_pmu_unregister(priv);
>> +}
>> +
>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>> +{
>> +	char *pmu_name;
>> +	int ret;
>> +
>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>> +		.module		= THIS_MODULE,
>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>> +		.task_ctx_nr	= perf_sw_context,
>> +		.attr_groups	= hisi_ptt_pmu_groups,
>> +		.event_init	= hisi_ptt_pmu_event_init,
>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>> +		.start		= hisi_ptt_pmu_start,
>> +		.stop		= hisi_ptt_pmu_stop,
>> +		.add		= hisi_ptt_pmu_add,
>> +		.del		= hisi_ptt_pmu_del,
>> +	};
>> +
>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>> +	if (!pmu_name)
>> +		return -ENOMEM;
>> +
>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>> +					hisi_ptt_unregister_pmu,
>> +					&hisi_ptt->hisi_ptt_pmu);
>> +}
>> +
>> +/*
>> + * Get RMR address if provided by the firmware.
>> + * Return 0 if the IOMMU doesn't present or the policy of the
>> + * IOMMU domain is passthrough or we get a usable RMR region.
>> + * Otherwise a negative value is returned.
>> + */
>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>> +{
>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>> +	struct iommu_domain *iommu_domain;
>> +	struct iommu_resv_region *region;
>> +	LIST_HEAD(list);
>> +
>> +	/*
>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>> +	 * IOMMU domain is passthrough.
>> +	 */
>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>> +		return 0;
>> +
>> +	iommu_get_resv_regions(&pdev->dev, &list);
>> +	list_for_each_entry(region, &list, list)
>> +		if (region->type == IOMMU_RESV_DIRECT &&
>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>> +			break;
>> +		}
>> +
>> +	iommu_put_resv_regions(&pdev->dev, &list);
>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>> +}
>> +
>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>> +			  const struct pci_device_id *id)
>> +{
>> +	struct hisi_ptt *hisi_ptt;
>> +	int ret;
>> +
>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>> +	if (!hisi_ptt)
>> +		return -ENOMEM;
>> +
>> +	mutex_init(&hisi_ptt->mutex);
>> +	hisi_ptt->pdev = pdev;
>> +
>> +	/*
>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>> +	 * so directly reference to the pci name string.
>> +	 */
>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>> +	pci_set_drvdata(pdev, hisi_ptt);
>> +
>> +	ret = pcim_enable_device(pdev);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>> +
>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>> +	if (ret) {
>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>> +		return ret;
>> +	}
>> +	pci_set_master(pdev);
>> +
>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>> +	if (ret)
>> +		return ret;
>> +
>> +	hisi_ptt_init_ctrls(hisi_ptt);
>> +
>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>> +		return ret;
>> +	}
> 
> If this fails the pci_free_irq_vectors() is not called.
> 

It'll be called as we've registered a devm callback when
allocating the irq vector. See hisi_ptt_register_irq().

> I am out of time for today - I will continue tomorrow.
> 

Sure. Thanks!
Yicong

> Thanks,
> Mathieu
> 
>> +
>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>> +	if (ret) {
>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>> +		return ret;
>> +	}
>> +
>> +	/* Register the bus notifier for dynamically updating the filter list */
>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>> +	if (ret)
>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>> +
>> +	return 0;
>> +}
>> +
>> +void hisi_ptt_remove(struct pci_dev *pdev)
>> +{
>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>> +
>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>> +
>> +	/* Cancel any work that has been queued */
>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>> +
>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>> +		hisi_ptt_trace_end(hisi_ptt);
>> +
>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>> +	hisi_ptt_release_filters(hisi_ptt);
>> +}
>> +
>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>> +	{ }
>> +};
>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>> +
>> +static struct pci_driver hisi_ptt_driver = {
>> +	.name = "hisi_ptt",
>> +	.id_table = hisi_ptt_id_tbl,
>> +	.probe = hisi_ptt_probe,
>> +	.remove = hisi_ptt_remove,
>> +};
>> +module_pci_driver(hisi_ptt_driver);
>> +
>> +MODULE_LICENSE("GPL v2");
>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>> -- 
>> 2.33.0
>>
> .
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

On Thu, Nov 25, 2021 at 04:39:46PM +0800, Yicong Yang wrote:
> Hi Mathieu,
> 
> Thanks for the comments! Replies inline.
> 
> On 2021/11/25 2:51, Mathieu Poirier wrote:
> > Hi Yicong,
> > 
> > On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
> >> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> >> integrated Endpoint(RCiEP) device, providing the capability
> >> to dynamically monitor and tune the PCIe traffic(tune),
> >> and trace the TLP headers(trace).
> >>
> > 
> > Is there a reason to put "tune" and "trace" are whithin parentheses?  
> > 
> 
> yeah. I want to use these single word to denote the related function, and
> place them in the parantheses near the function's description to make sure
> readers can match them.
> 

Probably a better idea to simply write "TLP headers" and "PCIe traffic" when
referring to them.  Otherwise it is very confusing because the reader doesn't
know if you mean the acronym or the real definition of "tune" and "trace".

> >> Add the driver for the device to enable the trace function. The driver
> >> will create PMU device for each PTT device, and users can start trace
> >> through perf command.
> >>
> >> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> >> ---
> >>  drivers/Makefile                       |    1 +
> >>  drivers/hwtracing/Kconfig              |    2 +
> >>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
> >>  drivers/hwtracing/hisilicon/Makefile   |    2 +
> >>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
> > 
> > Not sure about the "hisilicon" directory.  Right now we have "coresight",
> > "intel_th" and "stm" - they concentrate on the technology rather than the
> > vendor.  I think "ptt" would be just fine: 
> > 
> > drivers/hwtracing/ptt/Kconfig 
> > drivers/hwtracing/ptt/Makefile
> > drivers/hwtracing/ptt/hisi_ptt.c
> > 
> > That way if other vendors want to introduce support for the same kind of
> > technology we can simply put them under "drivers/hwtracing/ptt/" and things are
> > still accurate.
> > 
> 
> sure. sounds sensible. will rename the directory to "ptt" if no objection.
> 
> > 
> >>  5 files changed, 1159 insertions(+)
> >>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
> >>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
> >>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
> >>
> >> diff --git a/drivers/Makefile b/drivers/Makefile
> >> index be5d40ae1488..bb0dc9b55ea2 100644
> >> --- a/drivers/Makefile
> >> +++ b/drivers/Makefile
> >> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
> >>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
> >>  obj-y				+= hwtracing/intel_th/
> >>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> >> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
> >>  obj-$(CONFIG_ANDROID)		+= android/
> >>  obj-$(CONFIG_NVMEM)		+= nvmem/
> >>  obj-$(CONFIG_FPGA)		+= fpga/
> >> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> >> index 13085835a636..e3796b17541a 100644
> >> --- a/drivers/hwtracing/Kconfig
> >> +++ b/drivers/hwtracing/Kconfig
> >> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
> >>  
> >>  source "drivers/hwtracing/intel_th/Kconfig"
> >>  
> >> +source "drivers/hwtracing/hisilicon/Kconfig"
> >> +
> >>  endmenu
> >> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
> >> new file mode 100644
> >> index 000000000000..9c3ab80d99fe
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/Kconfig
> >> @@ -0,0 +1,8 @@
> >> +# SPDX-License-Identifier: GPL-2.0-only
> >> +config HISI_PTT
> >> +	tristate "HiSilicon PCIe Tune and Trace Device"
> >> +	depends on PCI && HAS_DMA && HAS_IOMEM
> > 
> > What Arm architecture are you aiming this for?  Can it be found on both armv7
> > and armv8?
> > 
> 
> currently the device will appear on Kunpeng 930, which is armv8.

Ok, then please make it dependent on armv8.

> 
> >> +	help
> >> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
> >> +	  device, provides support for PCIe traffic tuning and
> >> +	  tracing TLP headers to the memory.
> > 
> > Please indicate what the name of the driver will be when compiled as a module.
> > 
> 
> sure. will add the module's name here.
> 
> >> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
> >> new file mode 100644
> >> index 000000000000..908c09a98161
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/Makefile
> >> @@ -0,0 +1,2 @@
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> >> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
> >> new file mode 100644
> >> index 000000000000..e11e9b6cc2a8
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
> >> @@ -0,0 +1,1146 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * Driver for HiSilicon PCIe tune and trace device
> >> + *
> >> + * Copyright (c) 2021 HiSilicon Limited.
> >> + */
> >> +
> >> +#include <linux/bitfield.h>
> >> +#include <linux/bitops.h>
> >> +#include <linux/delay.h>
> >> +#include <linux/dma-iommu.h>
> >> +#include <linux/dma-mapping.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/io.h>
> >> +#include <linux/iommu.h>
> >> +#include <linux/iopoll.h>
> >> +#include <linux/kfifo.h>
> >> +#include <linux/module.h>
> >> +#include <linux/notifier.h>
> >> +#include <linux/pci.h>
> >> +#include <linux/perf_event.h>
> >> +#include <linux/sizes.h>
> >> +#include <linux/sysfs.h>
> >> +#include <linux/vmalloc.h>
> >> +#include <linux/workqueue.h>
> >> +
> >> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> >> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> >> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> >> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> >> +#define HISI_PTT_TRACE_CTRL		0x0850
> >> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> >> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> >> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> >> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> >> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> >> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> >> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> >> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> >> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> >> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> >> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> >> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> >> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> >> +#define HISI_PTT_TRACE_STS		0x08b0
> >> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> >> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> >> +#define HISI_PTT_LOCATION		0x0fe8
> >> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> >> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> > 
> > Extra spaces between #define and the literals.  Please align the values with
> > that of HISI_PTT_TRACE_DMA_IRQ below.
> > 
> 
> Spaces are intended to distinguish the definition of the register offset and
> related register fields. also used in include/uapi/linux/pci_regs.h.
>

Interesting and definitely not the norm...

> HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
> in the mail text, maybe because the tab is not 8 characters when display.
> 

In my VI terminal the '0' of HISI_PTT_TRACE_DMA_IRQ is align with the '3' in
GENMASK(31, 16) and I suspect it is the same for other people.

> >> +
> >> +#define HISI_PTT_TRACE_DMA_IRQ			0
> >> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> >> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> >> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> >> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> >> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> >> +
> >> +/* Delay time for filter updating work */
> >> +#define HISI_PTT_WORK_DELAY_MS		100UL
> >> +/* Wait time for DMA hardware to reset */
> >> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> >> +/* Poll timeout and interval for waiting hardware work to finish */
> >> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> >> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> >> +
> >> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> >> +
> >> +enum hisi_ptt_trace_status {
> >> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> >> +	HISI_PTT_TRACE_STATUS_ON,
> >> +};
> >> +
> >> +struct hisi_ptt_dma_buflet {
> > 
> > I'm not sure what a "buflet" is...  Probably best to just call this
> > hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> > know due to the lack of proper structure documentation.  Please have a look at
> > how "coresight_device" is documented.  The same applies to the rest of the
> > structures declared below.
> > 
> 
> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
> (we call it buflet) of 4MiB ...".

I just noticed the documentation in patch 5 - I will read it before continuing
with this set.

> 
> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
> a 'buflet', which means a small buffer (there exists other words with similiar
> format: droplet, wavelet, ...).
> 
> The device is designed like this to make sure we won't lose any data when we have to change
> the buffer address, the device can continue to write to the next buflet and don't
> neet to pause.
> 
> >> +	struct list_head list;
> >> +	unsigned int size;
> >> +	dma_addr_t dma;
> >> +
> >> +	/*
> >> +	 * The address of the buflet holding the trace data.
> >> +	 * See Documentation/trace/hisi-ptt.rst for the details
> > 
> > As of this writing, hisi-ptt.rst doesn't exist.
> > 
> 
> The comment intends to direct the user to the documentation if the user
> want to know details about the data format. The data format is decribed
> in the section 4 of the doc.
> 
> >> +	 * of the data format.
> >> +	 */
> >> +	void *addr;
> >> +	int index;
> >> +};
> >> +
> >> +struct hisi_ptt_trace_ctrl {
> >> +	enum hisi_ptt_trace_status status;
> >> +	struct perf_output_handle handle;
> >> +	struct list_head trace_buf;
> >> +	/*
> >> +	 * The index of the buffer which trace data
> >> +	 * currently is writing to.
> >> +	 */
> >> +	u32 buf_index;
> >> +
> >> +	int default_cpu;
> >> +	u32 buflet_size;
> >> +	bool is_port;
> >> +	u32 format:1;		/* Format of the traced TLP headers */
> >> +	u32 type:4;		/* Type of the TLP headers to trace */
> >> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> >> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> >> +
> >> +	phys_addr_t rmr_addr;
> >> +	size_t rmr_length;
> >> +	bool has_rmr;
> >> +};
> >> +
> >> +struct hisi_ptt_filter_desc {
> >> +	struct list_head list;
> >> +	struct pci_dev *pdev;
> >> +	u16 val;
> >> +};
> >> +
> >> +struct hisi_ptt_pmu_buf {
> >> +	size_t length;
> >> +	int nr_pages;
> >> +	void *base;
> >> +	long pos;
> >> +};
> >> +
> >> +/* Structure containing the information for filter updating */
> >> +struct hisi_ptt_filter_update_info {
> >> +	struct pci_dev *pdev;
> >> +	u32 port_devid;
> >> +	bool is_port;
> >> +	bool is_add;
> >> +	u16 val;
> >> +};
> >> +
> >> +struct hisi_ptt {
> >> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> >> +	struct notifier_block hisi_ptt_nb;
> >> +	struct pmu hisi_ptt_pmu;
> >> +	void __iomem *iobase;
> >> +	struct pci_dev *pdev;
> >> +	/*
> >> +	 * Use the mutex to protect the filter list and
> >> +	 * serialize the perf process.
> >> +	 */
> >> +	struct mutex mutex;
> >> +	const char *name;
> >> +	u16 core_id;
> >> +	u16 sicl_id;
> >> +	/* PCI device range managed by the PTT device */
> >> +	u32 upper;
> >> +	u32 lower;
> >> +	u8 busnr;
> >> +
> >> +	/*
> >> +	 * The trace TLP headers can either be filtered by certain
> >> +	 * root port, or by the requester ID. Organize the filters
> >> +	 * by @port_filters and @req_filters here. The mask of all
> >> +	 * the valid ports is also cached for doing sanity check
> >> +	 * of user input.
> >> +	 */
> >> +	struct list_head port_filters;
> >> +	struct list_head req_filters;
> >> +	u16 port_mask;
> >> +
> >> +	/*
> >> +	 * We use a delayed work here to avoid indefinitely waiting for
> >> +	 * the hisi_ptt->mutex which protecting the filter list. The
> >> +	 * work will be delayed only if the mutex can not be held,
> >> +	 * otherwise no delay will be applied.
> >> +	 */
> >> +	struct delayed_work work;
> >> +	spinlock_t filter_update_lock;
> >> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> >> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> >> +};
> > 
> > It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> > 
> 
> I'm not sure whether it is common or permitted to provide a header which is only used by
> this driver with a single file.

I've seen it done many times.

> 
> >> +
> >> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> >> +{
> >> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> >> +}
> > 
> > I'm not sure there is any value to this inline function.  I suggest to simply
> > call container_of() whenever it is needed.
> > 
> 
> ok.
> 
> >> +
> >> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> >> +{
> >> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> >> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> >> +
> >> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> >> +}
> >> +
> >> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	u32 val;
> >> +
> >> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> >> +				  val & HISI_PTT_TRACE_IDLE,
> >> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> >> +				  HISI_PTT_WAIT_TIMEOUT_US);
> >> +}
> >> +
> >> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct device *dev = &hisi_ptt->pdev->dev;
> >> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> >> +
> >> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> >> +		list_del(&buflet->list);
> >> +
> >> +		if (ctrl->has_rmr)
> >> +			memunmap(buflet->addr);
> >> +		else
> >> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> >> +					  buflet->dma);
> >> +
> >> +		kfree(buflet);
> >> +	}
> >> +}
> >> +
> >> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct device *dev = &hisi_ptt->pdev->dev;
> >> +	struct hisi_ptt_dma_buflet *buflet;
> >> +	int i, ret;
> >> +
> >> +	hisi_ptt->trace_ctrl.buf_index = 0;
> >> +
> >> +	/* Make sure the trace buffer is empty before allocating */
> >> +	if (!list_empty(&ctrl->trace_buf)) {
> >> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> >> +			memset(buflet->addr, 0, buflet->size);
> >> +		return 0;
> >> +	}
> >> +
> >> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> >> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> >> +		if (!buflet) {
> >> +			ret = -ENOMEM;
> >> +			goto err;
> >> +		}
> >> +
> >> +		if (ctrl->has_rmr) {
> >> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> >> +
> >> +			buflet->dma = base;
> >> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> >> +		} else {
> >> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> >> +							  &buflet->dma, GFP_KERNEL);
> >> +		}
> >> +
> >> +		if (!buflet->addr) {
> >> +			kfree(buflet);
> >> +			ret = -ENOMEM;
> >> +			goto err;
> >> +		}
> >> +
> >> +		memset(buflet->addr, 0, buflet->size);
> >> +
> >> +		buflet->index = i;
> >> +		buflet->size = ctrl->buflet_size;
> >> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> >> +	}
> >> +
> >> +	return 0;
> >> +err:
> >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> >> +	return ret;
> >> +}
> >> +
> >> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct hisi_ptt_dma_buflet *cur;
> >> +	u32 val;
> >> +
> >> +	/* Check device idle before start trace */
> >> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> >> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> >> +		return -EBUSY;
> >> +	}
> >> +
> >> +	/* Reset the DMA before start tracing */
> >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	val |= HISI_PTT_TRACE_CTRL_RST;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	/*
> >> +	 * We'll be in the perf context where preemption is disabled,
> >> +	 * so use busy loop here.
> >> +	 */
> >> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> >> +
> >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	/* clear the interrupt status */
> >> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> >> +
> >> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> >> +		writel(lower_32_bits(cur->dma),
> >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> >> +		writel(upper_32_bits(cur->dma),
> >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> >> +	}
> >> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> >> +
> >> +	/* set the trace control register */
> >> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> >> +	if (!hisi_ptt->trace_ctrl.is_port)
> >> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> >> +
> >> +	val |= HISI_PTT_TRACE_CTRL_EN;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> >> +{
> >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> >> +	struct perf_event *event = handle->event;
> >> +	struct hisi_ptt_dma_buflet *cur;
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +
> >> +	buf = perf_get_aux(handle);
> >> +	if (!buf || !handle->size)
> >> +		return -EINVAL;
> >> +
> >> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> >> +		if (cur->index == index)
> >> +			break;
> >> +
> >> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> >> +	memset(cur->addr, 0, cur->size);
> >> +	buf->pos += cur->size;
> >> +
> >> +	if (stop) {
> >> +		perf_aux_output_end(handle, buf->pos);
> >> +	} else if (buf->length - buf->pos < cur->size) {
> >> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> >> +		perf_aux_output_end(handle, buf->pos);
> >> +
> >> +		buf = perf_aux_output_begin(handle, event);
> >> +		if (!buf)
> >> +			return -EINVAL;
> >> +
> >> +		buf->pos = handle->head % buf->length;
> >> +		if (buf->length - buf->pos < cur->size) {
> >> +			perf_aux_output_end(handle, 0);
> >> +			return -EINVAL;
> >> +		}
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = context;
> >> +	u32 status, buf_idx;
> >> +
> >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	buf_idx = ffs(status) - 1;
> >> +
> >> +	/* Clear the interrupt status of buflet @buf_idx */
> >> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +
> >> +	/*
> >> +	 * Update the AUX buffer and cache the current buflet index,
> >> +	 * as we need to know this and save the data when the trace
> >> +	 * is ended out of the interrupt handler. End the trace
> >> +	 * if the updating fails.
> >> +	 */
> >> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +	else
> >> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> >> +
> >> +	return IRQ_HANDLED;
> >> +}
> >> +
> >> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = context;
> >> +	u32 status;
> >> +
> >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> >> +		return IRQ_NONE;
> >> +
> >> +	return IRQ_WAKE_THREAD;
> >> +}
> >> +
> >> +static void hisi_ptt_irq_free_vectors(void *pdev)
> >> +{
> >> +	pci_free_irq_vectors(pdev);
> >> +}
> >> +
> >> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	int ret;
> >> +
> >> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> >> +	if (ret < 0) {
> >> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	ret = devm_request_threaded_irq(&pdev->dev,
> >> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> >> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> >> +					"hisi-ptt", hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> >> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_update_filters(struct work_struct *work)
> >> +{
> >> +	struct delayed_work *delayed_work = to_delayed_work(work);
> >> +	struct hisi_ptt_filter_update_info info;
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	struct list_head *target_list;
> >> +	struct hisi_ptt *hisi_ptt;
> >> +
> >> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> >> +
> >> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> >> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> >> +		return;
> >> +	}
> >> +
> >> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> >> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> >> +			      &hisi_ptt->req_filters;
> >> +
> >> +		if (info.is_add) {
> >> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> >> +			if (!filter)
> >> +				continue;
> >> +
> >> +			filter->pdev = info.pdev;
> >> +			filter->val = info.val;
> >> +
> >> +			list_add_tail(&filter->list, target_list);
> >> +		} else {
> >> +			list_for_each_entry(filter, target_list, list)
> >> +				if (filter->val == info.val) {
> >> +					list_del(&filter->list);
> >> +					kfree(filter);
> >> +					break;
> >> +				}
> >> +		}
> >> +
> >> +		/* Update the available port mask */
> >> +		if (!info.is_port)
> >> +			continue;
> >> +
> >> +		if (info.is_add)
> >> +			hisi_ptt->port_mask |= info.val;
> >> +		else
> >> +			hisi_ptt->port_mask &= ~info.val;
> >> +	}
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +}
> >> +
> >> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> >> +				    struct hisi_ptt_filter_update_info *info)
> >> +{
> >> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> >> +
> >> +	if (!root_port)
> >> +		return;
> >> +
> >> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> >> +	if (info->port_devid < hisi_ptt->lower ||
> >> +	    info->port_devid > hisi_ptt->upper)
> >> +		return;
> >> +
> >> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> >> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> >> +
> >> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> >> +				&hisi_ptt->filter_update_lock))
> >> +		schedule_delayed_work(&hisi_ptt->work, 0);
> >> +	else
> >> +		pci_warn(hisi_ptt->pdev,
> >> +			 "filter update fifo overflow for target %s\n",
> >> +			 pci_name(info->pdev));
> >> +}
> >> +
> >> +/*
> >> + * A PCI bus notifier is used here for dynamically updating the filter
> >> + * list.
> >> + */
> >> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> >> +				  void *data)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> >> +	struct hisi_ptt_filter_update_info info;
> >> +	struct device *dev = data;
> >> +	struct pci_dev *pdev = to_pci_dev(dev);
> >> +
> >> +	info.pdev = pdev;
> >> +
> >> +	switch (action) {
> >> +	case BUS_NOTIFY_ADD_DEVICE:
> >> +		info.is_add = true;
> >> +		break;
> >> +	case BUS_NOTIFY_DEL_DEVICE:
> >> +		info.is_add = false;
> >> +		break;
> >> +	default:
> >> +		return 0;
> >> +	}
> >> +
> >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> >> +{
> >> +	struct hisi_ptt_filter_update_info info = {
> >> +		.pdev = pdev,
> >> +		.is_add = true,
> >> +	};
> >> +	struct hisi_ptt *hisi_ptt = data;
> >> +
> >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> >> +
> >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> >> +		list_del(&filter->list);
> >> +		kfree(filter);
> >> +	}
> >> +
> >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> >> +		list_del(&filter->list);
> >> +		kfree(filter);
> >> +	}
> >> +}
> >> +
> >> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	struct pci_bus *bus;
> >> +	u32 reg;
> >> +
> >> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> >> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> >> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> >> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> >> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> >> +
> >> +	/*
> >> +	 * The device range register provides the information about the
> >> +	 * root ports which the RCiEP can control and trace. The RCiEP
> >> +	 * and the root ports it support are on the same PCIe core, with
> >> +	 * same domain number but maybe different bus number. The device
> >> +	 * range register will tell us which root ports we can support,
> >> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> >> +	 * while Bit[15:0] indicates the lower.
> >> +	 */
> >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> >> +	hisi_ptt->upper = reg >> 16;
> >> +	hisi_ptt->lower = reg & 0xffff;
> >> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> >> +
> >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> >> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> >> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> >> +
> >> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> >> +	if (bus)
> >> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> > 
> > What happens if bus is NULL?  Should this be reported to the caller and cause
> > hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> > 
> 
> It's ok if the list is NULL in the probe process,
> as the device maybe hotplugged after the PTT driver probe, in which case
> we can detect the event and update the list as we register a bus notifier.
> 

Please add the above as a comment in the code.


> >> +
> >> +	/* Initialize trace controls */
> >> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> >> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> >> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> >> +}
> >> +
> >> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> >> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> >> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> >> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> >> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> >> +
> >> +static ssize_t available_filters_show(struct device *dev,
> >> +				      struct device_attribute *attr,
> >> +				      char *buf)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	int pos = 0;
> >> +
> >> +	if (list_empty(&hisi_ptt->port_filters))
> >> +		return sysfs_emit(buf, "#### No available filter ####\n");
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> >> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> >> +				     pci_name(filter->pdev),
> >> +				     hisi_ptt_get_filter_val(filter->pdev) |
> >> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> >> +
> >> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> >> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> >> +				     pci_name(filter->pdev),
> >> +				     hisi_ptt_get_filter_val(filter->pdev));
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return pos;
> >> +}
> >> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> >> +
> >> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> >> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> >> +PMU_FORMAT_ATTR(type,		"config:24-31");
> >> +PMU_FORMAT_ATTR(format,		"config:32-35");
> >> +
> >> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> >> +	&format_attr_filter.attr,
> >> +	&format_attr_direction.attr,
> >> +	&format_attr_type.attr,
> >> +	&format_attr_format.attr,
> >> +	NULL
> >> +};
> >> +
> >> +static struct attribute_group hisi_ptt_pmu_format_group = {
> >> +	.name = "format",
> >> +	.attrs = hisi_ptt_pmu_format_attrs,
> >> +};
> >> +
> >> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> >> +	&dev_attr_available_filters.attr,
> >> +	NULL
> >> +};
> >> +
> >> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> >> +	.attrs = hisi_ptt_pmu_filter_attrs,
> >> +};
> >> +
> >> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> >> +	&hisi_ptt_pmu_format_group,
> >> +	&hisi_ptt_pmu_filter_group,
> >> +	NULL
> >> +};
> >> +
> >> +/*
> >> + * The supported value of the direction parameter. See hisi_ptt.rst
> >> + * documentation for more details.
> >> + */
> >> +static u32 hisi_ptt_trace_available_direction[] = {
> >> +	0,
> >> +	1,
> >> +	2,
> >> +	3,
> >> +};
> >> +
> >> +/* Different types can be set simultaneously */
> >> +static u32 hisi_ptt_trace_available_type[] = {
> >> +	1,	/* posted_request */
> >> +	2,	/* non-posted_request */
> >> +	4,	/* completion */
> >> +};
> >> +
> >> +static u32 hisi_ptt_trace_availble_format[] = {
> >> +	0,	/* 4DW */
> >> +	1,	/* 8DW */
> >> +};
> >> +
> >> +/*
> >> + * Check whether the config is valid or not. Some configs are multi-selectable
> >> + * and can be set simultaneously, while some are single selectable (onehot).
> >> + * Use this function to check the non-onehot configs while
> >> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> >> + */
> >> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> >> +{
> >> +	int i;
> >> +
> >> +	/*
> >> +	 * The non-onehot configs cannot be 0. Walk the available
> >> +	 * list and clear the valid bits of the config. If there
> >> +	 * is any resident bit after the walk then the config is
> >> +	 * invalid.
> >> +	 */
> >> +	for (i = 0; i < list_size; i++)
> >> +		val &= ~available_list[i];
> >> +
> >> +	return val ? -EINVAL : 0;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> >> +{
> >> +	int i, ret = -EINVAL;
> >> +
> >> +	for (i = 0; i < list_size; i++)
> >> +		if (val == available_list[i]) {
> >> +			ret = 0;
> >> +			break;
> >> +		}
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> >> +{
> >> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	int ret = -EINVAL;
> >> +
> >> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> >> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> >> +
> >> +	/*
> >> +	 * Port filters are defined as bit mask. For port filters, check
> >> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> >> +	 * and whether it's empty or not, otherwise user has specified
> >> +	 * some unsupported root ports.
> >> +	 *
> >> +	 * For Requester ID filters, walk the available filter list to see
> >> +	 * whether we have one matched.
> >> +	 */
> >> +	if (hisi_ptt->trace_ctrl.is_port &&
> >> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> >> +		ret = 0;
> >> +	} else {
> >> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> >> +			if (filter->val == val) {
> >> +				ret = 0;
> >> +				break;
> >> +			}
> >> +	}
> >> +
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	hisi_ptt->trace_ctrl.filter = val;
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	int ret;
> >> +	u32 val;
> >> +
> >> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> >> +		return -ENOENT;
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +
> >> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> >> +	if (ret < 0)
> >> +		goto out;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> >> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->direction = val;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> >> +
> >> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> >> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->type = val;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> >> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->format = val;
> >> +
> >> +out:
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return ret;
> >> +}
> >> +
> >> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> >> +				    int nr_pages, bool overwrite)
> >> +{
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +	struct page **pagelist;
> >> +	int i;
> >> +
> >> +	if (overwrite) {
> >> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> >> +		return NULL;
> >> +	}
> >> +
> >> +	/* If the pages size less than buflets, we cannot start trace */
> >> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> >> +		return NULL;
> >> +
> >> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> >> +	if (!buf)
> >> +		return NULL;
> >> +
> >> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> >> +	if (!pagelist) {
> >> +		kfree(buf);
> >> +		return NULL;
> >> +	}
> >> +
> >> +	for (i = 0; i < nr_pages; i++)
> >> +		pagelist[i] = virt_to_page(pages[i]);
> >> +
> >> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> >> +	if (!buf->base) {
> >> +		kfree(pagelist);
> >> +		kfree(buf);
> >> +		return NULL;
> >> +	}
> >> +
> >> +	buf->nr_pages = nr_pages;
> >> +	buf->length = nr_pages * PAGE_SIZE;
> >> +	buf->pos = 0;
> >> +
> >> +	kfree(pagelist);
> >> +	return buf;
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_free_aux(void *aux)
> >> +{
> >> +	struct hisi_ptt_pmu_buf *buf = aux;
> >> +
> >> +	vunmap(buf->base);
> >> +	kfree(buf);
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +	int cpu = event->cpu;
> >> +	int ret;
> >> +
> >> +	hwc->state = 0;
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> >> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> >> +		goto stop;
> >> +	}
> >> +
> >> +	if (cpu == -1)
> >> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> >> +
> >> +	/*
> >> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> >> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> >> +	 * core in event_function_local().
> >> +	 */
> >> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> >> +				 cpumask_of(cpu)));
> >> +
> >> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> >> +		goto stop;
> >> +	}
> >> +
> >> +	buf = perf_aux_output_begin(handle, event);
> >> +	if (!buf) {
> >> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> >> +		goto stop;
> >> +	}
> >> +
> >> +	buf->pos = handle->head % buf->length;
> >> +
> >> +	ret = hisi_ptt_trace_start(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> >> +		perf_aux_output_end(handle, 0);
> >> +		goto stop;
> >> +	}
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return;
> >> +stop:
> >> +	event->hw.state |= PERF_HES_STOPPED;
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +
> >> +	if (hwc->state & PERF_HES_STOPPED)
> >> +		return;
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> >> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> >> +	}
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +
> >> +	hwc->state |= PERF_HES_STOPPED;
> >> +	perf_event_update_userpage(event);
> >> +	hwc->state |= PERF_HES_UPTODATE;
> >> +}
> >> +
> >> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +	int cpu = event->cpu;
> >> +
> >> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> >> +		return 0;
> >> +
> >> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> >> +
> >> +	if (flags & PERF_EF_START) {
> >> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> >> +		if (hwc->state & PERF_HES_STOPPED)
> >> +			return -EINVAL;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> >> +{
> >> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> >> +}
> >> +
> >> +static void hisi_ptt_unregister_pmu(void *priv)
> >> +{
> >> +	perf_pmu_unregister(priv);
> >> +}
> >> +
> >> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	char *pmu_name;
> >> +	int ret;
> >> +
> >> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> >> +		.module		= THIS_MODULE,
> >> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> >> +		.task_ctx_nr	= perf_sw_context,
> >> +		.attr_groups	= hisi_ptt_pmu_groups,
> >> +		.event_init	= hisi_ptt_pmu_event_init,
> >> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> >> +		.free_aux	= hisi_ptt_pmu_free_aux,
> >> +		.start		= hisi_ptt_pmu_start,
> >> +		.stop		= hisi_ptt_pmu_stop,
> >> +		.add		= hisi_ptt_pmu_add,
> >> +		.del		= hisi_ptt_pmu_del,
> >> +	};
> >> +
> >> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> >> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> >> +	if (!pmu_name)
> >> +		return -ENOMEM;
> >> +
> >> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> >> +					hisi_ptt_unregister_pmu,
> >> +					&hisi_ptt->hisi_ptt_pmu);
> >> +}
> >> +
> >> +/*
> >> + * Get RMR address if provided by the firmware.
> >> + * Return 0 if the IOMMU doesn't present or the policy of the
> >> + * IOMMU domain is passthrough or we get a usable RMR region.
> >> + * Otherwise a negative value is returned.
> >> + */
> >> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	struct iommu_domain *iommu_domain;
> >> +	struct iommu_resv_region *region;
> >> +	LIST_HEAD(list);
> >> +
> >> +	/*
> >> +	 * Use direct DMA if IOMMU does not present or the policy of the
> >> +	 * IOMMU domain is passthrough.
> >> +	 */
> >> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> >> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> >> +		return 0;
> >> +
> >> +	iommu_get_resv_regions(&pdev->dev, &list);
> >> +	list_for_each_entry(region, &list, list)
> >> +		if (region->type == IOMMU_RESV_DIRECT &&
> >> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> >> +			hisi_ptt->trace_ctrl.has_rmr = true;
> >> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> >> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> >> +			break;
> >> +		}
> >> +
> >> +	iommu_put_resv_regions(&pdev->dev, &list);
> >> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> >> +}
> >> +
> >> +static int hisi_ptt_probe(struct pci_dev *pdev,
> >> +			  const struct pci_device_id *id)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt;
> >> +	int ret;
> >> +
> >> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> >> +	if (!hisi_ptt)
> >> +		return -ENOMEM;
> >> +
> >> +	mutex_init(&hisi_ptt->mutex);
> >> +	hisi_ptt->pdev = pdev;
> >> +
> >> +	/*
> >> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> >> +	 * so directly reference to the pci name string.
> >> +	 */
> >> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> >> +	pci_set_drvdata(pdev, hisi_ptt);
> >> +
> >> +	ret = pcim_enable_device(pdev);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> >> +
> >> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +	pci_set_master(pdev);
> >> +
> >> +	ret = hisi_ptt_register_irq(hisi_ptt);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	hisi_ptt_init_ctrls(hisi_ptt);
> >> +
> >> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> >> +		return ret;
> >> +	}
> > 
> > If this fails the pci_free_irq_vectors() is not called.
> > 
> 
> It'll be called as we've registered a devm callback when
> allocating the irq vector. See hisi_ptt_register_irq().

Ah yes - thanks for pointing that out.

> 
> > I am out of time for today - I will continue tomorrow.
> > 
> 
> Sure. Thanks!
> Yicong
> 
> > Thanks,
> > Mathieu
> > 
> >> +
> >> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	/* Register the bus notifier for dynamically updating the filter list */
> >> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> >> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> >> +	if (ret)
> >> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +void hisi_ptt_remove(struct pci_dev *pdev)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> >> +
> >> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> >> +
> >> +	/* Cancel any work that has been queued */
> >> +	cancel_delayed_work_sync(&hisi_ptt->work);
> >> +
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +
> >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> >> +	hisi_ptt_release_filters(hisi_ptt);
> >> +}
> >> +
> >> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> >> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> >> +	{ }
> >> +};
> >> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> >> +
> >> +static struct pci_driver hisi_ptt_driver = {
> >> +	.name = "hisi_ptt",
> >> +	.id_table = hisi_ptt_id_tbl,
> >> +	.probe = hisi_ptt_probe,
> >> +	.remove = hisi_ptt_remove,
> >> +};
> >> +module_pci_driver(hisi_ptt_driver);
> >> +
> >> +MODULE_LICENSE("GPL v2");
> >> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> >> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> >> -- 
> >> 2.33.0
> >>
> > .
> > 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

On Thu, Nov 25, 2021 at 04:39:46PM +0800, Yicong Yang wrote:
> Hi Mathieu,
> 
> Thanks for the comments! Replies inline.
> 
> On 2021/11/25 2:51, Mathieu Poirier wrote:
> > Hi Yicong,
> > 
> > On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
> >> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> >> integrated Endpoint(RCiEP) device, providing the capability
> >> to dynamically monitor and tune the PCIe traffic(tune),
> >> and trace the TLP headers(trace).
> >>
> > 
> > Is there a reason to put "tune" and "trace" are whithin parentheses?  
> > 
> 
> yeah. I want to use these single word to denote the related function, and
> place them in the parantheses near the function's description to make sure
> readers can match them.
> 

Probably a better idea to simply write "TLP headers" and "PCIe traffic" when
referring to them.  Otherwise it is very confusing because the reader doesn't
know if you mean the acronym or the real definition of "tune" and "trace".

> >> Add the driver for the device to enable the trace function. The driver
> >> will create PMU device for each PTT device, and users can start trace
> >> through perf command.
> >>
> >> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> >> ---
> >>  drivers/Makefile                       |    1 +
> >>  drivers/hwtracing/Kconfig              |    2 +
> >>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
> >>  drivers/hwtracing/hisilicon/Makefile   |    2 +
> >>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
> > 
> > Not sure about the "hisilicon" directory.  Right now we have "coresight",
> > "intel_th" and "stm" - they concentrate on the technology rather than the
> > vendor.  I think "ptt" would be just fine: 
> > 
> > drivers/hwtracing/ptt/Kconfig 
> > drivers/hwtracing/ptt/Makefile
> > drivers/hwtracing/ptt/hisi_ptt.c
> > 
> > That way if other vendors want to introduce support for the same kind of
> > technology we can simply put them under "drivers/hwtracing/ptt/" and things are
> > still accurate.
> > 
> 
> sure. sounds sensible. will rename the directory to "ptt" if no objection.
> 
> > 
> >>  5 files changed, 1159 insertions(+)
> >>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
> >>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
> >>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
> >>
> >> diff --git a/drivers/Makefile b/drivers/Makefile
> >> index be5d40ae1488..bb0dc9b55ea2 100644
> >> --- a/drivers/Makefile
> >> +++ b/drivers/Makefile
> >> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
> >>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
> >>  obj-y				+= hwtracing/intel_th/
> >>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> >> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
> >>  obj-$(CONFIG_ANDROID)		+= android/
> >>  obj-$(CONFIG_NVMEM)		+= nvmem/
> >>  obj-$(CONFIG_FPGA)		+= fpga/
> >> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> >> index 13085835a636..e3796b17541a 100644
> >> --- a/drivers/hwtracing/Kconfig
> >> +++ b/drivers/hwtracing/Kconfig
> >> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
> >>  
> >>  source "drivers/hwtracing/intel_th/Kconfig"
> >>  
> >> +source "drivers/hwtracing/hisilicon/Kconfig"
> >> +
> >>  endmenu
> >> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
> >> new file mode 100644
> >> index 000000000000..9c3ab80d99fe
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/Kconfig
> >> @@ -0,0 +1,8 @@
> >> +# SPDX-License-Identifier: GPL-2.0-only
> >> +config HISI_PTT
> >> +	tristate "HiSilicon PCIe Tune and Trace Device"
> >> +	depends on PCI && HAS_DMA && HAS_IOMEM
> > 
> > What Arm architecture are you aiming this for?  Can it be found on both armv7
> > and armv8?
> > 
> 
> currently the device will appear on Kunpeng 930, which is armv8.

Ok, then please make it dependent on armv8.

> 
> >> +	help
> >> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
> >> +	  device, provides support for PCIe traffic tuning and
> >> +	  tracing TLP headers to the memory.
> > 
> > Please indicate what the name of the driver will be when compiled as a module.
> > 
> 
> sure. will add the module's name here.
> 
> >> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
> >> new file mode 100644
> >> index 000000000000..908c09a98161
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/Makefile
> >> @@ -0,0 +1,2 @@
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> >> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
> >> new file mode 100644
> >> index 000000000000..e11e9b6cc2a8
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
> >> @@ -0,0 +1,1146 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * Driver for HiSilicon PCIe tune and trace device
> >> + *
> >> + * Copyright (c) 2021 HiSilicon Limited.
> >> + */
> >> +
> >> +#include <linux/bitfield.h>
> >> +#include <linux/bitops.h>
> >> +#include <linux/delay.h>
> >> +#include <linux/dma-iommu.h>
> >> +#include <linux/dma-mapping.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/io.h>
> >> +#include <linux/iommu.h>
> >> +#include <linux/iopoll.h>
> >> +#include <linux/kfifo.h>
> >> +#include <linux/module.h>
> >> +#include <linux/notifier.h>
> >> +#include <linux/pci.h>
> >> +#include <linux/perf_event.h>
> >> +#include <linux/sizes.h>
> >> +#include <linux/sysfs.h>
> >> +#include <linux/vmalloc.h>
> >> +#include <linux/workqueue.h>
> >> +
> >> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> >> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> >> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> >> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> >> +#define HISI_PTT_TRACE_CTRL		0x0850
> >> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> >> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> >> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> >> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> >> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> >> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> >> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> >> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> >> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> >> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> >> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> >> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> >> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> >> +#define HISI_PTT_TRACE_STS		0x08b0
> >> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> >> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> >> +#define HISI_PTT_LOCATION		0x0fe8
> >> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> >> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> > 
> > Extra spaces between #define and the literals.  Please align the values with
> > that of HISI_PTT_TRACE_DMA_IRQ below.
> > 
> 
> Spaces are intended to distinguish the definition of the register offset and
> related register fields. also used in include/uapi/linux/pci_regs.h.
>

Interesting and definitely not the norm...

> HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
> in the mail text, maybe because the tab is not 8 characters when display.
> 

In my VI terminal the '0' of HISI_PTT_TRACE_DMA_IRQ is align with the '3' in
GENMASK(31, 16) and I suspect it is the same for other people.

> >> +
> >> +#define HISI_PTT_TRACE_DMA_IRQ			0
> >> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> >> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> >> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> >> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> >> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> >> +
> >> +/* Delay time for filter updating work */
> >> +#define HISI_PTT_WORK_DELAY_MS		100UL
> >> +/* Wait time for DMA hardware to reset */
> >> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> >> +/* Poll timeout and interval for waiting hardware work to finish */
> >> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> >> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> >> +
> >> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> >> +
> >> +enum hisi_ptt_trace_status {
> >> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> >> +	HISI_PTT_TRACE_STATUS_ON,
> >> +};
> >> +
> >> +struct hisi_ptt_dma_buflet {
> > 
> > I'm not sure what a "buflet" is...  Probably best to just call this
> > hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> > know due to the lack of proper structure documentation.  Please have a look at
> > how "coresight_device" is documented.  The same applies to the rest of the
> > structures declared below.
> > 
> 
> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
> (we call it buflet) of 4MiB ...".

I just noticed the documentation in patch 5 - I will read it before continuing
with this set.

> 
> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
> a 'buflet', which means a small buffer (there exists other words with similiar
> format: droplet, wavelet, ...).
> 
> The device is designed like this to make sure we won't lose any data when we have to change
> the buffer address, the device can continue to write to the next buflet and don't
> neet to pause.
> 
> >> +	struct list_head list;
> >> +	unsigned int size;
> >> +	dma_addr_t dma;
> >> +
> >> +	/*
> >> +	 * The address of the buflet holding the trace data.
> >> +	 * See Documentation/trace/hisi-ptt.rst for the details
> > 
> > As of this writing, hisi-ptt.rst doesn't exist.
> > 
> 
> The comment intends to direct the user to the documentation if the user
> want to know details about the data format. The data format is decribed
> in the section 4 of the doc.
> 
> >> +	 * of the data format.
> >> +	 */
> >> +	void *addr;
> >> +	int index;
> >> +};
> >> +
> >> +struct hisi_ptt_trace_ctrl {
> >> +	enum hisi_ptt_trace_status status;
> >> +	struct perf_output_handle handle;
> >> +	struct list_head trace_buf;
> >> +	/*
> >> +	 * The index of the buffer which trace data
> >> +	 * currently is writing to.
> >> +	 */
> >> +	u32 buf_index;
> >> +
> >> +	int default_cpu;
> >> +	u32 buflet_size;
> >> +	bool is_port;
> >> +	u32 format:1;		/* Format of the traced TLP headers */
> >> +	u32 type:4;		/* Type of the TLP headers to trace */
> >> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> >> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> >> +
> >> +	phys_addr_t rmr_addr;
> >> +	size_t rmr_length;
> >> +	bool has_rmr;
> >> +};
> >> +
> >> +struct hisi_ptt_filter_desc {
> >> +	struct list_head list;
> >> +	struct pci_dev *pdev;
> >> +	u16 val;
> >> +};
> >> +
> >> +struct hisi_ptt_pmu_buf {
> >> +	size_t length;
> >> +	int nr_pages;
> >> +	void *base;
> >> +	long pos;
> >> +};
> >> +
> >> +/* Structure containing the information for filter updating */
> >> +struct hisi_ptt_filter_update_info {
> >> +	struct pci_dev *pdev;
> >> +	u32 port_devid;
> >> +	bool is_port;
> >> +	bool is_add;
> >> +	u16 val;
> >> +};
> >> +
> >> +struct hisi_ptt {
> >> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> >> +	struct notifier_block hisi_ptt_nb;
> >> +	struct pmu hisi_ptt_pmu;
> >> +	void __iomem *iobase;
> >> +	struct pci_dev *pdev;
> >> +	/*
> >> +	 * Use the mutex to protect the filter list and
> >> +	 * serialize the perf process.
> >> +	 */
> >> +	struct mutex mutex;
> >> +	const char *name;
> >> +	u16 core_id;
> >> +	u16 sicl_id;
> >> +	/* PCI device range managed by the PTT device */
> >> +	u32 upper;
> >> +	u32 lower;
> >> +	u8 busnr;
> >> +
> >> +	/*
> >> +	 * The trace TLP headers can either be filtered by certain
> >> +	 * root port, or by the requester ID. Organize the filters
> >> +	 * by @port_filters and @req_filters here. The mask of all
> >> +	 * the valid ports is also cached for doing sanity check
> >> +	 * of user input.
> >> +	 */
> >> +	struct list_head port_filters;
> >> +	struct list_head req_filters;
> >> +	u16 port_mask;
> >> +
> >> +	/*
> >> +	 * We use a delayed work here to avoid indefinitely waiting for
> >> +	 * the hisi_ptt->mutex which protecting the filter list. The
> >> +	 * work will be delayed only if the mutex can not be held,
> >> +	 * otherwise no delay will be applied.
> >> +	 */
> >> +	struct delayed_work work;
> >> +	spinlock_t filter_update_lock;
> >> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> >> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> >> +};
> > 
> > It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> > 
> 
> I'm not sure whether it is common or permitted to provide a header which is only used by
> this driver with a single file.

I've seen it done many times.

> 
> >> +
> >> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> >> +{
> >> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> >> +}
> > 
> > I'm not sure there is any value to this inline function.  I suggest to simply
> > call container_of() whenever it is needed.
> > 
> 
> ok.
> 
> >> +
> >> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> >> +{
> >> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> >> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> >> +
> >> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> >> +}
> >> +
> >> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	u32 val;
> >> +
> >> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> >> +				  val & HISI_PTT_TRACE_IDLE,
> >> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> >> +				  HISI_PTT_WAIT_TIMEOUT_US);
> >> +}
> >> +
> >> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct device *dev = &hisi_ptt->pdev->dev;
> >> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> >> +
> >> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> >> +		list_del(&buflet->list);
> >> +
> >> +		if (ctrl->has_rmr)
> >> +			memunmap(buflet->addr);
> >> +		else
> >> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> >> +					  buflet->dma);
> >> +
> >> +		kfree(buflet);
> >> +	}
> >> +}
> >> +
> >> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct device *dev = &hisi_ptt->pdev->dev;
> >> +	struct hisi_ptt_dma_buflet *buflet;
> >> +	int i, ret;
> >> +
> >> +	hisi_ptt->trace_ctrl.buf_index = 0;
> >> +
> >> +	/* Make sure the trace buffer is empty before allocating */
> >> +	if (!list_empty(&ctrl->trace_buf)) {
> >> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> >> +			memset(buflet->addr, 0, buflet->size);
> >> +		return 0;
> >> +	}
> >> +
> >> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> >> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> >> +		if (!buflet) {
> >> +			ret = -ENOMEM;
> >> +			goto err;
> >> +		}
> >> +
> >> +		if (ctrl->has_rmr) {
> >> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> >> +
> >> +			buflet->dma = base;
> >> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> >> +		} else {
> >> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> >> +							  &buflet->dma, GFP_KERNEL);
> >> +		}
> >> +
> >> +		if (!buflet->addr) {
> >> +			kfree(buflet);
> >> +			ret = -ENOMEM;
> >> +			goto err;
> >> +		}
> >> +
> >> +		memset(buflet->addr, 0, buflet->size);
> >> +
> >> +		buflet->index = i;
> >> +		buflet->size = ctrl->buflet_size;
> >> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> >> +	}
> >> +
> >> +	return 0;
> >> +err:
> >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> >> +	return ret;
> >> +}
> >> +
> >> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct hisi_ptt_dma_buflet *cur;
> >> +	u32 val;
> >> +
> >> +	/* Check device idle before start trace */
> >> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> >> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> >> +		return -EBUSY;
> >> +	}
> >> +
> >> +	/* Reset the DMA before start tracing */
> >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	val |= HISI_PTT_TRACE_CTRL_RST;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	/*
> >> +	 * We'll be in the perf context where preemption is disabled,
> >> +	 * so use busy loop here.
> >> +	 */
> >> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> >> +
> >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	/* clear the interrupt status */
> >> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> >> +
> >> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> >> +		writel(lower_32_bits(cur->dma),
> >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> >> +		writel(upper_32_bits(cur->dma),
> >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> >> +	}
> >> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> >> +
> >> +	/* set the trace control register */
> >> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> >> +	if (!hisi_ptt->trace_ctrl.is_port)
> >> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> >> +
> >> +	val |= HISI_PTT_TRACE_CTRL_EN;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> >> +{
> >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> >> +	struct perf_event *event = handle->event;
> >> +	struct hisi_ptt_dma_buflet *cur;
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +
> >> +	buf = perf_get_aux(handle);
> >> +	if (!buf || !handle->size)
> >> +		return -EINVAL;
> >> +
> >> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> >> +		if (cur->index == index)
> >> +			break;
> >> +
> >> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> >> +	memset(cur->addr, 0, cur->size);
> >> +	buf->pos += cur->size;
> >> +
> >> +	if (stop) {
> >> +		perf_aux_output_end(handle, buf->pos);
> >> +	} else if (buf->length - buf->pos < cur->size) {
> >> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> >> +		perf_aux_output_end(handle, buf->pos);
> >> +
> >> +		buf = perf_aux_output_begin(handle, event);
> >> +		if (!buf)
> >> +			return -EINVAL;
> >> +
> >> +		buf->pos = handle->head % buf->length;
> >> +		if (buf->length - buf->pos < cur->size) {
> >> +			perf_aux_output_end(handle, 0);
> >> +			return -EINVAL;
> >> +		}
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = context;
> >> +	u32 status, buf_idx;
> >> +
> >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	buf_idx = ffs(status) - 1;
> >> +
> >> +	/* Clear the interrupt status of buflet @buf_idx */
> >> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +
> >> +	/*
> >> +	 * Update the AUX buffer and cache the current buflet index,
> >> +	 * as we need to know this and save the data when the trace
> >> +	 * is ended out of the interrupt handler. End the trace
> >> +	 * if the updating fails.
> >> +	 */
> >> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +	else
> >> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> >> +
> >> +	return IRQ_HANDLED;
> >> +}
> >> +
> >> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = context;
> >> +	u32 status;
> >> +
> >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> >> +		return IRQ_NONE;
> >> +
> >> +	return IRQ_WAKE_THREAD;
> >> +}
> >> +
> >> +static void hisi_ptt_irq_free_vectors(void *pdev)
> >> +{
> >> +	pci_free_irq_vectors(pdev);
> >> +}
> >> +
> >> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	int ret;
> >> +
> >> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> >> +	if (ret < 0) {
> >> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	ret = devm_request_threaded_irq(&pdev->dev,
> >> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> >> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> >> +					"hisi-ptt", hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> >> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_update_filters(struct work_struct *work)
> >> +{
> >> +	struct delayed_work *delayed_work = to_delayed_work(work);
> >> +	struct hisi_ptt_filter_update_info info;
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	struct list_head *target_list;
> >> +	struct hisi_ptt *hisi_ptt;
> >> +
> >> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> >> +
> >> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> >> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> >> +		return;
> >> +	}
> >> +
> >> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> >> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> >> +			      &hisi_ptt->req_filters;
> >> +
> >> +		if (info.is_add) {
> >> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> >> +			if (!filter)
> >> +				continue;
> >> +
> >> +			filter->pdev = info.pdev;
> >> +			filter->val = info.val;
> >> +
> >> +			list_add_tail(&filter->list, target_list);
> >> +		} else {
> >> +			list_for_each_entry(filter, target_list, list)
> >> +				if (filter->val == info.val) {
> >> +					list_del(&filter->list);
> >> +					kfree(filter);
> >> +					break;
> >> +				}
> >> +		}
> >> +
> >> +		/* Update the available port mask */
> >> +		if (!info.is_port)
> >> +			continue;
> >> +
> >> +		if (info.is_add)
> >> +			hisi_ptt->port_mask |= info.val;
> >> +		else
> >> +			hisi_ptt->port_mask &= ~info.val;
> >> +	}
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +}
> >> +
> >> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> >> +				    struct hisi_ptt_filter_update_info *info)
> >> +{
> >> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> >> +
> >> +	if (!root_port)
> >> +		return;
> >> +
> >> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> >> +	if (info->port_devid < hisi_ptt->lower ||
> >> +	    info->port_devid > hisi_ptt->upper)
> >> +		return;
> >> +
> >> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> >> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> >> +
> >> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> >> +				&hisi_ptt->filter_update_lock))
> >> +		schedule_delayed_work(&hisi_ptt->work, 0);
> >> +	else
> >> +		pci_warn(hisi_ptt->pdev,
> >> +			 "filter update fifo overflow for target %s\n",
> >> +			 pci_name(info->pdev));
> >> +}
> >> +
> >> +/*
> >> + * A PCI bus notifier is used here for dynamically updating the filter
> >> + * list.
> >> + */
> >> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> >> +				  void *data)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> >> +	struct hisi_ptt_filter_update_info info;
> >> +	struct device *dev = data;
> >> +	struct pci_dev *pdev = to_pci_dev(dev);
> >> +
> >> +	info.pdev = pdev;
> >> +
> >> +	switch (action) {
> >> +	case BUS_NOTIFY_ADD_DEVICE:
> >> +		info.is_add = true;
> >> +		break;
> >> +	case BUS_NOTIFY_DEL_DEVICE:
> >> +		info.is_add = false;
> >> +		break;
> >> +	default:
> >> +		return 0;
> >> +	}
> >> +
> >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> >> +{
> >> +	struct hisi_ptt_filter_update_info info = {
> >> +		.pdev = pdev,
> >> +		.is_add = true,
> >> +	};
> >> +	struct hisi_ptt *hisi_ptt = data;
> >> +
> >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> >> +
> >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> >> +		list_del(&filter->list);
> >> +		kfree(filter);
> >> +	}
> >> +
> >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> >> +		list_del(&filter->list);
> >> +		kfree(filter);
> >> +	}
> >> +}
> >> +
> >> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	struct pci_bus *bus;
> >> +	u32 reg;
> >> +
> >> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> >> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> >> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> >> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> >> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> >> +
> >> +	/*
> >> +	 * The device range register provides the information about the
> >> +	 * root ports which the RCiEP can control and trace. The RCiEP
> >> +	 * and the root ports it support are on the same PCIe core, with
> >> +	 * same domain number but maybe different bus number. The device
> >> +	 * range register will tell us which root ports we can support,
> >> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> >> +	 * while Bit[15:0] indicates the lower.
> >> +	 */
> >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> >> +	hisi_ptt->upper = reg >> 16;
> >> +	hisi_ptt->lower = reg & 0xffff;
> >> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> >> +
> >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> >> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> >> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> >> +
> >> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> >> +	if (bus)
> >> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> > 
> > What happens if bus is NULL?  Should this be reported to the caller and cause
> > hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> > 
> 
> It's ok if the list is NULL in the probe process,
> as the device maybe hotplugged after the PTT driver probe, in which case
> we can detect the event and update the list as we register a bus notifier.
> 

Please add the above as a comment in the code.


> >> +
> >> +	/* Initialize trace controls */
> >> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> >> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> >> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> >> +}
> >> +
> >> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> >> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> >> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> >> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> >> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> >> +
> >> +static ssize_t available_filters_show(struct device *dev,
> >> +				      struct device_attribute *attr,
> >> +				      char *buf)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	int pos = 0;
> >> +
> >> +	if (list_empty(&hisi_ptt->port_filters))
> >> +		return sysfs_emit(buf, "#### No available filter ####\n");
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> >> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> >> +				     pci_name(filter->pdev),
> >> +				     hisi_ptt_get_filter_val(filter->pdev) |
> >> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> >> +
> >> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> >> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> >> +				     pci_name(filter->pdev),
> >> +				     hisi_ptt_get_filter_val(filter->pdev));
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return pos;
> >> +}
> >> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> >> +
> >> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> >> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> >> +PMU_FORMAT_ATTR(type,		"config:24-31");
> >> +PMU_FORMAT_ATTR(format,		"config:32-35");
> >> +
> >> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> >> +	&format_attr_filter.attr,
> >> +	&format_attr_direction.attr,
> >> +	&format_attr_type.attr,
> >> +	&format_attr_format.attr,
> >> +	NULL
> >> +};
> >> +
> >> +static struct attribute_group hisi_ptt_pmu_format_group = {
> >> +	.name = "format",
> >> +	.attrs = hisi_ptt_pmu_format_attrs,
> >> +};
> >> +
> >> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> >> +	&dev_attr_available_filters.attr,
> >> +	NULL
> >> +};
> >> +
> >> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> >> +	.attrs = hisi_ptt_pmu_filter_attrs,
> >> +};
> >> +
> >> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> >> +	&hisi_ptt_pmu_format_group,
> >> +	&hisi_ptt_pmu_filter_group,
> >> +	NULL
> >> +};
> >> +
> >> +/*
> >> + * The supported value of the direction parameter. See hisi_ptt.rst
> >> + * documentation for more details.
> >> + */
> >> +static u32 hisi_ptt_trace_available_direction[] = {
> >> +	0,
> >> +	1,
> >> +	2,
> >> +	3,
> >> +};
> >> +
> >> +/* Different types can be set simultaneously */
> >> +static u32 hisi_ptt_trace_available_type[] = {
> >> +	1,	/* posted_request */
> >> +	2,	/* non-posted_request */
> >> +	4,	/* completion */
> >> +};
> >> +
> >> +static u32 hisi_ptt_trace_availble_format[] = {
> >> +	0,	/* 4DW */
> >> +	1,	/* 8DW */
> >> +};
> >> +
> >> +/*
> >> + * Check whether the config is valid or not. Some configs are multi-selectable
> >> + * and can be set simultaneously, while some are single selectable (onehot).
> >> + * Use this function to check the non-onehot configs while
> >> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> >> + */
> >> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> >> +{
> >> +	int i;
> >> +
> >> +	/*
> >> +	 * The non-onehot configs cannot be 0. Walk the available
> >> +	 * list and clear the valid bits of the config. If there
> >> +	 * is any resident bit after the walk then the config is
> >> +	 * invalid.
> >> +	 */
> >> +	for (i = 0; i < list_size; i++)
> >> +		val &= ~available_list[i];
> >> +
> >> +	return val ? -EINVAL : 0;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> >> +{
> >> +	int i, ret = -EINVAL;
> >> +
> >> +	for (i = 0; i < list_size; i++)
> >> +		if (val == available_list[i]) {
> >> +			ret = 0;
> >> +			break;
> >> +		}
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> >> +{
> >> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	int ret = -EINVAL;
> >> +
> >> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> >> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> >> +
> >> +	/*
> >> +	 * Port filters are defined as bit mask. For port filters, check
> >> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> >> +	 * and whether it's empty or not, otherwise user has specified
> >> +	 * some unsupported root ports.
> >> +	 *
> >> +	 * For Requester ID filters, walk the available filter list to see
> >> +	 * whether we have one matched.
> >> +	 */
> >> +	if (hisi_ptt->trace_ctrl.is_port &&
> >> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> >> +		ret = 0;
> >> +	} else {
> >> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> >> +			if (filter->val == val) {
> >> +				ret = 0;
> >> +				break;
> >> +			}
> >> +	}
> >> +
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	hisi_ptt->trace_ctrl.filter = val;
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	int ret;
> >> +	u32 val;
> >> +
> >> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> >> +		return -ENOENT;
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +
> >> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> >> +	if (ret < 0)
> >> +		goto out;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> >> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->direction = val;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> >> +
> >> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> >> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->type = val;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> >> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->format = val;
> >> +
> >> +out:
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return ret;
> >> +}
> >> +
> >> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> >> +				    int nr_pages, bool overwrite)
> >> +{
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +	struct page **pagelist;
> >> +	int i;
> >> +
> >> +	if (overwrite) {
> >> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> >> +		return NULL;
> >> +	}
> >> +
> >> +	/* If the pages size less than buflets, we cannot start trace */
> >> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> >> +		return NULL;
> >> +
> >> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> >> +	if (!buf)
> >> +		return NULL;
> >> +
> >> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> >> +	if (!pagelist) {
> >> +		kfree(buf);
> >> +		return NULL;
> >> +	}
> >> +
> >> +	for (i = 0; i < nr_pages; i++)
> >> +		pagelist[i] = virt_to_page(pages[i]);
> >> +
> >> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> >> +	if (!buf->base) {
> >> +		kfree(pagelist);
> >> +		kfree(buf);
> >> +		return NULL;
> >> +	}
> >> +
> >> +	buf->nr_pages = nr_pages;
> >> +	buf->length = nr_pages * PAGE_SIZE;
> >> +	buf->pos = 0;
> >> +
> >> +	kfree(pagelist);
> >> +	return buf;
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_free_aux(void *aux)
> >> +{
> >> +	struct hisi_ptt_pmu_buf *buf = aux;
> >> +
> >> +	vunmap(buf->base);
> >> +	kfree(buf);
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +	int cpu = event->cpu;
> >> +	int ret;
> >> +
> >> +	hwc->state = 0;
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> >> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> >> +		goto stop;
> >> +	}
> >> +
> >> +	if (cpu == -1)
> >> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> >> +
> >> +	/*
> >> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> >> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> >> +	 * core in event_function_local().
> >> +	 */
> >> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> >> +				 cpumask_of(cpu)));
> >> +
> >> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> >> +		goto stop;
> >> +	}
> >> +
> >> +	buf = perf_aux_output_begin(handle, event);
> >> +	if (!buf) {
> >> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> >> +		goto stop;
> >> +	}
> >> +
> >> +	buf->pos = handle->head % buf->length;
> >> +
> >> +	ret = hisi_ptt_trace_start(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> >> +		perf_aux_output_end(handle, 0);
> >> +		goto stop;
> >> +	}
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return;
> >> +stop:
> >> +	event->hw.state |= PERF_HES_STOPPED;
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +
> >> +	if (hwc->state & PERF_HES_STOPPED)
> >> +		return;
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> >> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> >> +	}
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +
> >> +	hwc->state |= PERF_HES_STOPPED;
> >> +	perf_event_update_userpage(event);
> >> +	hwc->state |= PERF_HES_UPTODATE;
> >> +}
> >> +
> >> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +	int cpu = event->cpu;
> >> +
> >> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> >> +		return 0;
> >> +
> >> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> >> +
> >> +	if (flags & PERF_EF_START) {
> >> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> >> +		if (hwc->state & PERF_HES_STOPPED)
> >> +			return -EINVAL;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> >> +{
> >> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> >> +}
> >> +
> >> +static void hisi_ptt_unregister_pmu(void *priv)
> >> +{
> >> +	perf_pmu_unregister(priv);
> >> +}
> >> +
> >> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	char *pmu_name;
> >> +	int ret;
> >> +
> >> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> >> +		.module		= THIS_MODULE,
> >> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> >> +		.task_ctx_nr	= perf_sw_context,
> >> +		.attr_groups	= hisi_ptt_pmu_groups,
> >> +		.event_init	= hisi_ptt_pmu_event_init,
> >> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> >> +		.free_aux	= hisi_ptt_pmu_free_aux,
> >> +		.start		= hisi_ptt_pmu_start,
> >> +		.stop		= hisi_ptt_pmu_stop,
> >> +		.add		= hisi_ptt_pmu_add,
> >> +		.del		= hisi_ptt_pmu_del,
> >> +	};
> >> +
> >> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> >> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> >> +	if (!pmu_name)
> >> +		return -ENOMEM;
> >> +
> >> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> >> +					hisi_ptt_unregister_pmu,
> >> +					&hisi_ptt->hisi_ptt_pmu);
> >> +}
> >> +
> >> +/*
> >> + * Get RMR address if provided by the firmware.
> >> + * Return 0 if the IOMMU doesn't present or the policy of the
> >> + * IOMMU domain is passthrough or we get a usable RMR region.
> >> + * Otherwise a negative value is returned.
> >> + */
> >> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	struct iommu_domain *iommu_domain;
> >> +	struct iommu_resv_region *region;
> >> +	LIST_HEAD(list);
> >> +
> >> +	/*
> >> +	 * Use direct DMA if IOMMU does not present or the policy of the
> >> +	 * IOMMU domain is passthrough.
> >> +	 */
> >> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> >> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> >> +		return 0;
> >> +
> >> +	iommu_get_resv_regions(&pdev->dev, &list);
> >> +	list_for_each_entry(region, &list, list)
> >> +		if (region->type == IOMMU_RESV_DIRECT &&
> >> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> >> +			hisi_ptt->trace_ctrl.has_rmr = true;
> >> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> >> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> >> +			break;
> >> +		}
> >> +
> >> +	iommu_put_resv_regions(&pdev->dev, &list);
> >> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> >> +}
> >> +
> >> +static int hisi_ptt_probe(struct pci_dev *pdev,
> >> +			  const struct pci_device_id *id)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt;
> >> +	int ret;
> >> +
> >> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> >> +	if (!hisi_ptt)
> >> +		return -ENOMEM;
> >> +
> >> +	mutex_init(&hisi_ptt->mutex);
> >> +	hisi_ptt->pdev = pdev;
> >> +
> >> +	/*
> >> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> >> +	 * so directly reference to the pci name string.
> >> +	 */
> >> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> >> +	pci_set_drvdata(pdev, hisi_ptt);
> >> +
> >> +	ret = pcim_enable_device(pdev);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> >> +
> >> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +	pci_set_master(pdev);
> >> +
> >> +	ret = hisi_ptt_register_irq(hisi_ptt);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	hisi_ptt_init_ctrls(hisi_ptt);
> >> +
> >> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> >> +		return ret;
> >> +	}
> > 
> > If this fails the pci_free_irq_vectors() is not called.
> > 
> 
> It'll be called as we've registered a devm callback when
> allocating the irq vector. See hisi_ptt_register_irq().

Ah yes - thanks for pointing that out.

> 
> > I am out of time for today - I will continue tomorrow.
> > 
> 
> Sure. Thanks!
> Yicong
> 
> > Thanks,
> > Mathieu
> > 
> >> +
> >> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	/* Register the bus notifier for dynamically updating the filter list */
> >> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> >> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> >> +	if (ret)
> >> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +void hisi_ptt_remove(struct pci_dev *pdev)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> >> +
> >> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> >> +
> >> +	/* Cancel any work that has been queued */
> >> +	cancel_delayed_work_sync(&hisi_ptt->work);
> >> +
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +
> >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> >> +	hisi_ptt_release_filters(hisi_ptt);
> >> +}
> >> +
> >> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> >> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> >> +	{ }
> >> +};
> >> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> >> +
> >> +static struct pci_driver hisi_ptt_driver = {
> >> +	.name = "hisi_ptt",
> >> +	.id_table = hisi_ptt_id_tbl,
> >> +	.probe = hisi_ptt_probe,
> >> +	.remove = hisi_ptt_remove,
> >> +};
> >> +module_pci_driver(hisi_ptt_driver);
> >> +
> >> +MODULE_LICENSE("GPL v2");
> >> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> >> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> >> -- 
> >> 2.33.0
> >>
> > .
> > 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

On Thu, Nov 25, 2021 at 04:39:46PM +0800, Yicong Yang wrote:
> Hi Mathieu,
> 
> Thanks for the comments! Replies inline.
> 
> On 2021/11/25 2:51, Mathieu Poirier wrote:
> > Hi Yicong,
> > 
> > On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
> >> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
> >> integrated Endpoint(RCiEP) device, providing the capability
> >> to dynamically monitor and tune the PCIe traffic(tune),
> >> and trace the TLP headers(trace).
> >>
> > 
> > Is there a reason to put "tune" and "trace" are whithin parentheses?  
> > 
> 
> yeah. I want to use these single word to denote the related function, and
> place them in the parantheses near the function's description to make sure
> readers can match them.
> 

Probably a better idea to simply write "TLP headers" and "PCIe traffic" when
referring to them.  Otherwise it is very confusing because the reader doesn't
know if you mean the acronym or the real definition of "tune" and "trace".

> >> Add the driver for the device to enable the trace function. The driver
> >> will create PMU device for each PTT device, and users can start trace
> >> through perf command.
> >>
> >> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> >> ---
> >>  drivers/Makefile                       |    1 +
> >>  drivers/hwtracing/Kconfig              |    2 +
> >>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
> >>  drivers/hwtracing/hisilicon/Makefile   |    2 +
> >>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
> > 
> > Not sure about the "hisilicon" directory.  Right now we have "coresight",
> > "intel_th" and "stm" - they concentrate on the technology rather than the
> > vendor.  I think "ptt" would be just fine: 
> > 
> > drivers/hwtracing/ptt/Kconfig 
> > drivers/hwtracing/ptt/Makefile
> > drivers/hwtracing/ptt/hisi_ptt.c
> > 
> > That way if other vendors want to introduce support for the same kind of
> > technology we can simply put them under "drivers/hwtracing/ptt/" and things are
> > still accurate.
> > 
> 
> sure. sounds sensible. will rename the directory to "ptt" if no objection.
> 
> > 
> >>  5 files changed, 1159 insertions(+)
> >>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
> >>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
> >>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
> >>
> >> diff --git a/drivers/Makefile b/drivers/Makefile
> >> index be5d40ae1488..bb0dc9b55ea2 100644
> >> --- a/drivers/Makefile
> >> +++ b/drivers/Makefile
> >> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
> >>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
> >>  obj-y				+= hwtracing/intel_th/
> >>  obj-$(CONFIG_STM)		+= hwtracing/stm/
> >> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
> >>  obj-$(CONFIG_ANDROID)		+= android/
> >>  obj-$(CONFIG_NVMEM)		+= nvmem/
> >>  obj-$(CONFIG_FPGA)		+= fpga/
> >> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
> >> index 13085835a636..e3796b17541a 100644
> >> --- a/drivers/hwtracing/Kconfig
> >> +++ b/drivers/hwtracing/Kconfig
> >> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
> >>  
> >>  source "drivers/hwtracing/intel_th/Kconfig"
> >>  
> >> +source "drivers/hwtracing/hisilicon/Kconfig"
> >> +
> >>  endmenu
> >> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
> >> new file mode 100644
> >> index 000000000000..9c3ab80d99fe
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/Kconfig
> >> @@ -0,0 +1,8 @@
> >> +# SPDX-License-Identifier: GPL-2.0-only
> >> +config HISI_PTT
> >> +	tristate "HiSilicon PCIe Tune and Trace Device"
> >> +	depends on PCI && HAS_DMA && HAS_IOMEM
> > 
> > What Arm architecture are you aiming this for?  Can it be found on both armv7
> > and armv8?
> > 
> 
> currently the device will appear on Kunpeng 930, which is armv8.

Ok, then please make it dependent on armv8.

> 
> >> +	help
> >> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
> >> +	  device, provides support for PCIe traffic tuning and
> >> +	  tracing TLP headers to the memory.
> > 
> > Please indicate what the name of the driver will be when compiled as a module.
> > 
> 
> sure. will add the module's name here.
> 
> >> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
> >> new file mode 100644
> >> index 000000000000..908c09a98161
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/Makefile
> >> @@ -0,0 +1,2 @@
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
> >> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
> >> new file mode 100644
> >> index 000000000000..e11e9b6cc2a8
> >> --- /dev/null
> >> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
> >> @@ -0,0 +1,1146 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +/*
> >> + * Driver for HiSilicon PCIe tune and trace device
> >> + *
> >> + * Copyright (c) 2021 HiSilicon Limited.
> >> + */
> >> +
> >> +#include <linux/bitfield.h>
> >> +#include <linux/bitops.h>
> >> +#include <linux/delay.h>
> >> +#include <linux/dma-iommu.h>
> >> +#include <linux/dma-mapping.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/io.h>
> >> +#include <linux/iommu.h>
> >> +#include <linux/iopoll.h>
> >> +#include <linux/kfifo.h>
> >> +#include <linux/module.h>
> >> +#include <linux/notifier.h>
> >> +#include <linux/pci.h>
> >> +#include <linux/perf_event.h>
> >> +#include <linux/sizes.h>
> >> +#include <linux/sysfs.h>
> >> +#include <linux/vmalloc.h>
> >> +#include <linux/workqueue.h>
> >> +
> >> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
> >> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
> >> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
> >> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
> >> +#define HISI_PTT_TRACE_CTRL		0x0850
> >> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
> >> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
> >> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
> >> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
> >> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
> >> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
> >> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
> >> +#define HISI_PTT_TRACE_INT_STAT		0x0890
> >> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
> >> +#define HISI_PTT_TRACE_INT_MASK		0x0894
> >> +#define HISI_PTT_TRACE_WR_STS		0x08a0
> >> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
> >> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
> >> +#define HISI_PTT_TRACE_STS		0x08b0
> >> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
> >> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
> >> +#define HISI_PTT_LOCATION		0x0fe8
> >> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
> >> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
> > 
> > Extra spaces between #define and the literals.  Please align the values with
> > that of HISI_PTT_TRACE_DMA_IRQ below.
> > 
> 
> Spaces are intended to distinguish the definition of the register offset and
> related register fields. also used in include/uapi/linux/pci_regs.h.
>

Interesting and definitely not the norm...

> HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
> in the mail text, maybe because the tab is not 8 characters when display.
> 

In my VI terminal the '0' of HISI_PTT_TRACE_DMA_IRQ is align with the '3' in
GENMASK(31, 16) and I suspect it is the same for other people.

> >> +
> >> +#define HISI_PTT_TRACE_DMA_IRQ			0
> >> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> >> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> >> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> >> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> >> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> >> +
> >> +/* Delay time for filter updating work */
> >> +#define HISI_PTT_WORK_DELAY_MS		100UL
> >> +/* Wait time for DMA hardware to reset */
> >> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> >> +/* Poll timeout and interval for waiting hardware work to finish */
> >> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> >> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> >> +
> >> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> >> +
> >> +enum hisi_ptt_trace_status {
> >> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> >> +	HISI_PTT_TRACE_STATUS_ON,
> >> +};
> >> +
> >> +struct hisi_ptt_dma_buflet {
> > 
> > I'm not sure what a "buflet" is...  Probably best to just call this
> > hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> > know due to the lack of proper structure documentation.  Please have a look at
> > how "coresight_device" is documented.  The same applies to the rest of the
> > structures declared below.
> > 
> 
> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
> (we call it buflet) of 4MiB ...".

I just noticed the documentation in patch 5 - I will read it before continuing
with this set.

> 
> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
> a 'buflet', which means a small buffer (there exists other words with similiar
> format: droplet, wavelet, ...).
> 
> The device is designed like this to make sure we won't lose any data when we have to change
> the buffer address, the device can continue to write to the next buflet and don't
> neet to pause.
> 
> >> +	struct list_head list;
> >> +	unsigned int size;
> >> +	dma_addr_t dma;
> >> +
> >> +	/*
> >> +	 * The address of the buflet holding the trace data.
> >> +	 * See Documentation/trace/hisi-ptt.rst for the details
> > 
> > As of this writing, hisi-ptt.rst doesn't exist.
> > 
> 
> The comment intends to direct the user to the documentation if the user
> want to know details about the data format. The data format is decribed
> in the section 4 of the doc.
> 
> >> +	 * of the data format.
> >> +	 */
> >> +	void *addr;
> >> +	int index;
> >> +};
> >> +
> >> +struct hisi_ptt_trace_ctrl {
> >> +	enum hisi_ptt_trace_status status;
> >> +	struct perf_output_handle handle;
> >> +	struct list_head trace_buf;
> >> +	/*
> >> +	 * The index of the buffer which trace data
> >> +	 * currently is writing to.
> >> +	 */
> >> +	u32 buf_index;
> >> +
> >> +	int default_cpu;
> >> +	u32 buflet_size;
> >> +	bool is_port;
> >> +	u32 format:1;		/* Format of the traced TLP headers */
> >> +	u32 type:4;		/* Type of the TLP headers to trace */
> >> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> >> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> >> +
> >> +	phys_addr_t rmr_addr;
> >> +	size_t rmr_length;
> >> +	bool has_rmr;
> >> +};
> >> +
> >> +struct hisi_ptt_filter_desc {
> >> +	struct list_head list;
> >> +	struct pci_dev *pdev;
> >> +	u16 val;
> >> +};
> >> +
> >> +struct hisi_ptt_pmu_buf {
> >> +	size_t length;
> >> +	int nr_pages;
> >> +	void *base;
> >> +	long pos;
> >> +};
> >> +
> >> +/* Structure containing the information for filter updating */
> >> +struct hisi_ptt_filter_update_info {
> >> +	struct pci_dev *pdev;
> >> +	u32 port_devid;
> >> +	bool is_port;
> >> +	bool is_add;
> >> +	u16 val;
> >> +};
> >> +
> >> +struct hisi_ptt {
> >> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> >> +	struct notifier_block hisi_ptt_nb;
> >> +	struct pmu hisi_ptt_pmu;
> >> +	void __iomem *iobase;
> >> +	struct pci_dev *pdev;
> >> +	/*
> >> +	 * Use the mutex to protect the filter list and
> >> +	 * serialize the perf process.
> >> +	 */
> >> +	struct mutex mutex;
> >> +	const char *name;
> >> +	u16 core_id;
> >> +	u16 sicl_id;
> >> +	/* PCI device range managed by the PTT device */
> >> +	u32 upper;
> >> +	u32 lower;
> >> +	u8 busnr;
> >> +
> >> +	/*
> >> +	 * The trace TLP headers can either be filtered by certain
> >> +	 * root port, or by the requester ID. Organize the filters
> >> +	 * by @port_filters and @req_filters here. The mask of all
> >> +	 * the valid ports is also cached for doing sanity check
> >> +	 * of user input.
> >> +	 */
> >> +	struct list_head port_filters;
> >> +	struct list_head req_filters;
> >> +	u16 port_mask;
> >> +
> >> +	/*
> >> +	 * We use a delayed work here to avoid indefinitely waiting for
> >> +	 * the hisi_ptt->mutex which protecting the filter list. The
> >> +	 * work will be delayed only if the mutex can not be held,
> >> +	 * otherwise no delay will be applied.
> >> +	 */
> >> +	struct delayed_work work;
> >> +	spinlock_t filter_update_lock;
> >> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> >> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> >> +};
> > 
> > It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> > 
> 
> I'm not sure whether it is common or permitted to provide a header which is only used by
> this driver with a single file.

I've seen it done many times.

> 
> >> +
> >> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> >> +{
> >> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> >> +}
> > 
> > I'm not sure there is any value to this inline function.  I suggest to simply
> > call container_of() whenever it is needed.
> > 
> 
> ok.
> 
> >> +
> >> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> >> +{
> >> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> >> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> >> +
> >> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> >> +}
> >> +
> >> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	u32 val;
> >> +
> >> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> >> +				  val & HISI_PTT_TRACE_IDLE,
> >> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> >> +				  HISI_PTT_WAIT_TIMEOUT_US);
> >> +}
> >> +
> >> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct device *dev = &hisi_ptt->pdev->dev;
> >> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> >> +
> >> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> >> +		list_del(&buflet->list);
> >> +
> >> +		if (ctrl->has_rmr)
> >> +			memunmap(buflet->addr);
> >> +		else
> >> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> >> +					  buflet->dma);
> >> +
> >> +		kfree(buflet);
> >> +	}
> >> +}
> >> +
> >> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct device *dev = &hisi_ptt->pdev->dev;
> >> +	struct hisi_ptt_dma_buflet *buflet;
> >> +	int i, ret;
> >> +
> >> +	hisi_ptt->trace_ctrl.buf_index = 0;
> >> +
> >> +	/* Make sure the trace buffer is empty before allocating */
> >> +	if (!list_empty(&ctrl->trace_buf)) {
> >> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> >> +			memset(buflet->addr, 0, buflet->size);
> >> +		return 0;
> >> +	}
> >> +
> >> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> >> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> >> +		if (!buflet) {
> >> +			ret = -ENOMEM;
> >> +			goto err;
> >> +		}
> >> +
> >> +		if (ctrl->has_rmr) {
> >> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> >> +
> >> +			buflet->dma = base;
> >> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> >> +		} else {
> >> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> >> +							  &buflet->dma, GFP_KERNEL);
> >> +		}
> >> +
> >> +		if (!buflet->addr) {
> >> +			kfree(buflet);
> >> +			ret = -ENOMEM;
> >> +			goto err;
> >> +		}
> >> +
> >> +		memset(buflet->addr, 0, buflet->size);
> >> +
> >> +		buflet->index = i;
> >> +		buflet->size = ctrl->buflet_size;
> >> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> >> +	}
> >> +
> >> +	return 0;
> >> +err:
> >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> >> +	return ret;
> >> +}
> >> +
> >> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	struct hisi_ptt_dma_buflet *cur;
> >> +	u32 val;
> >> +
> >> +	/* Check device idle before start trace */
> >> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> >> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> >> +		return -EBUSY;
> >> +	}
> >> +
> >> +	/* Reset the DMA before start tracing */
> >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	val |= HISI_PTT_TRACE_CTRL_RST;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	/*
> >> +	 * We'll be in the perf context where preemption is disabled,
> >> +	 * so use busy loop here.
> >> +	 */
> >> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> >> +
> >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	/* clear the interrupt status */
> >> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> >> +
> >> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> >> +		writel(lower_32_bits(cur->dma),
> >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> >> +		writel(upper_32_bits(cur->dma),
> >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> >> +	}
> >> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> >> +
> >> +	/* set the trace control register */
> >> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> >> +	if (!hisi_ptt->trace_ctrl.is_port)
> >> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> >> +
> >> +	val |= HISI_PTT_TRACE_CTRL_EN;
> >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> >> +
> >> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> >> +{
> >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> >> +	struct perf_event *event = handle->event;
> >> +	struct hisi_ptt_dma_buflet *cur;
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +
> >> +	buf = perf_get_aux(handle);
> >> +	if (!buf || !handle->size)
> >> +		return -EINVAL;
> >> +
> >> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> >> +		if (cur->index == index)
> >> +			break;
> >> +
> >> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> >> +	memset(cur->addr, 0, cur->size);
> >> +	buf->pos += cur->size;
> >> +
> >> +	if (stop) {
> >> +		perf_aux_output_end(handle, buf->pos);
> >> +	} else if (buf->length - buf->pos < cur->size) {
> >> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> >> +		perf_aux_output_end(handle, buf->pos);
> >> +
> >> +		buf = perf_aux_output_begin(handle, event);
> >> +		if (!buf)
> >> +			return -EINVAL;
> >> +
> >> +		buf->pos = handle->head % buf->length;
> >> +		if (buf->length - buf->pos < cur->size) {
> >> +			perf_aux_output_end(handle, 0);
> >> +			return -EINVAL;
> >> +		}
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = context;
> >> +	u32 status, buf_idx;
> >> +
> >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	buf_idx = ffs(status) - 1;
> >> +
> >> +	/* Clear the interrupt status of buflet @buf_idx */
> >> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +
> >> +	/*
> >> +	 * Update the AUX buffer and cache the current buflet index,
> >> +	 * as we need to know this and save the data when the trace
> >> +	 * is ended out of the interrupt handler. End the trace
> >> +	 * if the updating fails.
> >> +	 */
> >> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +	else
> >> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> >> +
> >> +	return IRQ_HANDLED;
> >> +}
> >> +
> >> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = context;
> >> +	u32 status;
> >> +
> >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> >> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> >> +		return IRQ_NONE;
> >> +
> >> +	return IRQ_WAKE_THREAD;
> >> +}
> >> +
> >> +static void hisi_ptt_irq_free_vectors(void *pdev)
> >> +{
> >> +	pci_free_irq_vectors(pdev);
> >> +}
> >> +
> >> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	int ret;
> >> +
> >> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> >> +	if (ret < 0) {
> >> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	ret = devm_request_threaded_irq(&pdev->dev,
> >> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> >> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> >> +					"hisi-ptt", hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> >> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_update_filters(struct work_struct *work)
> >> +{
> >> +	struct delayed_work *delayed_work = to_delayed_work(work);
> >> +	struct hisi_ptt_filter_update_info info;
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	struct list_head *target_list;
> >> +	struct hisi_ptt *hisi_ptt;
> >> +
> >> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> >> +
> >> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> >> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> >> +		return;
> >> +	}
> >> +
> >> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> >> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> >> +			      &hisi_ptt->req_filters;
> >> +
> >> +		if (info.is_add) {
> >> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> >> +			if (!filter)
> >> +				continue;
> >> +
> >> +			filter->pdev = info.pdev;
> >> +			filter->val = info.val;
> >> +
> >> +			list_add_tail(&filter->list, target_list);
> >> +		} else {
> >> +			list_for_each_entry(filter, target_list, list)
> >> +				if (filter->val == info.val) {
> >> +					list_del(&filter->list);
> >> +					kfree(filter);
> >> +					break;
> >> +				}
> >> +		}
> >> +
> >> +		/* Update the available port mask */
> >> +		if (!info.is_port)
> >> +			continue;
> >> +
> >> +		if (info.is_add)
> >> +			hisi_ptt->port_mask |= info.val;
> >> +		else
> >> +			hisi_ptt->port_mask &= ~info.val;
> >> +	}
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +}
> >> +
> >> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> >> +				    struct hisi_ptt_filter_update_info *info)
> >> +{
> >> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> >> +
> >> +	if (!root_port)
> >> +		return;
> >> +
> >> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> >> +	if (info->port_devid < hisi_ptt->lower ||
> >> +	    info->port_devid > hisi_ptt->upper)
> >> +		return;
> >> +
> >> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> >> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> >> +
> >> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> >> +				&hisi_ptt->filter_update_lock))
> >> +		schedule_delayed_work(&hisi_ptt->work, 0);
> >> +	else
> >> +		pci_warn(hisi_ptt->pdev,
> >> +			 "filter update fifo overflow for target %s\n",
> >> +			 pci_name(info->pdev));
> >> +}
> >> +
> >> +/*
> >> + * A PCI bus notifier is used here for dynamically updating the filter
> >> + * list.
> >> + */
> >> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> >> +				  void *data)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> >> +	struct hisi_ptt_filter_update_info info;
> >> +	struct device *dev = data;
> >> +	struct pci_dev *pdev = to_pci_dev(dev);
> >> +
> >> +	info.pdev = pdev;
> >> +
> >> +	switch (action) {
> >> +	case BUS_NOTIFY_ADD_DEVICE:
> >> +		info.is_add = true;
> >> +		break;
> >> +	case BUS_NOTIFY_DEL_DEVICE:
> >> +		info.is_add = false;
> >> +		break;
> >> +	default:
> >> +		return 0;
> >> +	}
> >> +
> >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> >> +{
> >> +	struct hisi_ptt_filter_update_info info = {
> >> +		.pdev = pdev,
> >> +		.is_add = true,
> >> +	};
> >> +	struct hisi_ptt *hisi_ptt = data;
> >> +
> >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> >> +
> >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> >> +		list_del(&filter->list);
> >> +		kfree(filter);
> >> +	}
> >> +
> >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> >> +		list_del(&filter->list);
> >> +		kfree(filter);
> >> +	}
> >> +}
> >> +
> >> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	struct pci_bus *bus;
> >> +	u32 reg;
> >> +
> >> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> >> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> >> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> >> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> >> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> >> +
> >> +	/*
> >> +	 * The device range register provides the information about the
> >> +	 * root ports which the RCiEP can control and trace. The RCiEP
> >> +	 * and the root ports it support are on the same PCIe core, with
> >> +	 * same domain number but maybe different bus number. The device
> >> +	 * range register will tell us which root ports we can support,
> >> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> >> +	 * while Bit[15:0] indicates the lower.
> >> +	 */
> >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> >> +	hisi_ptt->upper = reg >> 16;
> >> +	hisi_ptt->lower = reg & 0xffff;
> >> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> >> +
> >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> >> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> >> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> >> +
> >> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> >> +	if (bus)
> >> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> > 
> > What happens if bus is NULL?  Should this be reported to the caller and cause
> > hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> > 
> 
> It's ok if the list is NULL in the probe process,
> as the device maybe hotplugged after the PTT driver probe, in which case
> we can detect the event and update the list as we register a bus notifier.
> 

Please add the above as a comment in the code.


> >> +
> >> +	/* Initialize trace controls */
> >> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> >> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> >> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> >> +}
> >> +
> >> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> >> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> >> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> >> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> >> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> >> +
> >> +static ssize_t available_filters_show(struct device *dev,
> >> +				      struct device_attribute *attr,
> >> +				      char *buf)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	int pos = 0;
> >> +
> >> +	if (list_empty(&hisi_ptt->port_filters))
> >> +		return sysfs_emit(buf, "#### No available filter ####\n");
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> >> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> >> +				     pci_name(filter->pdev),
> >> +				     hisi_ptt_get_filter_val(filter->pdev) |
> >> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> >> +
> >> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> >> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> >> +				     pci_name(filter->pdev),
> >> +				     hisi_ptt_get_filter_val(filter->pdev));
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return pos;
> >> +}
> >> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> >> +
> >> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> >> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> >> +PMU_FORMAT_ATTR(type,		"config:24-31");
> >> +PMU_FORMAT_ATTR(format,		"config:32-35");
> >> +
> >> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> >> +	&format_attr_filter.attr,
> >> +	&format_attr_direction.attr,
> >> +	&format_attr_type.attr,
> >> +	&format_attr_format.attr,
> >> +	NULL
> >> +};
> >> +
> >> +static struct attribute_group hisi_ptt_pmu_format_group = {
> >> +	.name = "format",
> >> +	.attrs = hisi_ptt_pmu_format_attrs,
> >> +};
> >> +
> >> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> >> +	&dev_attr_available_filters.attr,
> >> +	NULL
> >> +};
> >> +
> >> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> >> +	.attrs = hisi_ptt_pmu_filter_attrs,
> >> +};
> >> +
> >> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> >> +	&hisi_ptt_pmu_format_group,
> >> +	&hisi_ptt_pmu_filter_group,
> >> +	NULL
> >> +};
> >> +
> >> +/*
> >> + * The supported value of the direction parameter. See hisi_ptt.rst
> >> + * documentation for more details.
> >> + */
> >> +static u32 hisi_ptt_trace_available_direction[] = {
> >> +	0,
> >> +	1,
> >> +	2,
> >> +	3,
> >> +};
> >> +
> >> +/* Different types can be set simultaneously */
> >> +static u32 hisi_ptt_trace_available_type[] = {
> >> +	1,	/* posted_request */
> >> +	2,	/* non-posted_request */
> >> +	4,	/* completion */
> >> +};
> >> +
> >> +static u32 hisi_ptt_trace_availble_format[] = {
> >> +	0,	/* 4DW */
> >> +	1,	/* 8DW */
> >> +};
> >> +
> >> +/*
> >> + * Check whether the config is valid or not. Some configs are multi-selectable
> >> + * and can be set simultaneously, while some are single selectable (onehot).
> >> + * Use this function to check the non-onehot configs while
> >> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> >> + */
> >> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> >> +{
> >> +	int i;
> >> +
> >> +	/*
> >> +	 * The non-onehot configs cannot be 0. Walk the available
> >> +	 * list and clear the valid bits of the config. If there
> >> +	 * is any resident bit after the walk then the config is
> >> +	 * invalid.
> >> +	 */
> >> +	for (i = 0; i < list_size; i++)
> >> +		val &= ~available_list[i];
> >> +
> >> +	return val ? -EINVAL : 0;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> >> +{
> >> +	int i, ret = -EINVAL;
> >> +
> >> +	for (i = 0; i < list_size; i++)
> >> +		if (val == available_list[i]) {
> >> +			ret = 0;
> >> +			break;
> >> +		}
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> >> +{
> >> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> >> +	struct hisi_ptt_filter_desc *filter;
> >> +	int ret = -EINVAL;
> >> +
> >> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> >> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> >> +
> >> +	/*
> >> +	 * Port filters are defined as bit mask. For port filters, check
> >> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> >> +	 * and whether it's empty or not, otherwise user has specified
> >> +	 * some unsupported root ports.
> >> +	 *
> >> +	 * For Requester ID filters, walk the available filter list to see
> >> +	 * whether we have one matched.
> >> +	 */
> >> +	if (hisi_ptt->trace_ctrl.is_port &&
> >> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> >> +		ret = 0;
> >> +	} else {
> >> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> >> +			if (filter->val == val) {
> >> +				ret = 0;
> >> +				break;
> >> +			}
> >> +	}
> >> +
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	hisi_ptt->trace_ctrl.filter = val;
> >> +	return 0;
> >> +}
> >> +
> >> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> >> +	int ret;
> >> +	u32 val;
> >> +
> >> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> >> +		return -ENOENT;
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +
> >> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> >> +	if (ret < 0)
> >> +		goto out;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> >> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->direction = val;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> >> +
> >> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> >> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->type = val;
> >> +
> >> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> >> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> >> +	if (ret < 0)
> >> +		goto out;
> >> +	ctrl->format = val;
> >> +
> >> +out:
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return ret;
> >> +}
> >> +
> >> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> >> +				    int nr_pages, bool overwrite)
> >> +{
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +	struct page **pagelist;
> >> +	int i;
> >> +
> >> +	if (overwrite) {
> >> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> >> +		return NULL;
> >> +	}
> >> +
> >> +	/* If the pages size less than buflets, we cannot start trace */
> >> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> >> +		return NULL;
> >> +
> >> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> >> +	if (!buf)
> >> +		return NULL;
> >> +
> >> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> >> +	if (!pagelist) {
> >> +		kfree(buf);
> >> +		return NULL;
> >> +	}
> >> +
> >> +	for (i = 0; i < nr_pages; i++)
> >> +		pagelist[i] = virt_to_page(pages[i]);
> >> +
> >> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> >> +	if (!buf->base) {
> >> +		kfree(pagelist);
> >> +		kfree(buf);
> >> +		return NULL;
> >> +	}
> >> +
> >> +	buf->nr_pages = nr_pages;
> >> +	buf->length = nr_pages * PAGE_SIZE;
> >> +	buf->pos = 0;
> >> +
> >> +	kfree(pagelist);
> >> +	return buf;
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_free_aux(void *aux)
> >> +{
> >> +	struct hisi_ptt_pmu_buf *buf = aux;
> >> +
> >> +	vunmap(buf->base);
> >> +	kfree(buf);
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +	struct hisi_ptt_pmu_buf *buf;
> >> +	int cpu = event->cpu;
> >> +	int ret;
> >> +
> >> +	hwc->state = 0;
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> >> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> >> +		goto stop;
> >> +	}
> >> +
> >> +	if (cpu == -1)
> >> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> >> +
> >> +	/*
> >> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> >> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> >> +	 * core in event_function_local().
> >> +	 */
> >> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> >> +				 cpumask_of(cpu)));
> >> +
> >> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> >> +		goto stop;
> >> +	}
> >> +
> >> +	buf = perf_aux_output_begin(handle, event);
> >> +	if (!buf) {
> >> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> >> +		goto stop;
> >> +	}
> >> +
> >> +	buf->pos = handle->head % buf->length;
> >> +
> >> +	ret = hisi_ptt_trace_start(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> >> +		perf_aux_output_end(handle, 0);
> >> +		goto stop;
> >> +	}
> >> +
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +	return;
> >> +stop:
> >> +	event->hw.state |= PERF_HES_STOPPED;
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +
> >> +	if (hwc->state & PERF_HES_STOPPED)
> >> +		return;
> >> +
> >> +	mutex_lock(&hisi_ptt->mutex);
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> >> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> >> +	}
> >> +	mutex_unlock(&hisi_ptt->mutex);
> >> +
> >> +	hwc->state |= PERF_HES_STOPPED;
> >> +	perf_event_update_userpage(event);
> >> +	hwc->state |= PERF_HES_UPTODATE;
> >> +}
> >> +
> >> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> >> +	struct hw_perf_event *hwc = &event->hw;
> >> +	int cpu = event->cpu;
> >> +
> >> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> >> +		return 0;
> >> +
> >> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> >> +
> >> +	if (flags & PERF_EF_START) {
> >> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> >> +		if (hwc->state & PERF_HES_STOPPED)
> >> +			return -EINVAL;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> >> +{
> >> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> >> +}
> >> +
> >> +static void hisi_ptt_unregister_pmu(void *priv)
> >> +{
> >> +	perf_pmu_unregister(priv);
> >> +}
> >> +
> >> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	char *pmu_name;
> >> +	int ret;
> >> +
> >> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> >> +		.module		= THIS_MODULE,
> >> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> >> +		.task_ctx_nr	= perf_sw_context,
> >> +		.attr_groups	= hisi_ptt_pmu_groups,
> >> +		.event_init	= hisi_ptt_pmu_event_init,
> >> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> >> +		.free_aux	= hisi_ptt_pmu_free_aux,
> >> +		.start		= hisi_ptt_pmu_start,
> >> +		.stop		= hisi_ptt_pmu_stop,
> >> +		.add		= hisi_ptt_pmu_add,
> >> +		.del		= hisi_ptt_pmu_del,
> >> +	};
> >> +
> >> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> >> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> >> +	if (!pmu_name)
> >> +		return -ENOMEM;
> >> +
> >> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> >> +					hisi_ptt_unregister_pmu,
> >> +					&hisi_ptt->hisi_ptt_pmu);
> >> +}
> >> +
> >> +/*
> >> + * Get RMR address if provided by the firmware.
> >> + * Return 0 if the IOMMU doesn't present or the policy of the
> >> + * IOMMU domain is passthrough or we get a usable RMR region.
> >> + * Otherwise a negative value is returned.
> >> + */
> >> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> >> +{
> >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> >> +	struct iommu_domain *iommu_domain;
> >> +	struct iommu_resv_region *region;
> >> +	LIST_HEAD(list);
> >> +
> >> +	/*
> >> +	 * Use direct DMA if IOMMU does not present or the policy of the
> >> +	 * IOMMU domain is passthrough.
> >> +	 */
> >> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> >> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> >> +		return 0;
> >> +
> >> +	iommu_get_resv_regions(&pdev->dev, &list);
> >> +	list_for_each_entry(region, &list, list)
> >> +		if (region->type == IOMMU_RESV_DIRECT &&
> >> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> >> +			hisi_ptt->trace_ctrl.has_rmr = true;
> >> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> >> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> >> +			break;
> >> +		}
> >> +
> >> +	iommu_put_resv_regions(&pdev->dev, &list);
> >> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> >> +}
> >> +
> >> +static int hisi_ptt_probe(struct pci_dev *pdev,
> >> +			  const struct pci_device_id *id)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt;
> >> +	int ret;
> >> +
> >> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> >> +	if (!hisi_ptt)
> >> +		return -ENOMEM;
> >> +
> >> +	mutex_init(&hisi_ptt->mutex);
> >> +	hisi_ptt->pdev = pdev;
> >> +
> >> +	/*
> >> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> >> +	 * so directly reference to the pci name string.
> >> +	 */
> >> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> >> +	pci_set_drvdata(pdev, hisi_ptt);
> >> +
> >> +	ret = pcim_enable_device(pdev);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> >> +
> >> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> >> +		return ret;
> >> +	}
> >> +	pci_set_master(pdev);
> >> +
> >> +	ret = hisi_ptt_register_irq(hisi_ptt);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	hisi_ptt_init_ctrls(hisi_ptt);
> >> +
> >> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> >> +		return ret;
> >> +	}
> > 
> > If this fails the pci_free_irq_vectors() is not called.
> > 
> 
> It'll be called as we've registered a devm callback when
> allocating the irq vector. See hisi_ptt_register_irq().

Ah yes - thanks for pointing that out.

> 
> > I am out of time for today - I will continue tomorrow.
> > 
> 
> Sure. Thanks!
> Yicong
> 
> > Thanks,
> > Mathieu
> > 
> >> +
> >> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> >> +	if (ret) {
> >> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	/* Register the bus notifier for dynamically updating the filter list */
> >> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> >> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> >> +	if (ret)
> >> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +void hisi_ptt_remove(struct pci_dev *pdev)
> >> +{
> >> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> >> +
> >> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> >> +
> >> +	/* Cancel any work that has been queued */
> >> +	cancel_delayed_work_sync(&hisi_ptt->work);
> >> +
> >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> >> +		hisi_ptt_trace_end(hisi_ptt);
> >> +
> >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> >> +	hisi_ptt_release_filters(hisi_ptt);
> >> +}
> >> +
> >> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> >> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> >> +	{ }
> >> +};
> >> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> >> +
> >> +static struct pci_driver hisi_ptt_driver = {
> >> +	.name = "hisi_ptt",
> >> +	.id_table = hisi_ptt_id_tbl,
> >> +	.probe = hisi_ptt_probe,
> >> +	.remove = hisi_ptt_remove,
> >> +};
> >> +module_pci_driver(hisi_ptt_driver);
> >> +
> >> +MODULE_LICENSE("GPL v2");
> >> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> >> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> >> -- 
> >> 2.33.0
> >>
> > .
> > 

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

[...]

> 
> > >> +
> > >> +#define HISI_PTT_TRACE_DMA_IRQ			0
> > >> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> > >> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> > >> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> > >> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> > >> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> > >> +
> > >> +/* Delay time for filter updating work */
> > >> +#define HISI_PTT_WORK_DELAY_MS		100UL
> > >> +/* Wait time for DMA hardware to reset */
> > >> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> > >> +/* Poll timeout and interval for waiting hardware work to finish */
> > >> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> > >> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> > >> +
> > >> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> > >> +
> > >> +enum hisi_ptt_trace_status {
> > >> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> > >> +	HISI_PTT_TRACE_STATUS_ON,
> > >> +};
> > >> +
> > >> +struct hisi_ptt_dma_buflet {
> > > 
> > > I'm not sure what a "buflet" is...  Probably best to just call this
> > > hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> > > know due to the lack of proper structure documentation.  Please have a look at
> > > how "coresight_device" is documented.  The same applies to the rest of the
> > > structures declared below.
> > > 
> > 
> > It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
> > (we call it buflet) of 4MiB ...".
> 
> I just noticed the documentation in patch 5 - I will read it before continuing
> with this set.
>

As promised I read the documentation and it didn't do much to alliviate my
concern about the terminology.  If a "buflet" is a DMA buffer than simply call
it a DMA buffer, regardless of the size.  

The size of this patchset makes it hard enough to review as it is... On top
of things reviewers have to remember that PTT means "PCIe tune and trace
device", tune means "PCIe link's events" and trace means "TLP headers".

But we also have to keep in mind that a buflet is really just a DMA buffer...

I will not continue reviewing this set.  Please fix the terminology and spin
another revision.  Doing so I suggest you trim down the submission to the
absolute minimum feature set this driver should provide - enhancements can be
made later when a foundation has been laid out.

Robin has pointed out what seems to be serious problems with the
implemenation.  That will also have to be sorted out for the next iteration.

Regards,
Mathieu


> > 
> > The total DMA buffer of PTT device is divided into 4 parts, and each part is called
> > a 'buflet', which means a small buffer (there exists other words with similiar
> > format: droplet, wavelet, ...).
> > 
> > The device is designed like this to make sure we won't lose any data when we have to change
> > the buffer address, the device can continue to write to the next buflet and don't
> > neet to pause.
> > 
> > >> +	struct list_head list;
> > >> +	unsigned int size;
> > >> +	dma_addr_t dma;
> > >> +
> > >> +	/*
> > >> +	 * The address of the buflet holding the trace data.
> > >> +	 * See Documentation/trace/hisi-ptt.rst for the details
> > > 
> > > As of this writing, hisi-ptt.rst doesn't exist.
> > > 
> > 
> > The comment intends to direct the user to the documentation if the user
> > want to know details about the data format. The data format is decribed
> > in the section 4 of the doc.
> > 
> > >> +	 * of the data format.
> > >> +	 */
> > >> +	void *addr;
> > >> +	int index;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_trace_ctrl {
> > >> +	enum hisi_ptt_trace_status status;
> > >> +	struct perf_output_handle handle;
> > >> +	struct list_head trace_buf;
> > >> +	/*
> > >> +	 * The index of the buffer which trace data
> > >> +	 * currently is writing to.
> > >> +	 */
> > >> +	u32 buf_index;
> > >> +
> > >> +	int default_cpu;
> > >> +	u32 buflet_size;
> > >> +	bool is_port;
> > >> +	u32 format:1;		/* Format of the traced TLP headers */
> > >> +	u32 type:4;		/* Type of the TLP headers to trace */
> > >> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> > >> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> > >> +
> > >> +	phys_addr_t rmr_addr;
> > >> +	size_t rmr_length;
> > >> +	bool has_rmr;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_filter_desc {
> > >> +	struct list_head list;
> > >> +	struct pci_dev *pdev;
> > >> +	u16 val;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_pmu_buf {
> > >> +	size_t length;
> > >> +	int nr_pages;
> > >> +	void *base;
> > >> +	long pos;
> > >> +};
> > >> +
> > >> +/* Structure containing the information for filter updating */
> > >> +struct hisi_ptt_filter_update_info {
> > >> +	struct pci_dev *pdev;
> > >> +	u32 port_devid;
> > >> +	bool is_port;
> > >> +	bool is_add;
> > >> +	u16 val;
> > >> +};
> > >> +
> > >> +struct hisi_ptt {
> > >> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> > >> +	struct notifier_block hisi_ptt_nb;
> > >> +	struct pmu hisi_ptt_pmu;
> > >> +	void __iomem *iobase;
> > >> +	struct pci_dev *pdev;
> > >> +	/*
> > >> +	 * Use the mutex to protect the filter list and
> > >> +	 * serialize the perf process.
> > >> +	 */
> > >> +	struct mutex mutex;
> > >> +	const char *name;
> > >> +	u16 core_id;
> > >> +	u16 sicl_id;
> > >> +	/* PCI device range managed by the PTT device */
> > >> +	u32 upper;
> > >> +	u32 lower;
> > >> +	u8 busnr;
> > >> +
> > >> +	/*
> > >> +	 * The trace TLP headers can either be filtered by certain
> > >> +	 * root port, or by the requester ID. Organize the filters
> > >> +	 * by @port_filters and @req_filters here. The mask of all
> > >> +	 * the valid ports is also cached for doing sanity check
> > >> +	 * of user input.
> > >> +	 */
> > >> +	struct list_head port_filters;
> > >> +	struct list_head req_filters;
> > >> +	u16 port_mask;
> > >> +
> > >> +	/*
> > >> +	 * We use a delayed work here to avoid indefinitely waiting for
> > >> +	 * the hisi_ptt->mutex which protecting the filter list. The
> > >> +	 * work will be delayed only if the mutex can not be held,
> > >> +	 * otherwise no delay will be applied.
> > >> +	 */
> > >> +	struct delayed_work work;
> > >> +	spinlock_t filter_update_lock;
> > >> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> > >> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> > >> +};
> > > 
> > > It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> > > 
> > 
> > I'm not sure whether it is common or permitted to provide a header which is only used by
> > this driver with a single file.
> 
> I've seen it done many times.
> 
> > 
> > >> +
> > >> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> > >> +{
> > >> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> > >> +}
> > > 
> > > I'm not sure there is any value to this inline function.  I suggest to simply
> > > call container_of() whenever it is needed.
> > > 
> > 
> > ok.
> > 
> > >> +
> > >> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> > >> +{
> > >> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> > >> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> > >> +
> > >> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	u32 val;
> > >> +
> > >> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> > >> +				  val & HISI_PTT_TRACE_IDLE,
> > >> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> > >> +				  HISI_PTT_WAIT_TIMEOUT_US);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct device *dev = &hisi_ptt->pdev->dev;
> > >> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> > >> +
> > >> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> > >> +		list_del(&buflet->list);
> > >> +
> > >> +		if (ctrl->has_rmr)
> > >> +			memunmap(buflet->addr);
> > >> +		else
> > >> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> > >> +					  buflet->dma);
> > >> +
> > >> +		kfree(buflet);
> > >> +	}
> > >> +}
> > >> +
> > >> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct device *dev = &hisi_ptt->pdev->dev;
> > >> +	struct hisi_ptt_dma_buflet *buflet;
> > >> +	int i, ret;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.buf_index = 0;
> > >> +
> > >> +	/* Make sure the trace buffer is empty before allocating */
> > >> +	if (!list_empty(&ctrl->trace_buf)) {
> > >> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> > >> +			memset(buflet->addr, 0, buflet->size);
> > >> +		return 0;
> > >> +	}
> > >> +
> > >> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> > >> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> > >> +		if (!buflet) {
> > >> +			ret = -ENOMEM;
> > >> +			goto err;
> > >> +		}
> > >> +
> > >> +		if (ctrl->has_rmr) {
> > >> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> > >> +
> > >> +			buflet->dma = base;
> > >> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> > >> +		} else {
> > >> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> > >> +							  &buflet->dma, GFP_KERNEL);
> > >> +		}
> > >> +
> > >> +		if (!buflet->addr) {
> > >> +			kfree(buflet);
> > >> +			ret = -ENOMEM;
> > >> +			goto err;
> > >> +		}
> > >> +
> > >> +		memset(buflet->addr, 0, buflet->size);
> > >> +
> > >> +		buflet->index = i;
> > >> +		buflet->size = ctrl->buflet_size;
> > >> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +err:
> > >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct hisi_ptt_dma_buflet *cur;
> > >> +	u32 val;
> > >> +
> > >> +	/* Check device idle before start trace */
> > >> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> > >> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> > >> +		return -EBUSY;
> > >> +	}
> > >> +
> > >> +	/* Reset the DMA before start tracing */
> > >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	val |= HISI_PTT_TRACE_CTRL_RST;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	/*
> > >> +	 * We'll be in the perf context where preemption is disabled,
> > >> +	 * so use busy loop here.
> > >> +	 */
> > >> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> > >> +
> > >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	/* clear the interrupt status */
> > >> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> > >> +
> > >> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> > >> +		writel(lower_32_bits(cur->dma),
> > >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> > >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> > >> +		writel(upper_32_bits(cur->dma),
> > >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> > >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> > >> +	}
> > >> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> > >> +
> > >> +	/* set the trace control register */
> > >> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> > >> +	if (!hisi_ptt->trace_ctrl.is_port)
> > >> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> > >> +
> > >> +	val |= HISI_PTT_TRACE_CTRL_EN;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> > >> +{
> > >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> > >> +	struct perf_event *event = handle->event;
> > >> +	struct hisi_ptt_dma_buflet *cur;
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +
> > >> +	buf = perf_get_aux(handle);
> > >> +	if (!buf || !handle->size)
> > >> +		return -EINVAL;
> > >> +
> > >> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> > >> +		if (cur->index == index)
> > >> +			break;
> > >> +
> > >> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> > >> +	memset(cur->addr, 0, cur->size);
> > >> +	buf->pos += cur->size;
> > >> +
> > >> +	if (stop) {
> > >> +		perf_aux_output_end(handle, buf->pos);
> > >> +	} else if (buf->length - buf->pos < cur->size) {
> > >> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> > >> +		perf_aux_output_end(handle, buf->pos);
> > >> +
> > >> +		buf = perf_aux_output_begin(handle, event);
> > >> +		if (!buf)
> > >> +			return -EINVAL;
> > >> +
> > >> +		buf->pos = handle->head % buf->length;
> > >> +		if (buf->length - buf->pos < cur->size) {
> > >> +			perf_aux_output_end(handle, 0);
> > >> +			return -EINVAL;
> > >> +		}
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = context;
> > >> +	u32 status, buf_idx;
> > >> +
> > >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	buf_idx = ffs(status) - 1;
> > >> +
> > >> +	/* Clear the interrupt status of buflet @buf_idx */
> > >> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +
> > >> +	/*
> > >> +	 * Update the AUX buffer and cache the current buflet index,
> > >> +	 * as we need to know this and save the data when the trace
> > >> +	 * is ended out of the interrupt handler. End the trace
> > >> +	 * if the updating fails.
> > >> +	 */
> > >> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +	else
> > >> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> > >> +
> > >> +	return IRQ_HANDLED;
> > >> +}
> > >> +
> > >> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = context;
> > >> +	u32 status;
> > >> +
> > >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> > >> +		return IRQ_NONE;
> > >> +
> > >> +	return IRQ_WAKE_THREAD;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_irq_free_vectors(void *pdev)
> > >> +{
> > >> +	pci_free_irq_vectors(pdev);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	int ret;
> > >> +
> > >> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> > >> +	if (ret < 0) {
> > >> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> > >> +	if (ret < 0)
> > >> +		return ret;
> > >> +
> > >> +	ret = devm_request_threaded_irq(&pdev->dev,
> > >> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> > >> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> > >> +					"hisi-ptt", hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> > >> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_update_filters(struct work_struct *work)
> > >> +{
> > >> +	struct delayed_work *delayed_work = to_delayed_work(work);
> > >> +	struct hisi_ptt_filter_update_info info;
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	struct list_head *target_list;
> > >> +	struct hisi_ptt *hisi_ptt;
> > >> +
> > >> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> > >> +
> > >> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> > >> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> > >> +		return;
> > >> +	}
> > >> +
> > >> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> > >> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> > >> +			      &hisi_ptt->req_filters;
> > >> +
> > >> +		if (info.is_add) {
> > >> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> > >> +			if (!filter)
> > >> +				continue;
> > >> +
> > >> +			filter->pdev = info.pdev;
> > >> +			filter->val = info.val;
> > >> +
> > >> +			list_add_tail(&filter->list, target_list);
> > >> +		} else {
> > >> +			list_for_each_entry(filter, target_list, list)
> > >> +				if (filter->val == info.val) {
> > >> +					list_del(&filter->list);
> > >> +					kfree(filter);
> > >> +					break;
> > >> +				}
> > >> +		}
> > >> +
> > >> +		/* Update the available port mask */
> > >> +		if (!info.is_port)
> > >> +			continue;
> > >> +
> > >> +		if (info.is_add)
> > >> +			hisi_ptt->port_mask |= info.val;
> > >> +		else
> > >> +			hisi_ptt->port_mask &= ~info.val;
> > >> +	}
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> > >> +				    struct hisi_ptt_filter_update_info *info)
> > >> +{
> > >> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> > >> +
> > >> +	if (!root_port)
> > >> +		return;
> > >> +
> > >> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> > >> +	if (info->port_devid < hisi_ptt->lower ||
> > >> +	    info->port_devid > hisi_ptt->upper)
> > >> +		return;
> > >> +
> > >> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> > >> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> > >> +
> > >> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> > >> +				&hisi_ptt->filter_update_lock))
> > >> +		schedule_delayed_work(&hisi_ptt->work, 0);
> > >> +	else
> > >> +		pci_warn(hisi_ptt->pdev,
> > >> +			 "filter update fifo overflow for target %s\n",
> > >> +			 pci_name(info->pdev));
> > >> +}
> > >> +
> > >> +/*
> > >> + * A PCI bus notifier is used here for dynamically updating the filter
> > >> + * list.
> > >> + */
> > >> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> > >> +				  void *data)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> > >> +	struct hisi_ptt_filter_update_info info;
> > >> +	struct device *dev = data;
> > >> +	struct pci_dev *pdev = to_pci_dev(dev);
> > >> +
> > >> +	info.pdev = pdev;
> > >> +
> > >> +	switch (action) {
> > >> +	case BUS_NOTIFY_ADD_DEVICE:
> > >> +		info.is_add = true;
> > >> +		break;
> > >> +	case BUS_NOTIFY_DEL_DEVICE:
> > >> +		info.is_add = false;
> > >> +		break;
> > >> +	default:
> > >> +		return 0;
> > >> +	}
> > >> +
> > >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> > >> +{
> > >> +	struct hisi_ptt_filter_update_info info = {
> > >> +		.pdev = pdev,
> > >> +		.is_add = true,
> > >> +	};
> > >> +	struct hisi_ptt *hisi_ptt = data;
> > >> +
> > >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> > >> +
> > >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> > >> +		list_del(&filter->list);
> > >> +		kfree(filter);
> > >> +	}
> > >> +
> > >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> > >> +		list_del(&filter->list);
> > >> +		kfree(filter);
> > >> +	}
> > >> +}
> > >> +
> > >> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	struct pci_bus *bus;
> > >> +	u32 reg;
> > >> +
> > >> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> > >> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> > >> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> > >> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> > >> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> > >> +
> > >> +	/*
> > >> +	 * The device range register provides the information about the
> > >> +	 * root ports which the RCiEP can control and trace. The RCiEP
> > >> +	 * and the root ports it support are on the same PCIe core, with
> > >> +	 * same domain number but maybe different bus number. The device
> > >> +	 * range register will tell us which root ports we can support,
> > >> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> > >> +	 * while Bit[15:0] indicates the lower.
> > >> +	 */
> > >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> > >> +	hisi_ptt->upper = reg >> 16;
> > >> +	hisi_ptt->lower = reg & 0xffff;
> > >> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> > >> +
> > >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> > >> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> > >> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> > >> +
> > >> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> > >> +	if (bus)
> > >> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> > > 
> > > What happens if bus is NULL?  Should this be reported to the caller and cause
> > > hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> > > 
> > 
> > It's ok if the list is NULL in the probe process,
> > as the device maybe hotplugged after the PTT driver probe, in which case
> > we can detect the event and update the list as we register a bus notifier.
> > 
> 
> Please add the above as a comment in the code.
> 
> 
> > >> +
> > >> +	/* Initialize trace controls */
> > >> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> > >> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> > >> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> > >> +}
> > >> +
> > >> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> > >> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> > >> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> > >> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> > >> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> > >> +
> > >> +static ssize_t available_filters_show(struct device *dev,
> > >> +				      struct device_attribute *attr,
> > >> +				      char *buf)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	int pos = 0;
> > >> +
> > >> +	if (list_empty(&hisi_ptt->port_filters))
> > >> +		return sysfs_emit(buf, "#### No available filter ####\n");
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> > >> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> > >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> > >> +				     pci_name(filter->pdev),
> > >> +				     hisi_ptt_get_filter_val(filter->pdev) |
> > >> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> > >> +
> > >> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> > >> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> > >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> > >> +				     pci_name(filter->pdev),
> > >> +				     hisi_ptt_get_filter_val(filter->pdev));
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return pos;
> > >> +}
> > >> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> > >> +
> > >> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> > >> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> > >> +PMU_FORMAT_ATTR(type,		"config:24-31");
> > >> +PMU_FORMAT_ATTR(format,		"config:32-35");
> > >> +
> > >> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> > >> +	&format_attr_filter.attr,
> > >> +	&format_attr_direction.attr,
> > >> +	&format_attr_type.attr,
> > >> +	&format_attr_format.attr,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +static struct attribute_group hisi_ptt_pmu_format_group = {
> > >> +	.name = "format",
> > >> +	.attrs = hisi_ptt_pmu_format_attrs,
> > >> +};
> > >> +
> > >> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> > >> +	&dev_attr_available_filters.attr,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> > >> +	.attrs = hisi_ptt_pmu_filter_attrs,
> > >> +};
> > >> +
> > >> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> > >> +	&hisi_ptt_pmu_format_group,
> > >> +	&hisi_ptt_pmu_filter_group,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +/*
> > >> + * The supported value of the direction parameter. See hisi_ptt.rst
> > >> + * documentation for more details.
> > >> + */
> > >> +static u32 hisi_ptt_trace_available_direction[] = {
> > >> +	0,
> > >> +	1,
> > >> +	2,
> > >> +	3,
> > >> +};
> > >> +
> > >> +/* Different types can be set simultaneously */
> > >> +static u32 hisi_ptt_trace_available_type[] = {
> > >> +	1,	/* posted_request */
> > >> +	2,	/* non-posted_request */
> > >> +	4,	/* completion */
> > >> +};
> > >> +
> > >> +static u32 hisi_ptt_trace_availble_format[] = {
> > >> +	0,	/* 4DW */
> > >> +	1,	/* 8DW */
> > >> +};
> > >> +
> > >> +/*
> > >> + * Check whether the config is valid or not. Some configs are multi-selectable
> > >> + * and can be set simultaneously, while some are single selectable (onehot).
> > >> + * Use this function to check the non-onehot configs while
> > >> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> > >> + */
> > >> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> > >> +{
> > >> +	int i;
> > >> +
> > >> +	/*
> > >> +	 * The non-onehot configs cannot be 0. Walk the available
> > >> +	 * list and clear the valid bits of the config. If there
> > >> +	 * is any resident bit after the walk then the config is
> > >> +	 * invalid.
> > >> +	 */
> > >> +	for (i = 0; i < list_size; i++)
> > >> +		val &= ~available_list[i];
> > >> +
> > >> +	return val ? -EINVAL : 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> > >> +{
> > >> +	int i, ret = -EINVAL;
> > >> +
> > >> +	for (i = 0; i < list_size; i++)
> > >> +		if (val == available_list[i]) {
> > >> +			ret = 0;
> > >> +			break;
> > >> +		}
> > >> +
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> > >> +{
> > >> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	int ret = -EINVAL;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> > >> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> > >> +
> > >> +	/*
> > >> +	 * Port filters are defined as bit mask. For port filters, check
> > >> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> > >> +	 * and whether it's empty or not, otherwise user has specified
> > >> +	 * some unsupported root ports.
> > >> +	 *
> > >> +	 * For Requester ID filters, walk the available filter list to see
> > >> +	 * whether we have one matched.
> > >> +	 */
> > >> +	if (hisi_ptt->trace_ctrl.is_port &&
> > >> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> > >> +		ret = 0;
> > >> +	} else {
> > >> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> > >> +			if (filter->val == val) {
> > >> +				ret = 0;
> > >> +				break;
> > >> +			}
> > >> +	}
> > >> +
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.filter = val;
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	int ret;
> > >> +	u32 val;
> > >> +
> > >> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> > >> +		return -ENOENT;
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +
> > >> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> > >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> > >> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->direction = val;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> > >> +
> > >> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> > >> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->type = val;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> > >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> > >> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->format = val;
> > >> +
> > >> +out:
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> > >> +				    int nr_pages, bool overwrite)
> > >> +{
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +	struct page **pagelist;
> > >> +	int i;
> > >> +
> > >> +	if (overwrite) {
> > >> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	/* If the pages size less than buflets, we cannot start trace */
> > >> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> > >> +		return NULL;
> > >> +
> > >> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> > >> +	if (!buf)
> > >> +		return NULL;
> > >> +
> > >> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> > >> +	if (!pagelist) {
> > >> +		kfree(buf);
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	for (i = 0; i < nr_pages; i++)
> > >> +		pagelist[i] = virt_to_page(pages[i]);
> > >> +
> > >> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> > >> +	if (!buf->base) {
> > >> +		kfree(pagelist);
> > >> +		kfree(buf);
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	buf->nr_pages = nr_pages;
> > >> +	buf->length = nr_pages * PAGE_SIZE;
> > >> +	buf->pos = 0;
> > >> +
> > >> +	kfree(pagelist);
> > >> +	return buf;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_free_aux(void *aux)
> > >> +{
> > >> +	struct hisi_ptt_pmu_buf *buf = aux;
> > >> +
> > >> +	vunmap(buf->base);
> > >> +	kfree(buf);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +	int cpu = event->cpu;
> > >> +	int ret;
> > >> +
> > >> +	hwc->state = 0;
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> > >> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	if (cpu == -1)
> > >> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> > >> +
> > >> +	/*
> > >> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> > >> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> > >> +	 * core in event_function_local().
> > >> +	 */
> > >> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> > >> +				 cpumask_of(cpu)));
> > >> +
> > >> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	buf = perf_aux_output_begin(handle, event);
> > >> +	if (!buf) {
> > >> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	buf->pos = handle->head % buf->length;
> > >> +
> > >> +	ret = hisi_ptt_trace_start(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> > >> +		perf_aux_output_end(handle, 0);
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return;
> > >> +stop:
> > >> +	event->hw.state |= PERF_HES_STOPPED;
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +
> > >> +	if (hwc->state & PERF_HES_STOPPED)
> > >> +		return;
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> > >> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> > >> +	}
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +
> > >> +	hwc->state |= PERF_HES_STOPPED;
> > >> +	perf_event_update_userpage(event);
> > >> +	hwc->state |= PERF_HES_UPTODATE;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +	int cpu = event->cpu;
> > >> +
> > >> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> > >> +		return 0;
> > >> +
> > >> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> > >> +
> > >> +	if (flags & PERF_EF_START) {
> > >> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> > >> +		if (hwc->state & PERF_HES_STOPPED)
> > >> +			return -EINVAL;
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> > >> +{
> > >> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_unregister_pmu(void *priv)
> > >> +{
> > >> +	perf_pmu_unregister(priv);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	char *pmu_name;
> > >> +	int ret;
> > >> +
> > >> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> > >> +		.module		= THIS_MODULE,
> > >> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> > >> +		.task_ctx_nr	= perf_sw_context,
> > >> +		.attr_groups	= hisi_ptt_pmu_groups,
> > >> +		.event_init	= hisi_ptt_pmu_event_init,
> > >> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> > >> +		.free_aux	= hisi_ptt_pmu_free_aux,
> > >> +		.start		= hisi_ptt_pmu_start,
> > >> +		.stop		= hisi_ptt_pmu_stop,
> > >> +		.add		= hisi_ptt_pmu_add,
> > >> +		.del		= hisi_ptt_pmu_del,
> > >> +	};
> > >> +
> > >> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> > >> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> > >> +	if (!pmu_name)
> > >> +		return -ENOMEM;
> > >> +
> > >> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> > >> +					hisi_ptt_unregister_pmu,
> > >> +					&hisi_ptt->hisi_ptt_pmu);
> > >> +}
> > >> +
> > >> +/*
> > >> + * Get RMR address if provided by the firmware.
> > >> + * Return 0 if the IOMMU doesn't present or the policy of the
> > >> + * IOMMU domain is passthrough or we get a usable RMR region.
> > >> + * Otherwise a negative value is returned.
> > >> + */
> > >> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	struct iommu_domain *iommu_domain;
> > >> +	struct iommu_resv_region *region;
> > >> +	LIST_HEAD(list);
> > >> +
> > >> +	/*
> > >> +	 * Use direct DMA if IOMMU does not present or the policy of the
> > >> +	 * IOMMU domain is passthrough.
> > >> +	 */
> > >> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> > >> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> > >> +		return 0;
> > >> +
> > >> +	iommu_get_resv_regions(&pdev->dev, &list);
> > >> +	list_for_each_entry(region, &list, list)
> > >> +		if (region->type == IOMMU_RESV_DIRECT &&
> > >> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> > >> +			hisi_ptt->trace_ctrl.has_rmr = true;
> > >> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> > >> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> > >> +			break;
> > >> +		}
> > >> +
> > >> +	iommu_put_resv_regions(&pdev->dev, &list);
> > >> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_probe(struct pci_dev *pdev,
> > >> +			  const struct pci_device_id *id)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt;
> > >> +	int ret;
> > >> +
> > >> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> > >> +	if (!hisi_ptt)
> > >> +		return -ENOMEM;
> > >> +
> > >> +	mutex_init(&hisi_ptt->mutex);
> > >> +	hisi_ptt->pdev = pdev;
> > >> +
> > >> +	/*
> > >> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> > >> +	 * so directly reference to the pci name string.
> > >> +	 */
> > >> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> > >> +	pci_set_drvdata(pdev, hisi_ptt);
> > >> +
> > >> +	ret = pcim_enable_device(pdev);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> > >> +
> > >> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +	pci_set_master(pdev);
> > >> +
> > >> +	ret = hisi_ptt_register_irq(hisi_ptt);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	hisi_ptt_init_ctrls(hisi_ptt);
> > >> +
> > >> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> > >> +		return ret;
> > >> +	}
> > > 
> > > If this fails the pci_free_irq_vectors() is not called.
> > > 
> > 
> > It'll be called as we've registered a devm callback when
> > allocating the irq vector. See hisi_ptt_register_irq().
> 
> Ah yes - thanks for pointing that out.
> 
> > 
> > > I am out of time for today - I will continue tomorrow.
> > > 
> > 
> > Sure. Thanks!
> > Yicong
> > 
> > > Thanks,
> > > Mathieu
> > > 
> > >> +
> > >> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	/* Register the bus notifier for dynamically updating the filter list */
> > >> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> > >> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> > >> +	if (ret)
> > >> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +void hisi_ptt_remove(struct pci_dev *pdev)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> > >> +
> > >> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> > >> +
> > >> +	/* Cancel any work that has been queued */
> > >> +	cancel_delayed_work_sync(&hisi_ptt->work);
> > >> +
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +
> > >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> > >> +	hisi_ptt_release_filters(hisi_ptt);
> > >> +}
> > >> +
> > >> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> > >> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> > >> +	{ }
> > >> +};
> > >> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> > >> +
> > >> +static struct pci_driver hisi_ptt_driver = {
> > >> +	.name = "hisi_ptt",
> > >> +	.id_table = hisi_ptt_id_tbl,
> > >> +	.probe = hisi_ptt_probe,
> > >> +	.remove = hisi_ptt_remove,
> > >> +};
> > >> +module_pci_driver(hisi_ptt_driver);
> > >> +
> > >> +MODULE_LICENSE("GPL v2");
> > >> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> > >> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> > >> -- 
> > >> 2.33.0
> > >>
> > > .
> > > 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

[...]

> 
> > >> +
> > >> +#define HISI_PTT_TRACE_DMA_IRQ			0
> > >> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> > >> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> > >> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> > >> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> > >> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> > >> +
> > >> +/* Delay time for filter updating work */
> > >> +#define HISI_PTT_WORK_DELAY_MS		100UL
> > >> +/* Wait time for DMA hardware to reset */
> > >> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> > >> +/* Poll timeout and interval for waiting hardware work to finish */
> > >> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> > >> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> > >> +
> > >> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> > >> +
> > >> +enum hisi_ptt_trace_status {
> > >> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> > >> +	HISI_PTT_TRACE_STATUS_ON,
> > >> +};
> > >> +
> > >> +struct hisi_ptt_dma_buflet {
> > > 
> > > I'm not sure what a "buflet" is...  Probably best to just call this
> > > hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> > > know due to the lack of proper structure documentation.  Please have a look at
> > > how "coresight_device" is documented.  The same applies to the rest of the
> > > structures declared below.
> > > 
> > 
> > It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
> > (we call it buflet) of 4MiB ...".
> 
> I just noticed the documentation in patch 5 - I will read it before continuing
> with this set.
>

As promised I read the documentation and it didn't do much to alliviate my
concern about the terminology.  If a "buflet" is a DMA buffer than simply call
it a DMA buffer, regardless of the size.  

The size of this patchset makes it hard enough to review as it is... On top
of things reviewers have to remember that PTT means "PCIe tune and trace
device", tune means "PCIe link's events" and trace means "TLP headers".

But we also have to keep in mind that a buflet is really just a DMA buffer...

I will not continue reviewing this set.  Please fix the terminology and spin
another revision.  Doing so I suggest you trim down the submission to the
absolute minimum feature set this driver should provide - enhancements can be
made later when a foundation has been laid out.

Robin has pointed out what seems to be serious problems with the
implemenation.  That will also have to be sorted out for the next iteration.

Regards,
Mathieu


> > 
> > The total DMA buffer of PTT device is divided into 4 parts, and each part is called
> > a 'buflet', which means a small buffer (there exists other words with similiar
> > format: droplet, wavelet, ...).
> > 
> > The device is designed like this to make sure we won't lose any data when we have to change
> > the buffer address, the device can continue to write to the next buflet and don't
> > neet to pause.
> > 
> > >> +	struct list_head list;
> > >> +	unsigned int size;
> > >> +	dma_addr_t dma;
> > >> +
> > >> +	/*
> > >> +	 * The address of the buflet holding the trace data.
> > >> +	 * See Documentation/trace/hisi-ptt.rst for the details
> > > 
> > > As of this writing, hisi-ptt.rst doesn't exist.
> > > 
> > 
> > The comment intends to direct the user to the documentation if the user
> > want to know details about the data format. The data format is decribed
> > in the section 4 of the doc.
> > 
> > >> +	 * of the data format.
> > >> +	 */
> > >> +	void *addr;
> > >> +	int index;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_trace_ctrl {
> > >> +	enum hisi_ptt_trace_status status;
> > >> +	struct perf_output_handle handle;
> > >> +	struct list_head trace_buf;
> > >> +	/*
> > >> +	 * The index of the buffer which trace data
> > >> +	 * currently is writing to.
> > >> +	 */
> > >> +	u32 buf_index;
> > >> +
> > >> +	int default_cpu;
> > >> +	u32 buflet_size;
> > >> +	bool is_port;
> > >> +	u32 format:1;		/* Format of the traced TLP headers */
> > >> +	u32 type:4;		/* Type of the TLP headers to trace */
> > >> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> > >> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> > >> +
> > >> +	phys_addr_t rmr_addr;
> > >> +	size_t rmr_length;
> > >> +	bool has_rmr;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_filter_desc {
> > >> +	struct list_head list;
> > >> +	struct pci_dev *pdev;
> > >> +	u16 val;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_pmu_buf {
> > >> +	size_t length;
> > >> +	int nr_pages;
> > >> +	void *base;
> > >> +	long pos;
> > >> +};
> > >> +
> > >> +/* Structure containing the information for filter updating */
> > >> +struct hisi_ptt_filter_update_info {
> > >> +	struct pci_dev *pdev;
> > >> +	u32 port_devid;
> > >> +	bool is_port;
> > >> +	bool is_add;
> > >> +	u16 val;
> > >> +};
> > >> +
> > >> +struct hisi_ptt {
> > >> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> > >> +	struct notifier_block hisi_ptt_nb;
> > >> +	struct pmu hisi_ptt_pmu;
> > >> +	void __iomem *iobase;
> > >> +	struct pci_dev *pdev;
> > >> +	/*
> > >> +	 * Use the mutex to protect the filter list and
> > >> +	 * serialize the perf process.
> > >> +	 */
> > >> +	struct mutex mutex;
> > >> +	const char *name;
> > >> +	u16 core_id;
> > >> +	u16 sicl_id;
> > >> +	/* PCI device range managed by the PTT device */
> > >> +	u32 upper;
> > >> +	u32 lower;
> > >> +	u8 busnr;
> > >> +
> > >> +	/*
> > >> +	 * The trace TLP headers can either be filtered by certain
> > >> +	 * root port, or by the requester ID. Organize the filters
> > >> +	 * by @port_filters and @req_filters here. The mask of all
> > >> +	 * the valid ports is also cached for doing sanity check
> > >> +	 * of user input.
> > >> +	 */
> > >> +	struct list_head port_filters;
> > >> +	struct list_head req_filters;
> > >> +	u16 port_mask;
> > >> +
> > >> +	/*
> > >> +	 * We use a delayed work here to avoid indefinitely waiting for
> > >> +	 * the hisi_ptt->mutex which protecting the filter list. The
> > >> +	 * work will be delayed only if the mutex can not be held,
> > >> +	 * otherwise no delay will be applied.
> > >> +	 */
> > >> +	struct delayed_work work;
> > >> +	spinlock_t filter_update_lock;
> > >> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> > >> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> > >> +};
> > > 
> > > It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> > > 
> > 
> > I'm not sure whether it is common or permitted to provide a header which is only used by
> > this driver with a single file.
> 
> I've seen it done many times.
> 
> > 
> > >> +
> > >> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> > >> +{
> > >> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> > >> +}
> > > 
> > > I'm not sure there is any value to this inline function.  I suggest to simply
> > > call container_of() whenever it is needed.
> > > 
> > 
> > ok.
> > 
> > >> +
> > >> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> > >> +{
> > >> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> > >> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> > >> +
> > >> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	u32 val;
> > >> +
> > >> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> > >> +				  val & HISI_PTT_TRACE_IDLE,
> > >> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> > >> +				  HISI_PTT_WAIT_TIMEOUT_US);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct device *dev = &hisi_ptt->pdev->dev;
> > >> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> > >> +
> > >> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> > >> +		list_del(&buflet->list);
> > >> +
> > >> +		if (ctrl->has_rmr)
> > >> +			memunmap(buflet->addr);
> > >> +		else
> > >> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> > >> +					  buflet->dma);
> > >> +
> > >> +		kfree(buflet);
> > >> +	}
> > >> +}
> > >> +
> > >> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct device *dev = &hisi_ptt->pdev->dev;
> > >> +	struct hisi_ptt_dma_buflet *buflet;
> > >> +	int i, ret;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.buf_index = 0;
> > >> +
> > >> +	/* Make sure the trace buffer is empty before allocating */
> > >> +	if (!list_empty(&ctrl->trace_buf)) {
> > >> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> > >> +			memset(buflet->addr, 0, buflet->size);
> > >> +		return 0;
> > >> +	}
> > >> +
> > >> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> > >> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> > >> +		if (!buflet) {
> > >> +			ret = -ENOMEM;
> > >> +			goto err;
> > >> +		}
> > >> +
> > >> +		if (ctrl->has_rmr) {
> > >> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> > >> +
> > >> +			buflet->dma = base;
> > >> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> > >> +		} else {
> > >> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> > >> +							  &buflet->dma, GFP_KERNEL);
> > >> +		}
> > >> +
> > >> +		if (!buflet->addr) {
> > >> +			kfree(buflet);
> > >> +			ret = -ENOMEM;
> > >> +			goto err;
> > >> +		}
> > >> +
> > >> +		memset(buflet->addr, 0, buflet->size);
> > >> +
> > >> +		buflet->index = i;
> > >> +		buflet->size = ctrl->buflet_size;
> > >> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +err:
> > >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct hisi_ptt_dma_buflet *cur;
> > >> +	u32 val;
> > >> +
> > >> +	/* Check device idle before start trace */
> > >> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> > >> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> > >> +		return -EBUSY;
> > >> +	}
> > >> +
> > >> +	/* Reset the DMA before start tracing */
> > >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	val |= HISI_PTT_TRACE_CTRL_RST;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	/*
> > >> +	 * We'll be in the perf context where preemption is disabled,
> > >> +	 * so use busy loop here.
> > >> +	 */
> > >> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> > >> +
> > >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	/* clear the interrupt status */
> > >> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> > >> +
> > >> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> > >> +		writel(lower_32_bits(cur->dma),
> > >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> > >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> > >> +		writel(upper_32_bits(cur->dma),
> > >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> > >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> > >> +	}
> > >> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> > >> +
> > >> +	/* set the trace control register */
> > >> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> > >> +	if (!hisi_ptt->trace_ctrl.is_port)
> > >> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> > >> +
> > >> +	val |= HISI_PTT_TRACE_CTRL_EN;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> > >> +{
> > >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> > >> +	struct perf_event *event = handle->event;
> > >> +	struct hisi_ptt_dma_buflet *cur;
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +
> > >> +	buf = perf_get_aux(handle);
> > >> +	if (!buf || !handle->size)
> > >> +		return -EINVAL;
> > >> +
> > >> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> > >> +		if (cur->index == index)
> > >> +			break;
> > >> +
> > >> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> > >> +	memset(cur->addr, 0, cur->size);
> > >> +	buf->pos += cur->size;
> > >> +
> > >> +	if (stop) {
> > >> +		perf_aux_output_end(handle, buf->pos);
> > >> +	} else if (buf->length - buf->pos < cur->size) {
> > >> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> > >> +		perf_aux_output_end(handle, buf->pos);
> > >> +
> > >> +		buf = perf_aux_output_begin(handle, event);
> > >> +		if (!buf)
> > >> +			return -EINVAL;
> > >> +
> > >> +		buf->pos = handle->head % buf->length;
> > >> +		if (buf->length - buf->pos < cur->size) {
> > >> +			perf_aux_output_end(handle, 0);
> > >> +			return -EINVAL;
> > >> +		}
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = context;
> > >> +	u32 status, buf_idx;
> > >> +
> > >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	buf_idx = ffs(status) - 1;
> > >> +
> > >> +	/* Clear the interrupt status of buflet @buf_idx */
> > >> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +
> > >> +	/*
> > >> +	 * Update the AUX buffer and cache the current buflet index,
> > >> +	 * as we need to know this and save the data when the trace
> > >> +	 * is ended out of the interrupt handler. End the trace
> > >> +	 * if the updating fails.
> > >> +	 */
> > >> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +	else
> > >> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> > >> +
> > >> +	return IRQ_HANDLED;
> > >> +}
> > >> +
> > >> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = context;
> > >> +	u32 status;
> > >> +
> > >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> > >> +		return IRQ_NONE;
> > >> +
> > >> +	return IRQ_WAKE_THREAD;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_irq_free_vectors(void *pdev)
> > >> +{
> > >> +	pci_free_irq_vectors(pdev);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	int ret;
> > >> +
> > >> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> > >> +	if (ret < 0) {
> > >> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> > >> +	if (ret < 0)
> > >> +		return ret;
> > >> +
> > >> +	ret = devm_request_threaded_irq(&pdev->dev,
> > >> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> > >> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> > >> +					"hisi-ptt", hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> > >> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_update_filters(struct work_struct *work)
> > >> +{
> > >> +	struct delayed_work *delayed_work = to_delayed_work(work);
> > >> +	struct hisi_ptt_filter_update_info info;
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	struct list_head *target_list;
> > >> +	struct hisi_ptt *hisi_ptt;
> > >> +
> > >> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> > >> +
> > >> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> > >> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> > >> +		return;
> > >> +	}
> > >> +
> > >> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> > >> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> > >> +			      &hisi_ptt->req_filters;
> > >> +
> > >> +		if (info.is_add) {
> > >> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> > >> +			if (!filter)
> > >> +				continue;
> > >> +
> > >> +			filter->pdev = info.pdev;
> > >> +			filter->val = info.val;
> > >> +
> > >> +			list_add_tail(&filter->list, target_list);
> > >> +		} else {
> > >> +			list_for_each_entry(filter, target_list, list)
> > >> +				if (filter->val == info.val) {
> > >> +					list_del(&filter->list);
> > >> +					kfree(filter);
> > >> +					break;
> > >> +				}
> > >> +		}
> > >> +
> > >> +		/* Update the available port mask */
> > >> +		if (!info.is_port)
> > >> +			continue;
> > >> +
> > >> +		if (info.is_add)
> > >> +			hisi_ptt->port_mask |= info.val;
> > >> +		else
> > >> +			hisi_ptt->port_mask &= ~info.val;
> > >> +	}
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> > >> +				    struct hisi_ptt_filter_update_info *info)
> > >> +{
> > >> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> > >> +
> > >> +	if (!root_port)
> > >> +		return;
> > >> +
> > >> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> > >> +	if (info->port_devid < hisi_ptt->lower ||
> > >> +	    info->port_devid > hisi_ptt->upper)
> > >> +		return;
> > >> +
> > >> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> > >> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> > >> +
> > >> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> > >> +				&hisi_ptt->filter_update_lock))
> > >> +		schedule_delayed_work(&hisi_ptt->work, 0);
> > >> +	else
> > >> +		pci_warn(hisi_ptt->pdev,
> > >> +			 "filter update fifo overflow for target %s\n",
> > >> +			 pci_name(info->pdev));
> > >> +}
> > >> +
> > >> +/*
> > >> + * A PCI bus notifier is used here for dynamically updating the filter
> > >> + * list.
> > >> + */
> > >> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> > >> +				  void *data)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> > >> +	struct hisi_ptt_filter_update_info info;
> > >> +	struct device *dev = data;
> > >> +	struct pci_dev *pdev = to_pci_dev(dev);
> > >> +
> > >> +	info.pdev = pdev;
> > >> +
> > >> +	switch (action) {
> > >> +	case BUS_NOTIFY_ADD_DEVICE:
> > >> +		info.is_add = true;
> > >> +		break;
> > >> +	case BUS_NOTIFY_DEL_DEVICE:
> > >> +		info.is_add = false;
> > >> +		break;
> > >> +	default:
> > >> +		return 0;
> > >> +	}
> > >> +
> > >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> > >> +{
> > >> +	struct hisi_ptt_filter_update_info info = {
> > >> +		.pdev = pdev,
> > >> +		.is_add = true,
> > >> +	};
> > >> +	struct hisi_ptt *hisi_ptt = data;
> > >> +
> > >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> > >> +
> > >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> > >> +		list_del(&filter->list);
> > >> +		kfree(filter);
> > >> +	}
> > >> +
> > >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> > >> +		list_del(&filter->list);
> > >> +		kfree(filter);
> > >> +	}
> > >> +}
> > >> +
> > >> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	struct pci_bus *bus;
> > >> +	u32 reg;
> > >> +
> > >> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> > >> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> > >> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> > >> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> > >> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> > >> +
> > >> +	/*
> > >> +	 * The device range register provides the information about the
> > >> +	 * root ports which the RCiEP can control and trace. The RCiEP
> > >> +	 * and the root ports it support are on the same PCIe core, with
> > >> +	 * same domain number but maybe different bus number. The device
> > >> +	 * range register will tell us which root ports we can support,
> > >> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> > >> +	 * while Bit[15:0] indicates the lower.
> > >> +	 */
> > >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> > >> +	hisi_ptt->upper = reg >> 16;
> > >> +	hisi_ptt->lower = reg & 0xffff;
> > >> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> > >> +
> > >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> > >> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> > >> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> > >> +
> > >> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> > >> +	if (bus)
> > >> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> > > 
> > > What happens if bus is NULL?  Should this be reported to the caller and cause
> > > hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> > > 
> > 
> > It's ok if the list is NULL in the probe process,
> > as the device maybe hotplugged after the PTT driver probe, in which case
> > we can detect the event and update the list as we register a bus notifier.
> > 
> 
> Please add the above as a comment in the code.
> 
> 
> > >> +
> > >> +	/* Initialize trace controls */
> > >> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> > >> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> > >> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> > >> +}
> > >> +
> > >> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> > >> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> > >> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> > >> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> > >> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> > >> +
> > >> +static ssize_t available_filters_show(struct device *dev,
> > >> +				      struct device_attribute *attr,
> > >> +				      char *buf)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	int pos = 0;
> > >> +
> > >> +	if (list_empty(&hisi_ptt->port_filters))
> > >> +		return sysfs_emit(buf, "#### No available filter ####\n");
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> > >> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> > >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> > >> +				     pci_name(filter->pdev),
> > >> +				     hisi_ptt_get_filter_val(filter->pdev) |
> > >> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> > >> +
> > >> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> > >> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> > >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> > >> +				     pci_name(filter->pdev),
> > >> +				     hisi_ptt_get_filter_val(filter->pdev));
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return pos;
> > >> +}
> > >> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> > >> +
> > >> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> > >> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> > >> +PMU_FORMAT_ATTR(type,		"config:24-31");
> > >> +PMU_FORMAT_ATTR(format,		"config:32-35");
> > >> +
> > >> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> > >> +	&format_attr_filter.attr,
> > >> +	&format_attr_direction.attr,
> > >> +	&format_attr_type.attr,
> > >> +	&format_attr_format.attr,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +static struct attribute_group hisi_ptt_pmu_format_group = {
> > >> +	.name = "format",
> > >> +	.attrs = hisi_ptt_pmu_format_attrs,
> > >> +};
> > >> +
> > >> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> > >> +	&dev_attr_available_filters.attr,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> > >> +	.attrs = hisi_ptt_pmu_filter_attrs,
> > >> +};
> > >> +
> > >> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> > >> +	&hisi_ptt_pmu_format_group,
> > >> +	&hisi_ptt_pmu_filter_group,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +/*
> > >> + * The supported value of the direction parameter. See hisi_ptt.rst
> > >> + * documentation for more details.
> > >> + */
> > >> +static u32 hisi_ptt_trace_available_direction[] = {
> > >> +	0,
> > >> +	1,
> > >> +	2,
> > >> +	3,
> > >> +};
> > >> +
> > >> +/* Different types can be set simultaneously */
> > >> +static u32 hisi_ptt_trace_available_type[] = {
> > >> +	1,	/* posted_request */
> > >> +	2,	/* non-posted_request */
> > >> +	4,	/* completion */
> > >> +};
> > >> +
> > >> +static u32 hisi_ptt_trace_availble_format[] = {
> > >> +	0,	/* 4DW */
> > >> +	1,	/* 8DW */
> > >> +};
> > >> +
> > >> +/*
> > >> + * Check whether the config is valid or not. Some configs are multi-selectable
> > >> + * and can be set simultaneously, while some are single selectable (onehot).
> > >> + * Use this function to check the non-onehot configs while
> > >> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> > >> + */
> > >> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> > >> +{
> > >> +	int i;
> > >> +
> > >> +	/*
> > >> +	 * The non-onehot configs cannot be 0. Walk the available
> > >> +	 * list and clear the valid bits of the config. If there
> > >> +	 * is any resident bit after the walk then the config is
> > >> +	 * invalid.
> > >> +	 */
> > >> +	for (i = 0; i < list_size; i++)
> > >> +		val &= ~available_list[i];
> > >> +
> > >> +	return val ? -EINVAL : 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> > >> +{
> > >> +	int i, ret = -EINVAL;
> > >> +
> > >> +	for (i = 0; i < list_size; i++)
> > >> +		if (val == available_list[i]) {
> > >> +			ret = 0;
> > >> +			break;
> > >> +		}
> > >> +
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> > >> +{
> > >> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	int ret = -EINVAL;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> > >> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> > >> +
> > >> +	/*
> > >> +	 * Port filters are defined as bit mask. For port filters, check
> > >> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> > >> +	 * and whether it's empty or not, otherwise user has specified
> > >> +	 * some unsupported root ports.
> > >> +	 *
> > >> +	 * For Requester ID filters, walk the available filter list to see
> > >> +	 * whether we have one matched.
> > >> +	 */
> > >> +	if (hisi_ptt->trace_ctrl.is_port &&
> > >> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> > >> +		ret = 0;
> > >> +	} else {
> > >> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> > >> +			if (filter->val == val) {
> > >> +				ret = 0;
> > >> +				break;
> > >> +			}
> > >> +	}
> > >> +
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.filter = val;
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	int ret;
> > >> +	u32 val;
> > >> +
> > >> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> > >> +		return -ENOENT;
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +
> > >> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> > >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> > >> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->direction = val;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> > >> +
> > >> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> > >> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->type = val;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> > >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> > >> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->format = val;
> > >> +
> > >> +out:
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> > >> +				    int nr_pages, bool overwrite)
> > >> +{
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +	struct page **pagelist;
> > >> +	int i;
> > >> +
> > >> +	if (overwrite) {
> > >> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	/* If the pages size less than buflets, we cannot start trace */
> > >> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> > >> +		return NULL;
> > >> +
> > >> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> > >> +	if (!buf)
> > >> +		return NULL;
> > >> +
> > >> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> > >> +	if (!pagelist) {
> > >> +		kfree(buf);
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	for (i = 0; i < nr_pages; i++)
> > >> +		pagelist[i] = virt_to_page(pages[i]);
> > >> +
> > >> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> > >> +	if (!buf->base) {
> > >> +		kfree(pagelist);
> > >> +		kfree(buf);
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	buf->nr_pages = nr_pages;
> > >> +	buf->length = nr_pages * PAGE_SIZE;
> > >> +	buf->pos = 0;
> > >> +
> > >> +	kfree(pagelist);
> > >> +	return buf;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_free_aux(void *aux)
> > >> +{
> > >> +	struct hisi_ptt_pmu_buf *buf = aux;
> > >> +
> > >> +	vunmap(buf->base);
> > >> +	kfree(buf);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +	int cpu = event->cpu;
> > >> +	int ret;
> > >> +
> > >> +	hwc->state = 0;
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> > >> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	if (cpu == -1)
> > >> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> > >> +
> > >> +	/*
> > >> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> > >> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> > >> +	 * core in event_function_local().
> > >> +	 */
> > >> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> > >> +				 cpumask_of(cpu)));
> > >> +
> > >> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	buf = perf_aux_output_begin(handle, event);
> > >> +	if (!buf) {
> > >> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	buf->pos = handle->head % buf->length;
> > >> +
> > >> +	ret = hisi_ptt_trace_start(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> > >> +		perf_aux_output_end(handle, 0);
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return;
> > >> +stop:
> > >> +	event->hw.state |= PERF_HES_STOPPED;
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +
> > >> +	if (hwc->state & PERF_HES_STOPPED)
> > >> +		return;
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> > >> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> > >> +	}
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +
> > >> +	hwc->state |= PERF_HES_STOPPED;
> > >> +	perf_event_update_userpage(event);
> > >> +	hwc->state |= PERF_HES_UPTODATE;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +	int cpu = event->cpu;
> > >> +
> > >> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> > >> +		return 0;
> > >> +
> > >> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> > >> +
> > >> +	if (flags & PERF_EF_START) {
> > >> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> > >> +		if (hwc->state & PERF_HES_STOPPED)
> > >> +			return -EINVAL;
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> > >> +{
> > >> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_unregister_pmu(void *priv)
> > >> +{
> > >> +	perf_pmu_unregister(priv);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	char *pmu_name;
> > >> +	int ret;
> > >> +
> > >> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> > >> +		.module		= THIS_MODULE,
> > >> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> > >> +		.task_ctx_nr	= perf_sw_context,
> > >> +		.attr_groups	= hisi_ptt_pmu_groups,
> > >> +		.event_init	= hisi_ptt_pmu_event_init,
> > >> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> > >> +		.free_aux	= hisi_ptt_pmu_free_aux,
> > >> +		.start		= hisi_ptt_pmu_start,
> > >> +		.stop		= hisi_ptt_pmu_stop,
> > >> +		.add		= hisi_ptt_pmu_add,
> > >> +		.del		= hisi_ptt_pmu_del,
> > >> +	};
> > >> +
> > >> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> > >> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> > >> +	if (!pmu_name)
> > >> +		return -ENOMEM;
> > >> +
> > >> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> > >> +					hisi_ptt_unregister_pmu,
> > >> +					&hisi_ptt->hisi_ptt_pmu);
> > >> +}
> > >> +
> > >> +/*
> > >> + * Get RMR address if provided by the firmware.
> > >> + * Return 0 if the IOMMU doesn't present or the policy of the
> > >> + * IOMMU domain is passthrough or we get a usable RMR region.
> > >> + * Otherwise a negative value is returned.
> > >> + */
> > >> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	struct iommu_domain *iommu_domain;
> > >> +	struct iommu_resv_region *region;
> > >> +	LIST_HEAD(list);
> > >> +
> > >> +	/*
> > >> +	 * Use direct DMA if IOMMU does not present or the policy of the
> > >> +	 * IOMMU domain is passthrough.
> > >> +	 */
> > >> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> > >> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> > >> +		return 0;
> > >> +
> > >> +	iommu_get_resv_regions(&pdev->dev, &list);
> > >> +	list_for_each_entry(region, &list, list)
> > >> +		if (region->type == IOMMU_RESV_DIRECT &&
> > >> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> > >> +			hisi_ptt->trace_ctrl.has_rmr = true;
> > >> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> > >> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> > >> +			break;
> > >> +		}
> > >> +
> > >> +	iommu_put_resv_regions(&pdev->dev, &list);
> > >> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_probe(struct pci_dev *pdev,
> > >> +			  const struct pci_device_id *id)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt;
> > >> +	int ret;
> > >> +
> > >> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> > >> +	if (!hisi_ptt)
> > >> +		return -ENOMEM;
> > >> +
> > >> +	mutex_init(&hisi_ptt->mutex);
> > >> +	hisi_ptt->pdev = pdev;
> > >> +
> > >> +	/*
> > >> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> > >> +	 * so directly reference to the pci name string.
> > >> +	 */
> > >> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> > >> +	pci_set_drvdata(pdev, hisi_ptt);
> > >> +
> > >> +	ret = pcim_enable_device(pdev);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> > >> +
> > >> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +	pci_set_master(pdev);
> > >> +
> > >> +	ret = hisi_ptt_register_irq(hisi_ptt);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	hisi_ptt_init_ctrls(hisi_ptt);
> > >> +
> > >> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> > >> +		return ret;
> > >> +	}
> > > 
> > > If this fails the pci_free_irq_vectors() is not called.
> > > 
> > 
> > It'll be called as we've registered a devm callback when
> > allocating the irq vector. See hisi_ptt_register_irq().
> 
> Ah yes - thanks for pointing that out.
> 
> > 
> > > I am out of time for today - I will continue tomorrow.
> > > 
> > 
> > Sure. Thanks!
> > Yicong
> > 
> > > Thanks,
> > > Mathieu
> > > 
> > >> +
> > >> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	/* Register the bus notifier for dynamically updating the filter list */
> > >> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> > >> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> > >> +	if (ret)
> > >> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +void hisi_ptt_remove(struct pci_dev *pdev)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> > >> +
> > >> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> > >> +
> > >> +	/* Cancel any work that has been queued */
> > >> +	cancel_delayed_work_sync(&hisi_ptt->work);
> > >> +
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +
> > >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> > >> +	hisi_ptt_release_filters(hisi_ptt);
> > >> +}
> > >> +
> > >> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> > >> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> > >> +	{ }
> > >> +};
> > >> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> > >> +
> > >> +static struct pci_driver hisi_ptt_driver = {
> > >> +	.name = "hisi_ptt",
> > >> +	.id_table = hisi_ptt_id_tbl,
> > >> +	.probe = hisi_ptt_probe,
> > >> +	.remove = hisi_ptt_remove,
> > >> +};
> > >> +module_pci_driver(hisi_ptt_driver);
> > >> +
> > >> +MODULE_LICENSE("GPL v2");
> > >> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> > >> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> > >> -- 
> > >> 2.33.0
> > >>
> > > .
> > > 

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Mathieu Poirier]

[...]

> 
> > >> +
> > >> +#define HISI_PTT_TRACE_DMA_IRQ			0
> > >> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
> > >> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
> > >> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
> > >> +						 HISI_PTT_TRACE_BUFLETS_CNT)
> > >> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
> > >> +
> > >> +/* Delay time for filter updating work */
> > >> +#define HISI_PTT_WORK_DELAY_MS		100UL
> > >> +/* Wait time for DMA hardware to reset */
> > >> +#define HISI_PTT_RESET_WAIT_MS		1000UL
> > >> +/* Poll timeout and interval for waiting hardware work to finish */
> > >> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
> > >> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
> > >> +
> > >> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
> > >> +
> > >> +enum hisi_ptt_trace_status {
> > >> +	HISI_PTT_TRACE_STATUS_OFF = 0,
> > >> +	HISI_PTT_TRACE_STATUS_ON,
> > >> +};
> > >> +
> > >> +struct hisi_ptt_dma_buflet {
> > > 
> > > I'm not sure what a "buflet" is...  Probably best to just call this
> > > hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
> > > know due to the lack of proper structure documentation.  Please have a look at
> > > how "coresight_device" is documented.  The same applies to the rest of the
> > > structures declared below.
> > > 
> > 
> > It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
> > (we call it buflet) of 4MiB ...".
> 
> I just noticed the documentation in patch 5 - I will read it before continuing
> with this set.
>

As promised I read the documentation and it didn't do much to alliviate my
concern about the terminology.  If a "buflet" is a DMA buffer than simply call
it a DMA buffer, regardless of the size.  

The size of this patchset makes it hard enough to review as it is... On top
of things reviewers have to remember that PTT means "PCIe tune and trace
device", tune means "PCIe link's events" and trace means "TLP headers".

But we also have to keep in mind that a buflet is really just a DMA buffer...

I will not continue reviewing this set.  Please fix the terminology and spin
another revision.  Doing so I suggest you trim down the submission to the
absolute minimum feature set this driver should provide - enhancements can be
made later when a foundation has been laid out.

Robin has pointed out what seems to be serious problems with the
implemenation.  That will also have to be sorted out for the next iteration.

Regards,
Mathieu


> > 
> > The total DMA buffer of PTT device is divided into 4 parts, and each part is called
> > a 'buflet', which means a small buffer (there exists other words with similiar
> > format: droplet, wavelet, ...).
> > 
> > The device is designed like this to make sure we won't lose any data when we have to change
> > the buffer address, the device can continue to write to the next buflet and don't
> > neet to pause.
> > 
> > >> +	struct list_head list;
> > >> +	unsigned int size;
> > >> +	dma_addr_t dma;
> > >> +
> > >> +	/*
> > >> +	 * The address of the buflet holding the trace data.
> > >> +	 * See Documentation/trace/hisi-ptt.rst for the details
> > > 
> > > As of this writing, hisi-ptt.rst doesn't exist.
> > > 
> > 
> > The comment intends to direct the user to the documentation if the user
> > want to know details about the data format. The data format is decribed
> > in the section 4 of the doc.
> > 
> > >> +	 * of the data format.
> > >> +	 */
> > >> +	void *addr;
> > >> +	int index;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_trace_ctrl {
> > >> +	enum hisi_ptt_trace_status status;
> > >> +	struct perf_output_handle handle;
> > >> +	struct list_head trace_buf;
> > >> +	/*
> > >> +	 * The index of the buffer which trace data
> > >> +	 * currently is writing to.
> > >> +	 */
> > >> +	u32 buf_index;
> > >> +
> > >> +	int default_cpu;
> > >> +	u32 buflet_size;
> > >> +	bool is_port;
> > >> +	u32 format:1;		/* Format of the traced TLP headers */
> > >> +	u32 type:4;		/* Type of the TLP headers to trace */
> > >> +	u32 direction:2;	/* Direction of the TLP headers to trace */
> > >> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
> > >> +
> > >> +	phys_addr_t rmr_addr;
> > >> +	size_t rmr_length;
> > >> +	bool has_rmr;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_filter_desc {
> > >> +	struct list_head list;
> > >> +	struct pci_dev *pdev;
> > >> +	u16 val;
> > >> +};
> > >> +
> > >> +struct hisi_ptt_pmu_buf {
> > >> +	size_t length;
> > >> +	int nr_pages;
> > >> +	void *base;
> > >> +	long pos;
> > >> +};
> > >> +
> > >> +/* Structure containing the information for filter updating */
> > >> +struct hisi_ptt_filter_update_info {
> > >> +	struct pci_dev *pdev;
> > >> +	u32 port_devid;
> > >> +	bool is_port;
> > >> +	bool is_add;
> > >> +	u16 val;
> > >> +};
> > >> +
> > >> +struct hisi_ptt {
> > >> +	struct hisi_ptt_trace_ctrl trace_ctrl;
> > >> +	struct notifier_block hisi_ptt_nb;
> > >> +	struct pmu hisi_ptt_pmu;
> > >> +	void __iomem *iobase;
> > >> +	struct pci_dev *pdev;
> > >> +	/*
> > >> +	 * Use the mutex to protect the filter list and
> > >> +	 * serialize the perf process.
> > >> +	 */
> > >> +	struct mutex mutex;
> > >> +	const char *name;
> > >> +	u16 core_id;
> > >> +	u16 sicl_id;
> > >> +	/* PCI device range managed by the PTT device */
> > >> +	u32 upper;
> > >> +	u32 lower;
> > >> +	u8 busnr;
> > >> +
> > >> +	/*
> > >> +	 * The trace TLP headers can either be filtered by certain
> > >> +	 * root port, or by the requester ID. Organize the filters
> > >> +	 * by @port_filters and @req_filters here. The mask of all
> > >> +	 * the valid ports is also cached for doing sanity check
> > >> +	 * of user input.
> > >> +	 */
> > >> +	struct list_head port_filters;
> > >> +	struct list_head req_filters;
> > >> +	u16 port_mask;
> > >> +
> > >> +	/*
> > >> +	 * We use a delayed work here to avoid indefinitely waiting for
> > >> +	 * the hisi_ptt->mutex which protecting the filter list. The
> > >> +	 * work will be delayed only if the mutex can not be held,
> > >> +	 * otherwise no delay will be applied.
> > >> +	 */
> > >> +	struct delayed_work work;
> > >> +	spinlock_t filter_update_lock;
> > >> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
> > >> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
> > >> +};
> > > 
> > > It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
> > > 
> > 
> > I'm not sure whether it is common or permitted to provide a header which is only used by
> > this driver with a single file.
> 
> I've seen it done many times.
> 
> > 
> > >> +
> > >> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
> > >> +{
> > >> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
> > >> +}
> > > 
> > > I'm not sure there is any value to this inline function.  I suggest to simply
> > > call container_of() whenever it is needed.
> > > 
> > 
> > ok.
> > 
> > >> +
> > >> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
> > >> +{
> > >> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
> > >> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
> > >> +
> > >> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	u32 val;
> > >> +
> > >> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
> > >> +				  val & HISI_PTT_TRACE_IDLE,
> > >> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
> > >> +				  HISI_PTT_WAIT_TIMEOUT_US);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct device *dev = &hisi_ptt->pdev->dev;
> > >> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
> > >> +
> > >> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
> > >> +		list_del(&buflet->list);
> > >> +
> > >> +		if (ctrl->has_rmr)
> > >> +			memunmap(buflet->addr);
> > >> +		else
> > >> +			dma_free_coherent(dev, buflet->size, buflet->addr,
> > >> +					  buflet->dma);
> > >> +
> > >> +		kfree(buflet);
> > >> +	}
> > >> +}
> > >> +
> > >> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct device *dev = &hisi_ptt->pdev->dev;
> > >> +	struct hisi_ptt_dma_buflet *buflet;
> > >> +	int i, ret;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.buf_index = 0;
> > >> +
> > >> +	/* Make sure the trace buffer is empty before allocating */
> > >> +	if (!list_empty(&ctrl->trace_buf)) {
> > >> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
> > >> +			memset(buflet->addr, 0, buflet->size);
> > >> +		return 0;
> > >> +	}
> > >> +
> > >> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
> > >> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
> > >> +		if (!buflet) {
> > >> +			ret = -ENOMEM;
> > >> +			goto err;
> > >> +		}
> > >> +
> > >> +		if (ctrl->has_rmr) {
> > >> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
> > >> +
> > >> +			buflet->dma = base;
> > >> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
> > >> +		} else {
> > >> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
> > >> +							  &buflet->dma, GFP_KERNEL);
> > >> +		}
> > >> +
> > >> +		if (!buflet->addr) {
> > >> +			kfree(buflet);
> > >> +			ret = -ENOMEM;
> > >> +			goto err;
> > >> +		}
> > >> +
> > >> +		memset(buflet->addr, 0, buflet->size);
> > >> +
> > >> +		buflet->index = i;
> > >> +		buflet->size = ctrl->buflet_size;
> > >> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +err:
> > >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	struct hisi_ptt_dma_buflet *cur;
> > >> +	u32 val;
> > >> +
> > >> +	/* Check device idle before start trace */
> > >> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
> > >> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
> > >> +		return -EBUSY;
> > >> +	}
> > >> +
> > >> +	/* Reset the DMA before start tracing */
> > >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	val |= HISI_PTT_TRACE_CTRL_RST;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	/*
> > >> +	 * We'll be in the perf context where preemption is disabled,
> > >> +	 * so use busy loop here.
> > >> +	 */
> > >> +	mdelay(HISI_PTT_RESET_WAIT_MS);
> > >> +
> > >> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	/* clear the interrupt status */
> > >> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
> > >> +
> > >> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
> > >> +		writel(lower_32_bits(cur->dma),
> > >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
> > >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> > >> +		writel(upper_32_bits(cur->dma),
> > >> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
> > >> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
> > >> +	}
> > >> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
> > >> +
> > >> +	/* set the trace control register */
> > >> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
> > >> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
> > >> +	if (!hisi_ptt->trace_ctrl.is_port)
> > >> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
> > >> +
> > >> +	val |= HISI_PTT_TRACE_CTRL_EN;
> > >> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
> > >> +
> > >> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
> > >> +{
> > >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> > >> +	struct perf_event *event = handle->event;
> > >> +	struct hisi_ptt_dma_buflet *cur;
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +
> > >> +	buf = perf_get_aux(handle);
> > >> +	if (!buf || !handle->size)
> > >> +		return -EINVAL;
> > >> +
> > >> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
> > >> +		if (cur->index == index)
> > >> +			break;
> > >> +
> > >> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
> > >> +	memset(cur->addr, 0, cur->size);
> > >> +	buf->pos += cur->size;
> > >> +
> > >> +	if (stop) {
> > >> +		perf_aux_output_end(handle, buf->pos);
> > >> +	} else if (buf->length - buf->pos < cur->size) {
> > >> +		perf_aux_output_skip(handle, buf->length - buf->pos);
> > >> +		perf_aux_output_end(handle, buf->pos);
> > >> +
> > >> +		buf = perf_aux_output_begin(handle, event);
> > >> +		if (!buf)
> > >> +			return -EINVAL;
> > >> +
> > >> +		buf->pos = handle->head % buf->length;
> > >> +		if (buf->length - buf->pos < cur->size) {
> > >> +			perf_aux_output_end(handle, 0);
> > >> +			return -EINVAL;
> > >> +		}
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = context;
> > >> +	u32 status, buf_idx;
> > >> +
> > >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	buf_idx = ffs(status) - 1;
> > >> +
> > >> +	/* Clear the interrupt status of buflet @buf_idx */
> > >> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +
> > >> +	/*
> > >> +	 * Update the AUX buffer and cache the current buflet index,
> > >> +	 * as we need to know this and save the data when the trace
> > >> +	 * is ended out of the interrupt handler. End the trace
> > >> +	 * if the updating fails.
> > >> +	 */
> > >> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +	else
> > >> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
> > >> +
> > >> +	return IRQ_HANDLED;
> > >> +}
> > >> +
> > >> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = context;
> > >> +	u32 status;
> > >> +
> > >> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
> > >> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
> > >> +		return IRQ_NONE;
> > >> +
> > >> +	return IRQ_WAKE_THREAD;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_irq_free_vectors(void *pdev)
> > >> +{
> > >> +	pci_free_irq_vectors(pdev);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	int ret;
> > >> +
> > >> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
> > >> +	if (ret < 0) {
> > >> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
> > >> +	if (ret < 0)
> > >> +		return ret;
> > >> +
> > >> +	ret = devm_request_threaded_irq(&pdev->dev,
> > >> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
> > >> +					hisi_ptt_irq, hisi_ptt_isr, 0,
> > >> +					"hisi-ptt", hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
> > >> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_update_filters(struct work_struct *work)
> > >> +{
> > >> +	struct delayed_work *delayed_work = to_delayed_work(work);
> > >> +	struct hisi_ptt_filter_update_info info;
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	struct list_head *target_list;
> > >> +	struct hisi_ptt *hisi_ptt;
> > >> +
> > >> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
> > >> +
> > >> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
> > >> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
> > >> +		return;
> > >> +	}
> > >> +
> > >> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
> > >> +		target_list = info.is_port ? &hisi_ptt->port_filters :
> > >> +			      &hisi_ptt->req_filters;
> > >> +
> > >> +		if (info.is_add) {
> > >> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
> > >> +			if (!filter)
> > >> +				continue;
> > >> +
> > >> +			filter->pdev = info.pdev;
> > >> +			filter->val = info.val;
> > >> +
> > >> +			list_add_tail(&filter->list, target_list);
> > >> +		} else {
> > >> +			list_for_each_entry(filter, target_list, list)
> > >> +				if (filter->val == info.val) {
> > >> +					list_del(&filter->list);
> > >> +					kfree(filter);
> > >> +					break;
> > >> +				}
> > >> +		}
> > >> +
> > >> +		/* Update the available port mask */
> > >> +		if (!info.is_port)
> > >> +			continue;
> > >> +
> > >> +		if (info.is_add)
> > >> +			hisi_ptt->port_mask |= info.val;
> > >> +		else
> > >> +			hisi_ptt->port_mask &= ~info.val;
> > >> +	}
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
> > >> +				    struct hisi_ptt_filter_update_info *info)
> > >> +{
> > >> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
> > >> +
> > >> +	if (!root_port)
> > >> +		return;
> > >> +
> > >> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
> > >> +	if (info->port_devid < hisi_ptt->lower ||
> > >> +	    info->port_devid > hisi_ptt->upper)
> > >> +		return;
> > >> +
> > >> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
> > >> +	info->val = hisi_ptt_get_filter_val(info->pdev);
> > >> +
> > >> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
> > >> +				&hisi_ptt->filter_update_lock))
> > >> +		schedule_delayed_work(&hisi_ptt->work, 0);
> > >> +	else
> > >> +		pci_warn(hisi_ptt->pdev,
> > >> +			 "filter update fifo overflow for target %s\n",
> > >> +			 pci_name(info->pdev));
> > >> +}
> > >> +
> > >> +/*
> > >> + * A PCI bus notifier is used here for dynamically updating the filter
> > >> + * list.
> > >> + */
> > >> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
> > >> +				  void *data)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
> > >> +	struct hisi_ptt_filter_update_info info;
> > >> +	struct device *dev = data;
> > >> +	struct pci_dev *pdev = to_pci_dev(dev);
> > >> +
> > >> +	info.pdev = pdev;
> > >> +
> > >> +	switch (action) {
> > >> +	case BUS_NOTIFY_ADD_DEVICE:
> > >> +		info.is_add = true;
> > >> +		break;
> > >> +	case BUS_NOTIFY_DEL_DEVICE:
> > >> +		info.is_add = false;
> > >> +		break;
> > >> +	default:
> > >> +		return 0;
> > >> +	}
> > >> +
> > >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
> > >> +{
> > >> +	struct hisi_ptt_filter_update_info info = {
> > >> +		.pdev = pdev,
> > >> +		.is_add = true,
> > >> +	};
> > >> +	struct hisi_ptt *hisi_ptt = data;
> > >> +
> > >> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct hisi_ptt_filter_desc *filter, *tfilter;
> > >> +
> > >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
> > >> +		list_del(&filter->list);
> > >> +		kfree(filter);
> > >> +	}
> > >> +
> > >> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
> > >> +		list_del(&filter->list);
> > >> +		kfree(filter);
> > >> +	}
> > >> +}
> > >> +
> > >> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	struct pci_bus *bus;
> > >> +	u32 reg;
> > >> +
> > >> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
> > >> +	spin_lock_init(&hisi_ptt->filter_update_lock);
> > >> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
> > >> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
> > >> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
> > >> +
> > >> +	/*
> > >> +	 * The device range register provides the information about the
> > >> +	 * root ports which the RCiEP can control and trace. The RCiEP
> > >> +	 * and the root ports it support are on the same PCIe core, with
> > >> +	 * same domain number but maybe different bus number. The device
> > >> +	 * range register will tell us which root ports we can support,
> > >> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
> > >> +	 * while Bit[15:0] indicates the lower.
> > >> +	 */
> > >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
> > >> +	hisi_ptt->upper = reg >> 16;
> > >> +	hisi_ptt->lower = reg & 0xffff;
> > >> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
> > >> +
> > >> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
> > >> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
> > >> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
> > >> +
> > >> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
> > >> +	if (bus)
> > >> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
> > > 
> > > What happens if bus is NULL?  Should this be reported to the caller and cause
> > > hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
> > > 
> > 
> > It's ok if the list is NULL in the probe process,
> > as the device maybe hotplugged after the PTT driver probe, in which case
> > we can detect the event and update the list as we register a bus notifier.
> > 
> 
> Please add the above as a comment in the code.
> 
> 
> > >> +
> > >> +	/* Initialize trace controls */
> > >> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
> > >> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
> > >> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
> > >> +}
> > >> +
> > >> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
> > >> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
> > >> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
> > >> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
> > >> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
> > >> +
> > >> +static ssize_t available_filters_show(struct device *dev,
> > >> +				      struct device_attribute *attr,
> > >> +				      char *buf)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	int pos = 0;
> > >> +
> > >> +	if (list_empty(&hisi_ptt->port_filters))
> > >> +		return sysfs_emit(buf, "#### No available filter ####\n");
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
> > >> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
> > >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
> > >> +				     pci_name(filter->pdev),
> > >> +				     hisi_ptt_get_filter_val(filter->pdev) |
> > >> +				     HISI_PTT_PMU_FILTER_IS_PORT);
> > >> +
> > >> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
> > >> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> > >> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
> > >> +				     pci_name(filter->pdev),
> > >> +				     hisi_ptt_get_filter_val(filter->pdev));
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return pos;
> > >> +}
> > >> +static DEVICE_ATTR_ADMIN_RO(available_filters);
> > >> +
> > >> +PMU_FORMAT_ATTR(filter,		"config:0-19");
> > >> +PMU_FORMAT_ATTR(direction,	"config:20-23");
> > >> +PMU_FORMAT_ATTR(type,		"config:24-31");
> > >> +PMU_FORMAT_ATTR(format,		"config:32-35");
> > >> +
> > >> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
> > >> +	&format_attr_filter.attr,
> > >> +	&format_attr_direction.attr,
> > >> +	&format_attr_type.attr,
> > >> +	&format_attr_format.attr,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +static struct attribute_group hisi_ptt_pmu_format_group = {
> > >> +	.name = "format",
> > >> +	.attrs = hisi_ptt_pmu_format_attrs,
> > >> +};
> > >> +
> > >> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
> > >> +	&dev_attr_available_filters.attr,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +static struct attribute_group hisi_ptt_pmu_filter_group = {
> > >> +	.attrs = hisi_ptt_pmu_filter_attrs,
> > >> +};
> > >> +
> > >> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
> > >> +	&hisi_ptt_pmu_format_group,
> > >> +	&hisi_ptt_pmu_filter_group,
> > >> +	NULL
> > >> +};
> > >> +
> > >> +/*
> > >> + * The supported value of the direction parameter. See hisi_ptt.rst
> > >> + * documentation for more details.
> > >> + */
> > >> +static u32 hisi_ptt_trace_available_direction[] = {
> > >> +	0,
> > >> +	1,
> > >> +	2,
> > >> +	3,
> > >> +};
> > >> +
> > >> +/* Different types can be set simultaneously */
> > >> +static u32 hisi_ptt_trace_available_type[] = {
> > >> +	1,	/* posted_request */
> > >> +	2,	/* non-posted_request */
> > >> +	4,	/* completion */
> > >> +};
> > >> +
> > >> +static u32 hisi_ptt_trace_availble_format[] = {
> > >> +	0,	/* 4DW */
> > >> +	1,	/* 8DW */
> > >> +};
> > >> +
> > >> +/*
> > >> + * Check whether the config is valid or not. Some configs are multi-selectable
> > >> + * and can be set simultaneously, while some are single selectable (onehot).
> > >> + * Use this function to check the non-onehot configs while
> > >> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
> > >> + */
> > >> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
> > >> +{
> > >> +	int i;
> > >> +
> > >> +	/*
> > >> +	 * The non-onehot configs cannot be 0. Walk the available
> > >> +	 * list and clear the valid bits of the config. If there
> > >> +	 * is any resident bit after the walk then the config is
> > >> +	 * invalid.
> > >> +	 */
> > >> +	for (i = 0; i < list_size; i++)
> > >> +		val &= ~available_list[i];
> > >> +
> > >> +	return val ? -EINVAL : 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
> > >> +{
> > >> +	int i, ret = -EINVAL;
> > >> +
> > >> +	for (i = 0; i < list_size; i++)
> > >> +		if (val == available_list[i]) {
> > >> +			ret = 0;
> > >> +			break;
> > >> +		}
> > >> +
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
> > >> +{
> > >> +	unsigned long val, port_mask = hisi_ptt->port_mask;
> > >> +	struct hisi_ptt_filter_desc *filter;
> > >> +	int ret = -EINVAL;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
> > >> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
> > >> +
> > >> +	/*
> > >> +	 * Port filters are defined as bit mask. For port filters, check
> > >> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
> > >> +	 * and whether it's empty or not, otherwise user has specified
> > >> +	 * some unsupported root ports.
> > >> +	 *
> > >> +	 * For Requester ID filters, walk the available filter list to see
> > >> +	 * whether we have one matched.
> > >> +	 */
> > >> +	if (hisi_ptt->trace_ctrl.is_port &&
> > >> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
> > >> +		ret = 0;
> > >> +	} else {
> > >> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
> > >> +			if (filter->val == val) {
> > >> +				ret = 0;
> > >> +				break;
> > >> +			}
> > >> +	}
> > >> +
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	hisi_ptt->trace_ctrl.filter = val;
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
> > >> +	int ret;
> > >> +	u32 val;
> > >> +
> > >> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
> > >> +		return -ENOENT;
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +
> > >> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
> > >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
> > >> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->direction = val;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
> > >> +
> > >> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
> > >> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->type = val;
> > >> +
> > >> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
> > >> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
> > >> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
> > >> +	if (ret < 0)
> > >> +		goto out;
> > >> +	ctrl->format = val;
> > >> +
> > >> +out:
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return ret;
> > >> +}
> > >> +
> > >> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
> > >> +				    int nr_pages, bool overwrite)
> > >> +{
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +	struct page **pagelist;
> > >> +	int i;
> > >> +
> > >> +	if (overwrite) {
> > >> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	/* If the pages size less than buflets, we cannot start trace */
> > >> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
> > >> +		return NULL;
> > >> +
> > >> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
> > >> +	if (!buf)
> > >> +		return NULL;
> > >> +
> > >> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
> > >> +	if (!pagelist) {
> > >> +		kfree(buf);
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	for (i = 0; i < nr_pages; i++)
> > >> +		pagelist[i] = virt_to_page(pages[i]);
> > >> +
> > >> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
> > >> +	if (!buf->base) {
> > >> +		kfree(pagelist);
> > >> +		kfree(buf);
> > >> +		return NULL;
> > >> +	}
> > >> +
> > >> +	buf->nr_pages = nr_pages;
> > >> +	buf->length = nr_pages * PAGE_SIZE;
> > >> +	buf->pos = 0;
> > >> +
> > >> +	kfree(pagelist);
> > >> +	return buf;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_free_aux(void *aux)
> > >> +{
> > >> +	struct hisi_ptt_pmu_buf *buf = aux;
> > >> +
> > >> +	vunmap(buf->base);
> > >> +	kfree(buf);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +	struct hisi_ptt_pmu_buf *buf;
> > >> +	int cpu = event->cpu;
> > >> +	int ret;
> > >> +
> > >> +	hwc->state = 0;
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> > >> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	if (cpu == -1)
> > >> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
> > >> +
> > >> +	/*
> > >> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
> > >> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
> > >> +	 * core in event_function_local().
> > >> +	 */
> > >> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
> > >> +				 cpumask_of(cpu)));
> > >> +
> > >> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	buf = perf_aux_output_begin(handle, event);
> > >> +	if (!buf) {
> > >> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	buf->pos = handle->head % buf->length;
> > >> +
> > >> +	ret = hisi_ptt_trace_start(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
> > >> +		perf_aux_output_end(handle, 0);
> > >> +		goto stop;
> > >> +	}
> > >> +
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +	return;
> > >> +stop:
> > >> +	event->hw.state |= PERF_HES_STOPPED;
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +
> > >> +	if (hwc->state & PERF_HES_STOPPED)
> > >> +		return;
> > >> +
> > >> +	mutex_lock(&hisi_ptt->mutex);
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
> > >> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
> > >> +	}
> > >> +	mutex_unlock(&hisi_ptt->mutex);
> > >> +
> > >> +	hwc->state |= PERF_HES_STOPPED;
> > >> +	perf_event_update_userpage(event);
> > >> +	hwc->state |= PERF_HES_UPTODATE;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
> > >> +	struct hw_perf_event *hwc = &event->hw;
> > >> +	int cpu = event->cpu;
> > >> +
> > >> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
> > >> +		return 0;
> > >> +
> > >> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
> > >> +
> > >> +	if (flags & PERF_EF_START) {
> > >> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
> > >> +		if (hwc->state & PERF_HES_STOPPED)
> > >> +			return -EINVAL;
> > >> +	}
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
> > >> +{
> > >> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
> > >> +}
> > >> +
> > >> +static void hisi_ptt_unregister_pmu(void *priv)
> > >> +{
> > >> +	perf_pmu_unregister(priv);
> > >> +}
> > >> +
> > >> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	char *pmu_name;
> > >> +	int ret;
> > >> +
> > >> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
> > >> +		.module		= THIS_MODULE,
> > >> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
> > >> +		.task_ctx_nr	= perf_sw_context,
> > >> +		.attr_groups	= hisi_ptt_pmu_groups,
> > >> +		.event_init	= hisi_ptt_pmu_event_init,
> > >> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
> > >> +		.free_aux	= hisi_ptt_pmu_free_aux,
> > >> +		.start		= hisi_ptt_pmu_start,
> > >> +		.stop		= hisi_ptt_pmu_stop,
> > >> +		.add		= hisi_ptt_pmu_add,
> > >> +		.del		= hisi_ptt_pmu_del,
> > >> +	};
> > >> +
> > >> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
> > >> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
> > >> +	if (!pmu_name)
> > >> +		return -ENOMEM;
> > >> +
> > >> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
> > >> +					hisi_ptt_unregister_pmu,
> > >> +					&hisi_ptt->hisi_ptt_pmu);
> > >> +}
> > >> +
> > >> +/*
> > >> + * Get RMR address if provided by the firmware.
> > >> + * Return 0 if the IOMMU doesn't present or the policy of the
> > >> + * IOMMU domain is passthrough or we get a usable RMR region.
> > >> + * Otherwise a negative value is returned.
> > >> + */
> > >> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
> > >> +{
> > >> +	struct pci_dev *pdev = hisi_ptt->pdev;
> > >> +	struct iommu_domain *iommu_domain;
> > >> +	struct iommu_resv_region *region;
> > >> +	LIST_HEAD(list);
> > >> +
> > >> +	/*
> > >> +	 * Use direct DMA if IOMMU does not present or the policy of the
> > >> +	 * IOMMU domain is passthrough.
> > >> +	 */
> > >> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
> > >> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
> > >> +		return 0;
> > >> +
> > >> +	iommu_get_resv_regions(&pdev->dev, &list);
> > >> +	list_for_each_entry(region, &list, list)
> > >> +		if (region->type == IOMMU_RESV_DIRECT &&
> > >> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
> > >> +			hisi_ptt->trace_ctrl.has_rmr = true;
> > >> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
> > >> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
> > >> +			break;
> > >> +		}
> > >> +
> > >> +	iommu_put_resv_regions(&pdev->dev, &list);
> > >> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
> > >> +}
> > >> +
> > >> +static int hisi_ptt_probe(struct pci_dev *pdev,
> > >> +			  const struct pci_device_id *id)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt;
> > >> +	int ret;
> > >> +
> > >> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
> > >> +	if (!hisi_ptt)
> > >> +		return -ENOMEM;
> > >> +
> > >> +	mutex_init(&hisi_ptt->mutex);
> > >> +	hisi_ptt->pdev = pdev;
> > >> +
> > >> +	/*
> > >> +	 * Lifetime of pci_dev is longer than hisi_ptt,
> > >> +	 * so directly reference to the pci name string.
> > >> +	 */
> > >> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
> > >> +	pci_set_drvdata(pdev, hisi_ptt);
> > >> +
> > >> +	ret = pcim_enable_device(pdev);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
> > >> +
> > >> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
> > >> +		return ret;
> > >> +	}
> > >> +	pci_set_master(pdev);
> > >> +
> > >> +	ret = hisi_ptt_register_irq(hisi_ptt);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	hisi_ptt_init_ctrls(hisi_ptt);
> > >> +
> > >> +	ret = hisi_ptt_register_pmu(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
> > >> +		return ret;
> > >> +	}
> > > 
> > > If this fails the pci_free_irq_vectors() is not called.
> > > 
> > 
> > It'll be called as we've registered a devm callback when
> > allocating the irq vector. See hisi_ptt_register_irq().
> 
> Ah yes - thanks for pointing that out.
> 
> > 
> > > I am out of time for today - I will continue tomorrow.
> > > 
> > 
> > Sure. Thanks!
> > Yicong
> > 
> > > Thanks,
> > > Mathieu
> > > 
> > >> +
> > >> +	ret = hisi_ptt_get_rmr(hisi_ptt);
> > >> +	if (ret) {
> > >> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
> > >> +		return ret;
> > >> +	}
> > >> +
> > >> +	/* Register the bus notifier for dynamically updating the filter list */
> > >> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
> > >> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> > >> +	if (ret)
> > >> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +void hisi_ptt_remove(struct pci_dev *pdev)
> > >> +{
> > >> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
> > >> +
> > >> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
> > >> +
> > >> +	/* Cancel any work that has been queued */
> > >> +	cancel_delayed_work_sync(&hisi_ptt->work);
> > >> +
> > >> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
> > >> +		hisi_ptt_trace_end(hisi_ptt);
> > >> +
> > >> +	hisi_ptt_free_trace_buf(hisi_ptt);
> > >> +	hisi_ptt_release_filters(hisi_ptt);
> > >> +}
> > >> +
> > >> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
> > >> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
> > >> +	{ }
> > >> +};
> > >> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
> > >> +
> > >> +static struct pci_driver hisi_ptt_driver = {
> > >> +	.name = "hisi_ptt",
> > >> +	.id_table = hisi_ptt_id_tbl,
> > >> +	.probe = hisi_ptt_probe,
> > >> +	.remove = hisi_ptt_remove,
> > >> +};
> > >> +module_pci_driver(hisi_ptt_driver);
> > >> +
> > >> +MODULE_LICENSE("GPL v2");
> > >> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
> > >> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
> > >> -- 
> > >> 2.33.0
> > >>
> > > .
> > > 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

On 2021/11/27 1:10, Mathieu Poirier wrote:
> 
> [...]
> 
>>
>>>>> +
>>>>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>>>>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>>>>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>>>>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>>>>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>>>>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>>>>> +
>>>>> +/* Delay time for filter updating work */
>>>>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>>>>> +/* Wait time for DMA hardware to reset */
>>>>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>>>>> +/* Poll timeout and interval for waiting hardware work to finish */
>>>>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>>>>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>>>>> +
>>>>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>>>>> +
>>>>> +enum hisi_ptt_trace_status {
>>>>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>>>>> +	HISI_PTT_TRACE_STATUS_ON,
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_dma_buflet {
>>>>
>>>> I'm not sure what a "buflet" is...  Probably best to just call this
>>>> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
>>>> know due to the lack of proper structure documentation.  Please have a look at
>>>> how "coresight_device" is documented.  The same applies to the rest of the
>>>> structures declared below.
>>>>
>>>
>>> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
>>> (we call it buflet) of 4MiB ...".
>>
>> I just noticed the documentation in patch 5 - I will read it before continuing
>> with this set.
>>
> 
> As promised I read the documentation and it didn't do much to alliviate my
> concern about the terminology.  If a "buflet" is a DMA buffer than simply call
> it a DMA buffer, regardless of the size.  
> 

ok. If it's hard for the readers to understand, I'll use DMA buffer instead.

> The size of this patchset makes it hard enough to review as it is... On top
> of things reviewers have to remember that PTT means "PCIe tune and trace
> device", tune means "PCIe link's events" and trace means "TLP headers".
> 

ok. I used to make the complete function into one patch, will see how to split
it to make review easier.

Trace and tune denote the two function of the device. Trace denote the function
of tracing the TLP headers and tune denotes the function of read and modify
the PCIe link's events. It's unavoidable of knowing the acronyms for the user
of the device. :)

> But we also have to keep in mind that a buflet is really just a DMA buffer...
> 
> I will not continue reviewing this set.  Please fix the terminology and spin
> another revision.  Doing so I suggest you trim down the submission to the
> absolute minimum feature set this driver should provide - enhancements can be
> made later when a foundation has been laid out.
> 
> Robin has pointed out what seems to be serious problems with the
> implemenation.  That will also have to be sorted out for the next iteration.
> 
ok. Will discuss with robin with his concerns and with the comments fixed in
the next version.

Thanks,
Yicong

> Regards,
> Mathieu
> 
> 
>>>
>>> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
>>> a 'buflet', which means a small buffer (there exists other words with similiar
>>> format: droplet, wavelet, ...).
>>>
>>> The device is designed like this to make sure we won't lose any data when we have to change
>>> the buffer address, the device can continue to write to the next buflet and don't
>>> neet to pause.
>>>
>>>>> +	struct list_head list;
>>>>> +	unsigned int size;
>>>>> +	dma_addr_t dma;
>>>>> +
>>>>> +	/*
>>>>> +	 * The address of the buflet holding the trace data.
>>>>> +	 * See Documentation/trace/hisi-ptt.rst for the details
>>>>
>>>> As of this writing, hisi-ptt.rst doesn't exist.
>>>>
>>>
>>> The comment intends to direct the user to the documentation if the user
>>> want to know details about the data format. The data format is decribed
>>> in the section 4 of the doc.
>>>
>>>>> +	 * of the data format.
>>>>> +	 */
>>>>> +	void *addr;
>>>>> +	int index;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_trace_ctrl {
>>>>> +	enum hisi_ptt_trace_status status;
>>>>> +	struct perf_output_handle handle;
>>>>> +	struct list_head trace_buf;
>>>>> +	/*
>>>>> +	 * The index of the buffer which trace data
>>>>> +	 * currently is writing to.
>>>>> +	 */
>>>>> +	u32 buf_index;
>>>>> +
>>>>> +	int default_cpu;
>>>>> +	u32 buflet_size;
>>>>> +	bool is_port;
>>>>> +	u32 format:1;		/* Format of the traced TLP headers */
>>>>> +	u32 type:4;		/* Type of the TLP headers to trace */
>>>>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>>>>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>>>>> +
>>>>> +	phys_addr_t rmr_addr;
>>>>> +	size_t rmr_length;
>>>>> +	bool has_rmr;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_filter_desc {
>>>>> +	struct list_head list;
>>>>> +	struct pci_dev *pdev;
>>>>> +	u16 val;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_pmu_buf {
>>>>> +	size_t length;
>>>>> +	int nr_pages;
>>>>> +	void *base;
>>>>> +	long pos;
>>>>> +};
>>>>> +
>>>>> +/* Structure containing the information for filter updating */
>>>>> +struct hisi_ptt_filter_update_info {
>>>>> +	struct pci_dev *pdev;
>>>>> +	u32 port_devid;
>>>>> +	bool is_port;
>>>>> +	bool is_add;
>>>>> +	u16 val;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt {
>>>>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>>>>> +	struct notifier_block hisi_ptt_nb;
>>>>> +	struct pmu hisi_ptt_pmu;
>>>>> +	void __iomem *iobase;
>>>>> +	struct pci_dev *pdev;
>>>>> +	/*
>>>>> +	 * Use the mutex to protect the filter list and
>>>>> +	 * serialize the perf process.
>>>>> +	 */
>>>>> +	struct mutex mutex;
>>>>> +	const char *name;
>>>>> +	u16 core_id;
>>>>> +	u16 sicl_id;
>>>>> +	/* PCI device range managed by the PTT device */
>>>>> +	u32 upper;
>>>>> +	u32 lower;
>>>>> +	u8 busnr;
>>>>> +
>>>>> +	/*
>>>>> +	 * The trace TLP headers can either be filtered by certain
>>>>> +	 * root port, or by the requester ID. Organize the filters
>>>>> +	 * by @port_filters and @req_filters here. The mask of all
>>>>> +	 * the valid ports is also cached for doing sanity check
>>>>> +	 * of user input.
>>>>> +	 */
>>>>> +	struct list_head port_filters;
>>>>> +	struct list_head req_filters;
>>>>> +	u16 port_mask;
>>>>> +
>>>>> +	/*
>>>>> +	 * We use a delayed work here to avoid indefinitely waiting for
>>>>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>>>>> +	 * work will be delayed only if the mutex can not be held,
>>>>> +	 * otherwise no delay will be applied.
>>>>> +	 */
>>>>> +	struct delayed_work work;
>>>>> +	spinlock_t filter_update_lock;
>>>>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>>>>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>>>>> +};
>>>>
>>>> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
>>>>
>>>
>>> I'm not sure whether it is common or permitted to provide a header which is only used by
>>> this driver with a single file.
>>
>> I've seen it done many times.
>>
>>>
>>>>> +
>>>>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>>>>> +{
>>>>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>>>>> +}
>>>>
>>>> I'm not sure there is any value to this inline function.  I suggest to simply
>>>> call container_of() whenever it is needed.
>>>>
>>>
>>> ok.
>>>
>>>>> +
>>>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>>>> +{
>>>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>>>> +
>>>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	u32 val;
>>>>> +
>>>>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>>>>> +				  val & HISI_PTT_TRACE_IDLE,
>>>>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>>>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>>>>> +
>>>>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>>>>> +		list_del(&buflet->list);
>>>>> +
>>>>> +		if (ctrl->has_rmr)
>>>>> +			memunmap(buflet->addr);
>>>>> +		else
>>>>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>>>>> +					  buflet->dma);
>>>>> +
>>>>> +		kfree(buflet);
>>>>> +	}
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>>> +	struct hisi_ptt_dma_buflet *buflet;
>>>>> +	int i, ret;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>>>> +
>>>>> +	/* Make sure the trace buffer is empty before allocating */
>>>>> +	if (!list_empty(&ctrl->trace_buf)) {
>>>>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>>>>> +			memset(buflet->addr, 0, buflet->size);
>>>>> +		return 0;
>>>>> +	}
>>>>> +
>>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>>>>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>>>>> +		if (!buflet) {
>>>>> +			ret = -ENOMEM;
>>>>> +			goto err;
>>>>> +		}
>>>>> +
>>>>> +		if (ctrl->has_rmr) {
>>>>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>>>>> +
>>>>> +			buflet->dma = base;
>>>>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>>>>> +		} else {
>>>>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>>>>> +							  &buflet->dma, GFP_KERNEL);
>>>>> +		}
>>>>> +
>>>>> +		if (!buflet->addr) {
>>>>> +			kfree(buflet);
>>>>> +			ret = -ENOMEM;
>>>>> +			goto err;
>>>>> +		}
>>>>> +
>>>>> +		memset(buflet->addr, 0, buflet->size);
>>>>> +
>>>>> +		buflet->index = i;
>>>>> +		buflet->size = ctrl->buflet_size;
>>>>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +err:
>>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>>> +	u32 val;
>>>>> +
>>>>> +	/* Check device idle before start trace */
>>>>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>>>>> +		return -EBUSY;
>>>>> +	}
>>>>> +
>>>>> +	/* Reset the DMA before start tracing */
>>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	/*
>>>>> +	 * We'll be in the perf context where preemption is disabled,
>>>>> +	 * so use busy loop here.
>>>>> +	 */
>>>>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>>>>> +
>>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	/* clear the interrupt status */
>>>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>>> +
>>>>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>>>>> +		writel(lower_32_bits(cur->dma),
>>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>>> +		writel(upper_32_bits(cur->dma),
>>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>>> +	}
>>>>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>>>> +
>>>>> +	/* set the trace control register */
>>>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>>>> +
>>>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>>>> +{
>>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>>> +	struct perf_event *event = handle->event;
>>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +
>>>>> +	buf = perf_get_aux(handle);
>>>>> +	if (!buf || !handle->size)
>>>>> +		return -EINVAL;
>>>>> +
>>>>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>>>>> +		if (cur->index == index)
>>>>> +			break;
>>>>> +
>>>>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>>>>> +	memset(cur->addr, 0, cur->size);
>>>>> +	buf->pos += cur->size;
>>>>> +
>>>>> +	if (stop) {
>>>>> +		perf_aux_output_end(handle, buf->pos);
>>>>> +	} else if (buf->length - buf->pos < cur->size) {
>>>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>>>> +		perf_aux_output_end(handle, buf->pos);
>>>>> +
>>>>> +		buf = perf_aux_output_begin(handle, event);
>>>>> +		if (!buf)
>>>>> +			return -EINVAL;
>>>>> +
>>>>> +		buf->pos = handle->head % buf->length;
>>>>> +		if (buf->length - buf->pos < cur->size) {
>>>>> +			perf_aux_output_end(handle, 0);
>>>>> +			return -EINVAL;
>>>>> +		}
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>>> +	u32 status, buf_idx;
>>>>> +
>>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	buf_idx = ffs(status) - 1;
>>>>> +
>>>>> +	/* Clear the interrupt status of buflet @buf_idx */
>>>>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +
>>>>> +	/*
>>>>> +	 * Update the AUX buffer and cache the current buflet index,
>>>>> +	 * as we need to know this and save the data when the trace
>>>>> +	 * is ended out of the interrupt handler. End the trace
>>>>> +	 * if the updating fails.
>>>>> +	 */
>>>>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +	else
>>>>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>>>>> +
>>>>> +	return IRQ_HANDLED;
>>>>> +}
>>>>> +
>>>>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>>> +	u32 status;
>>>>> +
>>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>>>>> +		return IRQ_NONE;
>>>>> +
>>>>> +	return IRQ_WAKE_THREAD;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>>>>> +{
>>>>> +	pci_free_irq_vectors(pdev);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	int ret;
>>>>> +
>>>>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>>>>> +	if (ret < 0) {
>>>>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>>>>> +	if (ret < 0)
>>>>> +		return ret;
>>>>> +
>>>>> +	ret = devm_request_threaded_irq(&pdev->dev,
>>>>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>>>>> +					"hisi-ptt", hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>>>>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_update_filters(struct work_struct *work)
>>>>> +{
>>>>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>>>>> +	struct hisi_ptt_filter_update_info info;
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	struct list_head *target_list;
>>>>> +	struct hisi_ptt *hisi_ptt;
>>>>> +
>>>>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>>>>> +
>>>>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>>>>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>>>>> +		return;
>>>>> +	}
>>>>> +
>>>>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>>>>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>>>>> +			      &hisi_ptt->req_filters;
>>>>> +
>>>>> +		if (info.is_add) {
>>>>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>>>>> +			if (!filter)
>>>>> +				continue;
>>>>> +
>>>>> +			filter->pdev = info.pdev;
>>>>> +			filter->val = info.val;
>>>>> +
>>>>> +			list_add_tail(&filter->list, target_list);
>>>>> +		} else {
>>>>> +			list_for_each_entry(filter, target_list, list)
>>>>> +				if (filter->val == info.val) {
>>>>> +					list_del(&filter->list);
>>>>> +					kfree(filter);
>>>>> +					break;
>>>>> +				}
>>>>> +		}
>>>>> +
>>>>> +		/* Update the available port mask */
>>>>> +		if (!info.is_port)
>>>>> +			continue;
>>>>> +
>>>>> +		if (info.is_add)
>>>>> +			hisi_ptt->port_mask |= info.val;
>>>>> +		else
>>>>> +			hisi_ptt->port_mask &= ~info.val;
>>>>> +	}
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>>>>> +				    struct hisi_ptt_filter_update_info *info)
>>>>> +{
>>>>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>>>>> +
>>>>> +	if (!root_port)
>>>>> +		return;
>>>>> +
>>>>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>>>>> +	if (info->port_devid < hisi_ptt->lower ||
>>>>> +	    info->port_devid > hisi_ptt->upper)
>>>>> +		return;
>>>>> +
>>>>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>>>>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>>>>> +
>>>>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>>>>> +				&hisi_ptt->filter_update_lock))
>>>>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>>>>> +	else
>>>>> +		pci_warn(hisi_ptt->pdev,
>>>>> +			 "filter update fifo overflow for target %s\n",
>>>>> +			 pci_name(info->pdev));
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * A PCI bus notifier is used here for dynamically updating the filter
>>>>> + * list.
>>>>> + */
>>>>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>>>>> +				  void *data)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>>>>> +	struct hisi_ptt_filter_update_info info;
>>>>> +	struct device *dev = data;
>>>>> +	struct pci_dev *pdev = to_pci_dev(dev);
>>>>> +
>>>>> +	info.pdev = pdev;
>>>>> +
>>>>> +	switch (action) {
>>>>> +	case BUS_NOTIFY_ADD_DEVICE:
>>>>> +		info.is_add = true;
>>>>> +		break;
>>>>> +	case BUS_NOTIFY_DEL_DEVICE:
>>>>> +		info.is_add = false;
>>>>> +		break;
>>>>> +	default:
>>>>> +		return 0;
>>>>> +	}
>>>>> +
>>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>>>>> +{
>>>>> +	struct hisi_ptt_filter_update_info info = {
>>>>> +		.pdev = pdev,
>>>>> +		.is_add = true,
>>>>> +	};
>>>>> +	struct hisi_ptt *hisi_ptt = data;
>>>>> +
>>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>>>>> +
>>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>>>>> +		list_del(&filter->list);
>>>>> +		kfree(filter);
>>>>> +	}
>>>>> +
>>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>>>>> +		list_del(&filter->list);
>>>>> +		kfree(filter);
>>>>> +	}
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	struct pci_bus *bus;
>>>>> +	u32 reg;
>>>>> +
>>>>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>>>>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>>>>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>>>>> +
>>>>> +	/*
>>>>> +	 * The device range register provides the information about the
>>>>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>>>>> +	 * and the root ports it support are on the same PCIe core, with
>>>>> +	 * same domain number but maybe different bus number. The device
>>>>> +	 * range register will tell us which root ports we can support,
>>>>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>>>>> +	 * while Bit[15:0] indicates the lower.
>>>>> +	 */
>>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>>>>> +	hisi_ptt->upper = reg >> 16;
>>>>> +	hisi_ptt->lower = reg & 0xffff;
>>>>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>>>>> +
>>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>>>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>>>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>>>> +
>>>>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>>>>> +	if (bus)
>>>>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>>>>
>>>> What happens if bus is NULL?  Should this be reported to the caller and cause
>>>> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
>>>>
>>>
>>> It's ok if the list is NULL in the probe process,
>>> as the device maybe hotplugged after the PTT driver probe, in which case
>>> we can detect the event and update the list as we register a bus notifier.
>>>
>>
>> Please add the above as a comment in the code.
>>
>>
>>>>> +
>>>>> +	/* Initialize trace controls */
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>>>>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>>>>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>>>>> +}
>>>>> +
>>>>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>>>>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>>>>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>>>>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>>>>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>>>>> +
>>>>> +static ssize_t available_filters_show(struct device *dev,
>>>>> +				      struct device_attribute *attr,
>>>>> +				      char *buf)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	int pos = 0;
>>>>> +
>>>>> +	if (list_empty(&hisi_ptt->port_filters))
>>>>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>>>>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>>>>> +				     pci_name(filter->pdev),
>>>>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>>>>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>>>>> +
>>>>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>>>>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>>>>> +				     pci_name(filter->pdev),
>>>>> +				     hisi_ptt_get_filter_val(filter->pdev));
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return pos;
>>>>> +}
>>>>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>>>>> +
>>>>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>>>>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>>>>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>>>>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>>>>> +
>>>>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>>>>> +	&format_attr_filter.attr,
>>>>> +	&format_attr_direction.attr,
>>>>> +	&format_attr_type.attr,
>>>>> +	&format_attr_format.attr,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>>>>> +	.name = "format",
>>>>> +	.attrs = hisi_ptt_pmu_format_attrs,
>>>>> +};
>>>>> +
>>>>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>>>>> +	&dev_attr_available_filters.attr,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>>>>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>>>>> +};
>>>>> +
>>>>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>>>>> +	&hisi_ptt_pmu_format_group,
>>>>> +	&hisi_ptt_pmu_filter_group,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +/*
>>>>> + * The supported value of the direction parameter. See hisi_ptt.rst
>>>>> + * documentation for more details.
>>>>> + */
>>>>> +static u32 hisi_ptt_trace_available_direction[] = {
>>>>> +	0,
>>>>> +	1,
>>>>> +	2,
>>>>> +	3,
>>>>> +};
>>>>> +
>>>>> +/* Different types can be set simultaneously */
>>>>> +static u32 hisi_ptt_trace_available_type[] = {
>>>>> +	1,	/* posted_request */
>>>>> +	2,	/* non-posted_request */
>>>>> +	4,	/* completion */
>>>>> +};
>>>>> +
>>>>> +static u32 hisi_ptt_trace_availble_format[] = {
>>>>> +	0,	/* 4DW */
>>>>> +	1,	/* 8DW */
>>>>> +};
>>>>> +
>>>>> +/*
>>>>> + * Check whether the config is valid or not. Some configs are multi-selectable
>>>>> + * and can be set simultaneously, while some are single selectable (onehot).
>>>>> + * Use this function to check the non-onehot configs while
>>>>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>>>>> + */
>>>>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>>>>> +{
>>>>> +	int i;
>>>>> +
>>>>> +	/*
>>>>> +	 * The non-onehot configs cannot be 0. Walk the available
>>>>> +	 * list and clear the valid bits of the config. If there
>>>>> +	 * is any resident bit after the walk then the config is
>>>>> +	 * invalid.
>>>>> +	 */
>>>>> +	for (i = 0; i < list_size; i++)
>>>>> +		val &= ~available_list[i];
>>>>> +
>>>>> +	return val ? -EINVAL : 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>>>>> +{
>>>>> +	int i, ret = -EINVAL;
>>>>> +
>>>>> +	for (i = 0; i < list_size; i++)
>>>>> +		if (val == available_list[i]) {
>>>>> +			ret = 0;
>>>>> +			break;
>>>>> +		}
>>>>> +
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>>>>> +{
>>>>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	int ret = -EINVAL;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>>>>> +
>>>>> +	/*
>>>>> +	 * Port filters are defined as bit mask. For port filters, check
>>>>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>>>>> +	 * and whether it's empty or not, otherwise user has specified
>>>>> +	 * some unsupported root ports.
>>>>> +	 *
>>>>> +	 * For Requester ID filters, walk the available filter list to see
>>>>> +	 * whether we have one matched.
>>>>> +	 */
>>>>> +	if (hisi_ptt->trace_ctrl.is_port &&
>>>>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>>>>> +		ret = 0;
>>>>> +	} else {
>>>>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>>> +			if (filter->val == val) {
>>>>> +				ret = 0;
>>>>> +				break;
>>>>> +			}
>>>>> +	}
>>>>> +
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.filter = val;
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	int ret;
>>>>> +	u32 val;
>>>>> +
>>>>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>>>>> +		return -ENOENT;
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +
>>>>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>>>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->direction = val;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>>>> +
>>>>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>>>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->type = val;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>>>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->format = val;
>>>>> +
>>>>> +out:
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>>>> +				    int nr_pages, bool overwrite)
>>>>> +{
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +	struct page **pagelist;
>>>>> +	int i;
>>>>> +
>>>>> +	if (overwrite) {
>>>>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	/* If the pages size less than buflets, we cannot start trace */
>>>>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>>>>> +		return NULL;
>>>>> +
>>>>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>>>> +	if (!buf)
>>>>> +		return NULL;
>>>>> +
>>>>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>>>> +	if (!pagelist) {
>>>>> +		kfree(buf);
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	for (i = 0; i < nr_pages; i++)
>>>>> +		pagelist[i] = virt_to_page(pages[i]);
>>>>> +
>>>>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>>>> +	if (!buf->base) {
>>>>> +		kfree(pagelist);
>>>>> +		kfree(buf);
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	buf->nr_pages = nr_pages;
>>>>> +	buf->length = nr_pages * PAGE_SIZE;
>>>>> +	buf->pos = 0;
>>>>> +
>>>>> +	kfree(pagelist);
>>>>> +	return buf;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>>>> +{
>>>>> +	struct hisi_ptt_pmu_buf *buf = aux;
>>>>> +
>>>>> +	vunmap(buf->base);
>>>>> +	kfree(buf);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +	int cpu = event->cpu;
>>>>> +	int ret;
>>>>> +
>>>>> +	hwc->state = 0;
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	if (cpu == -1)
>>>>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>>>>> +
>>>>> +	/*
>>>>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>>>>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>>>>> +	 * core in event_function_local().
>>>>> +	 */
>>>>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>>> +				 cpumask_of(cpu)));
>>>>> +
>>>>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	buf = perf_aux_output_begin(handle, event);
>>>>> +	if (!buf) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	buf->pos = handle->head % buf->length;
>>>>> +
>>>>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>>>> +		perf_aux_output_end(handle, 0);
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return;
>>>>> +stop:
>>>>> +	event->hw.state |= PERF_HES_STOPPED;
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +
>>>>> +	if (hwc->state & PERF_HES_STOPPED)
>>>>> +		return;
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>>>>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>>>>> +	}
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +
>>>>> +	hwc->state |= PERF_HES_STOPPED;
>>>>> +	perf_event_update_userpage(event);
>>>>> +	hwc->state |= PERF_HES_UPTODATE;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +	int cpu = event->cpu;
>>>>> +
>>>>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>>>>> +		return 0;
>>>>> +
>>>>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>>>>> +
>>>>> +	if (flags & PERF_EF_START) {
>>>>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>>>>> +		if (hwc->state & PERF_HES_STOPPED)
>>>>> +			return -EINVAL;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_unregister_pmu(void *priv)
>>>>> +{
>>>>> +	perf_pmu_unregister(priv);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	char *pmu_name;
>>>>> +	int ret;
>>>>> +
>>>>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>>>>> +		.module		= THIS_MODULE,
>>>>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>>>>> +		.task_ctx_nr	= perf_sw_context,
>>>>> +		.attr_groups	= hisi_ptt_pmu_groups,
>>>>> +		.event_init	= hisi_ptt_pmu_event_init,
>>>>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>>>>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>>>>> +		.start		= hisi_ptt_pmu_start,
>>>>> +		.stop		= hisi_ptt_pmu_stop,
>>>>> +		.add		= hisi_ptt_pmu_add,
>>>>> +		.del		= hisi_ptt_pmu_del,
>>>>> +	};
>>>>> +
>>>>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>>>>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>>>>> +	if (!pmu_name)
>>>>> +		return -ENOMEM;
>>>>> +
>>>>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>>>>> +					hisi_ptt_unregister_pmu,
>>>>> +					&hisi_ptt->hisi_ptt_pmu);
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Get RMR address if provided by the firmware.
>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>> + * Otherwise a negative value is returned.
>>>>> + */
>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	struct iommu_domain *iommu_domain;
>>>>> +	struct iommu_resv_region *region;
>>>>> +	LIST_HEAD(list);
>>>>> +
>>>>> +	/*
>>>>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>>>>> +	 * IOMMU domain is passthrough.
>>>>> +	 */
>>>>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>> +		return 0;
>>>>> +
>>>>> +	iommu_get_resv_regions(&pdev->dev, &list);
>>>>> +	list_for_each_entry(region, &list, list)
>>>>> +		if (region->type == IOMMU_RESV_DIRECT &&
>>>>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>>>>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>> +			break;
>>>>> +		}
>>>>> +
>>>>> +	iommu_put_resv_regions(&pdev->dev, &list);
>>>>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>>>>> +			  const struct pci_device_id *id)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt;
>>>>> +	int ret;
>>>>> +
>>>>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>>>>> +	if (!hisi_ptt)
>>>>> +		return -ENOMEM;
>>>>> +
>>>>> +	mutex_init(&hisi_ptt->mutex);
>>>>> +	hisi_ptt->pdev = pdev;
>>>>> +
>>>>> +	/*
>>>>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>>>>> +	 * so directly reference to the pci name string.
>>>>> +	 */
>>>>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>>>>> +	pci_set_drvdata(pdev, hisi_ptt);
>>>>> +
>>>>> +	ret = pcim_enable_device(pdev);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>>>>> +
>>>>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +	pci_set_master(pdev);
>>>>> +
>>>>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	hisi_ptt_init_ctrls(hisi_ptt);
>>>>> +
>>>>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>
>>>> If this fails the pci_free_irq_vectors() is not called.
>>>>
>>>
>>> It'll be called as we've registered a devm callback when
>>> allocating the irq vector. See hisi_ptt_register_irq().
>>
>> Ah yes - thanks for pointing that out.
>>
>>>
>>>> I am out of time for today - I will continue tomorrow.
>>>>
>>>
>>> Sure. Thanks!
>>> Yicong
>>>
>>>> Thanks,
>>>> Mathieu
>>>>
>>>>> +
>>>>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	/* Register the bus notifier for dynamically updating the filter list */
>>>>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>>>>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>>> +	if (ret)
>>>>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +void hisi_ptt_remove(struct pci_dev *pdev)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>>>> +
>>>>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>>> +
>>>>> +	/* Cancel any work that has been queued */
>>>>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>>>>> +
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +
>>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>>> +	hisi_ptt_release_filters(hisi_ptt);
>>>>> +}
>>>>> +
>>>>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>>>>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>>>>> +	{ }
>>>>> +};
>>>>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>>>>> +
>>>>> +static struct pci_driver hisi_ptt_driver = {
>>>>> +	.name = "hisi_ptt",
>>>>> +	.id_table = hisi_ptt_id_tbl,
>>>>> +	.probe = hisi_ptt_probe,
>>>>> +	.remove = hisi_ptt_remove,
>>>>> +};
>>>>> +module_pci_driver(hisi_ptt_driver);
>>>>> +
>>>>> +MODULE_LICENSE("GPL v2");
>>>>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>>>>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>>>>> -- 
>>>>> 2.33.0
>>>>>
>>>> .
>>>>
> .
> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/27 1:10, Mathieu Poirier wrote:
> 
> [...]
> 
>>
>>>>> +
>>>>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>>>>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>>>>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>>>>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>>>>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>>>>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>>>>> +
>>>>> +/* Delay time for filter updating work */
>>>>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>>>>> +/* Wait time for DMA hardware to reset */
>>>>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>>>>> +/* Poll timeout and interval for waiting hardware work to finish */
>>>>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>>>>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>>>>> +
>>>>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>>>>> +
>>>>> +enum hisi_ptt_trace_status {
>>>>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>>>>> +	HISI_PTT_TRACE_STATUS_ON,
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_dma_buflet {
>>>>
>>>> I'm not sure what a "buflet" is...  Probably best to just call this
>>>> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
>>>> know due to the lack of proper structure documentation.  Please have a look at
>>>> how "coresight_device" is documented.  The same applies to the rest of the
>>>> structures declared below.
>>>>
>>>
>>> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
>>> (we call it buflet) of 4MiB ...".
>>
>> I just noticed the documentation in patch 5 - I will read it before continuing
>> with this set.
>>
> 
> As promised I read the documentation and it didn't do much to alliviate my
> concern about the terminology.  If a "buflet" is a DMA buffer than simply call
> it a DMA buffer, regardless of the size.  
> 

ok. If it's hard for the readers to understand, I'll use DMA buffer instead.

> The size of this patchset makes it hard enough to review as it is... On top
> of things reviewers have to remember that PTT means "PCIe tune and trace
> device", tune means "PCIe link's events" and trace means "TLP headers".
> 

ok. I used to make the complete function into one patch, will see how to split
it to make review easier.

Trace and tune denote the two function of the device. Trace denote the function
of tracing the TLP headers and tune denotes the function of read and modify
the PCIe link's events. It's unavoidable of knowing the acronyms for the user
of the device. :)

> But we also have to keep in mind that a buflet is really just a DMA buffer...
> 
> I will not continue reviewing this set.  Please fix the terminology and spin
> another revision.  Doing so I suggest you trim down the submission to the
> absolute minimum feature set this driver should provide - enhancements can be
> made later when a foundation has been laid out.
> 
> Robin has pointed out what seems to be serious problems with the
> implemenation.  That will also have to be sorted out for the next iteration.
> 
ok. Will discuss with robin with his concerns and with the comments fixed in
the next version.

Thanks,
Yicong

> Regards,
> Mathieu
> 
> 
>>>
>>> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
>>> a 'buflet', which means a small buffer (there exists other words with similiar
>>> format: droplet, wavelet, ...).
>>>
>>> The device is designed like this to make sure we won't lose any data when we have to change
>>> the buffer address, the device can continue to write to the next buflet and don't
>>> neet to pause.
>>>
>>>>> +	struct list_head list;
>>>>> +	unsigned int size;
>>>>> +	dma_addr_t dma;
>>>>> +
>>>>> +	/*
>>>>> +	 * The address of the buflet holding the trace data.
>>>>> +	 * See Documentation/trace/hisi-ptt.rst for the details
>>>>
>>>> As of this writing, hisi-ptt.rst doesn't exist.
>>>>
>>>
>>> The comment intends to direct the user to the documentation if the user
>>> want to know details about the data format. The data format is decribed
>>> in the section 4 of the doc.
>>>
>>>>> +	 * of the data format.
>>>>> +	 */
>>>>> +	void *addr;
>>>>> +	int index;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_trace_ctrl {
>>>>> +	enum hisi_ptt_trace_status status;
>>>>> +	struct perf_output_handle handle;
>>>>> +	struct list_head trace_buf;
>>>>> +	/*
>>>>> +	 * The index of the buffer which trace data
>>>>> +	 * currently is writing to.
>>>>> +	 */
>>>>> +	u32 buf_index;
>>>>> +
>>>>> +	int default_cpu;
>>>>> +	u32 buflet_size;
>>>>> +	bool is_port;
>>>>> +	u32 format:1;		/* Format of the traced TLP headers */
>>>>> +	u32 type:4;		/* Type of the TLP headers to trace */
>>>>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>>>>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>>>>> +
>>>>> +	phys_addr_t rmr_addr;
>>>>> +	size_t rmr_length;
>>>>> +	bool has_rmr;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_filter_desc {
>>>>> +	struct list_head list;
>>>>> +	struct pci_dev *pdev;
>>>>> +	u16 val;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_pmu_buf {
>>>>> +	size_t length;
>>>>> +	int nr_pages;
>>>>> +	void *base;
>>>>> +	long pos;
>>>>> +};
>>>>> +
>>>>> +/* Structure containing the information for filter updating */
>>>>> +struct hisi_ptt_filter_update_info {
>>>>> +	struct pci_dev *pdev;
>>>>> +	u32 port_devid;
>>>>> +	bool is_port;
>>>>> +	bool is_add;
>>>>> +	u16 val;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt {
>>>>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>>>>> +	struct notifier_block hisi_ptt_nb;
>>>>> +	struct pmu hisi_ptt_pmu;
>>>>> +	void __iomem *iobase;
>>>>> +	struct pci_dev *pdev;
>>>>> +	/*
>>>>> +	 * Use the mutex to protect the filter list and
>>>>> +	 * serialize the perf process.
>>>>> +	 */
>>>>> +	struct mutex mutex;
>>>>> +	const char *name;
>>>>> +	u16 core_id;
>>>>> +	u16 sicl_id;
>>>>> +	/* PCI device range managed by the PTT device */
>>>>> +	u32 upper;
>>>>> +	u32 lower;
>>>>> +	u8 busnr;
>>>>> +
>>>>> +	/*
>>>>> +	 * The trace TLP headers can either be filtered by certain
>>>>> +	 * root port, or by the requester ID. Organize the filters
>>>>> +	 * by @port_filters and @req_filters here. The mask of all
>>>>> +	 * the valid ports is also cached for doing sanity check
>>>>> +	 * of user input.
>>>>> +	 */
>>>>> +	struct list_head port_filters;
>>>>> +	struct list_head req_filters;
>>>>> +	u16 port_mask;
>>>>> +
>>>>> +	/*
>>>>> +	 * We use a delayed work here to avoid indefinitely waiting for
>>>>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>>>>> +	 * work will be delayed only if the mutex can not be held,
>>>>> +	 * otherwise no delay will be applied.
>>>>> +	 */
>>>>> +	struct delayed_work work;
>>>>> +	spinlock_t filter_update_lock;
>>>>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>>>>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>>>>> +};
>>>>
>>>> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
>>>>
>>>
>>> I'm not sure whether it is common or permitted to provide a header which is only used by
>>> this driver with a single file.
>>
>> I've seen it done many times.
>>
>>>
>>>>> +
>>>>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>>>>> +{
>>>>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>>>>> +}
>>>>
>>>> I'm not sure there is any value to this inline function.  I suggest to simply
>>>> call container_of() whenever it is needed.
>>>>
>>>
>>> ok.
>>>
>>>>> +
>>>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>>>> +{
>>>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>>>> +
>>>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	u32 val;
>>>>> +
>>>>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>>>>> +				  val & HISI_PTT_TRACE_IDLE,
>>>>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>>>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>>>>> +
>>>>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>>>>> +		list_del(&buflet->list);
>>>>> +
>>>>> +		if (ctrl->has_rmr)
>>>>> +			memunmap(buflet->addr);
>>>>> +		else
>>>>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>>>>> +					  buflet->dma);
>>>>> +
>>>>> +		kfree(buflet);
>>>>> +	}
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>>> +	struct hisi_ptt_dma_buflet *buflet;
>>>>> +	int i, ret;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>>>> +
>>>>> +	/* Make sure the trace buffer is empty before allocating */
>>>>> +	if (!list_empty(&ctrl->trace_buf)) {
>>>>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>>>>> +			memset(buflet->addr, 0, buflet->size);
>>>>> +		return 0;
>>>>> +	}
>>>>> +
>>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>>>>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>>>>> +		if (!buflet) {
>>>>> +			ret = -ENOMEM;
>>>>> +			goto err;
>>>>> +		}
>>>>> +
>>>>> +		if (ctrl->has_rmr) {
>>>>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>>>>> +
>>>>> +			buflet->dma = base;
>>>>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>>>>> +		} else {
>>>>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>>>>> +							  &buflet->dma, GFP_KERNEL);
>>>>> +		}
>>>>> +
>>>>> +		if (!buflet->addr) {
>>>>> +			kfree(buflet);
>>>>> +			ret = -ENOMEM;
>>>>> +			goto err;
>>>>> +		}
>>>>> +
>>>>> +		memset(buflet->addr, 0, buflet->size);
>>>>> +
>>>>> +		buflet->index = i;
>>>>> +		buflet->size = ctrl->buflet_size;
>>>>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +err:
>>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>>> +	u32 val;
>>>>> +
>>>>> +	/* Check device idle before start trace */
>>>>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>>>>> +		return -EBUSY;
>>>>> +	}
>>>>> +
>>>>> +	/* Reset the DMA before start tracing */
>>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	/*
>>>>> +	 * We'll be in the perf context where preemption is disabled,
>>>>> +	 * so use busy loop here.
>>>>> +	 */
>>>>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>>>>> +
>>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	/* clear the interrupt status */
>>>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>>> +
>>>>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>>>>> +		writel(lower_32_bits(cur->dma),
>>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>>> +		writel(upper_32_bits(cur->dma),
>>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>>> +	}
>>>>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>>>> +
>>>>> +	/* set the trace control register */
>>>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>>>> +
>>>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>>>> +{
>>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>>> +	struct perf_event *event = handle->event;
>>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +
>>>>> +	buf = perf_get_aux(handle);
>>>>> +	if (!buf || !handle->size)
>>>>> +		return -EINVAL;
>>>>> +
>>>>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>>>>> +		if (cur->index == index)
>>>>> +			break;
>>>>> +
>>>>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>>>>> +	memset(cur->addr, 0, cur->size);
>>>>> +	buf->pos += cur->size;
>>>>> +
>>>>> +	if (stop) {
>>>>> +		perf_aux_output_end(handle, buf->pos);
>>>>> +	} else if (buf->length - buf->pos < cur->size) {
>>>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>>>> +		perf_aux_output_end(handle, buf->pos);
>>>>> +
>>>>> +		buf = perf_aux_output_begin(handle, event);
>>>>> +		if (!buf)
>>>>> +			return -EINVAL;
>>>>> +
>>>>> +		buf->pos = handle->head % buf->length;
>>>>> +		if (buf->length - buf->pos < cur->size) {
>>>>> +			perf_aux_output_end(handle, 0);
>>>>> +			return -EINVAL;
>>>>> +		}
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>>> +	u32 status, buf_idx;
>>>>> +
>>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	buf_idx = ffs(status) - 1;
>>>>> +
>>>>> +	/* Clear the interrupt status of buflet @buf_idx */
>>>>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +
>>>>> +	/*
>>>>> +	 * Update the AUX buffer and cache the current buflet index,
>>>>> +	 * as we need to know this and save the data when the trace
>>>>> +	 * is ended out of the interrupt handler. End the trace
>>>>> +	 * if the updating fails.
>>>>> +	 */
>>>>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +	else
>>>>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>>>>> +
>>>>> +	return IRQ_HANDLED;
>>>>> +}
>>>>> +
>>>>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>>> +	u32 status;
>>>>> +
>>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>>>>> +		return IRQ_NONE;
>>>>> +
>>>>> +	return IRQ_WAKE_THREAD;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>>>>> +{
>>>>> +	pci_free_irq_vectors(pdev);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	int ret;
>>>>> +
>>>>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>>>>> +	if (ret < 0) {
>>>>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>>>>> +	if (ret < 0)
>>>>> +		return ret;
>>>>> +
>>>>> +	ret = devm_request_threaded_irq(&pdev->dev,
>>>>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>>>>> +					"hisi-ptt", hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>>>>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_update_filters(struct work_struct *work)
>>>>> +{
>>>>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>>>>> +	struct hisi_ptt_filter_update_info info;
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	struct list_head *target_list;
>>>>> +	struct hisi_ptt *hisi_ptt;
>>>>> +
>>>>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>>>>> +
>>>>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>>>>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>>>>> +		return;
>>>>> +	}
>>>>> +
>>>>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>>>>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>>>>> +			      &hisi_ptt->req_filters;
>>>>> +
>>>>> +		if (info.is_add) {
>>>>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>>>>> +			if (!filter)
>>>>> +				continue;
>>>>> +
>>>>> +			filter->pdev = info.pdev;
>>>>> +			filter->val = info.val;
>>>>> +
>>>>> +			list_add_tail(&filter->list, target_list);
>>>>> +		} else {
>>>>> +			list_for_each_entry(filter, target_list, list)
>>>>> +				if (filter->val == info.val) {
>>>>> +					list_del(&filter->list);
>>>>> +					kfree(filter);
>>>>> +					break;
>>>>> +				}
>>>>> +		}
>>>>> +
>>>>> +		/* Update the available port mask */
>>>>> +		if (!info.is_port)
>>>>> +			continue;
>>>>> +
>>>>> +		if (info.is_add)
>>>>> +			hisi_ptt->port_mask |= info.val;
>>>>> +		else
>>>>> +			hisi_ptt->port_mask &= ~info.val;
>>>>> +	}
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>>>>> +				    struct hisi_ptt_filter_update_info *info)
>>>>> +{
>>>>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>>>>> +
>>>>> +	if (!root_port)
>>>>> +		return;
>>>>> +
>>>>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>>>>> +	if (info->port_devid < hisi_ptt->lower ||
>>>>> +	    info->port_devid > hisi_ptt->upper)
>>>>> +		return;
>>>>> +
>>>>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>>>>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>>>>> +
>>>>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>>>>> +				&hisi_ptt->filter_update_lock))
>>>>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>>>>> +	else
>>>>> +		pci_warn(hisi_ptt->pdev,
>>>>> +			 "filter update fifo overflow for target %s\n",
>>>>> +			 pci_name(info->pdev));
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * A PCI bus notifier is used here for dynamically updating the filter
>>>>> + * list.
>>>>> + */
>>>>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>>>>> +				  void *data)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>>>>> +	struct hisi_ptt_filter_update_info info;
>>>>> +	struct device *dev = data;
>>>>> +	struct pci_dev *pdev = to_pci_dev(dev);
>>>>> +
>>>>> +	info.pdev = pdev;
>>>>> +
>>>>> +	switch (action) {
>>>>> +	case BUS_NOTIFY_ADD_DEVICE:
>>>>> +		info.is_add = true;
>>>>> +		break;
>>>>> +	case BUS_NOTIFY_DEL_DEVICE:
>>>>> +		info.is_add = false;
>>>>> +		break;
>>>>> +	default:
>>>>> +		return 0;
>>>>> +	}
>>>>> +
>>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>>>>> +{
>>>>> +	struct hisi_ptt_filter_update_info info = {
>>>>> +		.pdev = pdev,
>>>>> +		.is_add = true,
>>>>> +	};
>>>>> +	struct hisi_ptt *hisi_ptt = data;
>>>>> +
>>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>>>>> +
>>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>>>>> +		list_del(&filter->list);
>>>>> +		kfree(filter);
>>>>> +	}
>>>>> +
>>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>>>>> +		list_del(&filter->list);
>>>>> +		kfree(filter);
>>>>> +	}
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	struct pci_bus *bus;
>>>>> +	u32 reg;
>>>>> +
>>>>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>>>>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>>>>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>>>>> +
>>>>> +	/*
>>>>> +	 * The device range register provides the information about the
>>>>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>>>>> +	 * and the root ports it support are on the same PCIe core, with
>>>>> +	 * same domain number but maybe different bus number. The device
>>>>> +	 * range register will tell us which root ports we can support,
>>>>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>>>>> +	 * while Bit[15:0] indicates the lower.
>>>>> +	 */
>>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>>>>> +	hisi_ptt->upper = reg >> 16;
>>>>> +	hisi_ptt->lower = reg & 0xffff;
>>>>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>>>>> +
>>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>>>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>>>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>>>> +
>>>>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>>>>> +	if (bus)
>>>>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>>>>
>>>> What happens if bus is NULL?  Should this be reported to the caller and cause
>>>> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
>>>>
>>>
>>> It's ok if the list is NULL in the probe process,
>>> as the device maybe hotplugged after the PTT driver probe, in which case
>>> we can detect the event and update the list as we register a bus notifier.
>>>
>>
>> Please add the above as a comment in the code.
>>
>>
>>>>> +
>>>>> +	/* Initialize trace controls */
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>>>>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>>>>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>>>>> +}
>>>>> +
>>>>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>>>>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>>>>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>>>>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>>>>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>>>>> +
>>>>> +static ssize_t available_filters_show(struct device *dev,
>>>>> +				      struct device_attribute *attr,
>>>>> +				      char *buf)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	int pos = 0;
>>>>> +
>>>>> +	if (list_empty(&hisi_ptt->port_filters))
>>>>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>>>>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>>>>> +				     pci_name(filter->pdev),
>>>>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>>>>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>>>>> +
>>>>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>>>>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>>>>> +				     pci_name(filter->pdev),
>>>>> +				     hisi_ptt_get_filter_val(filter->pdev));
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return pos;
>>>>> +}
>>>>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>>>>> +
>>>>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>>>>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>>>>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>>>>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>>>>> +
>>>>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>>>>> +	&format_attr_filter.attr,
>>>>> +	&format_attr_direction.attr,
>>>>> +	&format_attr_type.attr,
>>>>> +	&format_attr_format.attr,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>>>>> +	.name = "format",
>>>>> +	.attrs = hisi_ptt_pmu_format_attrs,
>>>>> +};
>>>>> +
>>>>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>>>>> +	&dev_attr_available_filters.attr,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>>>>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>>>>> +};
>>>>> +
>>>>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>>>>> +	&hisi_ptt_pmu_format_group,
>>>>> +	&hisi_ptt_pmu_filter_group,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +/*
>>>>> + * The supported value of the direction parameter. See hisi_ptt.rst
>>>>> + * documentation for more details.
>>>>> + */
>>>>> +static u32 hisi_ptt_trace_available_direction[] = {
>>>>> +	0,
>>>>> +	1,
>>>>> +	2,
>>>>> +	3,
>>>>> +};
>>>>> +
>>>>> +/* Different types can be set simultaneously */
>>>>> +static u32 hisi_ptt_trace_available_type[] = {
>>>>> +	1,	/* posted_request */
>>>>> +	2,	/* non-posted_request */
>>>>> +	4,	/* completion */
>>>>> +};
>>>>> +
>>>>> +static u32 hisi_ptt_trace_availble_format[] = {
>>>>> +	0,	/* 4DW */
>>>>> +	1,	/* 8DW */
>>>>> +};
>>>>> +
>>>>> +/*
>>>>> + * Check whether the config is valid or not. Some configs are multi-selectable
>>>>> + * and can be set simultaneously, while some are single selectable (onehot).
>>>>> + * Use this function to check the non-onehot configs while
>>>>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>>>>> + */
>>>>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>>>>> +{
>>>>> +	int i;
>>>>> +
>>>>> +	/*
>>>>> +	 * The non-onehot configs cannot be 0. Walk the available
>>>>> +	 * list and clear the valid bits of the config. If there
>>>>> +	 * is any resident bit after the walk then the config is
>>>>> +	 * invalid.
>>>>> +	 */
>>>>> +	for (i = 0; i < list_size; i++)
>>>>> +		val &= ~available_list[i];
>>>>> +
>>>>> +	return val ? -EINVAL : 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>>>>> +{
>>>>> +	int i, ret = -EINVAL;
>>>>> +
>>>>> +	for (i = 0; i < list_size; i++)
>>>>> +		if (val == available_list[i]) {
>>>>> +			ret = 0;
>>>>> +			break;
>>>>> +		}
>>>>> +
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>>>>> +{
>>>>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	int ret = -EINVAL;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>>>>> +
>>>>> +	/*
>>>>> +	 * Port filters are defined as bit mask. For port filters, check
>>>>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>>>>> +	 * and whether it's empty or not, otherwise user has specified
>>>>> +	 * some unsupported root ports.
>>>>> +	 *
>>>>> +	 * For Requester ID filters, walk the available filter list to see
>>>>> +	 * whether we have one matched.
>>>>> +	 */
>>>>> +	if (hisi_ptt->trace_ctrl.is_port &&
>>>>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>>>>> +		ret = 0;
>>>>> +	} else {
>>>>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>>> +			if (filter->val == val) {
>>>>> +				ret = 0;
>>>>> +				break;
>>>>> +			}
>>>>> +	}
>>>>> +
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.filter = val;
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	int ret;
>>>>> +	u32 val;
>>>>> +
>>>>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>>>>> +		return -ENOENT;
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +
>>>>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>>>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->direction = val;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>>>> +
>>>>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>>>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->type = val;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>>>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->format = val;
>>>>> +
>>>>> +out:
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>>>> +				    int nr_pages, bool overwrite)
>>>>> +{
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +	struct page **pagelist;
>>>>> +	int i;
>>>>> +
>>>>> +	if (overwrite) {
>>>>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	/* If the pages size less than buflets, we cannot start trace */
>>>>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>>>>> +		return NULL;
>>>>> +
>>>>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>>>> +	if (!buf)
>>>>> +		return NULL;
>>>>> +
>>>>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>>>> +	if (!pagelist) {
>>>>> +		kfree(buf);
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	for (i = 0; i < nr_pages; i++)
>>>>> +		pagelist[i] = virt_to_page(pages[i]);
>>>>> +
>>>>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>>>> +	if (!buf->base) {
>>>>> +		kfree(pagelist);
>>>>> +		kfree(buf);
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	buf->nr_pages = nr_pages;
>>>>> +	buf->length = nr_pages * PAGE_SIZE;
>>>>> +	buf->pos = 0;
>>>>> +
>>>>> +	kfree(pagelist);
>>>>> +	return buf;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>>>> +{
>>>>> +	struct hisi_ptt_pmu_buf *buf = aux;
>>>>> +
>>>>> +	vunmap(buf->base);
>>>>> +	kfree(buf);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +	int cpu = event->cpu;
>>>>> +	int ret;
>>>>> +
>>>>> +	hwc->state = 0;
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	if (cpu == -1)
>>>>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>>>>> +
>>>>> +	/*
>>>>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>>>>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>>>>> +	 * core in event_function_local().
>>>>> +	 */
>>>>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>>> +				 cpumask_of(cpu)));
>>>>> +
>>>>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	buf = perf_aux_output_begin(handle, event);
>>>>> +	if (!buf) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	buf->pos = handle->head % buf->length;
>>>>> +
>>>>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>>>> +		perf_aux_output_end(handle, 0);
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return;
>>>>> +stop:
>>>>> +	event->hw.state |= PERF_HES_STOPPED;
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +
>>>>> +	if (hwc->state & PERF_HES_STOPPED)
>>>>> +		return;
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>>>>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>>>>> +	}
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +
>>>>> +	hwc->state |= PERF_HES_STOPPED;
>>>>> +	perf_event_update_userpage(event);
>>>>> +	hwc->state |= PERF_HES_UPTODATE;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +	int cpu = event->cpu;
>>>>> +
>>>>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>>>>> +		return 0;
>>>>> +
>>>>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>>>>> +
>>>>> +	if (flags & PERF_EF_START) {
>>>>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>>>>> +		if (hwc->state & PERF_HES_STOPPED)
>>>>> +			return -EINVAL;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_unregister_pmu(void *priv)
>>>>> +{
>>>>> +	perf_pmu_unregister(priv);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	char *pmu_name;
>>>>> +	int ret;
>>>>> +
>>>>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>>>>> +		.module		= THIS_MODULE,
>>>>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>>>>> +		.task_ctx_nr	= perf_sw_context,
>>>>> +		.attr_groups	= hisi_ptt_pmu_groups,
>>>>> +		.event_init	= hisi_ptt_pmu_event_init,
>>>>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>>>>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>>>>> +		.start		= hisi_ptt_pmu_start,
>>>>> +		.stop		= hisi_ptt_pmu_stop,
>>>>> +		.add		= hisi_ptt_pmu_add,
>>>>> +		.del		= hisi_ptt_pmu_del,
>>>>> +	};
>>>>> +
>>>>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>>>>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>>>>> +	if (!pmu_name)
>>>>> +		return -ENOMEM;
>>>>> +
>>>>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>>>>> +					hisi_ptt_unregister_pmu,
>>>>> +					&hisi_ptt->hisi_ptt_pmu);
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Get RMR address if provided by the firmware.
>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>> + * Otherwise a negative value is returned.
>>>>> + */
>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	struct iommu_domain *iommu_domain;
>>>>> +	struct iommu_resv_region *region;
>>>>> +	LIST_HEAD(list);
>>>>> +
>>>>> +	/*
>>>>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>>>>> +	 * IOMMU domain is passthrough.
>>>>> +	 */
>>>>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>> +		return 0;
>>>>> +
>>>>> +	iommu_get_resv_regions(&pdev->dev, &list);
>>>>> +	list_for_each_entry(region, &list, list)
>>>>> +		if (region->type == IOMMU_RESV_DIRECT &&
>>>>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>>>>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>> +			break;
>>>>> +		}
>>>>> +
>>>>> +	iommu_put_resv_regions(&pdev->dev, &list);
>>>>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>>>>> +			  const struct pci_device_id *id)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt;
>>>>> +	int ret;
>>>>> +
>>>>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>>>>> +	if (!hisi_ptt)
>>>>> +		return -ENOMEM;
>>>>> +
>>>>> +	mutex_init(&hisi_ptt->mutex);
>>>>> +	hisi_ptt->pdev = pdev;
>>>>> +
>>>>> +	/*
>>>>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>>>>> +	 * so directly reference to the pci name string.
>>>>> +	 */
>>>>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>>>>> +	pci_set_drvdata(pdev, hisi_ptt);
>>>>> +
>>>>> +	ret = pcim_enable_device(pdev);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>>>>> +
>>>>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +	pci_set_master(pdev);
>>>>> +
>>>>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	hisi_ptt_init_ctrls(hisi_ptt);
>>>>> +
>>>>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>
>>>> If this fails the pci_free_irq_vectors() is not called.
>>>>
>>>
>>> It'll be called as we've registered a devm callback when
>>> allocating the irq vector. See hisi_ptt_register_irq().
>>
>> Ah yes - thanks for pointing that out.
>>
>>>
>>>> I am out of time for today - I will continue tomorrow.
>>>>
>>>
>>> Sure. Thanks!
>>> Yicong
>>>
>>>> Thanks,
>>>> Mathieu
>>>>
>>>>> +
>>>>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	/* Register the bus notifier for dynamically updating the filter list */
>>>>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>>>>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>>> +	if (ret)
>>>>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +void hisi_ptt_remove(struct pci_dev *pdev)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>>>> +
>>>>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>>> +
>>>>> +	/* Cancel any work that has been queued */
>>>>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>>>>> +
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +
>>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>>> +	hisi_ptt_release_filters(hisi_ptt);
>>>>> +}
>>>>> +
>>>>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>>>>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>>>>> +	{ }
>>>>> +};
>>>>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>>>>> +
>>>>> +static struct pci_driver hisi_ptt_driver = {
>>>>> +	.name = "hisi_ptt",
>>>>> +	.id_table = hisi_ptt_id_tbl,
>>>>> +	.probe = hisi_ptt_probe,
>>>>> +	.remove = hisi_ptt_remove,
>>>>> +};
>>>>> +module_pci_driver(hisi_ptt_driver);
>>>>> +
>>>>> +MODULE_LICENSE("GPL v2");
>>>>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>>>>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>>>>> -- 
>>>>> 2.33.0
>>>>>
>>>> .
>>>>
> .
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/27 1:10, Mathieu Poirier wrote:
> 
> [...]
> 
>>
>>>>> +
>>>>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>>>>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>>>>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>>>>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>>>>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>>>>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>>>>> +
>>>>> +/* Delay time for filter updating work */
>>>>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>>>>> +/* Wait time for DMA hardware to reset */
>>>>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>>>>> +/* Poll timeout and interval for waiting hardware work to finish */
>>>>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>>>>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>>>>> +
>>>>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>>>>> +
>>>>> +enum hisi_ptt_trace_status {
>>>>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>>>>> +	HISI_PTT_TRACE_STATUS_ON,
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_dma_buflet {
>>>>
>>>> I'm not sure what a "buflet" is...  Probably best to just call this
>>>> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
>>>> know due to the lack of proper structure documentation.  Please have a look at
>>>> how "coresight_device" is documented.  The same applies to the rest of the
>>>> structures declared below.
>>>>
>>>
>>> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
>>> (we call it buflet) of 4MiB ...".
>>
>> I just noticed the documentation in patch 5 - I will read it before continuing
>> with this set.
>>
> 
> As promised I read the documentation and it didn't do much to alliviate my
> concern about the terminology.  If a "buflet" is a DMA buffer than simply call
> it a DMA buffer, regardless of the size.  
> 

ok. If it's hard for the readers to understand, I'll use DMA buffer instead.

> The size of this patchset makes it hard enough to review as it is... On top
> of things reviewers have to remember that PTT means "PCIe tune and trace
> device", tune means "PCIe link's events" and trace means "TLP headers".
> 

ok. I used to make the complete function into one patch, will see how to split
it to make review easier.

Trace and tune denote the two function of the device. Trace denote the function
of tracing the TLP headers and tune denotes the function of read and modify
the PCIe link's events. It's unavoidable of knowing the acronyms for the user
of the device. :)

> But we also have to keep in mind that a buflet is really just a DMA buffer...
> 
> I will not continue reviewing this set.  Please fix the terminology and spin
> another revision.  Doing so I suggest you trim down the submission to the
> absolute minimum feature set this driver should provide - enhancements can be
> made later when a foundation has been laid out.
> 
> Robin has pointed out what seems to be serious problems with the
> implemenation.  That will also have to be sorted out for the next iteration.
> 
ok. Will discuss with robin with his concerns and with the comments fixed in
the next version.

Thanks,
Yicong

> Regards,
> Mathieu
> 
> 
>>>
>>> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
>>> a 'buflet', which means a small buffer (there exists other words with similiar
>>> format: droplet, wavelet, ...).
>>>
>>> The device is designed like this to make sure we won't lose any data when we have to change
>>> the buffer address, the device can continue to write to the next buflet and don't
>>> neet to pause.
>>>
>>>>> +	struct list_head list;
>>>>> +	unsigned int size;
>>>>> +	dma_addr_t dma;
>>>>> +
>>>>> +	/*
>>>>> +	 * The address of the buflet holding the trace data.
>>>>> +	 * See Documentation/trace/hisi-ptt.rst for the details
>>>>
>>>> As of this writing, hisi-ptt.rst doesn't exist.
>>>>
>>>
>>> The comment intends to direct the user to the documentation if the user
>>> want to know details about the data format. The data format is decribed
>>> in the section 4 of the doc.
>>>
>>>>> +	 * of the data format.
>>>>> +	 */
>>>>> +	void *addr;
>>>>> +	int index;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_trace_ctrl {
>>>>> +	enum hisi_ptt_trace_status status;
>>>>> +	struct perf_output_handle handle;
>>>>> +	struct list_head trace_buf;
>>>>> +	/*
>>>>> +	 * The index of the buffer which trace data
>>>>> +	 * currently is writing to.
>>>>> +	 */
>>>>> +	u32 buf_index;
>>>>> +
>>>>> +	int default_cpu;
>>>>> +	u32 buflet_size;
>>>>> +	bool is_port;
>>>>> +	u32 format:1;		/* Format of the traced TLP headers */
>>>>> +	u32 type:4;		/* Type of the TLP headers to trace */
>>>>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>>>>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>>>>> +
>>>>> +	phys_addr_t rmr_addr;
>>>>> +	size_t rmr_length;
>>>>> +	bool has_rmr;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_filter_desc {
>>>>> +	struct list_head list;
>>>>> +	struct pci_dev *pdev;
>>>>> +	u16 val;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt_pmu_buf {
>>>>> +	size_t length;
>>>>> +	int nr_pages;
>>>>> +	void *base;
>>>>> +	long pos;
>>>>> +};
>>>>> +
>>>>> +/* Structure containing the information for filter updating */
>>>>> +struct hisi_ptt_filter_update_info {
>>>>> +	struct pci_dev *pdev;
>>>>> +	u32 port_devid;
>>>>> +	bool is_port;
>>>>> +	bool is_add;
>>>>> +	u16 val;
>>>>> +};
>>>>> +
>>>>> +struct hisi_ptt {
>>>>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>>>>> +	struct notifier_block hisi_ptt_nb;
>>>>> +	struct pmu hisi_ptt_pmu;
>>>>> +	void __iomem *iobase;
>>>>> +	struct pci_dev *pdev;
>>>>> +	/*
>>>>> +	 * Use the mutex to protect the filter list and
>>>>> +	 * serialize the perf process.
>>>>> +	 */
>>>>> +	struct mutex mutex;
>>>>> +	const char *name;
>>>>> +	u16 core_id;
>>>>> +	u16 sicl_id;
>>>>> +	/* PCI device range managed by the PTT device */
>>>>> +	u32 upper;
>>>>> +	u32 lower;
>>>>> +	u8 busnr;
>>>>> +
>>>>> +	/*
>>>>> +	 * The trace TLP headers can either be filtered by certain
>>>>> +	 * root port, or by the requester ID. Organize the filters
>>>>> +	 * by @port_filters and @req_filters here. The mask of all
>>>>> +	 * the valid ports is also cached for doing sanity check
>>>>> +	 * of user input.
>>>>> +	 */
>>>>> +	struct list_head port_filters;
>>>>> +	struct list_head req_filters;
>>>>> +	u16 port_mask;
>>>>> +
>>>>> +	/*
>>>>> +	 * We use a delayed work here to avoid indefinitely waiting for
>>>>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>>>>> +	 * work will be delayed only if the mutex can not be held,
>>>>> +	 * otherwise no delay will be applied.
>>>>> +	 */
>>>>> +	struct delayed_work work;
>>>>> +	spinlock_t filter_update_lock;
>>>>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>>>>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>>>>> +};
>>>>
>>>> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
>>>>
>>>
>>> I'm not sure whether it is common or permitted to provide a header which is only used by
>>> this driver with a single file.
>>
>> I've seen it done many times.
>>
>>>
>>>>> +
>>>>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>>>>> +{
>>>>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>>>>> +}
>>>>
>>>> I'm not sure there is any value to this inline function.  I suggest to simply
>>>> call container_of() whenever it is needed.
>>>>
>>>
>>> ok.
>>>
>>>>> +
>>>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>>>> +{
>>>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>>>> +
>>>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	u32 val;
>>>>> +
>>>>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>>>>> +				  val & HISI_PTT_TRACE_IDLE,
>>>>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>>>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>>>>> +
>>>>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>>>>> +		list_del(&buflet->list);
>>>>> +
>>>>> +		if (ctrl->has_rmr)
>>>>> +			memunmap(buflet->addr);
>>>>> +		else
>>>>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>>>>> +					  buflet->dma);
>>>>> +
>>>>> +		kfree(buflet);
>>>>> +	}
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>>> +	struct hisi_ptt_dma_buflet *buflet;
>>>>> +	int i, ret;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>>>> +
>>>>> +	/* Make sure the trace buffer is empty before allocating */
>>>>> +	if (!list_empty(&ctrl->trace_buf)) {
>>>>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>>>>> +			memset(buflet->addr, 0, buflet->size);
>>>>> +		return 0;
>>>>> +	}
>>>>> +
>>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>>>>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>>>>> +		if (!buflet) {
>>>>> +			ret = -ENOMEM;
>>>>> +			goto err;
>>>>> +		}
>>>>> +
>>>>> +		if (ctrl->has_rmr) {
>>>>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>>>>> +
>>>>> +			buflet->dma = base;
>>>>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>>>>> +		} else {
>>>>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>>>>> +							  &buflet->dma, GFP_KERNEL);
>>>>> +		}
>>>>> +
>>>>> +		if (!buflet->addr) {
>>>>> +			kfree(buflet);
>>>>> +			ret = -ENOMEM;
>>>>> +			goto err;
>>>>> +		}
>>>>> +
>>>>> +		memset(buflet->addr, 0, buflet->size);
>>>>> +
>>>>> +		buflet->index = i;
>>>>> +		buflet->size = ctrl->buflet_size;
>>>>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +err:
>>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>>> +	u32 val;
>>>>> +
>>>>> +	/* Check device idle before start trace */
>>>>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>>>>> +		return -EBUSY;
>>>>> +	}
>>>>> +
>>>>> +	/* Reset the DMA before start tracing */
>>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	/*
>>>>> +	 * We'll be in the perf context where preemption is disabled,
>>>>> +	 * so use busy loop here.
>>>>> +	 */
>>>>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>>>>> +
>>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	/* clear the interrupt status */
>>>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>>> +
>>>>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>>>>> +		writel(lower_32_bits(cur->dma),
>>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>>> +		writel(upper_32_bits(cur->dma),
>>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>>> +	}
>>>>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>>>> +
>>>>> +	/* set the trace control register */
>>>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>>>> +
>>>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>>> +
>>>>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>>>> +{
>>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>>> +	struct perf_event *event = handle->event;
>>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +
>>>>> +	buf = perf_get_aux(handle);
>>>>> +	if (!buf || !handle->size)
>>>>> +		return -EINVAL;
>>>>> +
>>>>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>>>>> +		if (cur->index == index)
>>>>> +			break;
>>>>> +
>>>>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>>>>> +	memset(cur->addr, 0, cur->size);
>>>>> +	buf->pos += cur->size;
>>>>> +
>>>>> +	if (stop) {
>>>>> +		perf_aux_output_end(handle, buf->pos);
>>>>> +	} else if (buf->length - buf->pos < cur->size) {
>>>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>>>> +		perf_aux_output_end(handle, buf->pos);
>>>>> +
>>>>> +		buf = perf_aux_output_begin(handle, event);
>>>>> +		if (!buf)
>>>>> +			return -EINVAL;
>>>>> +
>>>>> +		buf->pos = handle->head % buf->length;
>>>>> +		if (buf->length - buf->pos < cur->size) {
>>>>> +			perf_aux_output_end(handle, 0);
>>>>> +			return -EINVAL;
>>>>> +		}
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>>> +	u32 status, buf_idx;
>>>>> +
>>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	buf_idx = ffs(status) - 1;
>>>>> +
>>>>> +	/* Clear the interrupt status of buflet @buf_idx */
>>>>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +
>>>>> +	/*
>>>>> +	 * Update the AUX buffer and cache the current buflet index,
>>>>> +	 * as we need to know this and save the data when the trace
>>>>> +	 * is ended out of the interrupt handler. End the trace
>>>>> +	 * if the updating fails.
>>>>> +	 */
>>>>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +	else
>>>>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>>>>> +
>>>>> +	return IRQ_HANDLED;
>>>>> +}
>>>>> +
>>>>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>>> +	u32 status;
>>>>> +
>>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>>>>> +		return IRQ_NONE;
>>>>> +
>>>>> +	return IRQ_WAKE_THREAD;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>>>>> +{
>>>>> +	pci_free_irq_vectors(pdev);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	int ret;
>>>>> +
>>>>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>>>>> +	if (ret < 0) {
>>>>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>>>>> +	if (ret < 0)
>>>>> +		return ret;
>>>>> +
>>>>> +	ret = devm_request_threaded_irq(&pdev->dev,
>>>>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>>>>> +					"hisi-ptt", hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>>>>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_update_filters(struct work_struct *work)
>>>>> +{
>>>>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>>>>> +	struct hisi_ptt_filter_update_info info;
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	struct list_head *target_list;
>>>>> +	struct hisi_ptt *hisi_ptt;
>>>>> +
>>>>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>>>>> +
>>>>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>>>>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>>>>> +		return;
>>>>> +	}
>>>>> +
>>>>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>>>>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>>>>> +			      &hisi_ptt->req_filters;
>>>>> +
>>>>> +		if (info.is_add) {
>>>>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>>>>> +			if (!filter)
>>>>> +				continue;
>>>>> +
>>>>> +			filter->pdev = info.pdev;
>>>>> +			filter->val = info.val;
>>>>> +
>>>>> +			list_add_tail(&filter->list, target_list);
>>>>> +		} else {
>>>>> +			list_for_each_entry(filter, target_list, list)
>>>>> +				if (filter->val == info.val) {
>>>>> +					list_del(&filter->list);
>>>>> +					kfree(filter);
>>>>> +					break;
>>>>> +				}
>>>>> +		}
>>>>> +
>>>>> +		/* Update the available port mask */
>>>>> +		if (!info.is_port)
>>>>> +			continue;
>>>>> +
>>>>> +		if (info.is_add)
>>>>> +			hisi_ptt->port_mask |= info.val;
>>>>> +		else
>>>>> +			hisi_ptt->port_mask &= ~info.val;
>>>>> +	}
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>>>>> +				    struct hisi_ptt_filter_update_info *info)
>>>>> +{
>>>>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>>>>> +
>>>>> +	if (!root_port)
>>>>> +		return;
>>>>> +
>>>>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>>>>> +	if (info->port_devid < hisi_ptt->lower ||
>>>>> +	    info->port_devid > hisi_ptt->upper)
>>>>> +		return;
>>>>> +
>>>>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>>>>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>>>>> +
>>>>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>>>>> +				&hisi_ptt->filter_update_lock))
>>>>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>>>>> +	else
>>>>> +		pci_warn(hisi_ptt->pdev,
>>>>> +			 "filter update fifo overflow for target %s\n",
>>>>> +			 pci_name(info->pdev));
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * A PCI bus notifier is used here for dynamically updating the filter
>>>>> + * list.
>>>>> + */
>>>>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>>>>> +				  void *data)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>>>>> +	struct hisi_ptt_filter_update_info info;
>>>>> +	struct device *dev = data;
>>>>> +	struct pci_dev *pdev = to_pci_dev(dev);
>>>>> +
>>>>> +	info.pdev = pdev;
>>>>> +
>>>>> +	switch (action) {
>>>>> +	case BUS_NOTIFY_ADD_DEVICE:
>>>>> +		info.is_add = true;
>>>>> +		break;
>>>>> +	case BUS_NOTIFY_DEL_DEVICE:
>>>>> +		info.is_add = false;
>>>>> +		break;
>>>>> +	default:
>>>>> +		return 0;
>>>>> +	}
>>>>> +
>>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>>>>> +{
>>>>> +	struct hisi_ptt_filter_update_info info = {
>>>>> +		.pdev = pdev,
>>>>> +		.is_add = true,
>>>>> +	};
>>>>> +	struct hisi_ptt *hisi_ptt = data;
>>>>> +
>>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>>>>> +
>>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>>>>> +		list_del(&filter->list);
>>>>> +		kfree(filter);
>>>>> +	}
>>>>> +
>>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>>>>> +		list_del(&filter->list);
>>>>> +		kfree(filter);
>>>>> +	}
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	struct pci_bus *bus;
>>>>> +	u32 reg;
>>>>> +
>>>>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>>>>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>>>>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>>>>> +
>>>>> +	/*
>>>>> +	 * The device range register provides the information about the
>>>>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>>>>> +	 * and the root ports it support are on the same PCIe core, with
>>>>> +	 * same domain number but maybe different bus number. The device
>>>>> +	 * range register will tell us which root ports we can support,
>>>>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>>>>> +	 * while Bit[15:0] indicates the lower.
>>>>> +	 */
>>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>>>>> +	hisi_ptt->upper = reg >> 16;
>>>>> +	hisi_ptt->lower = reg & 0xffff;
>>>>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>>>>> +
>>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>>>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>>>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>>>> +
>>>>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>>>>> +	if (bus)
>>>>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>>>>
>>>> What happens if bus is NULL?  Should this be reported to the caller and cause
>>>> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
>>>>
>>>
>>> It's ok if the list is NULL in the probe process,
>>> as the device maybe hotplugged after the PTT driver probe, in which case
>>> we can detect the event and update the list as we register a bus notifier.
>>>
>>
>> Please add the above as a comment in the code.
>>
>>
>>>>> +
>>>>> +	/* Initialize trace controls */
>>>>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>>>>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>>>>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>>>>> +}
>>>>> +
>>>>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>>>>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>>>>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>>>>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>>>>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>>>>> +
>>>>> +static ssize_t available_filters_show(struct device *dev,
>>>>> +				      struct device_attribute *attr,
>>>>> +				      char *buf)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	int pos = 0;
>>>>> +
>>>>> +	if (list_empty(&hisi_ptt->port_filters))
>>>>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>>>>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>>>>> +				     pci_name(filter->pdev),
>>>>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>>>>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>>>>> +
>>>>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>>>>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>>>>> +				     pci_name(filter->pdev),
>>>>> +				     hisi_ptt_get_filter_val(filter->pdev));
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return pos;
>>>>> +}
>>>>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>>>>> +
>>>>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>>>>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>>>>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>>>>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>>>>> +
>>>>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>>>>> +	&format_attr_filter.attr,
>>>>> +	&format_attr_direction.attr,
>>>>> +	&format_attr_type.attr,
>>>>> +	&format_attr_format.attr,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>>>>> +	.name = "format",
>>>>> +	.attrs = hisi_ptt_pmu_format_attrs,
>>>>> +};
>>>>> +
>>>>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>>>>> +	&dev_attr_available_filters.attr,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>>>>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>>>>> +};
>>>>> +
>>>>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>>>>> +	&hisi_ptt_pmu_format_group,
>>>>> +	&hisi_ptt_pmu_filter_group,
>>>>> +	NULL
>>>>> +};
>>>>> +
>>>>> +/*
>>>>> + * The supported value of the direction parameter. See hisi_ptt.rst
>>>>> + * documentation for more details.
>>>>> + */
>>>>> +static u32 hisi_ptt_trace_available_direction[] = {
>>>>> +	0,
>>>>> +	1,
>>>>> +	2,
>>>>> +	3,
>>>>> +};
>>>>> +
>>>>> +/* Different types can be set simultaneously */
>>>>> +static u32 hisi_ptt_trace_available_type[] = {
>>>>> +	1,	/* posted_request */
>>>>> +	2,	/* non-posted_request */
>>>>> +	4,	/* completion */
>>>>> +};
>>>>> +
>>>>> +static u32 hisi_ptt_trace_availble_format[] = {
>>>>> +	0,	/* 4DW */
>>>>> +	1,	/* 8DW */
>>>>> +};
>>>>> +
>>>>> +/*
>>>>> + * Check whether the config is valid or not. Some configs are multi-selectable
>>>>> + * and can be set simultaneously, while some are single selectable (onehot).
>>>>> + * Use this function to check the non-onehot configs while
>>>>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>>>>> + */
>>>>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>>>>> +{
>>>>> +	int i;
>>>>> +
>>>>> +	/*
>>>>> +	 * The non-onehot configs cannot be 0. Walk the available
>>>>> +	 * list and clear the valid bits of the config. If there
>>>>> +	 * is any resident bit after the walk then the config is
>>>>> +	 * invalid.
>>>>> +	 */
>>>>> +	for (i = 0; i < list_size; i++)
>>>>> +		val &= ~available_list[i];
>>>>> +
>>>>> +	return val ? -EINVAL : 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>>>>> +{
>>>>> +	int i, ret = -EINVAL;
>>>>> +
>>>>> +	for (i = 0; i < list_size; i++)
>>>>> +		if (val == available_list[i]) {
>>>>> +			ret = 0;
>>>>> +			break;
>>>>> +		}
>>>>> +
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>>>>> +{
>>>>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>>>>> +	struct hisi_ptt_filter_desc *filter;
>>>>> +	int ret = -EINVAL;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>>>>> +
>>>>> +	/*
>>>>> +	 * Port filters are defined as bit mask. For port filters, check
>>>>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>>>>> +	 * and whether it's empty or not, otherwise user has specified
>>>>> +	 * some unsupported root ports.
>>>>> +	 *
>>>>> +	 * For Requester ID filters, walk the available filter list to see
>>>>> +	 * whether we have one matched.
>>>>> +	 */
>>>>> +	if (hisi_ptt->trace_ctrl.is_port &&
>>>>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>>>>> +		ret = 0;
>>>>> +	} else {
>>>>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>>> +			if (filter->val == val) {
>>>>> +				ret = 0;
>>>>> +				break;
>>>>> +			}
>>>>> +	}
>>>>> +
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	hisi_ptt->trace_ctrl.filter = val;
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>>> +	int ret;
>>>>> +	u32 val;
>>>>> +
>>>>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>>>>> +		return -ENOENT;
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +
>>>>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>>>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->direction = val;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>>>> +
>>>>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>>>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->type = val;
>>>>> +
>>>>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>>>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>>>> +	if (ret < 0)
>>>>> +		goto out;
>>>>> +	ctrl->format = val;
>>>>> +
>>>>> +out:
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>>>> +				    int nr_pages, bool overwrite)
>>>>> +{
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +	struct page **pagelist;
>>>>> +	int i;
>>>>> +
>>>>> +	if (overwrite) {
>>>>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	/* If the pages size less than buflets, we cannot start trace */
>>>>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>>>>> +		return NULL;
>>>>> +
>>>>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>>>> +	if (!buf)
>>>>> +		return NULL;
>>>>> +
>>>>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>>>> +	if (!pagelist) {
>>>>> +		kfree(buf);
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	for (i = 0; i < nr_pages; i++)
>>>>> +		pagelist[i] = virt_to_page(pages[i]);
>>>>> +
>>>>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>>>> +	if (!buf->base) {
>>>>> +		kfree(pagelist);
>>>>> +		kfree(buf);
>>>>> +		return NULL;
>>>>> +	}
>>>>> +
>>>>> +	buf->nr_pages = nr_pages;
>>>>> +	buf->length = nr_pages * PAGE_SIZE;
>>>>> +	buf->pos = 0;
>>>>> +
>>>>> +	kfree(pagelist);
>>>>> +	return buf;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>>>> +{
>>>>> +	struct hisi_ptt_pmu_buf *buf = aux;
>>>>> +
>>>>> +	vunmap(buf->base);
>>>>> +	kfree(buf);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>>> +	int cpu = event->cpu;
>>>>> +	int ret;
>>>>> +
>>>>> +	hwc->state = 0;
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	if (cpu == -1)
>>>>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>>>>> +
>>>>> +	/*
>>>>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>>>>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>>>>> +	 * core in event_function_local().
>>>>> +	 */
>>>>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>>> +				 cpumask_of(cpu)));
>>>>> +
>>>>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	buf = perf_aux_output_begin(handle, event);
>>>>> +	if (!buf) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	buf->pos = handle->head % buf->length;
>>>>> +
>>>>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>>>> +		perf_aux_output_end(handle, 0);
>>>>> +		goto stop;
>>>>> +	}
>>>>> +
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +	return;
>>>>> +stop:
>>>>> +	event->hw.state |= PERF_HES_STOPPED;
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +
>>>>> +	if (hwc->state & PERF_HES_STOPPED)
>>>>> +		return;
>>>>> +
>>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>>>>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>>>>> +	}
>>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>>> +
>>>>> +	hwc->state |= PERF_HES_STOPPED;
>>>>> +	perf_event_update_userpage(event);
>>>>> +	hwc->state |= PERF_HES_UPTODATE;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>>> +	int cpu = event->cpu;
>>>>> +
>>>>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>>>>> +		return 0;
>>>>> +
>>>>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>>>>> +
>>>>> +	if (flags & PERF_EF_START) {
>>>>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>>>>> +		if (hwc->state & PERF_HES_STOPPED)
>>>>> +			return -EINVAL;
>>>>> +	}
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>>>>> +{
>>>>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>>>>> +}
>>>>> +
>>>>> +static void hisi_ptt_unregister_pmu(void *priv)
>>>>> +{
>>>>> +	perf_pmu_unregister(priv);
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	char *pmu_name;
>>>>> +	int ret;
>>>>> +
>>>>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>>>>> +		.module		= THIS_MODULE,
>>>>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>>>>> +		.task_ctx_nr	= perf_sw_context,
>>>>> +		.attr_groups	= hisi_ptt_pmu_groups,
>>>>> +		.event_init	= hisi_ptt_pmu_event_init,
>>>>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>>>>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>>>>> +		.start		= hisi_ptt_pmu_start,
>>>>> +		.stop		= hisi_ptt_pmu_stop,
>>>>> +		.add		= hisi_ptt_pmu_add,
>>>>> +		.del		= hisi_ptt_pmu_del,
>>>>> +	};
>>>>> +
>>>>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>>>>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>>>>> +	if (!pmu_name)
>>>>> +		return -ENOMEM;
>>>>> +
>>>>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>>>>> +					hisi_ptt_unregister_pmu,
>>>>> +					&hisi_ptt->hisi_ptt_pmu);
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Get RMR address if provided by the firmware.
>>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>>> + * Otherwise a negative value is returned.
>>>>> + */
>>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>>> +{
>>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>>> +	struct iommu_domain *iommu_domain;
>>>>> +	struct iommu_resv_region *region;
>>>>> +	LIST_HEAD(list);
>>>>> +
>>>>> +	/*
>>>>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>>>>> +	 * IOMMU domain is passthrough.
>>>>> +	 */
>>>>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>>> +		return 0;
>>>>> +
>>>>> +	iommu_get_resv_regions(&pdev->dev, &list);
>>>>> +	list_for_each_entry(region, &list, list)
>>>>> +		if (region->type == IOMMU_RESV_DIRECT &&
>>>>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>>>>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>>> +			break;
>>>>> +		}
>>>>> +
>>>>> +	iommu_put_resv_regions(&pdev->dev, &list);
>>>>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>>> +}
>>>>> +
>>>>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>>>>> +			  const struct pci_device_id *id)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt;
>>>>> +	int ret;
>>>>> +
>>>>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>>>>> +	if (!hisi_ptt)
>>>>> +		return -ENOMEM;
>>>>> +
>>>>> +	mutex_init(&hisi_ptt->mutex);
>>>>> +	hisi_ptt->pdev = pdev;
>>>>> +
>>>>> +	/*
>>>>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>>>>> +	 * so directly reference to the pci name string.
>>>>> +	 */
>>>>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>>>>> +	pci_set_drvdata(pdev, hisi_ptt);
>>>>> +
>>>>> +	ret = pcim_enable_device(pdev);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>>>>> +
>>>>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +	pci_set_master(pdev);
>>>>> +
>>>>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>>>>> +	if (ret)
>>>>> +		return ret;
>>>>> +
>>>>> +	hisi_ptt_init_ctrls(hisi_ptt);
>>>>> +
>>>>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>
>>>> If this fails the pci_free_irq_vectors() is not called.
>>>>
>>>
>>> It'll be called as we've registered a devm callback when
>>> allocating the irq vector. See hisi_ptt_register_irq().
>>
>> Ah yes - thanks for pointing that out.
>>
>>>
>>>> I am out of time for today - I will continue tomorrow.
>>>>
>>>
>>> Sure. Thanks!
>>> Yicong
>>>
>>>> Thanks,
>>>> Mathieu
>>>>
>>>>> +
>>>>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>>>>> +	if (ret) {
>>>>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>>>>> +		return ret;
>>>>> +	}
>>>>> +
>>>>> +	/* Register the bus notifier for dynamically updating the filter list */
>>>>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>>>>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>>> +	if (ret)
>>>>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>> +void hisi_ptt_remove(struct pci_dev *pdev)
>>>>> +{
>>>>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>>>> +
>>>>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>>> +
>>>>> +	/* Cancel any work that has been queued */
>>>>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>>>>> +
>>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>>> +
>>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>>> +	hisi_ptt_release_filters(hisi_ptt);
>>>>> +}
>>>>> +
>>>>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>>>>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>>>>> +	{ }
>>>>> +};
>>>>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>>>>> +
>>>>> +static struct pci_driver hisi_ptt_driver = {
>>>>> +	.name = "hisi_ptt",
>>>>> +	.id_table = hisi_ptt_id_tbl,
>>>>> +	.probe = hisi_ptt_probe,
>>>>> +	.remove = hisi_ptt_remove,
>>>>> +};
>>>>> +module_pci_driver(hisi_ptt_driver);
>>>>> +
>>>>> +MODULE_LICENSE("GPL v2");
>>>>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>>>>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>>>>> -- 
>>>>> 2.33.0
>>>>>
>>>> .
>>>>
> .
> 

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

On 2021/11/26 2:50, Mathieu Poirier wrote:
> On Thu, Nov 25, 2021 at 04:39:46PM +0800, Yicong Yang wrote:
>> Hi Mathieu,
>>
>> Thanks for the comments! Replies inline.
>>
>> On 2021/11/25 2:51, Mathieu Poirier wrote:
>>> Hi Yicong,
>>>
>>> On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
>>>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>>>> integrated Endpoint(RCiEP) device, providing the capability
>>>> to dynamically monitor and tune the PCIe traffic(tune),
>>>> and trace the TLP headers(trace).
>>>>
>>>
>>> Is there a reason to put "tune" and "trace" are whithin parentheses?  
>>>
>>
>> yeah. I want to use these single word to denote the related function, and
>> place them in the parantheses near the function's description to make sure
>> readers can match them.
>>
> 
> Probably a better idea to simply write "TLP headers" and "PCIe traffic" when
> referring to them.  Otherwise it is very confusing because the reader doesn't
> know if you mean the acronym or the real definition of "tune" and "trace".
> 

ok I reread the commit and seems reader will know the acronym as it's mentioned
as "*tune* the PCIe traffic" and "*trace* the TLP headers". I think the
contents in the parantheses can be simply dropped.

>>>> Add the driver for the device to enable the trace function. The driver
>>>> will create PMU device for each PTT device, and users can start trace
>>>> through perf command.
>>>>
>>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>>> ---
>>>>  drivers/Makefile                       |    1 +
>>>>  drivers/hwtracing/Kconfig              |    2 +
>>>>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>>>>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>>>>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
>>>
>>> Not sure about the "hisilicon" directory.  Right now we have "coresight",
>>> "intel_th" and "stm" - they concentrate on the technology rather than the
>>> vendor.  I think "ptt" would be just fine: 
>>>
>>> drivers/hwtracing/ptt/Kconfig 
>>> drivers/hwtracing/ptt/Makefile
>>> drivers/hwtracing/ptt/hisi_ptt.c
>>>
>>> That way if other vendors want to introduce support for the same kind of
>>> technology we can simply put them under "drivers/hwtracing/ptt/" and things are
>>> still accurate.
>>>
>>
>> sure. sounds sensible. will rename the directory to "ptt" if no objection.
>>
>>>
>>>>  5 files changed, 1159 insertions(+)
>>>>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>>>>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>>>>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
>>>>
>>>> diff --git a/drivers/Makefile b/drivers/Makefile
>>>> index be5d40ae1488..bb0dc9b55ea2 100644
>>>> --- a/drivers/Makefile
>>>> +++ b/drivers/Makefile
>>>> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>>>  obj-y				+= hwtracing/intel_th/
>>>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>>>> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>>>>  obj-$(CONFIG_ANDROID)		+= android/
>>>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>>>  obj-$(CONFIG_FPGA)		+= fpga/
>>>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>>>> index 13085835a636..e3796b17541a 100644
>>>> --- a/drivers/hwtracing/Kconfig
>>>> +++ b/drivers/hwtracing/Kconfig
>>>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>>>  
>>>>  source "drivers/hwtracing/intel_th/Kconfig"
>>>>  
>>>> +source "drivers/hwtracing/hisilicon/Kconfig"
>>>> +
>>>>  endmenu
>>>> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..9c3ab80d99fe
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/Kconfig
>>>> @@ -0,0 +1,8 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +config HISI_PTT
>>>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>>>> +	depends on PCI && HAS_DMA && HAS_IOMEM
>>>
>>> What Arm architecture are you aiming this for?  Can it be found on both armv7
>>> and armv8?
>>>
>>
>> currently the device will appear on Kunpeng 930, which is armv8.
> 
> Ok, then please make it dependent on armv8.
> 

sure.

>>
>>>> +	help
>>>> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
>>>> +	  device, provides support for PCIe traffic tuning and
>>>> +	  tracing TLP headers to the memory.
>>>
>>> Please indicate what the name of the driver will be when compiled as a module.
>>>
>>
>> sure. will add the module's name here.
>>
>>>> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
>>>> new file mode 100644
>>>> index 000000000000..908c09a98161
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/Makefile
>>>> @@ -0,0 +1,2 @@
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>>>> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
>>>> new file mode 100644
>>>> index 000000000000..e11e9b6cc2a8
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
>>>> @@ -0,0 +1,1146 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * Driver for HiSilicon PCIe tune and trace device
>>>> + *
>>>> + * Copyright (c) 2021 HiSilicon Limited.
>>>> + */
>>>> +
>>>> +#include <linux/bitfield.h>
>>>> +#include <linux/bitops.h>
>>>> +#include <linux/delay.h>
>>>> +#include <linux/dma-iommu.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/io.h>
>>>> +#include <linux/iommu.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/kfifo.h>
>>>> +#include <linux/module.h>
>>>> +#include <linux/notifier.h>
>>>> +#include <linux/pci.h>
>>>> +#include <linux/perf_event.h>
>>>> +#include <linux/sizes.h>
>>>> +#include <linux/sysfs.h>
>>>> +#include <linux/vmalloc.h>
>>>> +#include <linux/workqueue.h>
>>>> +
>>>> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
>>>> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
>>>> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
>>>> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
>>>> +#define HISI_PTT_TRACE_CTRL		0x0850
>>>> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
>>>> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
>>>> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
>>>> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
>>>> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
>>>> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
>>>> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
>>>> +#define HISI_PTT_TRACE_INT_STAT		0x0890
>>>> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
>>>> +#define HISI_PTT_TRACE_INT_MASK		0x0894
>>>> +#define HISI_PTT_TRACE_WR_STS		0x08a0
>>>> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
>>>> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
>>>> +#define HISI_PTT_TRACE_STS		0x08b0
>>>> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
>>>> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
>>>> +#define HISI_PTT_LOCATION		0x0fe8
>>>> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
>>>> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
>>>
>>> Extra spaces between #define and the literals.  Please align the values with
>>> that of HISI_PTT_TRACE_DMA_IRQ below.
>>>
>>
>> Spaces are intended to distinguish the definition of the register offset and
>> related register fields. also used in include/uapi/linux/pci_regs.h.
>>
> 
> Interesting and definitely not the norm...
> 
>> HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
>> in the mail text, maybe because the tab is not 8 characters when display.
>>
> 
> In my VI terminal the '0' of HISI_PTT_TRACE_DMA_IRQ is align with the '3' in
> GENMASK(31, 16) and I suspect it is the same for other people.
> 

not sure what happens, with tab=8char it's aligned on my vim and vscode. and yes
it's also aligned with '3' in GENMASK(31,16).

#define HISI_PTT_TRACE_DMA_IRQ\t\t\t0
#define HISI_PTT_TRACE_BUFLETS_CNT\t\t4

>>>> +
>>>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>>>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>>>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>>>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>>>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>>>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>>>> +
>>>> +/* Delay time for filter updating work */
>>>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>>>> +/* Wait time for DMA hardware to reset */
>>>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>>>> +/* Poll timeout and interval for waiting hardware work to finish */
>>>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>>>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>>>> +
>>>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>>>> +
>>>> +enum hisi_ptt_trace_status {
>>>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>>>> +	HISI_PTT_TRACE_STATUS_ON,
>>>> +};
>>>> +
>>>> +struct hisi_ptt_dma_buflet {
>>>
>>> I'm not sure what a "buflet" is...  Probably best to just call this
>>> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
>>> know due to the lack of proper structure documentation.  Please have a look at
>>> how "coresight_device" is documented.  The same applies to the rest of the
>>> structures declared below.
>>>
>>
>> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
>> (we call it buflet) of 4MiB ...".
> 
> I just noticed the documentation in patch 5 - I will read it before continuing
> with this set.
> 
>>
>> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
>> a 'buflet', which means a small buffer (there exists other words with similiar
>> format: droplet, wavelet, ...).
>>
>> The device is designed like this to make sure we won't lose any data when we have to change
>> the buffer address, the device can continue to write to the next buflet and don't
>> neet to pause.
>>
>>>> +	struct list_head list;
>>>> +	unsigned int size;
>>>> +	dma_addr_t dma;
>>>> +
>>>> +	/*
>>>> +	 * The address of the buflet holding the trace data.
>>>> +	 * See Documentation/trace/hisi-ptt.rst for the details
>>>
>>> As of this writing, hisi-ptt.rst doesn't exist.
>>>
>>
>> The comment intends to direct the user to the documentation if the user
>> want to know details about the data format. The data format is decribed
>> in the section 4 of the doc.
>>
>>>> +	 * of the data format.
>>>> +	 */
>>>> +	void *addr;
>>>> +	int index;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_trace_ctrl {
>>>> +	enum hisi_ptt_trace_status status;
>>>> +	struct perf_output_handle handle;
>>>> +	struct list_head trace_buf;
>>>> +	/*
>>>> +	 * The index of the buffer which trace data
>>>> +	 * currently is writing to.
>>>> +	 */
>>>> +	u32 buf_index;
>>>> +
>>>> +	int default_cpu;
>>>> +	u32 buflet_size;
>>>> +	bool is_port;
>>>> +	u32 format:1;		/* Format of the traced TLP headers */
>>>> +	u32 type:4;		/* Type of the TLP headers to trace */
>>>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>>>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>>>> +
>>>> +	phys_addr_t rmr_addr;
>>>> +	size_t rmr_length;
>>>> +	bool has_rmr;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_filter_desc {
>>>> +	struct list_head list;
>>>> +	struct pci_dev *pdev;
>>>> +	u16 val;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_pmu_buf {
>>>> +	size_t length;
>>>> +	int nr_pages;
>>>> +	void *base;
>>>> +	long pos;
>>>> +};
>>>> +
>>>> +/* Structure containing the information for filter updating */
>>>> +struct hisi_ptt_filter_update_info {
>>>> +	struct pci_dev *pdev;
>>>> +	u32 port_devid;
>>>> +	bool is_port;
>>>> +	bool is_add;
>>>> +	u16 val;
>>>> +};
>>>> +
>>>> +struct hisi_ptt {
>>>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>>>> +	struct notifier_block hisi_ptt_nb;
>>>> +	struct pmu hisi_ptt_pmu;
>>>> +	void __iomem *iobase;
>>>> +	struct pci_dev *pdev;
>>>> +	/*
>>>> +	 * Use the mutex to protect the filter list and
>>>> +	 * serialize the perf process.
>>>> +	 */
>>>> +	struct mutex mutex;
>>>> +	const char *name;
>>>> +	u16 core_id;
>>>> +	u16 sicl_id;
>>>> +	/* PCI device range managed by the PTT device */
>>>> +	u32 upper;
>>>> +	u32 lower;
>>>> +	u8 busnr;
>>>> +
>>>> +	/*
>>>> +	 * The trace TLP headers can either be filtered by certain
>>>> +	 * root port, or by the requester ID. Organize the filters
>>>> +	 * by @port_filters and @req_filters here. The mask of all
>>>> +	 * the valid ports is also cached for doing sanity check
>>>> +	 * of user input.
>>>> +	 */
>>>> +	struct list_head port_filters;
>>>> +	struct list_head req_filters;
>>>> +	u16 port_mask;
>>>> +
>>>> +	/*
>>>> +	 * We use a delayed work here to avoid indefinitely waiting for
>>>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>>>> +	 * work will be delayed only if the mutex can not be held,
>>>> +	 * otherwise no delay will be applied.
>>>> +	 */
>>>> +	struct delayed_work work;
>>>> +	spinlock_t filter_update_lock;
>>>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>>>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>>>> +};
>>>
>>> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
>>>
>>
>> I'm not sure whether it is common or permitted to provide a header which is only used by
>> this driver with a single file.
> 
> I've seen it done many times.
> 

will split a header file for the driver.

>>
>>>> +
>>>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>>>> +{
>>>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>>>> +}
>>>
>>> I'm not sure there is any value to this inline function.  I suggest to simply
>>> call container_of() whenever it is needed.
>>>
>>
>> ok.
>>
>>>> +
>>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>>> +{
>>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>>> +
>>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	u32 val;
>>>> +
>>>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>>>> +				  val & HISI_PTT_TRACE_IDLE,
>>>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>>>> +
>>>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>>>> +		list_del(&buflet->list);
>>>> +
>>>> +		if (ctrl->has_rmr)
>>>> +			memunmap(buflet->addr);
>>>> +		else
>>>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>>>> +					  buflet->dma);
>>>> +
>>>> +		kfree(buflet);
>>>> +	}
>>>> +}
>>>> +
>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	struct hisi_ptt_dma_buflet *buflet;
>>>> +	int i, ret;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>>> +
>>>> +	/* Make sure the trace buffer is empty before allocating */
>>>> +	if (!list_empty(&ctrl->trace_buf)) {
>>>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>>>> +			memset(buflet->addr, 0, buflet->size);
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>>>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>>>> +		if (!buflet) {
>>>> +			ret = -ENOMEM;
>>>> +			goto err;
>>>> +		}
>>>> +
>>>> +		if (ctrl->has_rmr) {
>>>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>>>> +
>>>> +			buflet->dma = base;
>>>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>>>> +		} else {
>>>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>>>> +							  &buflet->dma, GFP_KERNEL);
>>>> +		}
>>>> +
>>>> +		if (!buflet->addr) {
>>>> +			kfree(buflet);
>>>> +			ret = -ENOMEM;
>>>> +			goto err;
>>>> +		}
>>>> +
>>>> +		memset(buflet->addr, 0, buflet->size);
>>>> +
>>>> +		buflet->index = i;
>>>> +		buflet->size = ctrl->buflet_size;
>>>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +err:
>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>> +	u32 val;
>>>> +
>>>> +	/* Check device idle before start trace */
>>>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>>>> +		return -EBUSY;
>>>> +	}
>>>> +
>>>> +	/* Reset the DMA before start tracing */
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	/*
>>>> +	 * We'll be in the perf context where preemption is disabled,
>>>> +	 * so use busy loop here.
>>>> +	 */
>>>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>>>> +
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	/* clear the interrupt status */
>>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>> +
>>>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>>>> +		writel(lower_32_bits(cur->dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +		writel(upper_32_bits(cur->dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +	}
>>>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>>> +
>>>> +	/* set the trace control register */
>>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>>> +
>>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>>> +{
>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>> +	struct perf_event *event = handle->event;
>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +
>>>> +	buf = perf_get_aux(handle);
>>>> +	if (!buf || !handle->size)
>>>> +		return -EINVAL;
>>>> +
>>>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>>>> +		if (cur->index == index)
>>>> +			break;
>>>> +
>>>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>>>> +	memset(cur->addr, 0, cur->size);
>>>> +	buf->pos += cur->size;
>>>> +
>>>> +	if (stop) {
>>>> +		perf_aux_output_end(handle, buf->pos);
>>>> +	} else if (buf->length - buf->pos < cur->size) {
>>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>>> +		perf_aux_output_end(handle, buf->pos);
>>>> +
>>>> +		buf = perf_aux_output_begin(handle, event);
>>>> +		if (!buf)
>>>> +			return -EINVAL;
>>>> +
>>>> +		buf->pos = handle->head % buf->length;
>>>> +		if (buf->length - buf->pos < cur->size) {
>>>> +			perf_aux_output_end(handle, 0);
>>>> +			return -EINVAL;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>> +	u32 status, buf_idx;
>>>> +
>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	buf_idx = ffs(status) - 1;
>>>> +
>>>> +	/* Clear the interrupt status of buflet @buf_idx */
>>>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +
>>>> +	/*
>>>> +	 * Update the AUX buffer and cache the current buflet index,
>>>> +	 * as we need to know this and save the data when the trace
>>>> +	 * is ended out of the interrupt handler. End the trace
>>>> +	 * if the updating fails.
>>>> +	 */
>>>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +	else
>>>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>>>> +
>>>> +	return IRQ_HANDLED;
>>>> +}
>>>> +
>>>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>> +	u32 status;
>>>> +
>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>>>> +		return IRQ_NONE;
>>>> +
>>>> +	return IRQ_WAKE_THREAD;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>>>> +{
>>>> +	pci_free_irq_vectors(pdev);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	int ret;
>>>> +
>>>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>>>> +	if (ret < 0) {
>>>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>>>> +	if (ret < 0)
>>>> +		return ret;
>>>> +
>>>> +	ret = devm_request_threaded_irq(&pdev->dev,
>>>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>>>> +					"hisi-ptt", hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>>>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_update_filters(struct work_struct *work)
>>>> +{
>>>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>>>> +	struct hisi_ptt_filter_update_info info;
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	struct list_head *target_list;
>>>> +	struct hisi_ptt *hisi_ptt;
>>>> +
>>>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>>>> +
>>>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>>>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>>>> +		return;
>>>> +	}
>>>> +
>>>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>>>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>>>> +			      &hisi_ptt->req_filters;
>>>> +
>>>> +		if (info.is_add) {
>>>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>>>> +			if (!filter)
>>>> +				continue;
>>>> +
>>>> +			filter->pdev = info.pdev;
>>>> +			filter->val = info.val;
>>>> +
>>>> +			list_add_tail(&filter->list, target_list);
>>>> +		} else {
>>>> +			list_for_each_entry(filter, target_list, list)
>>>> +				if (filter->val == info.val) {
>>>> +					list_del(&filter->list);
>>>> +					kfree(filter);
>>>> +					break;
>>>> +				}
>>>> +		}
>>>> +
>>>> +		/* Update the available port mask */
>>>> +		if (!info.is_port)
>>>> +			continue;
>>>> +
>>>> +		if (info.is_add)
>>>> +			hisi_ptt->port_mask |= info.val;
>>>> +		else
>>>> +			hisi_ptt->port_mask &= ~info.val;
>>>> +	}
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>>>> +				    struct hisi_ptt_filter_update_info *info)
>>>> +{
>>>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>>>> +
>>>> +	if (!root_port)
>>>> +		return;
>>>> +
>>>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>>>> +	if (info->port_devid < hisi_ptt->lower ||
>>>> +	    info->port_devid > hisi_ptt->upper)
>>>> +		return;
>>>> +
>>>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>>>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>>>> +
>>>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>>>> +				&hisi_ptt->filter_update_lock))
>>>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>>>> +	else
>>>> +		pci_warn(hisi_ptt->pdev,
>>>> +			 "filter update fifo overflow for target %s\n",
>>>> +			 pci_name(info->pdev));
>>>> +}
>>>> +
>>>> +/*
>>>> + * A PCI bus notifier is used here for dynamically updating the filter
>>>> + * list.
>>>> + */
>>>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>>>> +				  void *data)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>>>> +	struct hisi_ptt_filter_update_info info;
>>>> +	struct device *dev = data;
>>>> +	struct pci_dev *pdev = to_pci_dev(dev);
>>>> +
>>>> +	info.pdev = pdev;
>>>> +
>>>> +	switch (action) {
>>>> +	case BUS_NOTIFY_ADD_DEVICE:
>>>> +		info.is_add = true;
>>>> +		break;
>>>> +	case BUS_NOTIFY_DEL_DEVICE:
>>>> +		info.is_add = false;
>>>> +		break;
>>>> +	default:
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>>>> +{
>>>> +	struct hisi_ptt_filter_update_info info = {
>>>> +		.pdev = pdev,
>>>> +		.is_add = true,
>>>> +	};
>>>> +	struct hisi_ptt *hisi_ptt = data;
>>>> +
>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>>>> +
>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>>>> +		list_del(&filter->list);
>>>> +		kfree(filter);
>>>> +	}
>>>> +
>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>>>> +		list_del(&filter->list);
>>>> +		kfree(filter);
>>>> +	}
>>>> +}
>>>> +
>>>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	struct pci_bus *bus;
>>>> +	u32 reg;
>>>> +
>>>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>>>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>>>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>>>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>>>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>>>> +
>>>> +	/*
>>>> +	 * The device range register provides the information about the
>>>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>>>> +	 * and the root ports it support are on the same PCIe core, with
>>>> +	 * same domain number but maybe different bus number. The device
>>>> +	 * range register will tell us which root ports we can support,
>>>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>>>> +	 * while Bit[15:0] indicates the lower.
>>>> +	 */
>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>>>> +	hisi_ptt->upper = reg >> 16;
>>>> +	hisi_ptt->lower = reg & 0xffff;
>>>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>>>> +
>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>>> +
>>>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>>>> +	if (bus)
>>>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>>>
>>> What happens if bus is NULL?  Should this be reported to the caller and cause
>>> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
>>>
>>
>> It's ok if the list is NULL in the probe process,
>> as the device maybe hotplugged after the PTT driver probe, in which case
>> we can detect the event and update the list as we register a bus notifier.
>>
> 
> Please add the above as a comment in the code.
> 

sure. will do.

> 
>>>> +
>>>> +	/* Initialize trace controls */
>>>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>>>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>>>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>>>> +}
>>>> +
>>>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>>>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>>>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>>>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>>>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>>>> +
>>>> +static ssize_t available_filters_show(struct device *dev,
>>>> +				      struct device_attribute *attr,
>>>> +				      char *buf)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	int pos = 0;
>>>> +
>>>> +	if (list_empty(&hisi_ptt->port_filters))
>>>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>>>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>>>> +				     pci_name(filter->pdev),
>>>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>>>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>>>> +
>>>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>>>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>>>> +				     pci_name(filter->pdev),
>>>> +				     hisi_ptt_get_filter_val(filter->pdev));
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return pos;
>>>> +}
>>>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>>>> +
>>>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>>>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>>>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>>>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>>>> +
>>>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>>>> +	&format_attr_filter.attr,
>>>> +	&format_attr_direction.attr,
>>>> +	&format_attr_type.attr,
>>>> +	&format_attr_format.attr,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>>>> +	.name = "format",
>>>> +	.attrs = hisi_ptt_pmu_format_attrs,
>>>> +};
>>>> +
>>>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>>>> +	&dev_attr_available_filters.attr,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>>>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>>>> +};
>>>> +
>>>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>>>> +	&hisi_ptt_pmu_format_group,
>>>> +	&hisi_ptt_pmu_filter_group,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +/*
>>>> + * The supported value of the direction parameter. See hisi_ptt.rst
>>>> + * documentation for more details.
>>>> + */
>>>> +static u32 hisi_ptt_trace_available_direction[] = {
>>>> +	0,
>>>> +	1,
>>>> +	2,
>>>> +	3,
>>>> +};
>>>> +
>>>> +/* Different types can be set simultaneously */
>>>> +static u32 hisi_ptt_trace_available_type[] = {
>>>> +	1,	/* posted_request */
>>>> +	2,	/* non-posted_request */
>>>> +	4,	/* completion */
>>>> +};
>>>> +
>>>> +static u32 hisi_ptt_trace_availble_format[] = {
>>>> +	0,	/* 4DW */
>>>> +	1,	/* 8DW */
>>>> +};
>>>> +
>>>> +/*
>>>> + * Check whether the config is valid or not. Some configs are multi-selectable
>>>> + * and can be set simultaneously, while some are single selectable (onehot).
>>>> + * Use this function to check the non-onehot configs while
>>>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>>>> + */
>>>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>>>> +{
>>>> +	int i;
>>>> +
>>>> +	/*
>>>> +	 * The non-onehot configs cannot be 0. Walk the available
>>>> +	 * list and clear the valid bits of the config. If there
>>>> +	 * is any resident bit after the walk then the config is
>>>> +	 * invalid.
>>>> +	 */
>>>> +	for (i = 0; i < list_size; i++)
>>>> +		val &= ~available_list[i];
>>>> +
>>>> +	return val ? -EINVAL : 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>>>> +{
>>>> +	int i, ret = -EINVAL;
>>>> +
>>>> +	for (i = 0; i < list_size; i++)
>>>> +		if (val == available_list[i]) {
>>>> +			ret = 0;
>>>> +			break;
>>>> +		}
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>>>> +{
>>>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	int ret = -EINVAL;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>>>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>>>> +
>>>> +	/*
>>>> +	 * Port filters are defined as bit mask. For port filters, check
>>>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>>>> +	 * and whether it's empty or not, otherwise user has specified
>>>> +	 * some unsupported root ports.
>>>> +	 *
>>>> +	 * For Requester ID filters, walk the available filter list to see
>>>> +	 * whether we have one matched.
>>>> +	 */
>>>> +	if (hisi_ptt->trace_ctrl.is_port &&
>>>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>>>> +		ret = 0;
>>>> +	} else {
>>>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>> +			if (filter->val == val) {
>>>> +				ret = 0;
>>>> +				break;
>>>> +			}
>>>> +	}
>>>> +
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.filter = val;
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	int ret;
>>>> +	u32 val;
>>>> +
>>>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>>>> +		return -ENOENT;
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +
>>>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->direction = val;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>>> +
>>>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->type = val;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->format = val;
>>>> +
>>>> +out:
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>>> +				    int nr_pages, bool overwrite)
>>>> +{
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +	struct page **pagelist;
>>>> +	int i;
>>>> +
>>>> +	if (overwrite) {
>>>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	/* If the pages size less than buflets, we cannot start trace */
>>>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>>>> +		return NULL;
>>>> +
>>>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>>> +	if (!buf)
>>>> +		return NULL;
>>>> +
>>>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>>> +	if (!pagelist) {
>>>> +		kfree(buf);
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	for (i = 0; i < nr_pages; i++)
>>>> +		pagelist[i] = virt_to_page(pages[i]);
>>>> +
>>>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>>> +	if (!buf->base) {
>>>> +		kfree(pagelist);
>>>> +		kfree(buf);
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	buf->nr_pages = nr_pages;
>>>> +	buf->length = nr_pages * PAGE_SIZE;
>>>> +	buf->pos = 0;
>>>> +
>>>> +	kfree(pagelist);
>>>> +	return buf;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>>> +{
>>>> +	struct hisi_ptt_pmu_buf *buf = aux;
>>>> +
>>>> +	vunmap(buf->base);
>>>> +	kfree(buf);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +	int cpu = event->cpu;
>>>> +	int ret;
>>>> +
>>>> +	hwc->state = 0;
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	if (cpu == -1)
>>>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>>>> +
>>>> +	/*
>>>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>>>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>>>> +	 * core in event_function_local().
>>>> +	 */
>>>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>> +				 cpumask_of(cpu)));
>>>> +
>>>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	buf = perf_aux_output_begin(handle, event);
>>>> +	if (!buf) {
>>>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	buf->pos = handle->head % buf->length;
>>>> +
>>>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>>> +		perf_aux_output_end(handle, 0);
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return;
>>>> +stop:
>>>> +	event->hw.state |= PERF_HES_STOPPED;
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +
>>>> +	if (hwc->state & PERF_HES_STOPPED)
>>>> +		return;
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>>>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>>>> +	}
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +
>>>> +	hwc->state |= PERF_HES_STOPPED;
>>>> +	perf_event_update_userpage(event);
>>>> +	hwc->state |= PERF_HES_UPTODATE;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +	int cpu = event->cpu;
>>>> +
>>>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>>>> +		return 0;
>>>> +
>>>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>>>> +
>>>> +	if (flags & PERF_EF_START) {
>>>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>>>> +		if (hwc->state & PERF_HES_STOPPED)
>>>> +			return -EINVAL;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>>>> +{
>>>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_unregister_pmu(void *priv)
>>>> +{
>>>> +	perf_pmu_unregister(priv);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	char *pmu_name;
>>>> +	int ret;
>>>> +
>>>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>>>> +		.module		= THIS_MODULE,
>>>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>>>> +		.task_ctx_nr	= perf_sw_context,
>>>> +		.attr_groups	= hisi_ptt_pmu_groups,
>>>> +		.event_init	= hisi_ptt_pmu_event_init,
>>>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>>>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>>>> +		.start		= hisi_ptt_pmu_start,
>>>> +		.stop		= hisi_ptt_pmu_stop,
>>>> +		.add		= hisi_ptt_pmu_add,
>>>> +		.del		= hisi_ptt_pmu_del,
>>>> +	};
>>>> +
>>>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>>>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>>>> +	if (!pmu_name)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>>>> +					hisi_ptt_unregister_pmu,
>>>> +					&hisi_ptt->hisi_ptt_pmu);
>>>> +}
>>>> +
>>>> +/*
>>>> + * Get RMR address if provided by the firmware.
>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>> + * Otherwise a negative value is returned.
>>>> + */
>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	struct iommu_domain *iommu_domain;
>>>> +	struct iommu_resv_region *region;
>>>> +	LIST_HEAD(list);
>>>> +
>>>> +	/*
>>>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>>>> +	 * IOMMU domain is passthrough.
>>>> +	 */
>>>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>> +		return 0;
>>>> +
>>>> +	iommu_get_resv_regions(&pdev->dev, &list);
>>>> +	list_for_each_entry(region, &list, list)
>>>> +		if (region->type == IOMMU_RESV_DIRECT &&
>>>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>>>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>> +			break;
>>>> +		}
>>>> +
>>>> +	iommu_put_resv_regions(&pdev->dev, &list);
>>>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>>>> +			  const struct pci_device_id *id)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt;
>>>> +	int ret;
>>>> +
>>>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>>>> +	if (!hisi_ptt)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	mutex_init(&hisi_ptt->mutex);
>>>> +	hisi_ptt->pdev = pdev;
>>>> +
>>>> +	/*
>>>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>>>> +	 * so directly reference to the pci name string.
>>>> +	 */
>>>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>>>> +	pci_set_drvdata(pdev, hisi_ptt);
>>>> +
>>>> +	ret = pcim_enable_device(pdev);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>>>> +
>>>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +	pci_set_master(pdev);
>>>> +
>>>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	hisi_ptt_init_ctrls(hisi_ptt);
>>>> +
>>>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>>>> +		return ret;
>>>> +	}
>>>
>>> If this fails the pci_free_irq_vectors() is not called.
>>>
>>
>> It'll be called as we've registered a devm callback when
>> allocating the irq vector. See hisi_ptt_register_irq().
> 
> Ah yes - thanks for pointing that out.
> 
>>
>>> I am out of time for today - I will continue tomorrow.
>>>
>>
>> Sure. Thanks!
>> Yicong
>>
>>> Thanks,
>>> Mathieu
>>>
>>>> +
>>>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	/* Register the bus notifier for dynamically updating the filter list */
>>>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>>>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>> +	if (ret)
>>>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +void hisi_ptt_remove(struct pci_dev *pdev)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>>> +
>>>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>> +
>>>> +	/* Cancel any work that has been queued */
>>>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>>>> +
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +
>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +	hisi_ptt_release_filters(hisi_ptt);
>>>> +}
>>>> +
>>>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>>>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>>>> +	{ }
>>>> +};
>>>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>>>> +
>>>> +static struct pci_driver hisi_ptt_driver = {
>>>> +	.name = "hisi_ptt",
>>>> +	.id_table = hisi_ptt_id_tbl,
>>>> +	.probe = hisi_ptt_probe,
>>>> +	.remove = hisi_ptt_remove,
>>>> +};
>>>> +module_pci_driver(hisi_ptt_driver);
>>>> +
>>>> +MODULE_LICENSE("GPL v2");
>>>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>>>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>>>> -- 
>>>> 2.33.0
>>>>
>>> .
>>>
> .
> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/26 2:50, Mathieu Poirier wrote:
> On Thu, Nov 25, 2021 at 04:39:46PM +0800, Yicong Yang wrote:
>> Hi Mathieu,
>>
>> Thanks for the comments! Replies inline.
>>
>> On 2021/11/25 2:51, Mathieu Poirier wrote:
>>> Hi Yicong,
>>>
>>> On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
>>>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>>>> integrated Endpoint(RCiEP) device, providing the capability
>>>> to dynamically monitor and tune the PCIe traffic(tune),
>>>> and trace the TLP headers(trace).
>>>>
>>>
>>> Is there a reason to put "tune" and "trace" are whithin parentheses?  
>>>
>>
>> yeah. I want to use these single word to denote the related function, and
>> place them in the parantheses near the function's description to make sure
>> readers can match them.
>>
> 
> Probably a better idea to simply write "TLP headers" and "PCIe traffic" when
> referring to them.  Otherwise it is very confusing because the reader doesn't
> know if you mean the acronym or the real definition of "tune" and "trace".
> 

ok I reread the commit and seems reader will know the acronym as it's mentioned
as "*tune* the PCIe traffic" and "*trace* the TLP headers". I think the
contents in the parantheses can be simply dropped.

>>>> Add the driver for the device to enable the trace function. The driver
>>>> will create PMU device for each PTT device, and users can start trace
>>>> through perf command.
>>>>
>>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>>> ---
>>>>  drivers/Makefile                       |    1 +
>>>>  drivers/hwtracing/Kconfig              |    2 +
>>>>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>>>>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>>>>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
>>>
>>> Not sure about the "hisilicon" directory.  Right now we have "coresight",
>>> "intel_th" and "stm" - they concentrate on the technology rather than the
>>> vendor.  I think "ptt" would be just fine: 
>>>
>>> drivers/hwtracing/ptt/Kconfig 
>>> drivers/hwtracing/ptt/Makefile
>>> drivers/hwtracing/ptt/hisi_ptt.c
>>>
>>> That way if other vendors want to introduce support for the same kind of
>>> technology we can simply put them under "drivers/hwtracing/ptt/" and things are
>>> still accurate.
>>>
>>
>> sure. sounds sensible. will rename the directory to "ptt" if no objection.
>>
>>>
>>>>  5 files changed, 1159 insertions(+)
>>>>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>>>>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>>>>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
>>>>
>>>> diff --git a/drivers/Makefile b/drivers/Makefile
>>>> index be5d40ae1488..bb0dc9b55ea2 100644
>>>> --- a/drivers/Makefile
>>>> +++ b/drivers/Makefile
>>>> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>>>  obj-y				+= hwtracing/intel_th/
>>>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>>>> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>>>>  obj-$(CONFIG_ANDROID)		+= android/
>>>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>>>  obj-$(CONFIG_FPGA)		+= fpga/
>>>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>>>> index 13085835a636..e3796b17541a 100644
>>>> --- a/drivers/hwtracing/Kconfig
>>>> +++ b/drivers/hwtracing/Kconfig
>>>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>>>  
>>>>  source "drivers/hwtracing/intel_th/Kconfig"
>>>>  
>>>> +source "drivers/hwtracing/hisilicon/Kconfig"
>>>> +
>>>>  endmenu
>>>> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..9c3ab80d99fe
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/Kconfig
>>>> @@ -0,0 +1,8 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +config HISI_PTT
>>>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>>>> +	depends on PCI && HAS_DMA && HAS_IOMEM
>>>
>>> What Arm architecture are you aiming this for?  Can it be found on both armv7
>>> and armv8?
>>>
>>
>> currently the device will appear on Kunpeng 930, which is armv8.
> 
> Ok, then please make it dependent on armv8.
> 

sure.

>>
>>>> +	help
>>>> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
>>>> +	  device, provides support for PCIe traffic tuning and
>>>> +	  tracing TLP headers to the memory.
>>>
>>> Please indicate what the name of the driver will be when compiled as a module.
>>>
>>
>> sure. will add the module's name here.
>>
>>>> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
>>>> new file mode 100644
>>>> index 000000000000..908c09a98161
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/Makefile
>>>> @@ -0,0 +1,2 @@
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>>>> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
>>>> new file mode 100644
>>>> index 000000000000..e11e9b6cc2a8
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
>>>> @@ -0,0 +1,1146 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * Driver for HiSilicon PCIe tune and trace device
>>>> + *
>>>> + * Copyright (c) 2021 HiSilicon Limited.
>>>> + */
>>>> +
>>>> +#include <linux/bitfield.h>
>>>> +#include <linux/bitops.h>
>>>> +#include <linux/delay.h>
>>>> +#include <linux/dma-iommu.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/io.h>
>>>> +#include <linux/iommu.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/kfifo.h>
>>>> +#include <linux/module.h>
>>>> +#include <linux/notifier.h>
>>>> +#include <linux/pci.h>
>>>> +#include <linux/perf_event.h>
>>>> +#include <linux/sizes.h>
>>>> +#include <linux/sysfs.h>
>>>> +#include <linux/vmalloc.h>
>>>> +#include <linux/workqueue.h>
>>>> +
>>>> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
>>>> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
>>>> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
>>>> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
>>>> +#define HISI_PTT_TRACE_CTRL		0x0850
>>>> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
>>>> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
>>>> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
>>>> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
>>>> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
>>>> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
>>>> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
>>>> +#define HISI_PTT_TRACE_INT_STAT		0x0890
>>>> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
>>>> +#define HISI_PTT_TRACE_INT_MASK		0x0894
>>>> +#define HISI_PTT_TRACE_WR_STS		0x08a0
>>>> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
>>>> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
>>>> +#define HISI_PTT_TRACE_STS		0x08b0
>>>> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
>>>> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
>>>> +#define HISI_PTT_LOCATION		0x0fe8
>>>> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
>>>> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
>>>
>>> Extra spaces between #define and the literals.  Please align the values with
>>> that of HISI_PTT_TRACE_DMA_IRQ below.
>>>
>>
>> Spaces are intended to distinguish the definition of the register offset and
>> related register fields. also used in include/uapi/linux/pci_regs.h.
>>
> 
> Interesting and definitely not the norm...
> 
>> HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
>> in the mail text, maybe because the tab is not 8 characters when display.
>>
> 
> In my VI terminal the '0' of HISI_PTT_TRACE_DMA_IRQ is align with the '3' in
> GENMASK(31, 16) and I suspect it is the same for other people.
> 

not sure what happens, with tab=8char it's aligned on my vim and vscode. and yes
it's also aligned with '3' in GENMASK(31,16).

#define HISI_PTT_TRACE_DMA_IRQ\t\t\t0
#define HISI_PTT_TRACE_BUFLETS_CNT\t\t4

>>>> +
>>>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>>>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>>>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>>>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>>>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>>>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>>>> +
>>>> +/* Delay time for filter updating work */
>>>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>>>> +/* Wait time for DMA hardware to reset */
>>>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>>>> +/* Poll timeout and interval for waiting hardware work to finish */
>>>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>>>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>>>> +
>>>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>>>> +
>>>> +enum hisi_ptt_trace_status {
>>>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>>>> +	HISI_PTT_TRACE_STATUS_ON,
>>>> +};
>>>> +
>>>> +struct hisi_ptt_dma_buflet {
>>>
>>> I'm not sure what a "buflet" is...  Probably best to just call this
>>> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
>>> know due to the lack of proper structure documentation.  Please have a look at
>>> how "coresight_device" is documented.  The same applies to the rest of the
>>> structures declared below.
>>>
>>
>> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
>> (we call it buflet) of 4MiB ...".
> 
> I just noticed the documentation in patch 5 - I will read it before continuing
> with this set.
> 
>>
>> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
>> a 'buflet', which means a small buffer (there exists other words with similiar
>> format: droplet, wavelet, ...).
>>
>> The device is designed like this to make sure we won't lose any data when we have to change
>> the buffer address, the device can continue to write to the next buflet and don't
>> neet to pause.
>>
>>>> +	struct list_head list;
>>>> +	unsigned int size;
>>>> +	dma_addr_t dma;
>>>> +
>>>> +	/*
>>>> +	 * The address of the buflet holding the trace data.
>>>> +	 * See Documentation/trace/hisi-ptt.rst for the details
>>>
>>> As of this writing, hisi-ptt.rst doesn't exist.
>>>
>>
>> The comment intends to direct the user to the documentation if the user
>> want to know details about the data format. The data format is decribed
>> in the section 4 of the doc.
>>
>>>> +	 * of the data format.
>>>> +	 */
>>>> +	void *addr;
>>>> +	int index;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_trace_ctrl {
>>>> +	enum hisi_ptt_trace_status status;
>>>> +	struct perf_output_handle handle;
>>>> +	struct list_head trace_buf;
>>>> +	/*
>>>> +	 * The index of the buffer which trace data
>>>> +	 * currently is writing to.
>>>> +	 */
>>>> +	u32 buf_index;
>>>> +
>>>> +	int default_cpu;
>>>> +	u32 buflet_size;
>>>> +	bool is_port;
>>>> +	u32 format:1;		/* Format of the traced TLP headers */
>>>> +	u32 type:4;		/* Type of the TLP headers to trace */
>>>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>>>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>>>> +
>>>> +	phys_addr_t rmr_addr;
>>>> +	size_t rmr_length;
>>>> +	bool has_rmr;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_filter_desc {
>>>> +	struct list_head list;
>>>> +	struct pci_dev *pdev;
>>>> +	u16 val;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_pmu_buf {
>>>> +	size_t length;
>>>> +	int nr_pages;
>>>> +	void *base;
>>>> +	long pos;
>>>> +};
>>>> +
>>>> +/* Structure containing the information for filter updating */
>>>> +struct hisi_ptt_filter_update_info {
>>>> +	struct pci_dev *pdev;
>>>> +	u32 port_devid;
>>>> +	bool is_port;
>>>> +	bool is_add;
>>>> +	u16 val;
>>>> +};
>>>> +
>>>> +struct hisi_ptt {
>>>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>>>> +	struct notifier_block hisi_ptt_nb;
>>>> +	struct pmu hisi_ptt_pmu;
>>>> +	void __iomem *iobase;
>>>> +	struct pci_dev *pdev;
>>>> +	/*
>>>> +	 * Use the mutex to protect the filter list and
>>>> +	 * serialize the perf process.
>>>> +	 */
>>>> +	struct mutex mutex;
>>>> +	const char *name;
>>>> +	u16 core_id;
>>>> +	u16 sicl_id;
>>>> +	/* PCI device range managed by the PTT device */
>>>> +	u32 upper;
>>>> +	u32 lower;
>>>> +	u8 busnr;
>>>> +
>>>> +	/*
>>>> +	 * The trace TLP headers can either be filtered by certain
>>>> +	 * root port, or by the requester ID. Organize the filters
>>>> +	 * by @port_filters and @req_filters here. The mask of all
>>>> +	 * the valid ports is also cached for doing sanity check
>>>> +	 * of user input.
>>>> +	 */
>>>> +	struct list_head port_filters;
>>>> +	struct list_head req_filters;
>>>> +	u16 port_mask;
>>>> +
>>>> +	/*
>>>> +	 * We use a delayed work here to avoid indefinitely waiting for
>>>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>>>> +	 * work will be delayed only if the mutex can not be held,
>>>> +	 * otherwise no delay will be applied.
>>>> +	 */
>>>> +	struct delayed_work work;
>>>> +	spinlock_t filter_update_lock;
>>>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>>>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>>>> +};
>>>
>>> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
>>>
>>
>> I'm not sure whether it is common or permitted to provide a header which is only used by
>> this driver with a single file.
> 
> I've seen it done many times.
> 

will split a header file for the driver.

>>
>>>> +
>>>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>>>> +{
>>>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>>>> +}
>>>
>>> I'm not sure there is any value to this inline function.  I suggest to simply
>>> call container_of() whenever it is needed.
>>>
>>
>> ok.
>>
>>>> +
>>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>>> +{
>>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>>> +
>>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	u32 val;
>>>> +
>>>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>>>> +				  val & HISI_PTT_TRACE_IDLE,
>>>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>>>> +
>>>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>>>> +		list_del(&buflet->list);
>>>> +
>>>> +		if (ctrl->has_rmr)
>>>> +			memunmap(buflet->addr);
>>>> +		else
>>>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>>>> +					  buflet->dma);
>>>> +
>>>> +		kfree(buflet);
>>>> +	}
>>>> +}
>>>> +
>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	struct hisi_ptt_dma_buflet *buflet;
>>>> +	int i, ret;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>>> +
>>>> +	/* Make sure the trace buffer is empty before allocating */
>>>> +	if (!list_empty(&ctrl->trace_buf)) {
>>>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>>>> +			memset(buflet->addr, 0, buflet->size);
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>>>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>>>> +		if (!buflet) {
>>>> +			ret = -ENOMEM;
>>>> +			goto err;
>>>> +		}
>>>> +
>>>> +		if (ctrl->has_rmr) {
>>>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>>>> +
>>>> +			buflet->dma = base;
>>>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>>>> +		} else {
>>>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>>>> +							  &buflet->dma, GFP_KERNEL);
>>>> +		}
>>>> +
>>>> +		if (!buflet->addr) {
>>>> +			kfree(buflet);
>>>> +			ret = -ENOMEM;
>>>> +			goto err;
>>>> +		}
>>>> +
>>>> +		memset(buflet->addr, 0, buflet->size);
>>>> +
>>>> +		buflet->index = i;
>>>> +		buflet->size = ctrl->buflet_size;
>>>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +err:
>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>> +	u32 val;
>>>> +
>>>> +	/* Check device idle before start trace */
>>>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>>>> +		return -EBUSY;
>>>> +	}
>>>> +
>>>> +	/* Reset the DMA before start tracing */
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	/*
>>>> +	 * We'll be in the perf context where preemption is disabled,
>>>> +	 * so use busy loop here.
>>>> +	 */
>>>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>>>> +
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	/* clear the interrupt status */
>>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>> +
>>>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>>>> +		writel(lower_32_bits(cur->dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +		writel(upper_32_bits(cur->dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +	}
>>>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>>> +
>>>> +	/* set the trace control register */
>>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>>> +
>>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>>> +{
>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>> +	struct perf_event *event = handle->event;
>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +
>>>> +	buf = perf_get_aux(handle);
>>>> +	if (!buf || !handle->size)
>>>> +		return -EINVAL;
>>>> +
>>>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>>>> +		if (cur->index == index)
>>>> +			break;
>>>> +
>>>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>>>> +	memset(cur->addr, 0, cur->size);
>>>> +	buf->pos += cur->size;
>>>> +
>>>> +	if (stop) {
>>>> +		perf_aux_output_end(handle, buf->pos);
>>>> +	} else if (buf->length - buf->pos < cur->size) {
>>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>>> +		perf_aux_output_end(handle, buf->pos);
>>>> +
>>>> +		buf = perf_aux_output_begin(handle, event);
>>>> +		if (!buf)
>>>> +			return -EINVAL;
>>>> +
>>>> +		buf->pos = handle->head % buf->length;
>>>> +		if (buf->length - buf->pos < cur->size) {
>>>> +			perf_aux_output_end(handle, 0);
>>>> +			return -EINVAL;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>> +	u32 status, buf_idx;
>>>> +
>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	buf_idx = ffs(status) - 1;
>>>> +
>>>> +	/* Clear the interrupt status of buflet @buf_idx */
>>>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +
>>>> +	/*
>>>> +	 * Update the AUX buffer and cache the current buflet index,
>>>> +	 * as we need to know this and save the data when the trace
>>>> +	 * is ended out of the interrupt handler. End the trace
>>>> +	 * if the updating fails.
>>>> +	 */
>>>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +	else
>>>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>>>> +
>>>> +	return IRQ_HANDLED;
>>>> +}
>>>> +
>>>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>> +	u32 status;
>>>> +
>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>>>> +		return IRQ_NONE;
>>>> +
>>>> +	return IRQ_WAKE_THREAD;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>>>> +{
>>>> +	pci_free_irq_vectors(pdev);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	int ret;
>>>> +
>>>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>>>> +	if (ret < 0) {
>>>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>>>> +	if (ret < 0)
>>>> +		return ret;
>>>> +
>>>> +	ret = devm_request_threaded_irq(&pdev->dev,
>>>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>>>> +					"hisi-ptt", hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>>>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_update_filters(struct work_struct *work)
>>>> +{
>>>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>>>> +	struct hisi_ptt_filter_update_info info;
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	struct list_head *target_list;
>>>> +	struct hisi_ptt *hisi_ptt;
>>>> +
>>>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>>>> +
>>>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>>>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>>>> +		return;
>>>> +	}
>>>> +
>>>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>>>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>>>> +			      &hisi_ptt->req_filters;
>>>> +
>>>> +		if (info.is_add) {
>>>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>>>> +			if (!filter)
>>>> +				continue;
>>>> +
>>>> +			filter->pdev = info.pdev;
>>>> +			filter->val = info.val;
>>>> +
>>>> +			list_add_tail(&filter->list, target_list);
>>>> +		} else {
>>>> +			list_for_each_entry(filter, target_list, list)
>>>> +				if (filter->val == info.val) {
>>>> +					list_del(&filter->list);
>>>> +					kfree(filter);
>>>> +					break;
>>>> +				}
>>>> +		}
>>>> +
>>>> +		/* Update the available port mask */
>>>> +		if (!info.is_port)
>>>> +			continue;
>>>> +
>>>> +		if (info.is_add)
>>>> +			hisi_ptt->port_mask |= info.val;
>>>> +		else
>>>> +			hisi_ptt->port_mask &= ~info.val;
>>>> +	}
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>>>> +				    struct hisi_ptt_filter_update_info *info)
>>>> +{
>>>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>>>> +
>>>> +	if (!root_port)
>>>> +		return;
>>>> +
>>>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>>>> +	if (info->port_devid < hisi_ptt->lower ||
>>>> +	    info->port_devid > hisi_ptt->upper)
>>>> +		return;
>>>> +
>>>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>>>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>>>> +
>>>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>>>> +				&hisi_ptt->filter_update_lock))
>>>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>>>> +	else
>>>> +		pci_warn(hisi_ptt->pdev,
>>>> +			 "filter update fifo overflow for target %s\n",
>>>> +			 pci_name(info->pdev));
>>>> +}
>>>> +
>>>> +/*
>>>> + * A PCI bus notifier is used here for dynamically updating the filter
>>>> + * list.
>>>> + */
>>>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>>>> +				  void *data)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>>>> +	struct hisi_ptt_filter_update_info info;
>>>> +	struct device *dev = data;
>>>> +	struct pci_dev *pdev = to_pci_dev(dev);
>>>> +
>>>> +	info.pdev = pdev;
>>>> +
>>>> +	switch (action) {
>>>> +	case BUS_NOTIFY_ADD_DEVICE:
>>>> +		info.is_add = true;
>>>> +		break;
>>>> +	case BUS_NOTIFY_DEL_DEVICE:
>>>> +		info.is_add = false;
>>>> +		break;
>>>> +	default:
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>>>> +{
>>>> +	struct hisi_ptt_filter_update_info info = {
>>>> +		.pdev = pdev,
>>>> +		.is_add = true,
>>>> +	};
>>>> +	struct hisi_ptt *hisi_ptt = data;
>>>> +
>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>>>> +
>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>>>> +		list_del(&filter->list);
>>>> +		kfree(filter);
>>>> +	}
>>>> +
>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>>>> +		list_del(&filter->list);
>>>> +		kfree(filter);
>>>> +	}
>>>> +}
>>>> +
>>>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	struct pci_bus *bus;
>>>> +	u32 reg;
>>>> +
>>>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>>>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>>>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>>>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>>>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>>>> +
>>>> +	/*
>>>> +	 * The device range register provides the information about the
>>>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>>>> +	 * and the root ports it support are on the same PCIe core, with
>>>> +	 * same domain number but maybe different bus number. The device
>>>> +	 * range register will tell us which root ports we can support,
>>>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>>>> +	 * while Bit[15:0] indicates the lower.
>>>> +	 */
>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>>>> +	hisi_ptt->upper = reg >> 16;
>>>> +	hisi_ptt->lower = reg & 0xffff;
>>>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>>>> +
>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>>> +
>>>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>>>> +	if (bus)
>>>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>>>
>>> What happens if bus is NULL?  Should this be reported to the caller and cause
>>> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
>>>
>>
>> It's ok if the list is NULL in the probe process,
>> as the device maybe hotplugged after the PTT driver probe, in which case
>> we can detect the event and update the list as we register a bus notifier.
>>
> 
> Please add the above as a comment in the code.
> 

sure. will do.

> 
>>>> +
>>>> +	/* Initialize trace controls */
>>>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>>>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>>>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>>>> +}
>>>> +
>>>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>>>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>>>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>>>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>>>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>>>> +
>>>> +static ssize_t available_filters_show(struct device *dev,
>>>> +				      struct device_attribute *attr,
>>>> +				      char *buf)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	int pos = 0;
>>>> +
>>>> +	if (list_empty(&hisi_ptt->port_filters))
>>>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>>>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>>>> +				     pci_name(filter->pdev),
>>>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>>>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>>>> +
>>>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>>>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>>>> +				     pci_name(filter->pdev),
>>>> +				     hisi_ptt_get_filter_val(filter->pdev));
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return pos;
>>>> +}
>>>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>>>> +
>>>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>>>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>>>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>>>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>>>> +
>>>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>>>> +	&format_attr_filter.attr,
>>>> +	&format_attr_direction.attr,
>>>> +	&format_attr_type.attr,
>>>> +	&format_attr_format.attr,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>>>> +	.name = "format",
>>>> +	.attrs = hisi_ptt_pmu_format_attrs,
>>>> +};
>>>> +
>>>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>>>> +	&dev_attr_available_filters.attr,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>>>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>>>> +};
>>>> +
>>>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>>>> +	&hisi_ptt_pmu_format_group,
>>>> +	&hisi_ptt_pmu_filter_group,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +/*
>>>> + * The supported value of the direction parameter. See hisi_ptt.rst
>>>> + * documentation for more details.
>>>> + */
>>>> +static u32 hisi_ptt_trace_available_direction[] = {
>>>> +	0,
>>>> +	1,
>>>> +	2,
>>>> +	3,
>>>> +};
>>>> +
>>>> +/* Different types can be set simultaneously */
>>>> +static u32 hisi_ptt_trace_available_type[] = {
>>>> +	1,	/* posted_request */
>>>> +	2,	/* non-posted_request */
>>>> +	4,	/* completion */
>>>> +};
>>>> +
>>>> +static u32 hisi_ptt_trace_availble_format[] = {
>>>> +	0,	/* 4DW */
>>>> +	1,	/* 8DW */
>>>> +};
>>>> +
>>>> +/*
>>>> + * Check whether the config is valid or not. Some configs are multi-selectable
>>>> + * and can be set simultaneously, while some are single selectable (onehot).
>>>> + * Use this function to check the non-onehot configs while
>>>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>>>> + */
>>>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>>>> +{
>>>> +	int i;
>>>> +
>>>> +	/*
>>>> +	 * The non-onehot configs cannot be 0. Walk the available
>>>> +	 * list and clear the valid bits of the config. If there
>>>> +	 * is any resident bit after the walk then the config is
>>>> +	 * invalid.
>>>> +	 */
>>>> +	for (i = 0; i < list_size; i++)
>>>> +		val &= ~available_list[i];
>>>> +
>>>> +	return val ? -EINVAL : 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>>>> +{
>>>> +	int i, ret = -EINVAL;
>>>> +
>>>> +	for (i = 0; i < list_size; i++)
>>>> +		if (val == available_list[i]) {
>>>> +			ret = 0;
>>>> +			break;
>>>> +		}
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>>>> +{
>>>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	int ret = -EINVAL;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>>>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>>>> +
>>>> +	/*
>>>> +	 * Port filters are defined as bit mask. For port filters, check
>>>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>>>> +	 * and whether it's empty or not, otherwise user has specified
>>>> +	 * some unsupported root ports.
>>>> +	 *
>>>> +	 * For Requester ID filters, walk the available filter list to see
>>>> +	 * whether we have one matched.
>>>> +	 */
>>>> +	if (hisi_ptt->trace_ctrl.is_port &&
>>>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>>>> +		ret = 0;
>>>> +	} else {
>>>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>> +			if (filter->val == val) {
>>>> +				ret = 0;
>>>> +				break;
>>>> +			}
>>>> +	}
>>>> +
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.filter = val;
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	int ret;
>>>> +	u32 val;
>>>> +
>>>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>>>> +		return -ENOENT;
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +
>>>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->direction = val;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>>> +
>>>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->type = val;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->format = val;
>>>> +
>>>> +out:
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>>> +				    int nr_pages, bool overwrite)
>>>> +{
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +	struct page **pagelist;
>>>> +	int i;
>>>> +
>>>> +	if (overwrite) {
>>>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	/* If the pages size less than buflets, we cannot start trace */
>>>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>>>> +		return NULL;
>>>> +
>>>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>>> +	if (!buf)
>>>> +		return NULL;
>>>> +
>>>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>>> +	if (!pagelist) {
>>>> +		kfree(buf);
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	for (i = 0; i < nr_pages; i++)
>>>> +		pagelist[i] = virt_to_page(pages[i]);
>>>> +
>>>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>>> +	if (!buf->base) {
>>>> +		kfree(pagelist);
>>>> +		kfree(buf);
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	buf->nr_pages = nr_pages;
>>>> +	buf->length = nr_pages * PAGE_SIZE;
>>>> +	buf->pos = 0;
>>>> +
>>>> +	kfree(pagelist);
>>>> +	return buf;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>>> +{
>>>> +	struct hisi_ptt_pmu_buf *buf = aux;
>>>> +
>>>> +	vunmap(buf->base);
>>>> +	kfree(buf);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +	int cpu = event->cpu;
>>>> +	int ret;
>>>> +
>>>> +	hwc->state = 0;
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	if (cpu == -1)
>>>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>>>> +
>>>> +	/*
>>>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>>>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>>>> +	 * core in event_function_local().
>>>> +	 */
>>>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>> +				 cpumask_of(cpu)));
>>>> +
>>>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	buf = perf_aux_output_begin(handle, event);
>>>> +	if (!buf) {
>>>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	buf->pos = handle->head % buf->length;
>>>> +
>>>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>>> +		perf_aux_output_end(handle, 0);
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return;
>>>> +stop:
>>>> +	event->hw.state |= PERF_HES_STOPPED;
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +
>>>> +	if (hwc->state & PERF_HES_STOPPED)
>>>> +		return;
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>>>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>>>> +	}
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +
>>>> +	hwc->state |= PERF_HES_STOPPED;
>>>> +	perf_event_update_userpage(event);
>>>> +	hwc->state |= PERF_HES_UPTODATE;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +	int cpu = event->cpu;
>>>> +
>>>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>>>> +		return 0;
>>>> +
>>>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>>>> +
>>>> +	if (flags & PERF_EF_START) {
>>>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>>>> +		if (hwc->state & PERF_HES_STOPPED)
>>>> +			return -EINVAL;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>>>> +{
>>>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_unregister_pmu(void *priv)
>>>> +{
>>>> +	perf_pmu_unregister(priv);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	char *pmu_name;
>>>> +	int ret;
>>>> +
>>>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>>>> +		.module		= THIS_MODULE,
>>>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>>>> +		.task_ctx_nr	= perf_sw_context,
>>>> +		.attr_groups	= hisi_ptt_pmu_groups,
>>>> +		.event_init	= hisi_ptt_pmu_event_init,
>>>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>>>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>>>> +		.start		= hisi_ptt_pmu_start,
>>>> +		.stop		= hisi_ptt_pmu_stop,
>>>> +		.add		= hisi_ptt_pmu_add,
>>>> +		.del		= hisi_ptt_pmu_del,
>>>> +	};
>>>> +
>>>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>>>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>>>> +	if (!pmu_name)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>>>> +					hisi_ptt_unregister_pmu,
>>>> +					&hisi_ptt->hisi_ptt_pmu);
>>>> +}
>>>> +
>>>> +/*
>>>> + * Get RMR address if provided by the firmware.
>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>> + * Otherwise a negative value is returned.
>>>> + */
>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	struct iommu_domain *iommu_domain;
>>>> +	struct iommu_resv_region *region;
>>>> +	LIST_HEAD(list);
>>>> +
>>>> +	/*
>>>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>>>> +	 * IOMMU domain is passthrough.
>>>> +	 */
>>>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>> +		return 0;
>>>> +
>>>> +	iommu_get_resv_regions(&pdev->dev, &list);
>>>> +	list_for_each_entry(region, &list, list)
>>>> +		if (region->type == IOMMU_RESV_DIRECT &&
>>>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>>>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>> +			break;
>>>> +		}
>>>> +
>>>> +	iommu_put_resv_regions(&pdev->dev, &list);
>>>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>>>> +			  const struct pci_device_id *id)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt;
>>>> +	int ret;
>>>> +
>>>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>>>> +	if (!hisi_ptt)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	mutex_init(&hisi_ptt->mutex);
>>>> +	hisi_ptt->pdev = pdev;
>>>> +
>>>> +	/*
>>>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>>>> +	 * so directly reference to the pci name string.
>>>> +	 */
>>>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>>>> +	pci_set_drvdata(pdev, hisi_ptt);
>>>> +
>>>> +	ret = pcim_enable_device(pdev);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>>>> +
>>>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +	pci_set_master(pdev);
>>>> +
>>>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	hisi_ptt_init_ctrls(hisi_ptt);
>>>> +
>>>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>>>> +		return ret;
>>>> +	}
>>>
>>> If this fails the pci_free_irq_vectors() is not called.
>>>
>>
>> It'll be called as we've registered a devm callback when
>> allocating the irq vector. See hisi_ptt_register_irq().
> 
> Ah yes - thanks for pointing that out.
> 
>>
>>> I am out of time for today - I will continue tomorrow.
>>>
>>
>> Sure. Thanks!
>> Yicong
>>
>>> Thanks,
>>> Mathieu
>>>
>>>> +
>>>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	/* Register the bus notifier for dynamically updating the filter list */
>>>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>>>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>> +	if (ret)
>>>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +void hisi_ptt_remove(struct pci_dev *pdev)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>>> +
>>>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>> +
>>>> +	/* Cancel any work that has been queued */
>>>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>>>> +
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +
>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +	hisi_ptt_release_filters(hisi_ptt);
>>>> +}
>>>> +
>>>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>>>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>>>> +	{ }
>>>> +};
>>>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>>>> +
>>>> +static struct pci_driver hisi_ptt_driver = {
>>>> +	.name = "hisi_ptt",
>>>> +	.id_table = hisi_ptt_id_tbl,
>>>> +	.probe = hisi_ptt_probe,
>>>> +	.remove = hisi_ptt_remove,
>>>> +};
>>>> +module_pci_driver(hisi_ptt_driver);
>>>> +
>>>> +MODULE_LICENSE("GPL v2");
>>>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>>>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>>>> -- 
>>>> 2.33.0
>>>>
>>> .
>>>
> .
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 2/6] hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

On 2021/11/26 2:50, Mathieu Poirier wrote:
> On Thu, Nov 25, 2021 at 04:39:46PM +0800, Yicong Yang wrote:
>> Hi Mathieu,
>>
>> Thanks for the comments! Replies inline.
>>
>> On 2021/11/25 2:51, Mathieu Poirier wrote:
>>> Hi Yicong,
>>>
>>> On Tue, Nov 16, 2021 at 05:06:21PM +0800, Yicong Yang wrote:
>>>> HiSilicon PCIe tune and trace device(PTT) is a PCIe Root Complex
>>>> integrated Endpoint(RCiEP) device, providing the capability
>>>> to dynamically monitor and tune the PCIe traffic(tune),
>>>> and trace the TLP headers(trace).
>>>>
>>>
>>> Is there a reason to put "tune" and "trace" are whithin parentheses?  
>>>
>>
>> yeah. I want to use these single word to denote the related function, and
>> place them in the parantheses near the function's description to make sure
>> readers can match them.
>>
> 
> Probably a better idea to simply write "TLP headers" and "PCIe traffic" when
> referring to them.  Otherwise it is very confusing because the reader doesn't
> know if you mean the acronym or the real definition of "tune" and "trace".
> 

ok I reread the commit and seems reader will know the acronym as it's mentioned
as "*tune* the PCIe traffic" and "*trace* the TLP headers". I think the
contents in the parantheses can be simply dropped.

>>>> Add the driver for the device to enable the trace function. The driver
>>>> will create PMU device for each PTT device, and users can start trace
>>>> through perf command.
>>>>
>>>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>>>> ---
>>>>  drivers/Makefile                       |    1 +
>>>>  drivers/hwtracing/Kconfig              |    2 +
>>>>  drivers/hwtracing/hisilicon/Kconfig    |    8 +
>>>>  drivers/hwtracing/hisilicon/Makefile   |    2 +
>>>>  drivers/hwtracing/hisilicon/hisi_ptt.c | 1146 ++++++++++++++++++++++++
>>>
>>> Not sure about the "hisilicon" directory.  Right now we have "coresight",
>>> "intel_th" and "stm" - they concentrate on the technology rather than the
>>> vendor.  I think "ptt" would be just fine: 
>>>
>>> drivers/hwtracing/ptt/Kconfig 
>>> drivers/hwtracing/ptt/Makefile
>>> drivers/hwtracing/ptt/hisi_ptt.c
>>>
>>> That way if other vendors want to introduce support for the same kind of
>>> technology we can simply put them under "drivers/hwtracing/ptt/" and things are
>>> still accurate.
>>>
>>
>> sure. sounds sensible. will rename the directory to "ptt" if no objection.
>>
>>>
>>>>  5 files changed, 1159 insertions(+)
>>>>  create mode 100644 drivers/hwtracing/hisilicon/Kconfig
>>>>  create mode 100644 drivers/hwtracing/hisilicon/Makefile
>>>>  create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c
>>>>
>>>> diff --git a/drivers/Makefile b/drivers/Makefile
>>>> index be5d40ae1488..bb0dc9b55ea2 100644
>>>> --- a/drivers/Makefile
>>>> +++ b/drivers/Makefile
>>>> @@ -176,6 +176,7 @@ obj-$(CONFIG_USB4)		+= thunderbolt/
>>>>  obj-$(CONFIG_CORESIGHT)		+= hwtracing/coresight/
>>>>  obj-y				+= hwtracing/intel_th/
>>>>  obj-$(CONFIG_STM)		+= hwtracing/stm/
>>>> +obj-$(CONFIG_HISI_PTT)         += hwtracing/hisilicon/
>>>>  obj-$(CONFIG_ANDROID)		+= android/
>>>>  obj-$(CONFIG_NVMEM)		+= nvmem/
>>>>  obj-$(CONFIG_FPGA)		+= fpga/
>>>> diff --git a/drivers/hwtracing/Kconfig b/drivers/hwtracing/Kconfig
>>>> index 13085835a636..e3796b17541a 100644
>>>> --- a/drivers/hwtracing/Kconfig
>>>> +++ b/drivers/hwtracing/Kconfig
>>>> @@ -5,4 +5,6 @@ source "drivers/hwtracing/stm/Kconfig"
>>>>  
>>>>  source "drivers/hwtracing/intel_th/Kconfig"
>>>>  
>>>> +source "drivers/hwtracing/hisilicon/Kconfig"
>>>> +
>>>>  endmenu
>>>> diff --git a/drivers/hwtracing/hisilicon/Kconfig b/drivers/hwtracing/hisilicon/Kconfig
>>>> new file mode 100644
>>>> index 000000000000..9c3ab80d99fe
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/Kconfig
>>>> @@ -0,0 +1,8 @@
>>>> +# SPDX-License-Identifier: GPL-2.0-only
>>>> +config HISI_PTT
>>>> +	tristate "HiSilicon PCIe Tune and Trace Device"
>>>> +	depends on PCI && HAS_DMA && HAS_IOMEM
>>>
>>> What Arm architecture are you aiming this for?  Can it be found on both armv7
>>> and armv8?
>>>
>>
>> currently the device will appear on Kunpeng 930, which is armv8.
> 
> Ok, then please make it dependent on armv8.
> 

sure.

>>
>>>> +	help
>>>> +	  HiSilicon PCIe Tune and Trace Device exist as a PCIe RCiEP
>>>> +	  device, provides support for PCIe traffic tuning and
>>>> +	  tracing TLP headers to the memory.
>>>
>>> Please indicate what the name of the driver will be when compiled as a module.
>>>
>>
>> sure. will add the module's name here.
>>
>>>> diff --git a/drivers/hwtracing/hisilicon/Makefile b/drivers/hwtracing/hisilicon/Makefile
>>>> new file mode 100644
>>>> index 000000000000..908c09a98161
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/Makefile
>>>> @@ -0,0 +1,2 @@
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +obj-$(CONFIG_HISI_PTT) += hisi_ptt.o
>>>> diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
>>>> new file mode 100644
>>>> index 000000000000..e11e9b6cc2a8
>>>> --- /dev/null
>>>> +++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
>>>> @@ -0,0 +1,1146 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * Driver for HiSilicon PCIe tune and trace device
>>>> + *
>>>> + * Copyright (c) 2021 HiSilicon Limited.
>>>> + */
>>>> +
>>>> +#include <linux/bitfield.h>
>>>> +#include <linux/bitops.h>
>>>> +#include <linux/delay.h>
>>>> +#include <linux/dma-iommu.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +#include <linux/interrupt.h>
>>>> +#include <linux/io.h>
>>>> +#include <linux/iommu.h>
>>>> +#include <linux/iopoll.h>
>>>> +#include <linux/kfifo.h>
>>>> +#include <linux/module.h>
>>>> +#include <linux/notifier.h>
>>>> +#include <linux/pci.h>
>>>> +#include <linux/perf_event.h>
>>>> +#include <linux/sizes.h>
>>>> +#include <linux/sysfs.h>
>>>> +#include <linux/vmalloc.h>
>>>> +#include <linux/workqueue.h>
>>>> +
>>>> +#define HISI_PTT_TRACE_ADDR_SIZE	0x0800
>>>> +#define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
>>>> +#define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
>>>> +#define HISI_PTT_TRACE_ADDR_STRIDE	0x8
>>>> +#define HISI_PTT_TRACE_CTRL		0x0850
>>>> +#define   HISI_PTT_TRACE_CTRL_EN	BIT(0)
>>>> +#define   HISI_PTT_TRACE_CTRL_RST	BIT(1)
>>>> +#define   HISI_PTT_TRACE_CTRL_RXTX_SEL	GENMASK(3, 2)
>>>> +#define   HISI_PTT_TRACE_CTRL_TYPE_SEL	GENMASK(7, 4)
>>>> +#define   HISI_PTT_TRACE_CTRL_DATA_FORMAT	BIT(14)
>>>> +#define   HISI_PTT_TRACE_CTRL_FILTER_MODE	BIT(15)
>>>> +#define   HISI_PTT_TRACE_CTRL_TARGET_SEL	GENMASK(31, 16)
>>>> +#define HISI_PTT_TRACE_INT_STAT		0x0890
>>>> +#define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
>>>> +#define HISI_PTT_TRACE_INT_MASK		0x0894
>>>> +#define HISI_PTT_TRACE_WR_STS		0x08a0
>>>> +#define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
>>>> +#define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
>>>> +#define HISI_PTT_TRACE_STS		0x08b0
>>>> +#define   HISI_PTT_TRACE_IDLE		BIT(0)
>>>> +#define HISI_PTT_DEVICE_RANGE		0x0fe0
>>>> +#define HISI_PTT_LOCATION		0x0fe8
>>>> +#define   HISI_PTT_CORE_ID		GENMASK(15, 0)
>>>> +#define   HISI_PTT_SICL_ID		GENMASK(31, 16)
>>>
>>> Extra spaces between #define and the literals.  Please align the values with
>>> that of HISI_PTT_TRACE_DMA_IRQ below.
>>>
>>
>> Spaces are intended to distinguish the definition of the register offset and
>> related register fields. also used in include/uapi/linux/pci_regs.h.
>>
> 
> Interesting and definitely not the norm...
> 
>> HISI_PTT_TRACE_DMA_IRQ is well aligned with tab but seems with problem display
>> in the mail text, maybe because the tab is not 8 characters when display.
>>
> 
> In my VI terminal the '0' of HISI_PTT_TRACE_DMA_IRQ is align with the '3' in
> GENMASK(31, 16) and I suspect it is the same for other people.
> 

not sure what happens, with tab=8char it's aligned on my vim and vscode. and yes
it's also aligned with '3' in GENMASK(31,16).

#define HISI_PTT_TRACE_DMA_IRQ\t\t\t0
#define HISI_PTT_TRACE_BUFLETS_CNT\t\t4

>>>> +
>>>> +#define HISI_PTT_TRACE_DMA_IRQ			0
>>>> +#define HISI_PTT_TRACE_BUFLETS_CNT		4
>>>> +#define HISI_PTT_TRACE_BUFLET_SIZE		SZ_4M
>>>> +#define HISI_PTT_TRACE_BUFFER_SIZE		(HISI_PTT_TRACE_BUFLET_SIZE * \
>>>> +						 HISI_PTT_TRACE_BUFLETS_CNT)
>>>> +#define HISI_PTT_FILTER_UPDATE_FIFO_SIZE	16
>>>> +
>>>> +/* Delay time for filter updating work */
>>>> +#define HISI_PTT_WORK_DELAY_MS		100UL
>>>> +/* Wait time for DMA hardware to reset */
>>>> +#define HISI_PTT_RESET_WAIT_MS		1000UL
>>>> +/* Poll timeout and interval for waiting hardware work to finish */
>>>> +#define HISI_PTT_WAIT_TIMEOUT_US	1000000UL
>>>> +#define HISI_PTT_WAIT_POLL_INTERVAL_US	100UL
>>>> +
>>>> +#define HISI_PCIE_CORE_PORT_ID(devfn)	(PCI_FUNC(devfn) << 1)
>>>> +
>>>> +enum hisi_ptt_trace_status {
>>>> +	HISI_PTT_TRACE_STATUS_OFF = 0,
>>>> +	HISI_PTT_TRACE_STATUS_ON,
>>>> +};
>>>> +
>>>> +struct hisi_ptt_dma_buflet {
>>>
>>> I'm not sure what a "buflet" is...  Probably best to just call this
>>> hisi_ptt_dma_buffer" if it pertains to a buffer.  On that note it is hard to
>>> know due to the lack of proper structure documentation.  Please have a look at
>>> how "coresight_device" is documented.  The same applies to the rest of the
>>> structures declared below.
>>>
>>
>> It's mentioned in section 5 of hisi_ptt.rst as "Driver will allocate each DMA buffer
>> (we call it buflet) of 4MiB ...".
> 
> I just noticed the documentation in patch 5 - I will read it before continuing
> with this set.
> 
>>
>> The total DMA buffer of PTT device is divided into 4 parts, and each part is called
>> a 'buflet', which means a small buffer (there exists other words with similiar
>> format: droplet, wavelet, ...).
>>
>> The device is designed like this to make sure we won't lose any data when we have to change
>> the buffer address, the device can continue to write to the next buflet and don't
>> neet to pause.
>>
>>>> +	struct list_head list;
>>>> +	unsigned int size;
>>>> +	dma_addr_t dma;
>>>> +
>>>> +	/*
>>>> +	 * The address of the buflet holding the trace data.
>>>> +	 * See Documentation/trace/hisi-ptt.rst for the details
>>>
>>> As of this writing, hisi-ptt.rst doesn't exist.
>>>
>>
>> The comment intends to direct the user to the documentation if the user
>> want to know details about the data format. The data format is decribed
>> in the section 4 of the doc.
>>
>>>> +	 * of the data format.
>>>> +	 */
>>>> +	void *addr;
>>>> +	int index;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_trace_ctrl {
>>>> +	enum hisi_ptt_trace_status status;
>>>> +	struct perf_output_handle handle;
>>>> +	struct list_head trace_buf;
>>>> +	/*
>>>> +	 * The index of the buffer which trace data
>>>> +	 * currently is writing to.
>>>> +	 */
>>>> +	u32 buf_index;
>>>> +
>>>> +	int default_cpu;
>>>> +	u32 buflet_size;
>>>> +	bool is_port;
>>>> +	u32 format:1;		/* Format of the traced TLP headers */
>>>> +	u32 type:4;		/* Type of the TLP headers to trace */
>>>> +	u32 direction:2;	/* Direction of the TLP headers to trace */
>>>> +	u32 filter:16;		/* Root port or Requester to filter the TLP headers */
>>>> +
>>>> +	phys_addr_t rmr_addr;
>>>> +	size_t rmr_length;
>>>> +	bool has_rmr;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_filter_desc {
>>>> +	struct list_head list;
>>>> +	struct pci_dev *pdev;
>>>> +	u16 val;
>>>> +};
>>>> +
>>>> +struct hisi_ptt_pmu_buf {
>>>> +	size_t length;
>>>> +	int nr_pages;
>>>> +	void *base;
>>>> +	long pos;
>>>> +};
>>>> +
>>>> +/* Structure containing the information for filter updating */
>>>> +struct hisi_ptt_filter_update_info {
>>>> +	struct pci_dev *pdev;
>>>> +	u32 port_devid;
>>>> +	bool is_port;
>>>> +	bool is_add;
>>>> +	u16 val;
>>>> +};
>>>> +
>>>> +struct hisi_ptt {
>>>> +	struct hisi_ptt_trace_ctrl trace_ctrl;
>>>> +	struct notifier_block hisi_ptt_nb;
>>>> +	struct pmu hisi_ptt_pmu;
>>>> +	void __iomem *iobase;
>>>> +	struct pci_dev *pdev;
>>>> +	/*
>>>> +	 * Use the mutex to protect the filter list and
>>>> +	 * serialize the perf process.
>>>> +	 */
>>>> +	struct mutex mutex;
>>>> +	const char *name;
>>>> +	u16 core_id;
>>>> +	u16 sicl_id;
>>>> +	/* PCI device range managed by the PTT device */
>>>> +	u32 upper;
>>>> +	u32 lower;
>>>> +	u8 busnr;
>>>> +
>>>> +	/*
>>>> +	 * The trace TLP headers can either be filtered by certain
>>>> +	 * root port, or by the requester ID. Organize the filters
>>>> +	 * by @port_filters and @req_filters here. The mask of all
>>>> +	 * the valid ports is also cached for doing sanity check
>>>> +	 * of user input.
>>>> +	 */
>>>> +	struct list_head port_filters;
>>>> +	struct list_head req_filters;
>>>> +	u16 port_mask;
>>>> +
>>>> +	/*
>>>> +	 * We use a delayed work here to avoid indefinitely waiting for
>>>> +	 * the hisi_ptt->mutex which protecting the filter list. The
>>>> +	 * work will be delayed only if the mutex can not be held,
>>>> +	 * otherwise no delay will be applied.
>>>> +	 */
>>>> +	struct delayed_work work;
>>>> +	spinlock_t filter_update_lock;
>>>> +	DECLARE_KFIFO(filter_update_kfifo, struct hisi_ptt_filter_update_info,
>>>> +		      HISI_PTT_FILTER_UPDATE_FIFO_SIZE);
>>>> +};
>>>
>>> It might be a good idea to spinoff a hisi_ptt.h for the above declarations.
>>>
>>
>> I'm not sure whether it is common or permitted to provide a header which is only used by
>> this driver with a single file.
> 
> I've seen it done many times.
> 

will split a header file for the driver.

>>
>>>> +
>>>> +static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
>>>> +{
>>>> +	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
>>>> +}
>>>
>>> I'm not sure there is any value to this inline function.  I suggest to simply
>>> call container_of() whenever it is needed.
>>>
>>
>> ok.
>>
>>>> +
>>>> +static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
>>>> +{
>>>> +	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
>>>> +		return BIT(HISI_PCIE_CORE_PORT_ID(PCI_SLOT(pdev->devfn)));
>>>> +
>>>> +	return PCI_DEVID(pdev->bus->number, pdev->devfn);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_wait_trace_hw_idle(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	u32 val;
>>>> +
>>>> +	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TRACE_STS, val,
>>>> +				  val & HISI_PTT_TRACE_IDLE,
>>>> +				  HISI_PTT_WAIT_POLL_INTERVAL_US,
>>>> +				  HISI_PTT_WAIT_TIMEOUT_US);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_free_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	struct hisi_ptt_dma_buflet *buflet, *tbuflet;
>>>> +
>>>> +	list_for_each_entry_safe(buflet, tbuflet, &ctrl->trace_buf, list) {
>>>> +		list_del(&buflet->list);
>>>> +
>>>> +		if (ctrl->has_rmr)
>>>> +			memunmap(buflet->addr);
>>>> +		else
>>>> +			dma_free_coherent(dev, buflet->size, buflet->addr,
>>>> +					  buflet->dma);
>>>> +
>>>> +		kfree(buflet);
>>>> +	}
>>>> +}
>>>> +
>>>> +static int hisi_ptt_alloc_trace_buf(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct device *dev = &hisi_ptt->pdev->dev;
>>>> +	struct hisi_ptt_dma_buflet *buflet;
>>>> +	int i, ret;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.buf_index = 0;
>>>> +
>>>> +	/* Make sure the trace buffer is empty before allocating */
>>>> +	if (!list_empty(&ctrl->trace_buf)) {
>>>> +		list_for_each_entry(buflet, &ctrl->trace_buf, list)
>>>> +			memset(buflet->addr, 0, buflet->size);
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	for (i = 0; i < HISI_PTT_TRACE_BUFLETS_CNT; ++i) {
>>>> +		buflet = kzalloc(sizeof(*buflet), GFP_KERNEL);
>>>> +		if (!buflet) {
>>>> +			ret = -ENOMEM;
>>>> +			goto err;
>>>> +		}
>>>> +
>>>> +		if (ctrl->has_rmr) {
>>>> +			phys_addr_t base = ctrl->rmr_addr + i * ctrl->buflet_size;
>>>> +
>>>> +			buflet->dma = base;
>>>> +			buflet->addr = memremap(base, ctrl->buflet_size, MEMREMAP_WB);
>>>> +		} else {
>>>> +			buflet->addr = dma_alloc_coherent(dev, ctrl->buflet_size,
>>>> +							  &buflet->dma, GFP_KERNEL);
>>>> +		}
>>>> +
>>>> +		if (!buflet->addr) {
>>>> +			kfree(buflet);
>>>> +			ret = -ENOMEM;
>>>> +			goto err;
>>>> +		}
>>>> +
>>>> +		memset(buflet->addr, 0, buflet->size);
>>>> +
>>>> +		buflet->index = i;
>>>> +		buflet->size = ctrl->buflet_size;
>>>> +		list_add_tail(&buflet->list, &ctrl->trace_buf);
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +err:
>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_trace_end(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	hisi_ptt->trace_ctrl.status = HISI_PTT_TRACE_STATUS_OFF;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_start(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>> +	u32 val;
>>>> +
>>>> +	/* Check device idle before start trace */
>>>> +	if (hisi_ptt_wait_trace_hw_idle(hisi_ptt)) {
>>>> +		pci_err(hisi_ptt->pdev, "Failed to start trace, the device is still busy.\n");
>>>> +		return -EBUSY;
>>>> +	}
>>>> +
>>>> +	/* Reset the DMA before start tracing */
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val |= HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	/*
>>>> +	 * We'll be in the perf context where preemption is disabled,
>>>> +	 * so use busy loop here.
>>>> +	 */
>>>> +	mdelay(HISI_PTT_RESET_WAIT_MS);
>>>> +
>>>> +	val = readl(hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +	val &= ~HISI_PTT_TRACE_CTRL_RST;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	/* clear the interrupt status */
>>>> +	writel(HISI_PTT_TRACE_INT_STAT_MASK, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	writel(0, hisi_ptt->iobase + HISI_PTT_TRACE_INT_MASK);
>>>> +
>>>> +	list_for_each_entry(cur, &ctrl->trace_buf, list) {
>>>> +		writel(lower_32_bits(cur->dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_LO_0 +
>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +		writel(upper_32_bits(cur->dma),
>>>> +		       hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_BASE_HI_0 +
>>>> +		       cur->index * HISI_PTT_TRACE_ADDR_STRIDE);
>>>> +	}
>>>> +	writel(ctrl->buflet_size, hisi_ptt->iobase + HISI_PTT_TRACE_ADDR_SIZE);
>>>> +
>>>> +	/* set the trace control register */
>>>> +	val = FIELD_PREP(HISI_PTT_TRACE_CTRL_TYPE_SEL, ctrl->type);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_RXTX_SEL, ctrl->direction);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_DATA_FORMAT, ctrl->format);
>>>> +	val |= FIELD_PREP(HISI_PTT_TRACE_CTRL_TARGET_SEL, hisi_ptt->trace_ctrl.filter);
>>>> +	if (!hisi_ptt->trace_ctrl.is_port)
>>>> +		val |= HISI_PTT_TRACE_CTRL_FILTER_MODE;
>>>> +
>>>> +	val |= HISI_PTT_TRACE_CTRL_EN;
>>>> +	writel(val, hisi_ptt->iobase + HISI_PTT_TRACE_CTRL);
>>>> +
>>>> +	ctrl->status = HISI_PTT_TRACE_STATUS_ON;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_update_aux(struct hisi_ptt *hisi_ptt, int index, bool stop)
>>>> +{
>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>> +	struct perf_event *event = handle->event;
>>>> +	struct hisi_ptt_dma_buflet *cur;
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +
>>>> +	buf = perf_get_aux(handle);
>>>> +	if (!buf || !handle->size)
>>>> +		return -EINVAL;
>>>> +
>>>> +	list_for_each_entry(cur, &hisi_ptt->trace_ctrl.trace_buf, list)
>>>> +		if (cur->index == index)
>>>> +			break;
>>>> +
>>>> +	memcpy(buf->base + buf->pos, cur->addr, cur->size);
>>>> +	memset(cur->addr, 0, cur->size);
>>>> +	buf->pos += cur->size;
>>>> +
>>>> +	if (stop) {
>>>> +		perf_aux_output_end(handle, buf->pos);
>>>> +	} else if (buf->length - buf->pos < cur->size) {
>>>> +		perf_aux_output_skip(handle, buf->length - buf->pos);
>>>> +		perf_aux_output_end(handle, buf->pos);
>>>> +
>>>> +		buf = perf_aux_output_begin(handle, event);
>>>> +		if (!buf)
>>>> +			return -EINVAL;
>>>> +
>>>> +		buf->pos = handle->head % buf->length;
>>>> +		if (buf->length - buf->pos < cur->size) {
>>>> +			perf_aux_output_end(handle, 0);
>>>> +			return -EINVAL;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static irqreturn_t hisi_ptt_isr(int irq, void *context)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>> +	u32 status, buf_idx;
>>>> +
>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	buf_idx = ffs(status) - 1;
>>>> +
>>>> +	/* Clear the interrupt status of buflet @buf_idx */
>>>> +	writel(status, hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +
>>>> +	/*
>>>> +	 * Update the AUX buffer and cache the current buflet index,
>>>> +	 * as we need to know this and save the data when the trace
>>>> +	 * is ended out of the interrupt handler. End the trace
>>>> +	 * if the updating fails.
>>>> +	 */
>>>> +	if (hisi_ptt_update_aux(hisi_ptt, buf_idx, false))
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +	else
>>>> +		hisi_ptt->trace_ctrl.buf_index = (buf_idx + 1) % HISI_PTT_TRACE_BUFLETS_CNT;
>>>> +
>>>> +	return IRQ_HANDLED;
>>>> +}
>>>> +
>>>> +static irqreturn_t hisi_ptt_irq(int irq, void *context)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = context;
>>>> +	u32 status;
>>>> +
>>>> +	status = readl(hisi_ptt->iobase + HISI_PTT_TRACE_INT_STAT);
>>>> +	if (!(status & HISI_PTT_TRACE_INT_STAT_MASK))
>>>> +		return IRQ_NONE;
>>>> +
>>>> +	return IRQ_WAKE_THREAD;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_irq_free_vectors(void *pdev)
>>>> +{
>>>> +	pci_free_irq_vectors(pdev);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_register_irq(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	int ret;
>>>> +
>>>> +	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
>>>> +	if (ret < 0) {
>>>> +		pci_err(pdev, "failed to allocate irq vector, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	ret = devm_add_action_or_reset(&pdev->dev, hisi_ptt_irq_free_vectors, pdev);
>>>> +	if (ret < 0)
>>>> +		return ret;
>>>> +
>>>> +	ret = devm_request_threaded_irq(&pdev->dev,
>>>> +					pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>> +					hisi_ptt_irq, hisi_ptt_isr, 0,
>>>> +					"hisi-ptt", hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to request irq %d, ret = %d.\n",
>>>> +			pci_irq_vector(pdev, HISI_PTT_TRACE_DMA_IRQ), ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_update_filters(struct work_struct *work)
>>>> +{
>>>> +	struct delayed_work *delayed_work = to_delayed_work(work);
>>>> +	struct hisi_ptt_filter_update_info info;
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	struct list_head *target_list;
>>>> +	struct hisi_ptt *hisi_ptt;
>>>> +
>>>> +	hisi_ptt = container_of(delayed_work, struct hisi_ptt, work);
>>>> +
>>>> +	if (!mutex_trylock(&hisi_ptt->mutex)) {
>>>> +		schedule_delayed_work(&hisi_ptt->work, HISI_PTT_WORK_DELAY_MS);
>>>> +		return;
>>>> +	}
>>>> +
>>>> +	while (kfifo_get(&hisi_ptt->filter_update_kfifo, &info)) {
>>>> +		target_list = info.is_port ? &hisi_ptt->port_filters :
>>>> +			      &hisi_ptt->req_filters;
>>>> +
>>>> +		if (info.is_add) {
>>>> +			filter = kzalloc(sizeof(*filter), GFP_KERNEL);
>>>> +			if (!filter)
>>>> +				continue;
>>>> +
>>>> +			filter->pdev = info.pdev;
>>>> +			filter->val = info.val;
>>>> +
>>>> +			list_add_tail(&filter->list, target_list);
>>>> +		} else {
>>>> +			list_for_each_entry(filter, target_list, list)
>>>> +				if (filter->val == info.val) {
>>>> +					list_del(&filter->list);
>>>> +					kfree(filter);
>>>> +					break;
>>>> +				}
>>>> +		}
>>>> +
>>>> +		/* Update the available port mask */
>>>> +		if (!info.is_port)
>>>> +			continue;
>>>> +
>>>> +		if (info.is_add)
>>>> +			hisi_ptt->port_mask |= info.val;
>>>> +		else
>>>> +			hisi_ptt->port_mask &= ~info.val;
>>>> +	}
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_update_fifo_in(struct hisi_ptt *hisi_ptt,
>>>> +				    struct hisi_ptt_filter_update_info *info)
>>>> +{
>>>> +	struct pci_dev *root_port = pcie_find_root_port(info->pdev);
>>>> +
>>>> +	if (!root_port)
>>>> +		return;
>>>> +
>>>> +	info->port_devid = PCI_DEVID(root_port->bus->number, root_port->devfn);
>>>> +	if (info->port_devid < hisi_ptt->lower ||
>>>> +	    info->port_devid > hisi_ptt->upper)
>>>> +		return;
>>>> +
>>>> +	info->is_port = pci_pcie_type(info->pdev) == PCI_EXP_TYPE_ROOT_PORT;
>>>> +	info->val = hisi_ptt_get_filter_val(info->pdev);
>>>> +
>>>> +	if (kfifo_in_spinlocked(&hisi_ptt->filter_update_kfifo, info, 1,
>>>> +				&hisi_ptt->filter_update_lock))
>>>> +		schedule_delayed_work(&hisi_ptt->work, 0);
>>>> +	else
>>>> +		pci_warn(hisi_ptt->pdev,
>>>> +			 "filter update fifo overflow for target %s\n",
>>>> +			 pci_name(info->pdev));
>>>> +}
>>>> +
>>>> +/*
>>>> + * A PCI bus notifier is used here for dynamically updating the filter
>>>> + * list.
>>>> + */
>>>> +static int hisi_ptt_notifier_call(struct notifier_block *nb, unsigned long action,
>>>> +				  void *data)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = container_of(nb, struct hisi_ptt, hisi_ptt_nb);
>>>> +	struct hisi_ptt_filter_update_info info;
>>>> +	struct device *dev = data;
>>>> +	struct pci_dev *pdev = to_pci_dev(dev);
>>>> +
>>>> +	info.pdev = pdev;
>>>> +
>>>> +	switch (action) {
>>>> +	case BUS_NOTIFY_ADD_DEVICE:
>>>> +		info.is_add = true;
>>>> +		break;
>>>> +	case BUS_NOTIFY_DEL_DEVICE:
>>>> +		info.is_add = false;
>>>> +		break;
>>>> +	default:
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_init_filters(struct pci_dev *pdev, void *data)
>>>> +{
>>>> +	struct hisi_ptt_filter_update_info info = {
>>>> +		.pdev = pdev,
>>>> +		.is_add = true,
>>>> +	};
>>>> +	struct hisi_ptt *hisi_ptt = data;
>>>> +
>>>> +	hisi_ptt_update_fifo_in(hisi_ptt, &info);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_release_filters(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct hisi_ptt_filter_desc *filter, *tfilter;
>>>> +
>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->req_filters, list) {
>>>> +		list_del(&filter->list);
>>>> +		kfree(filter);
>>>> +	}
>>>> +
>>>> +	list_for_each_entry_safe(filter, tfilter, &hisi_ptt->port_filters, list) {
>>>> +		list_del(&filter->list);
>>>> +		kfree(filter);
>>>> +	}
>>>> +}
>>>> +
>>>> +static void hisi_ptt_init_ctrls(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	struct pci_bus *bus;
>>>> +	u32 reg;
>>>> +
>>>> +	INIT_DELAYED_WORK(&hisi_ptt->work, hisi_ptt_update_filters);
>>>> +	spin_lock_init(&hisi_ptt->filter_update_lock);
>>>> +	INIT_KFIFO(hisi_ptt->filter_update_kfifo);
>>>> +	INIT_LIST_HEAD(&hisi_ptt->port_filters);
>>>> +	INIT_LIST_HEAD(&hisi_ptt->req_filters);
>>>> +
>>>> +	/*
>>>> +	 * The device range register provides the information about the
>>>> +	 * root ports which the RCiEP can control and trace. The RCiEP
>>>> +	 * and the root ports it support are on the same PCIe core, with
>>>> +	 * same domain number but maybe different bus number. The device
>>>> +	 * range register will tell us which root ports we can support,
>>>> +	 * Bit[31:16] indicates the upper BDF numbers of the root port,
>>>> +	 * while Bit[15:0] indicates the lower.
>>>> +	 */
>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_DEVICE_RANGE);
>>>> +	hisi_ptt->upper = reg >> 16;
>>>> +	hisi_ptt->lower = reg & 0xffff;
>>>> +	hisi_ptt->busnr = PCI_BUS_NUM(hisi_ptt->upper);
>>>> +
>>>> +	reg = readl(hisi_ptt->iobase + HISI_PTT_LOCATION);
>>>> +	hisi_ptt->core_id = FIELD_GET(HISI_PTT_CORE_ID, reg);
>>>> +	hisi_ptt->sicl_id = FIELD_GET(HISI_PTT_SICL_ID, reg);
>>>> +
>>>> +	bus = pci_find_bus(pci_domain_nr(pdev->bus), hisi_ptt->busnr);
>>>> +	if (bus)
>>>> +		pci_walk_bus(bus, hisi_ptt_init_filters, hisi_ptt);
>>>
>>> What happens if bus is NULL?  Should this be reported to the caller and cause
>>> hisi_ptt_probe() to fail?  Please add a comment that describes the scenario.
>>>
>>
>> It's ok if the list is NULL in the probe process,
>> as the device maybe hotplugged after the PTT driver probe, in which case
>> we can detect the event and update the list as we register a bus notifier.
>>
> 
> Please add the above as a comment in the code.
> 

sure. will do.

> 
>>>> +
>>>> +	/* Initialize trace controls */
>>>> +	INIT_LIST_HEAD(&hisi_ptt->trace_ctrl.trace_buf);
>>>> +	hisi_ptt->trace_ctrl.buflet_size = HISI_PTT_TRACE_BUFLET_SIZE;
>>>> +	hisi_ptt->trace_ctrl.default_cpu = cpumask_first(cpumask_of_node(dev_to_node(&pdev->dev)));
>>>> +}
>>>> +
>>>> +#define HISI_PTT_PMU_FILTER_IS_PORT	BIT(19)
>>>> +#define HISI_PTT_PMU_FILTER_VAL_MASK	GENMASK(15, 0)
>>>> +#define HISI_PTT_PMU_DIRECTION_MASK	GENMASK(23, 20)
>>>> +#define HISI_PTT_PMU_TYPE_MASK		GENMASK(31, 24)
>>>> +#define HISI_PTT_PMU_FORMAT_MASK	GENMASK(35, 32)
>>>> +
>>>> +static ssize_t available_filters_show(struct device *dev,
>>>> +				      struct device_attribute *attr,
>>>> +				      char *buf)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	int pos = 0;
>>>> +
>>>> +	if (list_empty(&hisi_ptt->port_filters))
>>>> +		return sysfs_emit(buf, "#### No available filter ####\n");
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	pos += sysfs_emit_at(buf, pos, "#### Root Ports ####\n");
>>>> +	list_for_each_entry(filter, &hisi_ptt->port_filters, list)
>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05lx\n",
>>>> +				     pci_name(filter->pdev),
>>>> +				     hisi_ptt_get_filter_val(filter->pdev) |
>>>> +				     HISI_PTT_PMU_FILTER_IS_PORT);
>>>> +
>>>> +	pos += sysfs_emit_at(buf, pos, "#### Requesters ####\n");
>>>> +	list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>> +		pos += sysfs_emit_at(buf, pos, "%s	0x%05x\n",
>>>> +				     pci_name(filter->pdev),
>>>> +				     hisi_ptt_get_filter_val(filter->pdev));
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return pos;
>>>> +}
>>>> +static DEVICE_ATTR_ADMIN_RO(available_filters);
>>>> +
>>>> +PMU_FORMAT_ATTR(filter,		"config:0-19");
>>>> +PMU_FORMAT_ATTR(direction,	"config:20-23");
>>>> +PMU_FORMAT_ATTR(type,		"config:24-31");
>>>> +PMU_FORMAT_ATTR(format,		"config:32-35");
>>>> +
>>>> +static struct attribute *hisi_ptt_pmu_format_attrs[] = {
>>>> +	&format_attr_filter.attr,
>>>> +	&format_attr_direction.attr,
>>>> +	&format_attr_type.attr,
>>>> +	&format_attr_format.attr,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +static struct attribute_group hisi_ptt_pmu_format_group = {
>>>> +	.name = "format",
>>>> +	.attrs = hisi_ptt_pmu_format_attrs,
>>>> +};
>>>> +
>>>> +static struct attribute *hisi_ptt_pmu_filter_attrs[] = {
>>>> +	&dev_attr_available_filters.attr,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +static struct attribute_group hisi_ptt_pmu_filter_group = {
>>>> +	.attrs = hisi_ptt_pmu_filter_attrs,
>>>> +};
>>>> +
>>>> +static const struct attribute_group *hisi_ptt_pmu_groups[] = {
>>>> +	&hisi_ptt_pmu_format_group,
>>>> +	&hisi_ptt_pmu_filter_group,
>>>> +	NULL
>>>> +};
>>>> +
>>>> +/*
>>>> + * The supported value of the direction parameter. See hisi_ptt.rst
>>>> + * documentation for more details.
>>>> + */
>>>> +static u32 hisi_ptt_trace_available_direction[] = {
>>>> +	0,
>>>> +	1,
>>>> +	2,
>>>> +	3,
>>>> +};
>>>> +
>>>> +/* Different types can be set simultaneously */
>>>> +static u32 hisi_ptt_trace_available_type[] = {
>>>> +	1,	/* posted_request */
>>>> +	2,	/* non-posted_request */
>>>> +	4,	/* completion */
>>>> +};
>>>> +
>>>> +static u32 hisi_ptt_trace_availble_format[] = {
>>>> +	0,	/* 4DW */
>>>> +	1,	/* 8DW */
>>>> +};
>>>> +
>>>> +/*
>>>> + * Check whether the config is valid or not. Some configs are multi-selectable
>>>> + * and can be set simultaneously, while some are single selectable (onehot).
>>>> + * Use this function to check the non-onehot configs while
>>>> + * hisi_ptt_trace_valid_config_onehot() for the onehot ones.
>>>> + */
>>>> +static int hisi_ptt_trace_valid_config(u32 val, u32 *available_list, u32 list_size)
>>>> +{
>>>> +	int i;
>>>> +
>>>> +	/*
>>>> +	 * The non-onehot configs cannot be 0. Walk the available
>>>> +	 * list and clear the valid bits of the config. If there
>>>> +	 * is any resident bit after the walk then the config is
>>>> +	 * invalid.
>>>> +	 */
>>>> +	for (i = 0; i < list_size; i++)
>>>> +		val &= ~available_list[i];
>>>> +
>>>> +	return val ? -EINVAL : 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_valid_config_onehot(u32 val, u32 *available_list, u32 list_size)
>>>> +{
>>>> +	int i, ret = -EINVAL;
>>>> +
>>>> +	for (i = 0; i < list_size; i++)
>>>> +		if (val == available_list[i]) {
>>>> +			ret = 0;
>>>> +			break;
>>>> +		}
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_trace_init_filter(struct hisi_ptt *hisi_ptt, u64 config)
>>>> +{
>>>> +	unsigned long val, port_mask = hisi_ptt->port_mask;
>>>> +	struct hisi_ptt_filter_desc *filter;
>>>> +	int ret = -EINVAL;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.is_port = FIELD_GET(HISI_PTT_PMU_FILTER_IS_PORT, config);
>>>> +	val = FIELD_GET(HISI_PTT_PMU_FILTER_VAL_MASK, config);
>>>> +
>>>> +	/*
>>>> +	 * Port filters are defined as bit mask. For port filters, check
>>>> +	 * the bits in the @val are within the range of hisi_ptt->port_mask
>>>> +	 * and whether it's empty or not, otherwise user has specified
>>>> +	 * some unsupported root ports.
>>>> +	 *
>>>> +	 * For Requester ID filters, walk the available filter list to see
>>>> +	 * whether we have one matched.
>>>> +	 */
>>>> +	if (hisi_ptt->trace_ctrl.is_port &&
>>>> +	    bitmap_subset(&val, &port_mask, BITS_PER_LONG)) {
>>>> +		ret = 0;
>>>> +	} else {
>>>> +		list_for_each_entry(filter, &hisi_ptt->req_filters, list)
>>>> +			if (filter->val == val) {
>>>> +				ret = 0;
>>>> +				break;
>>>> +			}
>>>> +	}
>>>> +
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	hisi_ptt->trace_ctrl.filter = val;
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_pmu_event_init(struct perf_event *event)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hisi_ptt_trace_ctrl *ctrl = &hisi_ptt->trace_ctrl;
>>>> +	int ret;
>>>> +	u32 val;
>>>> +
>>>> +	if (event->attr.type != hisi_ptt->hisi_ptt_pmu.type)
>>>> +		return -ENOENT;
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +
>>>> +	ret = hisi_ptt_trace_init_filter(hisi_ptt, event->attr.config);
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_DIRECTION_MASK, event->attr.config);
>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_available_direction,
>>>> +						 ARRAY_SIZE(hisi_ptt_trace_available_direction));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->direction = val;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_TYPE_MASK, event->attr.config);
>>>> +
>>>> +	ret = hisi_ptt_trace_valid_config(val, hisi_ptt_trace_available_type,
>>>> +					  ARRAY_SIZE(hisi_ptt_trace_available_type));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->type = val;
>>>> +
>>>> +	val = FIELD_GET(HISI_PTT_PMU_FORMAT_MASK, event->attr.config);
>>>> +	ret = hisi_ptt_trace_valid_config_onehot(val, hisi_ptt_trace_availble_format,
>>>> +						 ARRAY_SIZE(hisi_ptt_trace_availble_format));
>>>> +	if (ret < 0)
>>>> +		goto out;
>>>> +	ctrl->format = val;
>>>> +
>>>> +out:
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static void *hisi_ptt_pmu_setup_aux(struct perf_event *event, void **pages,
>>>> +				    int nr_pages, bool overwrite)
>>>> +{
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +	struct page **pagelist;
>>>> +	int i;
>>>> +
>>>> +	if (overwrite) {
>>>> +		dev_warn(event->pmu->dev, "Overwrite mode is not supported\n");
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	/* If the pages size less than buflets, we cannot start trace */
>>>> +	if (nr_pages < HISI_PTT_TRACE_BUFFER_SIZE / PAGE_SIZE)
>>>> +		return NULL;
>>>> +
>>>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>>>> +	if (!buf)
>>>> +		return NULL;
>>>> +
>>>> +	pagelist = kcalloc(nr_pages, sizeof(*pagelist), GFP_KERNEL);
>>>> +	if (!pagelist) {
>>>> +		kfree(buf);
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	for (i = 0; i < nr_pages; i++)
>>>> +		pagelist[i] = virt_to_page(pages[i]);
>>>> +
>>>> +	buf->base = vmap(pagelist, nr_pages, VM_MAP, PAGE_KERNEL);
>>>> +	if (!buf->base) {
>>>> +		kfree(pagelist);
>>>> +		kfree(buf);
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	buf->nr_pages = nr_pages;
>>>> +	buf->length = nr_pages * PAGE_SIZE;
>>>> +	buf->pos = 0;
>>>> +
>>>> +	kfree(pagelist);
>>>> +	return buf;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_free_aux(void *aux)
>>>> +{
>>>> +	struct hisi_ptt_pmu_buf *buf = aux;
>>>> +
>>>> +	vunmap(buf->base);
>>>> +	kfree(buf);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_start(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct perf_output_handle *handle = &hisi_ptt->trace_ctrl.handle;
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +	struct hisi_ptt_pmu_buf *buf;
>>>> +	int cpu = event->cpu;
>>>> +	int ret;
>>>> +
>>>> +	hwc->state = 0;
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>> +		pci_dbg(hisi_ptt->pdev, "trace has already started\n");
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	if (cpu == -1)
>>>> +		cpu = hisi_ptt->trace_ctrl.default_cpu;
>>>> +
>>>> +	/*
>>>> +	 * Handle the interrupt on the same cpu which starts the trace to avoid
>>>> +	 * context mismatch. Otherwise we'll trigger the WARN from the perf
>>>> +	 * core in event_function_local().
>>>> +	 */
>>>> +	WARN_ON(irq_set_affinity(pci_irq_vector(hisi_ptt->pdev, HISI_PTT_TRACE_DMA_IRQ),
>>>> +				 cpumask_of(cpu)));
>>>> +
>>>> +	ret = hisi_ptt_alloc_trace_buf(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_dbg(hisi_ptt->pdev, "alloc trace buf failed, ret = %d\n", ret);
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	buf = perf_aux_output_begin(handle, event);
>>>> +	if (!buf) {
>>>> +		pci_dbg(hisi_ptt->pdev, "aux output begin failed\n");
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	buf->pos = handle->head % buf->length;
>>>> +
>>>> +	ret = hisi_ptt_trace_start(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_dbg(hisi_ptt->pdev, "trace start failed, ret = %d\n", ret);
>>>> +		perf_aux_output_end(handle, 0);
>>>> +		goto stop;
>>>> +	}
>>>> +
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +	return;
>>>> +stop:
>>>> +	event->hw.state |= PERF_HES_STOPPED;
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_stop(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +
>>>> +	if (hwc->state & PERF_HES_STOPPED)
>>>> +		return;
>>>> +
>>>> +	mutex_lock(&hisi_ptt->mutex);
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON) {
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +		WARN(hisi_ptt_wait_trace_hw_idle(hisi_ptt), "Device is still busy");
>>>> +		hisi_ptt_update_aux(hisi_ptt, hisi_ptt->trace_ctrl.buf_index, true);
>>>> +	}
>>>> +	mutex_unlock(&hisi_ptt->mutex);
>>>> +
>>>> +	hwc->state |= PERF_HES_STOPPED;
>>>> +	perf_event_update_userpage(event);
>>>> +	hwc->state |= PERF_HES_UPTODATE;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_pmu_add(struct perf_event *event, int flags)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = to_hisi_ptt(event->pmu);
>>>> +	struct hw_perf_event *hwc = &event->hw;
>>>> +	int cpu = event->cpu;
>>>> +
>>>> +	if (cpu == -1 && smp_processor_id() != hisi_ptt->trace_ctrl.default_cpu)
>>>> +		return 0;
>>>> +
>>>> +	hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
>>>> +
>>>> +	if (flags & PERF_EF_START) {
>>>> +		hisi_ptt_pmu_start(event, PERF_EF_RELOAD);
>>>> +		if (hwc->state & PERF_HES_STOPPED)
>>>> +			return -EINVAL;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static void hisi_ptt_pmu_del(struct perf_event *event, int flags)
>>>> +{
>>>> +	hisi_ptt_pmu_stop(event, PERF_EF_UPDATE);
>>>> +}
>>>> +
>>>> +static void hisi_ptt_unregister_pmu(void *priv)
>>>> +{
>>>> +	perf_pmu_unregister(priv);
>>>> +}
>>>> +
>>>> +static int hisi_ptt_register_pmu(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	char *pmu_name;
>>>> +	int ret;
>>>> +
>>>> +	hisi_ptt->hisi_ptt_pmu = (struct pmu) {
>>>> +		.module		= THIS_MODULE,
>>>> +		.capabilities	= PERF_PMU_CAP_EXCLUSIVE | PERF_PMU_CAP_ITRACE,
>>>> +		.task_ctx_nr	= perf_sw_context,
>>>> +		.attr_groups	= hisi_ptt_pmu_groups,
>>>> +		.event_init	= hisi_ptt_pmu_event_init,
>>>> +		.setup_aux	= hisi_ptt_pmu_setup_aux,
>>>> +		.free_aux	= hisi_ptt_pmu_free_aux,
>>>> +		.start		= hisi_ptt_pmu_start,
>>>> +		.stop		= hisi_ptt_pmu_stop,
>>>> +		.add		= hisi_ptt_pmu_add,
>>>> +		.del		= hisi_ptt_pmu_del,
>>>> +	};
>>>> +
>>>> +	pmu_name = devm_kasprintf(&hisi_ptt->pdev->dev, GFP_KERNEL, "hisi_ptt%u_%u",
>>>> +				  hisi_ptt->sicl_id, hisi_ptt->core_id);
>>>> +	if (!pmu_name)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	ret = perf_pmu_register(&hisi_ptt->hisi_ptt_pmu, pmu_name, -1);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	return devm_add_action_or_reset(&hisi_ptt->pdev->dev,
>>>> +					hisi_ptt_unregister_pmu,
>>>> +					&hisi_ptt->hisi_ptt_pmu);
>>>> +}
>>>> +
>>>> +/*
>>>> + * Get RMR address if provided by the firmware.
>>>> + * Return 0 if the IOMMU doesn't present or the policy of the
>>>> + * IOMMU domain is passthrough or we get a usable RMR region.
>>>> + * Otherwise a negative value is returned.
>>>> + */
>>>> +static int hisi_ptt_get_rmr(struct hisi_ptt *hisi_ptt)
>>>> +{
>>>> +	struct pci_dev *pdev = hisi_ptt->pdev;
>>>> +	struct iommu_domain *iommu_domain;
>>>> +	struct iommu_resv_region *region;
>>>> +	LIST_HEAD(list);
>>>> +
>>>> +	/*
>>>> +	 * Use direct DMA if IOMMU does not present or the policy of the
>>>> +	 * IOMMU domain is passthrough.
>>>> +	 */
>>>> +	iommu_domain = iommu_get_domain_for_dev(&pdev->dev);
>>>> +	if (!iommu_domain || iommu_domain->type == IOMMU_DOMAIN_IDENTITY)
>>>> +		return 0;
>>>> +
>>>> +	iommu_get_resv_regions(&pdev->dev, &list);
>>>> +	list_for_each_entry(region, &list, list)
>>>> +		if (region->type == IOMMU_RESV_DIRECT &&
>>>> +		    region->length >= HISI_PTT_TRACE_BUFFER_SIZE) {
>>>> +			hisi_ptt->trace_ctrl.has_rmr = true;
>>>> +			hisi_ptt->trace_ctrl.rmr_addr = region->start;
>>>> +			hisi_ptt->trace_ctrl.rmr_length = region->length;
>>>> +			break;
>>>> +		}
>>>> +
>>>> +	iommu_put_resv_regions(&pdev->dev, &list);
>>>> +	return hisi_ptt->trace_ctrl.has_rmr ? 0 : -ENOMEM;
>>>> +}
>>>> +
>>>> +static int hisi_ptt_probe(struct pci_dev *pdev,
>>>> +			  const struct pci_device_id *id)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt;
>>>> +	int ret;
>>>> +
>>>> +	hisi_ptt = devm_kzalloc(&pdev->dev, sizeof(*hisi_ptt), GFP_KERNEL);
>>>> +	if (!hisi_ptt)
>>>> +		return -ENOMEM;
>>>> +
>>>> +	mutex_init(&hisi_ptt->mutex);
>>>> +	hisi_ptt->pdev = pdev;
>>>> +
>>>> +	/*
>>>> +	 * Lifetime of pci_dev is longer than hisi_ptt,
>>>> +	 * so directly reference to the pci name string.
>>>> +	 */
>>>> +	hisi_ptt->name = pci_name(hisi_ptt->pdev);
>>>> +	pci_set_drvdata(pdev, hisi_ptt);
>>>> +
>>>> +	ret = pcim_enable_device(pdev);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to enable device, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	ret = pcim_iomap_regions(pdev, BIT(2), hisi_ptt->name);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to remap io memory, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	hisi_ptt->iobase = pcim_iomap_table(pdev)[2];
>>>> +
>>>> +	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to set 64 bit dma mask, ret = %d.\n", ret);
>>>> +		return ret;
>>>> +	}
>>>> +	pci_set_master(pdev);
>>>> +
>>>> +	ret = hisi_ptt_register_irq(hisi_ptt);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	hisi_ptt_init_ctrls(hisi_ptt);
>>>> +
>>>> +	ret = hisi_ptt_register_pmu(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to register pmu device, ret = %d", ret);
>>>> +		return ret;
>>>> +	}
>>>
>>> If this fails the pci_free_irq_vectors() is not called.
>>>
>>
>> It'll be called as we've registered a devm callback when
>> allocating the irq vector. See hisi_ptt_register_irq().
> 
> Ah yes - thanks for pointing that out.
> 
>>
>>> I am out of time for today - I will continue tomorrow.
>>>
>>
>> Sure. Thanks!
>> Yicong
>>
>>> Thanks,
>>> Mathieu
>>>
>>>> +
>>>> +	ret = hisi_ptt_get_rmr(hisi_ptt);
>>>> +	if (ret) {
>>>> +		pci_err(pdev, "failed to get RMR region, ret = %d", ret);
>>>> +		return ret;
>>>> +	}
>>>> +
>>>> +	/* Register the bus notifier for dynamically updating the filter list */
>>>> +	hisi_ptt->hisi_ptt_nb.notifier_call = hisi_ptt_notifier_call;
>>>> +	ret = bus_register_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>> +	if (ret)
>>>> +		pci_warn(pdev, "failed to register filter update notifier, ret = %d", ret);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +void hisi_ptt_remove(struct pci_dev *pdev)
>>>> +{
>>>> +	struct hisi_ptt *hisi_ptt = pci_get_drvdata(pdev);
>>>> +
>>>> +	bus_unregister_notifier(&pci_bus_type, &hisi_ptt->hisi_ptt_nb);
>>>> +
>>>> +	/* Cancel any work that has been queued */
>>>> +	cancel_delayed_work_sync(&hisi_ptt->work);
>>>> +
>>>> +	if (hisi_ptt->trace_ctrl.status == HISI_PTT_TRACE_STATUS_ON)
>>>> +		hisi_ptt_trace_end(hisi_ptt);
>>>> +
>>>> +	hisi_ptt_free_trace_buf(hisi_ptt);
>>>> +	hisi_ptt_release_filters(hisi_ptt);
>>>> +}
>>>> +
>>>> +static const struct pci_device_id hisi_ptt_id_tbl[] = {
>>>> +	{ PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa12e) },
>>>> +	{ }
>>>> +};
>>>> +MODULE_DEVICE_TABLE(pci, hisi_ptt_id_tbl);
>>>> +
>>>> +static struct pci_driver hisi_ptt_driver = {
>>>> +	.name = "hisi_ptt",
>>>> +	.id_table = hisi_ptt_id_tbl,
>>>> +	.probe = hisi_ptt_probe,
>>>> +	.remove = hisi_ptt_remove,
>>>> +};
>>>> +module_pci_driver(hisi_ptt_driver);
>>>> +
>>>> +MODULE_LICENSE("GPL v2");
>>>> +MODULE_AUTHOR("Yicong Yang <yangyicong@hisilicon.com>");
>>>> +MODULE_DESCRIPTION("Driver for HiSilicon PCIe tune and trace device");
>>>> -- 
>>>> 2.33.0
>>>>
>>> .
>>>
> .
> 

[PATCH v2 3/6] hwtracing: Add tune function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

Add tune function for the HiSilicon Tune and Trace device. The interface
of tune is exposed through sysfs attributes of PTT PMU device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/hwtracing/hisilicon/hisi_ptt.c | 167 +++++++++++++++++++++++++
 1 file changed, 167 insertions(+)

diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
index e11e9b6cc2a8..a1e1fb766376 100644
--- a/drivers/hwtracing/hisilicon/hisi_ptt.c
+++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
@@ -24,6 +24,11 @@
 #include <linux/vmalloc.h>
 #include <linux/workqueue.h>
 
+#define HISI_PTT_TUNING_CTRL		0x0000
+#define   HISI_PTT_TUNING_CTRL_CODE	GENMASK(15, 0)
+#define   HISI_PTT_TUNING_CTRL_SUB	GENMASK(23, 16)
+#define HISI_PTT_TUNING_DATA		0x0004
+#define   HISI_PTT_TUNING_DATA_VAL_MASK	GENMASK(15, 0)
 #define HISI_PTT_TRACE_ADDR_SIZE	0x0800
 #define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
 #define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
@@ -39,6 +44,8 @@
 #define HISI_PTT_TRACE_INT_STAT		0x0890
 #define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
 #define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TUNING_INT_STAT	0x0898
+#define   HISI_PTT_TUNING_INT_STAT_MASK	BIT(0)
 #define HISI_PTT_TRACE_WR_STS		0x08a0
 #define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
 #define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
@@ -71,6 +78,12 @@ enum hisi_ptt_trace_status {
 	HISI_PTT_TRACE_STATUS_ON,
 };
 
+struct hisi_ptt_tune_desc {
+	struct hisi_ptt *hisi_ptt;
+	const char *name;
+	u32 event_code;
+};
+
 struct hisi_ptt_dma_buflet {
 	struct list_head list;
 	unsigned int size;
@@ -177,6 +190,159 @@ static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
 	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
 }
 
+static int hisi_ptt_wait_tuning_finish(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TUNING_INT_STAT,
+				  val, !(val & HISI_PTT_TUNING_INT_STAT_MASK),
+				  HISI_PTT_WAIT_POLL_INTERVAL_US,
+				  HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static int hisi_ptt_tune_data_get(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 *data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	/* Write all 1 to indicates it's the read process */
+	writel(~0UL, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+	reg &= HISI_PTT_TUNING_DATA_VAL_MASK;
+	*data = (u16)reg;
+
+	return 0;
+}
+
+static int hisi_ptt_tune_data_set(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	reg = data;
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	return 0;
+}
+
+static ssize_t hisi_ptt_tune_attr_show(struct device *dev,
+				       struct device_attribute *attr,
+				       char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_get(hisi_ptt, desc->event_code, &val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : sysfs_emit(buf, "%u\n", val);
+}
+
+static ssize_t hisi_ptt_tune_attr_store(struct device *dev,
+					struct device_attribute *attr,
+					const char *buf, size_t count)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (kstrtou16(buf, 10, &val))
+		return -EINVAL;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_set(hisi_ptt, desc->event_code, val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : count;
+}
+
+#define HISI_PTT_TUNE_ATTR(_name, _val, _show, _store)			\
+	static struct hisi_ptt_tune_desc _name##_desc = {		\
+		.name = #_name,						\
+		.event_code = _val,					\
+	};								\
+	static struct dev_ext_attribute hisi_ptt_##_name##_attr = {	\
+		.attr	= __ATTR(_name, 0600, _show, _store),		\
+		.var	= &_name##_desc,				\
+	}
+
+#define HISI_PTT_TUNE_ATTR_COMMON(_name, _val)		\
+	HISI_PTT_TUNE_ATTR(_name, _val,			\
+			   hisi_ptt_tune_attr_show,	\
+			   hisi_ptt_tune_attr_store)
+
+/*
+ * The value of the tuning event are composed of two parts: main event code in bit[0,15] and
+ * subevent code in bit[16,23]. For example, qox_tx_cpl is a subevent of 'Tx path QoS control'
+ * which for tuning the weight of Tx completion TLPs. See hisi_ptt.rst documentation for
+ * more information.
+ */
+#define HISI_PTT_TUNE_QOS_TX_CPL				(0x4 | (3 << 16))
+#define HISI_PTT_TUNE_QOS_TX_NP					(0x4 | (4 << 16))
+#define HISI_PTT_TUNE_QOS_TX_P					(0x4 | (5 << 16))
+#define HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL	(0x5 | (6 << 16))
+#define HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL		(0x5 | (7 << 16))
+
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_cpl,
+			  HISI_PTT_TUNE_QOS_TX_CPL);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_np,
+			  HISI_PTT_TUNE_QOS_TX_NP);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_p,
+			  HISI_PTT_TUNE_QOS_TX_P);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_iob_rx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_tx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL);
+
+static struct attribute *hisi_ptt_tune_attrs[] = {
+	&hisi_ptt_qos_tx_cpl_attr.attr.attr,
+	&hisi_ptt_qos_tx_np_attr.attr.attr,
+	&hisi_ptt_qos_tx_p_attr.attr.attr,
+	&hisi_ptt_tx_path_iob_rx_req_alloc_buf_level_attr.attr.attr,
+	&hisi_ptt_tx_path_tx_req_alloc_buf_level_attr.attr.attr,
+	NULL,
+};
+
+static struct attribute_group hisi_ptt_tune_group = {
+	.attrs	= hisi_ptt_tune_attrs,
+	.name	= "tune",
+};
+
 static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
 {
 	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
@@ -677,6 +843,7 @@ static struct attribute_group hisi_ptt_pmu_filter_group = {
 static const struct attribute_group *hisi_ptt_pmu_groups[] = {
 	&hisi_ptt_pmu_format_group,
 	&hisi_ptt_pmu_filter_group,
+	&hisi_ptt_tune_group,
 	NULL
 };
 
-- 
2.33.0

[PATCH v2 3/6] hwtracing: Add tune function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang via iommu]

Add tune function for the HiSilicon Tune and Trace device. The interface
of tune is exposed through sysfs attributes of PTT PMU device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/hwtracing/hisilicon/hisi_ptt.c | 167 +++++++++++++++++++++++++
 1 file changed, 167 insertions(+)

diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
index e11e9b6cc2a8..a1e1fb766376 100644
--- a/drivers/hwtracing/hisilicon/hisi_ptt.c
+++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
@@ -24,6 +24,11 @@
 #include <linux/vmalloc.h>
 #include <linux/workqueue.h>
 
+#define HISI_PTT_TUNING_CTRL		0x0000
+#define   HISI_PTT_TUNING_CTRL_CODE	GENMASK(15, 0)
+#define   HISI_PTT_TUNING_CTRL_SUB	GENMASK(23, 16)
+#define HISI_PTT_TUNING_DATA		0x0004
+#define   HISI_PTT_TUNING_DATA_VAL_MASK	GENMASK(15, 0)
 #define HISI_PTT_TRACE_ADDR_SIZE	0x0800
 #define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
 #define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
@@ -39,6 +44,8 @@
 #define HISI_PTT_TRACE_INT_STAT		0x0890
 #define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
 #define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TUNING_INT_STAT	0x0898
+#define   HISI_PTT_TUNING_INT_STAT_MASK	BIT(0)
 #define HISI_PTT_TRACE_WR_STS		0x08a0
 #define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
 #define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
@@ -71,6 +78,12 @@ enum hisi_ptt_trace_status {
 	HISI_PTT_TRACE_STATUS_ON,
 };
 
+struct hisi_ptt_tune_desc {
+	struct hisi_ptt *hisi_ptt;
+	const char *name;
+	u32 event_code;
+};
+
 struct hisi_ptt_dma_buflet {
 	struct list_head list;
 	unsigned int size;
@@ -177,6 +190,159 @@ static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
 	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
 }
 
+static int hisi_ptt_wait_tuning_finish(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TUNING_INT_STAT,
+				  val, !(val & HISI_PTT_TUNING_INT_STAT_MASK),
+				  HISI_PTT_WAIT_POLL_INTERVAL_US,
+				  HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static int hisi_ptt_tune_data_get(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 *data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	/* Write all 1 to indicates it's the read process */
+	writel(~0UL, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+	reg &= HISI_PTT_TUNING_DATA_VAL_MASK;
+	*data = (u16)reg;
+
+	return 0;
+}
+
+static int hisi_ptt_tune_data_set(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	reg = data;
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	return 0;
+}
+
+static ssize_t hisi_ptt_tune_attr_show(struct device *dev,
+				       struct device_attribute *attr,
+				       char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_get(hisi_ptt, desc->event_code, &val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : sysfs_emit(buf, "%u\n", val);
+}
+
+static ssize_t hisi_ptt_tune_attr_store(struct device *dev,
+					struct device_attribute *attr,
+					const char *buf, size_t count)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (kstrtou16(buf, 10, &val))
+		return -EINVAL;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_set(hisi_ptt, desc->event_code, val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : count;
+}
+
+#define HISI_PTT_TUNE_ATTR(_name, _val, _show, _store)			\
+	static struct hisi_ptt_tune_desc _name##_desc = {		\
+		.name = #_name,						\
+		.event_code = _val,					\
+	};								\
+	static struct dev_ext_attribute hisi_ptt_##_name##_attr = {	\
+		.attr	= __ATTR(_name, 0600, _show, _store),		\
+		.var	= &_name##_desc,				\
+	}
+
+#define HISI_PTT_TUNE_ATTR_COMMON(_name, _val)		\
+	HISI_PTT_TUNE_ATTR(_name, _val,			\
+			   hisi_ptt_tune_attr_show,	\
+			   hisi_ptt_tune_attr_store)
+
+/*
+ * The value of the tuning event are composed of two parts: main event code in bit[0,15] and
+ * subevent code in bit[16,23]. For example, qox_tx_cpl is a subevent of 'Tx path QoS control'
+ * which for tuning the weight of Tx completion TLPs. See hisi_ptt.rst documentation for
+ * more information.
+ */
+#define HISI_PTT_TUNE_QOS_TX_CPL				(0x4 | (3 << 16))
+#define HISI_PTT_TUNE_QOS_TX_NP					(0x4 | (4 << 16))
+#define HISI_PTT_TUNE_QOS_TX_P					(0x4 | (5 << 16))
+#define HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL	(0x5 | (6 << 16))
+#define HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL		(0x5 | (7 << 16))
+
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_cpl,
+			  HISI_PTT_TUNE_QOS_TX_CPL);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_np,
+			  HISI_PTT_TUNE_QOS_TX_NP);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_p,
+			  HISI_PTT_TUNE_QOS_TX_P);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_iob_rx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_tx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL);
+
+static struct attribute *hisi_ptt_tune_attrs[] = {
+	&hisi_ptt_qos_tx_cpl_attr.attr.attr,
+	&hisi_ptt_qos_tx_np_attr.attr.attr,
+	&hisi_ptt_qos_tx_p_attr.attr.attr,
+	&hisi_ptt_tx_path_iob_rx_req_alloc_buf_level_attr.attr.attr,
+	&hisi_ptt_tx_path_tx_req_alloc_buf_level_attr.attr.attr,
+	NULL,
+};
+
+static struct attribute_group hisi_ptt_tune_group = {
+	.attrs	= hisi_ptt_tune_attrs,
+	.name	= "tune",
+};
+
 static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
 {
 	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
@@ -677,6 +843,7 @@ static struct attribute_group hisi_ptt_pmu_filter_group = {
 static const struct attribute_group *hisi_ptt_pmu_groups[] = {
 	&hisi_ptt_pmu_format_group,
 	&hisi_ptt_pmu_filter_group,
+	&hisi_ptt_tune_group,
 	NULL
 };
 
-- 
2.33.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 3/6] hwtracing: Add tune function support for HiSilicon PCIe Tune and Trace device

[Yicong Yang]

Add tune function for the HiSilicon Tune and Trace device. The interface
of tune is exposed through sysfs attributes of PTT PMU device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 drivers/hwtracing/hisilicon/hisi_ptt.c | 167 +++++++++++++++++++++++++
 1 file changed, 167 insertions(+)

diff --git a/drivers/hwtracing/hisilicon/hisi_ptt.c b/drivers/hwtracing/hisilicon/hisi_ptt.c
index e11e9b6cc2a8..a1e1fb766376 100644
--- a/drivers/hwtracing/hisilicon/hisi_ptt.c
+++ b/drivers/hwtracing/hisilicon/hisi_ptt.c
@@ -24,6 +24,11 @@
 #include <linux/vmalloc.h>
 #include <linux/workqueue.h>
 
+#define HISI_PTT_TUNING_CTRL		0x0000
+#define   HISI_PTT_TUNING_CTRL_CODE	GENMASK(15, 0)
+#define   HISI_PTT_TUNING_CTRL_SUB	GENMASK(23, 16)
+#define HISI_PTT_TUNING_DATA		0x0004
+#define   HISI_PTT_TUNING_DATA_VAL_MASK	GENMASK(15, 0)
 #define HISI_PTT_TRACE_ADDR_SIZE	0x0800
 #define HISI_PTT_TRACE_ADDR_BASE_LO_0	0x0810
 #define HISI_PTT_TRACE_ADDR_BASE_HI_0	0x0814
@@ -39,6 +44,8 @@
 #define HISI_PTT_TRACE_INT_STAT		0x0890
 #define   HISI_PTT_TRACE_INT_STAT_MASK	GENMASK(3, 0)
 #define HISI_PTT_TRACE_INT_MASK		0x0894
+#define HISI_PTT_TUNING_INT_STAT	0x0898
+#define   HISI_PTT_TUNING_INT_STAT_MASK	BIT(0)
 #define HISI_PTT_TRACE_WR_STS		0x08a0
 #define   HISI_PTT_TRACE_WR_STS_WRITE	GENMASK(27, 0)
 #define   HISI_PTT_TRACE_WR_STS_BUFFER	GENMASK(29, 28)
@@ -71,6 +78,12 @@ enum hisi_ptt_trace_status {
 	HISI_PTT_TRACE_STATUS_ON,
 };
 
+struct hisi_ptt_tune_desc {
+	struct hisi_ptt *hisi_ptt;
+	const char *name;
+	u32 event_code;
+};
+
 struct hisi_ptt_dma_buflet {
 	struct list_head list;
 	unsigned int size;
@@ -177,6 +190,159 @@ static inline struct hisi_ptt *to_hisi_ptt(struct pmu *pmu)
 	return container_of(pmu, struct hisi_ptt, hisi_ptt_pmu);
 }
 
+static int hisi_ptt_wait_tuning_finish(struct hisi_ptt *hisi_ptt)
+{
+	u32 val;
+
+	return readl_poll_timeout(hisi_ptt->iobase + HISI_PTT_TUNING_INT_STAT,
+				  val, !(val & HISI_PTT_TUNING_INT_STAT_MASK),
+				  HISI_PTT_WAIT_POLL_INTERVAL_US,
+				  HISI_PTT_WAIT_TIMEOUT_US);
+}
+
+static int hisi_ptt_tune_data_get(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 *data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	/* Write all 1 to indicates it's the read process */
+	writel(~0UL, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+	reg &= HISI_PTT_TUNING_DATA_VAL_MASK;
+	*data = (u16)reg;
+
+	return 0;
+}
+
+static int hisi_ptt_tune_data_set(struct hisi_ptt *hisi_ptt,
+				  u32 event, u16 data)
+{
+	u32 reg;
+
+	reg = readl(hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+	reg &= ~(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB);
+	reg |= FIELD_PREP(HISI_PTT_TUNING_CTRL_CODE | HISI_PTT_TUNING_CTRL_SUB,
+			  event);
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_CTRL);
+
+	reg = data;
+	writel(reg, hisi_ptt->iobase + HISI_PTT_TUNING_DATA);
+
+	if (hisi_ptt_wait_tuning_finish(hisi_ptt))
+		return -ETIMEDOUT;
+
+	return 0;
+}
+
+static ssize_t hisi_ptt_tune_attr_show(struct device *dev,
+				       struct device_attribute *attr,
+				       char *buf)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_get(hisi_ptt, desc->event_code, &val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : sysfs_emit(buf, "%u\n", val);
+}
+
+static ssize_t hisi_ptt_tune_attr_store(struct device *dev,
+					struct device_attribute *attr,
+					const char *buf, size_t count)
+{
+	struct hisi_ptt *hisi_ptt = to_hisi_ptt(dev_get_drvdata(dev));
+	struct dev_ext_attribute *ext_attr;
+	struct hisi_ptt_tune_desc *desc;
+	int ret;
+	u16 val;
+
+	ext_attr = container_of(attr, struct dev_ext_attribute, attr);
+	desc = ext_attr->var;
+
+	if (kstrtou16(buf, 10, &val))
+		return -EINVAL;
+
+	if (!mutex_trylock(&hisi_ptt->mutex))
+		return -EBUSY;
+
+	ret = hisi_ptt_tune_data_set(hisi_ptt, desc->event_code, val);
+
+	mutex_unlock(&hisi_ptt->mutex);
+	return ret ? ret : count;
+}
+
+#define HISI_PTT_TUNE_ATTR(_name, _val, _show, _store)			\
+	static struct hisi_ptt_tune_desc _name##_desc = {		\
+		.name = #_name,						\
+		.event_code = _val,					\
+	};								\
+	static struct dev_ext_attribute hisi_ptt_##_name##_attr = {	\
+		.attr	= __ATTR(_name, 0600, _show, _store),		\
+		.var	= &_name##_desc,				\
+	}
+
+#define HISI_PTT_TUNE_ATTR_COMMON(_name, _val)		\
+	HISI_PTT_TUNE_ATTR(_name, _val,			\
+			   hisi_ptt_tune_attr_show,	\
+			   hisi_ptt_tune_attr_store)
+
+/*
+ * The value of the tuning event are composed of two parts: main event code in bit[0,15] and
+ * subevent code in bit[16,23]. For example, qox_tx_cpl is a subevent of 'Tx path QoS control'
+ * which for tuning the weight of Tx completion TLPs. See hisi_ptt.rst documentation for
+ * more information.
+ */
+#define HISI_PTT_TUNE_QOS_TX_CPL				(0x4 | (3 << 16))
+#define HISI_PTT_TUNE_QOS_TX_NP					(0x4 | (4 << 16))
+#define HISI_PTT_TUNE_QOS_TX_P					(0x4 | (5 << 16))
+#define HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL	(0x5 | (6 << 16))
+#define HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL		(0x5 | (7 << 16))
+
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_cpl,
+			  HISI_PTT_TUNE_QOS_TX_CPL);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_np,
+			  HISI_PTT_TUNE_QOS_TX_NP);
+HISI_PTT_TUNE_ATTR_COMMON(qos_tx_p,
+			  HISI_PTT_TUNE_QOS_TX_P);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_iob_rx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_IOB_RX_REQ_ALLOC_BUF_LEVEL);
+HISI_PTT_TUNE_ATTR_COMMON(tx_path_tx_req_alloc_buf_level,
+			  HISI_PTT_TUNE_TX_PATH_TX_REQ_ALLOC_BUF_LEVEL);
+
+static struct attribute *hisi_ptt_tune_attrs[] = {
+	&hisi_ptt_qos_tx_cpl_attr.attr.attr,
+	&hisi_ptt_qos_tx_np_attr.attr.attr,
+	&hisi_ptt_qos_tx_p_attr.attr.attr,
+	&hisi_ptt_tx_path_iob_rx_req_alloc_buf_level_attr.attr.attr,
+	&hisi_ptt_tx_path_tx_req_alloc_buf_level_attr.attr.attr,
+	NULL,
+};
+
+static struct attribute_group hisi_ptt_tune_group = {
+	.attrs	= hisi_ptt_tune_attrs,
+	.name	= "tune",
+};
+
 static u16 hisi_ptt_get_filter_val(struct pci_dev *pdev)
 {
 	if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT)
@@ -677,6 +843,7 @@ static struct attribute_group hisi_ptt_pmu_filter_group = {
 static const struct attribute_group *hisi_ptt_pmu_groups[] = {
 	&hisi_ptt_pmu_format_group,
 	&hisi_ptt_pmu_filter_group,
+	&hisi_ptt_tune_group,
 	NULL
 };
 
-- 
2.33.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 4/6] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver

[Yicong Yang]

From: Qi Liu <liuqi115@huawei.com>

'perf record' and 'perf report --dump-raw-trace' supported in this
patch.

Example usage:

Output will contain raw PTT data and its textual representation, such
as:

0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
.
. ... HISI PTT data: size 4194304 bytes
.  00000000: 00 00 00 00                                 Prefix
.  00000004: 08 20 00 60                                 Header DW0
.  00000008: ff 02 00 01                                 Header DW1
.  0000000c: 20 08 00 00                                 Header DW2
.  00000010: 10 e7 44 ab                                 Header DW3
.  00000014: 2a a8 1e 01                                 Time
.  00000020: 00 00 00 00                                 Prefix
.  00000024: 01 00 00 60                                 Header DW0
.  00000028: 0f 1e 00 01                                 Header DW1
.  0000002c: 04 00 00 00                                 Header DW2
.  00000030: 40 00 81 02                                 Header DW3
.  00000034: ee 02 00 00                                 Time
....

Signed-off-by: Qi Liu <liuqi115@huawei.com>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 tools/perf/arch/arm/util/auxtrace.c           |  56 ++++-
 tools/perf/arch/arm/util/pmu.c                |   3 +
 tools/perf/arch/arm64/util/Build              |   2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         | 195 +++++++++++++++
 tools/perf/util/Build                         |   2 +
 tools/perf/util/auxtrace.c                    |   4 +
 tools/perf/util/auxtrace.h                    |   1 +
 tools/perf/util/hisi-ptt-decoder/Build        |   1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 +++++++++++++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
 tools/perf/util/hisi_ptt.c                    | 228 ++++++++++++++++++
 tools/perf/util/hisi_ptt.h                    |  28 +++
 12 files changed, 714 insertions(+), 4 deletions(-)
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
index 5fc6a2a3dbc5..dc9d2172464e 100644
--- a/tools/perf/arch/arm/util/auxtrace.c
+++ b/tools/perf/arch/arm/util/auxtrace.c
@@ -4,6 +4,7 @@
  * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
  */
 
+#include <dirent.h>
 #include <stdbool.h>
 #include <linux/coresight-pmu.h>
 #include <linux/zalloc.h>
@@ -14,6 +15,7 @@
 #include "../../../util/pmu.h"
 #include "cs-etm.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 
 static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 {
@@ -50,6 +52,39 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 	return arm_spe_pmus;
 }
 
+static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
+{
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+	int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+	DIR *dir = NULL;
+	struct dirent *dent;
+
+	hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * nr_cpus);
+	if (!hisi_ptt_pmus) {
+		pr_err("hisi_ptt alloc failed\n");
+		*err = -ENOMEM;
+		return NULL;
+	}
+
+	dir = opendir("/sys/devices");
+	dent = readdir(dir);
+	while (dent) {
+		if (strstr(dent->d_name, HISI_PTT_PMU_NAME)) {
+			hisi_ptt_pmus[*nr_ptts] = perf_pmu__find(dent->d_name);
+			if (hisi_ptt_pmus[*nr_ptts]) {
+				pr_debug2("%s %d: arm_spe_pmu %d type %d name %s\n",
+					  __func__, __LINE__, *nr_ptts,
+					  hisi_ptt_pmus[*nr_ptts]->type,
+					  hisi_ptt_pmus[*nr_ptts]->name);
+				(*nr_ptts)++;
+			}
+		}
+		dent = readdir(dir);
+	}
+
+	return hisi_ptt_pmus;
+}
+
 struct auxtrace_record
 *auxtrace_record__init(struct evlist *evlist, int *err)
 {
@@ -57,8 +92,12 @@ struct auxtrace_record
 	struct evsel *evsel;
 	bool found_etm = false;
 	struct perf_pmu *found_spe = NULL;
+	struct perf_pmu *found_ptt = NULL;
 	struct perf_pmu **arm_spe_pmus = NULL;
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+
 	int nr_spes = 0;
+	int nr_ptts = 0;
 	int i = 0;
 
 	if (!evlist)
@@ -66,13 +105,14 @@ struct auxtrace_record
 
 	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
 	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
+	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
 
 	evlist__for_each_entry(evlist, evsel) {
 		if (cs_etm_pmu &&
 		    evsel->core.attr.type == cs_etm_pmu->type)
 			found_etm = true;
 
-		if (!nr_spes || found_spe)
+		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
 			continue;
 
 		for (i = 0; i < nr_spes; i++) {
@@ -81,11 +121,18 @@ struct auxtrace_record
 				break;
 			}
 		}
+
+		for (i = 0; i < nr_ptts; i++) {
+			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
+				found_ptt = hisi_ptt_pmus[i];
+				break;
+			}
+		}
 	}
 	free(arm_spe_pmus);
 
-	if (found_etm && found_spe) {
-		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
+	if (found_etm && found_spe && found_ptt) {
+		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
 		*err = -EOPNOTSUPP;
 		return NULL;
 	}
@@ -96,6 +143,9 @@ struct auxtrace_record
 #if defined(__aarch64__)
 	if (found_spe)
 		return arm_spe_recording_init(err, found_spe);
+
+	if (found_ptt)
+		return hisi_ptt_recording_init(err, found_ptt);
 #endif
 
 	/*
diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
index b8b23b9dc598..89a3cedb4557 100644
--- a/tools/perf/arch/arm/util/pmu.c
+++ b/tools/perf/arch/arm/util/pmu.c
@@ -10,6 +10,7 @@
 #include <linux/string.h>
 
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "../../../util/pmu.h"
 
 struct perf_event_attr
@@ -22,6 +23,8 @@ struct perf_event_attr
 #if defined(__aarch64__)
 	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
 		return arm_spe_pmu_default_config(pmu);
+	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
+		pmu->selectable = true;
 #endif
 	}
 
diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
index 9fcb4e68add9..8b7fd1dc9f37 100644
--- a/tools/perf/arch/arm64/util/Build
+++ b/tools/perf/arch/arm64/util/Build
@@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
 perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
 			      ../../arm/util/auxtrace.o \
 			      ../../arm/util/cs-etm.o \
-			      arm-spe.o mem-events.o
+			      arm-spe.o mem-events.o hisi_ptt.o
diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
new file mode 100644
index 000000000000..337c61b1d637
--- /dev/null
+++ b/tools/perf/arch/arm64/util/hisi_ptt.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/log2.h>
+#include <linux/zalloc.h>
+#include <time.h>
+
+#include <internal/lib.h> // page_size
+#include "../../../util/auxtrace.h"
+#include "../../../util/cpumap.h"
+#include "../../../util/debug.h"
+#include "../../../util/event.h"
+#include "../../../util/evlist.h"
+#include "../../../util/evsel.h"
+#include "../../../util/hisi_ptt.h"
+#include "../../../util/pmu.h"
+#include "../../../util/record.h"
+#include "../../../util/session.h"
+#include "../../../util/tsc.h"
+
+#define DEFAULT_PAGE_SIZE 1024
+#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
+#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
+
+struct hisi_ptt_recording {
+	struct auxtrace_record	itr;
+	struct perf_pmu *hisi_ptt_pmu;
+	struct evlist *evlist;
+};
+
+static size_t
+hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
+			struct evlist *evlist __maybe_unused)
+{
+	return HISI_PTT_AUXTRACE_PRIV_SIZE;
+}
+
+static int hisi_ptt_info_fill(struct auxtrace_record *itr,
+			      struct perf_session *session,
+			      struct perf_record_auxtrace_info *auxtrace_info,
+			      size_t priv_size)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+
+	if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
+		return -EINVAL;
+
+	if (!session->evlist->core.nr_mmaps)
+		return -EINVAL;
+
+	auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
+	auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
+
+	return 0;
+}
+
+static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
+{
+	bool privileged = perf_event_paranoid_check(-1);
+
+	if (!opts->full_auxtrace)
+		return 0;
+
+	if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
+		if (privileged) {
+			opts->auxtrace_mmap_pages = MiB(16) / page_size;
+		} else {
+			opts->auxtrace_mmap_pages = KiB(128) / page_size;
+			if (opts->mmap_pages == UINT_MAX)
+				opts->mmap_pages = KiB(256) / page_size;
+		}
+	}
+
+	/* Validate auxtrace_mmap_pages */
+	if (opts->auxtrace_mmap_pages) {
+		size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
+		size_t min_sz = KiB(8);
+
+		if (sz < min_sz || !is_power_of_2(sz)) {
+			pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
+			       min_sz / DEFAULT_PAGE_SIZE);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_recording_options(struct auxtrace_record *itr,
+				      struct evlist *evlist,
+				      struct record_opts *opts)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+	struct perf_cpu_map *cpus = evlist->core.cpus;
+	struct evsel *evsel, *hisi_ptt_evsel = NULL;
+	struct evsel *tracking_evsel;
+	int err;
+
+	pttr->evlist = evlist;
+	evlist__for_each_entry(evlist, evsel) {
+		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
+			if (hisi_ptt_evsel) {
+				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
+				return -EINVAL;
+			}
+			evsel->core.attr.freq = 0;
+			evsel->core.attr.sample_period = 1;
+			hisi_ptt_evsel = evsel;
+			opts->full_auxtrace = true;
+		}
+	}
+
+	err = hisi_ptt_set_auxtrace_mmap_page(opts);
+	if (err)
+		return err;
+	/*
+	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
+	 * must come first.
+	 */
+	evlist__to_front(evlist, hisi_ptt_evsel);
+
+	if (!perf_cpu_map__empty(cpus)) {
+		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
+		evsel__set_sample_bit(hisi_ptt_evsel, CPU);
+	}
+
+	/* Add dummy event to keep tracking */
+	err = parse_events(evlist, "dummy:u", NULL);
+	if (err)
+		return err;
+
+	tracking_evsel = evlist__last(evlist);
+	evlist__set_tracking_event(evlist, tracking_evsel);
+
+	tracking_evsel->core.attr.freq = 0;
+	tracking_evsel->core.attr.sample_period = 1;
+
+	if (!perf_cpu_map__empty(cpus))
+		evsel__set_sample_bit(tracking_evsel, TIME);
+
+	return 0;
+}
+
+static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
+{
+	return rdtsc();
+}
+
+static void hisi_ptt_recording_free(struct auxtrace_record *itr)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+
+	free(pttr);
+}
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu)
+{
+	struct hisi_ptt_recording *pttr;
+
+	if (!hisi_ptt_pmu) {
+		*err = -ENODEV;
+		return NULL;
+	}
+
+	pttr = zalloc(sizeof(struct hisi_ptt_recording));
+	if (!pttr) {
+		*err = -ENOMEM;
+		return NULL;
+	}
+
+	pttr->hisi_ptt_pmu = hisi_ptt_pmu;
+	pttr->itr.pmu = hisi_ptt_pmu;
+	pttr->itr.recording_options = hisi_ptt_recording_options;
+	pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
+	pttr->itr.info_fill = hisi_ptt_info_fill;
+	pttr->itr.free = hisi_ptt_recording_free;
+	pttr->itr.reference = hisi_ptt_reference;
+	pttr->itr.read_finish = auxtrace_record__read_finish;
+	pttr->itr.alignment = 0;
+
+	*err = 0;
+	return &pttr->itr;
+}
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 2e5bfbb69960..4afeabd87483 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -114,6 +114,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
 perf-$(CONFIG_AUXTRACE) += intel-bts.o
 perf-$(CONFIG_AUXTRACE) += arm-spe.o
 perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
+perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
 perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
 
 ifdef CONFIG_LIBOPENCSD
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index c679394b898d..0826d7e05d63 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -51,6 +51,7 @@
 #include "intel-pt.h"
 #include "intel-bts.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "s390-cpumsf.h"
 #include "util/mmap.h"
 
@@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
 	case PERF_AUXTRACE_S390_CPUMSF:
 		err = s390_cpumsf_process_auxtrace_info(event, session);
 		break;
+	case PERF_AUXTRACE_HISI_PTT:
+		err = hisi_ptt_process_auxtrace_info(event, session);
+		break;
 	case PERF_AUXTRACE_UNKNOWN:
 	default:
 		return -EINVAL;
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index bbf0d78c6401..51d37b7dce9d 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -47,6 +47,7 @@ enum auxtrace_type {
 	PERF_AUXTRACE_CS_ETM,
 	PERF_AUXTRACE_ARM_SPE,
 	PERF_AUXTRACE_S390_CPUMSF,
+	PERF_AUXTRACE_HISI_PTT,
 };
 
 enum itrace_period_type {
diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
new file mode 100644
index 000000000000..db3db8b75033
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/Build
@@ -0,0 +1 @@
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
new file mode 100644
index 000000000000..ad439bf409bb
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <endian.h>
+#include <byteswap.h>
+#include <linux/bitops.h>
+#include <stdarg.h>
+
+#include "../color.h"
+#include "hisi-ptt-pkt-decoder.h"
+
+/*
+ * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
+ * used to distinguish the data format.
+ * 8DW format is like:
+ *   bits [                 31:11                 ][       10:0       ]
+ *        |---------------------------------------|-------------------|
+ *    DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+ *    DW1 [                       Prefix                              ]
+ *    DW2 [                     Header DW0                            ]
+ *    DW3 [                     Header DW1                            ]
+ *    DW4 [                     Header DW2                            ]
+ *    DW5 [                     Header DW3                            ]
+ *    DW6 [                   Reserved (0x0)                          ]
+ *    DW7 [                        Time                               ]
+ *
+ * 4DW format is like:
+ *   bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+ *        |-----|---------|---|---|---|---|-------------|-------------|
+ *    DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+ *    DW1 [                     Header DW1                            ]
+ *    DW2 [                     Header DW2                            ]
+ *    DW3 [                     Header DW3                            ]
+ */
+
+enum hisi_ptt_8dw_pkt_field_type {
+	HISI_PTT_8DW_PREFIX,
+	HISI_PTT_8DW_HEAD0,
+	HISI_PTT_8DW_HEAD1,
+	HISI_PTT_8DW_HEAD2,
+	HISI_PTT_8DW_HEAD3,
+	HISI_PTT_8DW_TIME,
+	HISI_PTT_8DW_TYPE_MAX
+};
+
+enum hisi_ptt_4dw_pkt_field_type {
+	HISI_PTT_4DW_HEAD1,
+	HISI_PTT_4DW_HEAD2,
+	HISI_PTT_4DW_HEAD3,
+	HISI_PTT_4DW_TYPE_MAX
+};
+
+static const char * const hisi_ptt_8dw_pkt_field_name[] = {
+	[HISI_PTT_8DW_PREFIX]	= "Prefix",
+	[HISI_PTT_8DW_HEAD0]	= "Header DW0",
+	[HISI_PTT_8DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_8DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_8DW_HEAD3]	= "Header DW3",
+	[HISI_PTT_8DW_TIME]	= "Time",
+};
+
+static const char * const hisi_ptt_4dw_pkt_field_name[] = {
+	[HISI_PTT_4DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_4DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_4DW_HEAD3]	= "Header DW3",
+};
+
+/* offset of each member is determined by format of 8dw packet. */
+static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
+	[HISI_PTT_8DW_PREFIX]	= 4,
+	[HISI_PTT_8DW_HEAD0]	= 4,
+	[HISI_PTT_8DW_HEAD1]	= 4,
+	[HISI_PTT_8DW_HEAD2]	= 4,
+	[HISI_PTT_8DW_HEAD3]	= 4,
+	[HISI_PTT_8DW_TIME]	= 8,
+};
+
+union hisi_ptt_4dw {
+	struct {
+		uint32_t format : 2;
+		uint32_t type : 5;
+		uint32_t t9 : 1;
+		uint32_t t8 : 1;
+		uint32_t th : 1;
+		uint32_t so : 1;
+		uint32_t len : 10;
+		uint32_t time : 11;
+	};
+	uint32_t value;
+};
+
+static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	int i;
+
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+	color_fprintf(stdout, color, "  %s\n", desc);
+	*pos += field_len;
+}
+
+static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
+		buf += hisi_ptt_8dw_pkt_field_offset[i];
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_8DW_PKT_SIZE;
+}
+
+static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	union hisi_ptt_4dw dw0;
+	int i;
+
+	dw0.value = *(uint32_t *)buf;
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+
+	color_fprintf(stdout, color,
+		      "  %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
+		      "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
+		      "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
+		      dw0.len, "Time", dw0.time);
+
+	*pos += field_len;
+}
+
+static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	hisi_ptt_4dw_print_dw0(buf, &pos);
+
+	for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
+		buf += sizeof(uint32_t);
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_4DW_PKT_SIZE;
+}
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
+{
+	if (type == HISI_PTT_8DW_PKT)
+		return hisi_ptt_8dw_kpt_desc(buf, pos);
+
+	return hisi_ptt_4dw_kpt_desc(buf, pos);
+}
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
new file mode 100644
index 000000000000..f9eda1f9d0d3
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
+#define INCLUDE__HISI_PTT_PKT_DECODER_H__
+
+#include <stddef.h>
+#include <stdint.h>
+
+#define HISI_PTT_PKT_DESC_MAX		256
+#define HISI_PTT_NEED_MORE_BYTES	-1
+#define HISI_PTT_8DW_CHECK_MASK		GENMASK(31, 11)
+#define HISI_PTT_IS_8DW_PKT		GENMASK(31, 11)
+#define HISI_PTT_8DW_PKT_SIZE		32
+#define HISI_PTT_4DW_PKT_SIZE		16
+#define HISI_PTT_MAX_SPACE_LEN		10
+
+enum hisi_ptt_pkt_type {
+	HISI_PTT_4DW_PKT,
+	HISI_PTT_8DW_PKT,
+};
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
+
+#endif
diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
new file mode 100644
index 000000000000..541e67f7259e
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.c
@@ -0,0 +1,228 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <byteswap.h>
+#include <endian.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/types.h>
+#include <linux/zalloc.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "auxtrace.h"
+#include "color.h"
+#include "debug.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "hisi_ptt.h"
+#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
+#include "machine.h"
+#include "session.h"
+#include "symbol.h"
+#include "tool.h"
+#include "util/synthetic-events.h"
+
+struct hisi_ptt {
+	struct auxtrace auxtrace;
+	struct auxtrace_queues queues;
+	u32 auxtrace_type;
+	struct perf_session *session;
+	struct machine *machine;
+	u32 pmu_type;
+};
+
+struct hisi_ptt_queue {
+	struct hisi_ptt *ptt;
+	struct auxtrace_buffer *buffer;
+};
+
+static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
+{
+	uint32_t head = *(uint32_t *)buf;
+
+	if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
+		return HISI_PTT_8DW_PKT;
+
+	return HISI_PTT_4DW_PKT;
+}
+
+static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
+			  unsigned char *buf, size_t len)
+{
+	const char *color = PERF_COLOR_BLUE;
+	enum hisi_ptt_pkt_type type;
+	size_t pos = 0;
+	int pkt_len;
+
+	color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
+		      len);
+
+	type = hisi_ptt_check_packet_type(buf);
+	while (len) {
+		pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
+		if (!pkt_len)
+			color_fprintf(stdout, color, " Bad packet!\n");
+
+		pos += pkt_len;
+		buf += pkt_len;
+		len -= pkt_len;
+	}
+}
+
+static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
+				size_t len)
+{
+	printf(".\n");
+
+	hisi_ptt_dump(ptt, buf, len);
+}
+
+static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
+				  union perf_event *event __maybe_unused,
+				  struct perf_sample *sample __maybe_unused,
+				  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
+					   union perf_event *event,
+					   struct perf_tool *tool __maybe_unused)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_buffer *buffer;
+	off_t data_offset;
+	int fd = perf_data__fd(session->data);
+	int err;
+
+	if (perf_data__is_pipe(session->data)) {
+		data_offset = 0;
+	} else {
+		data_offset = lseek(fd, 0, SEEK_CUR);
+		if (data_offset == -1)
+			return -errno;
+	}
+
+	err = auxtrace_queues__add_event(&ptt->queues, session, event,
+					 data_offset, &buffer);
+	if (err)
+		return err;
+
+	if (dump_trace) {
+		if (auxtrace_buffer__get_data(buffer, fd)) {
+			hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
+			auxtrace_buffer__put_data(buffer);
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
+			  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static void hisi_ptt_free_queue(void *priv)
+{
+	struct hisi_ptt_queue *pttq = priv;
+
+	if (!pttq)
+		return;
+
+	free(pttq);
+}
+
+static void hisi_ptt_free_events(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_queues *queues = &ptt->queues;
+	unsigned int i;
+
+	for (i = 0; i < queues->nr_queues; i++) {
+		hisi_ptt_free_queue(queues->queue_array[i].priv);
+		queues->queue_array[i].priv = NULL;
+	}
+	auxtrace_queues__free(queues);
+}
+
+static void hisi_ptt_free(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+
+	hisi_ptt_free_events(session);
+	session->auxtrace = NULL;
+	free(ptt);
+}
+
+static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
+				       struct evsel *evsel)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
+
+	return evsel->core.attr.type == ptt->pmu_type;
+}
+
+static const char * const hisi_ptt_info_fmts[] = {
+	[HISI_PTT_PMU_TYPE]		= "  PMU Type           %" PRId64 "\n",
+};
+
+static void hisi_ptt_print_info(__u64 *arr)
+{
+	if (!dump_trace)
+		return;
+
+	fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
+}
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session)
+{
+	struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
+	struct hisi_ptt *ptt;
+	int err;
+
+	if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
+				sizeof(struct perf_record_auxtrace_info))
+		return -EINVAL;
+
+	ptt = zalloc(sizeof(struct hisi_ptt));
+	if (!ptt)
+		return -ENOMEM;
+
+	err = auxtrace_queues__init(&ptt->queues);
+	if (err)
+		goto err_free;
+
+	ptt->session = session;
+	ptt->machine = &session->machines.host; /* No kvm support */
+	ptt->auxtrace_type = auxtrace_info->type;
+	ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
+
+	ptt->auxtrace.process_event = hisi_ptt_process_event;
+	ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
+	ptt->auxtrace.flush_events = hisi_ptt_flush;
+	ptt->auxtrace.free_events = hisi_ptt_free_events;
+	ptt->auxtrace.free = hisi_ptt_free;
+	ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
+	session->auxtrace = &ptt->auxtrace;
+
+	hisi_ptt_print_info(&auxtrace_info->priv[0]);
+
+	return 0;
+
+err_free:
+	free(ptt);
+	return err;
+}
diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
new file mode 100644
index 000000000000..6e13a8976d82
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#ifndef INCLUDE__PERF_HISI_PTT_H__
+#define INCLUDE__PERF_HISI_PTT_H__
+
+#define HISI_PTT_PMU_NAME "hisi_ptt"
+enum {
+	HISI_PTT_PMU_TYPE,
+	HISI_PTT_PER_CPU_MMAPS,
+	HISI_PTT_AUXTRACE_PRIV_MAX,
+};
+
+#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
+union perf_event;
+struct perf_session;
+struct perf_pmu;
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu);
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session);
+
+#endif
-- 
2.33.0

[PATCH v2 4/6] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver

[Yicong Yang via iommu]

From: Qi Liu <liuqi115@huawei.com>

'perf record' and 'perf report --dump-raw-trace' supported in this
patch.

Example usage:

Output will contain raw PTT data and its textual representation, such
as:

0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
.
. ... HISI PTT data: size 4194304 bytes
.  00000000: 00 00 00 00                                 Prefix
.  00000004: 08 20 00 60                                 Header DW0
.  00000008: ff 02 00 01                                 Header DW1
.  0000000c: 20 08 00 00                                 Header DW2
.  00000010: 10 e7 44 ab                                 Header DW3
.  00000014: 2a a8 1e 01                                 Time
.  00000020: 00 00 00 00                                 Prefix
.  00000024: 01 00 00 60                                 Header DW0
.  00000028: 0f 1e 00 01                                 Header DW1
.  0000002c: 04 00 00 00                                 Header DW2
.  00000030: 40 00 81 02                                 Header DW3
.  00000034: ee 02 00 00                                 Time
....

Signed-off-by: Qi Liu <liuqi115@huawei.com>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 tools/perf/arch/arm/util/auxtrace.c           |  56 ++++-
 tools/perf/arch/arm/util/pmu.c                |   3 +
 tools/perf/arch/arm64/util/Build              |   2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         | 195 +++++++++++++++
 tools/perf/util/Build                         |   2 +
 tools/perf/util/auxtrace.c                    |   4 +
 tools/perf/util/auxtrace.h                    |   1 +
 tools/perf/util/hisi-ptt-decoder/Build        |   1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 +++++++++++++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
 tools/perf/util/hisi_ptt.c                    | 228 ++++++++++++++++++
 tools/perf/util/hisi_ptt.h                    |  28 +++
 12 files changed, 714 insertions(+), 4 deletions(-)
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
index 5fc6a2a3dbc5..dc9d2172464e 100644
--- a/tools/perf/arch/arm/util/auxtrace.c
+++ b/tools/perf/arch/arm/util/auxtrace.c
@@ -4,6 +4,7 @@
  * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
  */
 
+#include <dirent.h>
 #include <stdbool.h>
 #include <linux/coresight-pmu.h>
 #include <linux/zalloc.h>
@@ -14,6 +15,7 @@
 #include "../../../util/pmu.h"
 #include "cs-etm.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 
 static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 {
@@ -50,6 +52,39 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 	return arm_spe_pmus;
 }
 
+static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
+{
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+	int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+	DIR *dir = NULL;
+	struct dirent *dent;
+
+	hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * nr_cpus);
+	if (!hisi_ptt_pmus) {
+		pr_err("hisi_ptt alloc failed\n");
+		*err = -ENOMEM;
+		return NULL;
+	}
+
+	dir = opendir("/sys/devices");
+	dent = readdir(dir);
+	while (dent) {
+		if (strstr(dent->d_name, HISI_PTT_PMU_NAME)) {
+			hisi_ptt_pmus[*nr_ptts] = perf_pmu__find(dent->d_name);
+			if (hisi_ptt_pmus[*nr_ptts]) {
+				pr_debug2("%s %d: arm_spe_pmu %d type %d name %s\n",
+					  __func__, __LINE__, *nr_ptts,
+					  hisi_ptt_pmus[*nr_ptts]->type,
+					  hisi_ptt_pmus[*nr_ptts]->name);
+				(*nr_ptts)++;
+			}
+		}
+		dent = readdir(dir);
+	}
+
+	return hisi_ptt_pmus;
+}
+
 struct auxtrace_record
 *auxtrace_record__init(struct evlist *evlist, int *err)
 {
@@ -57,8 +92,12 @@ struct auxtrace_record
 	struct evsel *evsel;
 	bool found_etm = false;
 	struct perf_pmu *found_spe = NULL;
+	struct perf_pmu *found_ptt = NULL;
 	struct perf_pmu **arm_spe_pmus = NULL;
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+
 	int nr_spes = 0;
+	int nr_ptts = 0;
 	int i = 0;
 
 	if (!evlist)
@@ -66,13 +105,14 @@ struct auxtrace_record
 
 	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
 	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
+	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
 
 	evlist__for_each_entry(evlist, evsel) {
 		if (cs_etm_pmu &&
 		    evsel->core.attr.type == cs_etm_pmu->type)
 			found_etm = true;
 
-		if (!nr_spes || found_spe)
+		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
 			continue;
 
 		for (i = 0; i < nr_spes; i++) {
@@ -81,11 +121,18 @@ struct auxtrace_record
 				break;
 			}
 		}
+
+		for (i = 0; i < nr_ptts; i++) {
+			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
+				found_ptt = hisi_ptt_pmus[i];
+				break;
+			}
+		}
 	}
 	free(arm_spe_pmus);
 
-	if (found_etm && found_spe) {
-		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
+	if (found_etm && found_spe && found_ptt) {
+		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
 		*err = -EOPNOTSUPP;
 		return NULL;
 	}
@@ -96,6 +143,9 @@ struct auxtrace_record
 #if defined(__aarch64__)
 	if (found_spe)
 		return arm_spe_recording_init(err, found_spe);
+
+	if (found_ptt)
+		return hisi_ptt_recording_init(err, found_ptt);
 #endif
 
 	/*
diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
index b8b23b9dc598..89a3cedb4557 100644
--- a/tools/perf/arch/arm/util/pmu.c
+++ b/tools/perf/arch/arm/util/pmu.c
@@ -10,6 +10,7 @@
 #include <linux/string.h>
 
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "../../../util/pmu.h"
 
 struct perf_event_attr
@@ -22,6 +23,8 @@ struct perf_event_attr
 #if defined(__aarch64__)
 	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
 		return arm_spe_pmu_default_config(pmu);
+	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
+		pmu->selectable = true;
 #endif
 	}
 
diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
index 9fcb4e68add9..8b7fd1dc9f37 100644
--- a/tools/perf/arch/arm64/util/Build
+++ b/tools/perf/arch/arm64/util/Build
@@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
 perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
 			      ../../arm/util/auxtrace.o \
 			      ../../arm/util/cs-etm.o \
-			      arm-spe.o mem-events.o
+			      arm-spe.o mem-events.o hisi_ptt.o
diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
new file mode 100644
index 000000000000..337c61b1d637
--- /dev/null
+++ b/tools/perf/arch/arm64/util/hisi_ptt.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/log2.h>
+#include <linux/zalloc.h>
+#include <time.h>
+
+#include <internal/lib.h> // page_size
+#include "../../../util/auxtrace.h"
+#include "../../../util/cpumap.h"
+#include "../../../util/debug.h"
+#include "../../../util/event.h"
+#include "../../../util/evlist.h"
+#include "../../../util/evsel.h"
+#include "../../../util/hisi_ptt.h"
+#include "../../../util/pmu.h"
+#include "../../../util/record.h"
+#include "../../../util/session.h"
+#include "../../../util/tsc.h"
+
+#define DEFAULT_PAGE_SIZE 1024
+#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
+#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
+
+struct hisi_ptt_recording {
+	struct auxtrace_record	itr;
+	struct perf_pmu *hisi_ptt_pmu;
+	struct evlist *evlist;
+};
+
+static size_t
+hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
+			struct evlist *evlist __maybe_unused)
+{
+	return HISI_PTT_AUXTRACE_PRIV_SIZE;
+}
+
+static int hisi_ptt_info_fill(struct auxtrace_record *itr,
+			      struct perf_session *session,
+			      struct perf_record_auxtrace_info *auxtrace_info,
+			      size_t priv_size)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+
+	if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
+		return -EINVAL;
+
+	if (!session->evlist->core.nr_mmaps)
+		return -EINVAL;
+
+	auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
+	auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
+
+	return 0;
+}
+
+static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
+{
+	bool privileged = perf_event_paranoid_check(-1);
+
+	if (!opts->full_auxtrace)
+		return 0;
+
+	if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
+		if (privileged) {
+			opts->auxtrace_mmap_pages = MiB(16) / page_size;
+		} else {
+			opts->auxtrace_mmap_pages = KiB(128) / page_size;
+			if (opts->mmap_pages == UINT_MAX)
+				opts->mmap_pages = KiB(256) / page_size;
+		}
+	}
+
+	/* Validate auxtrace_mmap_pages */
+	if (opts->auxtrace_mmap_pages) {
+		size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
+		size_t min_sz = KiB(8);
+
+		if (sz < min_sz || !is_power_of_2(sz)) {
+			pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
+			       min_sz / DEFAULT_PAGE_SIZE);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_recording_options(struct auxtrace_record *itr,
+				      struct evlist *evlist,
+				      struct record_opts *opts)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+	struct perf_cpu_map *cpus = evlist->core.cpus;
+	struct evsel *evsel, *hisi_ptt_evsel = NULL;
+	struct evsel *tracking_evsel;
+	int err;
+
+	pttr->evlist = evlist;
+	evlist__for_each_entry(evlist, evsel) {
+		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
+			if (hisi_ptt_evsel) {
+				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
+				return -EINVAL;
+			}
+			evsel->core.attr.freq = 0;
+			evsel->core.attr.sample_period = 1;
+			hisi_ptt_evsel = evsel;
+			opts->full_auxtrace = true;
+		}
+	}
+
+	err = hisi_ptt_set_auxtrace_mmap_page(opts);
+	if (err)
+		return err;
+	/*
+	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
+	 * must come first.
+	 */
+	evlist__to_front(evlist, hisi_ptt_evsel);
+
+	if (!perf_cpu_map__empty(cpus)) {
+		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
+		evsel__set_sample_bit(hisi_ptt_evsel, CPU);
+	}
+
+	/* Add dummy event to keep tracking */
+	err = parse_events(evlist, "dummy:u", NULL);
+	if (err)
+		return err;
+
+	tracking_evsel = evlist__last(evlist);
+	evlist__set_tracking_event(evlist, tracking_evsel);
+
+	tracking_evsel->core.attr.freq = 0;
+	tracking_evsel->core.attr.sample_period = 1;
+
+	if (!perf_cpu_map__empty(cpus))
+		evsel__set_sample_bit(tracking_evsel, TIME);
+
+	return 0;
+}
+
+static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
+{
+	return rdtsc();
+}
+
+static void hisi_ptt_recording_free(struct auxtrace_record *itr)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+
+	free(pttr);
+}
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu)
+{
+	struct hisi_ptt_recording *pttr;
+
+	if (!hisi_ptt_pmu) {
+		*err = -ENODEV;
+		return NULL;
+	}
+
+	pttr = zalloc(sizeof(struct hisi_ptt_recording));
+	if (!pttr) {
+		*err = -ENOMEM;
+		return NULL;
+	}
+
+	pttr->hisi_ptt_pmu = hisi_ptt_pmu;
+	pttr->itr.pmu = hisi_ptt_pmu;
+	pttr->itr.recording_options = hisi_ptt_recording_options;
+	pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
+	pttr->itr.info_fill = hisi_ptt_info_fill;
+	pttr->itr.free = hisi_ptt_recording_free;
+	pttr->itr.reference = hisi_ptt_reference;
+	pttr->itr.read_finish = auxtrace_record__read_finish;
+	pttr->itr.alignment = 0;
+
+	*err = 0;
+	return &pttr->itr;
+}
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 2e5bfbb69960..4afeabd87483 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -114,6 +114,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
 perf-$(CONFIG_AUXTRACE) += intel-bts.o
 perf-$(CONFIG_AUXTRACE) += arm-spe.o
 perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
+perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
 perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
 
 ifdef CONFIG_LIBOPENCSD
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index c679394b898d..0826d7e05d63 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -51,6 +51,7 @@
 #include "intel-pt.h"
 #include "intel-bts.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "s390-cpumsf.h"
 #include "util/mmap.h"
 
@@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
 	case PERF_AUXTRACE_S390_CPUMSF:
 		err = s390_cpumsf_process_auxtrace_info(event, session);
 		break;
+	case PERF_AUXTRACE_HISI_PTT:
+		err = hisi_ptt_process_auxtrace_info(event, session);
+		break;
 	case PERF_AUXTRACE_UNKNOWN:
 	default:
 		return -EINVAL;
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index bbf0d78c6401..51d37b7dce9d 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -47,6 +47,7 @@ enum auxtrace_type {
 	PERF_AUXTRACE_CS_ETM,
 	PERF_AUXTRACE_ARM_SPE,
 	PERF_AUXTRACE_S390_CPUMSF,
+	PERF_AUXTRACE_HISI_PTT,
 };
 
 enum itrace_period_type {
diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
new file mode 100644
index 000000000000..db3db8b75033
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/Build
@@ -0,0 +1 @@
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
new file mode 100644
index 000000000000..ad439bf409bb
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <endian.h>
+#include <byteswap.h>
+#include <linux/bitops.h>
+#include <stdarg.h>
+
+#include "../color.h"
+#include "hisi-ptt-pkt-decoder.h"
+
+/*
+ * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
+ * used to distinguish the data format.
+ * 8DW format is like:
+ *   bits [                 31:11                 ][       10:0       ]
+ *        |---------------------------------------|-------------------|
+ *    DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+ *    DW1 [                       Prefix                              ]
+ *    DW2 [                     Header DW0                            ]
+ *    DW3 [                     Header DW1                            ]
+ *    DW4 [                     Header DW2                            ]
+ *    DW5 [                     Header DW3                            ]
+ *    DW6 [                   Reserved (0x0)                          ]
+ *    DW7 [                        Time                               ]
+ *
+ * 4DW format is like:
+ *   bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+ *        |-----|---------|---|---|---|---|-------------|-------------|
+ *    DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+ *    DW1 [                     Header DW1                            ]
+ *    DW2 [                     Header DW2                            ]
+ *    DW3 [                     Header DW3                            ]
+ */
+
+enum hisi_ptt_8dw_pkt_field_type {
+	HISI_PTT_8DW_PREFIX,
+	HISI_PTT_8DW_HEAD0,
+	HISI_PTT_8DW_HEAD1,
+	HISI_PTT_8DW_HEAD2,
+	HISI_PTT_8DW_HEAD3,
+	HISI_PTT_8DW_TIME,
+	HISI_PTT_8DW_TYPE_MAX
+};
+
+enum hisi_ptt_4dw_pkt_field_type {
+	HISI_PTT_4DW_HEAD1,
+	HISI_PTT_4DW_HEAD2,
+	HISI_PTT_4DW_HEAD3,
+	HISI_PTT_4DW_TYPE_MAX
+};
+
+static const char * const hisi_ptt_8dw_pkt_field_name[] = {
+	[HISI_PTT_8DW_PREFIX]	= "Prefix",
+	[HISI_PTT_8DW_HEAD0]	= "Header DW0",
+	[HISI_PTT_8DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_8DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_8DW_HEAD3]	= "Header DW3",
+	[HISI_PTT_8DW_TIME]	= "Time",
+};
+
+static const char * const hisi_ptt_4dw_pkt_field_name[] = {
+	[HISI_PTT_4DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_4DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_4DW_HEAD3]	= "Header DW3",
+};
+
+/* offset of each member is determined by format of 8dw packet. */
+static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
+	[HISI_PTT_8DW_PREFIX]	= 4,
+	[HISI_PTT_8DW_HEAD0]	= 4,
+	[HISI_PTT_8DW_HEAD1]	= 4,
+	[HISI_PTT_8DW_HEAD2]	= 4,
+	[HISI_PTT_8DW_HEAD3]	= 4,
+	[HISI_PTT_8DW_TIME]	= 8,
+};
+
+union hisi_ptt_4dw {
+	struct {
+		uint32_t format : 2;
+		uint32_t type : 5;
+		uint32_t t9 : 1;
+		uint32_t t8 : 1;
+		uint32_t th : 1;
+		uint32_t so : 1;
+		uint32_t len : 10;
+		uint32_t time : 11;
+	};
+	uint32_t value;
+};
+
+static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	int i;
+
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+	color_fprintf(stdout, color, "  %s\n", desc);
+	*pos += field_len;
+}
+
+static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
+		buf += hisi_ptt_8dw_pkt_field_offset[i];
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_8DW_PKT_SIZE;
+}
+
+static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	union hisi_ptt_4dw dw0;
+	int i;
+
+	dw0.value = *(uint32_t *)buf;
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+
+	color_fprintf(stdout, color,
+		      "  %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
+		      "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
+		      "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
+		      dw0.len, "Time", dw0.time);
+
+	*pos += field_len;
+}
+
+static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	hisi_ptt_4dw_print_dw0(buf, &pos);
+
+	for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
+		buf += sizeof(uint32_t);
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_4DW_PKT_SIZE;
+}
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
+{
+	if (type == HISI_PTT_8DW_PKT)
+		return hisi_ptt_8dw_kpt_desc(buf, pos);
+
+	return hisi_ptt_4dw_kpt_desc(buf, pos);
+}
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
new file mode 100644
index 000000000000..f9eda1f9d0d3
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
+#define INCLUDE__HISI_PTT_PKT_DECODER_H__
+
+#include <stddef.h>
+#include <stdint.h>
+
+#define HISI_PTT_PKT_DESC_MAX		256
+#define HISI_PTT_NEED_MORE_BYTES	-1
+#define HISI_PTT_8DW_CHECK_MASK		GENMASK(31, 11)
+#define HISI_PTT_IS_8DW_PKT		GENMASK(31, 11)
+#define HISI_PTT_8DW_PKT_SIZE		32
+#define HISI_PTT_4DW_PKT_SIZE		16
+#define HISI_PTT_MAX_SPACE_LEN		10
+
+enum hisi_ptt_pkt_type {
+	HISI_PTT_4DW_PKT,
+	HISI_PTT_8DW_PKT,
+};
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
+
+#endif
diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
new file mode 100644
index 000000000000..541e67f7259e
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.c
@@ -0,0 +1,228 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <byteswap.h>
+#include <endian.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/types.h>
+#include <linux/zalloc.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "auxtrace.h"
+#include "color.h"
+#include "debug.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "hisi_ptt.h"
+#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
+#include "machine.h"
+#include "session.h"
+#include "symbol.h"
+#include "tool.h"
+#include "util/synthetic-events.h"
+
+struct hisi_ptt {
+	struct auxtrace auxtrace;
+	struct auxtrace_queues queues;
+	u32 auxtrace_type;
+	struct perf_session *session;
+	struct machine *machine;
+	u32 pmu_type;
+};
+
+struct hisi_ptt_queue {
+	struct hisi_ptt *ptt;
+	struct auxtrace_buffer *buffer;
+};
+
+static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
+{
+	uint32_t head = *(uint32_t *)buf;
+
+	if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
+		return HISI_PTT_8DW_PKT;
+
+	return HISI_PTT_4DW_PKT;
+}
+
+static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
+			  unsigned char *buf, size_t len)
+{
+	const char *color = PERF_COLOR_BLUE;
+	enum hisi_ptt_pkt_type type;
+	size_t pos = 0;
+	int pkt_len;
+
+	color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
+		      len);
+
+	type = hisi_ptt_check_packet_type(buf);
+	while (len) {
+		pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
+		if (!pkt_len)
+			color_fprintf(stdout, color, " Bad packet!\n");
+
+		pos += pkt_len;
+		buf += pkt_len;
+		len -= pkt_len;
+	}
+}
+
+static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
+				size_t len)
+{
+	printf(".\n");
+
+	hisi_ptt_dump(ptt, buf, len);
+}
+
+static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
+				  union perf_event *event __maybe_unused,
+				  struct perf_sample *sample __maybe_unused,
+				  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
+					   union perf_event *event,
+					   struct perf_tool *tool __maybe_unused)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_buffer *buffer;
+	off_t data_offset;
+	int fd = perf_data__fd(session->data);
+	int err;
+
+	if (perf_data__is_pipe(session->data)) {
+		data_offset = 0;
+	} else {
+		data_offset = lseek(fd, 0, SEEK_CUR);
+		if (data_offset == -1)
+			return -errno;
+	}
+
+	err = auxtrace_queues__add_event(&ptt->queues, session, event,
+					 data_offset, &buffer);
+	if (err)
+		return err;
+
+	if (dump_trace) {
+		if (auxtrace_buffer__get_data(buffer, fd)) {
+			hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
+			auxtrace_buffer__put_data(buffer);
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
+			  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static void hisi_ptt_free_queue(void *priv)
+{
+	struct hisi_ptt_queue *pttq = priv;
+
+	if (!pttq)
+		return;
+
+	free(pttq);
+}
+
+static void hisi_ptt_free_events(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_queues *queues = &ptt->queues;
+	unsigned int i;
+
+	for (i = 0; i < queues->nr_queues; i++) {
+		hisi_ptt_free_queue(queues->queue_array[i].priv);
+		queues->queue_array[i].priv = NULL;
+	}
+	auxtrace_queues__free(queues);
+}
+
+static void hisi_ptt_free(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+
+	hisi_ptt_free_events(session);
+	session->auxtrace = NULL;
+	free(ptt);
+}
+
+static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
+				       struct evsel *evsel)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
+
+	return evsel->core.attr.type == ptt->pmu_type;
+}
+
+static const char * const hisi_ptt_info_fmts[] = {
+	[HISI_PTT_PMU_TYPE]		= "  PMU Type           %" PRId64 "\n",
+};
+
+static void hisi_ptt_print_info(__u64 *arr)
+{
+	if (!dump_trace)
+		return;
+
+	fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
+}
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session)
+{
+	struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
+	struct hisi_ptt *ptt;
+	int err;
+
+	if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
+				sizeof(struct perf_record_auxtrace_info))
+		return -EINVAL;
+
+	ptt = zalloc(sizeof(struct hisi_ptt));
+	if (!ptt)
+		return -ENOMEM;
+
+	err = auxtrace_queues__init(&ptt->queues);
+	if (err)
+		goto err_free;
+
+	ptt->session = session;
+	ptt->machine = &session->machines.host; /* No kvm support */
+	ptt->auxtrace_type = auxtrace_info->type;
+	ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
+
+	ptt->auxtrace.process_event = hisi_ptt_process_event;
+	ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
+	ptt->auxtrace.flush_events = hisi_ptt_flush;
+	ptt->auxtrace.free_events = hisi_ptt_free_events;
+	ptt->auxtrace.free = hisi_ptt_free;
+	ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
+	session->auxtrace = &ptt->auxtrace;
+
+	hisi_ptt_print_info(&auxtrace_info->priv[0]);
+
+	return 0;
+
+err_free:
+	free(ptt);
+	return err;
+}
diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
new file mode 100644
index 000000000000..6e13a8976d82
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#ifndef INCLUDE__PERF_HISI_PTT_H__
+#define INCLUDE__PERF_HISI_PTT_H__
+
+#define HISI_PTT_PMU_NAME "hisi_ptt"
+enum {
+	HISI_PTT_PMU_TYPE,
+	HISI_PTT_PER_CPU_MMAPS,
+	HISI_PTT_AUXTRACE_PRIV_MAX,
+};
+
+#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
+union perf_event;
+struct perf_session;
+struct perf_pmu;
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu);
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session);
+
+#endif
-- 
2.33.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 4/6] perf tool: Add support for HiSilicon PCIe Tune and Trace device driver

[Yicong Yang]

From: Qi Liu <liuqi115@huawei.com>

'perf record' and 'perf report --dump-raw-trace' supported in this
patch.

Example usage:

Output will contain raw PTT data and its textual representation, such
as:

0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000  offset: 0
ref: 0xa5d50c725  idx: 0  tid: -1  cpu: 0
.
. ... HISI PTT data: size 4194304 bytes
.  00000000: 00 00 00 00                                 Prefix
.  00000004: 08 20 00 60                                 Header DW0
.  00000008: ff 02 00 01                                 Header DW1
.  0000000c: 20 08 00 00                                 Header DW2
.  00000010: 10 e7 44 ab                                 Header DW3
.  00000014: 2a a8 1e 01                                 Time
.  00000020: 00 00 00 00                                 Prefix
.  00000024: 01 00 00 60                                 Header DW0
.  00000028: 0f 1e 00 01                                 Header DW1
.  0000002c: 04 00 00 00                                 Header DW2
.  00000030: 40 00 81 02                                 Header DW3
.  00000034: ee 02 00 00                                 Time
....

Signed-off-by: Qi Liu <liuqi115@huawei.com>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 tools/perf/arch/arm/util/auxtrace.c           |  56 ++++-
 tools/perf/arch/arm/util/pmu.c                |   3 +
 tools/perf/arch/arm64/util/Build              |   2 +-
 tools/perf/arch/arm64/util/hisi_ptt.c         | 195 +++++++++++++++
 tools/perf/util/Build                         |   2 +
 tools/perf/util/auxtrace.c                    |   4 +
 tools/perf/util/auxtrace.h                    |   1 +
 tools/perf/util/hisi-ptt-decoder/Build        |   1 +
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c   | 170 +++++++++++++
 .../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h   |  28 +++
 tools/perf/util/hisi_ptt.c                    | 228 ++++++++++++++++++
 tools/perf/util/hisi_ptt.h                    |  28 +++
 12 files changed, 714 insertions(+), 4 deletions(-)
 create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
 create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
 create mode 100644 tools/perf/util/hisi_ptt.c
 create mode 100644 tools/perf/util/hisi_ptt.h

diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
index 5fc6a2a3dbc5..dc9d2172464e 100644
--- a/tools/perf/arch/arm/util/auxtrace.c
+++ b/tools/perf/arch/arm/util/auxtrace.c
@@ -4,6 +4,7 @@
  * Author: Mathieu Poirier <mathieu.poirier@linaro.org>
  */
 
+#include <dirent.h>
 #include <stdbool.h>
 #include <linux/coresight-pmu.h>
 #include <linux/zalloc.h>
@@ -14,6 +15,7 @@
 #include "../../../util/pmu.h"
 #include "cs-etm.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 
 static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 {
@@ -50,6 +52,39 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
 	return arm_spe_pmus;
 }
 
+static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
+{
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+	int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+	DIR *dir = NULL;
+	struct dirent *dent;
+
+	hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * nr_cpus);
+	if (!hisi_ptt_pmus) {
+		pr_err("hisi_ptt alloc failed\n");
+		*err = -ENOMEM;
+		return NULL;
+	}
+
+	dir = opendir("/sys/devices");
+	dent = readdir(dir);
+	while (dent) {
+		if (strstr(dent->d_name, HISI_PTT_PMU_NAME)) {
+			hisi_ptt_pmus[*nr_ptts] = perf_pmu__find(dent->d_name);
+			if (hisi_ptt_pmus[*nr_ptts]) {
+				pr_debug2("%s %d: arm_spe_pmu %d type %d name %s\n",
+					  __func__, __LINE__, *nr_ptts,
+					  hisi_ptt_pmus[*nr_ptts]->type,
+					  hisi_ptt_pmus[*nr_ptts]->name);
+				(*nr_ptts)++;
+			}
+		}
+		dent = readdir(dir);
+	}
+
+	return hisi_ptt_pmus;
+}
+
 struct auxtrace_record
 *auxtrace_record__init(struct evlist *evlist, int *err)
 {
@@ -57,8 +92,12 @@ struct auxtrace_record
 	struct evsel *evsel;
 	bool found_etm = false;
 	struct perf_pmu *found_spe = NULL;
+	struct perf_pmu *found_ptt = NULL;
 	struct perf_pmu **arm_spe_pmus = NULL;
+	struct perf_pmu **hisi_ptt_pmus = NULL;
+
 	int nr_spes = 0;
+	int nr_ptts = 0;
 	int i = 0;
 
 	if (!evlist)
@@ -66,13 +105,14 @@ struct auxtrace_record
 
 	cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
 	arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
+	hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
 
 	evlist__for_each_entry(evlist, evsel) {
 		if (cs_etm_pmu &&
 		    evsel->core.attr.type == cs_etm_pmu->type)
 			found_etm = true;
 
-		if (!nr_spes || found_spe)
+		if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
 			continue;
 
 		for (i = 0; i < nr_spes; i++) {
@@ -81,11 +121,18 @@ struct auxtrace_record
 				break;
 			}
 		}
+
+		for (i = 0; i < nr_ptts; i++) {
+			if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
+				found_ptt = hisi_ptt_pmus[i];
+				break;
+			}
+		}
 	}
 	free(arm_spe_pmus);
 
-	if (found_etm && found_spe) {
-		pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
+	if (found_etm && found_spe && found_ptt) {
+		pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
 		*err = -EOPNOTSUPP;
 		return NULL;
 	}
@@ -96,6 +143,9 @@ struct auxtrace_record
 #if defined(__aarch64__)
 	if (found_spe)
 		return arm_spe_recording_init(err, found_spe);
+
+	if (found_ptt)
+		return hisi_ptt_recording_init(err, found_ptt);
 #endif
 
 	/*
diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
index b8b23b9dc598..89a3cedb4557 100644
--- a/tools/perf/arch/arm/util/pmu.c
+++ b/tools/perf/arch/arm/util/pmu.c
@@ -10,6 +10,7 @@
 #include <linux/string.h>
 
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "../../../util/pmu.h"
 
 struct perf_event_attr
@@ -22,6 +23,8 @@ struct perf_event_attr
 #if defined(__aarch64__)
 	} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
 		return arm_spe_pmu_default_config(pmu);
+	} else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
+		pmu->selectable = true;
 #endif
 	}
 
diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
index 9fcb4e68add9..8b7fd1dc9f37 100644
--- a/tools/perf/arch/arm64/util/Build
+++ b/tools/perf/arch/arm64/util/Build
@@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
 perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
 			      ../../arm/util/auxtrace.o \
 			      ../../arm/util/cs-etm.o \
-			      arm-spe.o mem-events.o
+			      arm-spe.o mem-events.o hisi_ptt.o
diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
new file mode 100644
index 000000000000..337c61b1d637
--- /dev/null
+++ b/tools/perf/arch/arm64/util/hisi_ptt.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/log2.h>
+#include <linux/zalloc.h>
+#include <time.h>
+
+#include <internal/lib.h> // page_size
+#include "../../../util/auxtrace.h"
+#include "../../../util/cpumap.h"
+#include "../../../util/debug.h"
+#include "../../../util/event.h"
+#include "../../../util/evlist.h"
+#include "../../../util/evsel.h"
+#include "../../../util/hisi_ptt.h"
+#include "../../../util/pmu.h"
+#include "../../../util/record.h"
+#include "../../../util/session.h"
+#include "../../../util/tsc.h"
+
+#define DEFAULT_PAGE_SIZE 1024
+#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
+#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
+
+struct hisi_ptt_recording {
+	struct auxtrace_record	itr;
+	struct perf_pmu *hisi_ptt_pmu;
+	struct evlist *evlist;
+};
+
+static size_t
+hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
+			struct evlist *evlist __maybe_unused)
+{
+	return HISI_PTT_AUXTRACE_PRIV_SIZE;
+}
+
+static int hisi_ptt_info_fill(struct auxtrace_record *itr,
+			      struct perf_session *session,
+			      struct perf_record_auxtrace_info *auxtrace_info,
+			      size_t priv_size)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+
+	if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
+		return -EINVAL;
+
+	if (!session->evlist->core.nr_mmaps)
+		return -EINVAL;
+
+	auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
+	auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
+
+	return 0;
+}
+
+static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
+{
+	bool privileged = perf_event_paranoid_check(-1);
+
+	if (!opts->full_auxtrace)
+		return 0;
+
+	if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
+		if (privileged) {
+			opts->auxtrace_mmap_pages = MiB(16) / page_size;
+		} else {
+			opts->auxtrace_mmap_pages = KiB(128) / page_size;
+			if (opts->mmap_pages == UINT_MAX)
+				opts->mmap_pages = KiB(256) / page_size;
+		}
+	}
+
+	/* Validate auxtrace_mmap_pages */
+	if (opts->auxtrace_mmap_pages) {
+		size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
+		size_t min_sz = KiB(8);
+
+		if (sz < min_sz || !is_power_of_2(sz)) {
+			pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
+			       min_sz / DEFAULT_PAGE_SIZE);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_recording_options(struct auxtrace_record *itr,
+				      struct evlist *evlist,
+				      struct record_opts *opts)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+	struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+	struct perf_cpu_map *cpus = evlist->core.cpus;
+	struct evsel *evsel, *hisi_ptt_evsel = NULL;
+	struct evsel *tracking_evsel;
+	int err;
+
+	pttr->evlist = evlist;
+	evlist__for_each_entry(evlist, evsel) {
+		if (evsel->core.attr.type == hisi_ptt_pmu->type) {
+			if (hisi_ptt_evsel) {
+				pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
+				return -EINVAL;
+			}
+			evsel->core.attr.freq = 0;
+			evsel->core.attr.sample_period = 1;
+			hisi_ptt_evsel = evsel;
+			opts->full_auxtrace = true;
+		}
+	}
+
+	err = hisi_ptt_set_auxtrace_mmap_page(opts);
+	if (err)
+		return err;
+	/*
+	 * To obtain the auxtrace buffer file descriptor, the auxtrace event
+	 * must come first.
+	 */
+	evlist__to_front(evlist, hisi_ptt_evsel);
+
+	if (!perf_cpu_map__empty(cpus)) {
+		evsel__set_sample_bit(hisi_ptt_evsel, TIME);
+		evsel__set_sample_bit(hisi_ptt_evsel, CPU);
+	}
+
+	/* Add dummy event to keep tracking */
+	err = parse_events(evlist, "dummy:u", NULL);
+	if (err)
+		return err;
+
+	tracking_evsel = evlist__last(evlist);
+	evlist__set_tracking_event(evlist, tracking_evsel);
+
+	tracking_evsel->core.attr.freq = 0;
+	tracking_evsel->core.attr.sample_period = 1;
+
+	if (!perf_cpu_map__empty(cpus))
+		evsel__set_sample_bit(tracking_evsel, TIME);
+
+	return 0;
+}
+
+static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
+{
+	return rdtsc();
+}
+
+static void hisi_ptt_recording_free(struct auxtrace_record *itr)
+{
+	struct hisi_ptt_recording *pttr =
+			container_of(itr, struct hisi_ptt_recording, itr);
+
+	free(pttr);
+}
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu)
+{
+	struct hisi_ptt_recording *pttr;
+
+	if (!hisi_ptt_pmu) {
+		*err = -ENODEV;
+		return NULL;
+	}
+
+	pttr = zalloc(sizeof(struct hisi_ptt_recording));
+	if (!pttr) {
+		*err = -ENOMEM;
+		return NULL;
+	}
+
+	pttr->hisi_ptt_pmu = hisi_ptt_pmu;
+	pttr->itr.pmu = hisi_ptt_pmu;
+	pttr->itr.recording_options = hisi_ptt_recording_options;
+	pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
+	pttr->itr.info_fill = hisi_ptt_info_fill;
+	pttr->itr.free = hisi_ptt_recording_free;
+	pttr->itr.reference = hisi_ptt_reference;
+	pttr->itr.read_finish = auxtrace_record__read_finish;
+	pttr->itr.alignment = 0;
+
+	*err = 0;
+	return &pttr->itr;
+}
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 2e5bfbb69960..4afeabd87483 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -114,6 +114,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
 perf-$(CONFIG_AUXTRACE) += intel-bts.o
 perf-$(CONFIG_AUXTRACE) += arm-spe.o
 perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
+perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
 perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
 
 ifdef CONFIG_LIBOPENCSD
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index c679394b898d..0826d7e05d63 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -51,6 +51,7 @@
 #include "intel-pt.h"
 #include "intel-bts.h"
 #include "arm-spe.h"
+#include "hisi_ptt.h"
 #include "s390-cpumsf.h"
 #include "util/mmap.h"
 
@@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
 	case PERF_AUXTRACE_S390_CPUMSF:
 		err = s390_cpumsf_process_auxtrace_info(event, session);
 		break;
+	case PERF_AUXTRACE_HISI_PTT:
+		err = hisi_ptt_process_auxtrace_info(event, session);
+		break;
 	case PERF_AUXTRACE_UNKNOWN:
 	default:
 		return -EINVAL;
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index bbf0d78c6401..51d37b7dce9d 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -47,6 +47,7 @@ enum auxtrace_type {
 	PERF_AUXTRACE_CS_ETM,
 	PERF_AUXTRACE_ARM_SPE,
 	PERF_AUXTRACE_S390_CPUMSF,
+	PERF_AUXTRACE_HISI_PTT,
 };
 
 enum itrace_period_type {
diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
new file mode 100644
index 000000000000..db3db8b75033
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/Build
@@ -0,0 +1 @@
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
new file mode 100644
index 000000000000..ad439bf409bb
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <endian.h>
+#include <byteswap.h>
+#include <linux/bitops.h>
+#include <stdarg.h>
+
+#include "../color.h"
+#include "hisi-ptt-pkt-decoder.h"
+
+/*
+ * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
+ * used to distinguish the data format.
+ * 8DW format is like:
+ *   bits [                 31:11                 ][       10:0       ]
+ *        |---------------------------------------|-------------------|
+ *    DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+ *    DW1 [                       Prefix                              ]
+ *    DW2 [                     Header DW0                            ]
+ *    DW3 [                     Header DW1                            ]
+ *    DW4 [                     Header DW2                            ]
+ *    DW5 [                     Header DW3                            ]
+ *    DW6 [                   Reserved (0x0)                          ]
+ *    DW7 [                        Time                               ]
+ *
+ * 4DW format is like:
+ *   bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+ *        |-----|---------|---|---|---|---|-------------|-------------|
+ *    DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+ *    DW1 [                     Header DW1                            ]
+ *    DW2 [                     Header DW2                            ]
+ *    DW3 [                     Header DW3                            ]
+ */
+
+enum hisi_ptt_8dw_pkt_field_type {
+	HISI_PTT_8DW_PREFIX,
+	HISI_PTT_8DW_HEAD0,
+	HISI_PTT_8DW_HEAD1,
+	HISI_PTT_8DW_HEAD2,
+	HISI_PTT_8DW_HEAD3,
+	HISI_PTT_8DW_TIME,
+	HISI_PTT_8DW_TYPE_MAX
+};
+
+enum hisi_ptt_4dw_pkt_field_type {
+	HISI_PTT_4DW_HEAD1,
+	HISI_PTT_4DW_HEAD2,
+	HISI_PTT_4DW_HEAD3,
+	HISI_PTT_4DW_TYPE_MAX
+};
+
+static const char * const hisi_ptt_8dw_pkt_field_name[] = {
+	[HISI_PTT_8DW_PREFIX]	= "Prefix",
+	[HISI_PTT_8DW_HEAD0]	= "Header DW0",
+	[HISI_PTT_8DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_8DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_8DW_HEAD3]	= "Header DW3",
+	[HISI_PTT_8DW_TIME]	= "Time",
+};
+
+static const char * const hisi_ptt_4dw_pkt_field_name[] = {
+	[HISI_PTT_4DW_HEAD1]	= "Header DW1",
+	[HISI_PTT_4DW_HEAD2]	= "Header DW2",
+	[HISI_PTT_4DW_HEAD3]	= "Header DW3",
+};
+
+/* offset of each member is determined by format of 8dw packet. */
+static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
+	[HISI_PTT_8DW_PREFIX]	= 4,
+	[HISI_PTT_8DW_HEAD0]	= 4,
+	[HISI_PTT_8DW_HEAD1]	= 4,
+	[HISI_PTT_8DW_HEAD2]	= 4,
+	[HISI_PTT_8DW_HEAD3]	= 4,
+	[HISI_PTT_8DW_TIME]	= 8,
+};
+
+union hisi_ptt_4dw {
+	struct {
+		uint32_t format : 2;
+		uint32_t type : 5;
+		uint32_t t9 : 1;
+		uint32_t t8 : 1;
+		uint32_t th : 1;
+		uint32_t so : 1;
+		uint32_t len : 10;
+		uint32_t time : 11;
+	};
+	uint32_t value;
+};
+
+static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	int i;
+
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+	color_fprintf(stdout, color, "  %s\n", desc);
+	*pos += field_len;
+}
+
+static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
+		buf += hisi_ptt_8dw_pkt_field_offset[i];
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_8DW_PKT_SIZE;
+}
+
+static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
+{
+	const char *color = PERF_COLOR_BLUE;
+	int field_len = sizeof(uint32_t);
+	union hisi_ptt_4dw dw0;
+	int i;
+
+	dw0.value = *(uint32_t *)buf;
+	printf(".");
+	color_fprintf(stdout, color, "  %08x: ", *pos);
+	for (i = 0; i < field_len; i++)
+		color_fprintf(stdout, color, "%02x ", buf[i]);
+	for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+		color_fprintf(stdout, color, "   ");
+
+	color_fprintf(stdout, color,
+		      "  %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
+		      "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
+		      "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
+		      dw0.len, "Time", dw0.time);
+
+	*pos += field_len;
+}
+
+static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
+{
+	int i;
+
+	hisi_ptt_4dw_print_dw0(buf, &pos);
+
+	for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
+		buf += sizeof(uint32_t);
+		hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
+	}
+
+	return HISI_PTT_4DW_PKT_SIZE;
+}
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
+{
+	if (type == HISI_PTT_8DW_PKT)
+		return hisi_ptt_8dw_kpt_desc(buf, pos);
+
+	return hisi_ptt_4dw_kpt_desc(buf, pos);
+}
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
new file mode 100644
index 000000000000..f9eda1f9d0d3
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
+#define INCLUDE__HISI_PTT_PKT_DECODER_H__
+
+#include <stddef.h>
+#include <stdint.h>
+
+#define HISI_PTT_PKT_DESC_MAX		256
+#define HISI_PTT_NEED_MORE_BYTES	-1
+#define HISI_PTT_8DW_CHECK_MASK		GENMASK(31, 11)
+#define HISI_PTT_IS_8DW_PKT		GENMASK(31, 11)
+#define HISI_PTT_8DW_PKT_SIZE		32
+#define HISI_PTT_4DW_PKT_SIZE		16
+#define HISI_PTT_MAX_SPACE_LEN		10
+
+enum hisi_ptt_pkt_type {
+	HISI_PTT_4DW_PKT,
+	HISI_PTT_8DW_PKT,
+};
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
+
+#endif
diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
new file mode 100644
index 000000000000..541e67f7259e
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.c
@@ -0,0 +1,228 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#include <byteswap.h>
+#include <endian.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/types.h>
+#include <linux/zalloc.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "auxtrace.h"
+#include "color.h"
+#include "debug.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "hisi_ptt.h"
+#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
+#include "machine.h"
+#include "session.h"
+#include "symbol.h"
+#include "tool.h"
+#include "util/synthetic-events.h"
+
+struct hisi_ptt {
+	struct auxtrace auxtrace;
+	struct auxtrace_queues queues;
+	u32 auxtrace_type;
+	struct perf_session *session;
+	struct machine *machine;
+	u32 pmu_type;
+};
+
+struct hisi_ptt_queue {
+	struct hisi_ptt *ptt;
+	struct auxtrace_buffer *buffer;
+};
+
+static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
+{
+	uint32_t head = *(uint32_t *)buf;
+
+	if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
+		return HISI_PTT_8DW_PKT;
+
+	return HISI_PTT_4DW_PKT;
+}
+
+static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
+			  unsigned char *buf, size_t len)
+{
+	const char *color = PERF_COLOR_BLUE;
+	enum hisi_ptt_pkt_type type;
+	size_t pos = 0;
+	int pkt_len;
+
+	color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
+		      len);
+
+	type = hisi_ptt_check_packet_type(buf);
+	while (len) {
+		pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
+		if (!pkt_len)
+			color_fprintf(stdout, color, " Bad packet!\n");
+
+		pos += pkt_len;
+		buf += pkt_len;
+		len -= pkt_len;
+	}
+}
+
+static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
+				size_t len)
+{
+	printf(".\n");
+
+	hisi_ptt_dump(ptt, buf, len);
+}
+
+static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
+				  union perf_event *event __maybe_unused,
+				  struct perf_sample *sample __maybe_unused,
+				  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
+					   union perf_event *event,
+					   struct perf_tool *tool __maybe_unused)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_buffer *buffer;
+	off_t data_offset;
+	int fd = perf_data__fd(session->data);
+	int err;
+
+	if (perf_data__is_pipe(session->data)) {
+		data_offset = 0;
+	} else {
+		data_offset = lseek(fd, 0, SEEK_CUR);
+		if (data_offset == -1)
+			return -errno;
+	}
+
+	err = auxtrace_queues__add_event(&ptt->queues, session, event,
+					 data_offset, &buffer);
+	if (err)
+		return err;
+
+	if (dump_trace) {
+		if (auxtrace_buffer__get_data(buffer, fd)) {
+			hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
+			auxtrace_buffer__put_data(buffer);
+		}
+	}
+
+	return 0;
+}
+
+static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
+			  struct perf_tool *tool __maybe_unused)
+{
+	return 0;
+}
+
+static void hisi_ptt_free_queue(void *priv)
+{
+	struct hisi_ptt_queue *pttq = priv;
+
+	if (!pttq)
+		return;
+
+	free(pttq);
+}
+
+static void hisi_ptt_free_events(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+	struct auxtrace_queues *queues = &ptt->queues;
+	unsigned int i;
+
+	for (i = 0; i < queues->nr_queues; i++) {
+		hisi_ptt_free_queue(queues->queue_array[i].priv);
+		queues->queue_array[i].priv = NULL;
+	}
+	auxtrace_queues__free(queues);
+}
+
+static void hisi_ptt_free(struct perf_session *session)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+					    auxtrace);
+
+	hisi_ptt_free_events(session);
+	session->auxtrace = NULL;
+	free(ptt);
+}
+
+static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
+				       struct evsel *evsel)
+{
+	struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
+
+	return evsel->core.attr.type == ptt->pmu_type;
+}
+
+static const char * const hisi_ptt_info_fmts[] = {
+	[HISI_PTT_PMU_TYPE]		= "  PMU Type           %" PRId64 "\n",
+};
+
+static void hisi_ptt_print_info(__u64 *arr)
+{
+	if (!dump_trace)
+		return;
+
+	fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
+}
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session)
+{
+	struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
+	struct hisi_ptt *ptt;
+	int err;
+
+	if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
+				sizeof(struct perf_record_auxtrace_info))
+		return -EINVAL;
+
+	ptt = zalloc(sizeof(struct hisi_ptt));
+	if (!ptt)
+		return -ENOMEM;
+
+	err = auxtrace_queues__init(&ptt->queues);
+	if (err)
+		goto err_free;
+
+	ptt->session = session;
+	ptt->machine = &session->machines.host; /* No kvm support */
+	ptt->auxtrace_type = auxtrace_info->type;
+	ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
+
+	ptt->auxtrace.process_event = hisi_ptt_process_event;
+	ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
+	ptt->auxtrace.flush_events = hisi_ptt_flush;
+	ptt->auxtrace.free_events = hisi_ptt_free_events;
+	ptt->auxtrace.free = hisi_ptt_free;
+	ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
+	session->auxtrace = &ptt->auxtrace;
+
+	hisi_ptt_print_info(&auxtrace_info->priv[0]);
+
+	return 0;
+
+err_free:
+	free(ptt);
+	return err;
+}
diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
new file mode 100644
index 000000000000..6e13a8976d82
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2021 HiSilicon Limited
+ */
+
+#ifndef INCLUDE__PERF_HISI_PTT_H__
+#define INCLUDE__PERF_HISI_PTT_H__
+
+#define HISI_PTT_PMU_NAME "hisi_ptt"
+enum {
+	HISI_PTT_PMU_TYPE,
+	HISI_PTT_PER_CPU_MMAPS,
+	HISI_PTT_AUXTRACE_PRIV_MAX,
+};
+
+#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
+union perf_event;
+struct perf_session;
+struct perf_pmu;
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+						struct perf_pmu *hisi_ptt_pmu);
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+				   struct perf_session *session);
+
+#endif
-- 
2.33.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 5/6] docs: Add HiSilicon PTT device driver documentation

[Yicong Yang]

Document the introduction and usage of HiSilicon PTT device driver.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 Documentation/trace/hisi-ptt.rst | 305 +++++++++++++++++++++++++++++++
 1 file changed, 305 insertions(+)
 create mode 100644 Documentation/trace/hisi-ptt.rst

diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
new file mode 100644
index 000000000000..a7ef6438e4aa
--- /dev/null
+++ b/Documentation/trace/hisi-ptt.rst
@@ -0,0 +1,305 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+HiSilicon PCIe Tune and Trace device
+======================================
+
+Introduction
+============
+
+HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
+integrated Endpoint (RCiEP) device, providing the capability
+to dynamically monitor and tune the PCIe link's events (tune),
+and trace the TLP headers (trace). The two functions are independent,
+but is recommended to use them together to analyze and enhance the
+PCIe link's performance.
+
+On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
+PCIe cores. Each PCIe core includes several Root Ports and a PTT
+RCiEP, like below. The PTT device is capable of tuning and
+tracing the link of the PCIe core.
+::
+          +--------------Core 0-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+    Root Complex  |------Core 1-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[ Switch ]---[Endpoint]
+          |       |       [Root Port]---[Endpoint] `-[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          +---------------------------+
+
+The PTT device driver registers PMU device for each PTT device.
+The name of each PTT device is composed of 'hisi_ptt' prefix with
+the id of the SICL and the Core where it locates. The Kunpeng 930
+SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
+IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
+Complex for each SICL.
+::
+    /sys/devices/hisi_ptt<sicl_id>_<core_id>
+
+Tune
+====
+
+PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
+Currently we support events in 4 classes. The scope of the events
+covers the PCIe core to which the PTT device belongs.
+
+Each event is presented as a file under $(PTT PMU dir)/tune, and
+mostly a simple open/read/write/close cycle will be used to tune
+the event.
+::
+    $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
+    $ ls
+    qos_tx_cpl    qos_tx_np    qos_tx_p
+    tx_path_rx_req_alloc_buf_level
+    tx_path_tx_req_alloc_buf_level
+    $ cat qos_tx_dp
+    1
+    $ echo 2 > qos_tx_dp
+    $ cat qos_tx_dp
+    2
+
+Current value (numerical value) of the event can be simply read
+from the file, and the desired value written to the file to tune.
+
+1. Tx path QoS control
+------------------------
+
+The following files are provided to tune the QoS of the tx path of
+the PCIe core.
+
+- qos_tx_cpl: weight of Tx completion TLPs
+- qos_tx_np: weight of Tx non-posted TLPs
+- qos_tx_p: weight of Tx posted TLPs
+
+The weight influences the proportion of certain packets on the PCIe link.
+For example, for the storage scenario, increase the proportion
+of the completion packets on the link to enhance the performance as
+more completions are consumed.
+
+The available tune data of these events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+2. Tx path buffer control
+-------------------------
+
+Following files are provided to tune the buffer of tx path of the PCIe core.
+
+- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
+- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
+
+These events influence the watermark of the buffer allocated for each
+type. Rx means the inbound while Tx means outbound. The packets will
+be stored in the buffer first and then posted either when the watermark
+reached or when timed out. For a busy direction, you should increase
+the related buffer watermark to avoid frequently posting and thus
+enhance the performance. In most cases just keep the default value.
+
+The available tune data of above events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+Trace
+=====
+
+PTT trace is designed for dumping the TLP headers to the memory, which
+can be used to analyze the transactions and usage condition of the PCIe
+Link. You can choose to filter the traced headers by either requester ID,
+or those downstream of a set of Root Ports on the same core of the PTT
+device. It's also supported to trace the headers of certain type and of
+certain direction.
+
+You can use the perf command `perf record` to set the parameters, start
+trace and get the data. It's also supported to decode the trace
+data with `perf report`. The control parameters for trace is inputted
+as event code for each events, which will be further illustracted later.
+An example usage is like
+::
+    $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
+      format=1/ -- sleep 5
+
+This will trace the TLP headers downstream root port 0000:00:10.1 (event
+code for event 'filter' is 0x80001) with type of posted TLP requests,
+direction of inbound and traced data format of 8DW.
+
+1. filter
+---------
+
+The TLP headers to trace can be filtered by the Root Ports or the requester
+ID of the endpoints, which are locates on the same core of the PTT device.
+You can set the filter by spedifying the `filter` parameter which is required
+to start the trace. The parameter value is 20 bit. The supported filters and
+related values is outputted through `available_filters` sysfs attribute
+under related PTT PMU directory, classified as Root Ports and Requesters
+respectively.
+::
+    $ cat available_filters
+    #### Root Ports ####
+    0000:00:10.0	0x80001
+    0000:00:11.0	0x80004
+    #### Requesters ####
+    0000:01:00.0	0x00100
+    0000:01:00.1	0x00101
+
+Note that multiple Root Ports can be specified at one time, but only
+one Endpoint function can be specified in one trace. Specifying both
+Root Port and function at the same time is not supported.
+
+If no filter is available, reading the available_filters will get the hint.
+::
+    $ cat available_filters
+    #### No available filter ####
+
+The available_filters can be dynamically updated, which means you can always
+get correct filter information when hotplug events happen, or when you manually
+remove/rescan the devices.
+
+2. type
+-------
+
+You can trace the TLP headers of certain types by specifying the `type`
+parameter, which is required to start the trace. The parameter value is
+8 bit. Current supported types and related values are shown below:
+
+8'b00000001: posted requests (P)
+8'b00000010: non-posted requests (NP)
+8'b00000100: completions (CPL)
+
+You can specify multiple types when tracing inbound TLP headers, but can only
+specify one when tracing outbound TLP headers.
+
+3. direction
+------------
+
+You can trace the TLP headers from certain direction, which is relative
+to the Root Port or the PCIe core, by specifying the `direction` parameter.
+This is optional and the default parameter is inbound. The parameter value
+is 4 bit. When the desired format is 4DW, directions and related values
+supported are shown below:
+
+4'b0000: inbound TLPs (P, NP, CPL)
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
+4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
+
+When the desired format is 8DW, directions and related values supported are
+shown below:
+
+4'b0000: reserved
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: inbound TLPs (P, NP, CPL B)
+4'b0011: inbound TLPs (CPL A)
+
+Inbound completions are classifed into two types:
+
+completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
+completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
+
+4. format
+--------------
+
+You can change the format of the traced TLP headers by specifying the
+`format` parameter. This is optional and the default format is 4DW.
+The parameter value is 4 bit. Current supported formats and related
+values are shown below:
+
+4'b0000: 4DW length per TLP header
+4'b0001: 8DW length per TLP header
+
+The traced TLP header format is different from the PCIe standard.
+
+When using the 8DW data format, the entire TLP header is logged
+(Header DW0-3 shown below). For example, the TLP header for Memory
+Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
+the header for Configuration Requests is shown in Figure 2.20, etc.
+
+In addition, 8DW trace buffer entries contain a timestamp and
+possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
+Otherwise this field will be all 0.
+
+The bit[31:11] of DW0 is always 0x1fffff, which can be
+used to distinguish the data format. 8DW format is like
+::
+    bits [                 31:11                 ][       10:0       ]
+         |---------------------------------------|-------------------|
+     DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+     DW1 [                       Prefix                              ]
+     DW2 [                     Header DW0                            ]
+     DW3 [                     Header DW1                            ]
+     DW4 [                     Header DW2                            ]
+     DW5 [                     Header DW3                            ]
+     DW6 [                   Reserved (0x0)                          ]
+     DW7 [                        Time                               ]
+
+When using the 4DW data format, DW0 of the trace buffer entry
+contains selected fields of DW0 of the TLP, together with a
+timestamp.  DW1-DW3 of the trace buffer entry contain DW1-DW3
+directly from the TLP header.
+
+4DW format is like
+::
+    bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+         |-----|---------|---|---|---|---|-------------|-------------|
+     DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+     DW1 [                     Header DW1                            ]
+     DW2 [                     Header DW2                            ]
+     DW3 [                     Header DW3                            ]
+
+5. memory management
+--------------------
+
+The traced TLP headers will be written to the memory allocated
+by the driver. The hardware accepts 4 DMA address with same size,
+and writes the buflet sequentially like below. If DMA addr 3 is
+finished and the trace is still on, it will return to addr 0.
+::
+    +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
+    +---------------------------------------------------------+
+
+Driver will allocate each DMA buffer (we call it buflet) of 4MiB.
+The finished buflet will be copied to the perf AUX buffer allocated
+by the perf core. Once the AUX buffer is full while the trace is
+still on, driver will commit the AUX buffer first and then apply
+for a new one with the same size. The size of AUX buffer is default
+to 16MiB. User can adjust the size by specifying the `-m` parameter
+of the perf command.
+
+Note that there is a gap between committing the old AUX buffer and
+applying a new one, which means the trace is stopped during the
+moment and TLPs transferred in the moment cannot be traced. To avoid
+this situation, you should begin the trace with large AUX buffer
+enough to avoid this gap.
+
+6. decoding
+-----------
+
+You can decode the traced data with `perf report -D` command (currently
+only support to dump the raw trace data). The traced data will be decoded
+according to the format described previously (take 8DW as an example):
+::
+    [...perf headers and other information]
+    . ... HISI PTT data: size 4194304 bytes
+    .  00000000: 00 00 00 00                                 Prefix
+    .  00000004: 01 00 00 60                                 Header DW0
+    .  00000008: 0f 1e 00 01                                 Header DW1
+    .  0000000c: 04 00 00 00                                 Header DW2
+    .  00000010: 40 00 81 02                                 Header DW3
+    .  00000014: 33 c0 04 00                                 Time
+    .  00000020: 00 00 00 00                                 Prefix
+    .  00000024: 01 00 00 60                                 Header DW0
+    .  00000028: 0f 1e 00 01                                 Header DW1
+    .  0000002c: 04 00 00 00                                 Header DW2
+    .  00000030: 40 00 81 02                                 Header DW3
+    .  00000034: 02 00 00 00                                 Time
+    .  00000040: 00 00 00 00                                 Prefix
+    .  00000044: 01 00 00 60                                 Header DW0
+    .  00000048: 0f 1e 00 01                                 Header DW1
+    .  0000004c: 04 00 00 00                                 Header DW2
+    .  00000050: 40 00 81 02                                 Header DW3
+    [...]
-- 
2.33.0

[PATCH v2 5/6] docs: Add HiSilicon PTT device driver documentation

[Yicong Yang via iommu]

Document the introduction and usage of HiSilicon PTT device driver.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 Documentation/trace/hisi-ptt.rst | 305 +++++++++++++++++++++++++++++++
 1 file changed, 305 insertions(+)
 create mode 100644 Documentation/trace/hisi-ptt.rst

diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
new file mode 100644
index 000000000000..a7ef6438e4aa
--- /dev/null
+++ b/Documentation/trace/hisi-ptt.rst
@@ -0,0 +1,305 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+HiSilicon PCIe Tune and Trace device
+======================================
+
+Introduction
+============
+
+HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
+integrated Endpoint (RCiEP) device, providing the capability
+to dynamically monitor and tune the PCIe link's events (tune),
+and trace the TLP headers (trace). The two functions are independent,
+but is recommended to use them together to analyze and enhance the
+PCIe link's performance.
+
+On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
+PCIe cores. Each PCIe core includes several Root Ports and a PTT
+RCiEP, like below. The PTT device is capable of tuning and
+tracing the link of the PCIe core.
+::
+          +--------------Core 0-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+    Root Complex  |------Core 1-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[ Switch ]---[Endpoint]
+          |       |       [Root Port]---[Endpoint] `-[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          +---------------------------+
+
+The PTT device driver registers PMU device for each PTT device.
+The name of each PTT device is composed of 'hisi_ptt' prefix with
+the id of the SICL and the Core where it locates. The Kunpeng 930
+SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
+IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
+Complex for each SICL.
+::
+    /sys/devices/hisi_ptt<sicl_id>_<core_id>
+
+Tune
+====
+
+PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
+Currently we support events in 4 classes. The scope of the events
+covers the PCIe core to which the PTT device belongs.
+
+Each event is presented as a file under $(PTT PMU dir)/tune, and
+mostly a simple open/read/write/close cycle will be used to tune
+the event.
+::
+    $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
+    $ ls
+    qos_tx_cpl    qos_tx_np    qos_tx_p
+    tx_path_rx_req_alloc_buf_level
+    tx_path_tx_req_alloc_buf_level
+    $ cat qos_tx_dp
+    1
+    $ echo 2 > qos_tx_dp
+    $ cat qos_tx_dp
+    2
+
+Current value (numerical value) of the event can be simply read
+from the file, and the desired value written to the file to tune.
+
+1. Tx path QoS control
+------------------------
+
+The following files are provided to tune the QoS of the tx path of
+the PCIe core.
+
+- qos_tx_cpl: weight of Tx completion TLPs
+- qos_tx_np: weight of Tx non-posted TLPs
+- qos_tx_p: weight of Tx posted TLPs
+
+The weight influences the proportion of certain packets on the PCIe link.
+For example, for the storage scenario, increase the proportion
+of the completion packets on the link to enhance the performance as
+more completions are consumed.
+
+The available tune data of these events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+2. Tx path buffer control
+-------------------------
+
+Following files are provided to tune the buffer of tx path of the PCIe core.
+
+- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
+- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
+
+These events influence the watermark of the buffer allocated for each
+type. Rx means the inbound while Tx means outbound. The packets will
+be stored in the buffer first and then posted either when the watermark
+reached or when timed out. For a busy direction, you should increase
+the related buffer watermark to avoid frequently posting and thus
+enhance the performance. In most cases just keep the default value.
+
+The available tune data of above events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+Trace
+=====
+
+PTT trace is designed for dumping the TLP headers to the memory, which
+can be used to analyze the transactions and usage condition of the PCIe
+Link. You can choose to filter the traced headers by either requester ID,
+or those downstream of a set of Root Ports on the same core of the PTT
+device. It's also supported to trace the headers of certain type and of
+certain direction.
+
+You can use the perf command `perf record` to set the parameters, start
+trace and get the data. It's also supported to decode the trace
+data with `perf report`. The control parameters for trace is inputted
+as event code for each events, which will be further illustracted later.
+An example usage is like
+::
+    $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
+      format=1/ -- sleep 5
+
+This will trace the TLP headers downstream root port 0000:00:10.1 (event
+code for event 'filter' is 0x80001) with type of posted TLP requests,
+direction of inbound and traced data format of 8DW.
+
+1. filter
+---------
+
+The TLP headers to trace can be filtered by the Root Ports or the requester
+ID of the endpoints, which are locates on the same core of the PTT device.
+You can set the filter by spedifying the `filter` parameter which is required
+to start the trace. The parameter value is 20 bit. The supported filters and
+related values is outputted through `available_filters` sysfs attribute
+under related PTT PMU directory, classified as Root Ports and Requesters
+respectively.
+::
+    $ cat available_filters
+    #### Root Ports ####
+    0000:00:10.0	0x80001
+    0000:00:11.0	0x80004
+    #### Requesters ####
+    0000:01:00.0	0x00100
+    0000:01:00.1	0x00101
+
+Note that multiple Root Ports can be specified at one time, but only
+one Endpoint function can be specified in one trace. Specifying both
+Root Port and function at the same time is not supported.
+
+If no filter is available, reading the available_filters will get the hint.
+::
+    $ cat available_filters
+    #### No available filter ####
+
+The available_filters can be dynamically updated, which means you can always
+get correct filter information when hotplug events happen, or when you manually
+remove/rescan the devices.
+
+2. type
+-------
+
+You can trace the TLP headers of certain types by specifying the `type`
+parameter, which is required to start the trace. The parameter value is
+8 bit. Current supported types and related values are shown below:
+
+8'b00000001: posted requests (P)
+8'b00000010: non-posted requests (NP)
+8'b00000100: completions (CPL)
+
+You can specify multiple types when tracing inbound TLP headers, but can only
+specify one when tracing outbound TLP headers.
+
+3. direction
+------------
+
+You can trace the TLP headers from certain direction, which is relative
+to the Root Port or the PCIe core, by specifying the `direction` parameter.
+This is optional and the default parameter is inbound. The parameter value
+is 4 bit. When the desired format is 4DW, directions and related values
+supported are shown below:
+
+4'b0000: inbound TLPs (P, NP, CPL)
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
+4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
+
+When the desired format is 8DW, directions and related values supported are
+shown below:
+
+4'b0000: reserved
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: inbound TLPs (P, NP, CPL B)
+4'b0011: inbound TLPs (CPL A)
+
+Inbound completions are classifed into two types:
+
+completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
+completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
+
+4. format
+--------------
+
+You can change the format of the traced TLP headers by specifying the
+`format` parameter. This is optional and the default format is 4DW.
+The parameter value is 4 bit. Current supported formats and related
+values are shown below:
+
+4'b0000: 4DW length per TLP header
+4'b0001: 8DW length per TLP header
+
+The traced TLP header format is different from the PCIe standard.
+
+When using the 8DW data format, the entire TLP header is logged
+(Header DW0-3 shown below). For example, the TLP header for Memory
+Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
+the header for Configuration Requests is shown in Figure 2.20, etc.
+
+In addition, 8DW trace buffer entries contain a timestamp and
+possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
+Otherwise this field will be all 0.
+
+The bit[31:11] of DW0 is always 0x1fffff, which can be
+used to distinguish the data format. 8DW format is like
+::
+    bits [                 31:11                 ][       10:0       ]
+         |---------------------------------------|-------------------|
+     DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+     DW1 [                       Prefix                              ]
+     DW2 [                     Header DW0                            ]
+     DW3 [                     Header DW1                            ]
+     DW4 [                     Header DW2                            ]
+     DW5 [                     Header DW3                            ]
+     DW6 [                   Reserved (0x0)                          ]
+     DW7 [                        Time                               ]
+
+When using the 4DW data format, DW0 of the trace buffer entry
+contains selected fields of DW0 of the TLP, together with a
+timestamp.  DW1-DW3 of the trace buffer entry contain DW1-DW3
+directly from the TLP header.
+
+4DW format is like
+::
+    bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+         |-----|---------|---|---|---|---|-------------|-------------|
+     DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+     DW1 [                     Header DW1                            ]
+     DW2 [                     Header DW2                            ]
+     DW3 [                     Header DW3                            ]
+
+5. memory management
+--------------------
+
+The traced TLP headers will be written to the memory allocated
+by the driver. The hardware accepts 4 DMA address with same size,
+and writes the buflet sequentially like below. If DMA addr 3 is
+finished and the trace is still on, it will return to addr 0.
+::
+    +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
+    +---------------------------------------------------------+
+
+Driver will allocate each DMA buffer (we call it buflet) of 4MiB.
+The finished buflet will be copied to the perf AUX buffer allocated
+by the perf core. Once the AUX buffer is full while the trace is
+still on, driver will commit the AUX buffer first and then apply
+for a new one with the same size. The size of AUX buffer is default
+to 16MiB. User can adjust the size by specifying the `-m` parameter
+of the perf command.
+
+Note that there is a gap between committing the old AUX buffer and
+applying a new one, which means the trace is stopped during the
+moment and TLPs transferred in the moment cannot be traced. To avoid
+this situation, you should begin the trace with large AUX buffer
+enough to avoid this gap.
+
+6. decoding
+-----------
+
+You can decode the traced data with `perf report -D` command (currently
+only support to dump the raw trace data). The traced data will be decoded
+according to the format described previously (take 8DW as an example):
+::
+    [...perf headers and other information]
+    . ... HISI PTT data: size 4194304 bytes
+    .  00000000: 00 00 00 00                                 Prefix
+    .  00000004: 01 00 00 60                                 Header DW0
+    .  00000008: 0f 1e 00 01                                 Header DW1
+    .  0000000c: 04 00 00 00                                 Header DW2
+    .  00000010: 40 00 81 02                                 Header DW3
+    .  00000014: 33 c0 04 00                                 Time
+    .  00000020: 00 00 00 00                                 Prefix
+    .  00000024: 01 00 00 60                                 Header DW0
+    .  00000028: 0f 1e 00 01                                 Header DW1
+    .  0000002c: 04 00 00 00                                 Header DW2
+    .  00000030: 40 00 81 02                                 Header DW3
+    .  00000034: 02 00 00 00                                 Time
+    .  00000040: 00 00 00 00                                 Prefix
+    .  00000044: 01 00 00 60                                 Header DW0
+    .  00000048: 0f 1e 00 01                                 Header DW1
+    .  0000004c: 04 00 00 00                                 Header DW2
+    .  00000050: 40 00 81 02                                 Header DW3
+    [...]
-- 
2.33.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 5/6] docs: Add HiSilicon PTT device driver documentation

[Yicong Yang]

Document the introduction and usage of HiSilicon PTT device driver.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 Documentation/trace/hisi-ptt.rst | 305 +++++++++++++++++++++++++++++++
 1 file changed, 305 insertions(+)
 create mode 100644 Documentation/trace/hisi-ptt.rst

diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
new file mode 100644
index 000000000000..a7ef6438e4aa
--- /dev/null
+++ b/Documentation/trace/hisi-ptt.rst
@@ -0,0 +1,305 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+HiSilicon PCIe Tune and Trace device
+======================================
+
+Introduction
+============
+
+HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
+integrated Endpoint (RCiEP) device, providing the capability
+to dynamically monitor and tune the PCIe link's events (tune),
+and trace the TLP headers (trace). The two functions are independent,
+but is recommended to use them together to analyze and enhance the
+PCIe link's performance.
+
+On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
+PCIe cores. Each PCIe core includes several Root Ports and a PTT
+RCiEP, like below. The PTT device is capable of tuning and
+tracing the link of the PCIe core.
+::
+          +--------------Core 0-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+    Root Complex  |------Core 1-------+
+          |       |       [   PTT   ] |
+          |       |       [Root Port]---[ Switch ]---[Endpoint]
+          |       |       [Root Port]---[Endpoint] `-[Endpoint]
+          |       |       [Root Port]---[Endpoint]
+          +---------------------------+
+
+The PTT device driver registers PMU device for each PTT device.
+The name of each PTT device is composed of 'hisi_ptt' prefix with
+the id of the SICL and the Core where it locates. The Kunpeng 930
+SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
+IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
+Complex for each SICL.
+::
+    /sys/devices/hisi_ptt<sicl_id>_<core_id>
+
+Tune
+====
+
+PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
+Currently we support events in 4 classes. The scope of the events
+covers the PCIe core to which the PTT device belongs.
+
+Each event is presented as a file under $(PTT PMU dir)/tune, and
+mostly a simple open/read/write/close cycle will be used to tune
+the event.
+::
+    $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
+    $ ls
+    qos_tx_cpl    qos_tx_np    qos_tx_p
+    tx_path_rx_req_alloc_buf_level
+    tx_path_tx_req_alloc_buf_level
+    $ cat qos_tx_dp
+    1
+    $ echo 2 > qos_tx_dp
+    $ cat qos_tx_dp
+    2
+
+Current value (numerical value) of the event can be simply read
+from the file, and the desired value written to the file to tune.
+
+1. Tx path QoS control
+------------------------
+
+The following files are provided to tune the QoS of the tx path of
+the PCIe core.
+
+- qos_tx_cpl: weight of Tx completion TLPs
+- qos_tx_np: weight of Tx non-posted TLPs
+- qos_tx_p: weight of Tx posted TLPs
+
+The weight influences the proportion of certain packets on the PCIe link.
+For example, for the storage scenario, increase the proportion
+of the completion packets on the link to enhance the performance as
+more completions are consumed.
+
+The available tune data of these events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+2. Tx path buffer control
+-------------------------
+
+Following files are provided to tune the buffer of tx path of the PCIe core.
+
+- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
+- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
+
+These events influence the watermark of the buffer allocated for each
+type. Rx means the inbound while Tx means outbound. The packets will
+be stored in the buffer first and then posted either when the watermark
+reached or when timed out. For a busy direction, you should increase
+the related buffer watermark to avoid frequently posting and thus
+enhance the performance. In most cases just keep the default value.
+
+The available tune data of above events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+Trace
+=====
+
+PTT trace is designed for dumping the TLP headers to the memory, which
+can be used to analyze the transactions and usage condition of the PCIe
+Link. You can choose to filter the traced headers by either requester ID,
+or those downstream of a set of Root Ports on the same core of the PTT
+device. It's also supported to trace the headers of certain type and of
+certain direction.
+
+You can use the perf command `perf record` to set the parameters, start
+trace and get the data. It's also supported to decode the trace
+data with `perf report`. The control parameters for trace is inputted
+as event code for each events, which will be further illustracted later.
+An example usage is like
+::
+    $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
+      format=1/ -- sleep 5
+
+This will trace the TLP headers downstream root port 0000:00:10.1 (event
+code for event 'filter' is 0x80001) with type of posted TLP requests,
+direction of inbound and traced data format of 8DW.
+
+1. filter
+---------
+
+The TLP headers to trace can be filtered by the Root Ports or the requester
+ID of the endpoints, which are locates on the same core of the PTT device.
+You can set the filter by spedifying the `filter` parameter which is required
+to start the trace. The parameter value is 20 bit. The supported filters and
+related values is outputted through `available_filters` sysfs attribute
+under related PTT PMU directory, classified as Root Ports and Requesters
+respectively.
+::
+    $ cat available_filters
+    #### Root Ports ####
+    0000:00:10.0	0x80001
+    0000:00:11.0	0x80004
+    #### Requesters ####
+    0000:01:00.0	0x00100
+    0000:01:00.1	0x00101
+
+Note that multiple Root Ports can be specified at one time, but only
+one Endpoint function can be specified in one trace. Specifying both
+Root Port and function at the same time is not supported.
+
+If no filter is available, reading the available_filters will get the hint.
+::
+    $ cat available_filters
+    #### No available filter ####
+
+The available_filters can be dynamically updated, which means you can always
+get correct filter information when hotplug events happen, or when you manually
+remove/rescan the devices.
+
+2. type
+-------
+
+You can trace the TLP headers of certain types by specifying the `type`
+parameter, which is required to start the trace. The parameter value is
+8 bit. Current supported types and related values are shown below:
+
+8'b00000001: posted requests (P)
+8'b00000010: non-posted requests (NP)
+8'b00000100: completions (CPL)
+
+You can specify multiple types when tracing inbound TLP headers, but can only
+specify one when tracing outbound TLP headers.
+
+3. direction
+------------
+
+You can trace the TLP headers from certain direction, which is relative
+to the Root Port or the PCIe core, by specifying the `direction` parameter.
+This is optional and the default parameter is inbound. The parameter value
+is 4 bit. When the desired format is 4DW, directions and related values
+supported are shown below:
+
+4'b0000: inbound TLPs (P, NP, CPL)
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
+4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
+
+When the desired format is 8DW, directions and related values supported are
+shown below:
+
+4'b0000: reserved
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: inbound TLPs (P, NP, CPL B)
+4'b0011: inbound TLPs (CPL A)
+
+Inbound completions are classifed into two types:
+
+completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
+completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
+
+4. format
+--------------
+
+You can change the format of the traced TLP headers by specifying the
+`format` parameter. This is optional and the default format is 4DW.
+The parameter value is 4 bit. Current supported formats and related
+values are shown below:
+
+4'b0000: 4DW length per TLP header
+4'b0001: 8DW length per TLP header
+
+The traced TLP header format is different from the PCIe standard.
+
+When using the 8DW data format, the entire TLP header is logged
+(Header DW0-3 shown below). For example, the TLP header for Memory
+Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
+the header for Configuration Requests is shown in Figure 2.20, etc.
+
+In addition, 8DW trace buffer entries contain a timestamp and
+possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
+Otherwise this field will be all 0.
+
+The bit[31:11] of DW0 is always 0x1fffff, which can be
+used to distinguish the data format. 8DW format is like
+::
+    bits [                 31:11                 ][       10:0       ]
+         |---------------------------------------|-------------------|
+     DW0 [                0x1fffff               ][ Reserved (0x7ff) ]
+     DW1 [                       Prefix                              ]
+     DW2 [                     Header DW0                            ]
+     DW3 [                     Header DW1                            ]
+     DW4 [                     Header DW2                            ]
+     DW5 [                     Header DW3                            ]
+     DW6 [                   Reserved (0x0)                          ]
+     DW7 [                        Time                               ]
+
+When using the 4DW data format, DW0 of the trace buffer entry
+contains selected fields of DW0 of the TLP, together with a
+timestamp.  DW1-DW3 of the trace buffer entry contain DW1-DW3
+directly from the TLP header.
+
+4DW format is like
+::
+    bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]
+         |-----|---------|---|---|---|---|-------------|-------------|
+     DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ]
+     DW1 [                     Header DW1                            ]
+     DW2 [                     Header DW2                            ]
+     DW3 [                     Header DW3                            ]
+
+5. memory management
+--------------------
+
+The traced TLP headers will be written to the memory allocated
+by the driver. The hardware accepts 4 DMA address with same size,
+and writes the buflet sequentially like below. If DMA addr 3 is
+finished and the trace is still on, it will return to addr 0.
+::
+    +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
+    +---------------------------------------------------------+
+
+Driver will allocate each DMA buffer (we call it buflet) of 4MiB.
+The finished buflet will be copied to the perf AUX buffer allocated
+by the perf core. Once the AUX buffer is full while the trace is
+still on, driver will commit the AUX buffer first and then apply
+for a new one with the same size. The size of AUX buffer is default
+to 16MiB. User can adjust the size by specifying the `-m` parameter
+of the perf command.
+
+Note that there is a gap between committing the old AUX buffer and
+applying a new one, which means the trace is stopped during the
+moment and TLPs transferred in the moment cannot be traced. To avoid
+this situation, you should begin the trace with large AUX buffer
+enough to avoid this gap.
+
+6. decoding
+-----------
+
+You can decode the traced data with `perf report -D` command (currently
+only support to dump the raw trace data). The traced data will be decoded
+according to the format described previously (take 8DW as an example):
+::
+    [...perf headers and other information]
+    . ... HISI PTT data: size 4194304 bytes
+    .  00000000: 00 00 00 00                                 Prefix
+    .  00000004: 01 00 00 60                                 Header DW0
+    .  00000008: 0f 1e 00 01                                 Header DW1
+    .  0000000c: 04 00 00 00                                 Header DW2
+    .  00000010: 40 00 81 02                                 Header DW3
+    .  00000014: 33 c0 04 00                                 Time
+    .  00000020: 00 00 00 00                                 Prefix
+    .  00000024: 01 00 00 60                                 Header DW0
+    .  00000028: 0f 1e 00 01                                 Header DW1
+    .  0000002c: 04 00 00 00                                 Header DW2
+    .  00000030: 40 00 81 02                                 Header DW3
+    .  00000034: 02 00 00 00                                 Time
+    .  00000040: 00 00 00 00                                 Prefix
+    .  00000044: 01 00 00 60                                 Header DW0
+    .  00000048: 0f 1e 00 01                                 Header DW1
+    .  0000004c: 04 00 00 00                                 Header DW2
+    .  00000050: 40 00 81 02                                 Header DW3
+    [...]
-- 
2.33.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 6/6] MAINTAINERS: Add maintainer for HiSilicon PTT driver

[Yicong Yang]

Add maintainer for driver and documentation of HiSilicon PTT device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 7a2345ce8521..823d495ca0d5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8601,6 +8601,13 @@ W:	http://www.hisilicon.com
 F:	Documentation/admin-guide/perf/hisi-pmu.rst
 F:	drivers/perf/hisilicon
 
+HISILICON PTT DRIVER
+M:	Yicong Yang <yangyicong@hisilicon.com>
+L:	linux-kernel@vger.kernel.org
+S:	Maintained
+F:	Documentation/trace/hisi-ptt.rst
+F:	drivers/hwtracing/hisilicon/
+
 HISILICON QM AND ZIP Controller DRIVER
 M:	Zhou Wang <wangzhou1@hisilicon.com>
 L:	linux-crypto@vger.kernel.org
-- 
2.33.0

[PATCH v2 6/6] MAINTAINERS: Add maintainer for HiSilicon PTT driver

[Yicong Yang via iommu]

Add maintainer for driver and documentation of HiSilicon PTT device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 7a2345ce8521..823d495ca0d5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8601,6 +8601,13 @@ W:	http://www.hisilicon.com
 F:	Documentation/admin-guide/perf/hisi-pmu.rst
 F:	drivers/perf/hisilicon
 
+HISILICON PTT DRIVER
+M:	Yicong Yang <yangyicong@hisilicon.com>
+L:	linux-kernel@vger.kernel.org
+S:	Maintained
+F:	Documentation/trace/hisi-ptt.rst
+F:	drivers/hwtracing/hisilicon/
+
 HISILICON QM AND ZIP Controller DRIVER
 M:	Zhou Wang <wangzhou1@hisilicon.com>
 L:	linux-crypto@vger.kernel.org
-- 
2.33.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 6/6] MAINTAINERS: Add maintainer for HiSilicon PTT driver

[Yicong Yang]

Add maintainer for driver and documentation of HiSilicon PTT device.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 7a2345ce8521..823d495ca0d5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8601,6 +8601,13 @@ W:	http://www.hisilicon.com
 F:	Documentation/admin-guide/perf/hisi-pmu.rst
 F:	drivers/perf/hisilicon
 
+HISILICON PTT DRIVER
+M:	Yicong Yang <yangyicong@hisilicon.com>
+L:	linux-kernel@vger.kernel.org
+S:	Maintained
+F:	Documentation/trace/hisi-ptt.rst
+F:	drivers/hwtracing/hisilicon/
+
 HISILICON QM AND ZIP Controller DRIVER
 M:	Zhou Wang <wangzhou1@hisilicon.com>
 L:	linux-crypto@vger.kernel.org
-- 
2.33.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel