[PATCH v10 0/7] Cavium ARM64 uncore PMU support

[PATCH v10 0/7] Cavium ARM64 uncore PMU support

[Jan Glauber]

Add support for various PMU counters found on the Cavium ThunderX and
OcteonTx SoC.

The PMU driver provides common "uncore" functions to avoid code duplication
and support adding more device PMUs (like L2 cache) in the future.

Changes to v9:
- Fix build error in first EDAC patch
- Use alphabetic ordering in some places
- Store pmu name in struct pmu
- Use devm_*
- Remove mapping of LMC events, expose hardware event numbers
  directly
- Simplify event removal

Changes to v8:
- Wrapper for PCI devices

Jan Glauber (7):
  edac: thunderx: Remove suspend/resume support
  perf: export perf_event_update_userpage()
  edac,soc: thunderx: Add wrapper for EDAC LMC PCI device
  edac,soc: thunderx: Add wrapper for EDAC OCX PCI device
  perf: cavium: Support memory controller PMU counters
  perf: cavium: Support transmit-link PMU counters
  perf: cavium: Add Documentation

 Documentation/perf/cavium-pmu.txt |  75 +++++
 drivers/edac/Kconfig              |   3 +
 drivers/edac/thunderx_edac.c      |  92 +-----
 drivers/perf/Kconfig              |  15 +
 drivers/perf/Makefile             |   1 +
 drivers/perf/cavium_pmu.c         | 660 ++++++++++++++++++++++++++++++++++++++
 drivers/soc/Kconfig               |   1 +
 drivers/soc/Makefile              |   1 +
 drivers/soc/cavium/Kconfig        |  14 +
 drivers/soc/cavium/Makefile       |   2 +
 drivers/soc/cavium/cavium_lmc.c   |  53 +++
 drivers/soc/cavium/cavium_ocx.c   |  49 +++
 include/linux/cpuhotplug.h        |   1 +
 include/linux/soc/cavium/lmc.h    |  12 +
 include/linux/soc/cavium/ocx.h    |  12 +
 kernel/events/core.c              |   1 +
 16 files changed, 913 insertions(+), 79 deletions(-)
 create mode 100644 Documentation/perf/cavium-pmu.txt
 create mode 100644 drivers/perf/cavium_pmu.c
 create mode 100644 drivers/soc/cavium/Kconfig
 create mode 100644 drivers/soc/cavium/Makefile
 create mode 100644 drivers/soc/cavium/cavium_lmc.c
 create mode 100644 drivers/soc/cavium/cavium_ocx.c
 create mode 100644 include/linux/soc/cavium/lmc.h
 create mode 100644 include/linux/soc/cavium/ocx.h

-- 
2.9.0.rc0.21.g7777322

[PATCH v10 1/7] edac: thunderx: Remove suspend/resume support

[Jan Glauber]

The memory controller on ThunderX/OcteonTX systems does not
support power management. Therefore remove the suspend/resume
callbacks.

Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
 drivers/edac/thunderx_edac.c | 25 -------------------------
 1 file changed, 25 deletions(-)

diff --git a/drivers/edac/thunderx_edac.c b/drivers/edac/thunderx_edac.c
index f35d87519a3e..4803c6468bab 100644
--- a/drivers/edac/thunderx_edac.c
+++ b/drivers/edac/thunderx_edac.c
@@ -639,27 +639,6 @@ static irqreturn_t thunderx_lmc_threaded_isr(int irq, void *dev_id)
 	return ret;
 }
 
-#ifdef CONFIG_PM
-static int thunderx_lmc_suspend(struct pci_dev *pdev, pm_message_t state)
-{
-	pci_save_state(pdev);
-	pci_disable_device(pdev);
-
-	pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
-	return 0;
-}
-
-static int thunderx_lmc_resume(struct pci_dev *pdev)
-{
-	pci_set_power_state(pdev, PCI_D0);
-	pci_enable_wake(pdev, PCI_D0, 0);
-	pci_restore_state(pdev);
-
-	return 0;
-}
-#endif
-
 static const struct pci_device_id thunderx_lmc_pci_tbl[] = {
 	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_LMC) },
 	{ 0, },
@@ -834,10 +813,6 @@ static struct pci_driver thunderx_lmc_driver = {
 	.name     = "thunderx_lmc_edac",
 	.probe    = thunderx_lmc_probe,
 	.remove   = thunderx_lmc_remove,
-#ifdef CONFIG_PM
-	.suspend  = thunderx_lmc_suspend,
-	.resume   = thunderx_lmc_resume,
-#endif
 	.id_table = thunderx_lmc_pci_tbl,
 };
 
-- 
2.9.0.rc0.21.g7777322

[PATCH v10 2/7] perf: export perf_event_update_userpage()

[Jan Glauber]

Export perf_event_update_userpage(). This change is needed to allow
building a PMU driver as a kernel module.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>

Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
 kernel/events/core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 6bc21e202ae4..162f5ba756a9 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -4982,6 +4982,7 @@ void perf_event_update_userpage(struct perf_event *event)
 unlock:
 	rcu_read_unlock();
 }
+EXPORT_SYMBOL_GPL(perf_event_update_userpage);
 
 static int perf_mmap_fault(struct vm_fault *vmf)
 {
-- 
2.9.0.rc0.21.g7777322

[PATCH v10 3/7] edac,soc: thunderx: Add wrapper for EDAC LMC PCI device

[Jan Glauber]

Cavium SOCs contain a memory controller that is presented as a
PCI device. This PCI device will be used by an EDAC driver and
by a PMU driver.

To allow both subsystems to access the device a small wrapper is
introduced that multi-plexes PCI probe and removal calls of the
device to the EDAC driver.

The same mechanism will be used later to call the PMU driver.

The ThunderX EDAC driver is limited to only build as module
with this patch. The reason is that with multiple users of the
multi-plexer all users must be either builtin or modules.

Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
 drivers/edac/Kconfig            |  2 ++
 drivers/edac/thunderx_edac.c    | 27 ++++++-----------------
 drivers/soc/Kconfig             |  1 +
 drivers/soc/Makefile            |  1 +
 drivers/soc/cavium/Kconfig      |  6 +++++
 drivers/soc/cavium/Makefile     |  1 +
 drivers/soc/cavium/cavium_lmc.c | 49 +++++++++++++++++++++++++++++++++++++++++
 include/linux/soc/cavium/lmc.h  |  9 ++++++++
 8 files changed, 76 insertions(+), 20 deletions(-)
 create mode 100644 drivers/soc/cavium/Kconfig
 create mode 100644 drivers/soc/cavium/Makefile
 create mode 100644 drivers/soc/cavium/cavium_lmc.c
 create mode 100644 include/linux/soc/cavium/lmc.h

diff --git a/drivers/edac/Kconfig b/drivers/edac/Kconfig
index 96afb2aeed18..7330447c43d1 100644
--- a/drivers/edac/Kconfig
+++ b/drivers/edac/Kconfig
@@ -371,6 +371,8 @@ config EDAC_THUNDERX
 	tristate "Cavium ThunderX EDAC"
 	depends on ARM64
 	depends on PCI
+	depends on m
+	select CAVIUM_LMC
 	help
 	  Support for error detection and correction on the
 	  Cavium ThunderX memory controllers (LMC), Cache
diff --git a/drivers/edac/thunderx_edac.c b/drivers/edac/thunderx_edac.c
index 4803c6468bab..a6a89bf0a457 100644
--- a/drivers/edac/thunderx_edac.c
+++ b/drivers/edac/thunderx_edac.c
@@ -12,6 +12,7 @@
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/edac.h>
+#include <linux/export.h>
 #include <linux/interrupt.h>
 #include <linux/string.h>
 #include <linux/stop_machine.h>
@@ -20,6 +21,7 @@
 #include <linux/atomic.h>
 #include <linux/bitfield.h>
 #include <linux/circ_buf.h>
+#include <linux/soc/cavium/lmc.h>
 
 #include <asm/page.h>
 
@@ -654,8 +656,7 @@ static inline int pci_dev_to_mc_idx(struct pci_dev *pdev)
 	return ret;
 }
 
-static int thunderx_lmc_probe(struct pci_dev *pdev,
-				const struct pci_device_id *id)
+int thunderx_edac_lmc_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct thunderx_lmc *lmc;
 	struct edac_mc_layer layer;
@@ -795,8 +796,9 @@ static int thunderx_lmc_probe(struct pci_dev *pdev,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(thunderx_edac_lmc_probe);
 
-static void thunderx_lmc_remove(struct pci_dev *pdev)
+void thunderx_edac_lmc_remove(struct pci_dev *pdev)
 {
 	struct mem_ctl_info *mci = pci_get_drvdata(pdev);
 	struct thunderx_lmc *lmc = mci->pvt_info;
@@ -806,15 +808,7 @@ static void thunderx_lmc_remove(struct pci_dev *pdev)
 	edac_mc_del_mc(&pdev->dev);
 	edac_mc_free(mci);
 }
-
-MODULE_DEVICE_TABLE(pci, thunderx_lmc_pci_tbl);
-
-static struct pci_driver thunderx_lmc_driver = {
-	.name     = "thunderx_lmc_edac",
-	.probe    = thunderx_lmc_probe,
-	.remove   = thunderx_lmc_remove,
-	.id_table = thunderx_lmc_pci_tbl,
-};
+EXPORT_SYMBOL_GPL(thunderx_edac_lmc_remove);
 
 /*---------------------- OCX driver ---------------------------------*/
 
@@ -2110,13 +2104,9 @@ static int __init thunderx_edac_init(void)
 {
 	int rc = 0;
 
-	rc = pci_register_driver(&thunderx_lmc_driver);
-	if (rc)
-		return rc;
-
 	rc = pci_register_driver(&thunderx_ocx_driver);
 	if (rc)
-		goto err_lmc;
+		return rc;
 
 	rc = pci_register_driver(&thunderx_l2c_driver);
 	if (rc)
@@ -2125,8 +2115,6 @@ static int __init thunderx_edac_init(void)
 	return rc;
 err_ocx:
 	pci_unregister_driver(&thunderx_ocx_driver);
-err_lmc:
-	pci_unregister_driver(&thunderx_lmc_driver);
 
 	return rc;
 }
@@ -2135,7 +2123,6 @@ static void __exit thunderx_edac_exit(void)
 {
 	pci_unregister_driver(&thunderx_l2c_driver);
 	pci_unregister_driver(&thunderx_ocx_driver);
-	pci_unregister_driver(&thunderx_lmc_driver);
 
 }
 
diff --git a/drivers/soc/Kconfig b/drivers/soc/Kconfig
index fc9e98047421..f19f6237f336 100644
--- a/drivers/soc/Kconfig
+++ b/drivers/soc/Kconfig
@@ -4,6 +4,7 @@ source "drivers/soc/actions/Kconfig"
 source "drivers/soc/amlogic/Kconfig"
 source "drivers/soc/atmel/Kconfig"
 source "drivers/soc/bcm/Kconfig"
+source "drivers/soc/cavium/Kconfig"
 source "drivers/soc/fsl/Kconfig"
 source "drivers/soc/imx/Kconfig"
 source "drivers/soc/mediatek/Kconfig"
diff --git a/drivers/soc/Makefile b/drivers/soc/Makefile
index 2fcaff864584..a2027196d0fb 100644
--- a/drivers/soc/Makefile
+++ b/drivers/soc/Makefile
@@ -5,6 +5,7 @@
 obj-$(CONFIG_ARCH_ACTIONS)	+= actions/
 obj-$(CONFIG_ARCH_AT91)		+= atmel/
 obj-y				+= bcm/
+obj-$(CONFIG_ARCH_THUNDER)	+= cavium/
 obj-$(CONFIG_ARCH_DOVE)		+= dove/
 obj-$(CONFIG_MACH_DOVE)		+= dove/
 obj-y				+= fsl/
diff --git a/drivers/soc/cavium/Kconfig b/drivers/soc/cavium/Kconfig
new file mode 100644
index 000000000000..46ded89fb696
--- /dev/null
+++ b/drivers/soc/cavium/Kconfig
@@ -0,0 +1,6 @@
+#
+# Cavium ThunderX Soc drivers
+#
+config CAVIUM_LMC
+        depends on ARCH_THUNDER
+	def_tristate m
diff --git a/drivers/soc/cavium/Makefile b/drivers/soc/cavium/Makefile
new file mode 100644
index 000000000000..4ad0c7f923fa
--- /dev/null
+++ b/drivers/soc/cavium/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_CAVIUM_LMC) += cavium_lmc.o
diff --git a/drivers/soc/cavium/cavium_lmc.c b/drivers/soc/cavium/cavium_lmc.c
new file mode 100644
index 000000000000..87248e83c55b
--- /dev/null
+++ b/drivers/soc/cavium/cavium_lmc.c
@@ -0,0 +1,49 @@
+/*
+ * These PCI devices contain RAS functionality and PMU counters. To allow
+ * independent RAS and PMU drivers this driver registers for the PCI devices
+ * and multi-plexes probe and removal.
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright: Cavium, Inc. (C) 2017
+ *
+ */
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/soc/cavium/lmc.h>
+
+static int cvm_lmc_probe(struct pci_dev *pdev,
+			 const struct pci_device_id *ent)
+{
+	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
+		thunderx_edac_lmc_probe(pdev, ent);
+	return 0;
+}
+
+static void cvm_lmc_remove(struct pci_dev *pdev)
+{
+	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
+		thunderx_edac_lmc_remove(pdev);
+}
+
+static const struct pci_device_id cvm_lmc_pci_table[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, 0xa022) },
+	{ 0, },
+};
+
+MODULE_DEVICE_TABLE(pci, cvm_lmc_pci_table);
+
+static struct pci_driver cvm_lmc_pci_driver = {
+	.name     = "Cavium SOC memory controller",
+	.id_table = cvm_lmc_pci_table,
+	.probe    = cvm_lmc_probe,
+	.remove   = cvm_lmc_remove,
+};
+
+module_pci_driver(cvm_lmc_pci_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Cavium, Inc.");
+MODULE_DESCRIPTION("PCI driver for Cavium SOC memory controller");
diff --git a/include/linux/soc/cavium/lmc.h b/include/linux/soc/cavium/lmc.h
new file mode 100644
index 000000000000..336f467e154f
--- /dev/null
+++ b/include/linux/soc/cavium/lmc.h
@@ -0,0 +1,9 @@
+#ifndef _LMC_H
+#define _LMC_H
+
+#include <linux/pci.h>
+
+int thunderx_edac_lmc_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
+void thunderx_edac_lmc_remove(struct pci_dev *pdev);
+
+#endif
-- 
2.9.0.rc0.21.g7777322

[PATCH v10 4/7] edac,soc: thunderx: Add wrapper for EDAC OCX PCI device

[Jan Glauber]

Cavium SOCs contain an processor interconnect that is presented as a
PCI device. This PCI device will be used by an EDAC driver and
by a PMU driver.

To allow both subsystems to access the device a small wrapper is
introduced that multi-plexes PCI probe and removal calls of the
device to the EDAC driver.

Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
 drivers/edac/Kconfig            |  1 +
 drivers/edac/thunderx_edac.c    | 42 +++++++-------------------------------
 drivers/soc/cavium/Kconfig      |  4 ++++
 drivers/soc/cavium/Makefile     |  1 +
 drivers/soc/cavium/cavium_ocx.c | 45 +++++++++++++++++++++++++++++++++++++++++
 include/linux/soc/cavium/ocx.h  |  9 +++++++++
 6 files changed, 67 insertions(+), 35 deletions(-)
 create mode 100644 drivers/soc/cavium/cavium_ocx.c
 create mode 100644 include/linux/soc/cavium/ocx.h

diff --git a/drivers/edac/Kconfig b/drivers/edac/Kconfig
index 7330447c43d1..830421208374 100644
--- a/drivers/edac/Kconfig
+++ b/drivers/edac/Kconfig
@@ -373,6 +373,7 @@ config EDAC_THUNDERX
 	depends on PCI
 	depends on m
 	select CAVIUM_LMC
+	select CAVIUM_OCX
 	help
 	  Support for error detection and correction on the
 	  Cavium ThunderX memory controllers (LMC), Cache
diff --git a/drivers/edac/thunderx_edac.c b/drivers/edac/thunderx_edac.c
index a6a89bf0a457..581517488a80 100644
--- a/drivers/edac/thunderx_edac.c
+++ b/drivers/edac/thunderx_edac.c
@@ -22,6 +22,7 @@
 #include <linux/bitfield.h>
 #include <linux/circ_buf.h>
 #include <linux/soc/cavium/lmc.h>
+#include <linux/soc/cavium/ocx.h>
 
 #include <asm/page.h>
 
@@ -812,8 +813,6 @@ EXPORT_SYMBOL_GPL(thunderx_edac_lmc_remove);
 
 /*---------------------- OCX driver ---------------------------------*/
 
-#define PCI_DEVICE_ID_THUNDER_OCX 0xa013
-
 #define OCX_LINK_INTS		3
 #define OCX_INTS		(OCX_LINK_INTS + 1)
 #define OCX_RX_LANES		24
@@ -1309,11 +1308,6 @@ struct debugfs_entry *ocx_dfs_ents[] = {
 	&debugfs_com_int,
 };
 
-static const struct pci_device_id thunderx_ocx_pci_tbl[] = {
-	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_OCX) },
-	{ 0, },
-};
-
 static void thunderx_ocx_clearstats(struct thunderx_ocx *ocx)
 {
 	int lane, stat, cfg;
@@ -1329,8 +1323,8 @@ static void thunderx_ocx_clearstats(struct thunderx_ocx *ocx)
 	}
 }
 
-static int thunderx_ocx_probe(struct pci_dev *pdev,
-			      const struct pci_device_id *id)
+int thunderx_edac_ocx_probe(struct pci_dev *pdev,
+			    const struct pci_device_id *id)
 {
 	struct thunderx_ocx *ocx;
 	struct edac_device_ctl_info *edac_dev;
@@ -1459,8 +1453,9 @@ static int thunderx_ocx_probe(struct pci_dev *pdev,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(thunderx_edac_ocx_probe);
 
-static void thunderx_ocx_remove(struct pci_dev *pdev)
+void thunderx_edac_ocx_remove(struct pci_dev *pdev)
 {
 	struct edac_device_ctl_info *edac_dev = pci_get_drvdata(pdev);
 	struct thunderx_ocx *ocx = edac_dev->pvt_info;
@@ -1478,15 +1473,7 @@ static void thunderx_ocx_remove(struct pci_dev *pdev)
 	edac_device_del_device(&pdev->dev);
 	edac_device_free_ctl_info(edac_dev);
 }
-
-MODULE_DEVICE_TABLE(pci, thunderx_ocx_pci_tbl);
-
-static struct pci_driver thunderx_ocx_driver = {
-	.name     = "thunderx_ocx_edac",
-	.probe    = thunderx_ocx_probe,
-	.remove   = thunderx_ocx_remove,
-	.id_table = thunderx_ocx_pci_tbl,
-};
+EXPORT_SYMBOL_GPL(thunderx_edac_ocx_remove);
 
 /*---------------------- L2C driver ---------------------------------*/
 
@@ -2102,27 +2089,12 @@ static struct pci_driver thunderx_l2c_driver = {
 
 static int __init thunderx_edac_init(void)
 {
-	int rc = 0;
-
-	rc = pci_register_driver(&thunderx_ocx_driver);
-	if (rc)
-		return rc;
-
-	rc = pci_register_driver(&thunderx_l2c_driver);
-	if (rc)
-		goto err_ocx;
-
-	return rc;
-err_ocx:
-	pci_unregister_driver(&thunderx_ocx_driver);
-
-	return rc;
+	return pci_register_driver(&thunderx_l2c_driver);
 }
 
 static void __exit thunderx_edac_exit(void)
 {
 	pci_unregister_driver(&thunderx_l2c_driver);
-	pci_unregister_driver(&thunderx_ocx_driver);
 
 }
 
diff --git a/drivers/soc/cavium/Kconfig b/drivers/soc/cavium/Kconfig
index 46ded89fb696..fe56503d20f4 100644
--- a/drivers/soc/cavium/Kconfig
+++ b/drivers/soc/cavium/Kconfig
@@ -4,3 +4,7 @@
 config CAVIUM_LMC
         depends on ARCH_THUNDER
 	def_tristate m
+
+config CAVIUM_OCX
+	depends on ARCH_THUNDER
+	def_tristate m
diff --git a/drivers/soc/cavium/Makefile b/drivers/soc/cavium/Makefile
index 4ad0c7f923fa..bf7ba252ebb1 100644
--- a/drivers/soc/cavium/Makefile
+++ b/drivers/soc/cavium/Makefile
@@ -1 +1,2 @@
 obj-$(CONFIG_CAVIUM_LMC) += cavium_lmc.o
+obj-$(CONFIG_CAVIUM_OCX) += cavium_ocx.o
diff --git a/drivers/soc/cavium/cavium_ocx.c b/drivers/soc/cavium/cavium_ocx.c
new file mode 100644
index 000000000000..fa3341b0744f
--- /dev/null
+++ b/drivers/soc/cavium/cavium_ocx.c
@@ -0,0 +1,45 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright: Cavium, Inc. (C) 2017
+ *
+ */
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/soc/cavium/ocx.h>
+
+static int cvm_ocx_probe(struct pci_dev *pdev,
+			 const struct pci_device_id *ent)
+{
+	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
+		thunderx_edac_ocx_probe(pdev, ent);
+	return 0;
+}
+
+static void cvm_ocx_remove(struct pci_dev *pdev)
+{
+	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
+		thunderx_edac_ocx_remove(pdev);
+}
+
+static const struct pci_device_id cvm_ocx_pci_table[] = {
+	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, 0xa013) },
+	{ 0, },
+};
+
+MODULE_DEVICE_TABLE(pci, cvm_ocx_pci_table);
+
+static struct pci_driver cvm_ocx_pci_driver = {
+	.name     = "Cavium ThunderX interconnect",
+	.id_table = cvm_ocx_pci_table,
+	.probe    = cvm_ocx_probe,
+	.remove   = cvm_ocx_remove,
+};
+
+module_pci_driver(cvm_ocx_pci_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Cavium, Inc.");
+MODULE_DESCRIPTION("PCI driver for Cavium ThunderX interconnect");
diff --git a/include/linux/soc/cavium/ocx.h b/include/linux/soc/cavium/ocx.h
new file mode 100644
index 000000000000..29f55b3d3171
--- /dev/null
+++ b/include/linux/soc/cavium/ocx.h
@@ -0,0 +1,9 @@
+#ifndef _OCX_H
+#define _OCX_H
+
+#include <linux/pci.h>
+
+int thunderx_edac_ocx_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
+void thunderx_edac_ocx_remove(struct pci_dev *pdev);
+
+#endif
-- 
2.9.0.rc0.21.g7777322

[PATCH v10 5/7] perf: cavium: Support memory controller PMU counters

[Jan Glauber]

Add support for the PMU counters on Cavium SOC memory controllers.

This patch also adds generic functions to allow supporting more
devices with PMU counters.

Properties of the LMC PMU counters:
- not stoppable
- fixed purpose
- read-only
- one PCI device per memory controller

Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
 drivers/perf/Kconfig            |   8 +
 drivers/perf/Makefile           |   1 +
 drivers/perf/cavium_pmu.c       | 430 ++++++++++++++++++++++++++++++++++++++++
 drivers/soc/cavium/cavium_lmc.c |   4 +
 include/linux/cpuhotplug.h      |   1 +
 include/linux/soc/cavium/lmc.h  |   3 +
 6 files changed, 447 insertions(+)
 create mode 100644 drivers/perf/cavium_pmu.c

diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
index e5197ffb7422..a787562c5432 100644
--- a/drivers/perf/Kconfig
+++ b/drivers/perf/Kconfig
@@ -43,4 +43,12 @@ config XGENE_PMU
         help
           Say y if you want to use APM X-Gene SoC performance monitors.
 
+config CAVIUM_PMU_LMC
+	tristate "Cavium SOC memory controller PMU"
+	depends on ARCH_THUNDER && m
+	select CAVIUM_LMC
+	help
+	  Provides PMU counters for the memory controller on
+	  Cavium ThunderX or OcteonTX SOCs.
+
 endmenu
diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile
index 6420bd4394d5..cd616785047f 100644
--- a/drivers/perf/Makefile
+++ b/drivers/perf/Makefile
@@ -1,5 +1,6 @@
 obj-$(CONFIG_ARM_PMU) += arm_pmu.o arm_pmu_platform.o
 obj-$(CONFIG_ARM_PMU_ACPI) += arm_pmu_acpi.o
+obj-$(CONFIG_CAVIUM_PMU_LMC) += cavium_pmu.o
 obj-$(CONFIG_QCOM_L2_PMU)	+= qcom_l2_pmu.o
 obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o
 obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
diff --git a/drivers/perf/cavium_pmu.c b/drivers/perf/cavium_pmu.c
new file mode 100644
index 000000000000..45d1766db474
--- /dev/null
+++ b/drivers/perf/cavium_pmu.c
@@ -0,0 +1,430 @@
+/*
+ * Cavium ARM SOC "uncore" PMU counters
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright Cavium, Inc. 2017
+ * Author(s): Jan Glauber <jan.glauber@cavium.com>
+ *
+ */
+#include <linux/cpuhotplug.h>
+#include <linux/cpumask.h>
+#include <linux/io.h>
+#include <linux/export.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/perf_event.h>
+#include <linux/slab.h>
+#include <linux/soc/cavium/lmc.h>
+
+enum cvm_pmu_type {
+	CVM_PMU_LMC,
+};
+
+/* maximum number of parallel hardware counters for all pmu types */
+#define CVM_PMU_MAX_COUNTERS 64
+
+/* generic struct to cover the different pmu types */
+struct cvm_pmu_dev {
+	struct pmu pmu;
+	bool (*event_valid)(u64);
+	void __iomem *map;
+	struct pci_dev *pdev;
+	int num_counters;
+	struct perf_event *events[CVM_PMU_MAX_COUNTERS];
+	struct list_head entry;
+	struct hlist_node cpuhp_node;
+	cpumask_t active_mask;
+};
+
+static struct list_head cvm_pmu_lmcs;
+static struct list_head cvm_pmu_tlks;
+
+/*
+ * Common Cavium PMU stuff
+ *
+ * Shared properties of the different PMU types:
+ * - all counters are 64 bit long
+ * - there are no overflow interrupts
+ * - all devices with PMU counters appear as PCI devices
+ *
+ * Counter control, access and device association depends on the
+ * PMU type.
+ */
+
+#define to_pmu_dev(x) container_of((x), struct cvm_pmu_dev, pmu)
+
+static int cvm_pmu_event_init(struct perf_event *event)
+{
+	struct hw_perf_event *hwc = &event->hw;
+	struct cvm_pmu_dev *pmu_dev;
+	struct perf_event *sibling;
+
+	if (event->attr.type != event->pmu->type)
+		return -ENOENT;
+
+	/* we do not support sampling */
+	if (is_sampling_event(event))
+		return -EINVAL;
+
+	/* PMU counters do not support any these bits */
+	if (event->attr.exclude_user	||
+	    event->attr.exclude_kernel	||
+	    event->attr.exclude_host	||
+	    event->attr.exclude_guest	||
+	    event->attr.exclude_hv	||
+	    event->attr.exclude_idle)
+		return -EINVAL;
+
+	pmu_dev = to_pmu_dev(event->pmu);
+	if (!pmu_dev->event_valid(event->attr.config))
+		return -EINVAL;
+
+	/*
+	 * Forbid groups containing mixed PMUs, software events are acceptable.
+	 */
+	if (event->group_leader->pmu != event->pmu &&
+	    !is_software_event(event->group_leader))
+		return -EINVAL;
+
+	list_for_each_entry(sibling, &event->group_leader->sibling_list,
+			    group_entry)
+		if (sibling->pmu != event->pmu &&
+		    !is_software_event(sibling))
+			return -EINVAL;
+
+	hwc->config = event->attr.config;
+	hwc->idx = -1;
+	return 0;
+}
+
+static void cvm_pmu_read(struct perf_event *event)
+{
+	struct cvm_pmu_dev *pmu_dev = to_pmu_dev(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+	u64 prev, delta, new;
+
+again:
+	prev = local64_read(&hwc->prev_count);
+	new = readq(hwc->event_base + pmu_dev->map);
+
+	if (local64_cmpxchg(&hwc->prev_count, prev, new) != prev)
+		goto again;
+
+	delta = new - prev;
+	local64_add(delta, &event->count);
+}
+
+static void cvm_pmu_start(struct perf_event *event, int flags)
+{
+	struct cvm_pmu_dev *pmu_dev = to_pmu_dev(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+	u64 new;
+
+	if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
+		return;
+
+	WARN_ON_ONCE(!(hwc->state & PERF_HES_UPTODATE));
+	hwc->state = 0;
+
+	/* update prev_count always in order support unstoppable counters */
+	new = readq(hwc->event_base + pmu_dev->map);
+	local64_set(&hwc->prev_count, new);
+
+	perf_event_update_userpage(event);
+}
+
+static void cvm_pmu_stop(struct perf_event *event, int flags)
+{
+	struct hw_perf_event *hwc = &event->hw;
+
+	WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
+	hwc->state |= PERF_HES_STOPPED;
+
+	if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) {
+		cvm_pmu_read(event);
+		hwc->state |= PERF_HES_UPTODATE;
+	}
+}
+
+static int cvm_pmu_add(struct perf_event *event, int flags, u64 event_base)
+{
+	struct cvm_pmu_dev *pmu_dev = to_pmu_dev(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+	int i;
+
+	for (i = 0; i < pmu_dev->num_counters; i++)
+		if (!cmpxchg(&pmu_dev->events[i], NULL, event)) {
+			hwc->idx = i;
+			break;
+		}
+
+	if (hwc->idx == -1)
+		return -EBUSY;
+
+	hwc->event_base = event_base;
+	hwc->state = PERF_HES_UPTODATE | PERF_HES_STOPPED;
+
+	if (flags & PERF_EF_START)
+		pmu_dev->pmu.start(event, PERF_EF_RELOAD);
+
+	return 0;
+}
+
+static void cvm_pmu_del(struct perf_event *event, int flags)
+{
+	struct cvm_pmu_dev *pmu_dev = to_pmu_dev(event->pmu);
+	struct hw_perf_event *hwc = &event->hw;
+
+	event->pmu->stop(event, PERF_EF_UPDATE);
+
+	pmu_dev->events[hwc->idx] = NULL;
+	hwc->idx = -1;
+	perf_event_update_userpage(event);
+}
+
+static ssize_t cvm_pmu_event_sysfs_show(struct device *dev,
+					struct device_attribute *attr,
+					char *page)
+{
+	struct perf_pmu_events_attr *pmu_attr =
+		container_of(attr, struct perf_pmu_events_attr, attr);
+
+	if (pmu_attr->event_str)
+		return sprintf(page, "%s", pmu_attr->event_str);
+
+	return 0;
+}
+
+/*
+ * The pmu events are independent from CPUs. Provide a cpumask
+ * nevertheless to prevent perf from adding the event per-cpu and just
+ * set the mask to one online CPU. Use the same cpumask for all "uncore"
+ * devices.
+ *
+ * There is a performance penalty for accessing a device from a CPU on
+ * another socket, but we do not care.
+ */
+static int cvm_pmu_offline_cpu(unsigned int old_cpu, struct hlist_node *node)
+{
+	struct cvm_pmu_dev *pmu_dev;
+	int new_cpu;
+
+	pmu_dev = hlist_entry_safe(node, struct cvm_pmu_dev, cpuhp_node);
+	if (!cpumask_test_and_clear_cpu(old_cpu, &pmu_dev->active_mask))
+		return 0;
+
+	new_cpu = cpumask_any_but(cpu_online_mask, old_cpu);
+	if (new_cpu >= nr_cpu_ids)
+		return 0;
+
+	perf_pmu_migrate_context(&pmu_dev->pmu, old_cpu, new_cpu);
+	cpumask_set_cpu(new_cpu, &pmu_dev->active_mask);
+
+	return 0;
+}
+
+static ssize_t cvm_pmu_attr_show_cpumask(struct device *dev,
+					 struct device_attribute *attr,
+					 char *buf)
+{
+	struct pmu *pmu = dev_get_drvdata(dev);
+	struct cvm_pmu_dev *pmu_dev = container_of(pmu, struct cvm_pmu_dev, pmu);
+
+	return cpumap_print_to_pagebuf(true, buf, &pmu_dev->active_mask);
+}
+
+static DEVICE_ATTR(cpumask, S_IRUGO, cvm_pmu_attr_show_cpumask, NULL);
+
+static struct attribute *cvm_pmu_attrs[] = {
+	&dev_attr_cpumask.attr,
+	NULL,
+};
+
+static struct attribute_group cvm_pmu_attr_group = {
+	.attrs = cvm_pmu_attrs,
+};
+
+/*
+ * LMC (memory controller) counters:
+ * - not stoppable, always on, read-only
+ * - one PCI device per memory controller
+ */
+#define LMC_CONFIG_OFFSET		0x188
+#define LMC_CONFIG_RESET_BIT		BIT(17)
+
+/* LMC events */
+#define LMC_EVENT_IFB_CNT		0x1d0
+#define LMC_EVENT_OPS_CNT		0x1d8
+#define LMC_EVENT_DCLK_CNT		0x1e0
+#define LMC_EVENT_BANK_CONFLICT1	0x360
+#define LMC_EVENT_BANK_CONFLICT2	0x368
+
+#define CVM_PMU_LMC_EVENT_ATTR(_name, _id)						\
+	&((struct perf_pmu_events_attr[]) {						\
+		{									\
+			__ATTR(_name, S_IRUGO, cvm_pmu_event_sysfs_show, NULL),		\
+			_id,								\
+			"lmc_event=" __stringify(_id),					\
+		}									\
+	})[0].attr.attr
+
+static int cvm_pmu_lmc_add(struct perf_event *event, int flags)
+{
+	struct hw_perf_event *hwc = &event->hw;
+
+	return cvm_pmu_add(event, flags, hwc->config);
+}
+
+PMU_FORMAT_ATTR(lmc_event, "config:0-9");
+
+static struct attribute *cvm_pmu_lmc_format_attr[] = {
+	&format_attr_lmc_event.attr,
+	NULL,
+};
+
+static struct attribute_group cvm_pmu_lmc_format_group = {
+	.name = "format",
+	.attrs = cvm_pmu_lmc_format_attr,
+};
+
+static struct attribute *cvm_pmu_lmc_events_attr[] = {
+	CVM_PMU_LMC_EVENT_ATTR(ifb_cnt,		0x1d0),
+	CVM_PMU_LMC_EVENT_ATTR(ops_cnt,		0x1d8),
+	CVM_PMU_LMC_EVENT_ATTR(dclk_cnt,	0x1e0),
+	CVM_PMU_LMC_EVENT_ATTR(bank_conflict1,	0x360),
+	CVM_PMU_LMC_EVENT_ATTR(bank_conflict2,	0x368),
+	NULL,
+};
+
+static struct attribute_group cvm_pmu_lmc_events_group = {
+	.name = "events",
+	.attrs = cvm_pmu_lmc_events_attr,
+};
+
+static const struct attribute_group *cvm_pmu_lmc_attr_groups[] = {
+	&cvm_pmu_attr_group,
+	&cvm_pmu_lmc_format_group,
+	&cvm_pmu_lmc_events_group,
+	NULL,
+};
+
+static bool cvm_pmu_lmc_event_valid(u64 config)
+{
+	struct perf_pmu_events_attr *attr;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(cvm_pmu_lmc_events_attr) -1; i++) {
+		attr = (struct perf_pmu_events_attr *)cvm_pmu_lmc_events_attr[i];
+		if (attr->id == config)
+			return true;
+	}
+	return false;
+}
+
+int cvm_lmc_pmu_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+	struct cvm_pmu_dev *next, *lmc;
+	int nr = 0, ret = -ENOMEM;
+	char *name;
+
+	lmc = devm_kzalloc(&pdev->dev, sizeof(*lmc), GFP_KERNEL);
+	if (!lmc)
+		return -ENOMEM;
+
+	lmc->map = devm_ioremap(&pdev->dev, pci_resource_start(pdev, 0),
+				pci_resource_len(pdev, 0));
+	if (!lmc->map)
+		return -EINVAL;
+
+	list_for_each_entry(next, &cvm_pmu_lmcs, entry)
+		nr++;
+	name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "lmc%d", nr);
+	if (!name)
+		return -ENOMEM;
+
+	lmc->pdev = pdev;
+	lmc->num_counters = ARRAY_SIZE(cvm_pmu_lmc_events_attr) - 1;
+	lmc->pmu = (struct pmu) {
+		.task_ctx_nr    = perf_invalid_context,
+		.name		= name,
+		.event_init	= cvm_pmu_event_init,
+		.add		= cvm_pmu_lmc_add,
+		.del		= cvm_pmu_del,
+		.start		= cvm_pmu_start,
+		.stop		= cvm_pmu_stop,
+		.read		= cvm_pmu_read,
+		.attr_groups	= cvm_pmu_lmc_attr_groups,
+	};
+
+	cpuhp_state_add_instance_nocalls(CPUHP_AP_PERF_ARM_CVM_ONLINE,
+					 &lmc->cpuhp_node);
+
+	/*
+	 * perf PMU is CPU dependent so pick a random CPU and migrate away
+	 * if it goes offline.
+	 */
+	cpumask_set_cpu(smp_processor_id(), &lmc->active_mask);
+
+	list_add(&lmc->entry, &cvm_pmu_lmcs);
+	lmc->event_valid = cvm_pmu_lmc_event_valid;
+
+	ret = perf_pmu_register(&lmc->pmu, name, -1);
+	if (ret)
+		goto fail_pmu;
+
+	dev_info(&pdev->dev, "Enabled %s PMU with %d counters\n",
+		 name, lmc->num_counters);
+	return 0;
+
+fail_pmu:
+	list_del(&lmc->entry);
+	cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_CVM_ONLINE,
+				    &lmc->cpuhp_node);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(cvm_lmc_pmu_probe);
+
+void cvm_lmc_pmu_remove(struct pci_dev *pdev)
+{
+	struct list_head *l, *tmp;
+	struct cvm_pmu_dev *lmc;
+
+	list_for_each_safe(l, tmp, &cvm_pmu_lmcs) {
+		lmc = list_entry(l, struct cvm_pmu_dev, entry);
+		if (pdev != lmc->pdev)
+			continue;
+
+		perf_pmu_unregister(&lmc->pmu);
+		cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_CVM_ONLINE,
+					    &lmc->cpuhp_node);
+		list_del(&lmc->entry);
+	}
+}
+EXPORT_SYMBOL_GPL(cvm_lmc_pmu_remove);
+
+static int __init cvm_pmu_init(void)
+{
+	INIT_LIST_HEAD(&cvm_pmu_lmcs);
+	INIT_LIST_HEAD(&cvm_pmu_tlks);
+
+	return cpuhp_setup_state_multi(CPUHP_AP_PERF_ARM_CVM_ONLINE,
+				       "perf/arm/cvm:online", NULL,
+				       cvm_pmu_offline_cpu);
+}
+
+static void __exit cvm_pmu_exit(void)
+{
+	cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_CVM_ONLINE);
+}
+
+module_init(cvm_pmu_init);
+module_exit(cvm_pmu_exit);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Cavium, Inc.");
+MODULE_DESCRIPTION("PMU Driver for Cavium ThunderX SOC");
diff --git a/drivers/soc/cavium/cavium_lmc.c b/drivers/soc/cavium/cavium_lmc.c
index 87248e83c55b..d21d59c24321 100644
--- a/drivers/soc/cavium/cavium_lmc.c
+++ b/drivers/soc/cavium/cavium_lmc.c
@@ -17,6 +17,8 @@
 static int cvm_lmc_probe(struct pci_dev *pdev,
 			 const struct pci_device_id *ent)
 {
+	if (IS_ENABLED(CONFIG_CAVIUM_PMU_LMC))
+		cvm_lmc_pmu_probe(pdev, ent);
 	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
 		thunderx_edac_lmc_probe(pdev, ent);
 	return 0;
@@ -24,6 +26,8 @@ static int cvm_lmc_probe(struct pci_dev *pdev,
 
 static void cvm_lmc_remove(struct pci_dev *pdev)
 {
+	if (IS_ENABLED(CONFIG_CAVIUM_PMU_LMC))
+		cvm_lmc_pmu_remove(pdev);
 	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
 		thunderx_edac_lmc_remove(pdev);
 }
diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index f24bfb2b9a2d..3362c158fe5c 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -134,6 +134,7 @@ enum cpuhp_state {
 	CPUHP_AP_PERF_S390_SF_ONLINE,
 	CPUHP_AP_PERF_ARM_CCI_ONLINE,
 	CPUHP_AP_PERF_ARM_CCN_ONLINE,
+	CPUHP_AP_PERF_ARM_CVM_ONLINE,
 	CPUHP_AP_PERF_ARM_L2X0_ONLINE,
 	CPUHP_AP_PERF_ARM_QCOM_L2_ONLINE,
 	CPUHP_AP_PERF_ARM_QCOM_L3_ONLINE,
diff --git a/include/linux/soc/cavium/lmc.h b/include/linux/soc/cavium/lmc.h
index 336f467e154f..e5ad6507fd36 100644
--- a/include/linux/soc/cavium/lmc.h
+++ b/include/linux/soc/cavium/lmc.h
@@ -3,6 +3,9 @@
 
 #include <linux/pci.h>
 
+int cvm_lmc_pmu_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
+void cvm_lmc_pmu_remove(struct pci_dev *pdev);
+
 int thunderx_edac_lmc_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
 void thunderx_edac_lmc_remove(struct pci_dev *pdev);
 
-- 
2.9.0.rc0.21.g7777322

[PATCH v10 6/7] perf: cavium: Support transmit-link PMU counters

[Jan Glauber]

Add support for the transmit-link (OCX TLK) PMU counters found
on Caviums SOCs with a processor interconnect.

Properties of the OCX TLK counters:
- per-unit control
- fixed purpose
- writable
- one PCI device with multiple TLK units

Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
 drivers/perf/Kconfig            |   7 ++
 drivers/perf/cavium_pmu.c       | 230 ++++++++++++++++++++++++++++++++++++++++
 drivers/soc/cavium/Kconfig      |   4 +
 drivers/soc/cavium/cavium_ocx.c |   4 +
 include/linux/soc/cavium/ocx.h  |   3 +
 5 files changed, 248 insertions(+)

diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
index a787562c5432..efb2ace649c1 100644
--- a/drivers/perf/Kconfig
+++ b/drivers/perf/Kconfig
@@ -51,4 +51,11 @@ config CAVIUM_PMU_LMC
 	  Provides PMU counters for the memory controller on
 	  Cavium ThunderX or OcteonTX SOCs.
 
+config CAVIUM_PMU_OCX_TLK
+	tristate "Cavium ThunderX interconnect PMU"
+	depends on ARCH_THUNDER && m
+	select CAVIUM_OCX
+	help
+	  Provides PMU counters for the processor interconnect on
+	  Cavium ThunderX processors.
 endmenu
diff --git a/drivers/perf/cavium_pmu.c b/drivers/perf/cavium_pmu.c
index 45d1766db474..1a112b0aeaa6 100644
--- a/drivers/perf/cavium_pmu.c
+++ b/drivers/perf/cavium_pmu.c
@@ -20,9 +20,11 @@
 #include <linux/perf_event.h>
 #include <linux/slab.h>
 #include <linux/soc/cavium/lmc.h>
+#include <linux/soc/cavium/ocx.h>
 
 enum cvm_pmu_type {
 	CVM_PMU_LMC,
+	CVM_PMU_TLK,
 };
 
 /* maximum number of parallel hardware counters for all pmu types */
@@ -407,6 +409,234 @@ void cvm_lmc_pmu_remove(struct pci_dev *pdev)
 }
 EXPORT_SYMBOL_GPL(cvm_lmc_pmu_remove);
 
+/*
+ * CCPI interface controller (OCX) Transmit link (TLK) counters:
+ * - per-unit control
+ * - writable
+ * - one PCI device with multiple TLK units
+ */
+
+#define TLK_NR_UNITS			3
+#define TLK_UNIT_OFFSET			0x2000
+#define TLK_UNIT_LEN			0x7ff
+#define TLK_START_ADDR			0x10000
+#define TLK_STAT_CTL_OFFSET		0x40
+#define TLK_STAT_OFFSET			0x400
+
+#define TLK_STAT_ENABLE_BIT		BIT(0)
+#define TLK_STAT_RESET_BIT		BIT(1)
+
+#define CVM_PMU_TLK_EVENT_ATTR(_name, _id)						\
+	&((struct perf_pmu_events_attr[]) {						\
+		{									\
+			__ATTR(_name, S_IRUGO, cvm_pmu_event_sysfs_show, NULL),		\
+			_id,								\
+			"tlk_event=" __stringify(_id),					\
+		}									\
+	})[0].attr.attr
+
+static void cvm_pmu_tlk_enable_pmu(struct pmu *pmu)
+{
+	struct cvm_pmu_dev *pmu_dev = container_of(pmu, struct cvm_pmu_dev, pmu);
+
+	/* enable all counters */
+	writeb(TLK_STAT_ENABLE_BIT, pmu_dev->map + TLK_STAT_CTL_OFFSET);
+}
+
+static void cvm_pmu_tlk_disable_pmu(struct pmu *pmu)
+{
+	struct cvm_pmu_dev *pmu_dev = container_of(pmu, struct cvm_pmu_dev, pmu);
+
+	/* disable all counters */
+	writeb(0, pmu_dev->map + TLK_STAT_CTL_OFFSET);
+}
+
+static int cvm_pmu_tlk_add(struct perf_event *event, int flags)
+{
+	struct hw_perf_event *hwc = &event->hw;
+
+	return cvm_pmu_add(event, flags, TLK_STAT_OFFSET + hwc->config * 8);
+}
+
+PMU_FORMAT_ATTR(tlk_event, "config:0-5");
+
+static struct attribute *cvm_pmu_tlk_format_attr[] = {
+	&format_attr_tlk_event.attr,
+	NULL,
+};
+
+static struct attribute_group cvm_pmu_tlk_format_group = {
+	.name = "format",
+	.attrs = cvm_pmu_tlk_format_attr,
+};
+
+static struct attribute *cvm_pmu_tlk_events_attr[] = {
+	CVM_PMU_TLK_EVENT_ATTR(idle_cnt,	0x00),
+	CVM_PMU_TLK_EVENT_ATTR(data_cnt,	0x01),
+	CVM_PMU_TLK_EVENT_ATTR(sync_cnt,	0x02),
+	CVM_PMU_TLK_EVENT_ATTR(retry_cnt,	0x03),
+	CVM_PMU_TLK_EVENT_ATTR(err_cnt,		0x04),
+	CVM_PMU_TLK_EVENT_ATTR(mat0_cnt,	0x08),
+	CVM_PMU_TLK_EVENT_ATTR(mat1_cnt,	0x09),
+	CVM_PMU_TLK_EVENT_ATTR(mat2_cnt,	0x0a),
+	CVM_PMU_TLK_EVENT_ATTR(mat3_cnt,	0x0b),
+	CVM_PMU_TLK_EVENT_ATTR(vc0_cmd,		0x10),
+	CVM_PMU_TLK_EVENT_ATTR(vc1_cmd,		0x11),
+	CVM_PMU_TLK_EVENT_ATTR(vc2_cmd,		0x12),
+	CVM_PMU_TLK_EVENT_ATTR(vc3_cmd,		0x13),
+	CVM_PMU_TLK_EVENT_ATTR(vc4_cmd,		0x14),
+	CVM_PMU_TLK_EVENT_ATTR(vc5_cmd,		0x15),
+	CVM_PMU_TLK_EVENT_ATTR(vc0_pkt,		0x20),
+	CVM_PMU_TLK_EVENT_ATTR(vc1_pkt,		0x21),
+	CVM_PMU_TLK_EVENT_ATTR(vc2_pkt,		0x22),
+	CVM_PMU_TLK_EVENT_ATTR(vc3_pkt,		0x23),
+	CVM_PMU_TLK_EVENT_ATTR(vc4_pkt,		0x24),
+	CVM_PMU_TLK_EVENT_ATTR(vc5_pkt,		0x25),
+	CVM_PMU_TLK_EVENT_ATTR(vc6_pkt,		0x26),
+	CVM_PMU_TLK_EVENT_ATTR(vc7_pkt,		0x27),
+	CVM_PMU_TLK_EVENT_ATTR(vc8_pkt,		0x28),
+	CVM_PMU_TLK_EVENT_ATTR(vc9_pkt,		0x29),
+	CVM_PMU_TLK_EVENT_ATTR(vc10_pkt,	0x2a),
+	CVM_PMU_TLK_EVENT_ATTR(vc11_pkt,	0x2b),
+	CVM_PMU_TLK_EVENT_ATTR(vc12_pkt,	0x2c),
+	CVM_PMU_TLK_EVENT_ATTR(vc13_pkt,	0x2d),
+	CVM_PMU_TLK_EVENT_ATTR(vc0_con,		0x30),
+	CVM_PMU_TLK_EVENT_ATTR(vc1_con,		0x31),
+	CVM_PMU_TLK_EVENT_ATTR(vc2_con,		0x32),
+	CVM_PMU_TLK_EVENT_ATTR(vc3_con,		0x33),
+	CVM_PMU_TLK_EVENT_ATTR(vc4_con,		0x34),
+	CVM_PMU_TLK_EVENT_ATTR(vc5_con,		0x35),
+	CVM_PMU_TLK_EVENT_ATTR(vc6_con,		0x36),
+	CVM_PMU_TLK_EVENT_ATTR(vc7_con,		0x37),
+	CVM_PMU_TLK_EVENT_ATTR(vc8_con,		0x38),
+	CVM_PMU_TLK_EVENT_ATTR(vc9_con,		0x39),
+	CVM_PMU_TLK_EVENT_ATTR(vc10_con,	0x3a),
+	CVM_PMU_TLK_EVENT_ATTR(vc11_con,	0x3b),
+	CVM_PMU_TLK_EVENT_ATTR(vc12_con,	0x3c),
+	CVM_PMU_TLK_EVENT_ATTR(vc13_con,	0x3d),
+	NULL,
+};
+
+static struct attribute_group cvm_pmu_tlk_events_group = {
+	.name = "events",
+	.attrs = cvm_pmu_tlk_events_attr,
+};
+
+static const struct attribute_group *cvm_pmu_tlk_attr_groups[] = {
+	&cvm_pmu_attr_group,
+	&cvm_pmu_tlk_format_group,
+	&cvm_pmu_tlk_events_group,
+	NULL,
+};
+
+static bool cvm_pmu_tlk_event_valid(u64 config)
+{
+	struct perf_pmu_events_attr *attr;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(cvm_pmu_tlk_events_attr) - 1; i++) {
+		attr = (struct perf_pmu_events_attr *)cvm_pmu_tlk_events_attr[i];
+		if (attr->id == config)
+			return true;
+	}
+	return false;
+}
+
+static int cvm_pmu_tlk_probe_unit(struct pci_dev *pdev, int nr)
+{
+	struct cvm_pmu_dev *tlk;
+	int ret = -ENOMEM;
+	char *name;
+
+	tlk = devm_kzalloc(&pdev->dev, sizeof(*tlk), GFP_KERNEL);
+	if (!tlk)
+		return -ENOMEM;
+
+	tlk->map = devm_ioremap(&pdev->dev,
+				pci_resource_start(pdev, 0) + TLK_START_ADDR +
+				nr * TLK_UNIT_OFFSET, TLK_UNIT_LEN);
+	if (!tlk->map)
+		return -EINVAL;
+
+	name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "ocx_tlk%d", nr);
+	if (!name)
+		return -ENOMEM;
+
+	tlk->pdev = pdev;
+	tlk->num_counters = ARRAY_SIZE(cvm_pmu_tlk_events_attr) - 1;
+	tlk->pmu = (struct pmu) {
+		.task_ctx_nr    = perf_invalid_context,
+		.name		= name,
+		.pmu_enable	= cvm_pmu_tlk_enable_pmu,
+		.pmu_disable	= cvm_pmu_tlk_disable_pmu,
+		.event_init	= cvm_pmu_event_init,
+		.add		= cvm_pmu_tlk_add,
+		.del		= cvm_pmu_del,
+		.start		= cvm_pmu_start,
+		.stop		= cvm_pmu_stop,
+		.read		= cvm_pmu_read,
+		.attr_groups	= cvm_pmu_tlk_attr_groups,
+	};
+
+	cpuhp_state_add_instance_nocalls(CPUHP_AP_PERF_ARM_CVM_ONLINE,
+					 &tlk->cpuhp_node);
+
+	/*
+	 * perf PMU is CPU dependent so pick a random CPU and migrate away
+	 * if it goes offline.
+	 */
+	cpumask_set_cpu(smp_processor_id(), &tlk->active_mask);
+
+	list_add(&tlk->entry, &cvm_pmu_tlks);
+	tlk->event_valid = cvm_pmu_tlk_event_valid;
+
+	ret = perf_pmu_register(&tlk->pmu, name, -1);
+	if (ret)
+		goto fail_pmu;
+
+	dev_info(&pdev->dev, "Enabled %s PMU with %d counters\n",
+		 name, tlk->num_counters);
+	return 0;
+
+fail_pmu:
+	list_del(&tlk->entry);
+	cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_CVM_ONLINE,
+				    &tlk->cpuhp_node);
+	return ret;
+}
+
+int cvm_ocx_tlk_pmu_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+	int rc, i;
+
+	for (i = 0; i < TLK_NR_UNITS; i++) {
+		rc = cvm_pmu_tlk_probe_unit(pdev, i);
+		if (rc)
+			return rc;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cvm_ocx_tlk_pmu_probe);
+
+void cvm_ocx_tlk_pmu_remove(struct pci_dev *pdev)
+{
+	struct list_head *l, *tmp;
+	struct cvm_pmu_dev *tlk;
+
+	list_for_each_safe(l, tmp, &cvm_pmu_tlks) {
+		tlk = list_entry(l, struct cvm_pmu_dev, entry);
+
+		if (pdev != tlk->pdev)
+			continue;
+
+		perf_pmu_unregister(&tlk->pmu);
+		cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_CVM_ONLINE,
+					    &tlk->cpuhp_node);
+		list_del(&tlk->entry);
+	}
+}
+EXPORT_SYMBOL_GPL(cvm_ocx_tlk_pmu_remove);
+
 static int __init cvm_pmu_init(void)
 {
 	INIT_LIST_HEAD(&cvm_pmu_lmcs);
diff --git a/drivers/soc/cavium/Kconfig b/drivers/soc/cavium/Kconfig
index fe56503d20f4..2c7406872b18 100644
--- a/drivers/soc/cavium/Kconfig
+++ b/drivers/soc/cavium/Kconfig
@@ -8,3 +8,7 @@ config CAVIUM_LMC
 config CAVIUM_OCX
 	depends on ARCH_THUNDER
 	def_tristate m
+
+config CAVIUM_OCX
+        depends on ARCH_THUNDER
+	def_tristate m
diff --git a/drivers/soc/cavium/cavium_ocx.c b/drivers/soc/cavium/cavium_ocx.c
index fa3341b0744f..de1ad146c0dd 100644
--- a/drivers/soc/cavium/cavium_ocx.c
+++ b/drivers/soc/cavium/cavium_ocx.c
@@ -13,6 +13,8 @@
 static int cvm_ocx_probe(struct pci_dev *pdev,
 			 const struct pci_device_id *ent)
 {
+	if (IS_ENABLED(CONFIG_CAVIUM_PMU_OCX_TLK))
+		cvm_ocx_tlk_pmu_probe(pdev, ent);
 	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
 		thunderx_edac_ocx_probe(pdev, ent);
 	return 0;
@@ -20,6 +22,8 @@ static int cvm_ocx_probe(struct pci_dev *pdev,
 
 static void cvm_ocx_remove(struct pci_dev *pdev)
 {
+	if (IS_ENABLED(CONFIG_CAVIUM_PMU_OCX_TLK))
+		cvm_ocx_tlk_pmu_remove(pdev);
 	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
 		thunderx_edac_ocx_remove(pdev);
 }
diff --git a/include/linux/soc/cavium/ocx.h b/include/linux/soc/cavium/ocx.h
index 29f55b3d3171..f7b2caada671 100644
--- a/include/linux/soc/cavium/ocx.h
+++ b/include/linux/soc/cavium/ocx.h
@@ -3,6 +3,9 @@
 
 #include <linux/pci.h>
 
+int cvm_ocx_tlk_pmu_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
+void cvm_ocx_tlk_pmu_remove(struct pci_dev *pdev);
+
 int thunderx_edac_ocx_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
 void thunderx_edac_ocx_remove(struct pci_dev *pdev);
 
-- 
2.9.0.rc0.21.g7777322

[PATCH v10 7/7] perf: cavium: Add Documentation

[Jan Glauber]

Document Cavium SoC PMUs.

Signed-off-by: Jan Glauber <jglauber@cavium.com>
---
 Documentation/perf/cavium-pmu.txt | 75 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
 create mode 100644 Documentation/perf/cavium-pmu.txt

diff --git a/Documentation/perf/cavium-pmu.txt b/Documentation/perf/cavium-pmu.txt
new file mode 100644
index 000000000000..6fbf824ee4fd
--- /dev/null
+++ b/Documentation/perf/cavium-pmu.txt
@@ -0,0 +1,75 @@
+Cavium ThunderX and OcteonTx Performance Monitoring Unit (PMU)
+==============================================================
+
+Cavium SoCs contain various system devices such as L2 caches, processor
+interconnect and memory controllers. Unfortunately the PMU counters
+are not following a common design so each device has a slightly different
+approach how to control and use the PMU counters.
+
+Common properties of all devices carrying PMU counters:
+- The devices are PCI devices and the counters are embedded somewhere
+  in the PCI register space.
+- All counters are 64 bit wide.
+- There are no overflow interrupts (unnecessary because of the 64 bit wide
+  counters).
+
+Properties depending on the device type:
+- How to start/stop the counters
+- Programmable vs. fixed purpose counters
+- Stoppable vs. always running counters
+- Independent vs. grouped counters
+- Read-only vs. writable counters
+- PCI device to PMU group relationship
+
+
+Devices with PMU counters
+-------------------------
+
+Memory controller (LMC):
+- one PCI device per LMC
+- fixed-purpose counters
+- always running counters without start/stop/reset control
+- read-only counters
+
+CCPI interface controller (OCX) Transmit link (TLK) counters:
+- writable counters
+- only one PCI device exposes multiple TLK units (3 units on T88)
+- start/stop control per unit
+- only present on multi-socket systems
+
+PMU (perf) driver
+-----------------
+
+The cavium-pmu driver registers several perf PMU drivers. Each of the perf
+driver provides description of its available events and configuration options
+in sysfs, see /sys/devices/<lmcX/ocx_tlkX>/.
+
+The "format" directory describes format of the config (event ID),
+The "events" directory shows the names of the events and provides configuration
+templates for all supported event types that can be used with perf tool. For
+example, "lmc0/dclk_cnt/" is an equivalent of "lmc0/config=2/".
+
+Each perf driver also provides a "cpumask" sysfs attribute, which contains a
+single CPU ID of the processor which will be used to handle all the PMU events.
+
+Example for perf tool use:
+
+ / # perf list | grep -e lmc
+   lmc0/bank_conflict1/                               [Kernel PMU event]
+   lmc0/bank_conflict2/                               [Kernel PMU event]
+   lmc0/dclk_cnt/                                     [Kernel PMU event]
+   lmc0/ifb_cnt/                                      [Kernel PMU event]
+   lmc0/ops_cnt/                                      [Kernel PMU event]
+
+ / # perf stat -a -e lmc0/ops_cnt/,lmc0/dclk_cnt/ -- sleep 1
+
+   Performance counter stats for 'system wide':
+
+             176,133      lmc0/ops_cnt/                                               
+         670,243,653      lmc0/dclk_cnt/                                              
+
+         1.005479295 seconds time elapsed
+
+The driver does not support sampling, therefore "perf record" will
+not work. System wide mode ("-a") must be used as per-task (without "-a")
+perf sessions are not supported.
-- 
2.9.0.rc0.21.g7777322

Re: [PATCH v10 1/7] edac: thunderx: Remove suspend/resume support

[Borislav Petkov]

On Mon, Sep 25, 2017 at 02:34:56PM +0200, Jan Glauber wrote:
> The memory controller on ThunderX/OcteonTX systems does not
> support power management. Therefore remove the suspend/resume
> callbacks.
> 
> Signed-off-by: Jan Glauber <jglauber@cavium.com>
> ---
>  drivers/edac/thunderx_edac.c | 25 -------------------------
>  1 file changed, 25 deletions(-)

Applied, thanks.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

Re: [PATCH v10 6/7] perf: cavium: Support transmit-link PMU counters

[Jonathan Cameron]

On Mon, 25 Sep 2017 14:35:01 +0200
Jan Glauber <jglauber@cavium.com> wrote:

> Add support for the transmit-link (OCX TLK) PMU counters found
> on Caviums SOCs with a processor interconnect.
> 
> Properties of the OCX TLK counters:
> - per-unit control
> - fixed purpose
> - writable
> - one PCI device with multiple TLK units
> 
> Signed-off-by: Jan Glauber <jglauber@cavium.com>

Something odd happened in
drivers/soc/cavium/Kconfig...

Otherwise whole series looks sensible to me.

Jonathan

> ---
>  drivers/perf/Kconfig            |   7 ++
>  drivers/perf/cavium_pmu.c       | 230 ++++++++++++++++++++++++++++++++++++++++
>  drivers/soc/cavium/Kconfig      |   4 +
>  drivers/soc/cavium/cavium_ocx.c |   4 +
>  include/linux/soc/cavium/ocx.h  |   3 +
>  5 files changed, 248 insertions(+)
> 

<snip>

>  static int __init cvm_pmu_init(void)
>  {
>  	INIT_LIST_HEAD(&cvm_pmu_lmcs);
> diff --git a/drivers/soc/cavium/Kconfig b/drivers/soc/cavium/Kconfig
> index fe56503d20f4..2c7406872b18 100644
> --- a/drivers/soc/cavium/Kconfig
> +++ b/drivers/soc/cavium/Kconfig
> @@ -8,3 +8,7 @@ config CAVIUM_LMC
>  config CAVIUM_OCX
>  	depends on ARCH_THUNDER
>  	def_tristate m
> +
> +config CAVIUM_OCX
> +        depends on ARCH_THUNDER
> +	def_tristate m

Something odd going on here...

> diff --git a/drivers/soc/cavium/cavium_ocx.c b/drivers/soc/cavium/cavium_ocx.c
> index fa3341b0744f..de1ad146c0dd 100644
> --- a/drivers/soc/cavium/cavium_ocx.c
> +++ b/drivers/soc/cavium/cavium_ocx.c
> @@ -13,6 +13,8 @@
>  static int cvm_ocx_probe(struct pci_dev *pdev,
>  			 const struct pci_device_id *ent)
>  {
> +	if (IS_ENABLED(CONFIG_CAVIUM_PMU_OCX_TLK))
> +		cvm_ocx_tlk_pmu_probe(pdev, ent);
>  	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
>  		thunderx_edac_ocx_probe(pdev, ent);
>  	return 0;
> @@ -20,6 +22,8 @@ static int cvm_ocx_probe(struct pci_dev *pdev,
>  
>  static void cvm_ocx_remove(struct pci_dev *pdev)
>  {
> +	if (IS_ENABLED(CONFIG_CAVIUM_PMU_OCX_TLK))
> +		cvm_ocx_tlk_pmu_remove(pdev);
>  	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
>  		thunderx_edac_ocx_remove(pdev);
>  }
> diff --git a/include/linux/soc/cavium/ocx.h b/include/linux/soc/cavium/ocx.h
> index 29f55b3d3171..f7b2caada671 100644
> --- a/include/linux/soc/cavium/ocx.h
> +++ b/include/linux/soc/cavium/ocx.h
> @@ -3,6 +3,9 @@
>  
>  #include <linux/pci.h>
>  
> +int cvm_ocx_tlk_pmu_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
> +void cvm_ocx_tlk_pmu_remove(struct pci_dev *pdev);
> +
>  int thunderx_edac_ocx_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
>  void thunderx_edac_ocx_remove(struct pci_dev *pdev);
>  

Re: [PATCH v10 3/7] edac,soc: thunderx: Add wrapper for EDAC LMC PCI device

[Borislav Petkov]

On Mon, Sep 25, 2017 at 02:34:58PM +0200, Jan Glauber wrote:
> Cavium SOCs contain a memory controller that is presented as a
> PCI device. This PCI device will be used by an EDAC driver and
> by a PMU driver.
> 
> To allow both subsystems to access the device a small wrapper is
> introduced that multi-plexes PCI probe and removal calls of the
> device to the EDAC driver.
> 
> The same mechanism will be used later to call the PMU driver.
> 
> The ThunderX EDAC driver is limited to only build as module
> with this patch. The reason is that with multiple users of the
> multi-plexer all users must be either builtin or modules.
> 
> Signed-off-by: Jan Glauber <jglauber@cavium.com>
> ---

...

> diff --git a/drivers/soc/cavium/cavium_lmc.c b/drivers/soc/cavium/cavium_lmc.c
> new file mode 100644
> index 000000000000..87248e83c55b
> --- /dev/null
> +++ b/drivers/soc/cavium/cavium_lmc.c
> @@ -0,0 +1,49 @@
> +/*
> + * These PCI devices contain RAS functionality and PMU counters. To allow
> + * independent RAS and PMU drivers this driver registers for the PCI devices
> + * and multi-plexes probe and removal.
> + *
> + * This file is subject to the terms and conditions of the GNU General Public
> + * License.  See the file "COPYING" in the main directory of this archive
> + * for more details.
> + *
> + * Copyright: Cavium, Inc. (C) 2017
> + *
> + */
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/soc/cavium/lmc.h>
> +
> +static int cvm_lmc_probe(struct pci_dev *pdev,
> +			 const struct pci_device_id *ent)
> +{
> +	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
> +		thunderx_edac_lmc_probe(pdev, ent);

You could save yourself the if (IS_ENABLED()) here by adding stubs in
the lmc.h header for those functions for the !CONFIG_EDAC_THUNDERX case.

One thing I'm not clear on though, is the design of the whole thing:
cvm_lmc_probe() probes the EDAC driver during its own probe, which
means, thunderx_edac needs to be loaded first. And the other things that
get loaded, do the same.

What I was expecting is those small cavium_lmc.c and cavium_ocx.c
wrappers to probe and register the respective PCI device and then its
*users* - EDAC and PMU drivers to go and request the PCI device from
them:

cavium_lmc_get_pci_dev()
cavium_ocx_get_pci_dev()

and so on. Those will be exported to modules. And the small stubs can
also be built-in too.

This way you can do reference counting and whatever else.

If the above calls fail, neither EDAC nor PMU will load properly but you
solve the multiplexing issue by having those wrappers arbitrate access
to the PCI devices.

Because right now the wrappers are simply weakly hiding the calls into
EDAC and that's exactly what I was opposing to.

Hmmm?

> +	return 0;
> +}
> +
> +static void cvm_lmc_remove(struct pci_dev *pdev)
> +{
> +	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
> +		thunderx_edac_lmc_remove(pdev);
> +}
> +
> +static const struct pci_device_id cvm_lmc_pci_table[] = {
> +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, 0xa022) },

        { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_LMC) },

You already have that PCI device id define.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

Re: [PATCH v10 3/7] edac,soc: thunderx: Add wrapper for EDAC LMC PCI device

[Jan Glauber]

On Wed, Sep 27, 2017 at 06:19:01PM +0200, Borislav Petkov wrote:
> On Mon, Sep 25, 2017 at 02:34:58PM +0200, Jan Glauber wrote:
> > Cavium SOCs contain a memory controller that is presented as a
> > PCI device. This PCI device will be used by an EDAC driver and
> > by a PMU driver.
> > 
> > To allow both subsystems to access the device a small wrapper is
> > introduced that multi-plexes PCI probe and removal calls of the
> > device to the EDAC driver.
> > 
> > The same mechanism will be used later to call the PMU driver.
> > 
> > The ThunderX EDAC driver is limited to only build as module
> > with this patch. The reason is that with multiple users of the
> > multi-plexer all users must be either builtin or modules.
> > 
> > Signed-off-by: Jan Glauber <jglauber@cavium.com>
> > ---
> 
> ...
> 
> > diff --git a/drivers/soc/cavium/cavium_lmc.c b/drivers/soc/cavium/cavium_lmc.c
> > new file mode 100644
> > index 000000000000..87248e83c55b
> > --- /dev/null
> > +++ b/drivers/soc/cavium/cavium_lmc.c
> > @@ -0,0 +1,49 @@
> > +/*
> > + * These PCI devices contain RAS functionality and PMU counters. To allow
> > + * independent RAS and PMU drivers this driver registers for the PCI devices
> > + * and multi-plexes probe and removal.
> > + *
> > + * This file is subject to the terms and conditions of the GNU General Public
> > + * License.  See the file "COPYING" in the main directory of this archive
> > + * for more details.
> > + *
> > + * Copyright: Cavium, Inc. (C) 2017
> > + *
> > + */
> > +#include <linux/module.h>
> > +#include <linux/pci.h>
> > +#include <linux/soc/cavium/lmc.h>
> > +
> > +static int cvm_lmc_probe(struct pci_dev *pdev,
> > +			 const struct pci_device_id *ent)
> > +{
> > +	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
> > +		thunderx_edac_lmc_probe(pdev, ent);
> 
> You could save yourself the if (IS_ENABLED()) here by adding stubs in
> the lmc.h header for those functions for the !CONFIG_EDAC_THUNDERX case.

OK.

> One thing I'm not clear on though, is the design of the whole thing:
> cvm_lmc_probe() probes the EDAC driver during its own probe, which
> means, thunderx_edac needs to be loaded first. And the other things that
> get loaded, do the same.

Yes. That seems to work fine with the limitation of not being
able to have mixed build options, everything needs to be modular or
built-in. So I forced all parts to be buildable only as modules.

I went for this as the simplest solution, the probing is completely
synchronous and no state needs to be stored in the wrapper.

> What I was expecting is those small cavium_lmc.c and cavium_ocx.c
> wrappers to probe and register the respective PCI device and then its
> *users* - EDAC and PMU drivers to go and request the PCI device from
> them:
> 
> cavium_lmc_get_pci_dev()
> cavium_ocx_get_pci_dev()
> 
> and so on. Those will be exported to modules. And the small stubs can
> also be built-in too.

So this is the opposite approach which would be asynchronous. What I
don't understand with that approach:

1. What will trigger probing the edac (or perf) driver part?
   Right now the trigger is the PCI device ID. If the wrapper
   does not call into edac how should we load the ThunderX edac/perf drivers?
   The only option I see is a initcall in edac/perf to look for their devices.

2. The probe & register is _very_ specific to perf/edac and very different.
   The only part that would fit in the wrapper is pci_enable_device().
   So is that what you have in mind?

> This way you can do reference counting and whatever else.
> 
> If the above calls fail, neither EDAC nor PMU will load properly but you
> solve the multiplexing issue by having those wrappers arbitrate access
> to the PCI devices.
> 
> Because right now the wrappers are simply weakly hiding the calls into
> EDAC and that's exactly what I was opposing to.
> 
> Hmmm?

> > +	return 0;
> > +}
> > +
> > +static void cvm_lmc_remove(struct pci_dev *pdev)
> > +{
> > +	if (IS_ENABLED(CONFIG_EDAC_THUNDERX))
> > +		thunderx_edac_lmc_remove(pdev);
> > +}
> > +
> > +static const struct pci_device_id cvm_lmc_pci_table[] = {
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, 0xa022) },
> 
>         { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_LMC) },
> 
> You already have that PCI device id define.

True, will use it.

Thanks for looking at this!

--Jan

> -- 
> Regards/Gruss,
>     Boris.
> 
> Good mailing practices for 400: avoid top-posting and trim the reply.

Re: [PATCH v10 3/7] edac,soc: thunderx: Add wrapper for EDAC LMC PCI device

[Borislav Petkov]

On Mon, Oct 02, 2017 at 05:17:56PM +0200, Jan Glauber wrote:
> I went for this as the simplest solution, the probing is completely
> synchronous and no state needs to be stored in the wrapper.

What state would you need to store? The wrapper simply gives out the
struct pci_dev * to the callers or NULL if not present.

> 1. What will trigger probing the edac (or perf) driver part?
>    Right now the trigger is the PCI device ID. If the wrapper
>    does not call into edac how should we load the ThunderX edac/perf drivers?
>    The only option I see is a initcall in edac/perf to look for their devices.

The wrapper loads on the PCI dev ID. EDAC loads later and calls the
wrapper function to get the struct pci_dev *. Simple.

> 2. The probe & register is _very_ specific to perf/edac and very different.
>    The only part that would fit in the wrapper is pci_enable_device().
>    So is that what you have in mind?

No, see above. Instead of getting the PCI device IDs from the PCI core,
you use the wrapper, which gets those from the PCI core. Thus it is
called a "wrapper". :)

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

Re: [PATCH v10 5/7] perf: cavium: Support memory controller PMU counters

[Suzuki K Poulose]

On 25/09/17 13:35, Jan Glauber wrote:
> Add support for the PMU counters on Cavium SOC memory controllers.
> 
> This patch also adds generic functions to allow supporting more
> devices with PMU counters.
> 
> Properties of the LMC PMU counters:
> - not stoppable
> - fixed purpose
> - read-only
> - one PCI device per memory controller
> 
> Signed-off-by: Jan Glauber <jglauber@cavium.com>
> ---
>   drivers/perf/Kconfig            |   8 +
>   drivers/perf/Makefile           |   1 +
>   drivers/perf/cavium_pmu.c       | 430 ++++++++++++++++++++++++++++++++++++++++
>   drivers/soc/cavium/cavium_lmc.c |   4 +
>   include/linux/cpuhotplug.h      |   1 +
>   include/linux/soc/cavium/lmc.h  |   3 +
>   6 files changed, 447 insertions(+)
>   create mode 100644 drivers/perf/cavium_pmu.c

> +int cvm_lmc_pmu_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
> +{
> +	struct cvm_pmu_dev *next, *lmc;
> +	int nr = 0, ret = -ENOMEM;
> +	char *name;
> +
> +	lmc = devm_kzalloc(&pdev->dev, sizeof(*lmc), GFP_KERNEL);
> +	if (!lmc)
> +		return -ENOMEM;
> +
> +	lmc->map = devm_ioremap(&pdev->dev, pci_resource_start(pdev, 0),
> +				pci_resource_len(pdev, 0));
> +	if (!lmc->map)
> +		return -EINVAL;
> +
> +	list_for_each_entry(next, &cvm_pmu_lmcs, entry)
> +		nr++;
> +	name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "lmc%d", nr);
> +	if (!name)
> +		return -ENOMEM;
> +
> +	lmc->pdev = pdev;
> +	lmc->num_counters = ARRAY_SIZE(cvm_pmu_lmc_events_attr) - 1;
> +	lmc->pmu = (struct pmu) {
> +		.task_ctx_nr    = perf_invalid_context,
> +		.name		= name,
> +		.event_init	= cvm_pmu_event_init,
> +		.add		= cvm_pmu_lmc_add,
> +		.del		= cvm_pmu_del,
> +		.start		= cvm_pmu_start,
> +		.stop		= cvm_pmu_stop,
> +		.read		= cvm_pmu_read,
> +		.attr_groups	= cvm_pmu_lmc_attr_groups,
> +	};
> +

You need to fill in the "module" field of the PMU to prevent the module from
being unloaded while the PMU is active.

See :

http://lists.infradead.org/pipermail/linux-arm-kernel/2017-November/540647.html

Suzuki