From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753922AbdDCO74 (ORCPT ); Mon, 3 Apr 2017 10:59:56 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:49534 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751803AbdDCO42 (ORCPT ); Mon, 3 Apr 2017 10:56:28 -0400 From: Madhavan Srinivasan To: mpe@ellerman.id.au Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, ego@linux.vnet.ibm.com, bsingharora@gmail.com, benh@kernel.crashing.org, paulus@samba.org, anton@samba.org, sukadev@linux.vnet.ibm.com, mikey@neuling.org, stewart@linux.vnet.ibm.com, dja@axtens.net, eranian@google.com, Madhavan Srinivasan Subject: [PATCH v6 00/11] IMC Instrumentation Support Date: Mon, 3 Apr 2017 20:24:57 +0530 X-Mailer: git-send-email 2.7.4 X-TM-AS-MML: disable x-cbid: 17040314-0052-0000-0000-0000022B0877 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17040314-0053-0000-0000-000008040C7D Message-Id: <1491231308-15282-1-git-send-email-maddy@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-03_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1704030133 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Power9 has In-Memory-Collection (IMC) infrastructure which contains various Performance Monitoring Units (PMUs) at Nest level (these are on-chip but off-core), Core level and Thread level. The Nest PMU counters are handled by a Nest IMC microcode which runs in the OCC (On-Chip Controller) complex. The microcode collects the counter data and moves the nest IMC counter data to memory. The Core and Thread IMC PMU counters are handled in the core. Core level PMU counters give us the IMC counters' data per core and thread level PMU counters give us the IMC counters' data per CPU thread. This patchset enables the nest IMC, core IMC and thread IMC PMUs and is based on the initial work done by Madhavan Srinivasan. "Nest Instrumentation Support" : https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-August/132078.html v1 for this patchset can be found here : https://lwn.net/Articles/705475/ Nest events: Per-chip nest instrumentation provides various per-chip metrics such as memory, powerbus, Xlink and Alink bandwidth. Core events: Per-core IMC instrumentation provides various per-core metrics such as non-idle cycles, non-idle instructions, various cache and memory related metrics etc. Thread events: All the events for thread level are same as core level with the difference being in the domain. These are per-cpu metrics. PMU Events' Information: OPAL obtains the IMC PMU and event information from the IMC Catalog and passes on to the kernel via the device tree. The events' information contains : - Event name - Event Offset - Event description and, maybe : - Event scale - Event unit Some PMUs may have a common scale and unit values for all their supported events. For those cases, the scale and unit properties for those events must be inherited from the PMU. The event offset in the memory is where the counter data gets accumulated. The OPAL-side patches are posted upstream : https://lists.ozlabs.org/pipermail/skiboot/2017-March/006531.html The kernel discovers the IMC counters information in the device tree at the "imc-counters" device node which has a compatible field "ibm,opal-in-memory-counters". Parsing of the Events' information: To parse the IMC PMUs and events information, the kernel has to discover the "imc-counters" node and walk through the pmu and event nodes. Here is an excerpt of the dt showing the imc-counters with mcs0 (nest), core and thread node: https://github.com/open-power/ima-catalog/blob/master/81E00612.4E0100.dts /dts-v1/; [...] /dts-v1/; / { name = ""; compatible = "ibm,opal-in-memory-counters"; #address-cells = <0x1>; #size-cells = <0x1>; imc-nest-offset = <0x320000>; imc-nest-size = <0x30000>; version-id = ""; NEST_MCS: nest-mcs-events { #address-cells = <0x1>; #size-cells = <0x1>; event@0 { event-name = "RRTO_QFULL_NO_DISP" ; reg = <0x0 0x8>; desc = "RRTO not dispatched in MCS0 due to capacity - pulses once for each time a valid RRTO op is not dispatched due to a command list full condition" ; }; event@8 { event-name = "WRTO_QFULL_NO_DISP" ; reg = <0x8 0x8>; desc = "WRTO not dispatched in MCS0 due to capacity - pulses once for each time a valid WRTO op is not dispatched due to a command list full condition" ; }; [...] mcs0 { compatible = "ibm,imc-counters-nest"; events-prefix = "PM_MCS0_"; unit = ""; scale = ""; reg = <0x118 0x8>; events = < &NEST_MCS >; }; mcs1 { compatible = "ibm,imc-counters-nest"; events-prefix = "PM_MCS1_"; unit = ""; scale = ""; reg = <0x198 0x8>; events = < &NEST_MCS >; }; [...] CORE_EVENTS: core-events { #address-cells = <0x1>; #size-cells = <0x1>; event@e0 { event-name = "0THRD_NON_IDLE_PCYC" ; reg = <0xe0 0x8>; desc = "The number of processor cycles when all threads are idle" ; }; event@120 { event-name = "1THRD_NON_IDLE_PCYC" ; reg = <0x120 0x8>; desc = "The number of processor cycles when exactly one SMT thread is executing non-idle code" ; }; [...] core { compatible = "ibm,imc-counters-core"; events-prefix = "CPM_"; unit = ""; scale = ""; reg = <0x0 0x8>; events = < &CORE_EVENTS >; }; thread { compatible = "ibm,imc-counters-core"; events-prefix = "CPM_"; unit = ""; scale = ""; reg = <0x0 0x8>; events = < &CORE_EVENTS >; }; }; >>From the device tree, the kernel parses the PMUs and their events' information. After parsing the IMC PMUs and their events, the PMUs and their attributes are registered in the kernel. This patchset (patches 9 and 10) configure the thread level IMC PMUs to count for tasks, which give us the thread level metric values per task. Example Usage : # perf list [...] nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0/ [Kernel PMU event] nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0_LAST_SAMPLE/ [Kernel PMU event] [...] core_imc/CPM_NON_IDLE_INST/ [Kernel PMU event] core_imc/CPM_NON_IDLE_PCYC/ [Kernel PMU event] [...] thread_imc/CPM_NON_IDLE_INST/ [Kernel PMU event] thread_imc/CPM_NON_IDLE_PCYC/ [Kernel PMU event] To see per chip data for nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0/ : # perf stat -e "nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0/" -a --per-socket To see non-idle instructions for core 0 : # ./perf stat -e "core_imc/CPM_NON_IDLE_INST/" -C 0 -I 1000 To see non-idle instructions for a "make" : # ./perf stat -e "thread_imc/CPM_NON_IDLE_PCYC/" make Comments/feedback/suggestions are welcome. TODO: 1)Add a sysfs interface to disable the Core imc (both for ldbar and pdbar) 2)Add disable_imc_nest and diable_imc_core kernel parameters to help out in debug Changelog: v5 -> v6: - merged few patches for the readability and code flow - Updated the commit message and code comments. - Added kdump check. - updated cpuhotplug code and added checks for perf migration context - Added READ_ONCE() when reading the counter data. - replaced of_property_read_u32() with of_get_address() for "reg" property read - replaced UNKNOWN_DOMAIN with IMC_DOMAIN_UNKNOWN v4 -> v5: - Updated opal call numbers - Added a patch to disable Core-IMC device using shutdown callback - Added patch to support cpuhotplug for thread-imc - Added patch to disable and enable core imc engine in cpuhot plug path v3 -> v4 : - Changed the events parser code to discover the PMU and events because of the changed format of the IMC DTS file (Patch 3). - Implemented the two TODOs to include core and thread IMC support with this patchset (Patches 7 through 10). - Changed the CPU hotplug code of Nest IMC PMUs to include a new state CPUHP_AP_PERF_POWERPC_NEST_ONLINE (Patch 6). v2 -> v3 : - Changed all references for IMA (In-Memory Accumulation) to IMC (In-Memory Collection). v1 -> v2 : - Account for the cases where a PMU can have a common scale and unit values for all its supported events (Patch 3/6). - Fixed a Build error (for maple_defconfig) by enabling imc_pmu.o only for CONFIG_PPC_POWERNV=y (Patch 4/6) - Read from the "event-name" property instead of "name" for an event node (Patch 3/6). Anju T Sudhakar (1): powerpc/perf: Thread imc cpuhotplug support Hemant Kumar (10): powerpc/powernv: Data structure and macros definitions powerpc/powernv: Autoload IMC device driver module powerpc/powernv: Detect supported IMC units and its events powerpc/perf: Add event attribute and group to IMC pmus powerpc/perf: Generic imc pmu event functions powerpc/perf: IMC pmu cpumask and cpu hotplug support powerpc/powernv: Core IMC events detection powerpc/perf: PMU functions for Core IMC and hotplugging powerpc/powernv: Thread IMC events detection powerpc/perf: Thread IMC PMU functions arch/powerpc/include/asm/imc-pmu.h | 81 +++ arch/powerpc/include/asm/opal-api.h | 21 +- arch/powerpc/include/asm/opal.h | 5 + arch/powerpc/perf/Makefile | 6 +- arch/powerpc/perf/imc-pmu.c | 837 +++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Makefile | 2 +- arch/powerpc/platforms/powernv/opal-imc.c | 575 +++++++++++++++++ arch/powerpc/platforms/powernv/opal-wrappers.S | 2 + arch/powerpc/platforms/powernv/opal.c | 14 + include/linux/cpuhotplug.h | 3 + 10 files changed, 1543 insertions(+), 3 deletions(-) create mode 100644 arch/powerpc/include/asm/imc-pmu.h create mode 100644 arch/powerpc/perf/imc-pmu.c create mode 100644 arch/powerpc/platforms/powernv/opal-imc.c -- 2.7.4