[3/3] perf/x86/rapl: Fix psys-energy event on Intel SPR platform
diff mbox series

Message ID 20210115092208.20866-3-rui.zhang@intel.com
State New, archived
Headers show
Series
  • [1/3] perf/x86/rapl: Add msr mask support
Related show

Commit Message

Zhang, Rui Jan. 15, 2021, 9:22 a.m. UTC
There are several things special for the RAPL Psys energy counter, on
Intel Sapphire Rapids platform.
1. it contains one Psys master package, and only CPUs on the master
   package can read valid value of the Psys energy counter, reading the
   MSR on CPUs in the slave package returns 0.
2. The master package does not have to be Physical package 0. And when
   all the CPUs on the Psys master package are offlined, we lose the Psys
   energy counter, at runtime.
3. The Psys energy counter can be disabled by BIOS, while all the other
   energy counters are not affected.

It is not easy to handle all of these in the current RAPL PMU design
because
a) perf_msr_probe() validates the MSR on some random CPU, which may either
   be in the Psys master package or in the Psys slave package.
b) all the RAPL events share the same PMU, and there is not API to remove
   the psys-energy event cleanly, without affecting the other events in
   the same PMU.

This patch addresses the problems in a simple way.

First, by setting .no_check bit for RAPL Psys MSR, the psys-energy event
is always added, so we don't have to check the Psys ENERGY_STATUS MSR on
master package.

Then, rapl_not_visible() is removed because
1. it is useless for RAPL MSRs with .no_check cleared, because the
   .is_visible() callbacks is always overridden in perf_msr_probe().
2. it is useless for RAPL MSRs with .no_check set, because we actually
   want the sysfs attributes always be visible for those MSRs.

With the above changes, we always probe the psys-energy event on Intel SPR
platform. Difference is that the event counter returns 0 when the Psys
RAPL Domain is disabled by BIOS, or the Psys master package is offlined.

Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Reviewed-by: Andi Kleen <ak@linux.intel.com>
---
 arch/x86/events/rapl.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

Comments

Peter Zijlstra Jan. 16, 2021, 12:50 p.m. UTC | #1
On Fri, Jan 15, 2021 at 05:22:08PM +0800, Zhang Rui wrote:
> There are several things special for the RAPL Psys energy counter, on
> Intel Sapphire Rapids platform.
> 1. it contains one Psys master package, and only CPUs on the master
>    package can read valid value of the Psys energy counter, reading the
>    MSR on CPUs in the slave package returns 0.
> 2. The master package does not have to be Physical package 0. And when
>    all the CPUs on the Psys master package are offlined, we lose the Psys
>    energy counter, at runtime.
> 3. The Psys energy counter can be disabled by BIOS, while all the other
>    energy counters are not affected.
> 
> It is not easy to handle all of these in the current RAPL PMU design
> because
> a) perf_msr_probe() validates the MSR on some random CPU, which may either
>    be in the Psys master package or in the Psys slave package.
> b) all the RAPL events share the same PMU, and there is not API to remove
>    the psys-energy event cleanly, without affecting the other events in
>    the same PMU.
> 
> This patch addresses the problems in a simple way.
> 
> First, by setting .no_check bit for RAPL Psys MSR, the psys-energy event
> is always added, so we don't have to check the Psys ENERGY_STATUS MSR on
> master package.
> 
> Then, rapl_not_visible() is removed because
> 1. it is useless for RAPL MSRs with .no_check cleared, because the
>    .is_visible() callbacks is always overridden in perf_msr_probe().
> 2. it is useless for RAPL MSRs with .no_check set, because we actually
>    want the sysfs attributes always be visible for those MSRs.
> 
> With the above changes, we always probe the psys-energy event on Intel SPR
> platform. Difference is that the event counter returns 0 when the Psys
> RAPL Domain is disabled by BIOS, or the Psys master package is offlined.

Maybe I'm too tired, but I cannot follow. How does this cure the fact
that the rapl_cpu_mask might not include that master thing. And how can
software detect what the master thing is to begin with?
Zhang, Rui Jan. 17, 2021, 2:33 p.m. UTC | #2
Hi, Peter,

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Saturday, January 16, 2021 8:50 PM
> To: Zhang, Rui <rui.zhang@intel.com>
> Cc: mingo@redhat.com; acme@kernel.org; mark.rutland@arm.com;
> alexander.shishkin@linux.intel.com; jolsa@redhat.com;
> namhyung@kernel.org; linux-kernel@vger.kernel.org; x86@kernel.org;
> kan.liang@linux.intel.com; ak@linux.intel.com
> Subject: Re: [PATCH 3/3] perf/x86/rapl: Fix psys-energy event on Intel SPR
> platform
> Importance: High
> 
> On Fri, Jan 15, 2021 at 05:22:08PM +0800, Zhang Rui wrote:
> > There are several things special for the RAPL Psys energy counter, on
> > Intel Sapphire Rapids platform.
> > 1. it contains one Psys master package, and only CPUs on the master
> >    package can read valid value of the Psys energy counter, reading the
> >    MSR on CPUs in the slave package returns 0.
> > 2. The master package does not have to be Physical package 0. And when
> >    all the CPUs on the Psys master package are offlined, we lose the Psys
> >    energy counter, at runtime.
> > 3. The Psys energy counter can be disabled by BIOS, while all the other
> >    energy counters are not affected.
> >
> > It is not easy to handle all of these in the current RAPL PMU design
> > because
> > a) perf_msr_probe() validates the MSR on some random CPU, which may
> either
> >    be in the Psys master package or in the Psys slave package.
> > b) all the RAPL events share the same PMU, and there is not API to remove
> >    the psys-energy event cleanly, without affecting the other events in
> >    the same PMU.
> >
> > This patch addresses the problems in a simple way.
> >
> > First, by setting .no_check bit for RAPL Psys MSR, the psys-energy
> > event is always added, so we don't have to check the Psys
> > ENERGY_STATUS MSR on master package.
> >
> > Then, rapl_not_visible() is removed because 1. it is useless for RAPL
> > MSRs with .no_check cleared, because the
> >    .is_visible() callbacks is always overridden in perf_msr_probe().
> > 2. it is useless for RAPL MSRs with .no_check set, because we actually
> >    want the sysfs attributes always be visible for those MSRs.
> >
> > With the above changes, we always probe the psys-energy event on Intel
> > SPR platform. Difference is that the event counter returns 0 when the
> > Psys RAPL Domain is disabled by BIOS, or the Psys master package is
> offlined.
> 
> Maybe I'm too tired, but I cannot follow. How does this cure the fact that the
> rapl_cpu_mask might not include that master thing. And how can software
> detect what the master thing is to begin with?

To make things simple, I ignore the master thing, and probe the psys-energy counter blindly on SPR.
So rapl_cpu_mask still includes all the online CPUs.
This means that psys-energy is "valid" on all packages, and it just returns different values on different packages.
AKA, whole system power consumption on Psys master package, and Zero on Psys slave packages.

Not sure if I answered your question or not.

Thanks,
rui
Zhang, Rui Jan. 25, 2021, 6:11 a.m. UTC | #3
Hi, Peter,

> -----Original Message-----
> From: Zhang, Rui
> Sent: Sunday, January 17, 2021 10:34 PM
> To: 'Peter Zijlstra' <peterz@infradead.org>
> Cc: mingo@redhat.com; acme@kernel.org; mark.rutland@arm.com;
> alexander.shishkin@linux.intel.com; jolsa@redhat.com;
> namhyung@kernel.org; linux-kernel@vger.kernel.org; x86@kernel.org;
> kan.liang@linux.intel.com; ak@linux.intel.com
> Subject: RE: [PATCH 3/3] perf/x86/rapl: Fix psys-energy event on Intel SPR
> platform
> 
> Hi, Peter,
> 
> > -----Original Message-----
> > From: Peter Zijlstra <peterz@infradead.org>
> > Sent: Saturday, January 16, 2021 8:50 PM
> > To: Zhang, Rui <rui.zhang@intel.com>
> > Cc: mingo@redhat.com; acme@kernel.org; mark.rutland@arm.com;
> > alexander.shishkin@linux.intel.com; jolsa@redhat.com;
> > namhyung@kernel.org; linux-kernel@vger.kernel.org; x86@kernel.org;
> > kan.liang@linux.intel.com; ak@linux.intel.com
> > Subject: Re: [PATCH 3/3] perf/x86/rapl: Fix psys-energy event on Intel
> > SPR platform
> > Importance: High
> >
> > On Fri, Jan 15, 2021 at 05:22:08PM +0800, Zhang Rui wrote:
> > > There are several things special for the RAPL Psys energy counter,
> > > on Intel Sapphire Rapids platform.
> > > 1. it contains one Psys master package, and only CPUs on the master
> > >    package can read valid value of the Psys energy counter, reading the
> > >    MSR on CPUs in the slave package returns 0.
> > > 2. The master package does not have to be Physical package 0. And when
> > >    all the CPUs on the Psys master package are offlined, we lose the Psys
> > >    energy counter, at runtime.
> > > 3. The Psys energy counter can be disabled by BIOS, while all the other
> > >    energy counters are not affected.
> > >
> > > It is not easy to handle all of these in the current RAPL PMU design
> > > because
> > > a) perf_msr_probe() validates the MSR on some random CPU, which may
> > either
> > >    be in the Psys master package or in the Psys slave package.
> > > b) all the RAPL events share the same PMU, and there is not API to
> remove
> > >    the psys-energy event cleanly, without affecting the other events in
> > >    the same PMU.
> > >
> > > This patch addresses the problems in a simple way.
> > >
> > > First, by setting .no_check bit for RAPL Psys MSR, the psys-energy
> > > event is always added, so we don't have to check the Psys
> > > ENERGY_STATUS MSR on master package.
> > >
> > > Then, rapl_not_visible() is removed because 1. it is useless for
> > > RAPL MSRs with .no_check cleared, because the
> > >    .is_visible() callbacks is always overridden in perf_msr_probe().
> > > 2. it is useless for RAPL MSRs with .no_check set, because we actually
> > >    want the sysfs attributes always be visible for those MSRs.
> > >
> > > With the above changes, we always probe the psys-energy event on
> > > Intel SPR platform. Difference is that the event counter returns 0
> > > when the Psys RAPL Domain is disabled by BIOS, or the Psys master
> > > package is
> > offlined.
> >
> > Maybe I'm too tired, but I cannot follow. How does this cure the fact
> > that the rapl_cpu_mask might not include that master thing. And how
> > can software detect what the master thing is to begin with?
> 
> To make things simple, I ignore the master thing, and probe the psys-energy
> counter blindly on SPR.
> So rapl_cpu_mask still includes all the online CPUs.
> This means that psys-energy is "valid" on all packages, and it just returns
> different values on different packages.
> AKA, whole system power consumption on Psys master package, and Zero
> on Psys slave packages.
> 
In short, the current code does not allow RAPL energy counter to return 0. And all the work I do is to allow Psys energy counter to return 0.
In this way, the Psys event is "valid" on all CPUs, so we don't need to handle the master thing.
The drawback is that we still see psys-energy event, but with 0 readout, when Psys counter is not available (master package offlined, or psys disabled).

TBH, I'm not quite sure if I understand your original question correctly or not, so please let me know if there is still something unclear.

Thanks,
rui
> 
> Thanks,
> rui
Zhang, Rui Feb. 3, 2021, 2:17 p.m. UTC | #4
Hi, Peter,

> -----Original Message-----
> From: Zhang, Rui
> Sent: Monday, January 25, 2021 2:11 PM
> To: 'Peter Zijlstra' <peterz@infradead.org>
> Cc: 'mingo@redhat.com' <mingo@redhat.com>; 'acme@kernel.org'
> <acme@kernel.org>; 'mark.rutland@arm.com' <mark.rutland@arm.com>;
> 'alexander.shishkin@linux.intel.com' <alexander.shishkin@linux.intel.com>;
> 'jolsa@redhat.com' <jolsa@redhat.com>; 'namhyung@kernel.org'
> <namhyung@kernel.org>; 'linux-kernel@vger.kernel.org' <linux-
> kernel@vger.kernel.org>; 'x86@kernel.org' <x86@kernel.org>;
> 'kan.liang@linux.intel.com' <kan.liang@linux.intel.com>; 'ak@linux.intel.com'
> <ak@linux.intel.com>
> Subject: RE: [PATCH 3/3] perf/x86/rapl: Fix psys-energy event on Intel SPR
> platform
> 
> Hi, Peter,
> 
> > -----Original Message-----
> > From: Zhang, Rui
> > Sent: Sunday, January 17, 2021 10:34 PM
> > To: 'Peter Zijlstra' <peterz@infradead.org>
> > Cc: mingo@redhat.com; acme@kernel.org; mark.rutland@arm.com;
> > alexander.shishkin@linux.intel.com; jolsa@redhat.com;
> > namhyung@kernel.org; linux-kernel@vger.kernel.org; x86@kernel.org;
> > kan.liang@linux.intel.com; ak@linux.intel.com
> > Subject: RE: [PATCH 3/3] perf/x86/rapl: Fix psys-energy event on Intel
> > SPR platform
> >
> > Hi, Peter,
> >
> > > -----Original Message-----
> > > From: Peter Zijlstra <peterz@infradead.org>
> > > Sent: Saturday, January 16, 2021 8:50 PM
> > > To: Zhang, Rui <rui.zhang@intel.com>
> > > Cc: mingo@redhat.com; acme@kernel.org; mark.rutland@arm.com;
> > > alexander.shishkin@linux.intel.com; jolsa@redhat.com;
> > > namhyung@kernel.org; linux-kernel@vger.kernel.org; x86@kernel.org;
> > > kan.liang@linux.intel.com; ak@linux.intel.com
> > > Subject: Re: [PATCH 3/3] perf/x86/rapl: Fix psys-energy event on
> > > Intel SPR platform
> > > Importance: High
> > >
> > > On Fri, Jan 15, 2021 at 05:22:08PM +0800, Zhang Rui wrote:
> > > > There are several things special for the RAPL Psys energy counter,
> > > > on Intel Sapphire Rapids platform.
> > > > 1. it contains one Psys master package, and only CPUs on the master
> > > >    package can read valid value of the Psys energy counter, reading the
> > > >    MSR on CPUs in the slave package returns 0.
> > > > 2. The master package does not have to be Physical package 0. And
> when
> > > >    all the CPUs on the Psys master package are offlined, we lose the Psys
> > > >    energy counter, at runtime.
> > > > 3. The Psys energy counter can be disabled by BIOS, while all the other
> > > >    energy counters are not affected.
> > > >
> > > > It is not easy to handle all of these in the current RAPL PMU
> > > > design because
> > > > a) perf_msr_probe() validates the MSR on some random CPU, which
> > > > may
> > > either
> > > >    be in the Psys master package or in the Psys slave package.
> > > > b) all the RAPL events share the same PMU, and there is not API to
> > remove
> > > >    the psys-energy event cleanly, without affecting the other events in
> > > >    the same PMU.
> > > >
> > > > This patch addresses the problems in a simple way.
> > > >
> > > > First, by setting .no_check bit for RAPL Psys MSR, the psys-energy
> > > > event is always added, so we don't have to check the Psys
> > > > ENERGY_STATUS MSR on master package.
> > > >
> > > > Then, rapl_not_visible() is removed because 1. it is useless for
> > > > RAPL MSRs with .no_check cleared, because the
> > > >    .is_visible() callbacks is always overridden in perf_msr_probe().
> > > > 2. it is useless for RAPL MSRs with .no_check set, because we actually
> > > >    want the sysfs attributes always be visible for those MSRs.
> > > >
> > > > With the above changes, we always probe the psys-energy event on
> > > > Intel SPR platform. Difference is that the event counter returns 0
> > > > when the Psys RAPL Domain is disabled by BIOS, or the Psys master
> > > > package is
> > > offlined.
> > >
> > > Maybe I'm too tired, but I cannot follow. How does this cure the
> > > fact that the rapl_cpu_mask might not include that master thing. And
> > > how can software detect what the master thing is to begin with?
> >
> > To make things simple, I ignore the master thing, and probe the
> > psys-energy counter blindly on SPR.
> > So rapl_cpu_mask still includes all the online CPUs.
> > This means that psys-energy is "valid" on all packages, and it just
> > returns different values on different packages.
> > AKA, whole system power consumption on Psys master package, and Zero
> > on Psys slave packages.
> >
> In short, the current code does not allow RAPL energy counter to return 0.
> And all the work I do is to allow Psys energy counter to return 0.
> In this way, the Psys event is "valid" on all CPUs, so we don't need to handle
> the master thing.
> The drawback is that we still see psys-energy event, but with 0 readout,
> when Psys counter is not available (master package offlined, or psys
> disabled).
> 
> TBH, I'm not quite sure if I understand your original question correctly or not,
> so please let me know if there is still something unclear.
> 
Sorry to bother, may I know your concern about this patch series?

Thanks,
rui
> Thanks,
> rui
> >
> > Thanks,
> > rui
Peter Zijlstra Feb. 3, 2021, 2:47 p.m. UTC | #5
FWIW, your email is malformed, please wrap at 78 chars.

On Mon, Jan 25, 2021 at 06:11:14AM +0000, Zhang, Rui wrote:
> In short, the current code does not allow RAPL energy counter to
> return 0. And all the work I do is to allow Psys energy counter to
> return 0.

Ok.

> In this way, the Psys event is "valid" on all CPUs, so we don't need
> to handle the master thing.

So RAPL is mapped to DIEs, and IIRC we can have multiple DIEs per
Package. But the master thing is a Package.

Is this all moot because SPR has one DIE per Package? Because if it
would have more, there's be more interesting problems I suppose.
Zhang, Rui Feb. 4, 2021, 4:04 p.m. UTC | #6
Hi, Peter,

On Wed, 2021-02-03 at 15:47 +0100, Peter Zijlstra wrote:
> FWIW, your email is malformed, please wrap at 78 chars.
> 
> On Mon, Jan 25, 2021 at 06:11:14AM +0000, Zhang, Rui wrote:
> > In short, the current code does not allow RAPL energy counter to
> > return 0. And all the work I do is to allow Psys energy counter to
> > return 0.
> 
> Ok.
> 
> > In this way, the Psys event is "valid" on all CPUs, so we don't
> > need
> > to handle the master thing.
> 
> So RAPL is mapped to DIEs, and IIRC we can have multiple DIEs per
> Package. But the master thing is a Package.
> 
> Is this all moot because SPR has one DIE per Package?

Oh, right.
This is not a problem on SPR because it is a single-die
platform.

>  Because if it
> would have more, there's be more interesting problems I suppose.

Agreed.

thanks,
rui

Patch
diff mbox series

diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
index 7ed25b2ba05f..f42a70496a24 100644
--- a/arch/x86/events/rapl.c
+++ b/arch/x86/events/rapl.c
@@ -454,16 +454,9 @@  static struct attribute *rapl_events_cores[] = {
 	NULL,
 };
 
-static umode_t
-rapl_not_visible(struct kobject *kobj, struct attribute *attr, int i)
-{
-	return 0;
-}
-
 static struct attribute_group rapl_events_cores_group = {
 	.name  = "events",
 	.attrs = rapl_events_cores,
-	.is_visible = rapl_not_visible,
 };
 
 static struct attribute *rapl_events_pkg[] = {
@@ -476,7 +469,6 @@  static struct attribute *rapl_events_pkg[] = {
 static struct attribute_group rapl_events_pkg_group = {
 	.name  = "events",
 	.attrs = rapl_events_pkg,
-	.is_visible = rapl_not_visible,
 };
 
 static struct attribute *rapl_events_ram[] = {
@@ -489,7 +481,6 @@  static struct attribute *rapl_events_ram[] = {
 static struct attribute_group rapl_events_ram_group = {
 	.name  = "events",
 	.attrs = rapl_events_ram,
-	.is_visible = rapl_not_visible,
 };
 
 static struct attribute *rapl_events_gpu[] = {
@@ -502,7 +493,6 @@  static struct attribute *rapl_events_gpu[] = {
 static struct attribute_group rapl_events_gpu_group = {
 	.name  = "events",
 	.attrs = rapl_events_gpu,
-	.is_visible = rapl_not_visible,
 };
 
 static struct attribute *rapl_events_psys[] = {
@@ -515,7 +505,6 @@  static struct attribute *rapl_events_psys[] = {
 static struct attribute_group rapl_events_psys_group = {
 	.name  = "events",
 	.attrs = rapl_events_psys,
-	.is_visible = rapl_not_visible,
 };
 
 static bool test_msr(int idx, void *data)
@@ -534,6 +523,14 @@  static struct perf_msr intel_rapl_msrs[] = {
 	[PERF_RAPL_PSYS] = { MSR_PLATFORM_ENERGY_STATUS, &rapl_events_psys_group,  test_msr, false, RAPL_MSR_MASK },
 };
 
+static struct perf_msr intel_rapl_spr_msrs[] = {
+	[PERF_RAPL_PP0]  = { MSR_PP0_ENERGY_STATUS,      &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
+	[PERF_RAPL_PKG]  = { MSR_PKG_ENERGY_STATUS,      &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
+	[PERF_RAPL_RAM]  = { MSR_DRAM_ENERGY_STATUS,     &rapl_events_ram_group,   test_msr, false, RAPL_MSR_MASK },
+	[PERF_RAPL_PP1]  = { MSR_PP1_ENERGY_STATUS,      &rapl_events_gpu_group,   test_msr, false, RAPL_MSR_MASK },
+	[PERF_RAPL_PSYS] = { MSR_PLATFORM_ENERGY_STATUS, &rapl_events_psys_group,  test_msr, true, RAPL_MSR_MASK },
+};
+
 /*
  * Force to PERF_RAPL_MAX size due to:
  * - perf_msr_probe(PERF_RAPL_MAX)
@@ -764,7 +761,7 @@  static struct rapl_model model_spr = {
 			  BIT(PERF_RAPL_PSYS),
 	.unit_quirk	= RAPL_UNIT_QUIRK_INTEL_SPR,
 	.msr_power_unit = MSR_RAPL_POWER_UNIT,
-	.rapl_msrs      = intel_rapl_msrs,
+	.rapl_msrs      = intel_rapl_spr_msrs,
 };
 
 static struct rapl_model model_amd_fam17h = {