All of lore.kernel.org
 help / color / mirror / Atom feed
* DRAM power consumption with turbostat
@ 2022-01-30 10:27 Julia Lawall
  2022-02-06  8:29 ` Zhang Rui
  0 siblings, 1 reply; 4+ messages in thread
From: Julia Lawall @ 2022-01-30 10:27 UTC (permalink / raw)
  To: Len Brown; +Cc: linux-pm

Hello,

I was wondering whether the DRAM power consumption reported by turbostat
is reliable on recent Intel machines?

In particular, I observed that turbostat reported a high DRAM energy
comsumption on a machine (Intel 5128) with persistent memory, but where
the persistent memory was not being used.  A colleague did an experiemnt
on another machine with persistent memory, and reported:

-----

I didn't run the test on troll but on another server equipped with PM
where I was able to reproduce the bug and by reading directly the msr
registers, I see that:
CPU Energy units = 0.00006104J
DRAM Energy units = 0.00001526J

However turbostat assumes that the DRAM  Energy units is 0.00006104J when
it runs the computation to obtain Joules (hence the too-high value
returned by turbostat)

-----

I see the code in turbostat that just uses the CPU energy units value
(rapl_dram_energy_units_probe), but I don't know what was the MSR used to
collect the above information.  Overall, I am wondering if the DRAM energy
consumption values are reliable in cases with and without persistent
memory.

thanks,
julia

^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: DRAM power consumption with turbostat
  2022-01-30 10:27 DRAM power consumption with turbostat Julia Lawall
@ 2022-02-06  8:29 ` Zhang Rui
  2022-02-06  8:34   ` Julia Lawall
  2022-02-06  9:32   ` Julia Lawall
  0 siblings, 2 replies; 4+ messages in thread
From: Zhang Rui @ 2022-02-06  8:29 UTC (permalink / raw)
  To: Julia Lawall, Len Brown; +Cc: linux-pm

Hi, Julia,

Thanks for reporting this.

On Sun, 2022-01-30 at 11:27 +0100, Julia Lawall wrote:
> Hello,
> 
> I was wondering whether the DRAM power consumption reported by
> turbostat
> is reliable on recent Intel machines?
> 
> In particular, I observed that turbostat reported a high DRAM energy
> comsumption on a machine (Intel 5128)

are you sure it is 5128? google tells me that 5128 is pretty old, in
around 2006.

can you please paste the lscpu output?

>  with persistent memory, but where
> the persistent memory was not being used.  A colleague did an
> experiemnt
> on another machine with persistent memory, and reported:
> 
> -----
> 
> I didn't run the test on troll but on another server equipped with PM
> where I was able to reproduce the bug and by reading directly the msr
> registers, I see that:
> CPU Energy units = 0.00006104J
> DRAM Energy units = 0.00001526J
> 
> However turbostat assumes that the DRAM  Energy units is 0.00006104J
> when
> it runs the computation to obtain Joules (hence the too-high value
> returned by turbostat)

And can you please try latest version of turbostat?

The problem should be fixed by the below commit

commit abdc75ab53b7fd2ef42c79e88cf0caf2d007c4f2
Author:     Zhang Rui <rui.zhang@intel.com>
AuthorDate: Thu Mar 11 10:05:13 2021 +0800
Commit:     Len Brown <len.brown@intel.com>
CommitDate: Tue May 4 18:23:14 2021 -0400

    tools/power turbostat: Fix DRAM Energy Unit on SKX
    
    SKX uses fixed DRAM Energy Unit, just like HSX and BDX.
    
    Signed-off-by: Zhang Rui <rui.zhang@intel.com>
    Signed-off-by: Len Brown <len.brown@intel.com>

thanks,
rui
> 
> -----
> 
> I see the code in turbostat that just uses the CPU energy units value
> (rapl_dram_energy_units_probe), but I don't know what was the MSR
> used to
> collect the above information.  Overall, I am wondering if the DRAM
> energy
> consumption values are reliable in cases with and without persistent
> memory.
> 
> thanks,
> julia


^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: DRAM power consumption with turbostat
  2022-02-06  8:29 ` Zhang Rui
@ 2022-02-06  8:34   ` Julia Lawall
  2022-02-06  9:32   ` Julia Lawall
  1 sibling, 0 replies; 4+ messages in thread
From: Julia Lawall @ 2022-02-06  8:34 UTC (permalink / raw)
  To: Zhang Rui; +Cc: Len Brown, linux-pm



On Sun, 6 Feb 2022, Zhang Rui wrote:

> Hi, Julia,
>
> Thanks for reporting this.
>
> On Sun, 2022-01-30 at 11:27 +0100, Julia Lawall wrote:
> > Hello,
> >
> > I was wondering whether the DRAM power consumption reported by
> > turbostat
> > is reliable on recent Intel machines?
> >
> > In particular, I observed that turbostat reported a high DRAM energy
> > comsumption on a machine (Intel 5128)
>
> are you sure it is 5128? google tells me that 5128 is pretty old, in
> around 2006.

Oops!  5218

Thanks for the pointer to the new version.  I will try that.

julia

>
> can you please paste the lscpu output?
>
> >  with persistent memory, but where
> > the persistent memory was not being used.  A colleague did an
> > experiemnt
> > on another machine with persistent memory, and reported:
> >
> > -----
> >
> > I didn't run the test on troll but on another server equipped with PM
> > where I was able to reproduce the bug and by reading directly the msr
> > registers, I see that:
> > CPU Energy units = 0.00006104J
> > DRAM Energy units = 0.00001526J
> >
> > However turbostat assumes that the DRAM  Energy units is 0.00006104J
> > when
> > it runs the computation to obtain Joules (hence the too-high value
> > returned by turbostat)
>
> And can you please try latest version of turbostat?
>
> The problem should be fixed by the below commit
>
> commit abdc75ab53b7fd2ef42c79e88cf0caf2d007c4f2
> Author:     Zhang Rui <rui.zhang@intel.com>
> AuthorDate: Thu Mar 11 10:05:13 2021 +0800
> Commit:     Len Brown <len.brown@intel.com>
> CommitDate: Tue May 4 18:23:14 2021 -0400
>
>     tools/power turbostat: Fix DRAM Energy Unit on SKX
>
>     SKX uses fixed DRAM Energy Unit, just like HSX and BDX.
>
>     Signed-off-by: Zhang Rui <rui.zhang@intel.com>
>     Signed-off-by: Len Brown <len.brown@intel.com>
>
> thanks,
> rui
> >
> > -----
> >
> > I see the code in turbostat that just uses the CPU energy units value
> > (rapl_dram_energy_units_probe), but I don't know what was the MSR
> > used to
> > collect the above information.  Overall, I am wondering if the DRAM
> > energy
> > consumption values are reliable in cases with and without persistent
> > memory.
> >
> > thanks,
> > julia
>
>

^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: DRAM power consumption with turbostat
  2022-02-06  8:29 ` Zhang Rui
  2022-02-06  8:34   ` Julia Lawall
@ 2022-02-06  9:32   ` Julia Lawall
  1 sibling, 0 replies; 4+ messages in thread
From: Julia Lawall @ 2022-02-06  9:32 UTC (permalink / raw)
  To: Zhang Rui; +Cc: Len Brown, linux-pm



On Sun, 6 Feb 2022, Zhang Rui wrote:

> Hi, Julia,
>
> Thanks for reporting this.
>
> On Sun, 2022-01-30 at 11:27 +0100, Julia Lawall wrote:
> > Hello,
> >
> > I was wondering whether the DRAM power consumption reported by
> > turbostat
> > is reliable on recent Intel machines?
> >
> > In particular, I observed that turbostat reported a high DRAM energy
> > comsumption on a machine (Intel 5128)
>
> are you sure it is 5128? google tells me that 5128 is pretty old, in
> around 2006.
>
> can you please paste the lscpu output?
>
> >  with persistent memory, but where
> > the persistent memory was not being used.  A colleague did an
> > experiemnt
> > on another machine with persistent memory, and reported:
> >
> > -----
> >
> > I didn't run the test on troll but on another server equipped with PM
> > where I was able to reproduce the bug and by reading directly the msr
> > registers, I see that:
> > CPU Energy units = 0.00006104J
> > DRAM Energy units = 0.00001526J
> >
> > However turbostat assumes that the DRAM  Energy units is 0.00006104J
> > when
> > it runs the computation to obtain Joules (hence the too-high value
> > returned by turbostat)
>
> And can you please try latest version of turbostat?
>
> The problem should be fixed by the below commit

The result indeed looks better, thanks!

The following just spins one on core for 5 seconds:

root@troll-4:/home/jlawall/spin# turbostat --version
turbostat version 20.09.30 - Len Brown <lenb@kernel.org>
root@troll-4:/home/jlawall/spin# head troll_turbo
5.011506 sec
Package	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	POLL	C1	C1E	C6	POLL%	C1%	C1E%	C6%	CPU%c1	CPU%c6	CoreTmp	PkgTmp	Pkg%pc2	Pkg%pc6	Pkg_J	RAM_J	PKG_%	RAM_%
-	-	-	61	1.59	3853	2294	5176	0	37	1493	153	2853	0.00	0.22	0.03	98.14	2.73	95.68	38	40	39.42	0.00	459.78	1336.73	0.00	0.00
0	0	0	4	0.36	1107	2294	298	0	0	0	0	564	0.00	0.00	0.00	99.69	4.86	94.78	36	40	78.85	0.00	228.32	664.98	0.00	0.00
0	0	32	1	0.06	1263	2294	49	0	0	1	14	46	0.00	0.00	0.69	99.26	5.15
0	1	4	0	0.03	1395	2294	29	0	0	8	0	44	0.00	0.59	0.00	99.36	1.03	98.94	34
0	1	36	0	0.02	1245	2294	22	0	0	0	1	25	0.00	0.00	0.02	99.95	1.04
0	2	8	0	0.02	1390	2294	22	0	0	0	0	27	0.00	0.00	0.00	99.97	0.37	99.61	32
0	2	40	0	0.03	1392	2294	24	0	0	0	0	36	0.00	0.00	0.00	99.96	0.36
0	3	12	0	0.02	1348	2294	23	0	0	0	0	26	0.00	0.00	0.00	99.97	0.23	99.76	36


root@troll-4:/home/jlawall/spin# myturbostat --version
turbostat version 21.05.04 - Len Brown <lenb@kernel.org>
root@troll-4:/home/jlawall/spin# head troll_new_turbo
5.014223 sec
Package	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IPC	IRQ	SMI	POLL	C1	C1E	C6	POLL%	C1%	C1E%	C6%	CPU%c1	CPU%c6	CoreTmp	PkgTmp	Pkg%pc2	Pkg%pc6	Pkg_J	RAM_J	PKG_%	RAM_%
-	-	-	61	1.58	3863	2294	1.00	2800	0	0	27	68	1927	0.00	0.02	0.02	98.37	2.14	96.28	37	39	46.29	0.00	457.55	335.42	0.00	0.00
0	0	0	5	0.44	1054	2295	0.63	351	0	0	0	0	668	0.00	0.00	0.00	99.63	4.53	95.04	34	39	92.52	0.00	225.67	166.64	0.00	0.00
0	0	32	1	0.05	1060	2295	0.12	25	0	0	0	1	25	0.00	0.00	0.02	99.97	4.91
0	1	4	0	0.01	1461	2295	0.38	6	0	0	0	1	6	0.00	0.00	0.01	99.97	0.19	99.80	33
0	1	36	0	0.02	1295	2295	0.56	19	0	0	0	2	15	0.00	0.00	0.04	99.94	0.18
0	2	8	0	0.01	1764	2295	0.31	9	0	0	0	1	10	0.00	0.00	0.01	99.98	0.18	99.81	30
0	2	40	0	0.01	1761	2295	0.32	10	0	0	0	2	15	0.00	0.00	0.03	99.95	0.18
0	3	12	0	0.03	1304	2295	0.41	32	0	0	0	21	42	0.00	0.00	0.62	99.35	0.95	99.02	35


It seems that the problem was not limited to machines with persuistent
memory.  Here are the same results on a 4-socket 6130:

Old version:
root@yeti-4:/home/jlawall/spin# head yeti_turbo
5.031399 sec
Package	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	POLL	C1	C1E	C6	POLL%	C1%	C1E%	C6%	CPU%c1	CPU%c6	CoreTmp	PkgTmp	Pkg%pc2	Pkg%pc6	PkgWatt	RAMWatt	PKG_%	RAM_%
-	-	-	29	0.80	3630	2096	4751	0	0	8	56	3585	0.00	0.00	0.01	99.23	1.23	97.97	30	31	72.53	0.00	193.12	111.17	0.00	0.00
0	0	0	4	0.41	1027	2095	290	0	0	0	10	536	0.00	0.00	0.66	98.98	4.19	95.40	28	31	92.84	0.00	44.66	25.49	0.00	0.00
0	0	64	0	0.04	1073	2095	9	0	0	0	0	9	0.00	0.00	0.00	99.98	4.56
0	1	8	0	0.01	1922	2095	9	0	0	0	0	10	0.00	0.00	0.00	99.99	0.23	99.76	29
0	1	72	0	0.02	1380	2095	27	0	0	0	0	29	0.00	0.00	0.00	99.98	0.22
0	2	16	0	0.02	1574	2095	24	0	0	0	0	26	0.00	0.00	0.00	99.98	0.91	99.07	29
0	2	80	0	0.03	1290	2095	55	0	0	8	1	45	0.00	0.58	0.04	99.36	0.90
0	3	24	0	0.02	1460	2095	34	0	0	0	8	30	0.00	0.00	0.59	99.39	0.83	99.15	29

New version:
root@yeti-4:/home/jlawall/spin# head yeti_new_turbo
5.030216 sec
Package	Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IPC	IRQ	SMI	POLL	C1	C1E	C6	POLL%	C1%	C1E%	C6%	CPU%c1	CPU%c6	CoreTmp	PkgTmp	Pkg%pc2	Pkg%pc6	PkgWatt	RAMWatt	PKG_%	RAM_%
-	-	-	51	1.49	3427	2107	1.04	23135	0	8	1524	2664	23370	0.00	0.07	0.22	98.79	5.71	92.80	31	32	59.79	0.00	205.38	30.23	0.00	0.00
0	0	0	7	0.52	1312	2095	0.56	397	0	0	0	3	690	0.00	0.00	0.01	99.53	5.58	93.90	29	32	80.47	0.00	45.69	6.78	0.00	0.00
0	0	64	1	0.04	1278	2095	0.10	9	0	0	0	0	11	0.00	0.00	0.00	99.99	6.06
0	1	8	3	0.12	2865	2095	0.71	44	0	0	0	1	51	0.00	0.00	0.00	99.88	0.64	99.24	30
0	1	72	0	0.01	1835	2095	0.30	18	0	0	0	0	18	0.00	0.00	0.00	99.99	0.75
0	2	16	0	0.02	1537	2095	0.16	8	0	0	0	0	9	0.00	0.00	0.00	99.99	12.87	87.11	29
0	2	80	38	2.08	1847	2095	0.16	8262	0	3	1426	98	6552	0.00	6.45	0.12	91.60	10.81
0	3	24	0	0.01	2043	2095	0.24	9	0	0	0	0	13	0.00	0.00	0.00	99.99	1.41	98.58	30

On the other hand, the energy consumption is still much higher on the
machine with the persistent memory, even though the persistent memory is
not being used.  But I guess it is reasonable that it should still consume
something.

thanks,
julia

^ permalink raw reply	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-02-06  9:32 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-30 10:27 DRAM power consumption with turbostat Julia Lawall
2022-02-06  8:29 ` Zhang Rui
2022-02-06  8:34   ` Julia Lawall
2022-02-06  9:32   ` Julia Lawall

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.