[Xen-devel] CPU frequency throttling based on the temperature

[Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

Hello,

My objective is to get CPU frequency throttling based on the
temperature in a Xen/OpenWRT(dom0) system.

After to expend hours reading Xen's wiki, mailing list archives,
commits, googling and asking in the IRC channel I'm coming here asking
for help because I hope there is something I miss and you could point
it.

My first question is, there is any way to do CPU frequency throttling
based on the temperature?

If the answer for first question is not, then there is any way to get
the CPU temperature from Dom0? (this way we could use Intel's thermald
and modify it to use xenpm to change the cpufreq).

If one and two are not, do you think that the right path is to modify
 linux/drivers/xen/xen-acpi-processor.c or you have any other ideas?

Thanks

-- 
Fredy Pulido,
Consultant en logiciel libre
Infrastructure, Infonuagique et architecture de systèmes
Savoir-faire Linux, Montréal, Qc
Bureau : (+ 1) 514 276-5468 p.410 

Message de confidentialité :
Ce courriel (de même que les fichiers joints) est strictement réservé à
l'usage de la personne ou de l'entité à qui il est adressé et peut
contenir de l'information privilégiée et confidentielle.
Toute divulgation, distribution ou copie de ce courriel est strictement
prohibée.
Si vous avez reçu ce courriel par erreur, veuillez nous en aviser sur-
le-champ, détruire toutes les copies et le supprimer de votre système
informatique.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Wed, Jul 24, 2019 at 10:01:40AM -0400, Fredy P. wrote:
> Hello,
> 
> My objective is to get CPU frequency throttling based on the
> temperature in a Xen/OpenWRT(dom0) system.
> 
> After to expend hours reading Xen's wiki, mailing list archives,
> commits, googling and asking in the IRC channel I'm coming here asking
> for help because I hope there is something I miss and you could point
> it.

That seems like an interesting project, I guess your focus is some
kind of low-power device? (not that it matters much for the context of
the question).

Anyway, thanks for your interest on Xen and ways to improve it!

> My first question is, there is any way to do CPU frequency throttling
> based on the temperature?

I don't think there's such governor ATM implemented in Xen, the more
that I think all frequency throttling is supposed to be done by dom0
using xenpm, but not Xen itself?

> If the answer for first question is not, then there is any way to get
> the CPU temperature from Dom0? (this way we could use Intel's thermald
> and modify it to use xenpm to change the cpufreq).

What hardware interface does thermald (or the driver in Linux if
there's one) use to get the temperature data?

Is it exposed in a mmio region somewhere? Or maybe exposed as a pci
device?

> If one and two are not, do you think that the right path is to modify
>  linux/drivers/xen/xen-acpi-processor.c or you have any other ideas?

I think it depends on how this data is exposed by the hardware.

> Thanks
> 
> -- 
> Fredy Pulido,
> Consultant en logiciel libre
> Infrastructure, Infonuagique et architecture de systèmes
> Savoir-faire Linux, Montréal, Qc
> Bureau : (+ 1) 514 276-5468 p.410 
> 
> Message de confidentialité :
> Ce courriel (de même que les fichiers joints) est strictement réservé à
> l'usage de la personne ou de l'entité à qui il est adressé et peut
> contenir de l'information privilégiée et confidentielle.
> Toute divulgation, distribution ou copie de ce courriel est strictement
> prohibée.
> Si vous avez reçu ce courriel par erreur, veuillez nous en aviser sur-
> le-champ, détruire toutes les copies et le supprimer de votre système
> informatique.

Adding a confidentiality footer to an email sent to a public mailing
list is pointless, do you know the whole mailing list archives are
publicly accessible at:

https://lists.xenproject.org/archives/html/xen-devel/

For anyone to read?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Jan Beulich]

On 24.07.2019 16:36, Roger Pau Monné  wrote:
> On Wed, Jul 24, 2019 at 10:01:40AM -0400, Fredy P. wrote:
>> My objective is to get CPU frequency throttling based on the
>> temperature in a Xen/OpenWRT(dom0) system.
>>
>> After to expend hours reading Xen's wiki, mailing list archives,
>> commits, googling and asking in the IRC channel I'm coming here asking
>> for help because I hope there is something I miss and you could point
>> it.
> 
> That seems like an interesting project, I guess your focus is some
> kind of low-power device? (not that it matters much for the context of
> the question).
> 
> Anyway, thanks for your interest on Xen and ways to improve it!
> 
>> My first question is, there is any way to do CPU frequency throttling
>> based on the temperature?
> 
> I don't think there's such governor ATM implemented in Xen, the more
> that I think all frequency throttling is supposed to be done by dom0
> using xenpm, but not Xen itself?

The original authors of P- and C-state handling look to have
assumed that T-state handling should work similarly, i.e. by
Dom0 uploading relevant data. See public/platform.h starting at

#define XENPF_set_processor_pminfo      54

where in particular you'll find

#define XEN_PM_TX   2

Additionally "done by dom0 using xenpm" still means Xen carrying
out the actual operation. And xenpm of course is only an
auxiliary tool by which an admin can influence overall behavior,
it doesn't replace a proper governor.

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

-- 
Fredy Pulido,
Consultant en logiciel libre
Infrastructure, Infonuagique et architecture de systèmes
Savoir-faire Linux, Montréal, Qc
Bureau : (+ 1) 514 276-5468 p.410 

Message de confidentialité :
Ce courriel (de même que les fichiers joints) est strictement réservé à l'usage de la personne ou de l'entité à qui il est adressé et peut contenir de l'information privilégiée et confidentielle.
Toute divulgation, distribution ou copie de ce courriel est strictement prohibée.
Si vous avez reçu ce courriel par erreur, veuillez nous en aviser sur-le-champ, détruire toutes les copies et le supprimer de votre système informatique.

-------- Forwarded Message --------
From: Fredy P. <fredy.pulido@savoirfairelinux.com>
To: Roger Pau Monné <roger.pau@citrix.com>
Subject: Re: [Xen-devel] CPU frequency throttling based on the
temperature
Date: Wed, 24 Jul 2019 11:00:51 -0400

Hello, answering between lines
On Wed, 2019-07-24 at 16:36 +0200, Roger Pau Monné wrote:
> On Wed, Jul 24, 2019 at 10:01:40AM -0400, Fredy P. wrote:
> > Hello,
> > 
> > My objective is to get CPU frequency throttling based on the
> > temperature in a Xen/OpenWRT(dom0) system.
> > 
> > After to expend hours reading Xen's wiki, mailing list archives,
> > commits, googling and asking in the IRC channel I'm coming here
> > asking
> > for help because I hope there is something I miss and you could
> > point
> > it.
> That seems like an interesting project, I guess your focus is some
> kind of low-power device? (not that it matters much for the context
> of
> the question).
Is a X86 server for aircrafts, we have it running Linux and will add
Xen to have a VM running inside, is a kind of big embedded system.

> Anyway, thanks for your interest on Xen and ways to improve it!
> 
> > My first question is, there is any way to do CPU frequency
> > throttling
> > based on the temperature?
> 
> I don't think there's such governor ATM implemented in Xen, the more
> that I think all frequency throttling is supposed to be done by dom0
> using xenpm, but not Xen itself?

Yes that is clear for us but to do active thermal throttling need to be
capable to read the CPU temperature.

> > If the answer for first question is not, then there is any way to
> > get
> > the CPU temperature from Dom0? (this way we could use Intel's
> > thermald
> > and modify it to use xenpm to change the cpufreq).
> 
> What hardware interface does thermald (or the driver in Linux if
> there's one) use to get the temperature data?

That is the main problem, in a POC we did years ago for our client
using Xen 4.8.x we took it from /sys/class/hwmon but that was not
accurate and Xen removes it on 4.9.x. this is the question I'm doing
here.

> Is it exposed in a mmio region somewhere? Or maybe exposed as a pci
> device?
> 
> > If one and two are not, do you think that the right path is to
> > modify
> >  linux/drivers/xen/xen-acpi-processor.c or you have any other
> > ideas?
> 
> I think it depends on how this data is exposed by the hardware.

Is a Intel(R) Core(TM) i5-5350U CPU @ 1.80GHz processor, when you say
the hardware is the board itself? or the processor?

> > Thanks
> > 
> > -- 
> > Fredy Pulido,
> > Consultant en logiciel libre
> > Infrastructure, Infonuagique et architecture de systèmes
> > Savoir-faire Linux, Montréal, Qc
> > Bureau : (+ 1) 514 276-5468 p.410 
> > 
> > Message de confidentialité :
> > Ce courriel (de même que les fichiers joints) est strictement
> > réservé à
> > l'usage de la personne ou de l'entité à qui il est adressé et peut
> > contenir de l'information privilégiée et confidentielle.
> > Toute divulgation, distribution ou copie de ce courriel est
> > strictement
> > prohibée.
> > Si vous avez reçu ce courriel par erreur, veuillez nous en aviser
> > sur-
> > le-champ, détruire toutes les copies et le supprimer de votre
> > système
> > informatique.
> 
> Adding a confidentiality footer to an email sent to a public mailing
> list is pointless, do you know the whole mailing list archives are
> publicly accessible at:
> 
> https://lists.xenproject.org/archives/html/xen-devel/
> 
> For anyone to read?
> 
> Roger.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Wed, Jul 24, 2019 at 02:47:19PM +0000, Jan Beulich wrote:
> On 24.07.2019 16:36, Roger Pau Monné  wrote:
> > On Wed, Jul 24, 2019 at 10:01:40AM -0400, Fredy P. wrote:
> >> My objective is to get CPU frequency throttling based on the
> >> temperature in a Xen/OpenWRT(dom0) system.
> >>
> >> After to expend hours reading Xen's wiki, mailing list archives,
> >> commits, googling and asking in the IRC channel I'm coming here asking
> >> for help because I hope there is something I miss and you could point
> >> it.
> > 
> > That seems like an interesting project, I guess your focus is some
> > kind of low-power device? (not that it matters much for the context of
> > the question).
> > 
> > Anyway, thanks for your interest on Xen and ways to improve it!
> > 
> >> My first question is, there is any way to do CPU frequency throttling
> >> based on the temperature?
> > 
> > I don't think there's such governor ATM implemented in Xen, the more
> > that I think all frequency throttling is supposed to be done by dom0
> > using xenpm, but not Xen itself?
> 
> The original authors of P- and C-state handling look to have
> assumed that T-state handling should work similarly, i.e. by
> Dom0 uploading relevant data. See public/platform.h starting at
> 
> #define XENPF_set_processor_pminfo      54
> 
> where in particular you'll find
> 
> #define XEN_PM_TX   2

OK, I assumed the question was about reading the CPU temperature and
then changing the frequency of the CPU, but not related to T-states.

FWIW, there's an Intel article about T-states from 2013:

https://software.intel.com/en-us/blogs/2013/10/15/c-states-p-states-where-the-heck-are-those-t-states

That claims T-states are basically dead, and no modern processors
support them.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Wed, Jul 24, 2019 at 11:25:55AM -0400, Fredy P. wrote:
> Hello, answering between lines
> On Wed, 2019-07-24 at 16:36 +0200, Roger Pau Monné wrote:
> > On Wed, Jul 24, 2019 at 10:01:40AM -0400, Fredy P. wrote:
> > > If the answer for first question is not, then there is any way to
> > > get
> > > the CPU temperature from Dom0? (this way we could use Intel's
> > > thermald
> > > and modify it to use xenpm to change the cpufreq).
> > 
> > What hardware interface does thermald (or the driver in Linux if
> > there's one) use to get the temperature data?
> 
> That is the main problem, in a POC we did years ago for our client
> using Xen 4.8.x we took it from /sys/class/hwmon but that was not
> accurate and Xen removes it on 4.9.x. this is the question I'm doing
> here.
> 
> > Is it exposed in a mmio region somewhere? Or maybe exposed as a pci
> > device?

You haven't answered this, which I think it's quite relevant in order
to know how to move forward. How is the temperature data exposed by
the hardware will likely determine how to read it, and whether Xen or
dom0 should access it.

If such data (or part of it) comes from ACPI dynamic tables then it
must be dom0 the one that reads it, if it is otherwise exposed as a
PCI device or maybe as a mmio region somewhere it could be Xen the
one to read such information.

> > > If one and two are not, do you think that the right path is to
> > > modify
> > >  linux/drivers/xen/xen-acpi-processor.c or you have any other
> > > ideas?
> > 
> > I think it depends on how this data is exposed by the hardware.
> 
> Is a Intel(R) Core(TM) i5-5350U CPU @ 1.80GHz processor, when you say
> the hardware is the board itself? or the processor?

The model itself is not that relevant, but rather how is the
temperature exposed by the CPU, see my comment above.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Jan Beulich]

On 24.07.2019 17:35, Roger Pau Monné  wrote:
> On Wed, Jul 24, 2019 at 02:47:19PM +0000, Jan Beulich wrote:
>> On 24.07.2019 16:36, Roger Pau Monné  wrote:
>>> On Wed, Jul 24, 2019 at 10:01:40AM -0400, Fredy P. wrote:
>>>> My objective is to get CPU frequency throttling based on the
>>>> temperature in a Xen/OpenWRT(dom0) system.
>>>>
>>>> After to expend hours reading Xen's wiki, mailing list archives,
>>>> commits, googling and asking in the IRC channel I'm coming here asking
>>>> for help because I hope there is something I miss and you could point
>>>> it.
>>>
>>> That seems like an interesting project, I guess your focus is some
>>> kind of low-power device? (not that it matters much for the context of
>>> the question).
>>>
>>> Anyway, thanks for your interest on Xen and ways to improve it!
>>>
>>>> My first question is, there is any way to do CPU frequency throttling
>>>> based on the temperature?
>>>
>>> I don't think there's such governor ATM implemented in Xen, the more
>>> that I think all frequency throttling is supposed to be done by dom0
>>> using xenpm, but not Xen itself?
>>
>> The original authors of P- and C-state handling look to have
>> assumed that T-state handling should work similarly, i.e. by
>> Dom0 uploading relevant data. See public/platform.h starting at
>>
>> #define XENPF_set_processor_pminfo      54
>>
>> where in particular you'll find
>>
>> #define XEN_PM_TX   2
> 
> OK, I assumed the question was about reading the CPU temperature and
> then changing the frequency of the CPU, but not related to T-states.

Well, except that iirc T-states are (were) a means to control this via
some governor, rather than "manually".

Obtaining the CPU temperature should work (perhaps with some tweaks)
the same way under Xen or on bare hardware. I also don't think hwmon
devices get unintentionally "unexposed" when running under Xen. Their
drivers may be written in ways that make them not work properly when
run under Xen, though.

> FWIW, there's an Intel article about T-states from 2013:
> 
> https://software.intel.com/en-us/blogs/2013/10/15/c-states-p-states-where-the-heck-are-those-t-states
> 
> That claims T-states are basically dead, and no modern processors
> support them.

Interesting; I wasn't aware of this.

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

On Wed, 2019-07-24 at 17:35 +0200, Roger Pau Monné wrote:
> OK, I assumed the question was about reading the CPU temperature and
> then changing the frequency of the CPU, but not related to T-states.

Yes, the question is about reading the CPU temperature, How to change
it is already "solved" (we will modify thermald to use xenpm)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:

> > > What hardware interface does thermald (or the driver in Linux if
> > > there's one) use to get the temperature data?

In our initial POC using Xen 4.8.x we where using Linux coretemp driver
reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6

Our ideal fix is to get back the /class/sys/hwmon/hwmon0 to feed
thermald in the same way we did in our POC. We understand that just
reverting that commit is not the right way, the question is what is the
right way?

> > > Is it exposed in a mmio region somewhere? Or maybe exposed as a
> > > pci
> > > device?
> 
> You haven't answered this, which I think it's quite relevant in order
> to know how to move forward. How is the temperature data exposed by
> the hardware will likely determine how to read it, and whether Xen or
> dom0 should access it.

I'm not sure if this is the right answer for that question but, we are
talking about the DTS (Digital Thermal Sensor) provided by the Intel
processor[1]

> If such data (or part of it) comes from ACPI dynamic tables then it
> must be dom0 the one that reads it, if it is otherwise exposed as a
> PCI device or maybe as a mmio region somewhere it could be Xen the
> one to read such information.

I'm not sure if I'll said something stupid but after to overview [1]
and commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6 my guess is we
retrieve DTS via Model Specific Register (MSR) and it is the x86/cpuid
leaf 0x6.

I think is a kind of gray zone because is the processor itself
(supposed to be handled by Xen) but is a kind of I/O (supposed to be
dom0)

> > > I think it depends on how this data is exposed by the hardware.
> > 
> > Is a Intel(R) Core(TM) i5-5350U CPU @ 1.80GHz processor, when you
> > say
> > the hardware is the board itself? or the processor?
> 
> The model itself is not that relevant, but rather how is the
> temperature exposed by the CPU, see my comment above.

I think could be because the DTS is inside the processor itself to be
honest I'm not a specialist in thermal systems for processors but it
looks Intel have good documents about the subject, by example [2]

> Thanks, Roger.

[1] Pag 68 
https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/5th-gen-core-family-datasheet-vol-1.pdf
[2] 
https://digitallibrary.intel.com/content/dam/ccl/public/cpu-monitoring-dts-peci-paper.pdf?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjb250ZW50SWQiOiI2MDA0MjAiLCJlbnRlcnByaXNlSWQiOiIyMDguODguMTEwLjQ2IiwiQUNDVF9OTSI6IiIsIkNOREFfTkJSIjoiIiwiaWF0IjoxNTY0MDU1NjkwfQ.TBPkSTO1CtkOZ1TqtbVe6IljK7hr6ius2iDDfj_SUEI


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Jan Beulich]

On 25.07.2019 14:44,  Fredy P.  wrote:
> On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
>>>> What hardware interface does thermald (or the driver in Linux if
>>>> there's one) use to get the temperature data?
> 
> In our initial POC using Xen 4.8.x we where using Linux coretemp driver
> reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
> deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6

Hmm, I wouldn't call this deprecation, but a regression. I would
say we want to re-expose this leaf to Dom0, the more that the
commit also only mentions unprivileged domains. Andrew?

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

On IRC Andrew told me

"I can see why thermal monitoring is useful from dom0, and we should figure out some way to fix it, but simply undoing that change isn't the right thing to do"
-- 
Fredy Pulido,
Consultant en logiciel libre
Infrastructure, Infonuagique et architecture de systèmes
Savoir-faire Linux, Montréal, Qc
Bureau : (+ 1) 514 276-5468 p.410 

On Thu, 2019-07-25 at 12:54 +0000, Jan Beulich wrote:
> Hmm, I wouldn't call this deprecation, but a regression. I would
> say we want to re-expose this leaf to Dom0, the more that the
> commit also only mentions unprivileged domains. Andrew?



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
> On 25.07.2019 14:44,  Fredy P.  wrote:
> > On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
> >>>> What hardware interface does thermald (or the driver in Linux if
> >>>> there's one) use to get the temperature data?
> > 
> > In our initial POC using Xen 4.8.x we where using Linux coretemp driver
> > reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
> > deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> 
> Hmm, I wouldn't call this deprecation, but a regression. I would
> say we want to re-expose this leaf to Dom0, the more that the
> commit also only mentions unprivileged domains. Andrew?

AFAICT from the documents provided by Fredy the temperature is read
from a MSR that reports the current temperature of the core on which
the MSR is read from. When running on Xen this will only work
correctly if dom0 is given the same vCPUs as pCPUs and those are
identity pinned.

Not sure how common this MSR interface is in order to read thermal
values, if the interface it's common maybe it's something that could
be implemented in Xen, and exported somehow to dom0, maybe using
sysctl?

Or else having an hypercall that allows dom0 to request Xen to execute
MSR read/writes on a given pCPU.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

On Thu, 2019-07-25 at 15:13 +0200, Roger Pau Monné wrote:
> On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
> > On 25.07.2019 14:44,  Fredy P.  wrote:
> > > On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
> > > > > > What hardware interface does thermald (or the driver in
> > > > > > Linux if
> > > > > > there's one) use to get the temperature data?
> > > 
> > > In our initial POC using Xen 4.8.x we where using Linux coretemp
> > > driver
> > > reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
> > > deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> > 
> > Hmm, I wouldn't call this deprecation, but a regression. I would
> > say we want to re-expose this leaf to Dom0, the more that the
> > commit also only mentions unprivileged domains. Andrew?
> 
> AFAICT from the documents provided by Fredy the temperature is read
> from a MSR that reports the current temperature of the core on which
> the MSR is read from. When running on Xen this will only work
> correctly if dom0 is given the same vCPUs as pCPUs and those are
> identity pinned.

I just want to be sure I got it correctly, by saying "When running on
Xen this will only work correctly if ..." means in a future
implementation or that right now could work if I pin this v/pCPUS?

> Not sure how common this MSR interface is in order to read thermal
> values, if the interface it's common maybe it's something that could
> be implemented in Xen, and exported somehow to dom0, maybe using
> sysctl?
> 
> Or else having an hypercall that allows dom0 to request Xen to
> execute
> MSR read/writes on a given pCPU.
> 
> Thanks, Roger.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Jan Beulich]

On 25.07.2019 15:13, Roger Pau Monné  wrote:
> On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
>> On 25.07.2019 14:44,  Fredy P.  wrote:
>>> On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
>>>>>> What hardware interface does thermald (or the driver in Linux if
>>>>>> there's one) use to get the temperature data?
>>>
>>> In our initial POC using Xen 4.8.x we where using Linux coretemp driver
>>> reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
>>> deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
>>
>> Hmm, I wouldn't call this deprecation, but a regression. I would
>> say we want to re-expose this leaf to Dom0, the more that the
>> commit also only mentions unprivileged domains. Andrew?
> 
> AFAICT from the documents provided by Fredy the temperature is read
> from a MSR that reports the current temperature of the core on which
> the MSR is read from. When running on Xen this will only work
> correctly if dom0 is given the same vCPUs as pCPUs and those are
> identity pinned.
> 
> Not sure how common this MSR interface is in order to read thermal
> values, if the interface it's common maybe it's something that could
> be implemented in Xen, and exported somehow to dom0, maybe using
> sysctl?
> 
> Or else having an hypercall that allows dom0 to request Xen to execute
> MSR read/writes on a given pCPU.

This would look to require just a small extension to
XEN_RESOURCE_OP_MSR_READ. Question is whether the Linux driver
maintainers would accept a change using this Xen-specific
alternative access mechanism (in whatever shape).

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Thu, Jul 25, 2019 at 09:29:01AM -0400, Fredy P. wrote:
> On Thu, 2019-07-25 at 15:13 +0200, Roger Pau Monné wrote:
> > On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
> > > On 25.07.2019 14:44,  Fredy P.  wrote:
> > > > On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
> > > > > > > What hardware interface does thermald (or the driver in
> > > > > > > Linux if
> > > > > > > there's one) use to get the temperature data?
> > > > 
> > > > In our initial POC using Xen 4.8.x we where using Linux coretemp
> > > > driver
> > > > reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
> > > > deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> > > 
> > > Hmm, I wouldn't call this deprecation, but a regression. I would
> > > say we want to re-expose this leaf to Dom0, the more that the
> > > commit also only mentions unprivileged domains. Andrew?
> > 
> > AFAICT from the documents provided by Fredy the temperature is read
> > from a MSR that reports the current temperature of the core on which
> > the MSR is read from. When running on Xen this will only work
> > correctly if dom0 is given the same vCPUs as pCPUs and those are
> > identity pinned.
> 
> I just want to be sure I got it correctly, by saying "When running on
> Xen this will only work correctly if ..." means in a future
> implementation or that right now could work if I pin this v/pCPUS?

No, right now there's no way to get this data from dom0, regardless of
the pinning.

The commit you mention simply removes exposing the feature on CPUID,
but I'm not sure whether access to the actual MSR is also forbidden. I
think so since we do MSR white listing IIRC, and I don't seem to find
MSR_IA32_THERM_STATUS white listed anywhere.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Jan Beulich]

On 25.07.2019 15:47, Roger Pau Monné  wrote:
> On Thu, Jul 25, 2019 at 09:29:01AM -0400, Fredy P. wrote:
>> On Thu, 2019-07-25 at 15:13 +0200, Roger Pau Monné wrote:
>>> On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
>>>> On 25.07.2019 14:44,  Fredy P.  wrote:
>>>>> On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
>>>>>>>> What hardware interface does thermald (or the driver in
>>>>>>>> Linux if
>>>>>>>> there's one) use to get the temperature data?
>>>>>
>>>>> In our initial POC using Xen 4.8.x we where using Linux coretemp
>>>>> driver
>>>>> reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
>>>>> deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
>>>>
>>>> Hmm, I wouldn't call this deprecation, but a regression. I would
>>>> say we want to re-expose this leaf to Dom0, the more that the
>>>> commit also only mentions unprivileged domains. Andrew?
>>>
>>> AFAICT from the documents provided by Fredy the temperature is read
>>> from a MSR that reports the current temperature of the core on which
>>> the MSR is read from. When running on Xen this will only work
>>> correctly if dom0 is given the same vCPUs as pCPUs and those are
>>> identity pinned.
>>
>> I just want to be sure I got it correctly, by saying "When running on
>> Xen this will only work correctly if ..." means in a future
>> implementation or that right now could work if I pin this v/pCPUS?
> 
> No, right now there's no way to get this data from dom0, regardless of
> the pinning.

Of course you can, using the MSR "device" Linux optionally
provides (plus perhaps the rdmsr utility from the msr-tools
package).

> The commit you mention simply removes exposing the feature on CPUID,
> but I'm not sure whether access to the actual MSR is also forbidden. I
> think so since we do MSR white listing IIRC, and I don't seem to find
> MSR_IA32_THERM_STATUS white listed anywhere.

At least for PV we continue to let everything not specially
handled shine through - see the bottom of pv/emul-pro-op.c:read_msr().

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Thu, Jul 25, 2019 at 01:43:34PM +0000, Jan Beulich wrote:
> On 25.07.2019 15:13, Roger Pau Monné  wrote:
> > On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
> >> On 25.07.2019 14:44,  Fredy P.  wrote:
> >>> On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
> >>>>>> What hardware interface does thermald (or the driver in Linux if
> >>>>>> there's one) use to get the temperature data?
> >>>
> >>> In our initial POC using Xen 4.8.x we where using Linux coretemp driver
> >>> reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
> >>> deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> >>
> >> Hmm, I wouldn't call this deprecation, but a regression. I would
> >> say we want to re-expose this leaf to Dom0, the more that the
> >> commit also only mentions unprivileged domains. Andrew?
> > 
> > AFAICT from the documents provided by Fredy the temperature is read
> > from a MSR that reports the current temperature of the core on which
> > the MSR is read from. When running on Xen this will only work
> > correctly if dom0 is given the same vCPUs as pCPUs and those are
> > identity pinned.
> > 
> > Not sure how common this MSR interface is in order to read thermal
> > values, if the interface it's common maybe it's something that could
> > be implemented in Xen, and exported somehow to dom0, maybe using
> > sysctl?
> > 
> > Or else having an hypercall that allows dom0 to request Xen to execute
> > MSR read/writes on a given pCPU.
> 
> This would look to require just a small extension to
> XEN_RESOURCE_OP_MSR_READ. Question is whether the Linux driver
> maintainers would accept a change using this Xen-specific
> alternative access mechanism (in whatever shape).

Right, there's also the fact that all pCPUs should be reported in the
thermal driver, while dom0 might have less vCPUs than pCPUs on the
system.

Do you think you can take a look into this Fredy?

It will involve looking into the Linux driver in order to make use of
an hypercall instead of a rdmsr. I think it should be fine to expose
the CPUID leaf to dom0 as long as reads are performed from the
hypercall, in order to assure that Linux gets consistent values.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Thu, Jul 25, 2019 at 01:59:22PM +0000, Jan Beulich wrote:
> On 25.07.2019 15:47, Roger Pau Monné  wrote:
> > On Thu, Jul 25, 2019 at 09:29:01AM -0400, Fredy P. wrote:
> >> On Thu, 2019-07-25 at 15:13 +0200, Roger Pau Monné wrote:
> >>> On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
> >>>> On 25.07.2019 14:44,  Fredy P.  wrote:
> >>>>> On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
> >>>>>>>> What hardware interface does thermald (or the driver in
> >>>>>>>> Linux if
> >>>>>>>> there's one) use to get the temperature data?
> >>>>>
> >>>>> In our initial POC using Xen 4.8.x we where using Linux coretemp
> >>>>> driver
> >>>>> reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
> >>>>> deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> >>>>
> >>>> Hmm, I wouldn't call this deprecation, but a regression. I would
> >>>> say we want to re-expose this leaf to Dom0, the more that the
> >>>> commit also only mentions unprivileged domains. Andrew?
> >>>
> >>> AFAICT from the documents provided by Fredy the temperature is read
> >>> from a MSR that reports the current temperature of the core on which
> >>> the MSR is read from. When running on Xen this will only work
> >>> correctly if dom0 is given the same vCPUs as pCPUs and those are
> >>> identity pinned.
> >>
> >> I just want to be sure I got it correctly, by saying "When running on
> >> Xen this will only work correctly if ..." means in a future
> >> implementation or that right now could work if I pin this v/pCPUS?
> > 
> > No, right now there's no way to get this data from dom0, regardless of
> > the pinning.
> 
> Of course you can, using the MSR "device" Linux optionally
> provides (plus perhaps the rdmsr utility from the msr-tools
> package).

But you won't get coherent results, since the vCPU might be jumping
from pCPU to pCPU, thus returning values from multiple different pCPUs
regardless of whether all rdmsr have been executed from the same vCPU
from dom0 PoV.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

-- 
Fredy Pulido,
Consultant en logiciel libre
Infrastructure, Infonuagique et architecture de systèmes
Savoir-faire Linux, Montréal, Qc
Bureau : (+ 1) 514 276-5468 p.410 

Message de confidentialité :
Ce courriel (de même que les fichiers joints) est strictement réservé à
l'usage de la personne ou de l'entité à qui il est adressé et peut
contenir de l'information privilégiée et confidentielle.
Toute divulgation, distribution ou copie de ce courriel est strictement
prohibée.
Si vous avez reçu ce courriel par erreur, veuillez nous en aviser sur-
le-champ, détruire toutes les copies et le supprimer de votre système
informatique.

On Thu, 2019-07-25 at 16:07 +0200, Roger Pau Monné wrote:
> On Thu, Jul 25, 2019 at 01:43:34PM +0000, Jan Beulich wrote:
> > On 25.07.2019 15:13, Roger Pau Monné  wrote:
> > > On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
> > > > On 25.07.2019 14:44,  Fredy P.  wrote:
> > > > > On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
> > > > > > > > What hardware interface does thermald (or the driver in
> > > > > > > > Linux if
> > > > > > > > there's one) use to get the temperature data?
> > > > > 
> > > > > In our initial POC using Xen 4.8.x we where using Linux
> > > > > coretemp driver
> > > > > reading by example /class/sys/hwmon/hwmon0/temp3_input but it
> > > > > got
> > > > > deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> > > > 
> > > > Hmm, I wouldn't call this deprecation, but a regression. I
> > > > would
> > > > say we want to re-expose this leaf to Dom0, the more that the
> > > > commit also only mentions unprivileged domains. Andrew?
> > > 
> > > AFAICT from the documents provided by Fredy the temperature is
> > > read
> > > from a MSR that reports the current temperature of the core on
> > > which
> > > the MSR is read from. When running on Xen this will only work
> > > correctly if dom0 is given the same vCPUs as pCPUs and those are
> > > identity pinned.
> > > 
> > > Not sure how common this MSR interface is in order to read
> > > thermal
> > > values, if the interface it's common maybe it's something that
> > > could
> > > be implemented in Xen, and exported somehow to dom0, maybe using
> > > sysctl?
> > > 
> > > Or else having an hypercall that allows dom0 to request Xen to
> > > execute
> > > MSR read/writes on a given pCPU.
> > 
> > This would look to require just a small extension to
> > XEN_RESOURCE_OP_MSR_READ. Question is whether the Linux driver
> > maintainers would accept a change using this Xen-specific
> > alternative access mechanism (in whatever shape).
> 
> Right, there's also the fact that all pCPUs should be reported in the
> thermal driver, while dom0 might have less vCPUs than pCPUs on the
> system.
> 
> Do you think you can take a look into this Fredy?

I think I can but will takes long, short history I'm "just a sysadmin
doing a security update" then I'll try to find if some one of the guys
that wrote low level code to help me with this.

> It will involve looking into the Linux driver in order to make use of
> an hypercall instead of a rdmsr. I think it should be fine to expose
> the CPUID leaf to dom0 as long as reads are performed from the
> hypercall, in order to assure that Linux gets consistent values.
> 
> Roger.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Jan Beulich]

On 25.07.2019 16:17, Roger Pau Monné  wrote:
> On Thu, Jul 25, 2019 at 01:59:22PM +0000, Jan Beulich wrote:
>> On 25.07.2019 15:47, Roger Pau Monné  wrote:
>>> On Thu, Jul 25, 2019 at 09:29:01AM -0400, Fredy P. wrote:
>>>> On Thu, 2019-07-25 at 15:13 +0200, Roger Pau Monné wrote:
>>>>> On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
>>>>>> On 25.07.2019 14:44,  Fredy P.  wrote:
>>>>>>> On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
>>>>>>>>>> What hardware interface does thermald (or the driver in
>>>>>>>>>> Linux if
>>>>>>>>>> there's one) use to get the temperature data?
>>>>>>>
>>>>>>> In our initial POC using Xen 4.8.x we where using Linux coretemp
>>>>>>> driver
>>>>>>> reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
>>>>>>> deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
>>>>>>
>>>>>> Hmm, I wouldn't call this deprecation, but a regression. I would
>>>>>> say we want to re-expose this leaf to Dom0, the more that the
>>>>>> commit also only mentions unprivileged domains. Andrew?
>>>>>
>>>>> AFAICT from the documents provided by Fredy the temperature is read
>>>>> from a MSR that reports the current temperature of the core on which
>>>>> the MSR is read from. When running on Xen this will only work
>>>>> correctly if dom0 is given the same vCPUs as pCPUs and those are
>>>>> identity pinned.
>>>>
>>>> I just want to be sure I got it correctly, by saying "When running on
>>>> Xen this will only work correctly if ..." means in a future
>>>> implementation or that right now could work if I pin this v/pCPUS?
>>>
>>> No, right now there's no way to get this data from dom0, regardless of
>>> the pinning.
>>
>> Of course you can, using the MSR "device" Linux optionally
>> provides (plus perhaps the rdmsr utility from the msr-tools
>> package).
> 
> But you won't get coherent results, since the vCPU might be jumping
> from pCPU to pCPU, thus returning values from multiple different pCPUs
> regardless of whether all rdmsr have been executed from the same vCPU
> from dom0 PoV.

I don't understand. Earlier you said "regardless of the pinning".
That's what my response was to, i.e. I was implying vCPU-s to be
pinned.

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Roger Pau Monné]

On Thu, Jul 25, 2019 at 02:31:40PM +0000, Jan Beulich wrote:
> On 25.07.2019 16:17, Roger Pau Monné  wrote:
> > On Thu, Jul 25, 2019 at 01:59:22PM +0000, Jan Beulich wrote:
> >> On 25.07.2019 15:47, Roger Pau Monné  wrote:
> >>> On Thu, Jul 25, 2019 at 09:29:01AM -0400, Fredy P. wrote:
> >>>> On Thu, 2019-07-25 at 15:13 +0200, Roger Pau Monné wrote:
> >>>>> On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich wrote:
> >>>>>> On 25.07.2019 14:44,  Fredy P.  wrote:
> >>>>>>> On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné wrote:
> >>>>>>>>>> What hardware interface does thermald (or the driver in
> >>>>>>>>>> Linux if
> >>>>>>>>>> there's one) use to get the temperature data?
> >>>>>>>
> >>>>>>> In our initial POC using Xen 4.8.x we where using Linux coretemp
> >>>>>>> driver
> >>>>>>> reading by example /class/sys/hwmon/hwmon0/temp3_input but it got
> >>>>>>> deprecated at commit 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> >>>>>>
> >>>>>> Hmm, I wouldn't call this deprecation, but a regression. I would
> >>>>>> say we want to re-expose this leaf to Dom0, the more that the
> >>>>>> commit also only mentions unprivileged domains. Andrew?
> >>>>>
> >>>>> AFAICT from the documents provided by Fredy the temperature is read
> >>>>> from a MSR that reports the current temperature of the core on which
> >>>>> the MSR is read from. When running on Xen this will only work
> >>>>> correctly if dom0 is given the same vCPUs as pCPUs and those are
> >>>>> identity pinned.
> >>>>
> >>>> I just want to be sure I got it correctly, by saying "When running on
> >>>> Xen this will only work correctly if ..." means in a future
> >>>> implementation or that right now could work if I pin this v/pCPUS?
> >>>
> >>> No, right now there's no way to get this data from dom0, regardless of
> >>> the pinning.
> >>
> >> Of course you can, using the MSR "device" Linux optionally
> >> provides (plus perhaps the rdmsr utility from the msr-tools
> >> package).
> > 
> > But you won't get coherent results, since the vCPU might be jumping
> > from pCPU to pCPU, thus returning values from multiple different pCPUs
> > regardless of whether all rdmsr have been executed from the same vCPU
> > from dom0 PoV.
> 
> I don't understand. Earlier you said "regardless of the pinning".
> That's what my response was to, i.e. I was implying vCPU-s to be
> pinned.

Oh sorry, that was me not taking into account the earlier context, you
are right. To summarize and make things easier for Fredy I think the
options are:

 - Create dom0 with vCPUs == pCPUs and identity pin them. Then you
   *could* expose CPUID leaf 6 to dom0 and things should be OK IMO,
   either when using the Linux driver or when reading values directly
   from user-space using the MSR device pointed out by Jan.

 - Modify the Linux thermal driver to report the temperature for all
   pCPUs (which might be different than dom0 vCPUs) using
   XENPF_resource_op and XEN_RESOURCE_OP_MSR_READ. AFAICT you will
   also need to expose CPUID leaf 6 to dom0 so that the thermal driver
   attaches.

 - Import a thermal driver into Xen and expose the thermal data
   somewhere, ie: a XENPF hypercall maybe.

Maybe someone can come up with more ideas, but there's likely some
coding to be done in order to get this working.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] CPU frequency throttling based on the temperature

[Fredy P.]

On Thu, 2019-07-25 at 17:34 +0200, Roger Pau Monné wrote:
> On Thu, Jul 25, 2019 at 02:31:40PM +0000, Jan Beulich wrote:
> > On 25.07.2019 16:17, Roger Pau Monné  wrote:
> > > On Thu, Jul 25, 2019 at 01:59:22PM +0000, Jan Beulich wrote:
> > > > On 25.07.2019 15:47, Roger Pau Monné  wrote:
> > > > > On Thu, Jul 25, 2019 at 09:29:01AM -0400, Fredy P. wrote:
> > > > > > On Thu, 2019-07-25 at 15:13 +0200, Roger Pau Monné wrote:
> > > > > > > On Thu, Jul 25, 2019 at 12:54:46PM +0000, Jan Beulich
> > > > > > > wrote:
> > > > > > > > On 25.07.2019 14:44,  Fredy P.  wrote:
> > > > > > > > > On Wed, 2019-07-24 at 17:41 +0200, Roger Pau Monné
> > > > > > > > > wrote:
> > > > > > > > > > > > What hardware interface does thermald (or the
> > > > > > > > > > > > driver in
> > > > > > > > > > > > Linux if
> > > > > > > > > > > > there's one) use to get the temperature data?
> > > > > > > > > 
> > > > > > > > > In our initial POC using Xen 4.8.x we where using
> > > > > > > > > Linux coretemp
> > > > > > > > > driver
> > > > > > > > > reading by example
> > > > > > > > > /class/sys/hwmon/hwmon0/temp3_input but it got
> > > > > > > > > deprecated at commit
> > > > > > > > > 72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> > > > > > > > 
> > > > > > > > Hmm, I wouldn't call this deprecation, but a
> > > > > > > > regression. I would
> > > > > > > > say we want to re-expose this leaf to Dom0, the more
> > > > > > > > that the
> > > > > > > > commit also only mentions unprivileged domains. Andrew?
> > > > > > > 
> > > > > > > AFAICT from the documents provided by Fredy the
> > > > > > > temperature is read
> > > > > > > from a MSR that reports the current temperature of the
> > > > > > > core on which
> > > > > > > the MSR is read from. When running on Xen this will only
> > > > > > > work
> > > > > > > correctly if dom0 is given the same vCPUs as pCPUs and
> > > > > > > those are
> > > > > > > identity pinned.
> > > > > > 
> > > > > > I just want to be sure I got it correctly, by saying "When
> > > > > > running on
> > > > > > Xen this will only work correctly if ..." means in a future
> > > > > > implementation or that right now could work if I pin this
> > > > > > v/pCPUS?
> > > > > 
> > > > > No, right now there's no way to get this data from dom0,
> > > > > regardless of
> > > > > the pinning.
> > > > 
> > > > Of course you can, using the MSR "device" Linux optionally
> > > > provides (plus perhaps the rdmsr utility from the msr-tools
> > > > package).
> > > 
> > > But you won't get coherent results, since the vCPU might be
> > > jumping
> > > from pCPU to pCPU, thus returning values from multiple different
> > > pCPUs
> > > regardless of whether all rdmsr have been executed from the same
> > > vCPU
> > > from dom0 PoV.
> > 
> > I don't understand. Earlier you said "regardless of the pinning".
> > That's what my response was to, i.e. I was implying vCPU-s to be
> > pinned.
> 
> Oh sorry, that was me not taking into account the earlier context,
> you
> are right. To summarize and make things easier for Fredy I think the
> options are:
> 
>  - Create dom0 with vCPUs == pCPUs and identity pin them. Then you
>    *could* expose CPUID leaf 6 to dom0 and things should be OK IMO,
>    either when using the Linux driver or when reading values directly
>    from user-space using the MSR device pointed out by Jan.
> 
>  - Modify the Linux thermal driver to report the temperature for all
>    pCPUs (which might be different than dom0 vCPUs) using
>    XENPF_resource_op and XEN_RESOURCE_OP_MSR_READ. AFAICT you will
>    also need to expose CPUID leaf 6 to dom0 so that the thermal
> driver
>    attaches.
> 
>  - Import a thermal driver into Xen and expose the thermal data
>    somewhere, ie: a XENPF hypercall maybe.
> 
> Maybe someone can come up with more ideas, but there's likely some
> coding to be done in order to get this working.

Thanks Roger, to me first look like a workaround that could save our
project delivery right now. Second look the right path.

I'll continue reading, It looks they are different ways to get thermal
Information form Linux and per driver there are different ways to, to
be more clear:

Thermald (Intel) [1] can work using this kernel drivers:

###############################
Prerequisites:
	Kernel
		Prefers kernel with
			Intel RAPL power capping driver : Available
from Linux kernel 3.13.rc1
			Intel P State driver (Available in Linux kernel
stable release)
			Intel Power clamp driver (Available in Linux
kernel stable release)
			Intel INT340X drivers
			Intel RAPL-mmio power capping driver: Available
from 5.3-rc1
##############################

MSR looks like part of RAPL and this other document [2] mention 3 ways
to get information from RAPL driver and mention that this information
is per package (procesor) and not per core:

#############################################
Linux support is via powercap/sysfs (as provided by
drivers/powercap/intel_rapl.c) and perf_event (as provided by
arch/x86/events/intel/rapl.c). Also users often access the relevant MSR
registers directly via the /dev/msr or safe-msr interfaces.

Various values are provided, not all chips support all values. The
"package" value is for one processor package (which may contain many
cores; a system might have multiple packages). The PP0/"cores" value is
power usage by all of the cores in the package (you cannot break down
to individual cores). The PP1 value is the uncore, in non-server chips
this often provides info for the integrated GPU. The "DRAM" value is
for the DRAM in the system. The "Psys" value is the entire SoC (system
on chip). 
#############################################

I'll try to get a more clear understanding and if I have ideas or
solutions I'll bring them here.


[1] https://github.com/intel/thermal_daemon
[2] http://web.eece.maine.edu/~vweaver/projects/rapl/rapl_support.html

> 
> Roger.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel