[Xen-devel] Xen hiding thermal capabilities from Dom0

[Xen-devel] Xen hiding thermal capabilities from Dom0

[Rishi]

Hi,

In some of our hosts, Xen is not correctly exposing processor thermal
capabilities to Dom0.
Please refer: https://bugzilla.kernel.org/show_bug.cgi?id=205347

The flag
/* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
X86_FEATURE_DTHERM (14*32+ 0)

is returned 0 via PVOP_VCALL4 vs returns 1 via native_cpuid() call from Dom0.

Sample output via PVCALL vs Native call.
[63291.688755] cpuid_eax 6 :  a 0
[63291.688759] native_cpuid : a 77

Is this a bug or needs some special feature to be enabled from Xen command line?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 14.11.2019 17:07, Rishi wrote:
> In some of our hosts, Xen is not correctly exposing processor thermal
> capabilities to Dom0.
> Please refer: https://bugzilla.kernel.org/show_bug.cgi?id=205347
> 
> The flag
> /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
> X86_FEATURE_DTHERM (14*32+ 0)
> 
> is returned 0 via PVOP_VCALL4 vs returns 1 via native_cpuid() call from Dom0.
> 
> Sample output via PVCALL vs Native call.
> [63291.688755] cpuid_eax 6 :  a 0
> [63291.688759] native_cpuid : a 77
> 
> Is this a bug or needs some special feature to be enabled from Xen command line?

Exposing this to guests (including Dom0) would imply properly
virtualizing the respective behavior. In
xen/arch/x86/cpuid.c:recalculate_misc() we specifically have

    p->basic.raw[0x6] = EMPTY_LEAF; /* Therm/Power not exposed to guests. */

If you wanted this exposed, you'd first of all need to come up
with a sane virtualization model of this functionality. Simply
exposing the CPUID bits is not going to be an option.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Rishi]

On Thu, Nov 14, 2019 at 10:05 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 14.11.2019 17:07, Rishi wrote:
> > In some of our hosts, Xen is not correctly exposing processor thermal
> > capabilities to Dom0.
> > Please refer: https://bugzilla.kernel.org/show_bug.cgi?id=205347
> >
> > The flag
> > /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
> > X86_FEATURE_DTHERM (14*32+ 0)
> >
> > is returned 0 via PVOP_VCALL4 vs returns 1 via native_cpuid() call from Dom0.
> >
> > Sample output via PVCALL vs Native call.
> > [63291.688755] cpuid_eax 6 :  a 0
> > [63291.688759] native_cpuid : a 77
> >
> > Is this a bug or needs some special feature to be enabled from Xen command line?
>
> Exposing this to guests (including Dom0) would imply properly
> virtualizing the respective behavior. In
> xen/arch/x86/cpuid.c:recalculate_misc() we specifically have
>
>     p->basic.raw[0x6] = EMPTY_LEAF; /* Therm/Power not exposed to guests. */
>
> If you wanted this exposed, you'd first of all need to come up
> with a sane virtualization model of this functionality. Simply
> exposing the CPUID bits is not going to be an option.
>
> Jan

Is it due to any security risks? If so, are there any advisories around it?

Should it be allowed that Xen is hiding CPU flags (in this case
Therm), yet a modification to Dom0 kernel allows them to be brought
back?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 15.11.2019 20:19, Rishi wrote:
> On Thu, Nov 14, 2019 at 10:05 PM Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 14.11.2019 17:07, Rishi wrote:
>>> In some of our hosts, Xen is not correctly exposing processor thermal
>>> capabilities to Dom0.
>>> Please refer: https://bugzilla.kernel.org/show_bug.cgi?id=205347
>>>
>>> The flag
>>> /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
>>> X86_FEATURE_DTHERM (14*32+ 0)
>>>
>>> is returned 0 via PVOP_VCALL4 vs returns 1 via native_cpuid() call from Dom0.
>>>
>>> Sample output via PVCALL vs Native call.
>>> [63291.688755] cpuid_eax 6 :  a 0
>>> [63291.688759] native_cpuid : a 77
>>>
>>> Is this a bug or needs some special feature to be enabled from Xen command line?
>>
>> Exposing this to guests (including Dom0) would imply properly
>> virtualizing the respective behavior. In
>> xen/arch/x86/cpuid.c:recalculate_misc() we specifically have
>>
>>     p->basic.raw[0x6] = EMPTY_LEAF; /* Therm/Power not exposed to guests. */
>>
>> If you wanted this exposed, you'd first of all need to come up
>> with a sane virtualization model of this functionality. Simply
>> exposing the CPUID bits is not going to be an option.
> 
> Is it due to any security risks? If so, are there any advisories around it?

No, it's because of the split of responsibilities: It's Xen to manage
the CPUs.

> Should it be allowed that Xen is hiding CPU flags (in this case
> Therm), yet a modification to Dom0 kernel allows them to be brought
> back?

Yes, a PV kernel is supposed to do things the PV way, i.e. in
particular use the PV form of CPUID. (Note that on hardware supporting
CPUID faulting you can't "bring back" the flags via a kernel change,
since then _all_ CPUID invocations will be handled by Xen instead of
by bare hardware. So _not_ doing things the PV way in the kernel sets
you up for surprises.)

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Rishi]

On Mon, Nov 18, 2019 at 3:08 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 15.11.2019 20:19, Rishi wrote:
> > On Thu, Nov 14, 2019 at 10:05 PM Jan Beulich <jbeulich@suse.com> wrote:
> >>
> >> On 14.11.2019 17:07, Rishi wrote:
> >>> In some of our hosts, Xen is not correctly exposing processor thermal
> >>> capabilities to Dom0.
> >>> Please refer: https://bugzilla.kernel.org/show_bug.cgi?id=205347
> >>>
> >>> The flag
> >>> /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
> >>> X86_FEATURE_DTHERM (14*32+ 0)
> >>>
> >>> is returned 0 via PVOP_VCALL4 vs returns 1 via native_cpuid() call from Dom0.
> >>>
> >>> Sample output via PVCALL vs Native call.
> >>> [63291.688755] cpuid_eax 6 :  a 0
> >>> [63291.688759] native_cpuid : a 77
> >>>
> >>> Is this a bug or needs some special feature to be enabled from Xen command line?
> >>
> >> Exposing this to guests (including Dom0) would imply properly
> >> virtualizing the respective behavior. In
> >> xen/arch/x86/cpuid.c:recalculate_misc() we specifically have
> >>
> >>     p->basic.raw[0x6] = EMPTY_LEAF; /* Therm/Power not exposed to guests. */
> >>
> >> If you wanted this exposed, you'd first of all need to come up
> >> with a sane virtualization model of this functionality. Simply
> >> exposing the CPUID bits is not going to be an option.
> >
> > Is it due to any security risks? If so, are there any advisories around it?
>
> No, it's because of the split of responsibilities: It's Xen to manage
> the CPUs.
>
> > Should it be allowed that Xen is hiding CPU flags (in this case
> > Therm), yet a modification to Dom0 kernel allows them to be brought
> > back?
>
> Yes, a PV kernel is supposed to do things the PV way, i.e. in
> particular use the PV form of CPUID. (Note that on hardware supporting
> CPUID faulting you can't "bring back" the flags via a kernel change,
> since then _all_ CPUID invocations will be handled by Xen instead of
> by bare hardware. So _not_ doing things the PV way in the kernel sets
> you up for surprises.)
>
> Jan

ok, thanks for clearing it up. Would a patch be accepted if this
option of showing EAX leaf is selectively done through command line
(default disabled)?
On longer run, what is an expected sane model of virtualizing this?
With some guidance, may be I or someone else can code to bring the
functionality back.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 19.11.2019 06:23, Rishi wrote:
> ok, thanks for clearing it up. Would a patch be accepted if this
> option of showing EAX leaf is selectively done through command line
> (default disabled)?

In general I'd expect this to be rather unlikely, but I guess much
would depend on the actual reasoning done in the description.

> On longer run, what is an expected sane model of virtualizing this?
> With some guidance, may be I or someone else can code to bring the
> functionality back.

Which functionality? So far you've talked of only CPUID bits I
think, without explaining at all what functionality you want to
have that depends on these. In general, as said earlier, CPU
management is the hypervisor's responsibility, so I'd rather
not see this virtualized, but the hypervisor be put into a
position of doing whatever is needed.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Rishi]

On Tue, Nov 19, 2019 at 2:47 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 19.11.2019 06:23, Rishi wrote:
> > ok, thanks for clearing it up. Would a patch be accepted if this
> > option of showing EAX leaf is selectively done through command line
> > (default disabled)?
>
> In general I'd expect this to be rather unlikely, but I guess much
> would depend on the actual reasoning done in the description.
>
> > On longer run, what is an expected sane model of virtualizing this?
> > With some guidance, may be I or someone else can code to bring the
> > functionality back.
>
> Which functionality? So far you've talked of only CPUID bits I
> think, without explaining at all what functionality you want to
> have that depends on these. In general, as said earlier, CPU
> management is the hypervisor's responsibility, so I'd rather
> not see this virtualized, but the hypervisor be put into a
> position of doing whatever is needed.
>
> Jan

The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
Power management.
Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
temperature or do Thermal management - including but not limited to
operating Fan speed.
Dom0 has to rely on other possible ways such as ipmi or BIOS which are
optionally available.

From the patch description
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
it seems that the change was introduced to not expose EAX(0x06h) to
unprivileged PV guests but nothing is said for Dom0 itself. I think
you already mentioned that the flag is hid from Dom0 as well
intentionally.

So unless hypervisor wants to do thermal management of the CPU board,
it would inhibit Dom0's ability to do this function.

What is an alternative way for coretemp kernel module to detect
"DTHERM" processor flag and/or proceed for safe reading of MSR to do
further temperature value reads?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Roger Pau Monné]

On Thu, Nov 21, 2019 at 07:09:31PM +0530, Rishi wrote:
> On Tue, Nov 19, 2019 at 2:47 PM Jan Beulich <jbeulich@suse.com> wrote:
> >
> > On 19.11.2019 06:23, Rishi wrote:
> > > ok, thanks for clearing it up. Would a patch be accepted if this
> > > option of showing EAX leaf is selectively done through command line
> > > (default disabled)?
> >
> > In general I'd expect this to be rather unlikely, but I guess much
> > would depend on the actual reasoning done in the description.
> >
> > > On longer run, what is an expected sane model of virtualizing this?
> > > With some guidance, may be I or someone else can code to bring the
> > > functionality back.
> >
> > Which functionality? So far you've talked of only CPUID bits I
> > think, without explaining at all what functionality you want to
> > have that depends on these. In general, as said earlier, CPU
> > management is the hypervisor's responsibility, so I'd rather
> > not see this virtualized, but the hypervisor be put into a
> > position of doing whatever is needed.
> >
> > Jan
> 
> The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
> Power management.
> Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
> temperature or do Thermal management - including but not limited to
> operating Fan speed.
> Dom0 has to rely on other possible ways such as ipmi or BIOS which are
> optionally available.
>
> From the patch description
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> it seems that the change was introduced to not expose EAX(0x06h) to
> unprivileged PV guests but nothing is said for Dom0 itself. I think
> you already mentioned that the flag is hid from Dom0 as well
> intentionally.
> 
> So unless hypervisor wants to do thermal management of the CPU board,
> it would inhibit Dom0's ability to do this function.

That's likely what you want, on a Xen system dom0 is a special guest,
but still a guest, so it's not feasible for a native dom0 driver to do
power or temperature management without having Xen specific
knowledge. For instance the load on dom0 doesn't match the actual
load on the hardware.

I think we had a very similar discussion at:

https://marc.info/?l=xen-devel&m=156397696413230&w=2

I would recommend reading the full thread and the
conclusions/proposals.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Rishi]

On Thu, Nov 21, 2019 at 7:26 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Thu, Nov 21, 2019 at 07:09:31PM +0530, Rishi wrote:
> > On Tue, Nov 19, 2019 at 2:47 PM Jan Beulich <jbeulich@suse.com> wrote:
> > >
> > > On 19.11.2019 06:23, Rishi wrote:
> > > > ok, thanks for clearing it up. Would a patch be accepted if this
> > > > option of showing EAX leaf is selectively done through command line
> > > > (default disabled)?
> > >
> > > In general I'd expect this to be rather unlikely, but I guess much
> > > would depend on the actual reasoning done in the description.
> > >
> > > > On longer run, what is an expected sane model of virtualizing this?
> > > > With some guidance, may be I or someone else can code to bring the
> > > > functionality back.
> > >
> > > Which functionality? So far you've talked of only CPUID bits I
> > > think, without explaining at all what functionality you want to
> > > have that depends on these. In general, as said earlier, CPU
> > > management is the hypervisor's responsibility, so I'd rather
> > > not see this virtualized, but the hypervisor be put into a
> > > position of doing whatever is needed.
> > >
> > > Jan
> >
> > The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
> > Power management.
> > Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
> > temperature or do Thermal management - including but not limited to
> > operating Fan speed.
> > Dom0 has to rely on other possible ways such as ipmi or BIOS which are
> > optionally available.
> >
> > From the patch description
> > https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> > it seems that the change was introduced to not expose EAX(0x06h) to
> > unprivileged PV guests but nothing is said for Dom0 itself. I think
> > you already mentioned that the flag is hid from Dom0 as well
> > intentionally.
> >
> > So unless hypervisor wants to do thermal management of the CPU board,
> > it would inhibit Dom0's ability to do this function.
>
> That's likely what you want, on a Xen system dom0 is a special guest,
> but still a guest, so it's not feasible for a native dom0 driver to do
> power or temperature management without having Xen specific
> knowledge. For instance the load on dom0 doesn't match the actual
> load on the hardware.
>
> I think we had a very similar discussion at:
>
> https://marc.info/?l=xen-devel&m=156397696413230&w=2
>
> I would recommend reading the full thread and the
> conclusions/proposals.
>
> Roger.

Thank Roger for the reference and conclusion. Repeating here for
having directed discussion.

> It will involve looking into the Linux driver in order to make use of
> an hypercall instead of a rdmsr. I think it should be fine to expose
> the CPUID leaf to dom0 as long as reads are performed from the
> hypercall, in order to assure that Linux gets consistent values.

The affected Linux driver in my case is coretemp.ko (drivers/hwmon/coretemp.c)

It's init depends on checking presence of X86_FEATURE_DTHERM

        /*
         * CPUID.06H.EAX[0] indicates whether the CPU has thermal
         * sensors. We check this bit only, all the early CPUs
         * without thermal sensors will be filtered out.
         */

It seems to use below MSR
MSR_IA32_PACKAGE_THERM_STATUS
MSR_IA32_THERM_STATUS
MSR_IA32_TEMPERATURE_TARGET

I'm not sure how can CPUID.06H.EAX[0] be read, should Xen provide a
hypercall interface to read this?
Even if a hypercall is given, coretemp will have to be modified to
separate MSR calls.
Does having a pv temperature reader (pvcoretemp) altogether make
sense? This would have hypercall for DTS detection and XEN MSR reads
for values.
For compatibility, can it use the same sys path as that of coretemp.ko
to write thermal info?
/sys/devices/platform/pvcoretemp.0/hwmon/hwmon2/temp1_input

This will let lm_sensors lib (SNMP/CLI) interpret temps regularly.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Roger Pau Monné]

On Thu, Nov 21, 2019 at 08:56:41PM +0530, Rishi wrote:
> On Thu, Nov 21, 2019 at 7:26 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Thu, Nov 21, 2019 at 07:09:31PM +0530, Rishi wrote:
> > > On Tue, Nov 19, 2019 at 2:47 PM Jan Beulich <jbeulich@suse.com> wrote:
> > > >
> > > > On 19.11.2019 06:23, Rishi wrote:
> > > > > ok, thanks for clearing it up. Would a patch be accepted if this
> > > > > option of showing EAX leaf is selectively done through command line
> > > > > (default disabled)?
> > > >
> > > > In general I'd expect this to be rather unlikely, but I guess much
> > > > would depend on the actual reasoning done in the description.
> > > >
> > > > > On longer run, what is an expected sane model of virtualizing this?
> > > > > With some guidance, may be I or someone else can code to bring the
> > > > > functionality back.
> > > >
> > > > Which functionality? So far you've talked of only CPUID bits I
> > > > think, without explaining at all what functionality you want to
> > > > have that depends on these. In general, as said earlier, CPU
> > > > management is the hypervisor's responsibility, so I'd rather
> > > > not see this virtualized, but the hypervisor be put into a
> > > > position of doing whatever is needed.
> > > >
> > > > Jan
> > >
> > > The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
> > > Power management.
> > > Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
> > > temperature or do Thermal management - including but not limited to
> > > operating Fan speed.
> > > Dom0 has to rely on other possible ways such as ipmi or BIOS which are
> > > optionally available.
> > >
> > > From the patch description
> > > https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> > > it seems that the change was introduced to not expose EAX(0x06h) to
> > > unprivileged PV guests but nothing is said for Dom0 itself. I think
> > > you already mentioned that the flag is hid from Dom0 as well
> > > intentionally.
> > >
> > > So unless hypervisor wants to do thermal management of the CPU board,
> > > it would inhibit Dom0's ability to do this function.
> >
> > That's likely what you want, on a Xen system dom0 is a special guest,
> > but still a guest, so it's not feasible for a native dom0 driver to do
> > power or temperature management without having Xen specific
> > knowledge. For instance the load on dom0 doesn't match the actual
> > load on the hardware.
> >
> > I think we had a very similar discussion at:
> >
> > https://marc.info/?l=xen-devel&m=156397696413230&w=2
> >
> > I would recommend reading the full thread and the
> > conclusions/proposals.
> >
> > Roger.
> 
> Thank Roger for the reference and conclusion. Repeating here for
> having directed discussion.
> 
> > It will involve looking into the Linux driver in order to make use of
> > an hypercall instead of a rdmsr. I think it should be fine to expose
> > the CPUID leaf to dom0 as long as reads are performed from the
> > hypercall, in order to assure that Linux gets consistent values.
> 
> The affected Linux driver in my case is coretemp.ko (drivers/hwmon/coretemp.c)
> 
> It's init depends on checking presence of X86_FEATURE_DTHERM
> 
>         /*
>          * CPUID.06H.EAX[0] indicates whether the CPU has thermal
>          * sensors. We check this bit only, all the early CPUs
>          * without thermal sensors will be filtered out.
>          */
> 
> It seems to use below MSR
> MSR_IA32_PACKAGE_THERM_STATUS
> MSR_IA32_THERM_STATUS
> MSR_IA32_TEMPERATURE_TARGET
> 
> I'm not sure how can CPUID.06H.EAX[0] be read, should Xen provide a
> hypercall interface to read this?

I think there's already some interface to read the raw cpuid policy,
but maybe that's a domctl. Note that on PV executing a plain cpuid
instruction without the Xen prefix will get you the native policy,
albeit I won't rely on that since it will go away with PVH.

You will also need an interface to request Xen to execute rdmsr/wrmsr
on specific pCPUs in order to get and set the thermal data.

> Even if a hypercall is given, coretemp will have to be modified to
> separate MSR calls.

Not only this, keep in mind that dom0 vCPUs != pCPUs, so you will
likely need a way to tell coretemp about the real number of CPUs in
the system, which might not match the number of vCPUs dom0 has
assigned.

> Does having a pv temperature reader (pvcoretemp) altogether make
> sense? This would have hypercall for DTS detection and XEN MSR reads
> for values.
> For compatibility, can it use the same sys path as that of coretemp.ko
> to write thermal info?
> /sys/devices/platform/pvcoretemp.0/hwmon/hwmon2/temp1_input

That's a Linux kernel question IMO, as a Xen developer I cannot tell
whether that's suitable or not since it's a detail of the Linux
implementation.

> 
> This will let lm_sensors lib (SNMP/CLI) interpret temps regularly.

Seems feasible, but you will also have to check that lm_sensors
doesn't make any assumptions about the number of CPUs by reading
/proc/cpuinfo for example.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 21.11.2019 16:45, Roger Pau Monné  wrote:
> On Thu, Nov 21, 2019 at 08:56:41PM +0530, Rishi wrote:
>> On Thu, Nov 21, 2019 at 7:26 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Thu, Nov 21, 2019 at 07:09:31PM +0530, Rishi wrote:
>>>> On Tue, Nov 19, 2019 at 2:47 PM Jan Beulich <jbeulich@suse.com> wrote:
>>>>>
>>>>> On 19.11.2019 06:23, Rishi wrote:
>>>>>> ok, thanks for clearing it up. Would a patch be accepted if this
>>>>>> option of showing EAX leaf is selectively done through command line
>>>>>> (default disabled)?
>>>>>
>>>>> In general I'd expect this to be rather unlikely, but I guess much
>>>>> would depend on the actual reasoning done in the description.
>>>>>
>>>>>> On longer run, what is an expected sane model of virtualizing this?
>>>>>> With some guidance, may be I or someone else can code to bring the
>>>>>> functionality back.
>>>>>
>>>>> Which functionality? So far you've talked of only CPUID bits I
>>>>> think, without explaining at all what functionality you want to
>>>>> have that depends on these. In general, as said earlier, CPU
>>>>> management is the hypervisor's responsibility, so I'd rather
>>>>> not see this virtualized, but the hypervisor be put into a
>>>>> position of doing whatever is needed.
>>>>>
>>>>> Jan
>>>>
>>>> The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
>>>> Power management.
>>>> Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
>>>> temperature or do Thermal management - including but not limited to
>>>> operating Fan speed.
>>>> Dom0 has to rely on other possible ways such as ipmi or BIOS which are
>>>> optionally available.
>>>>
>>>> From the patch description
>>>> https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
>>>> it seems that the change was introduced to not expose EAX(0x06h) to
>>>> unprivileged PV guests but nothing is said for Dom0 itself. I think
>>>> you already mentioned that the flag is hid from Dom0 as well
>>>> intentionally.
>>>>
>>>> So unless hypervisor wants to do thermal management of the CPU board,
>>>> it would inhibit Dom0's ability to do this function.
>>>
>>> That's likely what you want, on a Xen system dom0 is a special guest,
>>> but still a guest, so it's not feasible for a native dom0 driver to do
>>> power or temperature management without having Xen specific
>>> knowledge. For instance the load on dom0 doesn't match the actual
>>> load on the hardware.
>>>
>>> I think we had a very similar discussion at:
>>>
>>> https://marc.info/?l=xen-devel&m=156397696413230&w=2
>>>
>>> I would recommend reading the full thread and the
>>> conclusions/proposals.
>>>
>>> Roger.
>>
>> Thank Roger for the reference and conclusion. Repeating here for
>> having directed discussion.
>>
>>> It will involve looking into the Linux driver in order to make use of
>>> an hypercall instead of a rdmsr. I think it should be fine to expose
>>> the CPUID leaf to dom0 as long as reads are performed from the
>>> hypercall, in order to assure that Linux gets consistent values.
>>
>> The affected Linux driver in my case is coretemp.ko (drivers/hwmon/coretemp.c)
>>
>> It's init depends on checking presence of X86_FEATURE_DTHERM
>>
>>         /*
>>          * CPUID.06H.EAX[0] indicates whether the CPU has thermal
>>          * sensors. We check this bit only, all the early CPUs
>>          * without thermal sensors will be filtered out.
>>          */
>>
>> It seems to use below MSR
>> MSR_IA32_PACKAGE_THERM_STATUS
>> MSR_IA32_THERM_STATUS
>> MSR_IA32_TEMPERATURE_TARGET
>>
>> I'm not sure how can CPUID.06H.EAX[0] be read, should Xen provide a
>> hypercall interface to read this?
> 
> I think there's already some interface to read the raw cpuid policy,
> but maybe that's a domctl. Note that on PV executing a plain cpuid
> instruction without the Xen prefix will get you the native policy,
> albeit I won't rely on that since it will go away with PVH.

Nor would you see the real value in PV mode when the hardware
supports CPUID faulting (and you don't suppress this via
"dom0=no-cpuid-faulting").

> You will also need an interface to request Xen to execute rdmsr/wrmsr
> on specific pCPUs in order to get and set the thermal data.

As said elsewhere on this thread, we have such an interface already
(with a white listed set of MSRs).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 21.11.2019 16:26, Rishi wrote:
> The affected Linux driver in my case is coretemp.ko (drivers/hwmon/coretemp.c)
> 
> It's init depends on checking presence of X86_FEATURE_DTHERM
> 
>         /*
>          * CPUID.06H.EAX[0] indicates whether the CPU has thermal
>          * sensors. We check this bit only, all the early CPUs
>          * without thermal sensors will be filtered out.
>          */
> 
> It seems to use below MSR
> MSR_IA32_PACKAGE_THERM_STATUS
> MSR_IA32_THERM_STATUS
> MSR_IA32_TEMPERATURE_TARGET
> 
> I'm not sure how can CPUID.06H.EAX[0] be read, should Xen provide a
> hypercall interface to read this?
> Even if a hypercall is given, coretemp will have to be modified to
> separate MSR calls.

A hypercall exists (XENPF_resource_op), and would just need
extension. It could e.g. permit access to said MSRs when
CPUID.06H.EAX[0] is 1. The coretemp driver would of course
need modification (or cloning).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jürgen Groß]

On 21.11.19 14:39, Rishi wrote:
> On Tue, Nov 19, 2019 at 2:47 PM Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 19.11.2019 06:23, Rishi wrote:
>>> ok, thanks for clearing it up. Would a patch be accepted if this
>>> option of showing EAX leaf is selectively done through command line
>>> (default disabled)?
>>
>> In general I'd expect this to be rather unlikely, but I guess much
>> would depend on the actual reasoning done in the description.
>>
>>> On longer run, what is an expected sane model of virtualizing this?
>>> With some guidance, may be I or someone else can code to bring the
>>> functionality back.
>>
>> Which functionality? So far you've talked of only CPUID bits I
>> think, without explaining at all what functionality you want to
>> have that depends on these. In general, as said earlier, CPU
>> management is the hypervisor's responsibility, so I'd rather
>> not see this virtualized, but the hypervisor be put into a
>> position of doing whatever is needed.
>>
>> Jan
> 
> The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
> Power management.
> Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
> temperature or do Thermal management - including but not limited to
> operating Fan speed.
> Dom0 has to rely on other possible ways such as ipmi or BIOS which are
> optionally available.

You are aware that dom0 can't easily control on which _physical_ cpu it
is just running? So it could easily be that you are sampling lets say
3 MSRs in a function, but you are accessing different cpus each time due
to the hypervisor re-scheduling the vcpu in between.

And in case you want to adjust settings you can hit another cpu again.

So: no, just giving dom0 access to the management hardware isn't going
to fly. You need to have a proper virtualization layer for that purpose.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Rishi]

On Thu, Nov 21, 2019 at 7:54 PM Jürgen Groß <jgross@suse.com> wrote:
>
> On 21.11.19 14:39, Rishi wrote:
> > On Tue, Nov 19, 2019 at 2:47 PM Jan Beulich <jbeulich@suse.com> wrote:
> >>
> >> On 19.11.2019 06:23, Rishi wrote:
> >>> ok, thanks for clearing it up. Would a patch be accepted if this
> >>> option of showing EAX leaf is selectively done through command line
> >>> (default disabled)?
> >>
> >> In general I'd expect this to be rather unlikely, but I guess much
> >> would depend on the actual reasoning done in the description.
> >>
> >>> On longer run, what is an expected sane model of virtualizing this?
> >>> With some guidance, may be I or someone else can code to bring the
> >>> functionality back.
> >>
> >> Which functionality? So far you've talked of only CPUID bits I
> >> think, without explaining at all what functionality you want to
> >> have that depends on these. In general, as said earlier, CPU
> >> management is the hypervisor's responsibility, so I'd rather
> >> not see this virtualized, but the hypervisor be put into a
> >> position of doing whatever is needed.
> >>
> >> Jan
> >
> > The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
> > Power management.
> > Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
> > temperature or do Thermal management - including but not limited to
> > operating Fan speed.
> > Dom0 has to rely on other possible ways such as ipmi or BIOS which are
> > optionally available.
>
> You are aware that dom0 can't easily control on which _physical_ cpu it
> is just running? So it could easily be that you are sampling lets say
> 3 MSRs in a function, but you are accessing different cpus each time due
> to the hypervisor re-scheduling the vcpu in between.
>
> And in case you want to adjust settings you can hit another cpu again.
>
> So: no, just giving dom0 access to the management hardware isn't going
> to fly. You need to have a proper virtualization layer for that purpose.
>
>
> Juergen

I now understand it, thanks.
As mentioned in other reply, I think a pv version of coretemp Linux
driver may be a way to bring back thermal information values for a
given pCPU.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 21.11.2019 15:24, Jürgen Groß wrote:
> So: no, just giving dom0 access to the management hardware isn't going
> to fly. You need to have a proper virtualization layer for that purpose.

Or, like I had done in our XenoLinux forward port, you need to
go through hoops to make the coretemp driver actually understand
the environment it's running in.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jürgen Groß]

On 21.11.19 16:36, Jan Beulich wrote:
> On 21.11.2019 15:24, Jürgen Groß wrote:
>> So: no, just giving dom0 access to the management hardware isn't going
>> to fly. You need to have a proper virtualization layer for that purpose.
> 
> Or, like I had done in our XenoLinux forward port, you need to
> go through hoops to make the coretemp driver actually understand
> the environment it's running in.

This will still not guarantee you'll be able to reach all physical
cpus. IIRC you pinned the vcpu to the respective physical cpu for
performing its duty, but with cpupools this might not be possible for
all physical cpus in the system.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 21.11.2019 16:46, Jürgen Groß wrote:
> On 21.11.19 16:36, Jan Beulich wrote:
>> On 21.11.2019 15:24, Jürgen Groß wrote:
>>> So: no, just giving dom0 access to the management hardware isn't going
>>> to fly. You need to have a proper virtualization layer for that purpose.
>>
>> Or, like I had done in our XenoLinux forward port, you need to
>> go through hoops to make the coretemp driver actually understand
>> the environment it's running in.
> 
> This will still not guarantee you'll be able to reach all physical
> cpus. IIRC you pinned the vcpu to the respective physical cpu for
> performing its duty, but with cpupools this might not be possible for
> all physical cpus in the system.

Correct - that work predated CPU pools by quite a bit, I think.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Elliott Mitchell]

On Thu, Nov 21, 2019 at 04:46:21PM +0100, J??rgen Gro?? wrote:
> On 21.11.19 16:36, Jan Beulich wrote:
> > On 21.11.2019 15:24, J??rgen Gro?? wrote:
> >> So: no, just giving dom0 access to the management hardware isn't going
> >> to fly. You need to have a proper virtualization layer for that purpose.
> > 
> > Or, like I had done in our XenoLinux forward port, you need to
> > go through hoops to make the coretemp driver actually understand
> > the environment it's running in.
> 
> This will still not guarantee you'll be able to reach all physical
> cpus. IIRC you pinned the vcpu to the respective physical cpu for
> performing its duty, but with cpupools this might not be possible for
> all physical cpus in the system.

Similar to the issue of MCE support, might it instead be better to have
*less* virtualization here instead of more?  The original idea behind Xen
was to leave the hard to virtualize bits visible and work with Domain 0.

Might it be better to expose this functionality to Domain 0, then
intercept the kernel calls?  Just needs 1 vcpu which can be scheduled on
any processor and that can be moved around to retrieve the data.  This
way Xen wouldn't need a proper driver for the management hardware.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jürgen Groß]

On 23.11.19 05:29, Elliott Mitchell wrote:
> On Thu, Nov 21, 2019 at 04:46:21PM +0100, J??rgen Gro?? wrote:
>> On 21.11.19 16:36, Jan Beulich wrote:
>>> On 21.11.2019 15:24, J??rgen Gro?? wrote:
>>>> So: no, just giving dom0 access to the management hardware isn't going
>>>> to fly. You need to have a proper virtualization layer for that purpose.
>>>
>>> Or, like I had done in our XenoLinux forward port, you need to
>>> go through hoops to make the coretemp driver actually understand
>>> the environment it's running in.
>>
>> This will still not guarantee you'll be able to reach all physical
>> cpus. IIRC you pinned the vcpu to the respective physical cpu for
>> performing its duty, but with cpupools this might not be possible for
>> all physical cpus in the system.
> 
> Similar to the issue of MCE support, might it instead be better to have
> *less* virtualization here instead of more?  The original idea behind Xen
> was to leave the hard to virtualize bits visible and work with Domain 0.
> 
> Might it be better to expose this functionality to Domain 0, then
> intercept the kernel calls?  Just needs 1 vcpu which can be scheduled on
> any processor and that can be moved around to retrieve the data.  This
> way Xen wouldn't need a proper driver for the management hardware.

In case dom0 is to handle this then it would be much easier to have a
way for dom0 to specify which physical cpu the data should be retrieved
from. Forcing a dom0 vcpu to run on a specific physical cpu would need
a major rework of the Xen scheduling (especially regarding cpupools, let
alone core scheduling).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Rishi]

On Mon, Nov 25, 2019 at 1:55 PM Jürgen Groß <jgross@suse.com> wrote:
>
> On 23.11.19 05:29, Elliott Mitchell wrote:
> > On Thu, Nov 21, 2019 at 04:46:21PM +0100, J??rgen Gro?? wrote:
> >> On 21.11.19 16:36, Jan Beulich wrote:
> >>> On 21.11.2019 15:24, J??rgen Gro?? wrote:
> >>>> So: no, just giving dom0 access to the management hardware isn't going
> >>>> to fly. You need to have a proper virtualization layer for that purpose.
> >>>
> >>> Or, like I had done in our XenoLinux forward port, you need to
> >>> go through hoops to make the coretemp driver actually understand
> >>> the environment it's running in.
> >>
> >> This will still not guarantee you'll be able to reach all physical
> >> cpus. IIRC you pinned the vcpu to the respective physical cpu for
> >> performing its duty, but with cpupools this might not be possible for
> >> all physical cpus in the system.
> >
> > Similar to the issue of MCE support, might it instead be better to have
> > *less* virtualization here instead of more?  The original idea behind Xen
> > was to leave the hard to virtualize bits visible and work with Domain 0.
> >
> > Might it be better to expose this functionality to Domain 0, then
> > intercept the kernel calls?  Just needs 1 vcpu which can be scheduled on
> > any processor and that can be moved around to retrieve the data.  This
> > way Xen wouldn't need a proper driver for the management hardware.
>
> In case dom0 is to handle this then it would be much easier to have a
> way for dom0 to specify which physical cpu the data should be retrieved
> from. Forcing a dom0 vcpu to run on a specific physical cpu would need
> a major rework of the Xen scheduling (especially regarding cpupools, let
> alone core scheduling).
>
>
> Juergen

While modifying coretemp driver, following CPU flags came across
X86_FEATURE_DTHERM
X86_FEATURE_PTS

Need to replace/get these via Xen hypercall. This only detects if
DTHERM and PTS support is present. Currently bypassing them in code
and will wait for a proper EAX exposure.

Next is the MSR read for actual temperature values. Currently
rdmsr_safe_on_cpu is being used, does it already get converted to a
Hypercall to be able to detect values?
While tracing the function calls from code, rdmsr_safe_on_cpu() ->
rdmsr_safe() -> native_read_msr_safe() -> asm volatile() comes up.
I can see xen_read_msr_safe() but not sure if this or its any other
variant can be called.

I haven't gone into depth of this and would appreciate pointers to
understand more.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 25.11.2019 16:43, Rishi wrote:
> Next is the MSR read for actual temperature values. Currently
> rdmsr_safe_on_cpu is being used, does it already get converted to a
> Hypercall to be able to detect values?
> While tracing the function calls from code, rdmsr_safe_on_cpu() ->
> rdmsr_safe() -> native_read_msr_safe() -> asm volatile() comes up.
> I can see xen_read_msr_safe() but not sure if this or its any other
> variant can be called.

None of these are suitable, as mentioned before, as they're all
acting in terms of vCPU-s, while you want to act on pCPU-s.
You're unlikely to be able to make this work without first
making Xen allow Dom0 access to these MSRs via the indicated
platform-op (or yet something more intrusive).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen hiding thermal capabilities from Dom0

[Jan Beulich]

On 21.11.2019 14:39, Rishi wrote:
> The reasoning to have EAX(0x06h) exposed to Dom0 is for Thermal and
> Power management.

I understand this.

> Without EAX(0x06h) Dom0 is unable to sense presence of CPU core
> temperature or do Thermal management - including but not limited to
> operating Fan speed.

But you don't seem to understand that this is, from a layering
perspective, not Dom0's job. (As per the other reply sent a
moment ago, it can be made work, but it's not as simple as you
appear to think.) In principle - repeating this just another
time - it's Xen's job to handle this.

> Dom0 has to rely on other possible ways such as ipmi or BIOS which are
> optionally available.
> 
> From the patch description
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=72e038450d3d5de1a39f0cfa2d2b0f9b3d43c6c6
> it seems that the change was introduced to not expose EAX(0x06h) to
> unprivileged PV guests but nothing is said for Dom0 itself. I think
> you already mentioned that the flag is hid from Dom0 as well
> intentionally.
> 
> So unless hypervisor wants to do thermal management of the CPU board,
> it would inhibit Dom0's ability to do this function.
> 
> What is an alternative way for coretemp kernel module to detect
> "DTHERM" processor flag and/or proceed for safe reading of MSR to do
> further temperature value reads?

Introduction of a proper interface between Xen and Dom0 by which
Xen becomes aware of this part of CPU management getting delegated
to Dom0.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel