[Xen-devel] wall clock drift on C24x mainboard, best practices

[Xen-devel] wall clock drift on C24x mainboard, best practices

[Andreas Kinzler]

Hello all, hello Paul,

On a certain new mainboard with chipset C242 and Intel Xeon E-2136 I 
notice a severe clock drift. This is from dom0:

# uptime
  20:13:52 up 81 days,  1:41,  1 user,  load average: 0.00, 0.00, 0.00
# hwclock
2019-10-12 20:27:37.204966+02:00
# date
Sat Oct 12 20:07:19 CEST 2019

Kernel is 4.13.16 vanilla, Xen 4.10.2

So after 81 days uptime there is a difference of over 20 minutes between 
"date" and "hwclock". I operate many Xen servers and have never seen 
such a great drift except on this type of mainboard. What could be the 
reason?

In general, what is the current best practice for NTP sync? Run it in 
dom0? In domU? Both? How does the domU type (Linux HVM/PVM/PVH or 
Windows HVM with WinPV drivers) make a difference?

Regards Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] wall clock drift on Coffee Lake / C24x mainboard (HPET broken?), best practices

[Andreas Kinzler]

Hello All,

I came across the following: https://lkml.org/lkml/2019/8/29/536

Could that be the reason for the problem mentioned below? Xen is using 
HPET as clocksource on the platform/mainboard. Is there an (easy) way to 
verify if Xen uses PC10?

Regards Andreas

On 12.10.2019 20:47, Andreas Kinzler wrote:
> Hello all, hello Paul,
>
> On a certain new mainboard with chipset C242 and Intel Xeon E-2136 I 
> notice a severe clock drift. This is from dom0:
>
> # uptime
>  20:13:52 up 81 days,  1:41,  1 user,  load average: 0.00, 0.00, 0.00
> # hwclock
> 2019-10-12 20:27:37.204966+02:00
> # date
> Sat Oct 12 20:07:19 CEST 2019
>
> Kernel is 4.13.16 vanilla, Xen 4.10.2
>
> So after 81 days uptime there is a difference of over 20 minutes 
> between "date" and "hwclock". I operate many Xen servers and have 
> never seen such a great drift except on this type of mainboard. What 
> could be the reason?
>
> In general, what is the current best practice for NTP sync? Run it in 
> dom0? In domU? Both? How does the domU type (Linux HVM/PVM/PVH or 
> Windows HVM with WinPV drivers) make a difference?
>
> Regards Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] wall clock drift on Coffee Lake / C24x mainboard (HPET broken?), best practices

[Jan Beulich]

On 14.11.2019 00:10, Andreas Kinzler wrote:
> I came across the following: https://lkml.org/lkml/2019/8/29/536
> 
> Could that be the reason for the problem mentioned below? Xen is using 
> HPET as clocksource on the platform/mainboard. Is there an (easy) way to 
> verify if Xen uses PC10?

In principle this can be obtained via both the xenpm utility and
the 'c' debug key. For Coffee Lake, however, I can't find any
indication in the SDM that a PC10 residency MSR would exist.
Hence I can only suggest that you try again with limited or no
use of C states, to at least get a hint as to a possible
connection.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] wall clock drift on Coffee Lake / C24x mainboard (HPET broken?), best practices

[Andreas Kinzler]

On 14.11.2019 12:29, Jan Beulich wrote:
> On 14.11.2019 00:10, Andreas Kinzler wrote:
>> I came across the following: https://lkml.org/lkml/2019/8/29/536
>> Could that be the reason for the problem mentioned below? Xen is using
>> HPET as clocksource on the platform/mainboard. Is there an (easy) way to
>> verify if Xen uses PC10?
> In principle this can be obtained via both the xenpm utility and
> the 'c' debug key.

Both xenpm and 'c' debug key show only up to level 7 in Xen 4.10.x 
(unmodified code).

> For Coffee Lake, however, I can't find any
> indication in the SDM that a PC10 residency MSR would exist.

I used turbostat 
(https://github.com/torvalds/linux/blob/master/tools/power/x86/turbostat/turbostat.c) 
as a help. See functions has_c8910_msrs and intel_model_duplicates.

I then added Coffee Lake with PC8/9/10 to do_get_hw_residencies and then 
I got high counts in PC8+PC9 and zero in PC10.

> Hence I can only suggest that you try again with limited or no
> use of C states, to at least get a hint as to a possible

I changed the BIOS setting to a limit of PC7 and it is now running. I 
have to wait for the result. Thanks.

Regards Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] wall clock drift on Coffee Lake / C24x mainboard (HPET broken?), best practices

[Andreas Kinzler]

On 15.11.2019 12:01, Andreas Kinzler wrote:
> On 14.11.2019 12:29, Jan Beulich wrote:
>> On 14.11.2019 00:10, Andreas Kinzler wrote:
>>> I came across the following: https://lkml.org/lkml/2019/8/29/536
>>> Could that be the reason for the problem mentioned below? Xen is using
>>> HPET as clocksource on the platform/mainboard. Is there an (easy) way to
>>> verify if Xen uses PC10?
>> Hence I can only suggest that you try again with limited or no
>> use of C states, to at least get a hint as to a possible
> I changed the BIOS setting to a limit of PC7 and it is now running. I 
> have to wait for the result. Thanks.

Previously the drift after 4 days uptime was 60 sec. Now after 4 days 
uptime drift is 9 sec. So setting the package c-state limit to PC7 was a 
success.

Regards Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] wall clock drift on Coffee Lake / C24x mainboard (HPET broken?), best practices

[Jan Beulich]

On 18.11.2019 20:35, Andreas Kinzler wrote:
> On 15.11.2019 12:01, Andreas Kinzler wrote:
>> On 14.11.2019 12:29, Jan Beulich wrote:
>>> On 14.11.2019 00:10, Andreas Kinzler wrote:
>>>> I came across the following: https://lkml.org/lkml/2019/8/29/536
>>>> Could that be the reason for the problem mentioned below? Xen is using
>>>> HPET as clocksource on the platform/mainboard. Is there an (easy) way to
>>>> verify if Xen uses PC10?
>>> Hence I can only suggest that you try again with limited or no
>>> use of C states, to at least get a hint as to a possible
>> I changed the BIOS setting to a limit of PC7 and it is now running. I 
>> have to wait for the result. Thanks.
> 
> Previously the drift after 4 days uptime was 60 sec. Now after 4 days 
> uptime drift is 9 sec. So setting the package c-state limit to PC7 was a 
> success.

9s still seems quite a lot to me, but yes, it's an improvement.
Now would you be up to checking whether, rather than via BIOS
settings (which not all BIOSes may offer) the same can be
achieved by using Xen's command line option "max_cstate="?

Also did you check whether further limiting C state use would
further improve the situation? And did you possibly also check
whether telling Xen not to use the HPET would make a difference?

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] wall clock drift on Coffee Lake / C24x mainboard (HPET broken?), best practices

[Andreas Kinzler]

On 19.11.2019 10:29, Jan Beulich wrote:
> On 18.11.2019 20:35, Andreas Kinzler wrote:
>> On 15.11.2019 12:01, Andreas Kinzler wrote:
>>> On 14.11.2019 12:29, Jan Beulich wrote:
>>>> On 14.11.2019 00:10, Andreas Kinzler wrote:
>>>>> I came across the following: https://lkml.org/lkml/2019/8/29/536
>>>>> Could that be the reason for the problem mentioned below? Xen is using
>>>>> HPET as clocksource on the platform/mainboard. Is there an (easy) way to
>>>>> verify if Xen uses PC10?
>>>> Hence I can only suggest that you try again with limited or no
>>>> use of C states, to at least get a hint as to a possible
>>> I changed the BIOS setting to a limit of PC7 and it is now running. I
>>> have to wait for the result. Thanks.
>>
>> Previously the drift after 4 days uptime was 60 sec. Now after 4 days
>> uptime drift is 9 sec. So setting the package c-state limit to PC7 was a
>> success.
> 
> 9s still seems quite a lot to me, but yes, it's an improvement.

It seems it is even better than some other platforms now. Some snapshot 
measurements from running systems:
Xeon E3-1230v5 (Skylake): drift of 4 sec per day (23.999MHz HPET)
Xeon E3-1240v6 (Kaby Lake): drift of 1.9 sec per day (23.999MHz HPET)
Xeon E3-1240v5 (Skylake): drift of 4.85 sec per day (23.999MHz HPET)
Xeon E5-1620v4 (Broadwell): drift of 2.7 sec per day (14.318MHz HPET)

All these values are not great, but it is OK for me.

> Now would you be up to checking whether, rather than via BIOS
> settings (which not all BIOSes may offer) the same can be
> achieved by using Xen's command line option "max_cstate="?
> Also did you check whether further limiting C state use would

I cannot try on production machines. I may have a slot on lab machines 
but I cannot promise.

 > further improve the situation? And did you possibly also check
 > whether telling Xen not to use the HPET would make a difference?

Which other clocksource do you prefer? Is Xen tested (field-proven) on 
that other clocksource?

Regards Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] wall clock drift on Coffee Lake / C24x mainboard (HPET broken?), best practices

[Jan Beulich]

On 19.11.2019 15:31, Andreas Kinzler wrote:
> On 19.11.2019 10:29, Jan Beulich wrote:
>> Now would you be up to checking whether, rather than via BIOS
>> settings (which not all BIOSes may offer) the same can be
>> achieved by using Xen's command line option "max_cstate="?
>> Also did you check whether further limiting C state use would
> 
> I cannot try on production machines. I may have a slot on lab machines 
> but I cannot promise.
> 
>  > further improve the situation? And did you possibly also check
>  > whether telling Xen not to use the HPET would make a difference?
> 
> Which other clocksource do you prefer? Is Xen tested (field-proven) on 
> that other clocksource?

"acpi" (i.e. the PM timer) ought to be fine. That's what Xen was
primarily using before HPET became commonly exposed by the ACPI
tables. "tsc" ought to be fine too on single-socket systems.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel