Re: tsc: Fast TSC calibration failed with sever AMD Ryzen processor (2200G, 2400G, Ryzen 7 1700)

[Paul Menzel <pmenzel@molgen.mpg.de>]

[-- Attachment #1.1: Type: text/plain, Size: 5729 bytes --]

Dear Tom,


On 01/22/19 21:24, Lendacky, Thomas wrote:
> On 1/22/19 10:53 AM, Paul Menzel wrote:
>> [Adding Tom to CC]

>> On 01/14/19 11:09, Paul Menzel wrote:
>>
>>> On 01/11/19 21:43, Thomas Gleixner wrote:
>>>
>>>> On Mon, 7 Jan 2019, Paul Menzel wrote:
>>>>> On 01/07/19 16:24, Thomas Gleixner wrote:
>>>>>>> Linux 4.19.13 from Debian Sid/unstable logs the message below on the board MSI
>>>>>>> MS-7A37/B350M MORTAR with the processor AMD Ryzen 3 2200G.
>>>>>>>
>>>>>>> As a result, the early time stamps do not seem to be working.
>>>>>>
>>>>>>>> [    0.000000] DMI: Micro-Star International Co., Ltd. MS-7A37/B350M MORTAR (MS-7A37), BIOS 1.I0 11/06/2018
>>>>>>>> [    0.000000] tsc: Fast TSC calibration failed
>>>>>>
>>>>>> And the further boot log says:
>>>>>>
>>>>>> [    0.036000] tsc: Unable to calibrate against PIT
>>>>>> [    0.036000] tsc: using HPET reference calibration
>>>>>> [    0.036000] tsc: Detected 3500.117 MHz processor
>>>>>>
>>>>>> So the quick calibration in early boot fails because the PIT seems not to
>>>>>> do what the kernel expects. Nothing we can cure :(
>>>>>
>>>>> I see. Can AMD confirm that this is the expected behavior? If yes, should
>>>>> the fast TSC calibration be skipped on these devices?

> It's not expected behavior. All of the systems that I have access to do
> not exhibit this issue. Having said that, I have a limited number of
> systems available to me.

But as a data point, what Ryzen systems did you test with? Just to know, if
there are configurations where the same processor behaves inconsistently.

Can you request one of the failing systems mentioned below to reproduce the
problem?

> I don't have much experience in this area, but if it is something that
> consistently occurs, you might try to see if you can better identify why
> it fails. The message is issued in pit_hpet_ptimer_calibrate_cpu() in file
> arch/x86/kernel/tsc.c.

With the attached patch applied, I get:

    [    0.000000] tsc: quick_pit_calibrate: break in if !pit_expect_msb, i = 42
    [    0.000000] tsc: Fast TSC calibration failed, i = 42
    [    0.000000] tsc: Using PIT calibration value

The functions `pit_verify_msb()` and `pit_expect_msb()` are:

```
static inline int pit_verify_msb(unsigned char val)
{
        /* Ignore LSB */
        inb(0x42);
        return inb(0x42) == val;
}

static inline int pit_expect_msb(unsigned char val, u64 *tscp, unsigned long *deltap)
{
        int count;
        u64 tsc = 0, prev_tsc = 0;

        for (count = 0; count < 50000; count++) {
                if (!pit_verify_msb(val))
                        break;
                prev_tsc = tsc;
                tsc = get_cycles();
        }
        *deltap = get_cycles() - prev_tsc;
        *tscp = tsc;

        /*
         * We require _some_ success, but the quality control
         * will be based on the error terms on the TSC values.
         */
        return count > 5;
}
```

So count is smaller than or equal to 5, and `pit_verify_msb(val)` failed early,
right?

>>>> It should work and we really don't want to add cpu family/model based
>>>> decisions whether we invoke something or not. Those tables are stale before
>>>> they hit mainline.
>>>
>>> Understood. If it’s supposed to work, any hints on how to debug this?
>>>
>>> Does some Linux kernel developers have an AMD Ryzen system, and can reproduce
>>> the issue?
>>>
>>> It seems to fail with an AMD Ryzen 2400G too [1].
>>
>> We now have an HP EliteDesk 705 G4 MT with that processsor, showing the same
>> problem.
>>
>> ```
>> [    0.000000] Linux version 4.20.0.mx64.238 (root@elcattivo.molgen.mpg.de) (gcc version 7.3.0 (GCC)) #1 SMP Mon Dec 24 14:50:00 CET 2018
>> […]
>> [    0.000000] NX (Execute Disable) protection: active
>> [    0.000000] SMBIOS 3.1 present.
>> [    0.000000] DMI: HP HP EliteDesk 705 G4 MT/83E7, BIOS Q06 Ver. 02.04.01 09/14/2018
>> [    0.000000] tsc: Fast TSC calibration failed
>> [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
>> […]
>> [    0.017860] smpboot: CPU0: AMD Ryzen 5 PRO 2400G with Radeon Vega Graphics (family: 0x17, model: 0x11, stepping: 0x0)
>> […]
>> ```
>>
>>> It also fails on an AMD Ryzen 7 1700 [2].
>>>
>>> ```
>>> [ 0.000000] Linux version 4.15.0-kali3-amd64 (devel@kali.org) (gcc version 7.3.0 (Debian 7.3.0-16)) #1 SMP Debian 4.15.17-1kali1 (2018-04-25)
>>> […]
>>> [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>>> [ 0.028000] tsc: Fast TSC calibration failed
>>> [ 0.032000] tsc: PIT calibration matches HPET. 1 loops
>>> [ 0.032000] tsc: Detected 2994.246 MHz processor
>>> […]
>>> [ 0.044000] smpboot: CPU0: AMD Ryzen 7 1700 Eight-Core Processor (family: 0x17, model: 0x1, stepping: 0x1)
>>> ```
>>>
>>> It *works* here on one system with AMD Ryzen 5 PRO 1500 and Linux 4.14.87.
>>>
>>> ```
>>> [    0.000000] Linux version 4.14.87.mx64.236 (root@likearollingstone.molgen.mpg.de) (gcc version 7.3.0 (GCC)) #1 SMP Mon Dec 10 09:48:57 CET 2018
>>> […]
>>> [    0.000000] tsc: Fast TSC calibration using PIT
>>> […]
>>> [    0.035000] smpboot: CPU0: AMD Ryzen 5 PRO 1500 Quad-Core Processor (family: 0x17, model: 0x1, stepping: 0x1)
>>> ```
>>
>> How to continue from here? Is documentation for that available from AMD?
>> I didn’t find a BKDG (Bios Kernel Developer Guide) at [3].


Kind regards,

Paul


>>> [1]: https://bbs.archlinux.org/viewtopic.php?pid=1781282#p1781282
>>> [2]: https://forums.kali.org/showthread.php?40444-error-loading-amdgpu-drivers-AMD-RX580-driver[3]: https://developer.amd.com/resources/developer-guides-manuals/

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-x86-kernel-tsc-Debug-early-TSC-calibration.patch --]
[-- Type: text/x-patch; name="0001-x86-kernel-tsc-Debug-early-TSC-calibration.patch", Size: 1414 bytes --]

From af43aa1191dd7abcfece56b712966bb19df94653 Mon Sep 17 00:00:00 2001
From: Paul Menzel <pmenzel@molgen.mpg.de>
Date: Wed, 23 Jan 2019 00:24:37 +0100
Subject: [PATCH] x86/kernel/tsc: Debug early TSC calibration

Signed-off-by: Paul Menzel <pmenzel@molgen.mpg.de>
---
 arch/x86/kernel/tsc.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index e9f777bfed40..28d0b4b29668 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -549,8 +549,10 @@ static unsigned long quick_pit_calibrate(void)
 
 	if (pit_expect_msb(0xff, &tsc, &d1)) {
 		for (i = 1; i <= MAX_QUICK_PIT_ITERATIONS; i++) {
-			if (!pit_expect_msb(0xff-i, &delta, &d2))
+			if (!pit_expect_msb(0xff-i, &delta, &d2)) {
+				pr_err("%s: break in if !pit_expect_msb, i = %d\n", __func__, i);
 				break;
+			}
 
 			delta -= tsc;
 
@@ -575,12 +577,14 @@ static unsigned long quick_pit_calibrate(void)
 			 * This also guarantees serialization of the
 			 * last cycle read ('d2') in pit_expect_msb.
 			 */
-			if (!pit_verify_msb(0xfe - i))
+			if (!pit_verify_msb(0xfe - i)) {
+				pr_err("%s: break in if !pit_verify_msb\n", __func__);
 				break;
+			}
 			goto success;
 		}
 	}
-	pr_info("Fast TSC calibration failed\n");
+	pr_info("Fast TSC calibration failed, i = %d\n", i);
 	return 0;
 
 success:
-- 
2.20.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]