[PATCH 0/2] Fix TSC ADJUST breakage causing TSC failure

[PATCH 0/2] Fix TSC ADJUST breakage causing TSC failure

[Mike Travis]

Fix a breakage caused by enabling early tsc initialization which bypasses
a check that disables the forcing of TSC ADJUST to 0 for chassis 0.
This is common on systems where all the chassis start up asynchronously
so which chassis should have a TSC ADJUST value of 0 is not predictable.

The solution is to add a check earlier than this early tsc init to
disable the potential of it incorrectly adjusting TSC ADJUST values that
are already correctly initialized.

*  Patch 1 adds an early callable function (after efi_init) that will
   check if this system might be a UV system.

*  Patch 2 adds code to tsc_early_init() which disables adjusting the
   TSC ADJUST value if it's a UV system.  This allows the later tsc_init
   function to test the tsc_async_resets flag that indicates the system
   chassis start up asynchronously, so which chassis should have a TSC
   ADJUST value of 0 is not predictable.  Further references are in
   the patch.

-- 

[PATCH 1/2] x86/platform/uv: Add is_early_uv_system check

[Mike Travis]

[-- Attachment #1: add_is_early_uv_system --]
[-- Type: text/plain, Size: 1391 bytes --]

Introduce is_early_uv_system() which uses efi.uv_systab to decide early
in the boot process whether we're on a UV system.

This is needed to skip other early setup/init code that might break
the UV platform if done too early such as before necessary ACPI tables
parsing takes place.

Signed-off-by: Mike Travis <mike.travis@hpe.com>
Suggested-by: Hedi Berriche <hedi.berriche@hpe.com>
Reviewed-by: Russ Anderson <rja@hpe.com>
Reviewed-by: Dimitri Sivanich <sivanich@hpe.com>
---
 arch/x86/include/asm/uv/uv.h |    6 ++++++
 1 file changed, 6 insertions(+)

--- linux.orig/arch/x86/include/asm/uv/uv.h
+++ linux/arch/x86/include/asm/uv/uv.h
@@ -10,8 +10,13 @@ struct cpumask;
 struct mm_struct;
 
 #ifdef CONFIG_X86_UV
+#include <linux/efi.h>
 
 extern enum uv_system_type get_uv_system_type(void);
+static inline bool is_early_uv_system(void)
+{
+	return !((efi.uv_systab == EFI_INVALID_TABLE_ADDR) || !efi.uv_systab);
+}
 extern int is_uv_system(void);
 extern int is_uv_hubless(void);
 extern void uv_cpu_init(void);
@@ -23,6 +28,7 @@ extern const struct cpumask *uv_flush_tl
 #else	/* X86_UV */
 
 static inline enum uv_system_type get_uv_system_type(void) { return UV_NONE; }
+static inline bool is_early_uv_system(void)	{ return 0; }
 static inline int is_uv_system(void)	{ return 0; }
 static inline int is_uv_hubless(void)	{ return 0; }
 static inline void uv_cpu_init(void)	{ }

-- 

[PATCH 2/2] x86/tsc: Fix UV TSC initialization

[Mike Travis]

[-- Attachment #1: fix_tsc_early_init --]
[-- Type: text/plain, Size: 1511 bytes --]

Fix regression introduced by

commit cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")

as it added a call to tsc_early_init() which initializes the TSC ADJUST
values before acpi_boot_table_init().  In the case of UV systems,
that is a necessary step thats calls uv_system_init().  This informs
tsc_sanitize_first_cpu() that we're on a platform with async TSC resets
as documented in

commit 341102c3ef29 ("x86/tsc: Add option that TSC on Socket 0 being non-zero is valid")

Fix by skipping the early tsc initialization on UV systems and let TSC
init tests take place later in tsc_init().

Fixes: cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once") 
Signed-off-by: Mike Travis <mike.travis@hpe.com>
Suggested-by: Hedi Berriche <hedi.berriche@hpe.com>
Reviewed-by: Russ Anderson <rja@hpe.com>
Reviewed-by: Dimitri Sivanich <sivanich@hpe.com>
---
 arch/x86/kernel/tsc.c |    4 ++++
 1 file changed, 4 insertions(+)

--- linux.orig/arch/x86/kernel/tsc.c
+++ linux/arch/x86/kernel/tsc.c
@@ -26,6 +26,7 @@
 #include <asm/apic.h>
 #include <asm/intel-family.h>
 #include <asm/i8259.h>
+#include <asm/uv/uv.h>
 
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);
@@ -1433,6 +1434,9 @@ void __init tsc_early_init(void)
 {
 	if (!boot_cpu_has(X86_FEATURE_TSC))
 		return;
+	/* Don't change UV TSC multi-chassis synchronization */
+	if (is_early_uv_system())
+		return;
 	if (!determine_cpu_tsc_frequencies(true))
 		return;
 	loops_per_jiffy = get_loops_per_jiffy();

-- 

[tip:x86/urgent] x86/platform/uv: Provide is_early_uv_system()

[tip-bot for Mike Travis]

Commit-ID:  20a8378aa9dd108a01cb0e695599f5257a885c4b
Gitweb:     https://git.kernel.org/tip/20a8378aa9dd108a01cb0e695599f5257a885c4b
Author:     Mike Travis <mike.travis@hpe.com>
AuthorDate: Tue, 2 Oct 2018 13:01:45 -0500
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Tue, 2 Oct 2018 21:29:16 +0200

x86/platform/uv: Provide is_early_uv_system()

Introduce is_early_uv_system() which uses efi.uv_systab to decide early
in the boot process whether the kernel runs on a UV system.

This is needed to skip other early setup/init code that might break
the UV platform if done too early such as before necessary ACPI tables
parsing takes place.

Suggested-by: Hedi Berriche <hedi.berriche@hpe.com>
Signed-off-by: Mike Travis <mike.travis@hpe.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Russ Anderson <rja@hpe.com>
Reviewed-by: Dimitri Sivanich <sivanich@hpe.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Russ Anderson <russ.anderson@hpe.com>
Cc: Dimitri Sivanich <dimitri.sivanich@hpe.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Kate Stewart <kstewart@linuxfoundation.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Philippe Ombredanne <pombredanne@nexb.com>
Cc: Pavel Tatashin <pasha.tatashin@oracle.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Len Brown <len.brown@intel.com>
Cc: Dou Liyang <douly.fnst@cn.fujitsu.com>
Cc: Xiaoming Gao <gxm.linux.kernel@gmail.com>
Cc: Rajvi Jingar <rajvi.jingar@intel.com>
Link: https://lkml.kernel.org/r/20181002180144.801700401@stormcage.americas.sgi.com

---
 arch/x86/include/asm/uv/uv.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/x86/include/asm/uv/uv.h b/arch/x86/include/asm/uv/uv.h
index a80c0673798f..e60c45fd3679 100644
--- a/arch/x86/include/asm/uv/uv.h
+++ b/arch/x86/include/asm/uv/uv.h
@@ -10,8 +10,13 @@ struct cpumask;
 struct mm_struct;
 
 #ifdef CONFIG_X86_UV
+#include <linux/efi.h>
 
 extern enum uv_system_type get_uv_system_type(void);
+static inline bool is_early_uv_system(void)
+{
+	return !((efi.uv_systab == EFI_INVALID_TABLE_ADDR) || !efi.uv_systab);
+}
 extern int is_uv_system(void);
 extern int is_uv_hubless(void);
 extern void uv_cpu_init(void);
@@ -23,6 +28,7 @@ extern const struct cpumask *uv_flush_tlb_others(const struct cpumask *cpumask,
 #else	/* X86_UV */
 
 static inline enum uv_system_type get_uv_system_type(void) { return UV_NONE; }
+static inline bool is_early_uv_system(void)	{ return 0; }
 static inline int is_uv_system(void)	{ return 0; }
 static inline int is_uv_hubless(void)	{ return 0; }
 static inline void uv_cpu_init(void)	{ }

[tip:x86/urgent] x86/tsc: Fix UV TSC initialization

[tip-bot for Mike Travis]

Commit-ID:  2647c43c7f3ba4b752bfce261d53b16e2f5bc9e3
Gitweb:     https://git.kernel.org/tip/2647c43c7f3ba4b752bfce261d53b16e2f5bc9e3
Author:     Mike Travis <mike.travis@hpe.com>
AuthorDate: Tue, 2 Oct 2018 13:01:46 -0500
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Tue, 2 Oct 2018 21:29:16 +0200

x86/tsc: Fix UV TSC initialization

The recent rework of the TSC calibration code introduced a regression on UV
systems as it added a call to tsc_early_init() which initializes the TSC
ADJUST values before acpi_boot_table_init().  In the case of UV systems,
that is a necessary step that calls uv_system_init().  This informs
tsc_sanitize_first_cpu() that the kernel runs on a platform with async TSC
resets as documented in commit 341102c3ef29 ("x86/tsc: Add option that TSC
on Socket 0 being non-zero is valid")

Fix it by skipping the early tsc initialization on UV systems and let TSC
init tests take place later in tsc_init().

Fixes: cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")
Suggested-by: Hedi Berriche <hedi.berriche@hpe.com>
Signed-off-by: Mike Travis <mike.travis@hpe.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Russ Anderson <rja@hpe.com>
Reviewed-by: Dimitri Sivanich <sivanich@hpe.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Russ Anderson <russ.anderson@hpe.com>
Cc: Dimitri Sivanich <dimitri.sivanich@hpe.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Kate Stewart <kstewart@linuxfoundation.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Philippe Ombredanne <pombredanne@nexb.com>
Cc: Pavel Tatashin <pasha.tatashin@oracle.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Len Brown <len.brown@intel.com>
Cc: Dou Liyang <douly.fnst@cn.fujitsu.com>
Cc: Xiaoming Gao <gxm.linux.kernel@gmail.com>
Cc: Rajvi Jingar <rajvi.jingar@intel.com>
Link: https://lkml.kernel.org/r/20181002180144.923579706@stormcage.americas.sgi.com

---
 arch/x86/kernel/tsc.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index 6490f618e096..b52bd2b6cdb4 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -26,6 +26,7 @@
 #include <asm/apic.h>
 #include <asm/intel-family.h>
 #include <asm/i8259.h>
+#include <asm/uv/uv.h>
 
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);
@@ -1433,6 +1434,9 @@ void __init tsc_early_init(void)
 {
 	if (!boot_cpu_has(X86_FEATURE_TSC))
 		return;
+	/* Don't change UV TSC multi-chassis synchronization */
+	if (is_early_uv_system())
+		return;
 	if (!determine_cpu_tsc_frequencies(true))
 		return;
 	loops_per_jiffy = get_loops_per_jiffy();

Re: [PATCH 2/2] x86/tsc: Fix UV TSC initialization

[Mike Travis]

On 10/1/2018 11:22 PM, Thomas Gleixner wrote:
> On Mon, 1 Oct 2018, Mike Travis wrote:
> 
>> Fix regression introduced by
>>
>> commit cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")
>>
>> as it changed setup_arch() so that it now calls tsc_early_init() before
>> acpi_boot_table_init() which is a necessary step, in the case of UV
>> systems, to inform tsc_sanitize_first_cpu() that we're on a platform
>> with async TSC resets as documented in
>>
>> commit 341102c3ef29 ("x86/tsc: Add option that TSC on Socket 0 being non-zero is valid")
>>
>> Fix by skipping tsc_early_init() on UV systems and let TSC initialization
>> take place later in tsc_init().
>>
>> Fixes: cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")
>> Signed-off-by: Mike Travis <mike.travis@hpe.com>
>> Signed-off-by: Hedi Berriche <hedi.berriche@hpe.com>
> 
> See previous mail.
> 
>> Reviewed-by: Russ Anderson <rja@hpe.com>
>> Reviewed-by: Dimitri Sivanich <sivanich@hpe.com>
>> ---
>>   arch/x86/kernel/setup.c |    6 +++++-
>>   1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> --- linux.orig/arch/x86/kernel/setup.c
>> +++ linux/arch/x86/kernel/setup.c
>> @@ -117,6 +117,7 @@
>>   #include <asm/microcode.h>
>>   #include <asm/kaslr.h>
>>   #include <asm/unwind.h>
>> +#include <asm/uv/uv.h>
>>   
>>   /*
>>    * max_low_pfn_mapped: highest direct mapped pfn under 4GB
>> @@ -1015,7 +1016,10 @@ void __init setup_arch(char **cmdline_p)
>>   	 */
>>   	init_hypervisor_platform();
>>   
>> -	tsc_early_init();
>> +	/* UV TSC multi-chassis synchronization already set, don't change it */
>> +	if (!is_early_uv_system())
>> +		tsc_early_init();
> 
> tsc_early_init() has already a check which makes it not invoke the
> calibration code, so please put your uv check into that.

Will do, thanks.

> 
> Thanks,
> 
> 	tglx
> 

Re: [PATCH 2/2] x86/tsc: Fix UV TSC initialization

[Thomas Gleixner]

On Mon, 1 Oct 2018, Mike Travis wrote:

> Fix regression introduced by
> 
> commit cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")
> 
> as it changed setup_arch() so that it now calls tsc_early_init() before
> acpi_boot_table_init() which is a necessary step, in the case of UV
> systems, to inform tsc_sanitize_first_cpu() that we're on a platform
> with async TSC resets as documented in
> 
> commit 341102c3ef29 ("x86/tsc: Add option that TSC on Socket 0 being non-zero is valid")
> 
> Fix by skipping tsc_early_init() on UV systems and let TSC initialization
> take place later in tsc_init().
> 
> Fixes: cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once") 
> Signed-off-by: Mike Travis <mike.travis@hpe.com>
> Signed-off-by: Hedi Berriche <hedi.berriche@hpe.com>

See previous mail.

> Reviewed-by: Russ Anderson <rja@hpe.com>
> Reviewed-by: Dimitri Sivanich <sivanich@hpe.com>
> ---
>  arch/x86/kernel/setup.c |    6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> --- linux.orig/arch/x86/kernel/setup.c
> +++ linux/arch/x86/kernel/setup.c
> @@ -117,6 +117,7 @@
>  #include <asm/microcode.h>
>  #include <asm/kaslr.h>
>  #include <asm/unwind.h>
> +#include <asm/uv/uv.h>
>  
>  /*
>   * max_low_pfn_mapped: highest direct mapped pfn under 4GB
> @@ -1015,7 +1016,10 @@ void __init setup_arch(char **cmdline_p)
>  	 */
>  	init_hypervisor_platform();
>  
> -	tsc_early_init();
> +	/* UV TSC multi-chassis synchronization already set, don't change it */
> +	if (!is_early_uv_system())
> +		tsc_early_init();

tsc_early_init() has already a check which makes it not invoke the
calibration code, so please put your uv check into that.

Thanks,

	tglx

[PATCH 2/2] x86/tsc: Fix UV TSC initialization

[Mike Travis]

[-- Attachment #1: fix_tsc_early_init --]
[-- Type: text/plain, Size: 1477 bytes --]

Fix regression introduced by

commit cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")

as it changed setup_arch() so that it now calls tsc_early_init() before
acpi_boot_table_init() which is a necessary step, in the case of UV
systems, to inform tsc_sanitize_first_cpu() that we're on a platform
with async TSC resets as documented in

commit 341102c3ef29 ("x86/tsc: Add option that TSC on Socket 0 being non-zero is valid")

Fix by skipping tsc_early_init() on UV systems and let TSC initialization
take place later in tsc_init().

Fixes: cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once") 
Signed-off-by: Mike Travis <mike.travis@hpe.com>
Signed-off-by: Hedi Berriche <hedi.berriche@hpe.com>
Reviewed-by: Russ Anderson <rja@hpe.com>
Reviewed-by: Dimitri Sivanich <sivanich@hpe.com>
---
 arch/x86/kernel/setup.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

--- linux.orig/arch/x86/kernel/setup.c
+++ linux/arch/x86/kernel/setup.c
@@ -117,6 +117,7 @@
 #include <asm/microcode.h>
 #include <asm/kaslr.h>
 #include <asm/unwind.h>
+#include <asm/uv/uv.h>
 
 /*
  * max_low_pfn_mapped: highest direct mapped pfn under 4GB
@@ -1015,7 +1016,10 @@ void __init setup_arch(char **cmdline_p)
 	 */
 	init_hypervisor_platform();
 
-	tsc_early_init();
+	/* UV TSC multi-chassis synchronization already set, don't change it */
+	if (!is_early_uv_system())
+		tsc_early_init();
+
 	x86_init.resources.probe_roms();
 
 	/* after parse_early_param, so could debug it */

--