Re: [PATCH 1/1] x86/boot/compressed/64: Set EFER.LME=1 in 32-bit trampoline code before returning to long mode

Re: [PATCH 1/1] x86/boot/compressed/64: Set EFER.LME=1 in 32-bit trampoline code before returning to long mode

[Wei Huang]

[adding lkml and linux-x86_64]

On 1/7/19 2:25 AM, Kirill A. Shutemov wrote:
> On Fri, Jan 04, 2019 at 05:44:11AM +0000, Wei Huang wrote:
>> In some old AMD KVM implementation, guest's EFER.LME bit is cleared by KVM
>> when the hypervsior detects guest sets CR0.PG to 0. This causes guest OS
>> to reboot when it tries to return from 32-bit trampoline code because CPU
>> is in incorrect state: CR4.PAE=1, CR0.PG=1, CS.L=1, but EFER.LME=0.
>> As a precaution, this patch sets EFER.LME=1 as part of long mode
>> activation procedure. This extra step won't cause any harm when Linux is
>> booting on bare-metal machine.
>>
>> Signed-off-by: Wei Huang <wei@redhat.com>
> 
> Thanks for tracking this down.

BTW I think this patch _might_ be related the recent reboot issue
reported in https://lkml.org/lkml/2018/7/1/836 since the symptoms are
exactly the same.

> 
> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> Fixes: 34bbb0009f3b ("x86/boot/compressed: Enable 5-level paging during decompression stage")
> 

Re: [PATCH 1/1] x86/boot/compressed/64: Set EFER.LME=1 in 32-bit trampoline code before returning to long mode

[Benjamin Gilbert]

On Mon, Jan 07, 2019 at 02:03:15PM -0600, Wei Huang wrote:
> On 1/7/19 2:25 AM, Kirill A. Shutemov wrote:
> > On Fri, Jan 04, 2019 at 05:44:11AM +0000, Wei Huang wrote:
> >> In some old AMD KVM implementation, guest's EFER.LME bit is cleared by KVM
> >> when the hypervsior detects guest sets CR0.PG to 0. This causes guest OS
> >> to reboot when it tries to return from 32-bit trampoline code because CPU
> >> is in incorrect state: CR4.PAE=1, CR0.PG=1, CS.L=1, but EFER.LME=0.
> >> As a precaution, this patch sets EFER.LME=1 as part of long mode
> >> activation procedure. This extra step won't cause any harm when Linux is
> >> booting on bare-metal machine.
> >>
> >> Signed-off-by: Wei Huang <wei@redhat.com>
> > 
> > Thanks for tracking this down.
> 
> BTW I think this patch _might_ be related the recent reboot issue
> reported in https://lkml.org/lkml/2018/7/1/836 since the symptoms are
> exactly the same.

The problem in that case turned out to be https://lkml.org/lkml/2018/7/4/723
which was fixed by d503ac531a.

--Benjamin Gilbert

Re: [PATCH 1/1] x86/boot/compressed/64: Set EFER.LME=1 in 32-bit trampoline code before returning to long mode

[Wei Huang]

On 1/7/19 3:53 PM, Benjamin Gilbert wrote:
> On Mon, Jan 07, 2019 at 02:03:15PM -0600, Wei Huang wrote:
>> On 1/7/19 2:25 AM, Kirill A. Shutemov wrote:
>>> On Fri, Jan 04, 2019 at 05:44:11AM +0000, Wei Huang wrote:
>>>> In some old AMD KVM implementation, guest's EFER.LME bit is cleared by KVM
>>>> when the hypervsior detects guest sets CR0.PG to 0. This causes guest OS
>>>> to reboot when it tries to return from 32-bit trampoline code because CPU
>>>> is in incorrect state: CR4.PAE=1, CR0.PG=1, CS.L=1, but EFER.LME=0.
>>>> As a precaution, this patch sets EFER.LME=1 as part of long mode
>>>> activation procedure. This extra step won't cause any harm when Linux is
>>>> booting on bare-metal machine.
>>>>
>>>> Signed-off-by: Wei Huang <wei@redhat.com>
>>>
>>> Thanks for tracking this down.
>>
>> BTW I think this patch _might_ be related the recent reboot issue
>> reported in https://lkml.org/lkml/2018/7/1/836 since the symptoms are
>> exactly the same.
> 
> The problem in that case turned out to be https://lkml.org/lkml/2018/7/4/723
> which was fixed by d503ac531a.

OK, then it is a different problem. For this specific patch, without it,
the latest kernel can't boot on RHEL6 (and other old KVM distros) as a
guest VM on AMD box.

> 
> --Benjamin Gilbert
> 

[tip:x86/urgent] x86/boot/compressed/64: Set EFER.LME=1 in 32-bit trampoline before returning to long mode

[tip-bot for Wei Huang]

Commit-ID:  b677dfae5aa197afc5191755a76a8727ffca538a
Gitweb:     https://git.kernel.org/tip/b677dfae5aa197afc5191755a76a8727ffca538a
Author:     Wei Huang <wei@redhat.com>
AuthorDate: Thu, 3 Jan 2019 23:44:11 -0600
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Tue, 29 Jan 2019 21:58:59 +0100

x86/boot/compressed/64: Set EFER.LME=1 in 32-bit trampoline before returning to long mode

In some old AMD KVM implementation, guest's EFER.LME bit is cleared by KVM
when the hypervsior detects that the guest sets CR0.PG to 0. This causes
the guest OS to reboot when it tries to return from 32-bit trampoline code
because the CPU is in incorrect state: CR4.PAE=1, CR0.PG=1, CS.L=1, but
EFER.LME=0.  As a precaution, set EFER.LME=1 as part of long mode
activation procedure. This extra step won't cause any harm when Linux is
booted on a bare-metal machine.

Signed-off-by: Wei Huang <wei@redhat.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: bp@alien8.de
Cc: hpa@zytor.com
Link: https://lkml.kernel.org/r/20190104054411.12489-1-wei@redhat.com

---
 arch/x86/boot/compressed/head_64.S | 8 ++++++++
 arch/x86/boot/compressed/pgtable.h | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 64037895b085..f105ae8651c9 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -600,6 +600,14 @@ ENTRY(trampoline_32bit_src)
 	leal	TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax
 	movl	%eax, %cr3
 3:
+	/* Set EFER.LME=1 as a precaution in case hypervsior pulls the rug */
+	pushl	%ecx
+	movl	$MSR_EFER, %ecx
+	rdmsr
+	btsl	$_EFER_LME, %eax
+	wrmsr
+	popl	%ecx
+
 	/* Enable PAE and LA57 (if required) paging modes */
 	movl	$X86_CR4_PAE, %eax
 	cmpl	$0, %edx
diff --git a/arch/x86/boot/compressed/pgtable.h b/arch/x86/boot/compressed/pgtable.h
index 91f75638f6e6..6ff7e81b5628 100644
--- a/arch/x86/boot/compressed/pgtable.h
+++ b/arch/x86/boot/compressed/pgtable.h
@@ -6,7 +6,7 @@
 #define TRAMPOLINE_32BIT_PGTABLE_OFFSET	0
 
 #define TRAMPOLINE_32BIT_CODE_OFFSET	PAGE_SIZE
-#define TRAMPOLINE_32BIT_CODE_SIZE	0x60
+#define TRAMPOLINE_32BIT_CODE_SIZE	0x70
 
 #define TRAMPOLINE_32BIT_STACK_END	TRAMPOLINE_32BIT_SIZE