Re: [PATCH 05/23] x86, kaiser: unmap kernel from userspace page tables (core patch)

[Andrea Arcangeli <aarcange@redhat.com>]

Hi Yisheng and Dave,

On Thu, Jan 04, 2018 at 10:29:53PM -0800, Dave Hansen wrote:
> On 01/04/2018 10:16 PM, Yisheng Xie wrote:
> > BTW, we have just reported a bug caused by kaiser[1], which looks like
> > caused by SMEP. Could you please help to have a look?
> > 
> > [1] https://lkml.org/lkml/2018/1/5/3
> 
> Please report that to your kernel vendor.  Your EFI page tables have the
> NX bit set on the low addresses.  There have been a bunch of iterations
> of this, but you need to make sure that the EFI kernel mappings don't
> get _PAGE_NX set on them.  Look at what __pti_set_user_pgd() does in
> mainline.

Yisheng could you file a report on the vendor bz?

>From my part of course I'm fine to discuss it here, but it's not fair
to use lkml bandwidth for this, sorry for the noise.

The vast majority of the hardware boots fine and isn't running into
this. This is the first time I hear about this, sorry about that.

I fixed it with the upstream solution, greatly appreciated the pointer
Dave. I don't have hardware to verify it though so we've to follow up
on bz.

Thanks,
Andrea

>From 74e2d799b7c22f00a8d3158958e3d6d9fa45c1d2 Mon Sep 17 00:00:00 2001
From: Andrea Arcangeli <aarcange@redhat.com>
Date: Fri, 5 Jan 2018 11:39:40 +0100
Subject: [RHEL7.5 PATCH 1/1] x86/pti/mm: don't set NX on EFI mapping without
 _PAGE_USER

The kernel must be able to execute EFI code in userland (positive
virtual address space) without _PAGE_USER set, so don't set NX on
it. This only selectively disables the NX poisoning in kernel pgd so
there's no effect whatsoever on the page table isolation from userland
point of view.

Solves this crash at boot:

[    0.039130] BUG: unable to handle kernel paging request at 000000005b835f90
[    0.046101] IP: [<000000005b835f90>] 0x5b835f8f
[    0.050637] PGD 8000000001f61067 PUD 190ffefff067 PMD 190ffeffd067 PTE 5b835063
[    0.057989] Oops: 0011 [#1] SMP
[    0.061241] Modules linked in:
[    0.064304] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-327.59.59.46.h42.x86_64 #1
[    0.072280] Hardware name: Huawei FusionServer9032/IT91SMUB, BIOS BLXSV316 11/14/2017
[    0.080082] task: ffffffff8196e440 ti: ffffffff81958000 task.ti: ffffffff81958000
[    0.087539] RIP: 0010:[<000000005b835f90>]  [<000000005b835f90>] 0x5b835f8f
[    0.094494] RSP: 0000:ffffffff8195be28  EFLAGS: 00010046
[    0.099788] RAX: 0000000080050033 RBX: ffff910fbc802000 RCX: 00000000000002d0
[    0.106897] RDX: 0000000000000030 RSI: 00000000000002d0 RDI: 000000005b835f90
[    0.114006] RBP: ffffffff8195bf38 R08: 0000000000000001 R09: 0000090fbc802000
[    0.121116] R10: ffff88ffbcc07340 R11: 0000000000000001 R12: 0000000000000001
[    0.128225] R13: 0000090fbc802000 R14: 00000000000002d0 R15: 0000000000000001
[    0.135336] FS:  0000000000000000(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000
[    0.143398] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.149124] CR2: 000000005b835f90 CR3: 0000000001966000 CR4: 00000000000606b0
[    0.156234] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.163344] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    0.170454] Call Trace:
[    0.172899]  [<ffffffff8107512c>] ? efi_call4+0x6c/0xf0
[    0.178108]  [<ffffffff8105b3fe>] ? native_flush_tlb_global+0x8e/0xc0
[    0.184527]  [<ffffffff810652b3>] ? set_memory_x+0x43/0x50
[    0.189997]  [<ffffffff81acf91f>] ? efi_enter_virtual_mode+0x3bc/0x538
[    0.196505]  [<ffffffff81ab104b>] start_kernel+0x39f/0x44f
[    0.201972]  [<ffffffff81ab0ab5>] ? repair_env_string+0x5c/0x5c
[    0.207872]  [<ffffffff81ab0120>] ? early_idt_handlers+0x120/0x120
[    0.214030]  [<ffffffff81ab066c>] x86_64_start_reservations+0x2a/0x2c
[    0.220449]  [<ffffffff81ab07c0>] x86_64_start_kernel+0x152/0x175
[    0.226521] Code:  Bad RIP value.
[    0.229860] RIP  [<000000005b835f90>] 0x5b835f8f
[    0.234478]  RSP <ffffffff8195be28>
[    0.237955] CR2: 000000005b835f90
[    0.241266] ---[ end trace 8178226af3e802ca ]---
[    0.245869] Kernel panic - not syncing: Fatal exception

Reported-by: Yisheng Xie <xieyisheng1@huawei.com>
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---
 arch/x86/include/asm/pgtable_64.h | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h
index 7c8bc5c23664..132176fe45e2 100644
--- a/arch/x86/include/asm/pgtable_64.h
+++ b/arch/x86/include/asm/pgtable_64.h
@@ -189,28 +189,34 @@ static inline bool pgd_userspace_access(pgd_t pgd)
 	return pgd.pgd & _PAGE_USER;
 }
 
+#define _PAGE_PTI_CAN_NX (_PAGE_PRESENT|_PAGE_USER)
+
 static inline void kaiser_poison_pgd(pgd_t *pgd)
 {
-	if (pgd->pgd & _PAGE_PRESENT && __supported_pte_mask & _PAGE_NX)
+	if ((pgd->pgd & _PAGE_PTI_CAN_NX) == _PAGE_PTI_CAN_NX &&
+	    __supported_pte_mask & _PAGE_NX)
 		pgd->pgd |= _PAGE_NX;
 }
 
 static inline void kaiser_unpoison_pgd(pgd_t *pgd)
 {
-	if (pgd->pgd & _PAGE_PRESENT && __supported_pte_mask & _PAGE_NX)
+	if ((pgd->pgd & _PAGE_PTI_CAN_NX) == _PAGE_PTI_CAN_NX &&
+	    __supported_pte_mask & _PAGE_NX)
 		pgd->pgd &= ~_PAGE_NX;
 }
 
 static inline void kaiser_poison_pgd_atomic(pgd_t *pgd)
 {
 	BUILD_BUG_ON(_PAGE_NX == 0);
-	if (pgd->pgd & _PAGE_PRESENT && __supported_pte_mask & _PAGE_NX)
+	if ((pgd->pgd & _PAGE_PTI_CAN_NX) == _PAGE_PTI_CAN_NX &&
+	    __supported_pte_mask & _PAGE_NX)
 		set_bit(_PAGE_BIT_NX, &pgd->pgd);
 }
 
 static inline void kaiser_unpoison_pgd_atomic(pgd_t *pgd)
 {
-	if (pgd->pgd & _PAGE_PRESENT && __supported_pte_mask & _PAGE_NX)
+	if ((pgd->pgd & _PAGE_PTI_CAN_NX) == _PAGE_PTI_CAN_NX &&
+	    __supported_pte_mask & _PAGE_NX)
 		clear_bit(_PAGE_BIT_NX, &pgd->pgd);
 }