efi boot failures due to PTI with 32 bit builds and Intel CPUs

efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Guenter Roeck]

Hi all,

I see boot failures on mainline when trying to boot x86 images with an efi
bios on Intel CPUs in qemu. Behavior is quite unusual: qemu dies silently
after the kernel displays "Run /sbin/init as init process". With debugging
enabled, qemu reports a CR3 update followed by a triple fault.
Here is the end of the log file:

----------------
IN:
0xc75f1d1a:  66 90                    nop
0xc75f1d1c:  0f 20 d8                 movl     %cr3, %eax
0xc75f1d1f:  0d 00 10 00 00           orl      $0x1000, %eax
0xc75f1d24:  0f 22 d8                 movl     %eax, %cr3

CR3 update: CR3=0e39b000
----------------
IN:
0xc75f1d27:  5b                       popl     %ebx
0xc75f1d28:  59                       popl     %ecx
0xc75f1d29:  5a                       popl     %edx
0xc75f1d2a:  5e                       popl     %esi
0xc75f1d2b:  5f                       popl     %edi
0xc75f1d2c:  5d                       popl     %ebp
0xc75f1d2d:  58                       popl     %eax
0xc75f1d2e:  1f                       popl     %ds

Triple fault

This happens with both qemu 2.12 and 3.0. More detailed logs (not really
showing anything) are at http://kerneltests.org/builders; look for x86
boot reports for master and next towards the end of the page.

Here is an example qemu command line:

qemu-system-i386 -kernel arch/x86/boot/bzImage -M q35 -cpu core2duo \
	-no-reboot -m 256 \
	-bios OVMF-pure-efi-32.fd \
	-usb -device usb-storage,drive=d0 \
	-drive file=rootfs.ext2,if=none,id=d0,format=raw \
	--append 'root=/dev/sda rw rootwait mem=256M console=ttyS0 console=tty noreboot' \
	-nographic

The problem is only seen in mainline (v4.19-rc1). It is not seen in earlier
kernels. It does not really matter what to boot from as long as the boot is
with an efi bios and an Intel CPU (AMD CPUs boot fine). Bisect was a bit
tricky (see multiple runs below), but ultimately points to commit
7757d607c6b31 ("x86/pti: Allow CONFIG_PAGE_TABLE_ISOLATION for x86_32")
as the culprit. Reverting this commit indeed fixes the problem.

Please let me know if I can help tracking down the underlying issue.

Thanks,
Guenter

---
# bad: [3f16503b7d2274ac8cbab11163047ac0b4c66cfe] Merge branch 'fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/evalenti/linux-soc-thermal
# good: [94710cac0ef4ee177a63b5227664b38c95bbf703] Linux 4.18
git bisect start 'HEAD' 'v4.18'
# bad: [54dbe75bbf1e189982516de179147208e90b5e45] Merge tag 'drm-next-2018-08-15' of git://anongit.freedesktop.org/drm/drm
git bisect bad 54dbe75bbf1e189982516de179147208e90b5e45
# bad: [0a957467c5fd46142bc9c52758ffc552d4c5e2f7] x86: i8259: Add missing include file
git bisect bad 0a957467c5fd46142bc9c52758ffc552d4c5e2f7
# bad: [958f338e96f874a0d29442396d6adf9c1e17aa2d] Merge branch 'l1tf-final' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 958f338e96f874a0d29442396d6adf9c1e17aa2d
# bad: [85a0b791bc17f7a49280b33e2905d109c062a47b] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
git bisect bad 85a0b791bc17f7a49280b33e2905d109c062a47b
# good: [8603596a327c978534f5c45db135e6c36b4b1425] Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 8603596a327c978534f5c45db135e6c36b4b1425
# bad: [eac341194426ba7ead3444923b9eba491ae4feeb] Merge branch 'x86/pti' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad eac341194426ba7ead3444923b9eba491ae4feeb
# good: [30de24c7dd21348b142ee977b687afc70b392af6] Merge branch 'x86-cache-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 30de24c7dd21348b142ee977b687afc70b392af6
# bad: [8c934e01a7ce685d98e970880f5941d79272c654] x86/pti: Check the return value of pti_user_pagetable_walk_pmd()
git bisect bad 8c934e01a7ce685d98e970880f5941d79272c654
# bad: [fcbbd977572cfe5a3dcc97d663bf7480431a07ca] x86/pgtable: Move pti_set_user_pgtbl() to pgtable.h
git bisect bad fcbbd977572cfe5a3dcc97d663bf7480431a07ca
# bad: [e5862d0515ad970ccec6208ecf5bb0cffe291ea3] x86/entry/32: Leave the kernel via trampoline stack
git bisect bad e5862d0515ad970ccec6208ecf5bb0cffe291ea3
# bad: [a6b744f3ce9d017dd86b28355de2d8e0d36496d4] x86/entry/32: Load task stack from x86_tss.sp1 in SYSENTER handler
git bisect bad a6b744f3ce9d017dd86b28355de2d8e0d36496d4
# bad: [d9f4426c73002957be5dd39936f44a09498f7560] x86/speculation: Remove SPECTRE_V2_IBRS in enum spectre_v2_mitigation
git bisect bad d9f4426c73002957be5dd39936f44a09498f7560
# bad: [21279157efffe5e7258483809942d576cb802768] x86/pti: Make pti_set_kernel_image_nonglobal() static
git bisect bad 21279157efffe5e7258483809942d576cb802768
# first bad commit: [21279157efffe5e7258483809942d576cb802768] x86/pti: Make pti_set_kernel_image_nonglobal() static

This doesn't really mean anything: The incoming merge is already broken
due to commit e181ae0c5db9, but that should be fixed in mainline.

---
# bad: [eac341194426ba7ead3444923b9eba491ae4feeb] Merge branch 'x86/pti' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
# good: [d191c82d4d9bd0bb3b945fc458cc65053ef868a0] Merge branch 'x86-vdso-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect start 'eac341194426' 'd191c82d4d9b'
# bad: [b976690f5db26fbc7c2be413bfa0fbd270547a94] x86/mm/pti: Introduce pti_finalize()
git bisect bad b976690f5db26fbc7c2be413bfa0fbd270547a94
# bad: [b65bef400689ceee7108c2d47fb97ae91f4d1440] x86/entry/32: Add PTI CR3 switches to NMI handler code
git bisect bad b65bef400689ceee7108c2d47fb97ae91f4d1440
# bad: [8e676ced31e9d1448d3ffc4159586a259cc67f30] x86/entry/32: Unshare NMI return path
git bisect bad 8e676ced31e9d1448d3ffc4159586a259cc67f30
# bad: [9e97b73fdb235345a826519862a52a7398c89eb8] x86/asm-offsets: Move TSS_sp0 and TSS_sp1 to asm-offsets.c
git bisect bad 9e97b73fdb235345a826519862a52a7398c89eb8
# bad: [d9f4426c73002957be5dd39936f44a09498f7560] x86/speculation: Remove SPECTRE_V2_IBRS in enum spectre_v2_mitigation
git bisect bad d9f4426c73002957be5dd39936f44a09498f7560
# bad: [21279157efffe5e7258483809942d576cb802768] x86/pti: Make pti_set_kernel_image_nonglobal() static
git bisect bad 21279157efffe5e7258483809942d576cb802768
# first bad commit: [21279157efffe5e7258483809942d576cb802768] x86/pti: Make pti_set_kernel_image_nonglobal() static

---
# bad: [21279157efffe5e7258483809942d576cb802768] x86/pti: Make pti_set_kernel_image_nonglobal() static
# good: [1e4b044d22517cae7047c99038abb444423243ca] Linux 4.18-rc4
git bisect start '21279157efffe5e7258483809942d576cb802768' 'v4.18-rc4'
# good: [35a84f34cf41915a0b2d0a3688b20761580f8ce4] Merge tag 'trace-v4.18-rc3-3' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace
git bisect good 35a84f34cf41915a0b2d0a3688b20761580f8ce4
# good: [75adbd1386796c1234035996c6aec3ede4060eb2] Merge branch 'efi-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 75adbd1386796c1234035996c6aec3ede4060eb2
# good: [2db39a2f491a48ec740e0214a7dd584eefc2137d] Merge branch 'i2c/for-current' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect good 2db39a2f491a48ec740e0214a7dd584eefc2137d
# good: [fe10e398e860955bac4d28ec031b701d358465e4] reiserfs: fix buffer overflow with long warning messages
git bisect good fe10e398e860955bac4d28ec031b701d358465e4
# bad: [c31496dbacc2b6352750937afc20a8dbe22b27a4] Merge tag 'for-linus-4.18-rc5-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
git bisect bad c31496dbacc2b6352750937afc20a8dbe22b27a4
# bad: [2da8c426d90355eef1d42d974d2dccf0f5f7f21d] Merge tag 'for-linus-20180713' of git://git.kernel.dk/linux-block
git bisect bad 2da8c426d90355eef1d42d974d2dccf0f5f7f21d
# bad: [f353078f028fbfe9acd4b747b4a19c69ef6846cd] Merge branch 'akpm' (patches from Andrew)
git bisect bad f353078f028fbfe9acd4b747b4a19c69ef6846cd
# bad: [e181ae0c5db9544de9c53239eb22bc012ce75033] mm: zero unavailable pages before memmap init
git bisect bad e181ae0c5db9544de9c53239eb22bc012ce75033
# first bad commit: [e181ae0c5db9544de9c53239eb22bc012ce75033] mm: zero unavailable pages before memmap init

This 'bad' patch created a problem with 32-bit images which was later
fixed with commit d1b47a7c9efc ("mm: don't do zero_resv_unavail if memmap
is not allocated"). Reverting those two patches in mainline does _not_
fix the problem. With that in mind, I applied commit d1b47a7c9efc on top
of 21279157efffe and ran another test. This test passed, so the above is
a false positive.

---
Another bisect run, this time applying d1b47a7c9efc if 21279157efffe
is in the image but d1b47a7c9efc isn't.

# bad: [781fca5b104693bc9242199cc47c690dcaf6a4cb] Merge tag 'xfs-4.19-merge-6' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
# good: [94710cac0ef4ee177a63b5227664b38c95bbf703] Linux 4.18
git bisect start 'HEAD' 'v4.18'
# bad: [85a0b791bc17f7a49280b33e2905d109c062a47b] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
git bisect bad 85a0b791bc17f7a49280b33e2905d109c062a47b
# good: [8603596a327c978534f5c45db135e6c36b4b1425] Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 8603596a327c978534f5c45db135e6c36b4b1425
# bad: [eac341194426ba7ead3444923b9eba491ae4feeb] Merge branch 'x86/pti' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad eac341194426ba7ead3444923b9eba491ae4feeb
# good: [30de24c7dd21348b142ee977b687afc70b392af6] Merge branch 'x86-cache-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 30de24c7dd21348b142ee977b687afc70b392af6
# bad: [8c934e01a7ce685d98e970880f5941d79272c654] x86/pti: Check the return value of pti_user_pagetable_walk_pmd()
git bisect bad 8c934e01a7ce685d98e970880f5941d79272c654
# good: [fcbbd977572cfe5a3dcc97d663bf7480431a07ca] x86/pgtable: Move pti_set_user_pgtbl() to pgtable.h
git bisect good fcbbd977572cfe5a3dcc97d663bf7480431a07ca
# good: [ba0364e260ab37c02975557dbecc014a26072236] x86/mm/pti: Clone entry-text again in pti_finalize()
git bisect good ba0364e260ab37c02975557dbecc014a26072236
# good: [9bae3197e15dd5e03ce8e237db6fe4486b08a775] x86/ldt: Split out sanity check in map_ldt_struct()
git bisect good 9bae3197e15dd5e03ce8e237db6fe4486b08a775
# bad: [5e8105950a8b3e03e805299b4d05020ee4eda31a] x86/mm/pti: Add Warning when booting on a PCID capable CPU
git bisect bad 5e8105950a8b3e03e805299b4d05020ee4eda31a
# bad: [7757d607c6b31867777de42e1fb0210b9c5d8b70] x86/pti: Allow CONFIG_PAGE_TABLE_ISOLATION for x86_32
git bisect bad 7757d607c6b31867777de42e1fb0210b9c5d8b70
# good: [6df934b92a549cb3badb6d576f71aeb133e2f110] x86/ldt: Enable LDT user-mapping for PAE
git bisect good 6df934b92a549cb3badb6d576f71aeb133e2f110
# first bad commit: [7757d607c6b31867777de42e1fb0210b9c5d8b70] x86/pti: Allow CONFIG_PAGE_TABLE_ISOLATION for x86_32

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Dave Hansen]

On 08/29/2018 01:16 PM, Guenter Roeck wrote:
> 
> I see boot failures on mainline when trying to boot x86 images with an efi
> bios on Intel CPUs in qemu. Behavior is quite unusual: qemu dies silently
> after the kernel displays "Run /sbin/init as init process". With debugging
> enabled, qemu reports a CR3 update followed by a triple fault.

My first thought would be that the EFI pgd is broken somehow.

Is 0e39b000 in your kernel binary, or was it dynamically allocated?

What was CR2 when things went bad?  Could you just share a full register
dump?

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Guenter Roeck]

On Wed, Aug 29, 2018 at 01:28:16PM -0700, Dave Hansen wrote:
> On 08/29/2018 01:16 PM, Guenter Roeck wrote:
> > 
> > I see boot failures on mainline when trying to boot x86 images with an efi
> > bios on Intel CPUs in qemu. Behavior is quite unusual: qemu dies silently
> > after the kernel displays "Run /sbin/init as init process". With debugging
> > enabled, qemu reports a CR3 update followed by a triple fault.
> 
> My first thought would be that the EFI pgd is broken somehow.
> 
> Is 0e39b000 in your kernel binary, or was it dynamically allocated?
> 
No idea. In the log below (taken after I recompiled with the latest upstream
kernel) the CR3 value is completely different. Where is the value expected
to come from ?

> What was CR2 when things went bad?  Could you just share a full register
> dump?

Here you are. I have a complete log file, but its size is about 1.7GB
(21MB compressed). Let me know if you need it, and I'll publish it
somewhere.

Thanks,
Guenter

---
0xce1f1cd9:  66 90                    nop      
0xce1f1cdb:  8b 44 24 38              movl     0x38(%esp), %eax
0xce1f1cdf:  8a 64 24 40              movb     0x40(%esp), %ah
0xce1f1ce3:  8a 44 24 34              movb     0x34(%esp), %al
0xce1f1ce7:  25 03 04 02 00           andl     $0x20403, %eax
0xce1f1cec:  3d 03 04 00 00           cmpl     $0x403, %eax
0xce1f1cf1:  75 27                    jne      0xce1f1d1a

EAX=f60b8000 EBX=ff8020bc ECX=00000000 EDX=00000000
ESI=f60c1ff8 EDI=ff802100 EBP=00000000 ESP=f60c1fb4
EIP=ce1f1cd7 EFL=00200006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0068 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
FS =00d8 2819b000 ffffffff 008f9300 DPL=0 DS16 [-WA]
GS =00e0 f67f29c0 00000018 00409100 DPL=0 DS   [--A]
LDT=0000 00000000 00000000 00008200 DPL=0 LDT
TR =0080 ff803000 0000206b 00008900 DPL=0 TSS32-avl
GDT=     f67e2000 000000ff
IDT=     ff800000 000007ff
CR0=80050033 CR2=b7f30854 CR3=35402000 CR4=000006d0
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
CCS=00000022 CCD=00000011 CCO=SARL    
EFER=0000000000000000
----------------
IN: 
0xce1f1d1a:  66 90                    nop      
0xce1f1d1c:  0f 20 d8                 movl     %cr3, %eax
0xce1f1d1f:  0d 00 10 00 00           orl      $0x1000, %eax
0xce1f1d24:  0f 22 d8                 movl     %eax, %cr3

EAX=00000003 EBX=ff8020bc ECX=00000000 EDX=00000000
ESI=f60c1ff8 EDI=ff802100 EBP=00000000 ESP=ff8020bc
EIP=ce1f1d1a EFL=00200087 [--S--PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0068 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
FS =00d8 2819b000 ffffffff 008f9300 DPL=0 DS16 [-WA]
GS =00e0 f67f29c0 00000018 00409100 DPL=0 DS   [--A]
LDT=0000 00000000 00000000 00008200 DPL=0 LDT
TR =0080 ff803000 0000206b 00008900 DPL=0 TSS32-avl
GDT=     f67e2000 000000ff
IDT=     ff800000 000007ff
CR0=80050033 CR2=b7f30854 CR3=35402000 CR4=000006d0
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
CCS=00000403 CCD=fffffc00 CCO=SUBL    
EFER=0000000000000000
CR3 update: CR3=35403000
----------------
IN: 
0xce1f1d27:  5b                       popl     %ebx
0xce1f1d28:  59                       popl     %ecx
0xce1f1d29:  5a                       popl     %edx
0xce1f1d2a:  5e                       popl     %esi
0xce1f1d2b:  5f                       popl     %edi
0xce1f1d2c:  5d                       popl     %ebp
0xce1f1d2d:  58                       popl     %eax
0xce1f1d2e:  1f                       popl     %ds

EAX=35403000 EBX=ff8020bc ECX=00000000 EDX=00000000
ESI=f60c1ff8 EDI=ff802100 EBP=00000000 ESP=ff8020bc
EIP=ce1f1d27 EFL=00200006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0068 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
FS =00d8 2819b000 ffffffff 008f9300 DPL=0 DS16 [-WA]
GS =00e0 f67f29c0 00000018 00409100 DPL=0 DS   [--A]
LDT=0000 00000000 00000000 00008200 DPL=0 LDT
TR =0080 ff803000 0000206b 00008900 DPL=0 TSS32-avl
GDT=     f67e2000 000000ff
IDT=     ff800000 000007ff
CR0=80050033 CR2=b7f30854 CR3=35403000 CR4=000006d0
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
CCS=00000403 CCD=35403000 CCO=LOGICL  
EFER=0000000000000000
EAX=35403000 EBX=ff8020bc ECX=00000000 EDX=00000000
ESI=f60c1ff8 EDI=ff802100 EBP=00000000 ESP=ff8020bc
EIP=ce1f1d27 EFL=00200006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
CS =0060 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0068 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =007b 00000000 ffffffff 00cff300 DPL=3 DS   [-WA]
FS =00d8 2819b000 ffffffff 008f9300 DPL=0 DS16 [-WA]
GS =00e0 f67f29c0 00000018 00409100 DPL=0 DS   [--A]
LDT=0000 00000000 00000000 00008200 DPL=0 LDT
TR =0080 ff803000 0000206b 00008900 DPL=0 TSS32-avl
GDT=     f67e2000 000000ff
IDT=     ff800000 000007ff
CR0=80050033 CR2=b7f30854 CR3=35403000 CR4=000006d0
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
CCS=00000004 CCD=35403000 CCO=EFLAGS  
EFER=0000000000000000
Triple fault

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Joerg Roedel]

Hey Guenter,

On Wed, Aug 29, 2018 at 01:16:59PM -0700, Guenter Roeck wrote:
> I see boot failures on mainline when trying to boot x86 images with an efi
> bios on Intel CPUs in qemu. Behavior is quite unusual: qemu dies silently
> after the kernel displays "Run /sbin/init as init process". With debugging
> enabled, qemu reports a CR3 update followed by a triple fault.
> Here is the end of the log file:
> 
> ----------------
> IN:
> 0xc75f1d1a:  66 90                    nop
> 0xc75f1d1c:  0f 20 d8                 movl     %cr3, %eax
> 0xc75f1d1f:  0d 00 10 00 00           orl      $0x1000, %eax
> 0xc75f1d24:  0f 22 d8                 movl     %eax, %cr3
> 
> CR3 update: CR3=0e39b000

Thanks for the bug report, from a quick glance it looks like the kernel
is switching to an empty page-table. But I need to debug that further.

> qemu-system-i386 -kernel arch/x86/boot/bzImage -M q35 -cpu core2duo \
> 	-no-reboot -m 256 \
> 	-bios OVMF-pure-efi-32.fd \
> 	-usb -device usb-storage,drive=d0 \
> 	-drive file=rootfs.ext2,if=none,id=d0,format=raw \
> 	--append 'root=/dev/sda rw rootwait mem=256M console=ttyS0 console=tty noreboot' \
> 	-nographic

Can you please provide the exact OVMF-pure-efi-32.fd image and the
.config of the kernel?

Thanks a lot,

       Joerg

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Guenter Roeck]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

On 08/30/2018 12:19 AM, Joerg Roedel wrote:
> Hey Guenter,
> 
> On Wed, Aug 29, 2018 at 01:16:59PM -0700, Guenter Roeck wrote:
>> I see boot failures on mainline when trying to boot x86 images with an efi
>> bios on Intel CPUs in qemu. Behavior is quite unusual: qemu dies silently
>> after the kernel displays "Run /sbin/init as init process". With debugging
>> enabled, qemu reports a CR3 update followed by a triple fault.
>> Here is the end of the log file:
>>
>> ----------------
>> IN:
>> 0xc75f1d1a:  66 90                    nop
>> 0xc75f1d1c:  0f 20 d8                 movl     %cr3, %eax
>> 0xc75f1d1f:  0d 00 10 00 00           orl      $0x1000, %eax
>> 0xc75f1d24:  0f 22 d8                 movl     %eax, %cr3
>>
>> CR3 update: CR3=0e39b000
> 
> Thanks for the bug report, from a quick glance it looks like the kernel
> is switching to an empty page-table. But I need to debug that further.
> 
>> qemu-system-i386 -kernel arch/x86/boot/bzImage -M q35 -cpu core2duo \
>> 	-no-reboot -m 256 \
>> 	-bios OVMF-pure-efi-32.fd \
>> 	-usb -device usb-storage,drive=d0 \
>> 	-drive file=rootfs.ext2,if=none,id=d0,format=raw \
>> 	--append 'root=/dev/sda rw rootwait mem=256M console=ttyS0 console=tty noreboot' \
>> 	-nographic
> 
> Can you please provide the exact OVMF-pure-efi-32.fd image and the
> .config of the kernel?
> 

OVMF image:

https://github.com/groeck/linux-build-test/blob/master/rootfs/firmware/OVMF-pure-efi-32.fd

root file system:

https://github.com/groeck/linux-build-test/blob/master/rootfs/x86/rootfs.ext2

Configuration is attached.

Guenter

[-- Attachment #2: defconfig --]
[-- Type: text/plain, Size: 6871 bytes --]

# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_AUDIT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_LOG_BUF_SHIFT=18
CONFIG_CGROUPS=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_BLK_DEV_INITRD=y
# CONFIG_COMPAT_BRK is not set
CONFIG_PROFILING=y
CONFIG_SMP=y
CONFIG_X86_GENERIC=y
CONFIG_HPET_TIMER=y
CONFIG_SCHED_SMT=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_REBOOTFIXUPS=y
CONFIG_MICROCODE_AMD=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_HIGHPTE=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_EFI=y
CONFIG_HZ_1000=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_HIBERNATION=y
CONFIG_PM_DEBUG=y
CONFIG_PM_TRACE_RTC=y
CONFIG_ACPI_DOCK=y
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCI_MSI=y
CONFIG_HOTPLUG_PCI=y
CONFIG_PCCARD=y
CONFIG_YENTA=y
CONFIG_EFI_VARS=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_SGI_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_BINFMT_MISC=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM_USER=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_DIAG is not set
CONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not set
CONFIG_TCP_MD5SIG=y
CONFIG_INET6_AH=y
CONFIG_INET6_ESP=y
CONFIG_NETLABEL=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_ADVANCED is not set
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_CONNTRACK_IRC=y
CONFIG_NF_CONNTRACK_SIP=y
CONFIG_NF_CT_NETLINK=y
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_SECMARK=y
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_POLICY=y
CONFIG_NETFILTER_XT_MATCH_STATE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_MANGLE=y
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
CONFIG_IP6_NF_FILTER=y
CONFIG_IP6_NF_TARGET_REJECT=y
CONFIG_IP6_NF_MANGLE=y
CONFIG_NET_SCHED=y
CONFIG_NET_EMATCH=y
CONFIG_NET_CLS_ACT=y
CONFIG_HAMRADIO=y
CONFIG_CFG80211=y
CONFIG_MAC80211=y
CONFIG_MAC80211_LEDS=y
CONFIG_RFKILL=y
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_DEBUG_DEVRES=y
CONFIG_CONNECTOR=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_BLK_SCSI=y
CONFIG_BLK_DEV_NVME=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_MEGARAID_SAS=y
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_SCSI_DC395x=y
CONFIG_SCSI_AM53C974=y
CONFIG_SCSI_VIRTIO=y
CONFIG_ATA=y
CONFIG_SATA_AHCI=y
CONFIG_ATA_PIIX=y
CONFIG_PATA_AMD=y
CONFIG_PATA_OLDPIIX=y
CONFIG_PATA_SCH=y
CONFIG_PATA_MPIIX=y
CONFIG_ATA_GENERIC=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_DM_MIRROR=y
CONFIG_DM_ZERO=y
CONFIG_FUSION=y
CONFIG_FUSION_SAS=y
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_NETCONSOLE=y
CONFIG_BNX2=y
CONFIG_TIGON3=y
CONFIG_NET_TULIP=y
CONFIG_E100=y
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_SKY2=y
CONFIG_NE2K_PCI=y
CONFIG_FORCEDETH=y
CONFIG_8139TOO=y
# CONFIG_8139TOO_PIO is not set
CONFIG_R8169=y
CONFIG_FDDI=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_JOYSTICK=y
CONFIG_INPUT_TABLET=y
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_INPUT_MISC=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_RSA=y
CONFIG_HW_RANDOM=y
CONFIG_NVRAM=y
CONFIG_HPET=y
# CONFIG_HPET_MMAP is not set
CONFIG_I2C_I801=y
CONFIG_WATCHDOG=y
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_DRM=y
CONFIG_DRM_I915=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
CONFIG_FB_EFI=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_SOUND=y
CONFIG_SND=y
CONFIG_SND_HRTIMER=y
CONFIG_SND_SEQUENCER=y
CONFIG_SND_SEQ_DUMMY=y
CONFIG_SND_HDA_INTEL=y
CONFIG_SND_HDA_HWDEP=y
CONFIG_HIDRAW=y
CONFIG_HID_GYRATION=y
CONFIG_LOGITECH_FF=y
CONFIG_HID_NTRIG=y
CONFIG_HID_PANTHERLORD=y
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
CONFIG_HID_SUNPLUS=y
CONFIG_HID_TOPSEED=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y
CONFIG_USB=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
CONFIG_USB_MON=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_PRINTER=y
CONFIG_USB_STORAGE=y
CONFIG_USB_UAS=y
CONFIG_MMC=y
CONFIG_MMC_SDHCI=y
CONFIG_MMC_SDHCI_PCI=y
CONFIG_EDAC=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not set
CONFIG_DMADEVICES=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=y
CONFIG_EEEPC_LAPTOP=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
CONFIG_QFMT_V2=y
CONFIG_AUTOFS4_FS=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_PROC_KCORE=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_ROOT_NFS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_UTF8=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_CRYPTO_AES_586=y
CONFIG_PRINTK_TIME=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_SCHED_DEBUG is not set
CONFIG_SCHEDSTATS=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_OPTIMIZE_INLINING=y

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Joerg Roedel]

On Thu, Aug 30, 2018 at 03:11:26AM -0700, Guenter Roeck wrote:
> OVMF image:
> 
> https://github.com/groeck/linux-build-test/blob/master/rootfs/firmware/OVMF-pure-efi-32.fd

Thanks, with this image I can reproduce the issue. Investigating now...


Regards,

	Joerg

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Guenter Roeck]

On 08/30/2018 04:30 AM, Joerg Roedel wrote:
> On Thu, Aug 30, 2018 at 03:11:26AM -0700, Guenter Roeck wrote:
>> OVMF image:
>>
>> https://github.com/groeck/linux-build-test/blob/master/rootfs/firmware/OVMF-pure-efi-32.fd
> 
> Thanks, with this image I can reproduce the issue. Investigating now...
> 

Excellent.

In case it matters: I can use the same 32-bit efi image to boot a 64-bit kernel
without problems.

Guenter

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Joerg Roedel]

On Thu, Aug 30, 2018 at 05:20:22AM -0700, Guenter Roeck wrote:
> Excellent.
> 
> In case it matters: I can use the same 32-bit efi image to boot a 64-bit kernel
> without problems.

Okay, I tracked it down. The 32-bit EFI code switches the CPU to the
read-write version of the GDT which is not mapped in the user
page-table. This obviously faults when the kernel entry/exit code
restores the segment registers.

Without a mapped GDT the #PF and #DF handlers also can't be started, so
the machine triple-faults. Below diff fixes it for me, I'll send a
proper patch tomorrow.

diff --git a/arch/x86/platform/efi/efi_32.c b/arch/x86/platform/efi/efi_32.c
index 324b93328b37..892edb4664fb 100644
--- a/arch/x86/platform/efi/efi_32.c
+++ b/arch/x86/platform/efi/efi_32.c
@@ -87,7 +87,7 @@ void __init efi_call_phys_epilog(pgd_t *save_pgd)
 {
 	struct desc_ptr gdt_descr;
 
-	gdt_descr.address = (unsigned long)get_cpu_gdt_rw(0);
+	gdt_descr.address = (unsigned long)get_cpu_gdt_ro(0);
 	gdt_descr.size = GDT_SIZE - 1;
 	load_gdt(&gdt_descr);
 

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Linus Torvalds]

On Thu, Aug 30, 2018 at 11:08 AM Joerg Roedel <jroedel@suse.de> wrote:
>
> Without a mapped GDT the #PF and #DF handlers also can't be started, so
> the machine triple-faults. Below diff fixes it for me, I'll send a
> proper patch tomorrow.

Hmm. Is there any reason why this code doesn't just use

        load_fixmap_gdt(0);

and shouldn't it do it after loading the new %cr3?

               Linus

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Joerg Roedel]

On Thu, Aug 30, 2018 at 11:21:49AM -0700, Linus Torvalds wrote:
> On Thu, Aug 30, 2018 at 11:08 AM Joerg Roedel <jroedel@suse.de> wrote:
> >
> > Without a mapped GDT the #PF and #DF handlers also can't be started, so
> > the machine triple-faults. Below diff fixes it for me, I'll send a
> > proper patch tomorrow.
> 
> Hmm. Is there any reason why this code doesn't just use
> 
>         load_fixmap_gdt(0);

No idea, probably the function didn't exist when the code was written?
I can change that when writing the patch.

> and shouldn't it do it after loading the new %cr3?

That seems more robust, yes. No sure if the old %cr3
(initial_page_table) has the fixmap gdt mapped at all.


	Joerg

Re: efi boot failures due to PTI with 32 bit builds and Intel CPUs

[Guenter Roeck]

On Thu, Aug 30, 2018 at 08:46:39PM +0200, Joerg Roedel wrote:
> On Thu, Aug 30, 2018 at 11:21:49AM -0700, Linus Torvalds wrote:
> > On Thu, Aug 30, 2018 at 11:08 AM Joerg Roedel <jroedel@suse.de> wrote:
> > >
> > > Without a mapped GDT the #PF and #DF handlers also can't be started, so
> > > the machine triple-faults. Below diff fixes it for me, I'll send a
> > > proper patch tomorrow.
> > 
> > Hmm. Is there any reason why this code doesn't just use
> > 
> >         load_fixmap_gdt(0);
> 
> No idea, probably the function didn't exist when the code was written?
> I can change that when writing the patch.
> 
> > and shouldn't it do it after loading the new %cr3?
> 
> That seems more robust, yes. No sure if the old %cr3
> (initial_page_table) has the fixmap gdt mapped at all.

All three variants (hardcoded, call load_fixmap_gdt(0) first, call
load_fixmap_gdt(0) after load_cr3()) work for me. Feel free to add

Tested-by: Guenter Roeck <linux@roeck-us.net>

when you submit the patch.

Thanks a lot for tracking this down!

Guenter