KPTI backport to 3.16

KPTI backport to 3.16

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

I have a backport of KPTI/KAISER to 3.16, based on Hugh Dickins's work
for 3.18, some upstream changes between 3.16 and 3.18, and other
patches that went into 4.4.75.

I sent this out for review on the stable list after quite minimal
testing, but have done more since then.  On bare metal (Sandy Bridge,
with pcid but not invpcid) it crashes at boot.  In fact it
reboots without any panic message, suggesting a triple fault, as soon
as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.

Using the 'nopcid' kernel parameter gets it to boot but it's somewhat
unstable even after that - once I start another kernel build I see
programs segfaulting.  So I'm guessing I've screwed up some of the TLB
stuff.

I'm going to continue investigating this myself before making a
release, but would really appreciate any time people can spare to
review this patch series.

(I haven't found any such regression in 3.2.98.)

Ben.

-- 
Ben Hutchings
friends: People who know you well, but like you anyway.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Hugh Dickins]

On Sun, Jan 7, 2018 at 3:35 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> I have a backport of KPTI/KAISER to 3.16, based on Hugh Dickins's work
> for 3.18, some upstream changes between 3.16 and 3.18, and other
> patches that went into 4.4.75.
>
> I sent this out for review on the stable list after quite minimal
> testing, but have done more since then.  On bare metal (Sandy Bridge,
> with pcid but not invpcid) it crashes at boot.  In fact it
> reboots without any panic message, suggesting a triple fault, as soon
> as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.
>
> Using the 'nopcid' kernel parameter gets it to boot but it's somewhat
> unstable even after that - once I start another kernel build I see
> programs segfaulting.  So I'm guessing I've screwed up some of the TLB
> stuff.
>
> I'm going to continue investigating this myself before making a
> release, but would really appreciate any time people can spare to
> review this patch series.
>
> (I haven't found any such regression in 3.2.98.)
>
> Ben.
>
> --
> Ben Hutchings
> friends: People who know you well, but like you anyway.
>
> --
> You received this message because you are subscribed to the Google Groups "kaiser-discuss-external" group.
> To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/kaiser-discuss-external/1515368108.13978.37.camel%40decadent.org.uk.

Hi Ben, I'm not on that "stable list", and don't see where it is
archived, so this is the first I've heard of your 3.16 port (which
does sound from your description like you've approached it in exactly
the right way).

Could you please attach a tarfile of git-format-patches since 3.16.52
that we could look through - that will be easier to manage than
downloading 40 mails or whatever - thanks.

Hugh

Re: [kaiser-discuss] KPTI backport to 3.16

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Sun, 2018-01-07 at 16:05 -0800, Hugh Dickins wrote:
[...]
> Hi Ben, I'm not on that "stable list", and don't see where it is
> archived, so this is the first I've heard of your 3.16 port (which
> does sound from your description like you've approached it in exactly
> the right way).
> 
> Could you please attach a tarfile of git-format-patches since 3.16.52
> that we could look through - that will be easier to manage than
> downloading 40 mails or whatever - thanks.

Thanks, they're on the linux-3.16.y-rc branch at:
https://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-stable-rc.git

Ben.

-- 
Ben Hutchings
This sentence contradicts itself - no actually it doesn't.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Dave Hansen]

On 01/07/2018 03:35 PM, Ben Hutchings wrote:
> I sent this out for review on the stable list after quite minimal
> testing, but have done more since then.  On bare metal (Sandy Bridge,
> with pcid but not invpcid) it crashes at boot.  In fact it
> reboots without any panic message, suggesting a triple fault, as soon
> as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.

My first guess would be something around this stuff:

> commit c7ad5ad297e644601747d6dbee978bf85e14f7bc
> Author: Andy Lutomirski <luto@kernel.org>
> Date:   Sun Sep 10 17:48:27 2017 -0700
> 
>     x86/mm/64: Initialize CR4.PCIDE early

But, if you also want to toss a set of binaries up somewhere that I can
test I can give them a quick run in the simulator or with a hardware
debugger attached.  It's been very useful in getting these things
debugged, especially when normal debugging techniques fail.

Re: [kaiser-discuss] KPTI backport to 3.16

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]

On Sun, 2018-01-07 at 19:03 -0800, Dave Hansen wrote:
> On 01/07/2018 03:35 PM, Ben Hutchings wrote:
> > I sent this out for review on the stable list after quite minimal
> > testing, but have done more since then.  On bare metal (Sandy Bridge,
> > with pcid but not invpcid) it crashes at boot.  In fact it
> > reboots without any panic message, suggesting a triple fault, as soon
> > as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.
> 
> My first guess would be something around this stuff:
> 
> > commit c7ad5ad297e644601747d6dbee978bf85e14f7bc
> > Author: Andy Lutomirski <luto@kernel.org>
> > Date:   Sun Sep 10 17:48:27 2017 -0700
> > 
> >     x86/mm/64: Initialize CR4.PCIDE early
> 
> But, if you also want to toss a set of binaries up somewhere that I can
> test I can give them a quick run in the simulator or with a hardware
> debugger attached.  It's been very useful in getting these things
> debugged, especially when normal debugging techniques fail.

This is with the full patch set applied (and a fix for NMI handling
that wasn't in 3.16.53-rc1):
https://www.decadent.org.uk/ben/tmp/linux-image-3.16.52_3.16.52-50_amd64.deb

Ben.

-- 
Ben Hutchings
This sentence contradicts itself - no actually it doesn't.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Juerg Haefliger]

[-- Attachment #1.1: Type: text/plain, Size: 1889 bytes --]

Ben,

On 01/08/2018 12:35 AM, Ben Hutchings wrote:
> I have a backport of KPTI/KAISER to 3.16, based on Hugh Dickins's work
> for 3.18, some upstream changes between 3.16 and 3.18, and other
> patches that went into 4.4.75.
> 
> I sent this out for review on the stable list after quite minimal
> testing, but have done more since then.  On bare metal (Sandy Bridge,
> with pcid but not invpcid) it crashes at boot.  In fact it
> reboots without any panic message, suggesting a triple fault, as soon
> as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.

I've seen this as well with my 3.13 tree. As soon as PCID is set on the
first (non-boot) CPU, the kernel reboots. Note that it seems to work
fine with maxcpus=1.

I've checked the other versions, your 3.2 doesn't have this issue and
Hugh's 3.18 doesn't have it either. After some bisecting, I found that
the problem was introduced in 3.15 by:
cda846f101fb ('x86, realmode: read cr4 and EFER from kernel for 64-bit
trampoline')
and then later fixed again in 4.0 by:
375074cc736a ('x86: Clean up cr4 manipulation')

I've backported 375074cc736a to 3.13 which seems to fix this particular
issue but I'm still seeing userspace segfaults.

FWIW:
My tree: https://code.launchpad.net/~juergh/+git/linux-pti
pti/3.13 is my 3.13 tree, pti/3.2.97-bwh and pti/3.16.52-bwh are yours
and pti/3.18.72 is Hugh's.

...Juerg

> Using the 'nopcid' kernel parameter gets it to boot but it's somewhat
> unstable even after that - once I start another kernel build I see
> programs segfaulting.  So I'm guessing I've screwed up some of the TLB
> stuff.
> 
> I'm going to continue investigating this myself before making a
> release, but would really appreciate any time people can spare to
> review this patch series.
> 
> (I haven't found any such regression in 3.2.98.)
> 
> Ben.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 845 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Mon, 2018-01-08 at 03:25 +0000, Ben Hutchings wrote:
> This is with the full patch set applied (and a fix for NMI handling
> that wasn't in 3.16.53-rc1):
> https://www.decadent.org.uk/ben/tmp/linux-image-3.16.52_3.16.52-50_amd64.deb

Hi Ben (and others)

I tried this on a ThinkPad X230 (Ivy Bridge CPU, i7 3520M, so pcid but not
invpcid) on full UEFI mode and it doesn't boot at all. grub-efi loads the
kernel and initrd, but I don't have any message after “Loading initial
ramdisk” (even with debug earlyprintk=vga).

I've also tried with nopcid/pcid=no but it doesn't change anything.

I've tried maxcpus=1 and it does work so maybe backporting 375074cc736a could
help (I can take a look today).

Regards,
-- 
Yves-Alexis

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

On Mon, 2018-01-08 at 10:05 +0100, Yves-Alexis Perez wrote:
> On Mon, 2018-01-08 at 03:25 +0000, Ben Hutchings wrote:
> > This is with the full patch set applied (and a fix for NMI handling
> > that wasn't in 3.16.53-rc1):
> > https://www.decadent.org.uk/ben/tmp/linux-image-3.16.52_3.16.52-50_amd64.deb
> 
> Hi Ben (and others)
> 
> I tried this on a ThinkPad X230 (Ivy Bridge CPU, i7 3520M, so pcid but not
> invpcid) on full UEFI mode and it doesn't boot at all. grub-efi loads the
> kernel and initrd, but I don't have any message after “Loading initial
> ramdisk” (even with debug earlyprintk=vga).

If you boot with EFI you need to use earlyprintk=efi.

Ben.

> I've also tried with nopcid/pcid=no but it doesn't change anything.
> 
> I've tried maxcpus=1 and it does work so maybe backporting 375074cc736a could
> help (I can take a look today).
> 
> Regards,
-- 
Ben Hutchings
This sentence contradicts itself - no actually it doesn't.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

On Mon, 2018-01-08 at 15:19 +0000, Ben Hutchings wrote:
> > I tried this on a ThinkPad X230 (Ivy Bridge CPU, i7 3520M, so pcid but not
> > invpcid) on full UEFI mode and it doesn't boot at all. grub-efi loads the
> > kernel and initrd, but I don't have any message after “Loading initial
> > ramdisk” (even with debug earlyprintk=vga).
> 
> If you boot with EFI you need to use earlyprintk=efi.

Thanks! With that, I can see the last few log lines before it hangs:

Kernel/User page tables isolation: enabled
[…]
bootconsole [earlyefi0] disabled

and then nothing.

Also I must have done something wrong earlier today, because when booting with
 'nopcid' it does work (which is consistent with other people tests, afaict).

Regards,
-- 
Yves-Alexis

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Dave Hansen]

On 01/08/2018 07:28 AM, Yves-Alexis Perez wrote:
> On Mon, 2018-01-08 at 15:19 +0000, Ben Hutchings wrote:
>>> I tried this on a ThinkPad X230 (Ivy Bridge CPU, i7 3520M, so pcid but not
>>> invpcid) on full UEFI mode and it doesn't boot at all. grub-efi loads the
>>> kernel and initrd, but I don't have any message after “Loading initial
>>> ramdisk” (even with debug earlyprintk=vga).
>> If you boot with EFI you need to use earlyprintk=efi.
> Thanks! With that, I can see the last few log lines before it hangs:
> 
> Kernel/User page tables isolation: enabled
> […]
> bootconsole [earlyefi0] disabled

Does "earlyprintk=efi,keep" help get more out to the console?

> Also I must have done something wrong earlier today, because when booting with
>  'nopcid' it does work (which is consistent with other people tests, afaict).

If I had to guess, I'd say we're probably putting a PCID value into the
low bits of CR3 before we've _enabled_ PCID support in the CPU.

Re: [kaiser-discuss] KPTI backport to 3.16

[Dave Hansen]

On 01/08/2018 01:05 AM, Yves-Alexis Perez wrote:
> On Mon, 2018-01-08 at 03:25 +0000, Ben Hutchings wrote:
>> This is with the full patch set applied (and a fix for NMI handling
>> that wasn't in 3.16.53-rc1):
>> https://www.decadent.org.uk/ben/tmp/linux-image-3.16.52_3.16.52-50_amd64.deb

I booted this.  It crashes in *secondary* CPU startup when it sets
CR4.PCIDE while still in 32-bit protected mode.  That's illegal.

Plain 3.16 doesn't do this:

https://git.kernel.org/pub/scm/linux/kernel/git/daveh/x86-kaiser.git/tree/arch/x86/kernel/head_64.S?h=v3.16

> 	/* Enable PAE mode and PGE */
> 	movl	$(X86_CR4_PAE | X86_CR4_PGE), %ecx
> 	movq	%rcx, %cr4

So I suspect the "Enable PAE and PGE" area is wrong.

Re: [kaiser-discuss] KPTI backport to 3.16

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1960 bytes --]

On Mon, 2018-01-08 at 08:10 +0100, Juerg Haefliger wrote:
> Ben,
> 
> On 01/08/2018 12:35 AM, Ben Hutchings wrote:
> > I have a backport of KPTI/KAISER to 3.16, based on Hugh Dickins's work
> > for 3.18, some upstream changes between 3.16 and 3.18, and other
> > patches that went into 4.4.75.
> > 
> > I sent this out for review on the stable list after quite minimal
> > testing, but have done more since then.  On bare metal (Sandy Bridge,
> > with pcid but not invpcid) it crashes at boot.  In fact it
> > reboots without any panic message, suggesting a triple fault, as soon
> > as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.
> 
> I've seen this as well with my 3.13 tree. As soon as PCID is set on the
> first (non-boot) CPU, the kernel reboots. Note that it seems to work
> fine with maxcpus=1.

I see, this makes sense.

> I've checked the other versions, your 3.2 doesn't have this issue and
> Hugh's 3.18 doesn't have it either. After some bisecting, I found that
> the problem was introduced in 3.15 by:
> cda846f101fb ('x86, realmode: read cr4 and EFER from kernel for 64-bit
> trampoline')
> and then later fixed again in 4.0 by:
> 375074cc736a ('x86: Clean up cr4 manipulation')

Thanks!  This plus obvious changes to the patches using
{clear,set}_in_cr4() gets me a kernel that boots on the SNB system.

> I've backported 375074cc736a to 3.13 which seems to fix this particular
> issue but I'm still seeing userspace segfaults.

With the above changes I'm not seeing any user-space segfaults either
(so far).

> FWIW:
> My tree: https://code.launchpad.net/~juergh/+git/linux-pti
> pti/3.13 is my 3.13 tree, pti/3.2.97-bwh and pti/3.16.52-bwh are yours
> and pti/3.18.72 is Hugh's.

I've just pushed 3.16.53-rc2 to
https://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-stable-rc.git

Ben.

-- 
Ben Hutchings
This sentence contradicts itself - no actually it doesn't.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Juerg Haefliger]

[-- Attachment #1.1: Type: text/plain, Size: 2225 bytes --]



On 01/08/2018 09:39 PM, Ben Hutchings wrote:
> On Mon, 2018-01-08 at 08:10 +0100, Juerg Haefliger wrote:
>> Ben,
>>
>> On 01/08/2018 12:35 AM, Ben Hutchings wrote:
>>> I have a backport of KPTI/KAISER to 3.16, based on Hugh Dickins's work
>>> for 3.18, some upstream changes between 3.16 and 3.18, and other
>>> patches that went into 4.4.75.
>>>
>>> I sent this out for review on the stable list after quite minimal
>>> testing, but have done more since then.  On bare metal (Sandy Bridge,
>>> with pcid but not invpcid) it crashes at boot.  In fact it
>>> reboots without any panic message, suggesting a triple fault, as soon
>>> as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.
>>
>> I've seen this as well with my 3.13 tree. As soon as PCID is set on the
>> first (non-boot) CPU, the kernel reboots. Note that it seems to work
>> fine with maxcpus=1.
> 
> I see, this makes sense.
> 
>> I've checked the other versions, your 3.2 doesn't have this issue and
>> Hugh's 3.18 doesn't have it either. After some bisecting, I found that
>> the problem was introduced in 3.15 by:
>> cda846f101fb ('x86, realmode: read cr4 and EFER from kernel for 64-bit
>> trampoline')
>> and then later fixed again in 4.0 by:
>> 375074cc736a ('x86: Clean up cr4 manipulation')
> 
> Thanks!  This plus obvious changes to the patches using
> {clear,set}_in_cr4() gets me a kernel that boots on the SNB system.

Do you have debs I could run some tests against?


>> I've backported 375074cc736a to 3.13 which seems to fix this particular
>> issue but I'm still seeing userspace segfaults.
> 
> With the above changes I'm not seeing any user-space segfaults either
> (so far).

Hmm... I was hoping you did ;-) Bummer. I've added the TLB flushing
patches to 3.13 that you backported from 3.18 (plus some dependencies)
but I'm still seeing segfaults.

...Juerg


>> FWIW:
>> My tree: https://code.launchpad.net/~juergh/+git/linux-pti
>> pti/3.13 is my 3.13 tree, pti/3.2.97-bwh and pti/3.16.52-bwh are yours
>> and pti/3.18.72 is Hugh's.
> 
> I've just pushed 3.16.53-rc2 to
> https://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-stable-rc.git
> 
> Ben.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 845 bytes --]

Re: [kaiser-discuss] KPTI backport to 3.16

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1859 bytes --]

On Mon, 2018-01-08 at 22:13 +0100, Juerg Haefliger wrote:
> 
> On 01/08/2018 09:39 PM, Ben Hutchings wrote:
> > On Mon, 2018-01-08 at 08:10 +0100, Juerg Haefliger wrote:
> > > Ben,
> > > 
> > > On 01/08/2018 12:35 AM, Ben Hutchings wrote:
> > > > I have a backport of KPTI/KAISER to 3.16, based on Hugh Dickins's work
> > > > for 3.18, some upstream changes between 3.16 and 3.18, and other
> > > > patches that went into 4.4.75.
> > > > 
> > > > I sent this out for review on the stable list after quite minimal
> > > > testing, but have done more since then.  On bare metal (Sandy Bridge,
> > > > with pcid but not invpcid) it crashes at boot.  In fact it
> > > > reboots without any panic message, suggesting a triple fault, as soon
> > > > as I apply the patch that turns on CR4.PCIDE, i.e. without KPTI itself.
> > > 
> > > I've seen this as well with my 3.13 tree. As soon as PCID is set on the
> > > first (non-boot) CPU, the kernel reboots. Note that it seems to work
> > > fine with maxcpus=1.
> > 
> > I see, this makes sense.
> > 
> > > I've checked the other versions, your 3.2 doesn't have this issue and
> > > Hugh's 3.18 doesn't have it either. After some bisecting, I found that
> > > the problem was introduced in 3.15 by:
> > > cda846f101fb ('x86, realmode: read cr4 and EFER from kernel for 64-bit
> > > trampoline')
> > > and then later fixed again in 4.0 by:
> > > 375074cc736a ('x86: Clean up cr4 manipulation')
> > 
> > Thanks!  This plus obvious changes to the patches using
> > {clear,set}_in_cr4() gets me a kernel that boots on the SNB system.
> 
> Do you have debs I could run some tests against?
[...]

Here you go:
https://www.decadent.org.uk/ben/tmp/linux-image-3.16.53-rc2_3.16.53-rc2-53_amd64.deb

Ben.

-- 
Ben Hutchings
This sentence contradicts itself - no actually it doesn't.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]