Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Brian Woods]

I'm currently seeing an issue where when booting from a recent Linux
kernel without nospec_store_bypass_disable.  There's a NULL pointer
having to do with a lock.  I put some printks in and it seems that in
arch/x86/kernel/process.c
that speculative_store_bypass_ht_init isn't getting called which
initializes the spin lock.  Here's the serial output:

[    7.748191] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[    7.748202] PGD 0 P4D 0
[    7.748208] Oops: 0002 [#1] SMP NOPTI
[    7.748212] Modules linked in: ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto raid10 raid456 async_raid6_recov async_memcpy async_pq async_xore
[    7.748261] CPU: 4 PID: 321 Comm: (journald) Not tainted 4.17.2+ #1
[    7.748266] Hardware name: AMD Corporation Diesel Debug/Diesel Debug, BIOS TDD1007E 04/16/2018
[    7.748277] RIP: e030:_raw_spin_lock+0xc/0x20
[    7.748293] RSP: e02b:ffffc9004709feb8 EFLAGS: 00010046
[    7.748297] RAX: 0000000000000000 RBX: ffff880285715b30 RCX: ffffea0009ce30df
[    7.748302] RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000008
[    7.748308] RBP: 0000000000000400 R08: aaaaaaaaaaaaaaaa R09: 0000000000000007
[    7.748313] R10: 0000000000000040 R11: ffff88027a9bc800 R12: 0206800000000000
[    7.748318] R13: 0000000000000000 R14: ffff880273aa0080 R15: 0000000000000000
[    7.748331] FS:  00007fc72e830940(0000) GS:ffff880285700000(0000) knlGS:0000000000000000
[    7.748336] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[    7.748341] CR2: 0000000000000008 CR3: 0000000278c6c000 CR4: 0000000000040660
[    7.748347] Call Trace:
[    7.748354]  speculative_store_bypass_update+0x72/0x160
[    7.748361]  ssb_prctl_set+0x67/0xb0
[    7.748367]  do_seccomp+0x477/0x6c0
[    7.748385]  do_syscall_64+0x55/0x100
[    7.748390]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[    7.748395] RIP: 0033:0x7fc72ce13229
[    7.748399] RSP: 002b:00007ffda5166968 EFLAGS: 00000246 ORIG_RAX: 000000000000013d
[    7.748405] RAX: ffffffffffffffda RBX: 000055745ea1dfe0 RCX: 00007fc72ce13229
[    7.748410] RDX: 000055745ea1dfe0 RSI: 0000000000000000 RDI: 0000000000000001
[    7.748415] RBP: 000055745ea5b740 R08: 000055745ea1dfe0 R09: 000000004000003e
[    7.748421] R10: 000000000000000d R11: 0000000000000246 R12: 00007ffda51669c0
[    7.748426] R13: 00007ffda51669b8 R14: 00007fc72e560c14 R15: 000000000000002a
[    7.748431] Code: ff 01 00 00 75 05 48 89 d8 5b c3 e8 1f 8f 9f ff 48 89 d8 5b c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f
[    7.748483] RIP: _raw_spin_lock+0xc/0x20 RSP: ffffc9004709feb8
[    7.748487] CR2: 0000000000000008
[    7.748492] ---[ end trace cf886bf535fde244 ]---


With nospec_store_bypass_disable, it boots fine etc.  It seems to works
fine (at least Dom0 can boot).

Linux Kernel -> 4.17.2
Xen -> current HEAD on master

Is this a known or expected problem?

-- 
Brian Woods

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Boris Ostrovsky]

On 06/19/2018 05:30 PM, Brian Woods wrote:
> I'm currently seeing an issue where when booting from a recent Linux
> kernel without nospec_store_bypass_disable.  There's a NULL pointer
> having to do with a lock.  I put some printks in and it seems that in
> arch/x86/kernel/process.c
> that speculative_store_bypass_ht_init isn't getting called which
> initializes the spin lock.  



speculative_store_bypass_ht_init() is not called on PV. For BSP it is 
called from native_smp_prepare_cpus() and for APs it is called from 
start_secondary(), neither of which is in PV code path.

I think the most logical place to put it is in cpu_init().

-boris


Here's the serial output:
> 
> [    7.748191] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> [    7.748202] PGD 0 P4D 0
> [    7.748208] Oops: 0002 [#1] SMP NOPTI
> [    7.748212] Modules linked in: ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto raid10 raid456 async_raid6_recov async_memcpy async_pq async_xore
> [    7.748261] CPU: 4 PID: 321 Comm: (journald) Not tainted 4.17.2+ #1
> [    7.748266] Hardware name: AMD Corporation Diesel Debug/Diesel Debug, BIOS TDD1007E 04/16/2018
> [    7.748277] RIP: e030:_raw_spin_lock+0xc/0x20
> [    7.748293] RSP: e02b:ffffc9004709feb8 EFLAGS: 00010046
> [    7.748297] RAX: 0000000000000000 RBX: ffff880285715b30 RCX: ffffea0009ce30df
> [    7.748302] RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000008
> [    7.748308] RBP: 0000000000000400 R08: aaaaaaaaaaaaaaaa R09: 0000000000000007
> [    7.748313] R10: 0000000000000040 R11: ffff88027a9bc800 R12: 0206800000000000
> [    7.748318] R13: 0000000000000000 R14: ffff880273aa0080 R15: 0000000000000000
> [    7.748331] FS:  00007fc72e830940(0000) GS:ffff880285700000(0000) knlGS:0000000000000000
> [    7.748336] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> [    7.748341] CR2: 0000000000000008 CR3: 0000000278c6c000 CR4: 0000000000040660
> [    7.748347] Call Trace:
> [    7.748354]  speculative_store_bypass_update+0x72/0x160
> [    7.748361]  ssb_prctl_set+0x67/0xb0
> [    7.748367]  do_seccomp+0x477/0x6c0
> [    7.748385]  do_syscall_64+0x55/0x100
> [    7.748390]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [    7.748395] RIP: 0033:0x7fc72ce13229
> [    7.748399] RSP: 002b:00007ffda5166968 EFLAGS: 00000246 ORIG_RAX: 000000000000013d
> [    7.748405] RAX: ffffffffffffffda RBX: 000055745ea1dfe0 RCX: 00007fc72ce13229
> [    7.748410] RDX: 000055745ea1dfe0 RSI: 0000000000000000 RDI: 0000000000000001
> [    7.748415] RBP: 000055745ea5b740 R08: 000055745ea1dfe0 R09: 000000004000003e
> [    7.748421] R10: 000000000000000d R11: 0000000000000246 R12: 00007ffda51669c0
> [    7.748426] R13: 00007ffda51669b8 R14: 00007fc72e560c14 R15: 000000000000002a
> [    7.748431] Code: ff 01 00 00 75 05 48 89 d8 5b c3 e8 1f 8f 9f ff 48 89 d8 5b c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f
> [    7.748483] RIP: _raw_spin_lock+0xc/0x20 RSP: ffffc9004709feb8
> [    7.748487] CR2: 0000000000000008
> [    7.748492] ---[ end trace cf886bf535fde244 ]---
> 
> 
> With nospec_store_bypass_disable, it boots fine etc.  It seems to works
> fine (at least Dom0 can boot).
> 
> Linux Kernel -> 4.17.2
> Xen -> current HEAD on master
> 
> Is this a known or expected problem?
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Juergen Gross]

On 20/06/18 05:31, Boris Ostrovsky wrote:
> 
> 
> On 06/19/2018 05:30 PM, Brian Woods wrote:
>> I'm currently seeing an issue where when booting from a recent Linux
>> kernel without nospec_store_bypass_disable.  There's a NULL pointer
>> having to do with a lock.  I put some printks in and it seems that in
>> arch/x86/kernel/process.c
>> that speculative_store_bypass_ht_init isn't getting called which
>> initializes the spin lock.  
> 
> 
> 
> speculative_store_bypass_ht_init() is not called on PV. For BSP it is
> called from native_smp_prepare_cpus() and for APs it is called from
> start_secondary(), neither of which is in PV code path.
> 
> I think the most logical place to put it is in cpu_init().

I don't think so. speculative_store_bypass_ht_init() requires
topology_sibling_cpumask to be setup which happens only after the call
of cpu_init().

We should add the call to cpu_bringup() in smp_pv.c


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Juergen Gross]

[-- Attachment #1: Type: text/plain, Size: 441 bytes --]

On 19/06/18 23:30, Brian Woods wrote:
> I'm currently seeing an issue where when booting from a recent Linux
> kernel without nospec_store_bypass_disable.  There's a NULL pointer
> having to do with a lock.  I put some printks in and it seems that in
> arch/x86/kernel/process.c
> that speculative_store_bypass_ht_init isn't getting called which
> initializes the spin lock.

Brian, would you please give the attached patch a try?


Juergen

[-- Attachment #2: 0001-x86-xen-add-call-of-speculative_store_bypass_ht_init.patch --]
[-- Type: text/x-patch, Size: 1587 bytes --]

>From 90a49beee17fe77075a7fdf433c74c5672cfe0d2 Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
Date: Wed, 20 Jun 2018 08:21:33 +0200
Subject: [PATCH] x86/xen: add call of speculative_store_bypass_ht_init() to pv
 paths

Commit 1f50ddb4f4189243c05926b842dc1a0332195f31 ("x86/speculation:
Handle HT correctly on AMD") added speculative_store_bypass_ht_init()
to the per-cpu initialization sequence.

speculative_store_bypass_ht_init() needs to be called on each cpu for
pv guests, too.

Reported-by: Brian Woods <brian.woods@amd.com>
Fixes: 1f50ddb4f4189243c05926b842dc1a0332195f31 ("x86/speculation: Handle HT correctly on AMD")
Cc: <stable@vger.kernel.org>
Signed-off-by: Juergen Gross <jgross@suse.com>
---
 arch/x86/xen/smp_pv.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c
index 2e20ae2fa2d6..e3b18ad49889 100644
--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -32,6 +32,7 @@
 #include <xen/interface/vcpu.h>
 #include <xen/interface/xenpmu.h>
 
+#include <asm/spec-ctrl.h>
 #include <asm/xen/interface.h>
 #include <asm/xen/hypercall.h>
 
@@ -70,6 +71,8 @@ static void cpu_bringup(void)
 	cpu_data(cpu).x86_max_cores = 1;
 	set_cpu_sibling_map(cpu);
 
+	speculative_store_bypass_ht_init();
+
 	xen_setup_cpu_clockevents();
 
 	notify_cpu_starting(cpu);
@@ -250,6 +253,8 @@ static void __init xen_pv_smp_prepare_cpus(unsigned int max_cpus)
 	}
 	set_cpu_sibling_map(0);
 
+	speculative_store_bypass_ht_init();
+
 	xen_pmu_init(0);
 
 	if (xen_smp_intr_init(0) || xen_smp_intr_init_pv(0))
-- 
2.13.7


[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Boris Ostrovsky]

On 06/20/2018 02:17 AM, Juergen Gross wrote:
> On 20/06/18 05:31, Boris Ostrovsky wrote:
>>
>> On 06/19/2018 05:30 PM, Brian Woods wrote:
>>> I'm currently seeing an issue where when booting from a recent Linux
>>> kernel without nospec_store_bypass_disable.  There's a NULL pointer
>>> having to do with a lock.  I put some printks in and it seems that in
>>> arch/x86/kernel/process.c
>>> that speculative_store_bypass_ht_init isn't getting called which
>>> initializes the spin lock.  
>>
>>
>> speculative_store_bypass_ht_init() is not called on PV. For BSP it is
>> called from native_smp_prepare_cpus() and for APs it is called from
>> start_secondary(), neither of which is in PV code path.
>>
>> I think the most logical place to put it is in cpu_init().
> I don't think so. speculative_store_bypass_ht_init() requires
> topology_sibling_cpumask to be setup which happens only after the call
> of cpu_init().

Right.

>
> We should add the call to cpu_bringup() in smp_pv.c

That still leaves cpu0 uninitialized. xen_pv_smp_prepare_cpus() may need
this call as well if we are to follow barmetal example.

I also wonder whether this works at all for !SMP.

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Boris Ostrovsky]

On 06/20/2018 10:27 AM, Boris Ostrovsky wrote:
> On 06/20/2018 02:17 AM, Juergen Gross wrote:
>> On 20/06/18 05:31, Boris Ostrovsky wrote:
>>> On 06/19/2018 05:30 PM, Brian Woods wrote:
>>>> I'm currently seeing an issue where when booting from a recent Linux
>>>> kernel without nospec_store_bypass_disable.  There's a NULL pointer
>>>> having to do with a lock.  I put some printks in and it seems that in
>>>> arch/x86/kernel/process.c
>>>> that speculative_store_bypass_ht_init isn't getting called which
>>>> initializes the spin lock.  
>>>
>>> speculative_store_bypass_ht_init() is not called on PV. For BSP it is
>>> called from native_smp_prepare_cpus() and for APs it is called from
>>> start_secondary(), neither of which is in PV code path.
>>>
>>> I think the most logical place to put it is in cpu_init().
>> I don't think so. speculative_store_bypass_ht_init() requires
>> topology_sibling_cpumask to be setup which happens only after the call
>> of cpu_init().
> Right.
>
>> We should add the call to cpu_bringup() in smp_pv.c
> That still leaves cpu0 uninitialized. xen_pv_smp_prepare_cpus() may need
> this call as well if we are to follow barmetal example.

Oh, and I see now that you posted a patch that does just that.

-boris


>
> I also wonder whether this works at all for !SMP.
>
> -boris
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Brian Woods]

On Wed, Jun 20, 2018 at 08:34:13AM +0200, Juergen Gross wrote:
> Brian, would you please give the attached patch a try?
> 
> 
> Juergen

Juergen,

It works for Dom0.  You see a lot of messages that are like:
(XEN) emul-priv-op.c:1166:d0v1 Domain attempted WRMSR c0011020 from 0x0206800000000000 to 0x0206800000000400
(XEN) emul-priv-op.c:1166:d0v0 Domain attempted WRMSR c0011020 from 0x0206800000000000 to 0x0206800000000400

But it boots up.  I'll test a DomU with a newer kernel after I get back
from lunch.

-- 
Brian Woods

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Andrew Cooper]

On 21/06/18 01:20, Brian Woods wrote:
> On Wed, Jun 20, 2018 at 08:34:13AM +0200, Juergen Gross wrote:
>> Brian, would you please give the attached patch a try?
>>
>>
>> Juergen
> Juergen,
>
> It works for Dom0.  You see a lot of messages that are like:
> (XEN) emul-priv-op.c:1166:d0v1 Domain attempted WRMSR c0011020 from 0x0206800000000000 to 0x0206800000000400
> (XEN) emul-priv-op.c:1166:d0v0 Domain attempted WRMSR c0011020 from 0x0206800000000000 to 0x0206800000000400

That is Linux trying to enable the SSBD via the native mechanism (on
Fam17h hardware I'm guessing, give the bit position).

Like many of the other mitigations, a PV guest knows it is virtualised
and shouldn’t be playing with this MSR.

OTOH, Xen should be implementing the AMD SSBD spec and allowing for
virtualised control of SSBD.  Sadly, I was only show the whitepaper with
insufficient time before the SSBD embargo to implement it in XSA-263,
and haven't had time since.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Brian Woods]

On Wed, Jun 20, 2018 at 12:20:21PM -0500, Brian Woods wrote:
> On Wed, Jun 20, 2018 at 08:34:13AM +0200, Juergen Gross wrote:
> Juergen,
> 
> It works for Dom0.  You see a lot of messages that are like:
> (XEN) emul-priv-op.c:1166:d0v1 Domain attempted WRMSR c0011020 from 0x0206800000000000 to 0x0206800000000400
> (XEN) emul-priv-op.c:1166:d0v0 Domain attempted WRMSR c0011020 from 0x0206800000000000 to 0x0206800000000400
> 
> But it boots up.  I'll test a DomU with a newer kernel after I get back
> from lunch.
> 
> -- 
> Brian Woods

It works on PV and HVM guests.

I'll reply to Andy's email soon.  Have a meeting in 2 minutes...

-- 
Brian Woods

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 Failing to Boot with Recent Linux Kernels (Spectre Mitigations)

[Brian Woods]

On Thu, Jun 21, 2018 at 01:55:54AM +0800, Andrew Cooper wrote:
> That is Linux trying to enable the SSBD via the native mechanism (on
> Fam17h hardware I'm guessing, give the bit position).
> 
> Like many of the other mitigations, a PV guest knows it is virtualised
> and shouldn’t be playing with this MSR.
> 
> OTOH, Xen should be implementing the AMD SSBD spec and allowing for
> virtualised control of SSBD.  Sadly, I was only show the whitepaper with
> insufficient time before the SSBD embargo to implement it in XSA-263,
> and haven't had time since.
> 
> ~Andrew

I'll take a look into updating it for AMD related processors then.  I
will talk with some people internally who did the work for Linux and
see what needs to be done etc.

-- 
Brian Woods

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel