[PATCH 0/2] x86: Disable kexec for SEV-ES guests

[PATCH 0/2] x86: Disable kexec for SEV-ES guests

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

Hi,

two small patches to disable kexec on x86 when running as an SEV-ES
guest. Trying to kexec a new kernel would fail anyway because there is
no mechanism yet to hand over the APs from the old to the new kernel.
Supporting this needs changes in the Hypervisor and the guest kernel
as well.

This code is currently being work on, but disable kexec in SEV-ES
guests until it is ready.

Please review.

Regards,

	Joerg

Joerg Roedel (2):
  kexec: Allow architecture code to opt-out at runtime
  x86/kexec/64: Forbid kexec when running as an SEV-ES guest

 arch/x86/kernel/machine_kexec_64.c |  8 ++++++++
 kernel/kexec.c                     | 14 ++++++++++++++
 2 files changed, 22 insertions(+)

-- 
2.31.1

[PATCH 1/2] kexec: Allow architecture code to opt-out at runtime

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

Allow a runtime opt-out of kexec support for architecture code in case
the kernel is running in an environment where kexec is not properly
supported yet.

This will be used on x86 when the kernel is running as an SEV-ES
guest. SEV-ES guests need special handling for kexec to hand over all
CPUs to the new kernel. This requires special hypervisor support and
handling code in the guest which is not yet implemented.

Cc: stable@vger.kernel.org # v5.10+
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 kernel/kexec.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/kernel/kexec.c b/kernel/kexec.c
index c82c6c06f051..d03134160458 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -195,11 +195,25 @@ static int do_kexec_load(unsigned long entry, unsigned long nr_segments,
  * that to happen you need to do that yourself.
  */
 
+bool __weak arch_kexec_supported(void)
+{
+	return true;
+}
+
 static inline int kexec_load_check(unsigned long nr_segments,
 				   unsigned long flags)
 {
 	int result;
 
+	/*
+	 * The architecture may support kexec in general, but the kernel could
+	 * run in an environment where it is not (yet) possible to execute a new
+	 * kernel. Allow the architecture code to opt-out of kexec support when
+	 * it is running in such an environment.
+	 */
+	if (!arch_kexec_supported())
+		return -ENOSYS;
+
 	/* We only trust the superuser with rebooting the system. */
 	if (!capable(CAP_SYS_BOOT) || kexec_load_disabled)
 		return -EPERM;
-- 
2.31.1

[PATCH 2/2] x86/kexec/64: Forbid kexec when running as an SEV-ES guest

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

For now, kexec is not supported when running as an SEV-ES guest. Doing
so requires additional hypervisor support and special code to hand
over the CPUs to the new kernel in a safe way.

Until this is implemented, do not support kexec in SEV-ES guests.

Cc: stable@vger.kernel.org # v5.10+
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 arch/x86/kernel/machine_kexec_64.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index c078b0d3ab0e..f902cc9cc634 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -620,3 +620,11 @@ void arch_kexec_pre_free_pages(void *vaddr, unsigned int pages)
 	 */
 	set_memory_encrypted((unsigned long)vaddr, pages);
 }
+
+/*
+ * Kexec is not supported in SEV-ES guests yet
+ */
+bool arch_kexec_supported(void)
+{
+	return !sev_es_active();
+}
-- 
2.31.1

Re: [PATCH 1/2] kexec: Allow architecture code to opt-out at runtime

[Sean Christopherson]

On Thu, May 06, 2021, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
> 
> Allow a runtime opt-out of kexec support for architecture code in case
> the kernel is running in an environment where kexec is not properly
> supported yet.
> 
> This will be used on x86 when the kernel is running as an SEV-ES
> guest. SEV-ES guests need special handling for kexec to hand over all
> CPUs to the new kernel. This requires special hypervisor support and
> handling code in the guest which is not yet implemented.
> 
> Cc: stable@vger.kernel.org # v5.10+
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  kernel/kexec.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/kernel/kexec.c b/kernel/kexec.c
> index c82c6c06f051..d03134160458 100644
> --- a/kernel/kexec.c
> +++ b/kernel/kexec.c
> @@ -195,11 +195,25 @@ static int do_kexec_load(unsigned long entry, unsigned long nr_segments,
>   * that to happen you need to do that yourself.
>   */
>  
> +bool __weak arch_kexec_supported(void)
> +{
> +	return true;
> +}
> +
>  static inline int kexec_load_check(unsigned long nr_segments,
>  				   unsigned long flags)
>  {
>  	int result;
>  
> +	/*
> +	 * The architecture may support kexec in general, but the kernel could
> +	 * run in an environment where it is not (yet) possible to execute a new
> +	 * kernel. Allow the architecture code to opt-out of kexec support when
> +	 * it is running in such an environment.
> +	 */
> +	if (!arch_kexec_supported())
> +		return -ENOSYS;

This misses kexec_file_load.  Also, is a new hook really needed?  E.g. the
SEV-ES check be shoved into machine_kexec_prepare().  The downside is that we'd
do a fair amount of work before detecting failure, but that doesn't seem hugely
problematic.

> +
>  	/* We only trust the superuser with rebooting the system. */
>  	if (!capable(CAP_SYS_BOOT) || kexec_load_disabled)
>  		return -EPERM;
> -- 
> 2.31.1
> 

Re: [PATCH 2/2] x86/kexec/64: Forbid kexec when running as an SEV-ES guest

[Eric W. Biederman]

Joerg Roedel <joro@8bytes.org> writes:

> From: Joerg Roedel <jroedel@suse.de>
>
> For now, kexec is not supported when running as an SEV-ES guest. Doing
> so requires additional hypervisor support and special code to hand
> over the CPUs to the new kernel in a safe way.
>
> Until this is implemented, do not support kexec in SEV-ES guests.

I don't understand this.

Fundamentally kexec is about doing things more or less inspite of
what the firmware is doing.

I don't have any idea what a SEV-ES is.  But the normal x86 boot doesn't
do anything special.  Is cross cpu IPI emulation buggy?

If this is a move in your face hypervisor like Xen is sometimes I can
see perhaps needing a little bit of different work during bootup.
Perhaps handing back a cpu on system shutdown and asking for more cpus
on system boot up.

What is the actual problem you are trying to avoid?

And yes for a temporary hack the suggestion of putting code into
machine_kexec_prepare seems much more reasonable so we don't have to
carry special case infrastructure for the forseeable future.

Eric


> Cc: stable@vger.kernel.org # v5.10+
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  arch/x86/kernel/machine_kexec_64.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
> index c078b0d3ab0e..f902cc9cc634 100644
> --- a/arch/x86/kernel/machine_kexec_64.c
> +++ b/arch/x86/kernel/machine_kexec_64.c
> @@ -620,3 +620,11 @@ void arch_kexec_pre_free_pages(void *vaddr, unsigned int pages)
>  	 */
>  	set_memory_encrypted((unsigned long)vaddr, pages);
>  }
> +
> +/*
> + * Kexec is not supported in SEV-ES guests yet
> + */
> +bool arch_kexec_supported(void)
> +{
> +	return !sev_es_active();
> +}

Re: [PATCH 1/2] kexec: Allow architecture code to opt-out at runtime

[Joerg Roedel]

On Thu, May 06, 2021 at 03:43:23PM +0000, Sean Christopherson wrote:
> This misses kexec_file_load.

Right, thanks, I will fix that in the next version.

> Also, is a new hook really needed?  E.g. the SEV-ES check be shoved
> into machine_kexec_prepare().  The downside is that we'd do a fair
> amount of work before detecting failure, but that doesn't seem hugely
> problematic.

That could work, but I think its more user-friendly to just claim that
the syscalls are not supported at all.

Regards,

	Joerg

Re: [PATCH 2/2] x86/kexec/64: Forbid kexec when running as an SEV-ES guest

[Joerg Roedel]

On Thu, May 06, 2021 at 12:42:03PM -0500, Eric W. Biederman wrote:
> I don't understand this.
> 
> Fundamentally kexec is about doing things more or less inspite of
> what the firmware is doing.
> 
> I don't have any idea what a SEV-ES is.  But the normal x86 boot doesn't
> do anything special.  Is cross cpu IPI emulation buggy?

Under SEV-ES the normal SIPI-based sequence to re-initialize a CPU does
not work anymore. An SEV-ES guest is a special virtual machine where the
hardware encrypts the guest memory and the guest register state. The
hypervisor can't make any modifications to the guests registers at
runtime. Therefore it also can't emulate a SIPI sequence and reset the
vCPU.

The guest kernel has to reset the vCPU itself and hand it over from the
old kernel to the kexec'ed kernel. This isn't currently implemented and
therefore kexec needs to be disabled when running as an SEV-ES guest.

Implementing this also requires an extension to the guest-hypervisor
protocol (the GHCB Spec[1]) which is only available in version 2. So a
guest running on a hypervisor supporting only version 1 will never
properly support kexec.

> What is the actual problem you are trying to avoid?

Currently, if someone tries kexec in an SEV-ES guest, the kexec'ed
kernel will only be able to bring up the boot CPU, not the others. The
others will wake up with the old kernels CPU state in the new kernels
memory and do undefined things, most likely triple-fault because their
page-table is not existent anymore.

So since kexec currently does not work as expected under SEV-ES, it is
better to hide it until everything is implemented so it can do what the
user expects.

> And yes for a temporary hack the suggestion of putting code into
> machine_kexec_prepare seems much more reasonable so we don't have to
> carry special case infrastructure for the forseeable future.

As I said above, for protocol version 1 it will stay disabled, so it is
not only a temporary hack.

Regards,

	Joerg

[1] https://developer.amd.com/wp-content/resources/56421.pdf

Re: [PATCH 2/2] x86/kexec/64: Forbid kexec when running as an SEV-ES guest

[Eric W. Biederman]

Joerg Roedel <jroedel@suse.de> writes:

> On Thu, May 06, 2021 at 12:42:03PM -0500, Eric W. Biederman wrote:
>> I don't understand this.
>> 
>> Fundamentally kexec is about doing things more or less inspite of
>> what the firmware is doing.
>> 
>> I don't have any idea what a SEV-ES is.  But the normal x86 boot doesn't
>> do anything special.  Is cross cpu IPI emulation buggy?
>
> Under SEV-ES the normal SIPI-based sequence to re-initialize a CPU does
> not work anymore. An SEV-ES guest is a special virtual machine where the
> hardware encrypts the guest memory and the guest register state. The
> hypervisor can't make any modifications to the guests registers at
> runtime. Therefore it also can't emulate a SIPI sequence and reset the
> vCPU.
>
> The guest kernel has to reset the vCPU itself and hand it over from the
> old kernel to the kexec'ed kernel. This isn't currently implemented and
> therefore kexec needs to be disabled when running as an SEV-ES guest.
>
> Implementing this also requires an extension to the guest-hypervisor
> protocol (the GHCB Spec[1]) which is only available in version 2. So a
> guest running on a hypervisor supporting only version 1 will never
> properly support kexec.

Why does it need that?

Would it not make sense to instead teach kexec how to pass a cpu from
one kernel to another.  We could use that everywhere.

Even the kexec-on-panic case should work as even in that case we have
to touch the cpus as they go down.

The hardware simply worked well enough that it hasn't mattered enough
for us to do something like that, but given that we need to do something
anyway.  It seems like it would make most sense do something that
will work everywhere, and does not introduce unnecessary dependencies
on hypervisors.

>> What is the actual problem you are trying to avoid?
>
> Currently, if someone tries kexec in an SEV-ES guest, the kexec'ed
> kernel will only be able to bring up the boot CPU, not the others. The
> others will wake up with the old kernels CPU state in the new kernels
> memory and do undefined things, most likely triple-fault because their
> page-table is not existent anymore.
>
> So since kexec currently does not work as expected under SEV-ES, it is
> better to hide it until everything is implemented so it can do what the
> user expects.

I can understand temporarily disabling the functionality.

>> And yes for a temporary hack the suggestion of putting code into
>> machine_kexec_prepare seems much more reasonable so we don't have to
>> carry special case infrastructure for the forseeable future.
>
> As I said above, for protocol version 1 it will stay disabled, so it is
> not only a temporary hack.

Why does bringing up a cpu need hypervisor support?

I understand why we can't do what we do currently, but that doesn't seem
to preclude doing something without hypervisor support.

Eric

Re: [PATCH 2/2] x86/kexec/64: Forbid kexec when running as an SEV-ES guest

[Joerg Roedel]

On Thu, May 06, 2021 at 01:59:42PM -0500, Eric W. Biederman wrote:
> Joerg Roedel <jroedel@suse.de> writes:

> Why does it need that?
> 
> Would it not make sense to instead teach kexec how to pass a cpu from
> one kernel to another.  We could use that everywhere.
> 
> Even the kexec-on-panic case should work as even in that case we have
> to touch the cpus as they go down.
> 
> The hardware simply worked well enough that it hasn't mattered enough
> for us to do something like that, but given that we need to do something
> anyway.  It seems like it would make most sense do something that
> will work everywhere, and does not introduce unnecessary dependencies
> on hypervisors.

Well, I guess we could implement something that even works for non
SEV-ES guests and bare-metal. The question is what benefit we get from
that. Is the SIPI sequence currently used not reliable enough?

The benefit of being able to rely on the SIPI sequence is that the
kexec'ed kernel can use the same method to bring up APs as the first
kernel did.

Btw, the same is true for SEV-ES guests, The goal is bring the APs of
an SEV-ES guest into a state where they will use the SEV-ES method of
AP-bringup when they wake up again. This method involves a
firmware-owned page called the AP-jump-table, which contains the reset
vector for the AP in its first 4 bytes.

> > As I said above, for protocol version 1 it will stay disabled, so it is
> > not only a temporary hack.
> 
> Why does bringing up a cpu need hypervisor support?

When a CPU is taken offline under SEV-ES it will do a special hypercall
named AP-reset-hold. The hypervisor will put the CPU into a halt state
until the next SIPI arrives. In protocol version 1 this hypercall
requires a GHCB shared page to be set up for the CPU doing the hypercall
and upon CPU wakeup the HV will write to that shared page. Problem with
that is that the page which contains the GHCB is already owned by the
new kernel then and is probably not shared anymore, which can cause data
corruption in the new kernel.

Version 2 of the protocol adds a purely MSR based version of the
AP-reset-hold hypercall. This one does not need a GHCB page and has to
be used to bring the APs offline before doing the kexec. That is the
reason I plan to only support kexec when the hypervisor provides version
2 of the protocol.

Regards,

	Joerg