[PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Vitaly Kuznetsov]

Currently there is a number of issues preventing PVHVM Xen guests from
doing successful kexec/kdump:
- Bound event channels.
- Registered vcpu_info.
- PIRQ/emuirq mappings.
- shared_info frame after XENMAPSPACE_shared_info operation.
- Active grant mappings.
Basically, newly booted kernel stumbles upon already set up Xen interfaces
and there is no way to reestablish them. In Xen-4.7 a new feature called
'soft reset' is coming. A guest performing kexec/kdump operation is
supposed to call SCHEDOP_shutdown hypercall with SHUTDOWN_soft_reset reason
before jumping to new kernel. Hypervisor (with some help from toolstack)
will do full domain cleanup (but keeping its memory and vCPU contexts
intact) returning the guest to the state it had when it was first booted
and thus allowing it to start over.

Doing SHUTDOWN_soft_reset on Xen hypervisors which don't support it is
probably OK as by default all unknown shutdown reasons cause domain destroy
with a message in toolstack log:
'Unknown shutdown reason code 5. Destroying domain.'
which gives a clue to what the problem is and eliminates false
expectations.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 arch/x86/xen/enlighten.c      | 23 +++++++++++++++++++++++
 include/xen/interface/sched.h |  8 ++++++++
 2 files changed, 31 insertions(+)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 30d12af..e4be074 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -33,6 +33,10 @@
 #include <linux/memblock.h>
 #include <linux/edd.h>
 
+#ifdef CONFIG_KEXEC_CORE
+#include <linux/kexec.h>
+#endif
+
 #include <xen/xen.h>
 #include <xen/events.h>
 #include <xen/interface/xen.h>
@@ -1807,6 +1811,21 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+#ifdef CONFIG_KEXEC_CORE
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	native_machine_shutdown();
+	if (kexec_in_progress)
+		xen_reboot(SHUTDOWN_soft_reset);
+}
+
+static void xen_pvhvm_crash_shutdown(struct pt_regs *regs)
+{
+	native_machine_crash_shutdown(regs);
+	xen_reboot(SHUTDOWN_soft_reset);
+}
+#endif
+
 static void __init xen_hvm_guest_init(void)
 {
 	if (xen_pv_domain())
@@ -1826,6 +1845,10 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC_CORE
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+	machine_ops.crash_shutdown = xen_pvhvm_crash_shutdown;
+#endif
 }
 #endif
 
diff --git a/include/xen/interface/sched.h b/include/xen/interface/sched.h
index 9ce0839..f184909 100644
--- a/include/xen/interface/sched.h
+++ b/include/xen/interface/sched.h
@@ -107,5 +107,13 @@ struct sched_watchdog {
 #define SHUTDOWN_suspend    2  /* Clean up, save suspend info, kill.         */
 #define SHUTDOWN_crash      3  /* Tell controller we've crashed.             */
 #define SHUTDOWN_watchdog   4  /* Restart because watchdog time expired.     */
+/*
+ * Domain asked to perform 'soft reset' for it. The expected behavior is to
+ * reset internal Xen state for the domain returning it to the point where it
+ * was created but leaving the domain's memory contents and vCPU contexts
+ * intact. This will allow the domain to start over and set up all Xen specific
+ * interfaces again.
+ */
+#define SHUTDOWN_soft_reset 5
 
 #endif /* __XEN_PUBLIC_SCHED_H__ */
-- 
2.4.3

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Konrad Rzeszutek Wilk]

On Fri, Sep 25, 2015 at 11:59:52AM +0200, Vitaly Kuznetsov wrote:
> Currently there is a number of issues preventing PVHVM Xen guests from
> doing successful kexec/kdump:
> - Bound event channels.
> - Registered vcpu_info.
> - PIRQ/emuirq mappings.
> - shared_info frame after XENMAPSPACE_shared_info operation.
> - Active grant mappings.
> Basically, newly booted kernel stumbles upon already set up Xen interfaces
> and there is no way to reestablish them. In Xen-4.7 a new feature called
> 'soft reset' is coming. A guest performing kexec/kdump operation is
> supposed to call SCHEDOP_shutdown hypercall with SHUTDOWN_soft_reset reason
> before jumping to new kernel. Hypervisor (with some help from toolstack)
> will do full domain cleanup (but keeping its memory and vCPU contexts
> intact) returning the guest to the state it had when it was first booted
> and thus allowing it to start over.
> 
> Doing SHUTDOWN_soft_reset on Xen hypervisors which don't support it is
> probably OK as by default all unknown shutdown reasons cause domain destroy
> with a message in toolstack log:
> 'Unknown shutdown reason code 5. Destroying domain.'
> which gives a clue to what the problem is and eliminates false
> expectations.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

with s/pvhvm/hvm/

to match with the rest of the file.

Thanks.
> ---
>  arch/x86/xen/enlighten.c      | 23 +++++++++++++++++++++++
>  include/xen/interface/sched.h |  8 ++++++++
>  2 files changed, 31 insertions(+)
> 
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index 30d12af..e4be074 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -33,6 +33,10 @@
>  #include <linux/memblock.h>
>  #include <linux/edd.h>
>  
> +#ifdef CONFIG_KEXEC_CORE
> +#include <linux/kexec.h>
> +#endif
> +
>  #include <xen/xen.h>
>  #include <xen/events.h>
>  #include <xen/interface/xen.h>
> @@ -1807,6 +1811,21 @@ static struct notifier_block xen_hvm_cpu_notifier = {
>  	.notifier_call	= xen_hvm_cpu_notify,
>  };
>  
> +#ifdef CONFIG_KEXEC_CORE
> +static void xen_pvhvm_kexec_shutdown(void)
> +{
> +	native_machine_shutdown();
> +	if (kexec_in_progress)
> +		xen_reboot(SHUTDOWN_soft_reset);
> +}
> +
> +static void xen_pvhvm_crash_shutdown(struct pt_regs *regs)
> +{
> +	native_machine_crash_shutdown(regs);
> +	xen_reboot(SHUTDOWN_soft_reset);
> +}
> +#endif
> +
>  static void __init xen_hvm_guest_init(void)
>  {
>  	if (xen_pv_domain())
> @@ -1826,6 +1845,10 @@ static void __init xen_hvm_guest_init(void)
>  	x86_init.irqs.intr_init = xen_init_IRQ;
>  	xen_hvm_init_time_ops();
>  	xen_hvm_init_mmu_ops();
> +#ifdef CONFIG_KEXEC_CORE
> +	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
> +	machine_ops.crash_shutdown = xen_pvhvm_crash_shutdown;
> +#endif
>  }
>  #endif
>  
> diff --git a/include/xen/interface/sched.h b/include/xen/interface/sched.h
> index 9ce0839..f184909 100644
> --- a/include/xen/interface/sched.h
> +++ b/include/xen/interface/sched.h
> @@ -107,5 +107,13 @@ struct sched_watchdog {
>  #define SHUTDOWN_suspend    2  /* Clean up, save suspend info, kill.         */
>  #define SHUTDOWN_crash      3  /* Tell controller we've crashed.             */
>  #define SHUTDOWN_watchdog   4  /* Restart because watchdog time expired.     */
> +/*
> + * Domain asked to perform 'soft reset' for it. The expected behavior is to
> + * reset internal Xen state for the domain returning it to the point where it
> + * was created but leaving the domain's memory contents and vCPU contexts
> + * intact. This will allow the domain to start over and set up all Xen specific
> + * interfaces again.
> + */
> +#define SHUTDOWN_soft_reset 5
>  
>  #endif /* __XEN_PUBLIC_SCHED_H__ */
> -- 
> 2.4.3
> 

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Boris Ostrovsky]

On 09/25/2015 09:59 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 25, 2015 at 11:59:52AM +0200, Vitaly Kuznetsov wrote:
>> Currently there is a number of issues preventing PVHVM Xen guests from
>> doing successful kexec/kdump:
>> - Bound event channels.
>> - Registered vcpu_info.
>> - PIRQ/emuirq mappings.
>> - shared_info frame after XENMAPSPACE_shared_info operation.
>> - Active grant mappings.
>> Basically, newly booted kernel stumbles upon already set up Xen interfaces
>> and there is no way to reestablish them. In Xen-4.7 a new feature called
>> 'soft reset' is coming. A guest performing kexec/kdump operation is
>> supposed to call SCHEDOP_shutdown hypercall with SHUTDOWN_soft_reset reason
>> before jumping to new kernel. Hypervisor (with some help from toolstack)
>> will do full domain cleanup (but keeping its memory and vCPU contexts
>> intact) returning the guest to the state it had when it was first booted
>> and thus allowing it to start over.
>>
>> Doing SHUTDOWN_soft_reset on Xen hypervisors which don't support it is
>> probably OK as by default all unknown shutdown reasons cause domain destroy
>> with a message in toolstack log:
>> 'Unknown shutdown reason code 5. Destroying domain.'
>> which gives a clue to what the problem is and eliminates false
>> expectations.
>>
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>
> with s/pvhvm/hvm/
>
> to match with the rest of the file.
>
> Thanks.
>>   
>> +#ifdef CONFIG_KEXEC_CORE
>> +static void xen_pvhvm_kexec_shutdown(void)
>> +{
>> +	native_machine_shutdown();
>> +	if (kexec_in_progress)
>> +		xen_reboot(SHUTDOWN_soft_reset);
>> +}

This routine is not kexec-specific so "kexec_" should be dropped from 
the name.

Also, I am not sure I see how this new op will be used in the hypervisor 
--- currently AFAICS it is only processed under is_hardware_domain(). 
Are there other patches that will support HVM guests?


-boris

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Vitaly Kuznetsov]

Boris Ostrovsky <boris.ostrovsky@oracle.com> writes:

> On 09/25/2015 09:59 AM, Konrad Rzeszutek Wilk wrote:
>> On Fri, Sep 25, 2015 at 11:59:52AM +0200, Vitaly Kuznetsov wrote:
>>> Currently there is a number of issues preventing PVHVM Xen guests from
>>> doing successful kexec/kdump:
>>> - Bound event channels.
>>> - Registered vcpu_info.
>>> - PIRQ/emuirq mappings.
>>> - shared_info frame after XENMAPSPACE_shared_info operation.
>>> - Active grant mappings.
>>> Basically, newly booted kernel stumbles upon already set up Xen interfaces
>>> and there is no way to reestablish them. In Xen-4.7 a new feature called
>>> 'soft reset' is coming. A guest performing kexec/kdump operation is
>>> supposed to call SCHEDOP_shutdown hypercall with SHUTDOWN_soft_reset reason
>>> before jumping to new kernel. Hypervisor (with some help from toolstack)
>>> will do full domain cleanup (but keeping its memory and vCPU contexts
>>> intact) returning the guest to the state it had when it was first booted
>>> and thus allowing it to start over.
>>>
>>> Doing SHUTDOWN_soft_reset on Xen hypervisors which don't support it is
>>> probably OK as by default all unknown shutdown reasons cause domain destroy
>>> with a message in toolstack log:
>>> 'Unknown shutdown reason code 5. Destroying domain.'
>>> which gives a clue to what the problem is and eliminates false
>>> expectations.
>>>
>>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>>
>> with s/pvhvm/hvm/
>>
>> to match with the rest of the file.
>>
>> Thanks.
>>>   +#ifdef CONFIG_KEXEC_CORE
>>> +static void xen_pvhvm_kexec_shutdown(void)
>>> +{
>>> +	native_machine_shutdown();
>>> +	if (kexec_in_progress)
>>> +		xen_reboot(SHUTDOWN_soft_reset);
>>> +}
>
> This routine is not kexec-specific so "kexec_" should be dropped from
> the name.

Ok,

>
> Also, I am not sure I see how this new op will be used in the
> hypervisor --- currently AFAICS it is only processed under
> is_hardware_domain(). Are there other patches that will support HVM
> guests?

Please see my Xen series:
http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
(last 'full' submission).

All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
are already merged to xen.git (first 10 are already in 'master' and the
last one is in 'staging').

-- 
  Vitaly

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Boris Ostrovsky]

On 09/25/2015 12:07 PM, Vitaly Kuznetsov wrote:
>> Also, I am not sure I see how this new op will be used in the
>> hypervisor --- currently AFAICS it is only processed under
>> is_hardware_domain(). Are there other patches that will support HVM
>> guests?
> Please see my Xen series:
> http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
> (last 'full' submission).
>
> All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
> are already merged to xen.git (first 10 are already in 'master' and the
> last one is in 'staging').
>


OK, so I was looking at the right tree. Then I don't understand how 
SHUTDOWN_soft_reset would be reached for a non-privileged domain. The 
only path that I see is

     domain_shutdown()
     {
         ...
         if ( is_hardware_domain(d) )
             hwdom_shutdown(reason);
         ...
     }

Is there another path to handle this op?

Thanks.
-boris

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Konrad Rzeszutek Wilk]

On Fri, Sep 25, 2015 at 01:17:40PM -0400, Boris Ostrovsky wrote:
> On 09/25/2015 12:07 PM, Vitaly Kuznetsov wrote:
> >>Also, I am not sure I see how this new op will be used in the
> >>hypervisor --- currently AFAICS it is only processed under
> >>is_hardware_domain(). Are there other patches that will support HVM
> >>guests?
> >Please see my Xen series:
> >http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
> >(last 'full' submission).
> >
> >All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
> >are already merged to xen.git (first 10 are already in 'master' and the
> >last one is in 'staging').
> >
> 
> 
> OK, so I was looking at the right tree. Then I don't understand how
> SHUTDOWN_soft_reset would be reached for a non-privileged domain. The only
> path that I see is
> 
>     domain_shutdown()
>     {
>         ...
>         if ( is_hardware_domain(d) )
>             hwdom_shutdown(reason);
>         ...
>     }
> 
> Is there another path to handle this op?

Yes:
 e1bd9812966de9a16f30a58e7162b80bd6af361b libxc: support XEN_DOMCTL_soft_reset operation
and
 c57e6ebd8c3e490e353e68d96abec1dad01e72f5 (lib)xl: soft reset support

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Boris Ostrovsky]

On 09/25/2015 03:01 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 25, 2015 at 01:17:40PM -0400, Boris Ostrovsky wrote:
>> On 09/25/2015 12:07 PM, Vitaly Kuznetsov wrote:
>>>> Also, I am not sure I see how this new op will be used in the
>>>> hypervisor --- currently AFAICS it is only processed under
>>>> is_hardware_domain(). Are there other patches that will support HVM
>>>> guests?
>>> Please see my Xen series:
>>> http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
>>> (last 'full' submission).
>>>
>>> All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
>>> are already merged to xen.git (first 10 are already in 'master' and the
>>> last one is in 'staging').
>>>
>>
>> OK, so I was looking at the right tree. Then I don't understand how
>> SHUTDOWN_soft_reset would be reached for a non-privileged domain. The only
>> path that I see is
>>
>>      domain_shutdown()
>>      {
>>          ...
>>          if ( is_hardware_domain(d) )
>>              hwdom_shutdown(reason);
>>          ...
>>      }
>>
>> Is there another path to handle this op?
> Yes:
>   e1bd9812966de9a16f30a58e7162b80bd6af361b libxc: support XEN_DOMCTL_soft_reset operation
> and
>   c57e6ebd8c3e490e353e68d96abec1dad01e72f5 (lib)xl: soft reset support
>


That's toolstack issuing hypercalls from dom0.

I am asking about (non-privileged) guest itself calling SCHEDOP_shutdown.

-boris

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Konrad Rzeszutek Wilk]

On Fri, Sep 25, 2015 at 03:19:57PM -0400, Boris Ostrovsky wrote:
> On 09/25/2015 03:01 PM, Konrad Rzeszutek Wilk wrote:
> >On Fri, Sep 25, 2015 at 01:17:40PM -0400, Boris Ostrovsky wrote:
> >>On 09/25/2015 12:07 PM, Vitaly Kuznetsov wrote:
> >>>>Also, I am not sure I see how this new op will be used in the
> >>>>hypervisor --- currently AFAICS it is only processed under
> >>>>is_hardware_domain(). Are there other patches that will support HVM
> >>>>guests?
> >>>Please see my Xen series:
> >>>http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
> >>>(last 'full' submission).
> >>>
> >>>All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
> >>>are already merged to xen.git (first 10 are already in 'master' and the
> >>>last one is in 'staging').
> >>>
> >>
> >>OK, so I was looking at the right tree. Then I don't understand how
> >>SHUTDOWN_soft_reset would be reached for a non-privileged domain. The only
> >>path that I see is
> >>
> >>     domain_shutdown()
> >>     {
> >>         ...
> >>         if ( is_hardware_domain(d) )
> >>             hwdom_shutdown(reason);
> >>         ...
> >>     }
> >>
> >>Is there another path to handle this op?
> >Yes:
> >  e1bd9812966de9a16f30a58e7162b80bd6af361b libxc: support XEN_DOMCTL_soft_reset operation
> >and
> >  c57e6ebd8c3e490e353e68d96abec1dad01e72f5 (lib)xl: soft reset support
> >
> 
> 
> That's toolstack issuing hypercalls from dom0.
> 
> I am asking about (non-privileged) guest itself calling SCHEDOP_shutdown.

The hypervisor ends up calling:
__domain_finalise_shutdown which sends an VIRQ_DOM_EXC to dom0 which
makes the toolstack do all of those operations.

> 
> -boris

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Boris Ostrovsky]

On 09/25/2015 03:35 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 25, 2015 at 03:19:57PM -0400, Boris Ostrovsky wrote:
>> On 09/25/2015 03:01 PM, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Sep 25, 2015 at 01:17:40PM -0400, Boris Ostrovsky wrote:
>>>> On 09/25/2015 12:07 PM, Vitaly Kuznetsov wrote:
>>>>>> Also, I am not sure I see how this new op will be used in the
>>>>>> hypervisor --- currently AFAICS it is only processed under
>>>>>> is_hardware_domain(). Are there other patches that will support HVM
>>>>>> guests?
>>>>> Please see my Xen series:
>>>>> http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
>>>>> (last 'full' submission).
>>>>>
>>>>> All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
>>>>> are already merged to xen.git (first 10 are already in 'master' and the
>>>>> last one is in 'staging').
>>>>>
>>>> OK, so I was looking at the right tree. Then I don't understand how
>>>> SHUTDOWN_soft_reset would be reached for a non-privileged domain. The only
>>>> path that I see is
>>>>
>>>>      domain_shutdown()
>>>>      {
>>>>          ...
>>>>          if ( is_hardware_domain(d) )
>>>>              hwdom_shutdown(reason);
>>>>          ...
>>>>      }
>>>>
>>>> Is there another path to handle this op?
>>> Yes:
>>>   e1bd9812966de9a16f30a58e7162b80bd6af361b libxc: support XEN_DOMCTL_soft_reset operation
>>> and
>>>   c57e6ebd8c3e490e353e68d96abec1dad01e72f5 (lib)xl: soft reset support
>>>
>>
>> That's toolstack issuing hypercalls from dom0.
>>
>> I am asking about (non-privileged) guest itself calling SCHEDOP_shutdown.
> The hypervisor ends up calling:
> __domain_finalise_shutdown which sends an VIRQ_DOM_EXC to dom0 which
> makes the toolstack do all of those operations.

OK. But the I don't see anyone checking that 'reason' (or 
'shutdown_code') is SHUTDOWN_soft_reset. In other words, the guest can 
do 'xen_reboot(23)' or 'xen_reboot(154)'. Or, it seems, even 
'xen_reboot(SHUTDOWN_reboot)'? The only value it shouldn't be is 
SHUTDOWN_suspend.

-boris

Re: [Xen-devel] [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[David Vrabel]

On 25/09/15 10:59, Vitaly Kuznetsov wrote:
> Currently there is a number of issues preventing PVHVM Xen guests from
> doing successful kexec/kdump:
> - Bound event channels.
> - Registered vcpu_info.
> - PIRQ/emuirq mappings.
> - shared_info frame after XENMAPSPACE_shared_info operation.
> - Active grant mappings.
> Basically, newly booted kernel stumbles upon already set up Xen interfaces
> and there is no way to reestablish them. In Xen-4.7 a new feature called
> 'soft reset' is coming. A guest performing kexec/kdump operation is
> supposed to call SCHEDOP_shutdown hypercall with SHUTDOWN_soft_reset reason
> before jumping to new kernel. Hypervisor (with some help from toolstack)
> will do full domain cleanup (but keeping its memory and vCPU contexts
> intact) returning the guest to the state it had when it was first booted
> and thus allowing it to start over.
> 
> Doing SHUTDOWN_soft_reset on Xen hypervisors which don't support it is
> probably OK as by default all unknown shutdown reasons cause domain destroy
> with a message in toolstack log:
> 'Unknown shutdown reason code 5. Destroying domain.'
> which gives a clue to what the problem is and eliminates false
> expectations.

I've fixed up the naming (see below) and applied to for-linus-4.3b, thanks.

I've also tagged it for stable since this looks like something that
distros would want and I think this is useful to have in place when soft
reset support in Xen becomes available.

I can drop the stable tag if others disagree.

David

8<--------------------------------
x86/xen: Support kexec/kdump in HVM guests by doing a soft
 reset

Currently there is a number of issues preventing PVHVM Xen guests from
doing successful kexec/kdump:

  - Bound event channels.
  - Registered vcpu_info.
  - PIRQ/emuirq mappings.
  - shared_info frame after XENMAPSPACE_shared_info operation.
  - Active grant mappings.

Basically, newly booted kernel stumbles upon already set up Xen
interfaces and there is no way to reestablish them. In Xen-4.7 a new
feature called 'soft reset' is coming. A guest performing kexec/kdump
operation is supposed to call SCHEDOP_shutdown hypercall with
SHUTDOWN_soft_reset reason before jumping to new kernel. Hypervisor
(with some help from toolstack) will do full domain cleanup (but
keeping its memory and vCPU contexts intact) returning the guest to
the state it had when it was first booted and thus allowing it to
start over.

Doing SHUTDOWN_soft_reset on Xen hypervisors which don't support it is
probably OK as by default all unknown shutdown reasons cause domain
destroy with a message in toolstack log: 'Unknown shutdown reason code
5. Destroying domain.'  which gives a clue to what the problem is and
eliminates false expectations.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
---
 arch/x86/xen/enlighten.c      | 23 +++++++++++++++++++++++
 include/xen/interface/sched.h |  8 ++++++++
 2 files changed, 31 insertions(+)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 1f1dbd2..993b7a7 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -33,6 +33,10 @@
 #include <linux/memblock.h>
 #include <linux/edd.h>

+#ifdef CONFIG_KEXEC_CORE
+#include <linux/kexec.h>
+#endif
+
 #include <xen/xen.h>
 #include <xen/events.h>
 #include <xen/interface/xen.h>
@@ -1808,6 +1812,21 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };

+#ifdef CONFIG_KEXEC_CORE
+static void xen_hvm_shutdown(void)
+{
+	native_machine_shutdown();
+	if (kexec_in_progress)
+		xen_reboot(SHUTDOWN_soft_reset);
+}
+
+static void xen_hvm_crash_shutdown(struct pt_regs *regs)
+{
+	native_machine_crash_shutdown(regs);
+	xen_reboot(SHUTDOWN_soft_reset);
+}
+#endif
+
 static void __init xen_hvm_guest_init(void)
 {
 	if (xen_pv_domain())
@@ -1827,6 +1846,10 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC_CORE
+	machine_ops.shutdown = xen_hvm_shutdown;
+	machine_ops.crash_shutdown = xen_hvm_crash_shutdown;
+#endif
 }
 #endif

diff --git a/include/xen/interface/sched.h b/include/xen/interface/sched.h
index 9ce0839..f184909 100644
--- a/include/xen/interface/sched.h
+++ b/include/xen/interface/sched.h
@@ -107,5 +107,13 @@ struct sched_watchdog {
 #define SHUTDOWN_suspend    2  /* Clean up, save suspend info, kill.
      */
 #define SHUTDOWN_crash      3  /* Tell controller we've crashed.
      */
 #define SHUTDOWN_watchdog   4  /* Restart because watchdog time
expired.     */
+/*
+ * Domain asked to perform 'soft reset' for it. The expected behavior is to
+ * reset internal Xen state for the domain returning it to the point
where it
+ * was created but leaving the domain's memory contents and vCPU contexts
+ * intact. This will allow the domain to start over and set up all Xen
specific
+ * interfaces again.
+ */
+#define SHUTDOWN_soft_reset 5

 #endif /* __XEN_PUBLIC_SCHED_H__ */
-- 
2.1.4

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Vitaly Kuznetsov]

Boris Ostrovsky <boris.ostrovsky@oracle.com> writes:

> On 09/25/2015 03:35 PM, Konrad Rzeszutek Wilk wrote:
>> On Fri, Sep 25, 2015 at 03:19:57PM -0400, Boris Ostrovsky wrote:
>>> On 09/25/2015 03:01 PM, Konrad Rzeszutek Wilk wrote:
>>>> On Fri, Sep 25, 2015 at 01:17:40PM -0400, Boris Ostrovsky wrote:
>>>>> On 09/25/2015 12:07 PM, Vitaly Kuznetsov wrote:
>>>>>>> Also, I am not sure I see how this new op will be used in the
>>>>>>> hypervisor --- currently AFAICS it is only processed under
>>>>>>> is_hardware_domain(). Are there other patches that will support HVM
>>>>>>> guests?
>>>>>> Please see my Xen series:
>>>>>> http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
>>>>>> (last 'full' submission).
>>>>>>
>>>>>> All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
>>>>>> are already merged to xen.git (first 10 are already in 'master' and the
>>>>>> last one is in 'staging').
>>>>>>
>>>>> OK, so I was looking at the right tree. Then I don't understand how
>>>>> SHUTDOWN_soft_reset would be reached for a non-privileged domain. The only
>>>>> path that I see is
>>>>>
>>>>>      domain_shutdown()
>>>>>      {
>>>>>          ...
>>>>>          if ( is_hardware_domain(d) )
>>>>>              hwdom_shutdown(reason);
>>>>>          ...
>>>>>      }
>>>>>
>>>>> Is there another path to handle this op?
>>>> Yes:
>>>>   e1bd9812966de9a16f30a58e7162b80bd6af361b libxc: support XEN_DOMCTL_soft_reset operation
>>>> and
>>>>   c57e6ebd8c3e490e353e68d96abec1dad01e72f5 (lib)xl: soft reset support
>>>>
>>>
>>> That's toolstack issuing hypercalls from dom0.
>>>
>>> I am asking about (non-privileged) guest itself calling SCHEDOP_shutdown.
>> The hypervisor ends up calling:
>> __domain_finalise_shutdown which sends an VIRQ_DOM_EXC to dom0 which
>> makes the toolstack do all of those operations.
>
> OK. But the I don't see anyone checking that 'reason' (or
> 'shutdown_code') is SHUTDOWN_soft_reset. In other words, the guest can
> do 'xen_reboot(23)' or 'xen_reboot(154)'. Or, it seems, even
> 'xen_reboot(SHUTDOWN_reboot)'? The only value it shouldn't be is
> SHUTDOWN_suspend.

Xen hypervisor doesn't analyzie the reason, it is being analyzed by the
toolstack (XEN_DOMCTL_getdomaininfo returns this info encoded in flags
with XEN_DOMINF_shutdownshift shift).

-- 
  Vitaly

Re: [PATCH] xen/pvhvm: add soft reset on kexec/kdump support

[Boris Ostrovsky]

On 09/29/2015 07:39 AM, Vitaly Kuznetsov wrote:
> Boris Ostrovsky <boris.ostrovsky@oracle.com> writes:
>
>> On 09/25/2015 03:35 PM, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Sep 25, 2015 at 03:19:57PM -0400, Boris Ostrovsky wrote:
>>>> On 09/25/2015 03:01 PM, Konrad Rzeszutek Wilk wrote:
>>>>> On Fri, Sep 25, 2015 at 01:17:40PM -0400, Boris Ostrovsky wrote:
>>>>>> On 09/25/2015 12:07 PM, Vitaly Kuznetsov wrote:
>>>>>>>> Also, I am not sure I see how this new op will be used in the
>>>>>>>> hypervisor --- currently AFAICS it is only processed under
>>>>>>>> is_hardware_domain(). Are there other patches that will support HVM
>>>>>>>> guests?
>>>>>>> Please see my Xen series:
>>>>>>> http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00547.html
>>>>>>> (last 'full' submission).
>>>>>>>
>>>>>>> All patches from my 'toolstack-assisted approach to PVHVM guest kexec'
>>>>>>> are already merged to xen.git (first 10 are already in 'master' and the
>>>>>>> last one is in 'staging').
>>>>>>>
>>>>>> OK, so I was looking at the right tree. Then I don't understand how
>>>>>> SHUTDOWN_soft_reset would be reached for a non-privileged domain. The only
>>>>>> path that I see is
>>>>>>
>>>>>>       domain_shutdown()
>>>>>>       {
>>>>>>           ...
>>>>>>           if ( is_hardware_domain(d) )
>>>>>>               hwdom_shutdown(reason);
>>>>>>           ...
>>>>>>       }
>>>>>>
>>>>>> Is there another path to handle this op?
>>>>> Yes:
>>>>>    e1bd9812966de9a16f30a58e7162b80bd6af361b libxc: support XEN_DOMCTL_soft_reset operation
>>>>> and
>>>>>    c57e6ebd8c3e490e353e68d96abec1dad01e72f5 (lib)xl: soft reset support
>>>>>
>>>> That's toolstack issuing hypercalls from dom0.
>>>>
>>>> I am asking about (non-privileged) guest itself calling SCHEDOP_shutdown.
>>> The hypervisor ends up calling:
>>> __domain_finalise_shutdown which sends an VIRQ_DOM_EXC to dom0 which
>>> makes the toolstack do all of those operations.
>> OK. But the I don't see anyone checking that 'reason' (or
>> 'shutdown_code') is SHUTDOWN_soft_reset. In other words, the guest can
>> do 'xen_reboot(23)' or 'xen_reboot(154)'. Or, it seems, even
>> 'xen_reboot(SHUTDOWN_reboot)'? The only value it shouldn't be is
>> SHUTDOWN_suspend.
> Xen hypervisor doesn't analyzie the reason, it is being analyzed by the
> toolstack (XEN_DOMCTL_getdomaininfo returns this info encoded in flags
> with XEN_DOMINF_shutdownshift shift).

Thanks.

My problem was that I didn't realize that toolstack generates macros for 
shutdown reasons from libxl_types.idl and I couldn't see who looks for 
SHUTDOWN_soft_reset.

-boris