All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 03/19] xen/arm: While a domain is suspended put its watchdogs on pause
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
  2022-10-07 10:32 ` [PATCH 01/19] xen/arm: Implement PSCI system suspend Mykyta Poturai
  2022-10-07 10:32 ` [PATCH 02/19] watchdog: Introduce a separate struct for watchdog timers Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-10-07 10:32 ` [PATCH 04/19] xen/arm: Trigger Xen suspend when Dom0 completes suspend Mykyta Poturai
                   ` (15 subsequent siblings)
  18 siblings, 0 replies; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Andrew Cooper,
	George Dunlap, Ian Jackson, Jan Beulich, Wei Liu, Dario Faggioli,
	Saeed Nowshadi, Mykyta Poturai

From: Mirela Simonovic <mirela.simonovic@aggios.com>

While a domain is suspended its watchdogs must be paused. Otherwise,
if the domain stays in the suspend state for a longer period of time
compared to the watchdog period, the domain would be shutdown on resume.
Proper solution to this problem is to stop (suspend) the watchdog timers
after the domain suspends and to restart (resume) the watchdog timers
before the domain resumes. The suspend/resume of watchdog timers is done
in Xen and is invisible to the guests.
Just before the domain starts resuming the watchdog timers are programmed
with a new expire value. The new expire value is set according to the
last offset provided by the last hypercall before suspend was triggered,
effectively restarting the timer.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/arch/arm/suspend.c  |  4 ++++
 xen/common/sched/core.c | 36 ++++++++++++++++++++++++++++++++++++
 xen/include/xen/sched.h |  7 +++++++
 3 files changed, 47 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 987ba6ac11..d19545744f 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -128,6 +128,7 @@ void vcpu_resume(struct vcpu *v)
 
     /* Initialize VCPU registers */
     arch_set_info_guest(v, &ctxt);
+    watchdog_domain_resume(v->domain);
     clear_bit(_VPF_suspended, &v->pause_flags);
 }
 
@@ -162,6 +163,9 @@ int32_t domain_suspend(register_t epoint, register_t cid)
      */
     vcpu_suspend(epoint, cid);
 
+    /* Disable watchdogs of this domain */
+    watchdog_domain_suspend(d);
+
     /*
      * The calling domain is suspended by blocking its last running VCPU. If an
      * event is pending the domain will resume right away (VCPU will not block,
diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
index 0ecb41cfe1..ebed0ec49e 100644
--- a/xen/common/sched/core.c
+++ b/xen/common/sched/core.c
@@ -1616,6 +1616,42 @@ void watchdog_domain_destroy(struct domain *d)
         kill_timer(&d->watchdog_timer[i].timer);
 }
 
+void watchdog_domain_suspend(struct domain *d)
+{
+    unsigned int i;
+
+    spin_lock(&d->watchdog_lock);
+
+    for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
+    {
+        if ( test_bit(i, &d->watchdog_inuse_map) )
+        {
+            stop_timer(&d->watchdog_timer[i].timer);
+        }
+    }
+
+    spin_unlock(&d->watchdog_lock);
+}
+
+void watchdog_domain_resume(struct domain *d)
+{
+    unsigned int i;
+
+    spin_lock(&d->watchdog_lock);
+
+    for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
+    {
+        if ( test_bit(i, &d->watchdog_inuse_map) )
+        {
+            set_timer(&d->watchdog_timer[i].timer,
+                      NOW() + SECONDS(d->watchdog_timer[i].timeout));
+        }
+    }
+
+    spin_unlock(&d->watchdog_lock);
+}
+
+
 /*
  * Pin a vcpu temporarily to a specific CPU (or restore old pinning state if
  * cpu is NR_CPUS).
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 7a4aef4c17..00a939aa01 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -1025,6 +1025,13 @@ void scheduler_disable(void);
 void watchdog_domain_init(struct domain *d);
 void watchdog_domain_destroy(struct domain *d);
 
+/*
+ * Suspend/resume watchdogs of domain (while the domain is suspended its
+ * watchdogs should be on pause)
+ */
+void watchdog_domain_suspend(struct domain *d);
+void watchdog_domain_resume(struct domain *d);
+
 /*
  * Use this check when the following are both true:
  *  - Using this feature or interface requires full access to the hardware
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 01/19] xen/arm: Implement PSCI system suspend
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-10-07 10:59   ` Julien Grall
  2022-10-08 22:00   ` Volodymyr Babchuk
  2022-10-07 10:32 ` [PATCH 02/19] watchdog: Introduce a separate struct for watchdog timers Mykyta Poturai
                   ` (17 subsequent siblings)
  18 siblings, 2 replies; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Andrew Cooper,
	George Dunlap, Ian Jackson, Jan Beulich, Wei Liu, Dario Faggioli,
	Saeed Nowshadi, Mykyta Poturai

From: Mirela Simonovic <mirela.simonovic@aggios.com>

The implementation consists of:
-Adding PSCI system suspend call as new PSCI function
-Trapping PSCI system_suspend HVC
-Implementing PSCI system suspend call (virtual interface that allows
 guests to suspend themselves)

The PSCI system suspend should be called by a guest from its boot
VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
are left enabled by the guest are assumed to be its wake-up interrupts.
Therefore, a wake-up interrupt triggers the resume of the guest. Guest
should resume regardless of the state of Xen (suspended or not).

When a guest calls PSCI system suspend the respective domain will be
suspended if the following conditions are met:
1) Given resume entry point is not invalid
2) Other (if any) VCPUs of the calling guest are hot-unplugged

If the conditions above are met the calling domain is labeled as
suspended and the calling VCPU is blocked. If nothing else wouldn't
be done the suspended domain would resume from the place where it
called PSCI system suspend. This is expected if processing of the PSCI
system suspend call fails. However, in the case of success the calling
guest should resume (continue execution after the wake-up) from the entry
point which is given as the first argument of the PSCI system suspend
call. In addition to the entry point, the guest expects to start within
the environment whose state matches the state after reset. This means
that the guest should find reset register values, MMU disabled, etc.
Thereby, the context of VCPU should be 'reset' (as if the system is
comming out of reset), the program counter should contain entry point,
which is 1st argument, and r0/x0 should contain context ID which is 2nd
argument of PSCI system suspend call. The context of VCPU is set during
resume path, to prevent it being overwritten by ctxt_switch_from after
vcpu is blocked and scheduled out.

VCPU is marked as suspended with _VPF_suspended flag. A suspended domain
will resume after the Xen receives an interrupt which is targeted to the
domain, unblocks the domain's VCPU, and schedules it in. During the
vcpu_unblock execution the VCPU is checked for VPF_suspended flag. If
the flag is present, the context of that VCPU gets cleared and entry
point/cid are set.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mykyta Poturai <mykyta.poturai@epam.com>
---
 xen/arch/arm/Makefile            |   1 +
 xen/arch/arm/domain.c            |   4 +
 xen/arch/arm/suspend.c           | 182 +++++++++++++++++++++++++++++++
 xen/arch/arm/vpsci.c             |  28 +++++
 xen/common/sched/core.c          |   8 ++
 xen/include/asm-arm/domain.h     |   3 +
 xen/include/asm-arm/perfc_defn.h |   1 +
 xen/include/asm-arm/psci.h       |   2 +
 xen/include/asm-arm/suspend.h    |  17 +++
 xen/include/xen/sched.h          |   3 +
 10 files changed, 249 insertions(+)
 create mode 100644 xen/arch/arm/suspend.c
 create mode 100644 xen/include/asm-arm/suspend.h

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index b5913c9d39..07dbbd99a3 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -49,6 +49,7 @@ obj-y += setup.o
 obj-y += shutdown.o
 obj-y += smp.o
 obj-y += smpboot.o
+obj-y += suspend.o
 obj-y += sysctl.o
 obj-y += time.o
 obj-y += traps.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 23c8b345d4..4110154bda 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -40,6 +40,8 @@
 #include <asm/vtimer.h>
 #include <asm/vscmi.h>
 
+#include <public/sched.h>
+
 #include "vpci.h"
 #include "vuart.h"
 
@@ -101,6 +103,8 @@ static void ctxt_switch_from(struct vcpu *p)
     if ( is_idle_vcpu(p) )
         return;
 
+    /* VCPU's context should not be saved if its domain is suspended */
+
     p2m_save_state(p);
 
     /* CP 15 */
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
new file mode 100644
index 0000000000..987ba6ac11
--- /dev/null
+++ b/xen/arch/arm/suspend.c
@@ -0,0 +1,182 @@
+/*
+ * Copyright (C) 2022 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include <xen/sched.h>
+#include <asm/cpufeature.h>
+#include <asm/event.h>
+#include <asm/psci.h>
+#include <asm/suspend.h>
+#include <public/sched.h>
+
+struct cpu_context cpu_context;
+
+/* Reset values of VCPU architecture specific registers */
+static void vcpu_arch_reset(struct vcpu *v)
+{
+    v->arch.ttbr0 = 0;
+    v->arch.ttbr1 = 0;
+    v->arch.ttbcr = 0;
+
+    v->arch.csselr = 0;
+    v->arch.cpacr = 0;
+    v->arch.contextidr = 0;
+    v->arch.tpidr_el0 = 0;
+    v->arch.tpidrro_el0 = 0;
+    v->arch.tpidr_el1 = 0;
+    v->arch.vbar = 0;
+    v->arch.dacr = 0;
+    v->arch.par = 0;
+#if defined(CONFIG_ARM_32)
+    v->arch.mair0 = 0;
+    v->arch.mair1 = 0;
+    v->arch.amair0 = 0;
+    v->arch.amair1 = 0;
+#else
+    v->arch.mair = 0;
+    v->arch.amair = 0;
+#endif
+    /* Fault Status */
+#if defined(CONFIG_ARM_32)
+    v->arch.dfar = 0;
+    v->arch.ifar = 0;
+    v->arch.dfsr = 0;
+#elif defined(CONFIG_ARM_64)
+    v->arch.far = 0;
+    v->arch.esr = 0;
+#endif
+
+    v->arch.ifsr  = 0;
+    v->arch.afsr0 = 0;
+    v->arch.afsr1 = 0;
+
+#ifdef CONFIG_ARM_32
+    v->arch.joscr = 0;
+    v->arch.jmcr = 0;
+#endif
+
+    if ( is_32bit_domain(v->domain) && cpu_has_thumbee )
+    {
+        v->arch.teecr = 0;
+        v->arch.teehbr = 0;
+    }
+}
+
+
+static void vcpu_suspend(register_t epoint, register_t cid)
+{
+    struct vcpu *v = current;
+
+    v->arch.suspend_ep = epoint;
+    v->arch.suspend_cid = cid;
+    set_bit(_VPF_suspended, &v->pause_flags);
+    return;
+}
+
+/*
+ * This function sets the context of current VCPU to the state which is expected
+ * by the guest on resume. The expected VCPU state is:
+ * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
+ * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
+ * 3) All other general purpose and system registers should have reset values
+ */
+void vcpu_resume(struct vcpu *v)
+{
+
+    struct vcpu_guest_context ctxt;
+
+    /* Make sure that VCPU guest regs are zeroed */
+    memset(&ctxt, 0, sizeof(ctxt));
+
+    /* Set non-zero values to the registers prior to copying */
+    ctxt.user_regs.pc64 = (u64)v->arch.suspend_ep;
+
+    if ( is_32bit_domain(v->domain) )
+    {
+        ctxt.user_regs.r0_usr = v->arch.suspend_cid;
+        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
+
+        /* Thumb set is allowed only for 32-bit domain */
+        if ( v->arch.suspend_ep & 1 )
+        {
+            ctxt.user_regs.cpsr |= PSR_THUMB;
+            ctxt.user_regs.pc64 &= ~(u64)1;
+        }
+    }
+#ifdef CONFIG_ARM_64
+    else
+    {
+        ctxt.user_regs.x0 = v->arch.suspend_cid;
+        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;
+    }
+#endif
+    ctxt.sctlr = SCTLR_GUEST_INIT;
+    ctxt.flags = VGCF_online;
+
+    /* Reset architecture specific registers */
+    vcpu_arch_reset(v);
+
+    /* Initialize VCPU registers */
+    arch_set_info_guest(v, &ctxt);
+    clear_bit(_VPF_suspended, &v->pause_flags);
+}
+
+int32_t domain_suspend(register_t epoint, register_t cid)
+{
+    struct vcpu *v;
+    struct domain *d = current->domain;
+    bool is_thumb = epoint & 1;
+
+    dprintk(XENLOG_DEBUG,
+            "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
+            d->domain_id, epoint, cid);
+
+    /* THUMB set is not allowed with 64-bit domain */
+    if ( is_64bit_domain(d) && is_thumb )
+        return PSCI_INVALID_ADDRESS;
+
+    /* Ensure that all CPUs other than the calling one are offline */
+    for_each_vcpu ( d, v )
+    {
+        if ( v != current && is_vcpu_online(v) )
+            return PSCI_DENIED;
+    }
+
+    //TODO: add support for suspending from any VCPU
+    if (current->vcpu_id != 0)
+        return PSCI_DENIED;
+
+    /*
+     * Prepare the calling VCPU for suspend (reset its context, save entry point
+     * into pc and context ID into r0/x0 as specified by PSCI SYSTEM_SUSPEND)
+     */
+    vcpu_suspend(epoint, cid);
+
+    /*
+     * The calling domain is suspended by blocking its last running VCPU. If an
+     * event is pending the domain will resume right away (VCPU will not block,
+     * but when scheduled in it will resume from the given entry point).
+     */
+    vcpu_block_unless_event_pending(current);
+
+    return PSCI_SUCCESS;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vpsci.c b/xen/arch/arm/vpsci.c
index c1e250be59..f4e6e92873 100644
--- a/xen/arch/arm/vpsci.c
+++ b/xen/arch/arm/vpsci.c
@@ -18,6 +18,7 @@
 #include <asm/vgic.h>
 #include <asm/vpsci.h>
 #include <asm/event.h>
+#include <asm/suspend.h>
 
 #include <public/sched.h>
 
@@ -208,6 +209,11 @@ static void do_psci_0_2_system_reset(void)
     domain_shutdown(d,SHUTDOWN_reboot);
 }
 
+static int32_t do_psci_1_0_system_suspend(register_t epoint, register_t cid)
+{
+    return domain_suspend(epoint, cid);
+}
+
 static int32_t do_psci_1_0_features(uint32_t psci_func_id)
 {
     /* /!\ Ordered by function ID and not name */
@@ -225,6 +231,8 @@ static int32_t do_psci_1_0_features(uint32_t psci_func_id)
     case PSCI_0_2_FN32_SYSTEM_OFF:
     case PSCI_0_2_FN32_SYSTEM_RESET:
     case PSCI_1_0_FN32_PSCI_FEATURES:
+    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
+    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
     case ARM_SMCCC_VERSION_FID:
         return 0;
     default:
@@ -355,6 +363,26 @@ bool do_vpsci_0_2_call(struct cpu_user_regs *regs, uint32_t fid)
         return true;
     }
 
+    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
+    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
+    {
+        register_t epoint = PSCI_ARG(regs,1);
+        register_t cid = PSCI_ARG(regs,2);
+        register_t ret;
+
+        perfc_incr(vpsci_system_suspend);
+        /* Set the result to PSCI_SUCCESS if the call fails.
+         * Otherwise preserve the context_id in x0. For now 
+         * we don't support the case where the system is suspended
+         * to a shallower level and PSCI_SUCCESS is returned to the 
+         * caller.
+         */
+        ret = do_psci_1_0_system_suspend(epoint, cid);
+        if ( ret != PSCI_SUCCESS )
+            PSCI_SET_RESULT(regs, ret);
+        return true;
+    }
+
     default:
         return false;
     }
diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
index 8f4b1ca10d..4e1ea62c44 100644
--- a/xen/common/sched/core.c
+++ b/xen/common/sched/core.c
@@ -38,6 +38,10 @@
 #include <xsm/xsm.h>
 #include <xen/err.h>
 
+#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
+#include <asm/suspend.h>
+#endif
+
 #include "private.h"
 
 #ifdef CONFIG_XEN_GUEST
@@ -957,6 +961,10 @@ void vcpu_unblock(struct vcpu *v)
 {
     if ( !test_and_clear_bit(_VPF_blocked, &v->pause_flags) )
         return;
+#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
+    if ( test_bit(_VPF_suspended, &v->pause_flags) )
+        vcpu_resume(v);
+#endif
 
     /* Polling period ends when a VCPU is unblocked. */
     if ( unlikely(v->poll_evtchn != 0) )
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 413e5a2a18..715841e0b5 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -203,6 +203,9 @@ struct arch_vcpu
     struct vtimer virt_timer;
     bool   vtimer_initialized;
 
+    register_t suspend_ep;
+    register_t suspend_cid;
+
     /*
      * The full P2M may require some cleaning (e.g when emulation
      * set/way). As the action can take a long time, it requires
diff --git a/xen/include/asm-arm/perfc_defn.h b/xen/include/asm-arm/perfc_defn.h
index 31f071222b..d71e91a5e4 100644
--- a/xen/include/asm-arm/perfc_defn.h
+++ b/xen/include/asm-arm/perfc_defn.h
@@ -33,6 +33,7 @@ PERFCOUNTER(vpsci_system_reset,        "vpsci: system_reset")
 PERFCOUNTER(vpsci_cpu_suspend,         "vpsci: cpu_suspend")
 PERFCOUNTER(vpsci_cpu_affinity_info,   "vpsci: cpu_affinity_info")
 PERFCOUNTER(vpsci_features,            "vpsci: features")
+PERFCOUNTER(vpsci_system_suspend,      "vpsci: system_suspend")
 
 PERFCOUNTER(vcpu_kick,                 "vcpu: notify other vcpu")
 
diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
index 832f77afff..26462d0c47 100644
--- a/xen/include/asm-arm/psci.h
+++ b/xen/include/asm-arm/psci.h
@@ -43,10 +43,12 @@ void call_psci_system_reset(void);
 #define PSCI_0_2_FN32_SYSTEM_OFF          PSCI_0_2_FN32(8)
 #define PSCI_0_2_FN32_SYSTEM_RESET        PSCI_0_2_FN32(9)
 #define PSCI_1_0_FN32_PSCI_FEATURES       PSCI_0_2_FN32(10)
+#define PSCI_1_0_FN32_SYSTEM_SUSPEND      PSCI_0_2_FN32(14)
 
 #define PSCI_0_2_FN64_CPU_SUSPEND         PSCI_0_2_FN64(1)
 #define PSCI_0_2_FN64_CPU_ON              PSCI_0_2_FN64(3)
 #define PSCI_0_2_FN64_AFFINITY_INFO       PSCI_0_2_FN64(4)
+#define PSCI_1_0_FN64_SYSTEM_SUSPEND      PSCI_0_2_FN64(14)
 
 /* PSCI v0.2 affinity level state returned by AFFINITY_INFO */
 #define PSCI_0_2_AFFINITY_LEVEL_ON      0
diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
new file mode 100644
index 0000000000..fbaa414f0c
--- /dev/null
+++ b/xen/include/asm-arm/suspend.h
@@ -0,0 +1,17 @@
+#ifndef __ASM_ARM_SUSPEND_H__
+#define __ASM_ARM_SUSPEND_H__
+
+int32_t domain_suspend(register_t epoint, register_t cid);
+void vcpu_resume(struct vcpu *v);
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 3b4ed3a2ab..b2f6d1af28 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -903,6 +903,9 @@ static inline struct domain *next_domain_in_cpupool(
 /* VCPU is parked. */
 #define _VPF_parked          8
 #define VPF_parked           (1UL<<_VPF_parked)
+/* VCPU is suspended */
+#define _VPF_suspended       9
+#define VPF_suspended        (1UL<<_VPF_suspended)
 
 static inline bool vcpu_runnable(const struct vcpu *v)
 {
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 02/19] watchdog: Introduce a separate struct for watchdog timers
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
  2022-10-07 10:32 ` [PATCH 01/19] xen/arm: Implement PSCI system suspend Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-10-07 10:32 ` [PATCH 03/19] xen/arm: While a domain is suspended put its watchdogs on pause Mykyta Poturai
                   ` (16 subsequent siblings)
  18 siblings, 0 replies; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mykyta Poturai, Andrew Cooper, George Dunlap, Ian Jackson,
	Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu,
	Dario Faggioli

This change is needed to properly implement the suspend/resume actions
for the watchdog timers. To be able to restart watchdog timer after
suspend we need to remember their frequency somewhere. To not bloat the
struct timer a new struct watchdog_timer is introduced, containing the
original timer and the last set timeout.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/common/keyhandler.c    |  2 +-
 xen/common/sched/core.c    | 11 ++++++-----
 xen/include/xen/sched.h    |  3 ++-
 xen/include/xen/watchdog.h |  6 ++++++
 4 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/xen/common/keyhandler.c b/xen/common/keyhandler.c
index 8b9f378371..e9bd48fa5b 100644
--- a/xen/common/keyhandler.c
+++ b/xen/common/keyhandler.c
@@ -289,7 +289,7 @@ static void dump_domains(unsigned char key)
         for ( i = 0 ; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
             if ( test_bit(i, &d->watchdog_inuse_map) )
                 printk("    watchdog %d expires in %d seconds\n",
-                       i, (u32)((d->watchdog_timer[i].expires - NOW()) >> 30));
+                       i, (u32)((d->watchdog_timer[i].timer.expires - NOW()) >> 30));
 
         arch_dump_domain_info(d);
 
diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
index 4e1ea62c44..0ecb41cfe1 100644
--- a/xen/common/sched/core.c
+++ b/xen/common/sched/core.c
@@ -1567,7 +1567,8 @@ static long domain_watchdog(struct domain *d, uint32_t id, uint32_t timeout)
         {
             if ( test_and_set_bit(id, &d->watchdog_inuse_map) )
                 continue;
-            set_timer(&d->watchdog_timer[id], NOW() + SECONDS(timeout));
+            d->watchdog_timer[id].timeout = timeout;
+            set_timer(&d->watchdog_timer[id].timer, NOW() + SECONDS(timeout));
             break;
         }
         spin_unlock(&d->watchdog_lock);
@@ -1583,12 +1584,12 @@ static long domain_watchdog(struct domain *d, uint32_t id, uint32_t timeout)
 
     if ( timeout == 0 )
     {
-        stop_timer(&d->watchdog_timer[id]);
+        stop_timer(&d->watchdog_timer[id].timer);
         clear_bit(id, &d->watchdog_inuse_map);
     }
     else
     {
-        set_timer(&d->watchdog_timer[id], NOW() + SECONDS(timeout));
+        set_timer(&d->watchdog_timer[id].timer, NOW() + SECONDS(timeout));
     }
 
     spin_unlock(&d->watchdog_lock);
@@ -1604,7 +1605,7 @@ void watchdog_domain_init(struct domain *d)
     d->watchdog_inuse_map = 0;
 
     for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
-        init_timer(&d->watchdog_timer[i], domain_watchdog_timeout, d, 0);
+        init_timer(&d->watchdog_timer[i].timer, domain_watchdog_timeout, d, 0);
 }
 
 void watchdog_domain_destroy(struct domain *d)
@@ -1612,7 +1613,7 @@ void watchdog_domain_destroy(struct domain *d)
     unsigned int i;
 
     for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
-        kill_timer(&d->watchdog_timer[i]);
+        kill_timer(&d->watchdog_timer[i].timer);
 }
 
 /*
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index b2f6d1af28..7a4aef4c17 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -24,6 +24,7 @@
 #include <asm/atomic.h>
 #include <xen/vpci.h>
 #include <xen/wait.h>
+#include <xen/watchdog.h>
 #include <public/xen.h>
 #include <public/domctl.h>
 #include <public/sysctl.h>
@@ -517,7 +518,7 @@ struct domain
 #define NR_DOMAIN_WATCHDOG_TIMERS 2
     spinlock_t watchdog_lock;
     uint32_t watchdog_inuse_map;
-    struct timer watchdog_timer[NR_DOMAIN_WATCHDOG_TIMERS];
+    struct watchdog_timer watchdog_timer[NR_DOMAIN_WATCHDOG_TIMERS];
 
     struct rcu_head rcu;
 
diff --git a/xen/include/xen/watchdog.h b/xen/include/xen/watchdog.h
index 86fde0884a..3398dcba37 100644
--- a/xen/include/xen/watchdog.h
+++ b/xen/include/xen/watchdog.h
@@ -8,6 +8,12 @@
 #define __XEN_WATCHDOG_H__
 
 #include <xen/types.h>
+#include <xen/timer.h>
+
+struct watchdog_timer {
+    struct timer timer;
+    uint32_t timeout;
+};
 
 #ifdef CONFIG_WATCHDOG
 
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 04/19] xen/arm: Trigger Xen suspend when Dom0 completes suspend
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (2 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 03/19] xen/arm: While a domain is suspended put its watchdogs on pause Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-10-07 10:32 ` [PATCH 05/19] xen/x86: Move freeze/thaw_domains into common files Mykyta Poturai
                   ` (14 subsequent siblings)
  18 siblings, 0 replies; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi,
	Mykyta Poturai

From: Mirela Simonovic <mirela.simonovic@aggios.com>

When Dom0 finalizes its suspend procedure the suspend of Xen is
triggered by calling system_suspend(). Dom0 finalizes the suspend from
its boot core (VCPU#0), which could be mapped to any physical CPU,
i.e. the system_suspend() function could be executed by any physical
CPU. Since Xen suspend procedure has to be run by the boot CPU
(non-boot CPUs will be disabled at some point in suspend procedure),
system_suspend() execution has to continue on CPU#0.

Though it is not clearly stated that the PSCI suspend call should be
issued from cpu0, we assume that it is to simplify the process. This
assumption is based on a fact that most platforms call some form of
disable_nonboot_cpus routine before issuing the PSCI suspend call.

When the system_suspend() returns 0, it means that the system was
suspended and it is coming out of the resume procedure. Regardless
of the system_suspend() return value, after this function returns
Xen is fully functional, and its state, including all devices and data
structures, matches the state prior to calling system_suspend().
The status is returned by system_suspend() for debugging/logging
purposes and function prototype compatibility.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index d19545744f..b09bf319d0 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -132,11 +132,20 @@ void vcpu_resume(struct vcpu *v)
     clear_bit(_VPF_suspended, &v->pause_flags);
 }
 
+/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
+static long system_suspend(void *data)
+{
+    BUG_ON(system_state != SYS_STATE_active);
+
+    return -ENOSYS;
+}
+
 int32_t domain_suspend(register_t epoint, register_t cid)
 {
     struct vcpu *v;
     struct domain *d = current->domain;
     bool is_thumb = epoint & 1;
+    int status;
 
     dprintk(XENLOG_DEBUG,
             "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
@@ -173,6 +182,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
      */
     vcpu_block_unless_event_pending(current);
 
+    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
+    if ( is_hardware_domain(d) )
+    {
+        /*
+         * system_suspend should be called when Dom0 finalizes the suspend
+         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
+         * be mapped to any PCPU (this function could be executed by any PCPU).
+         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
+         * PCPUs will be disabled during the suspend).
+         */
+        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
+        /*
+         * If an error happened, there is nothing that needs to be done here
+         * because the system_suspend always returns in fully functional state
+         * no matter what the outcome of suspend procedure is. If the system
+         * suspended successfully the function will return 0 after the resume.
+         * Otherwise, if an error is returned it means Xen did not suspended,
+         * but it is still in the same state as if the system_suspend was never
+         * called. We dump a debug message in case of an error for debugging/
+         * logging purpose.
+         */
+        if ( status )
+            dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
+    }
+
     return PSCI_SUCCESS;
 }
 
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 05/19] xen/x86: Move freeze/thaw_domains into common files
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (3 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 04/19] xen/arm: Trigger Xen suspend when Dom0 completes suspend Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 20:36   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 09/19] xen/arm: Implement GIC suspend/resume functions (gicv2 only) Mykyta Poturai
                   ` (13 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Andrew Cooper, George Dunlap, Ian Jackson,
	Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu,
	Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

These functions will be reused by suspend/resume support for ARM.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/common/domain.c     | 29 +++++++++++++++++++++++++++++
 xen/include/xen/sched.h |  3 +++
 2 files changed, 32 insertions(+)

diff --git a/xen/common/domain.c b/xen/common/domain.c
index 56d47dd664..5e5894c468 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -1884,6 +1884,35 @@ int continue_hypercall_on_cpu(
     return 0;
 }
 
+
+void freeze_domains(void)
+{
+    struct domain *d;
+
+    rcu_read_lock(&domlist_read_lock);
+    /*
+     * Note that we iterate in order of domain-id. Hence we will pause dom0
+     * first which is required for correctness (as only dom0 can add domains to
+     * the domain list). Otherwise we could miss concurrently-created domains.
+     */
+    for_each_domain ( d )
+        domain_pause(d);
+    rcu_read_unlock(&domlist_read_lock);
+}
+
+void thaw_domains(void)
+{
+    struct domain *d;
+
+    rcu_read_lock(&domlist_read_lock);
+    for_each_domain ( d )
+    {
+        restore_vcpu_affinity(d);
+        domain_unpause(d);
+    }
+    rcu_read_unlock(&domlist_read_lock);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 00a939aa01..c8ddfdd51c 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -978,6 +978,9 @@ static inline struct vcpu *domain_vcpu(const struct domain *d,
     return vcpu_id >= d->max_vcpus ? NULL : d->vcpu[idx];
 }
 
+void freeze_domains(void);
+void thaw_domains(void);
+
 void cpu_init(void);
 
 /*
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 09/19] xen/arm: Implement GIC suspend/resume functions (gicv2 only)
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (4 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 05/19] xen/x86: Move freeze/thaw_domains into common files Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 21:18   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 08/19] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume Mykyta Poturai
                   ` (12 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

System suspend may lead to a state where GIC would be powered down.
Therefore, Xen should save/restore the context of GIC on suspend/resume.
Note that the context consists of states of registers which are
controlled by the hypervisor. Other GIC registers which are accessible
by guests are saved/restored on context switch.
Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
the GIC.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/gic-v2.c     | 138 +++++++++++++++++++++++++++++++++++++-
 xen/arch/arm/gic.c        |  27 ++++++++
 xen/include/asm-arm/gic.h |   8 +++
 3 files changed, 172 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index b2adc8ec9a..b077b66d92 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -123,6 +123,23 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
 /* Maximum cpu interface per GIC */
 #define NR_GIC_CPU_IF 8
 
+/* GICv2 registers to be saved/restored on system suspend/resume */
+struct gicv2_context {
+    /* GICC context */
+    uint32_t gicc_ctlr;
+    uint32_t gicc_pmr;
+    uint32_t gicc_bpr;
+    /* GICD context */
+    uint32_t gicd_ctlr;
+    uint32_t *gicd_isenabler;
+    uint32_t *gicd_isactiver;
+    uint32_t *gicd_ipriorityr;
+    uint32_t *gicd_itargetsr;
+    uint32_t *gicd_icfgr;
+};
+
+static struct gicv2_context gicv2_context;
+
 static inline void writeb_gicd(uint8_t val, unsigned int offset)
 {
     writeb_relaxed(val, gicv2.map_dbase + offset);
@@ -1251,6 +1268,40 @@ static void __init gicv2_acpi_init(void)
 static void __init gicv2_acpi_init(void) { }
 #endif
 
+static int gicv2_alloc_context(struct gicv2_context *gc)
+{
+    uint32_t n = gicv2_info.nr_lines;
+
+    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+    if ( !gc->gicd_isenabler )
+        goto err_free;
+
+    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+    if ( !gc->gicd_isactiver )
+        goto err_free;
+
+    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+    if ( !gc->gicd_itargetsr )
+        goto err_free;
+
+    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+    if ( !gc->gicd_ipriorityr )
+        goto err_free;
+
+    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
+    if ( !gc->gicd_icfgr )
+        goto err_free;
+
+    return 0;
+err_free:
+    xfree(gc->gicd_icfgr);
+    xfree(gc->gicd_ipriorityr);
+    xfree(gc->gicd_itargetsr);
+    xfree(gc->gicd_isactiver);
+    xfree(gc->gicd_isenabler);
+    return -ENOMEM;
+}
+
 static int __init gicv2_init(void)
 {
     uint32_t aliased_offset = 0;
@@ -1318,7 +1369,8 @@ static int __init gicv2_init(void)
 
     spin_unlock(&gicv2.lock);
 
-    return 0;
+    /* Allocate memory to be used for saving GIC context during the suspend */
+    return gicv2_alloc_context(&gicv2_context);
 }
 
 static void gicv2_do_LPI(unsigned int lpi)
@@ -1327,6 +1379,88 @@ static void gicv2_do_LPI(unsigned int lpi)
     BUG();
 }
 
+static int gicv2_suspend(void)
+{
+    int i;
+
+    ASSERT(gicv2_context.gicd_isenabler);
+    ASSERT(gicv2_context.gicd_isactiver);
+    ASSERT(gicv2_context.gicd_ipriorityr);
+    ASSERT(gicv2_context.gicd_itargetsr);
+    ASSERT(gicv2_context.gicd_icfgr);
+
+    /* Save GICC configuration */
+    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
+    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
+    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
+
+    /* Save GICD configuration */
+    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
+        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
+
+    return 0;
+}
+
+static void gicv2_resume(void)
+{
+    int i;
+
+    ASSERT(gicv2_context.gicd_isenabler);
+    ASSERT(gicv2_context.gicd_isactiver);
+    ASSERT(gicv2_context.gicd_ipriorityr);
+    ASSERT(gicv2_context.gicd_itargetsr);
+    ASSERT(gicv2_context.gicd_icfgr);
+
+    /* Disable CPU interface and distributor */
+    writel_gicc(0, GICC_CTLR);
+    writel_gicd(0, GICD_CTLR);
+
+    /* Restore GICD configuration */
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ ) {
+        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
+        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
+    }
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ ) {
+        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
+        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);
+    }
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
+        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
+
+    /* Make sure all registers are restored and enable distributor */
+    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
+
+    /* Restore GIC CPU interface configuration */
+    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
+    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
+
+    /* Enable GIC CPU interface */
+    writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | GICC_CTL_EOI,
+                GICC_CTLR);
+}
+
 const static struct gic_hw_operations gicv2_ops = {
     .info                = &gicv2_info,
     .init                = gicv2_init,
@@ -1361,6 +1495,8 @@ const static struct gic_hw_operations gicv2_ops = {
     .map_hwdom_extra_mappings = gicv2_map_hwdown_extra_mappings,
     .iomem_deny_access   = gicv2_iomem_deny_access,
     .do_LPI              = gicv2_do_LPI,
+    .suspend             = gicv2_suspend,
+    .resume              = gicv2_resume,
 };
 
 /* Set up the GIC */
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 3b0331b538..e9feb1fd3b 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -467,6 +467,33 @@ int gic_iomem_deny_access(const struct domain *d)
     return gic_hw_ops->iomem_deny_access(d);
 }
 
+int gic_suspend(void)
+{
+    /* Must be called by boot CPU#0 with interrupts disabled */
+    ASSERT(!local_irq_is_enabled());
+    ASSERT(!smp_processor_id());
+
+    if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
+        return -ENOSYS;
+
+    gic_hw_ops->suspend();
+
+    return 0;
+}
+
+void gic_resume(void)
+{
+    /*
+     * Must be called by boot CPU#0 with interrupts disabled after gic_suspend
+     * has returned successfully.
+     */
+    ASSERT(!local_irq_is_enabled());
+    ASSERT(!smp_processor_id());
+    ASSERT(gic_hw_ops->resume);
+
+    gic_hw_ops->resume();
+}
+
 static int cpu_gic_callback(struct notifier_block *nfb,
                             unsigned long action,
                             void *hcpu)
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index c7f0c343d1..113e39460d 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -275,6 +275,10 @@ extern int gicv_setup(struct domain *d);
 extern void gic_save_state(struct vcpu *v);
 extern void gic_restore_state(struct vcpu *v);
 
+/* Suspend/resume */
+extern int gic_suspend(void);
+extern void gic_resume(void);
+
 /* SGI (AKA IPIs) */
 enum gic_sgi {
     GIC_SGI_EVENT_CHECK,
@@ -390,6 +394,10 @@ struct gic_hw_operations {
     int (*iomem_deny_access)(const struct domain *d);
     /* Handle LPIs, which require special handling */
     void (*do_LPI)(unsigned int lpi);
+    /* Save GIC configuration due to the system suspend */
+    int (*suspend)(void);
+    /* Restore GIC configuration due to the system resume */
+    void (*resume)(void);
 };
 
 extern const struct gic_hw_operations *gic_hw_ops;
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 06/19] xen/arm: Freeze domains on suspend and thaw them on resume
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (6 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 08/19] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 20:47   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 07/19] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume Mykyta Poturai
                   ` (10 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

Freeze and thaw of domains is reused as implemented for x86. In
addition, system_state variable is updated to represent the actual
state of the system.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index b09bf319d0..2b94816b63 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -137,6 +137,14 @@ static long system_suspend(void *data)
 {
     BUG_ON(system_state != SYS_STATE_active);
 
+    system_state = SYS_STATE_suspend;
+    freeze_domains();
+
+    system_state = SYS_STATE_resume;
+
+    thaw_domains();
+    system_state = SYS_STATE_active;
+
     return -ENOSYS;
 }
 
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 07/19] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (7 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 06/19] xen/arm: Freeze domains on suspend and thaw them " Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 21:00   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 10/19] xen/arm: Suspend/resume GIC on system suspend/resume Mykyta Poturai
                   ` (9 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

Non-boot CPUs have to be disabled on suspend and enabled on resume
(hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
CPU_OFF to be called by each non-boot CPU. Depending on the underlying
platform capabilities, this may lead to the physical powering down of
CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
each non-boot CPU).

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 2b94816b63..0784979e4f 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -13,6 +13,7 @@
  */
 
 #include <xen/sched.h>
+#include <xen/cpu.h>
 #include <asm/cpufeature.h>
 #include <asm/event.h>
 #include <asm/psci.h>
@@ -135,17 +136,29 @@ void vcpu_resume(struct vcpu *v)
 /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
 static long system_suspend(void *data)
 {
+    int status;
+
     BUG_ON(system_state != SYS_STATE_active);
 
     system_state = SYS_STATE_suspend;
     freeze_domains();
 
+    status = disable_nonboot_cpus();
+    if ( status )
+    {
+        system_state = SYS_STATE_resume;
+        goto resume_nonboot_cpus;
+    }
+
     system_state = SYS_STATE_resume;
 
+resume_nonboot_cpus:
+    enable_nonboot_cpus();
     thaw_domains();
     system_state = SYS_STATE_active;
+    dsb(sy);
 
-    return -ENOSYS;
+    return status;
 }
 
 int32_t domain_suspend(register_t epoint, register_t cid)
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 08/19] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (5 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 09/19] xen/arm: Implement GIC suspend/resume functions (gicv2 only) Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-10-07 10:32 ` [PATCH 06/19] xen/arm: Freeze domains on suspend and thaw them " Mykyta Poturai
                   ` (11 subsequent siblings)
  18 siblings, 0 replies; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

The rcu_barrier() has to be added to ensure that the per cpu area is
freed before a non-boot CPU tries to initialize it (_free_percpu_area()
has to be called before the init_percpu_area()). This scenario occurs
when non-boot CPUs are hot-unplugged on suspend and hotplugged on resume.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 0784979e4f..0c16cfc750 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -153,6 +153,7 @@ static long system_suspend(void *data)
     system_state = SYS_STATE_resume;
 
 resume_nonboot_cpus:
+    rcu_barrier();
     enable_nonboot_cpus();
     thaw_domains();
     system_state = SYS_STATE_active;
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 10/19] xen/arm: Suspend/resume GIC on system suspend/resume
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (8 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 07/19] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 21:20   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 12/19] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface) Mykyta Poturai
                   ` (8 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

GIC state is saved on system suspend by calling gic_suspend
(this function does not change current state of the GIC but only
saves the values of configuration registers).
The state of GIC has to be restored by calling gic_resume, but only
if the gic_suspend has succeeded. If gic_suspend fails, we'll just
restore interrupts configuration and abort suspend.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/gic.c     |  4 +---
 xen/arch/arm/suspend.c | 14 ++++++++++++++
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index e9feb1fd3b..ef90664929 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -476,9 +476,7 @@ int gic_suspend(void)
     if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
         return -ENOSYS;
 
-    gic_hw_ops->suspend();
-
-    return 0;
+    return gic_hw_ops->suspend();
 }
 
 void gic_resume(void)
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 0c16cfc750..b94a6df86d 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -137,6 +137,7 @@ void vcpu_resume(struct vcpu *v)
 static long system_suspend(void *data)
 {
     int status;
+    unsigned long flags;
 
     BUG_ON(system_state != SYS_STATE_active);
 
@@ -150,8 +151,21 @@ static long system_suspend(void *data)
         goto resume_nonboot_cpus;
     }
 
+    local_irq_save(flags);
+    status = gic_suspend();
+    if ( status )
+    {
+        system_state = SYS_STATE_resume;
+        goto resume_irqs;
+    }
+
     system_state = SYS_STATE_resume;
 
+    gic_resume();
+
+resume_irqs:
+    local_irq_restore(flags);
+
 resume_nonboot_cpus:
     rcu_barrier();
     enable_nonboot_cpus();
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 11/19] xen/arm: Suspend/resume timer interrupt generation
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (10 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 12/19] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface) Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 21:29   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 15/19] xen/arm: Resume Dom0 after Xen resumes Mykyta Poturai
                   ` (6 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

Timer interrupts have to be disabled while the system is in suspend.
Otherwise, a timer interrupt would fire and wake-up the system.
Suspending the timer interrupts consists of disabling physical EL1
and EL2 timers. The resume consists only of raising timer softirq,
which will trigger the generic timer code to reprogram the EL2 timer
as needed. Enabling of EL1 physical timer will be triggered by an
entity which uses it.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c     |  4 ++++
 xen/arch/arm/time.c        | 22 ++++++++++++++++++++++
 xen/include/asm-arm/time.h |  3 +++
 3 files changed, 29 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index b94a6df86d..05c43ce502 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -151,6 +151,8 @@ static long system_suspend(void *data)
         goto resume_nonboot_cpus;
     }
 
+    time_suspend();
+
     local_irq_save(flags);
     status = gic_suspend();
     if ( status )
@@ -166,6 +168,8 @@ static long system_suspend(void *data)
 resume_irqs:
     local_irq_restore(flags);
 
+    time_resume();
+
 resume_nonboot_cpus:
     rcu_barrier();
     enable_nonboot_cpus();
diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index dec53b5f7d..ca54bcfe68 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -363,6 +363,28 @@ void domain_set_time_offset(struct domain *d, int64_t time_offset_seconds)
     /* XXX update guest visible wallclock time */
 }
 
+void time_suspend(void)
+{
+    /* Disable physical EL1 timer */
+    WRITE_SYSREG(0, CNTP_CTL_EL0);
+
+    /* Disable hypervisor's timer */
+    WRITE_SYSREG(0, CNTHP_CTL_EL2);
+    isb();
+}
+
+void time_resume(void)
+{
+    /*
+     * Raising timer softirq will trigger generic timer code to reprogram_timer
+     * with the correct timeout value (which is not known here). There is no
+     * need to do anything else in order to recover the time keeping from power
+     * down, because the system counter is not affected by the power down (it
+     * resides out of the ARM's cluster in an always-on part of the SoC).
+     */
+    raise_softirq(TIMER_SOFTIRQ);
+}
+
 static int cpu_time_callback(struct notifier_block *nfb,
                              unsigned long action,
                              void *hcpu)
diff --git a/xen/include/asm-arm/time.h b/xen/include/asm-arm/time.h
index 4b401c1110..ded93b490a 100644
--- a/xen/include/asm-arm/time.h
+++ b/xen/include/asm-arm/time.h
@@ -107,6 +107,9 @@ void preinit_xen_time(void);
 
 void force_update_vcpu_system_time(struct vcpu *v);
 
+void time_suspend(void);
+void time_resume(void);
+
 #endif /* __ARM_TIME_H__ */
 /*
  * Local variables:
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 12/19] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (9 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 10/19] xen/arm: Suspend/resume GIC on system suspend/resume Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 21:34   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 11/19] xen/arm: Suspend/resume timer interrupt generation Mykyta Poturai
                   ` (7 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi,
	Mykyta Poturai

From: Mirela Simonovic <mirela.simonovic@aggios.com>

PSCI system suspend function shall be invoked to finalize Xen suspend
procedure. Resume entry point, which needs to be passed via 1st argument
of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
is just a placeholder that will be implemented in assembly. Context ID,
which is 2nd argument of system suspend PSCI call, is unused, as in Linux.

Update: moved hyp_resume to head.S to place it near the rest of the
start code

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/arch/arm/arm64/head.S     |  3 +++
 xen/arch/arm/psci.c           | 16 ++++++++++++++++
 xen/arch/arm/suspend.c        |  4 ++++
 xen/include/asm-arm/psci.h    |  1 +
 xen/include/asm-arm/suspend.h |  1 +
 5 files changed, 25 insertions(+)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index aa1f88c764..8857955699 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -957,6 +957,9 @@ ENTRY(efi_xen_start)
         b     real_start_efi
 ENDPROC(efi_xen_start)
 
+ENTRY(hyp_resume)
+        b .
+
 /*
  * Local variables:
  * mode: ASM
diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
index 0c90c2305c..43a67eb345 100644
--- a/xen/arch/arm/psci.c
+++ b/xen/arch/arm/psci.c
@@ -25,6 +25,7 @@
 #include <asm/cpufeature.h>
 #include <asm/psci.h>
 #include <asm/acpi.h>
+#include <asm/suspend.h>
 
 /*
  * While a 64-bit OS can make calls with SMC32 calling conventions, for
@@ -68,6 +69,21 @@ void call_psci_cpu_off(void)
     }
 }
 
+int call_psci_system_suspend(void)
+{
+#ifdef CONFIG_ARM_64
+    struct arm_smccc_res res;
+
+    /* 2nd argument (context ID) is not used */
+    arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), &res);
+
+    return PSCI_RET(res);
+#else
+    /* not supported */
+    return 1;
+#endif
+}
+
 void call_psci_system_off(void)
 {
     if ( psci_ver > PSCI_VERSION(0, 1) )
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 05c43ce502..a0258befc9 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -161,6 +161,10 @@ static long system_suspend(void *data)
         goto resume_irqs;
     }
 
+    status = call_psci_system_suspend();
+    if ( status )
+        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
+
     system_state = SYS_STATE_resume;
 
     gic_resume();
diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
index 26462d0c47..9f6116a224 100644
--- a/xen/include/asm-arm/psci.h
+++ b/xen/include/asm-arm/psci.h
@@ -20,6 +20,7 @@ extern uint32_t psci_ver;
 
 int psci_init(void);
 int call_psci_cpu_on(int cpu);
+int call_psci_system_suspend(void);
 void call_psci_cpu_off(void);
 void call_psci_system_off(void);
 void call_psci_system_reset(void);
diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
index fbaa414f0c..29420e27fa 100644
--- a/xen/include/asm-arm/suspend.h
+++ b/xen/include/asm-arm/suspend.h
@@ -3,6 +3,7 @@
 
 int32_t domain_suspend(register_t epoint, register_t cid);
 void vcpu_resume(struct vcpu *v);
+void hyp_resume(void);
 
 #endif
 
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 13/19] xen/arm: Resume memory management on Xen resume
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (12 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 15/19] xen/arm: Resume Dom0 after Xen resumes Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 21:52   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 16/19] xen/arm: Suspend/resume console on Xen suspend/resume Mykyta Poturai
                   ` (4 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi,
	Mykyta Poturai

From: Mirela Simonovic <mirela.simonovic@aggios.com>

The MMU needs to be enabled in the resume flow before the context
can be restored (we need to be able to access the context data by
virtual address in order to restore it). The configuration of system
registers prior to branching to the routine that sets up the page
tables is copied from xen/arch/arm/arm64/head.S.
After the MMU is enabled, the content of TTBR0_EL2 is changed to
point to init_ttbr (page tables used at runtime).

At boot the init_ttbr variable is updated when a secondary CPU is
hotplugged. In the scenario where there is only one physical CPU in
the system, the init_ttbr would not be initialized for the use in
resume flow. To get the variable initialized in all scenarios in this
patch we add that the boot CPU updates init_ttbr after it sets the
page tables for runtime.

After the memory management is resumed, the SCTLR_WXN in SCTLR_EL2
has to be set in order to configure that a mapping cannot be both
writable and executable (this was configured prior to suspend).
This is done using an existing function (mmu_init_secondary_cpu).

Update: moved hyp_resume to head.S to place it near the rest of the
start code

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/arch/arm/arm64/entry.S      |  2 ++
 xen/arch/arm/arm64/head.S       | 30 ++++++++++++++++++++++++++++++
 xen/arch/arm/mm.c               |  1 +
 xen/arch/arm/suspend.c          |  6 ++++++
 xen/include/asm-arm/processor.h | 22 ++++++++++++++++++++++
 5 files changed, 61 insertions(+)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index fc3811ad0a..f49f1daf46 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -1,4 +1,6 @@
 #include <asm/current.h>
+#include <asm/macros.h>
+#include <asm/page.h>
 #include <asm/regs.h>
 #include <asm/alternative.h>
 #include <asm/smccc.h>
diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 8857955699..82d83214dc 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -958,6 +958,36 @@ ENTRY(efi_xen_start)
 ENDPROC(efi_xen_start)
 
 ENTRY(hyp_resume)
+        msr   DAIFSet, 0xf           /* Disable all interrupts */
+
+        tlbi  alle2
+        dsb   sy                     /* Ensure completion of TLB flush */
+        isb
+
+        ldr   x0, =start
+        adr   x19, start             /* x19 := paddr (start) */
+        sub   x20, x19, x0           /* x20 := phys-offset */
+
+        mov   x22, #0                /* x22 := is_secondary_cpu */
+
+        bl    check_cpu_mode
+        bl    cpu_init
+        bl    create_page_tables
+        bl    enable_mmu
+
+        ldr   x0, =mmu_resumed      /* Explicit vaddr, not RIP-relative */
+        br    x0                    /* Get a proper vaddr into PC */
+
+mmu_resumed:
+        ldr   x4, =init_ttbr         /* VA of TTBR0_EL2 stashed by CPU 0 */
+        ldr   x4, [x4]               /* Actual value */
+        dsb   sy
+        msr   TTBR0_EL2, x4
+        dsb   sy
+        isb
+        tlbi  alle2
+        dsb   sy                     /* Ensure completion of TLB flush */
+        isb
         b .
 
 /*
diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index eea926d823..29cdaff3bf 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -708,6 +708,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
     switch_ttbr(ttbr);
 
     xen_pt_enforce_wnx();
+    init_secondary_pagetables(0);
 
 #ifdef CONFIG_ARM_32
     per_cpu(xen_pgtable, 0) = cpu0_pgtable;
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index a0258befc9..aa5ee4714b 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -167,6 +167,12 @@ static long system_suspend(void *data)
 
     system_state = SYS_STATE_resume;
 
+    /*
+     * SCTLR_WXN needs to be set to configure that a mapping cannot be both
+     * writable and executable. This is done by mmu_init_secondary_cpu.
+     */
+    mmu_init_secondary_cpu();
+
     gic_resume();
 
 resume_irqs:
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index 8ab2940f68..ecf97f1ab4 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -133,6 +133,28 @@
 #define TTBCR_PD1       (_AC(1,U)<<5)
 
 /* SCTLR System Control Register. */
+/* HSCTLR is a subset of this. */
+#define SCTLR_TE        (_AC(1,U)<<30)
+#define SCTLR_AFE       (_AC(1,U)<<29)
+#define SCTLR_TRE       (_AC(1,U)<<28)
+#define SCTLR_NMFI      (_AC(1,U)<<27)
+#define SCTLR_EE        (_AC(1,U)<<25)
+#define SCTLR_VE        (_AC(1,U)<<24)
+#define SCTLR_U         (_AC(1,U)<<22)
+#define SCTLR_FI        (_AC(1,U)<<21)
+#define SCTLR_WXN       (_AC(1,U)<<19)
+#define SCTLR_HA        (_AC(1,U)<<17)
+#define SCTLR_RR        (_AC(1,U)<<14)
+#define SCTLR_V         (_AC(1,U)<<13)
+#define SCTLR_I         (_AC(1,U)<<12)
+#define SCTLR_Z         (_AC(1,U)<<11)
+#define SCTLR_SW        (_AC(1,U)<<10)
+#define SCTLR_B         (_AC(1,U)<<7)
+#define SCTLR_C         (_AC(1,U)<<2)
+#define SCTLR_A         (_AC(1,U)<<1)
+#define SCTLR_M         (_AC(1,U)<<0)
+
+#define HSCTLR_BASE     _AC(0x30c51878,U)
 
 /* Bits specific to SCTLR_EL1 for Arm32 */
 
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 15/19] xen/arm: Resume Dom0 after Xen resumes
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (11 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 11/19] xen/arm: Suspend/resume timer interrupt generation Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-10-07 10:32 ` [PATCH 13/19] xen/arm: Resume memory management on Xen resume Mykyta Poturai
                   ` (5 subsequent siblings)
  18 siblings, 0 replies; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

The resume of Dom0 should always follow Xen's resume. This is
done by unblocking the first vCPU of Dom0.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 13d1aba658..4a690eac3b 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -206,6 +206,9 @@ resume_nonboot_cpus:
     system_state = SYS_STATE_active;
     dsb(sy);
 
+    /* Wake-up hardware domain (should always resume after Xen) */
+    vcpu_unblock(hardware_domain->vcpu[0]);
+
     return status;
 }
 
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 14/19] xen/arm: Save/restore context on suspend/resume
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (14 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 16/19] xen/arm: Suspend/resume console on Xen suspend/resume Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 22:09   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 17/19] xen: don't free percpu areas during suspend Mykyta Poturai
                   ` (2 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi,
	Mykyta Poturai

From: Mirela Simonovic <mirela.simonovic@aggios.com>

The context of CPU general purpose and system control registers
has to be saved on suspend and restored on resume. This is
implemented in hyp_suspend and before the return from hyp_resume
function. The hyp_suspend is invoked just before the PSCI system
suspend call is issued to the ATF. The hyp_suspend has to return a
non-zero value so that the calling 'if' statement evaluates to true,
causing the system suspend to be invoked. Upon the resume, context
saved on suspend will be restored, including the link register.
Therefore, after restoring the context the control flow will
return to the address pointed by the saved link register, which
is the place from which the hyp_suspend was called. To ensure
that the calling 'if' statement doesn't again evaluate to true
and initiate system suspend, hyp_resume has to return a zero value
after restoring the context.
Note that the order of saving register context into cpu_context
structure has to match the order of restoring.
Since the suspend/resume is supported only for arm64, we define
a null cpu_context structure so arm32 could compile.

Update: moved hyp_resume to head.S to place it near the rest of the
start code

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/arch/arm/arm64/head.S     | 90 ++++++++++++++++++++++++++++++++++-
 xen/arch/arm/suspend.c        | 25 ++++++++--
 xen/include/asm-arm/suspend.h | 22 +++++++++
 3 files changed, 133 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 82d83214dc..e2c46a864c 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -957,6 +957,53 @@ ENTRY(efi_xen_start)
         b     real_start_efi
 ENDPROC(efi_xen_start)
 
+/*
+ * void hyp_suspend(struct cpu_context *ptr)
+ *
+ * x0 - pointer to the storage where callee's context will be saved
+ *
+ * CPU context saved here will be restored on resume in hyp_resume function.
+ * hyp_suspend shall return a non-zero value. Upon restoring context
+ * hyp_resume shall return value zero instead. From C code that invokes
+ * hyp_suspend, the return value is interpreted to determine whether the context
+ * is saved (hyp_suspend) or restored (hyp_resume).
+ */
+ENTRY(hyp_suspend)
+        /* Store callee-saved registers */
+        stp     x19, x20, [x0], #16
+        stp     x21, x22, [x0], #16
+        stp     x23, x24, [x0], #16
+        stp     x25, x26, [x0], #16
+        stp     x27, x28, [x0], #16
+        stp     x29, lr, [x0], #16
+
+        /* Store stack-pointer */
+        mov     x2, sp
+        str     x2, [x0], #8
+
+        /* Store system control registers */
+        mrs     x2, VBAR_EL2
+        str     x2, [x0], #8
+        mrs     x2, VTCR_EL2
+        str     x2, [x0], #8
+        mrs     x2, VTTBR_EL2
+        str     x2, [x0], #8
+        mrs     x2, TPIDR_EL2
+        str     x2, [x0], #8
+        mrs     x2, MDCR_EL2
+        str     x2, [x0], #8
+        mrs     x2, HSTR_EL2
+        str     x2, [x0], #8
+        mrs     x2, CPTR_EL2
+        str     x2, [x0], #8
+        mrs     x2, HCR_EL2
+        str     x2, [x0], #8
+
+        /* hyp_suspend must return a non-zero value */
+        mov     x0, #1
+        ret
+
+
 ENTRY(hyp_resume)
         msr   DAIFSet, 0xf           /* Disable all interrupts */
 
@@ -988,7 +1035,48 @@ mmu_resumed:
         tlbi  alle2
         dsb   sy                     /* Ensure completion of TLB flush */
         isb
-        b .
+
+        /* Now we can access the cpu_context, so restore the context here */
+        ldr     x0, =cpu_context
+
+        /* Restore callee-saved registers */
+        ldp     x19, x20, [x0], #16
+        ldp     x21, x22, [x0], #16
+        ldp     x23, x24, [x0], #16
+        ldp     x25, x26, [x0], #16
+        ldp     x27, x28, [x0], #16
+        ldp     x29, lr, [x0], #16
+
+        /* Restore stack pointer */
+        ldr     x2, [x0], #8
+        mov     sp, x2
+
+        /* Restore system control registers */
+        ldr     x2, [x0], #8
+        msr     VBAR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     VTCR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     VTTBR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     TPIDR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     MDCR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     HSTR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     CPTR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     HCR_EL2, x2
+        isb
+
+        /* Since context is restored return from this function will appear as
+         * return from hyp_suspend. To distinguish a return from hyp_suspend
+         * which is called upon finalizing the suspend, as opposed to return
+         * from this function which executes on resume, we need to return zero
+         * value here. */
+        mov x0, #0
+        ret
 
 /*
  * Local variables:
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index aa5ee4714b..13d1aba658 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -133,6 +133,11 @@ void vcpu_resume(struct vcpu *v)
     clear_bit(_VPF_suspended, &v->pause_flags);
 }
 
+#ifndef CONFIG_ARM_64
+/* not supported on ARM_32 */
+int32_t hyp_suspend(struct cpu_context *ptr) { return 1; }
+#endif
+
 /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
 static long system_suspend(void *data)
 {
@@ -161,9 +166,23 @@ static long system_suspend(void *data)
         goto resume_irqs;
     }
 
-    status = call_psci_system_suspend();
-    if ( status )
-        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
+    if ( hyp_suspend(&cpu_context) )
+    {
+        status = call_psci_system_suspend();
+        /*
+         * If suspend is finalized properly by above system suspend PSCI call,
+         * the code below in this 'if' branch will never execute. Execution
+         * will continue from hyp_resume which is the hypervisor's resume point.
+         * In hyp_resume CPU context will be restored and since link-register is
+         * restored as well, it will appear to return from hyp_suspend. The
+         * difference in returning from hyp_suspend on system suspend versus
+         * resume is in function's return value: on suspend, the return value is
+         * a non-zero value, on resume it is zero. That is why the control flow
+         * will not re-enter this 'if' branch on resume.
+         */
+        if ( status )
+            dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
+    }
 
     system_state = SYS_STATE_resume;
 
diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
index 29420e27fa..70dbf4e208 100644
--- a/xen/include/asm-arm/suspend.h
+++ b/xen/include/asm-arm/suspend.h
@@ -1,9 +1,31 @@
 #ifndef __ASM_ARM_SUSPEND_H__
 #define __ASM_ARM_SUSPEND_H__
 
+#ifdef CONFIG_ARM_64
+struct cpu_context {
+    uint64_t callee_regs[12];
+    uint64_t sp;
+    uint64_t vbar_el2;
+    uint64_t vtcr_el2;
+    uint64_t vttbr_el2;
+    uint64_t tpidr_el2;
+    uint64_t mdcr_el2;
+    uint64_t hstr_el2;
+    uint64_t cptr_el2;
+    uint64_t hcr_el2;
+} __aligned(16);
+#else
+struct cpu_context {
+    uint8_t pad;
+};
+#endif
+
+extern struct cpu_context cpu_context;
+
 int32_t domain_suspend(register_t epoint, register_t cid);
 void vcpu_resume(struct vcpu *v);
 void hyp_resume(void);
+int32_t hyp_suspend(struct cpu_context *ptr);
 
 #endif
 
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 16/19] xen/arm: Suspend/resume console on Xen suspend/resume
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (13 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 13/19] xen/arm: Resume memory management on Xen resume Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 22:12   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 14/19] xen/arm: Save/restore context on suspend/resume Mykyta Poturai
                   ` (3 subsequent siblings)
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Saeed Nowshadi

From: Mirela Simonovic <mirela.simonovic@aggios.com>

This is done using generic console_suspend/resume functions that cause
uart driver specific suspend/resume handlers to be called for each
initialized port (if the port has suspend/resume driver handlers
implemented).

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 4a690eac3b..cf3aab0099 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -14,6 +14,7 @@
 
 #include <xen/sched.h>
 #include <xen/cpu.h>
+#include <xen/console.h>
 #include <asm/cpufeature.h>
 #include <asm/event.h>
 #include <asm/psci.h>
@@ -166,6 +167,15 @@ static long system_suspend(void *data)
         goto resume_irqs;
     }
 
+    dprintk(XENLOG_DEBUG, "Suspend\n");
+    status = console_suspend();
+    if ( status )
+    {
+        dprintk(XENLOG_ERR, "Failed to suspend the console, err=%d\n", status);
+        system_state = SYS_STATE_resume;
+        goto resume_console;
+    }
+
     if ( hyp_suspend(&cpu_context) )
     {
         status = call_psci_system_suspend();
@@ -192,6 +202,10 @@ static long system_suspend(void *data)
      */
     mmu_init_secondary_cpu();
 
+resume_console:
+    console_resume();
+    dprintk(XENLOG_DEBUG, "Resume\n");
+
     gic_resume();
 
 resume_irqs:
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 17/19] xen: don't free percpu areas during suspend
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (15 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 14/19] xen/arm: Save/restore context on suspend/resume Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-10-07 11:17   ` Juergen Gross
  2022-10-07 10:32 ` [PATCH 19/19] Fix misleading indentation gcc warning Mykyta Poturai
  2022-10-07 10:32 ` [PATCH 18/19] timers: Don't migrate timers during suspend Mykyta Poturai
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Juergen Gross, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis, Dario Faggioli

From: Juergen Gross <jgross@suse.com>

Instead of freeing percpu areas during suspend and allocating them
again when resuming keep them. Only free an area in case a cpu didn't
come up again when resuming.

It should be noted that there is a potential change in behaviour as
the percpu areas are no longer zeroed out during suspend/resume. While
I have checked the called cpu notifier hooks to cope with that there
might be some well hidden dependency on the previous behaviour. OTOH
a component not registering itself for cpu down/up and expecting to
see a zeroed percpu variable after suspend/resume is kind of broken
already. And the opposite case, where a component is not registered
to be called for cpu down/up and is not expecting a percpu variable
suddenly to be zero due to suspend/resume is much more probable,
especially as the suspend/resume functionality seems not to be tested
that often.

Signed-off-by: Juergen Gross <jgross@suse.com>
Reviewed-by: Dario Faggioli <dfaggioli@suse.com>
---
 xen/arch/arm/percpu.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/xen/arch/arm/percpu.c b/xen/arch/arm/percpu.c
index 25442c48fe..0642705544 100644
--- a/xen/arch/arm/percpu.c
+++ b/xen/arch/arm/percpu.c
@@ -58,10 +58,13 @@ static int cpu_percpu_callback(
     switch ( action )
     {
     case CPU_UP_PREPARE:
+      if ( system_state != SYS_STATE_resume )
         rc = init_percpu_area(cpu);
         break;
     case CPU_UP_CANCELED:
     case CPU_DEAD:
+    case CPU_RESUME_FAILED:
+      if ( system_state != SYS_STATE_suspend )
         free_percpu_area(cpu);
         break;
     default:
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 18/19] timers: Don't migrate timers during suspend
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (17 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 19/19] Fix misleading indentation gcc warning Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 22:19   ` Julien Grall
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Mykyta Poturai, Andrew Cooper, George Dunlap, Ian Jackson,
	Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu

Migrating timers during suspend causes Dom0 to freeze after resume.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/common/timer.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/xen/common/timer.c b/xen/common/timer.c
index 1bb265ceea..52d4f72a76 100644
--- a/xen/common/timer.c
+++ b/xen/common/timer.c
@@ -658,7 +658,8 @@ static int cpu_callback(
     case CPU_UP_CANCELED:
     case CPU_DEAD:
     case CPU_RESUME_FAILED:
-        migrate_timers_from_cpu(cpu);
+        if ( system_state != SYS_STATE_suspend )
+            migrate_timers_from_cpu(cpu);
 
         if ( !park_offline_cpus && system_state != SYS_STATE_suspend )
             free_percpu_timers(cpu);
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 19/19] Fix misleading indentation gcc warning
       [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
                   ` (16 preceding siblings ...)
  2022-10-07 10:32 ` [PATCH 17/19] xen: don't free percpu areas during suspend Mykyta Poturai
@ 2022-10-07 10:32 ` Mykyta Poturai
  2022-12-06 22:17   ` Julien Grall
  2022-10-07 10:32 ` [PATCH 18/19] timers: Don't migrate timers during suspend Mykyta Poturai
  18 siblings, 1 reply; 37+ messages in thread
From: Mykyta Poturai @ 2022-10-07 10:32 UTC (permalink / raw)
  To: xen-devel
  Cc: Oleksandr Andrushchenko, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Bertrand Marquis

From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>

percpu.c: In function 'cpu_percpu_callback':
percpu.c:61:7: error: this 'if' clause does not guard... [-Werror=misleading-indentation]
       if ( system_state != SYS_STATE_resume )
       ^~
percpu.c:63:9: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
         break;
         ^~~~~
percpu.c:67:7: error: this 'if' clause does not guard... [-Werror=misleading-indentation]
       if ( system_state != SYS_STATE_suspend )
       ^~
percpu.c:69:9: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
         break;
         ^~~~~

Fixes: c3109b76d967 ("xen: don't free percpu areas during suspend")

Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
---
 xen/arch/arm/percpu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/percpu.c b/xen/arch/arm/percpu.c
index 0642705544..9eab1e6d7c 100644
--- a/xen/arch/arm/percpu.c
+++ b/xen/arch/arm/percpu.c
@@ -60,13 +60,13 @@ static int cpu_percpu_callback(
     case CPU_UP_PREPARE:
       if ( system_state != SYS_STATE_resume )
         rc = init_percpu_area(cpu);
-        break;
+      break;
     case CPU_UP_CANCELED:
     case CPU_DEAD:
     case CPU_RESUME_FAILED:
       if ( system_state != SYS_STATE_suspend )
         free_percpu_area(cpu);
-        break;
+      break;
     default:
         break;
     }
-- 
2.37.1


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* Re: [PATCH 01/19] xen/arm: Implement PSCI system suspend
  2022-10-07 10:32 ` [PATCH 01/19] xen/arm: Implement PSCI system suspend Mykyta Poturai
@ 2022-10-07 10:59   ` Julien Grall
  2022-10-08 22:00   ` Volodymyr Babchuk
  1 sibling, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-10-07 10:59 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Andrew Cooper, George Dunlap, Ian Jackson,
	Jan Beulich, Wei Liu, Dario Faggioli, Saeed Nowshadi,
	Mykyta Poturai

Hi Mykyta,

I couldn't find any cover letter for this series in neither my inbox nor 
xen-devel. Can you provide one explain the goal of this series (you seem 
to have a mix of domain suspend and host suspend)? If it is also based 
on an existing series, then it would be nice to mention it (this would 
be helpful to look at the existings and check if they were addressed).

Also, we are currently in the middle of the code freeze. So the review 
will likely be quite slow until 4.17 is out and I caught up with the 
rest of my backlog.

That said from a brief look, I see you included a patch that was merged 
around 4.12. So is this series actually based on staging?

If not, then the first step would be to rebase this series to the latest 
staging.

Cheers,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> The implementation consists of:
> -Adding PSCI system suspend call as new PSCI function
> -Trapping PSCI system_suspend HVC
> -Implementing PSCI system suspend call (virtual interface that allows
>   guests to suspend themselves)
> 
> The PSCI system suspend should be called by a guest from its boot
> VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
> CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
> are left enabled by the guest are assumed to be its wake-up interrupts.
> Therefore, a wake-up interrupt triggers the resume of the guest. Guest
> should resume regardless of the state of Xen (suspended or not).
> 
> When a guest calls PSCI system suspend the respective domain will be
> suspended if the following conditions are met:
> 1) Given resume entry point is not invalid
> 2) Other (if any) VCPUs of the calling guest are hot-unplugged
> 
> If the conditions above are met the calling domain is labeled as
> suspended and the calling VCPU is blocked. If nothing else wouldn't
> be done the suspended domain would resume from the place where it
> called PSCI system suspend. This is expected if processing of the PSCI
> system suspend call fails. However, in the case of success the calling
> guest should resume (continue execution after the wake-up) from the entry
> point which is given as the first argument of the PSCI system suspend
> call. In addition to the entry point, the guest expects to start within
> the environment whose state matches the state after reset. This means
> that the guest should find reset register values, MMU disabled, etc.
> Thereby, the context of VCPU should be 'reset' (as if the system is
> comming out of reset), the program counter should contain entry point,
> which is 1st argument, and r0/x0 should contain context ID which is 2nd
> argument of PSCI system suspend call. The context of VCPU is set during
> resume path, to prevent it being overwritten by ctxt_switch_from after
> vcpu is blocked and scheduled out.
> 
> VCPU is marked as suspended with _VPF_suspended flag. A suspended domain
> will resume after the Xen receives an interrupt which is targeted to the
> domain, unblocks the domain's VCPU, and schedules it in. During the
> vcpu_unblock execution the VCPU is checked for VPF_suspended flag. If
> the flag is present, the context of that VCPU gets cleared and entry
> point/cid are set.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> Signed-off-by: Mykyta Poturai <mykyta.poturai@epam.com>
> ---
>   xen/arch/arm/Makefile            |   1 +
>   xen/arch/arm/domain.c            |   4 +
>   xen/arch/arm/suspend.c           | 182 +++++++++++++++++++++++++++++++
>   xen/arch/arm/vpsci.c             |  28 +++++
>   xen/common/sched/core.c          |   8 ++
>   xen/include/asm-arm/domain.h     |   3 +
>   xen/include/asm-arm/perfc_defn.h |   1 +
>   xen/include/asm-arm/psci.h       |   2 +
>   xen/include/asm-arm/suspend.h    |  17 +++
>   xen/include/xen/sched.h          |   3 +
>   10 files changed, 249 insertions(+)
>   create mode 100644 xen/arch/arm/suspend.c
>   create mode 100644 xen/include/asm-arm/suspend.h
> 
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index b5913c9d39..07dbbd99a3 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -49,6 +49,7 @@ obj-y += setup.o
>   obj-y += shutdown.o
>   obj-y += smp.o
>   obj-y += smpboot.o
> +obj-y += suspend.o
>   obj-y += sysctl.o
>   obj-y += time.o
>   obj-y += traps.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 23c8b345d4..4110154bda 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -40,6 +40,8 @@
>   #include <asm/vtimer.h>
>   #include <asm/vscmi.h>
>   
> +#include <public/sched.h>
> +
>   #include "vpci.h"
>   #include "vuart.h"
>   
> @@ -101,6 +103,8 @@ static void ctxt_switch_from(struct vcpu *p)
>       if ( is_idle_vcpu(p) )
>           return;
>   
> +    /* VCPU's context should not be saved if its domain is suspended */
> +
>       p2m_save_state(p);
>   
>       /* CP 15 */
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> new file mode 100644
> index 0000000000..987ba6ac11
> --- /dev/null
> +++ b/xen/arch/arm/suspend.c
> @@ -0,0 +1,182 @@
> +/*
> + * Copyright (C) 2022 EPAM Systems Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2.1 only. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + */
> +
> +#include <xen/sched.h>
> +#include <asm/cpufeature.h>
> +#include <asm/event.h>
> +#include <asm/psci.h>
> +#include <asm/suspend.h>
> +#include <public/sched.h>
> +
> +struct cpu_context cpu_context;
> +
> +/* Reset values of VCPU architecture specific registers */
> +static void vcpu_arch_reset(struct vcpu *v)
> +{
> +    v->arch.ttbr0 = 0;
> +    v->arch.ttbr1 = 0;
> +    v->arch.ttbcr = 0;
> +
> +    v->arch.csselr = 0;
> +    v->arch.cpacr = 0;
> +    v->arch.contextidr = 0;
> +    v->arch.tpidr_el0 = 0;
> +    v->arch.tpidrro_el0 = 0;
> +    v->arch.tpidr_el1 = 0;
> +    v->arch.vbar = 0;
> +    v->arch.dacr = 0;
> +    v->arch.par = 0;
> +#if defined(CONFIG_ARM_32)
> +    v->arch.mair0 = 0;
> +    v->arch.mair1 = 0;
> +    v->arch.amair0 = 0;
> +    v->arch.amair1 = 0;
> +#else
> +    v->arch.mair = 0;
> +    v->arch.amair = 0;
> +#endif
> +    /* Fault Status */
> +#if defined(CONFIG_ARM_32)
> +    v->arch.dfar = 0;
> +    v->arch.ifar = 0;
> +    v->arch.dfsr = 0;
> +#elif defined(CONFIG_ARM_64)
> +    v->arch.far = 0;
> +    v->arch.esr = 0;
> +#endif
> +
> +    v->arch.ifsr  = 0;
> +    v->arch.afsr0 = 0;
> +    v->arch.afsr1 = 0;
> +
> +#ifdef CONFIG_ARM_32
> +    v->arch.joscr = 0;
> +    v->arch.jmcr = 0;
> +#endif
> +
> +    if ( is_32bit_domain(v->domain) && cpu_has_thumbee )
> +    {
> +        v->arch.teecr = 0;
> +        v->arch.teehbr = 0;
> +    }
> +}
> +
> +
> +static void vcpu_suspend(register_t epoint, register_t cid)
> +{
> +    struct vcpu *v = current;
> +
> +    v->arch.suspend_ep = epoint;
> +    v->arch.suspend_cid = cid;
> +    set_bit(_VPF_suspended, &v->pause_flags);
> +    return;
> +}
> +
> +/*
> + * This function sets the context of current VCPU to the state which is expected
> + * by the guest on resume. The expected VCPU state is:
> + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
> + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
> + * 3) All other general purpose and system registers should have reset values
> + */
> +void vcpu_resume(struct vcpu *v)
> +{
> +
> +    struct vcpu_guest_context ctxt;
> +
> +    /* Make sure that VCPU guest regs are zeroed */
> +    memset(&ctxt, 0, sizeof(ctxt));
> +
> +    /* Set non-zero values to the registers prior to copying */
> +    ctxt.user_regs.pc64 = (u64)v->arch.suspend_ep;
> +
> +    if ( is_32bit_domain(v->domain) )
> +    {
> +        ctxt.user_regs.r0_usr = v->arch.suspend_cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
> +
> +        /* Thumb set is allowed only for 32-bit domain */
> +        if ( v->arch.suspend_ep & 1 )
> +        {
> +            ctxt.user_regs.cpsr |= PSR_THUMB;
> +            ctxt.user_regs.pc64 &= ~(u64)1;
> +        }
> +    }
> +#ifdef CONFIG_ARM_64
> +    else
> +    {
> +        ctxt.user_regs.x0 = v->arch.suspend_cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;
> +    }
> +#endif
> +    ctxt.sctlr = SCTLR_GUEST_INIT;
> +    ctxt.flags = VGCF_online;
> +
> +    /* Reset architecture specific registers */
> +    vcpu_arch_reset(v);
> +
> +    /* Initialize VCPU registers */
> +    arch_set_info_guest(v, &ctxt);
> +    clear_bit(_VPF_suspended, &v->pause_flags);
> +}
> +
> +int32_t domain_suspend(register_t epoint, register_t cid)
> +{
> +    struct vcpu *v;
> +    struct domain *d = current->domain;
> +    bool is_thumb = epoint & 1;
> +
> +    dprintk(XENLOG_DEBUG,
> +            "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> +            d->domain_id, epoint, cid);
> +
> +    /* THUMB set is not allowed with 64-bit domain */
> +    if ( is_64bit_domain(d) && is_thumb )
> +        return PSCI_INVALID_ADDRESS;
> +
> +    /* Ensure that all CPUs other than the calling one are offline */
> +    for_each_vcpu ( d, v )
> +    {
> +        if ( v != current && is_vcpu_online(v) )
> +            return PSCI_DENIED;
> +    }
> +
> +    //TODO: add support for suspending from any VCPU
> +    if (current->vcpu_id != 0)
> +        return PSCI_DENIED;
> +
> +    /*
> +     * Prepare the calling VCPU for suspend (reset its context, save entry point
> +     * into pc and context ID into r0/x0 as specified by PSCI SYSTEM_SUSPEND)
> +     */
> +    vcpu_suspend(epoint, cid);
> +
> +    /*
> +     * The calling domain is suspended by blocking its last running VCPU. If an
> +     * event is pending the domain will resume right away (VCPU will not block,
> +     * but when scheduled in it will resume from the given entry point).
> +     */
> +    vcpu_block_unless_event_pending(current);
> +
> +    return PSCI_SUCCESS;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vpsci.c b/xen/arch/arm/vpsci.c
> index c1e250be59..f4e6e92873 100644
> --- a/xen/arch/arm/vpsci.c
> +++ b/xen/arch/arm/vpsci.c
> @@ -18,6 +18,7 @@
>   #include <asm/vgic.h>
>   #include <asm/vpsci.h>
>   #include <asm/event.h>
> +#include <asm/suspend.h>
>   
>   #include <public/sched.h>
>   
> @@ -208,6 +209,11 @@ static void do_psci_0_2_system_reset(void)
>       domain_shutdown(d,SHUTDOWN_reboot);
>   }
>   
> +static int32_t do_psci_1_0_system_suspend(register_t epoint, register_t cid)
> +{
> +    return domain_suspend(epoint, cid);
> +}
> +
>   static int32_t do_psci_1_0_features(uint32_t psci_func_id)
>   {
>       /* /!\ Ordered by function ID and not name */
> @@ -225,6 +231,8 @@ static int32_t do_psci_1_0_features(uint32_t psci_func_id)
>       case PSCI_0_2_FN32_SYSTEM_OFF:
>       case PSCI_0_2_FN32_SYSTEM_RESET:
>       case PSCI_1_0_FN32_PSCI_FEATURES:
> +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
>       case ARM_SMCCC_VERSION_FID:
>           return 0;
>       default:
> @@ -355,6 +363,26 @@ bool do_vpsci_0_2_call(struct cpu_user_regs *regs, uint32_t fid)
>           return true;
>       }
>   
> +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
> +    {
> +        register_t epoint = PSCI_ARG(regs,1);
> +        register_t cid = PSCI_ARG(regs,2);
> +        register_t ret;
> +
> +        perfc_incr(vpsci_system_suspend);
> +        /* Set the result to PSCI_SUCCESS if the call fails.
> +         * Otherwise preserve the context_id in x0. For now
> +         * we don't support the case where the system is suspended
> +         * to a shallower level and PSCI_SUCCESS is returned to the
> +         * caller.
> +         */
> +        ret = do_psci_1_0_system_suspend(epoint, cid);
> +        if ( ret != PSCI_SUCCESS )
> +            PSCI_SET_RESULT(regs, ret);
> +        return true;
> +    }
> +
>       default:
>           return false;
>       }
> diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
> index 8f4b1ca10d..4e1ea62c44 100644
> --- a/xen/common/sched/core.c
> +++ b/xen/common/sched/core.c
> @@ -38,6 +38,10 @@
>   #include <xsm/xsm.h>
>   #include <xen/err.h>
>   
> +#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
> +#include <asm/suspend.h>
> +#endif
> +
>   #include "private.h"
>   
>   #ifdef CONFIG_XEN_GUEST
> @@ -957,6 +961,10 @@ void vcpu_unblock(struct vcpu *v)
>   {
>       if ( !test_and_clear_bit(_VPF_blocked, &v->pause_flags) )
>           return;
> +#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
> +    if ( test_bit(_VPF_suspended, &v->pause_flags) )
> +        vcpu_resume(v);
> +#endif
>   
>       /* Polling period ends when a VCPU is unblocked. */
>       if ( unlikely(v->poll_evtchn != 0) )
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 413e5a2a18..715841e0b5 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -203,6 +203,9 @@ struct arch_vcpu
>       struct vtimer virt_timer;
>       bool   vtimer_initialized;
>   
> +    register_t suspend_ep;
> +    register_t suspend_cid;
> +
>       /*
>        * The full P2M may require some cleaning (e.g when emulation
>        * set/way). As the action can take a long time, it requires
> diff --git a/xen/include/asm-arm/perfc_defn.h b/xen/include/asm-arm/perfc_defn.h
> index 31f071222b..d71e91a5e4 100644
> --- a/xen/include/asm-arm/perfc_defn.h
> +++ b/xen/include/asm-arm/perfc_defn.h
> @@ -33,6 +33,7 @@ PERFCOUNTER(vpsci_system_reset,        "vpsci: system_reset")
>   PERFCOUNTER(vpsci_cpu_suspend,         "vpsci: cpu_suspend")
>   PERFCOUNTER(vpsci_cpu_affinity_info,   "vpsci: cpu_affinity_info")
>   PERFCOUNTER(vpsci_features,            "vpsci: features")
> +PERFCOUNTER(vpsci_system_suspend,      "vpsci: system_suspend")
>   
>   PERFCOUNTER(vcpu_kick,                 "vcpu: notify other vcpu")
>   
> diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> index 832f77afff..26462d0c47 100644
> --- a/xen/include/asm-arm/psci.h
> +++ b/xen/include/asm-arm/psci.h
> @@ -43,10 +43,12 @@ void call_psci_system_reset(void);
>   #define PSCI_0_2_FN32_SYSTEM_OFF          PSCI_0_2_FN32(8)
>   #define PSCI_0_2_FN32_SYSTEM_RESET        PSCI_0_2_FN32(9)
>   #define PSCI_1_0_FN32_PSCI_FEATURES       PSCI_0_2_FN32(10)
> +#define PSCI_1_0_FN32_SYSTEM_SUSPEND      PSCI_0_2_FN32(14)
>   
>   #define PSCI_0_2_FN64_CPU_SUSPEND         PSCI_0_2_FN64(1)
>   #define PSCI_0_2_FN64_CPU_ON              PSCI_0_2_FN64(3)
>   #define PSCI_0_2_FN64_AFFINITY_INFO       PSCI_0_2_FN64(4)
> +#define PSCI_1_0_FN64_SYSTEM_SUSPEND      PSCI_0_2_FN64(14)
>   
>   /* PSCI v0.2 affinity level state returned by AFFINITY_INFO */
>   #define PSCI_0_2_AFFINITY_LEVEL_ON      0
> diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> new file mode 100644
> index 0000000000..fbaa414f0c
> --- /dev/null
> +++ b/xen/include/asm-arm/suspend.h
> @@ -0,0 +1,17 @@
> +#ifndef __ASM_ARM_SUSPEND_H__
> +#define __ASM_ARM_SUSPEND_H__
> +
> +int32_t domain_suspend(register_t epoint, register_t cid);
> +void vcpu_resume(struct vcpu *v);
> +
> +#endif
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> index 3b4ed3a2ab..b2f6d1af28 100644
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -903,6 +903,9 @@ static inline struct domain *next_domain_in_cpupool(
>   /* VCPU is parked. */
>   #define _VPF_parked          8
>   #define VPF_parked           (1UL<<_VPF_parked)
> +/* VCPU is suspended */
> +#define _VPF_suspended       9
> +#define VPF_suspended        (1UL<<_VPF_suspended)
>   
>   static inline bool vcpu_runnable(const struct vcpu *v)
>   {

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 17/19] xen: don't free percpu areas during suspend
  2022-10-07 10:32 ` [PATCH 17/19] xen: don't free percpu areas during suspend Mykyta Poturai
@ 2022-10-07 11:17   ` Juergen Gross
  2022-10-21  9:46     ` Dario Faggioli
  0 siblings, 1 reply; 37+ messages in thread
From: Juergen Gross @ 2022-10-07 11:17 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
	Bertrand Marquis, Dario Faggioli


[-- Attachment #1.1.1: Type: text/plain, Size: 1930 bytes --]

On 07.10.22 12:32, Mykyta Poturai wrote:
> From: Juergen Gross <jgross@suse.com>
> 
> Instead of freeing percpu areas during suspend and allocating them
> again when resuming keep them. Only free an area in case a cpu didn't
> come up again when resuming.
> 
> It should be noted that there is a potential change in behaviour as
> the percpu areas are no longer zeroed out during suspend/resume. While
> I have checked the called cpu notifier hooks to cope with that there
> might be some well hidden dependency on the previous behaviour. OTOH
> a component not registering itself for cpu down/up and expecting to
> see a zeroed percpu variable after suspend/resume is kind of broken
> already. And the opposite case, where a component is not registered
> to be called for cpu down/up and is not expecting a percpu variable
> suddenly to be zero due to suspend/resume is much more probable,
> especially as the suspend/resume functionality seems not to be tested
> that often.
> 
> Signed-off-by: Juergen Gross <jgross@suse.com>

I can't remember having written this patch. The one I remember was for
x86.

> Reviewed-by: Dario Faggioli <dfaggioli@suse.com>

I doubt that, reasoning see above.


Juergen

> ---
>   xen/arch/arm/percpu.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/xen/arch/arm/percpu.c b/xen/arch/arm/percpu.c
> index 25442c48fe..0642705544 100644
> --- a/xen/arch/arm/percpu.c
> +++ b/xen/arch/arm/percpu.c
> @@ -58,10 +58,13 @@ static int cpu_percpu_callback(
>       switch ( action )
>       {
>       case CPU_UP_PREPARE:
> +      if ( system_state != SYS_STATE_resume )
>           rc = init_percpu_area(cpu);
>           break;
>       case CPU_UP_CANCELED:
>       case CPU_DEAD:
> +    case CPU_RESUME_FAILED:
> +      if ( system_state != SYS_STATE_suspend )
>           free_percpu_area(cpu);
>           break;
>       default:


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 01/19] xen/arm: Implement PSCI system suspend
  2022-10-07 10:32 ` [PATCH 01/19] xen/arm: Implement PSCI system suspend Mykyta Poturai
  2022-10-07 10:59   ` Julien Grall
@ 2022-10-08 22:00   ` Volodymyr Babchuk
  2022-10-09  9:30     ` Julien Grall
  1 sibling, 1 reply; 37+ messages in thread
From: Volodymyr Babchuk @ 2022-10-08 22:00 UTC (permalink / raw)
  To: Mykyta Poturai
  Cc: xen-devel, Mirela Simonovic, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Andrew Cooper, George Dunlap, Ian Jackson,
	Jan Beulich, Wei Liu, Dario Faggioli, Saeed Nowshadi,
	Mykyta Poturai


Hi Mykyta,

Mykyta Poturai <Mykyta_Poturai@epam.com> writes:

> From: Mirela Simonovic <mirela.simonovic@aggios.com>
>
> The implementation consists of:
> -Adding PSCI system suspend call as new PSCI function
> -Trapping PSCI system_suspend HVC

AFAIK, this can be SMC as well.

> -Implementing PSCI system suspend call (virtual interface that allows
>  guests to suspend themselves)
>
> The PSCI system suspend should be called by a guest from its boot
> VCPU.

Why such limititation? PSCI standart does not limit choise of a CPU.


> Non-boot VCPUs of the guest should be hot-unplugged using PSCI
> CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
> are left enabled by the guest are assumed to be its wake-up interrupts.
> Therefore, a wake-up interrupt triggers the resume of the guest. Guest
> should resume regardless of the state of Xen (suspended or not).
>
> When a guest calls PSCI system suspend the respective domain will be
> suspended if the following conditions are met:
> 1) Given resume entry point is not invalid
> 2) Other (if any) VCPUs of the calling guest are hot-unplugged
>
> If the conditions above are met the calling domain is labeled as
> suspended and the calling VCPU is blocked. If nothing else wouldn't
> be done the suspended domain would resume from the place where it
> called PSCI system suspend. This is expected if processing of the PSCI
> system suspend call fails. However, in the case of success the calling
> guest should resume (continue execution after the wake-up) from the entry
> point which is given as the first argument of the PSCI system suspend
> call. In addition to the entry point, the guest expects to start within
> the environment whose state matches the state after reset. This means
> that the guest should find reset register values, MMU disabled, etc.
> Thereby, the context of VCPU should be 'reset' (as if the system is
> comming out of reset), the program counter should contain entry point,
> which is 1st argument, and r0/x0 should contain context ID which is 2nd
> argument of PSCI system suspend call. The context of VCPU is set during
> resume path, to prevent it being overwritten by ctxt_switch_from after
> vcpu is blocked and scheduled out.
>
> VCPU is marked as suspended with _VPF_suspended flag. A suspended domain
> will resume after the Xen receives an interrupt which is targeted to the
> domain, unblocks the domain's VCPU, and schedules it in. During the
> vcpu_unblock execution the VCPU is checked for VPF_suspended flag. If
> the flag is present, the context of that VCPU gets cleared and entry
> point/cid are set.
>
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> Signed-off-by: Mykyta Poturai <mykyta.poturai@epam.com>
> ---
>  xen/arch/arm/Makefile            |   1 +
>  xen/arch/arm/domain.c            |   4 +
>  xen/arch/arm/suspend.c           | 182 +++++++++++++++++++++++++++++++
>  xen/arch/arm/vpsci.c             |  28 +++++
>  xen/common/sched/core.c          |   8 ++
>  xen/include/asm-arm/domain.h     |   3 +
>  xen/include/asm-arm/perfc_defn.h |   1 +
>  xen/include/asm-arm/psci.h       |   2 +
>  xen/include/asm-arm/suspend.h    |  17 +++
>  xen/include/xen/sched.h          |   3 +
>  10 files changed, 249 insertions(+)
>  create mode 100644 xen/arch/arm/suspend.c
>  create mode 100644 xen/include/asm-arm/suspend.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index b5913c9d39..07dbbd99a3 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -49,6 +49,7 @@ obj-y += setup.o
>  obj-y += shutdown.o
>  obj-y += smp.o
>  obj-y += smpboot.o
> +obj-y += suspend.o
>  obj-y += sysctl.o
>  obj-y += time.o
>  obj-y += traps.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 23c8b345d4..4110154bda 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -40,6 +40,8 @@
>  #include <asm/vtimer.h>
>  #include <asm/vscmi.h>
>  
> +#include <public/sched.h>
> +

Do you really need this include?

>  #include "vpci.h"
>  #include "vuart.h"
>  
> @@ -101,6 +103,8 @@ static void ctxt_switch_from(struct vcpu *p)
>      if ( is_idle_vcpu(p) )
>          return;
>  
> +    /* VCPU's context should not be saved if its domain is suspended */
> +

Is this a some leftover?

>      p2m_save_state(p);
>  
>      /* CP 15 */
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> new file mode 100644
> index 0000000000..987ba6ac11
> --- /dev/null
> +++ b/xen/arch/arm/suspend.c
> @@ -0,0 +1,182 @@
> +/*
> + * Copyright (C) 2022 EPAM Systems Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2.1 only. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + */
> +
> +#include <xen/sched.h>
> +#include <asm/cpufeature.h>
> +#include <asm/event.h>
> +#include <asm/psci.h>
> +#include <asm/suspend.h>
> +#include <public/sched.h>
> +
> +struct cpu_context cpu_context;
> +
> +/* Reset values of VCPU architecture specific registers */
> +static void vcpu_arch_reset(struct vcpu *v)
> +{
> +    v->arch.ttbr0 = 0;
> +    v->arch.ttbr1 = 0;
> +    v->arch.ttbcr = 0;
> +
> +    v->arch.csselr = 0;
> +    v->arch.cpacr = 0;
> +    v->arch.contextidr = 0;
> +    v->arch.tpidr_el0 = 0;
> +    v->arch.tpidrro_el0 = 0;
> +    v->arch.tpidr_el1 = 0;
> +    v->arch.vbar = 0;
> +    v->arch.dacr = 0;
> +    v->arch.par = 0;
> +#if defined(CONFIG_ARM_32)
> +    v->arch.mair0 = 0;
> +    v->arch.mair1 = 0;
> +    v->arch.amair0 = 0;
> +    v->arch.amair1 = 0;
> +#else
> +    v->arch.mair = 0;
> +    v->arch.amair = 0;
> +#endif
> +    /* Fault Status */
> +#if defined(CONFIG_ARM_32)
> +    v->arch.dfar = 0;
> +    v->arch.ifar = 0;
> +    v->arch.dfsr = 0;
> +#elif defined(CONFIG_ARM_64)
> +    v->arch.far = 0;
> +    v->arch.esr = 0;
> +#endif
> +
> +    v->arch.ifsr  = 0;
> +    v->arch.afsr0 = 0;
> +    v->arch.afsr1 = 0;
> +
> +#ifdef CONFIG_ARM_32
> +    v->arch.joscr = 0;
> +    v->arch.jmcr = 0;
> +#endif
> +
> +    if ( is_32bit_domain(v->domain) && cpu_has_thumbee )
> +    {
> +        v->arch.teecr = 0;
> +        v->arch.teehbr = 0;
> +    }
> +}
> +
> +
> +static void vcpu_suspend(register_t epoint, register_t cid)
> +{
> +    struct vcpu *v = current;
> +
> +    v->arch.suspend_ep = epoint;
> +    v->arch.suspend_cid = cid;
> +    set_bit(_VPF_suspended, &v->pause_flags);
> +    return;
> +}
> +
> +/*
> + * This function sets the context of current VCPU to the state which is expected
> + * by the guest on resume. The expected VCPU state is:
> + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
> + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
> + * 3) All other general purpose and system registers should have reset values
> + */
> +void vcpu_resume(struct vcpu *v)
> +{
> +
> +    struct vcpu_guest_context ctxt;
> +
> +    /* Make sure that VCPU guest regs are zeroed */
> +    memset(&ctxt, 0, sizeof(ctxt));
> +
> +    /* Set non-zero values to the registers prior to copying */
> +    ctxt.user_regs.pc64 = (u64)v->arch.suspend_ep;
> +
> +    if ( is_32bit_domain(v->domain) )
> +    {
> +        ctxt.user_regs.r0_usr = v->arch.suspend_cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
> +
> +        /* Thumb set is allowed only for 32-bit domain */
> +        if ( v->arch.suspend_ep & 1 )
> +        {
> +            ctxt.user_regs.cpsr |= PSR_THUMB;
> +            ctxt.user_regs.pc64 &= ~(u64)1;
> +        }
> +    }
> +#ifdef CONFIG_ARM_64
> +    else
> +    {
> +        ctxt.user_regs.x0 = v->arch.suspend_cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;
> +    }
> +#endif
> +    ctxt.sctlr = SCTLR_GUEST_INIT;
> +    ctxt.flags = VGCF_online;
> +
> +    /* Reset architecture specific registers */
> +    vcpu_arch_reset(v);
> +
> +    /* Initialize VCPU registers */
> +    arch_set_info_guest(v, &ctxt);
> +    clear_bit(_VPF_suspended, &v->pause_flags);
> +}
> +
> +int32_t domain_suspend(register_t epoint, register_t cid)
> +{
> +    struct vcpu *v;
> +    struct domain *d = current->domain;
> +    bool is_thumb = epoint & 1;
> +
> +    dprintk(XENLOG_DEBUG,
> +            "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> +            d->domain_id, epoint, cid);
> +
> +    /* THUMB set is not allowed with 64-bit domain */
> +    if ( is_64bit_domain(d) && is_thumb )
> +        return PSCI_INVALID_ADDRESS;
> +
> +    /* Ensure that all CPUs other than the calling one are offline */
> +    for_each_vcpu ( d, v )
> +    {
> +        if ( v != current && is_vcpu_online(v) )
> +            return PSCI_DENIED;
> +    }
> +
> +    //TODO: add support for suspending from any VCPU

How hard it would be to implement?

> +    if (current->vcpu_id != 0)
> +        return PSCI_DENIED;
> +
> +    /*
> +     * Prepare the calling VCPU for suspend (reset its context, save entry point
> +     * into pc and context ID into r0/x0 as specified by PSCI SYSTEM_SUSPEND)
> +     */
> +    vcpu_suspend(epoint, cid);
> +
> +    /*
> +     * The calling domain is suspended by blocking its last running VCPU. If an
> +     * event is pending the domain will resume right away (VCPU will not block,
> +     * but when scheduled in it will resume from the given entry point).
> +     */
> +    vcpu_block_unless_event_pending(current);
> +
> +    return PSCI_SUCCESS;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vpsci.c b/xen/arch/arm/vpsci.c
> index c1e250be59..f4e6e92873 100644
> --- a/xen/arch/arm/vpsci.c
> +++ b/xen/arch/arm/vpsci.c
> @@ -18,6 +18,7 @@
>  #include <asm/vgic.h>
>  #include <asm/vpsci.h>
>  #include <asm/event.h>
> +#include <asm/suspend.h>
>  
>  #include <public/sched.h>
>  
> @@ -208,6 +209,11 @@ static void do_psci_0_2_system_reset(void)
>      domain_shutdown(d,SHUTDOWN_reboot);
>  }
>  
> +static int32_t do_psci_1_0_system_suspend(register_t epoint, register_t cid)
> +{
> +    return domain_suspend(epoint, cid);
> +}
> +
>  static int32_t do_psci_1_0_features(uint32_t psci_func_id)
>  {
>      /* /!\ Ordered by function ID and not name */
> @@ -225,6 +231,8 @@ static int32_t do_psci_1_0_features(uint32_t psci_func_id)
>      case PSCI_0_2_FN32_SYSTEM_OFF:
>      case PSCI_0_2_FN32_SYSTEM_RESET:
>      case PSCI_1_0_FN32_PSCI_FEATURES:
> +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
>      case ARM_SMCCC_VERSION_FID:
>          return 0;
>      default:
> @@ -355,6 +363,26 @@ bool do_vpsci_0_2_call(struct cpu_user_regs *regs, uint32_t fid)
>          return true;
>      }
>  
> +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
> +    {
> +        register_t epoint = PSCI_ARG(regs,1);
> +        register_t cid = PSCI_ARG(regs,2);
> +        register_t ret;
> +
> +        perfc_incr(vpsci_system_suspend);
> +        /* Set the result to PSCI_SUCCESS if the call fails.
> +         * Otherwise preserve the context_id in x0. For now 

Looks like there is a trailing space                          ^

> +         * we don't support the case where the system is suspended
> +         * to a shallower level and PSCI_SUCCESS is returned to the 

And in this line also

> +         * caller.
> +         */
> +        ret = do_psci_1_0_system_suspend(epoint, cid);
> +        if ( ret != PSCI_SUCCESS )
> +            PSCI_SET_RESULT(regs, ret);
> +        return true;
> +    }
> +
>      default:
>          return false;
>      }
> diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
> index 8f4b1ca10d..4e1ea62c44 100644
> --- a/xen/common/sched/core.c
> +++ b/xen/common/sched/core.c
> @@ -38,6 +38,10 @@
>  #include <xsm/xsm.h>
>  #include <xen/err.h>
>  
> +#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
> +#include <asm/suspend.h>
> +#endif
> +
>  #include "private.h"
>  
>  #ifdef CONFIG_XEN_GUEST
> @@ -957,6 +961,10 @@ void vcpu_unblock(struct vcpu *v)
>  {
>      if ( !test_and_clear_bit(_VPF_blocked, &v->pause_flags) )
>          return;
> +#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
> +    if ( test_bit(_VPF_suspended, &v->pause_flags) )
> +        vcpu_resume(v);
> +#endif

This does not look good. I do understant that that was I, who suggested
to add this flag, but I didn't expected that it will get into common code.

Also, this is not justified in the commit message. I remeber that there was
some discussion about why vcpu_block()/vcpu_unblock() could not be used, and
I'd love to see its summary in the commit message.

>  
>      /* Polling period ends when a VCPU is unblocked. */
>      if ( unlikely(v->poll_evtchn != 0) )
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 413e5a2a18..715841e0b5 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -203,6 +203,9 @@ struct arch_vcpu
>      struct vtimer virt_timer;
>      bool   vtimer_initialized;
>  
> +    register_t suspend_ep;
> +    register_t suspend_cid;
> +
>      /*
>       * The full P2M may require some cleaning (e.g when emulation
>       * set/way). As the action can take a long time, it requires
> diff --git a/xen/include/asm-arm/perfc_defn.h b/xen/include/asm-arm/perfc_defn.h
> index 31f071222b..d71e91a5e4 100644
> --- a/xen/include/asm-arm/perfc_defn.h
> +++ b/xen/include/asm-arm/perfc_defn.h
> @@ -33,6 +33,7 @@ PERFCOUNTER(vpsci_system_reset,        "vpsci: system_reset")
>  PERFCOUNTER(vpsci_cpu_suspend,         "vpsci: cpu_suspend")
>  PERFCOUNTER(vpsci_cpu_affinity_info,   "vpsci: cpu_affinity_info")
>  PERFCOUNTER(vpsci_features,            "vpsci: features")
> +PERFCOUNTER(vpsci_system_suspend,      "vpsci: system_suspend")
>  
>  PERFCOUNTER(vcpu_kick,                 "vcpu: notify other vcpu")
>  
> diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> index 832f77afff..26462d0c47 100644
> --- a/xen/include/asm-arm/psci.h
> +++ b/xen/include/asm-arm/psci.h
> @@ -43,10 +43,12 @@ void call_psci_system_reset(void);
>  #define PSCI_0_2_FN32_SYSTEM_OFF          PSCI_0_2_FN32(8)
>  #define PSCI_0_2_FN32_SYSTEM_RESET        PSCI_0_2_FN32(9)
>  #define PSCI_1_0_FN32_PSCI_FEATURES       PSCI_0_2_FN32(10)
> +#define PSCI_1_0_FN32_SYSTEM_SUSPEND      PSCI_0_2_FN32(14)
>  
>  #define PSCI_0_2_FN64_CPU_SUSPEND         PSCI_0_2_FN64(1)
>  #define PSCI_0_2_FN64_CPU_ON              PSCI_0_2_FN64(3)
>  #define PSCI_0_2_FN64_AFFINITY_INFO       PSCI_0_2_FN64(4)
> +#define PSCI_1_0_FN64_SYSTEM_SUSPEND      PSCI_0_2_FN64(14)
>  
>  /* PSCI v0.2 affinity level state returned by AFFINITY_INFO */
>  #define PSCI_0_2_AFFINITY_LEVEL_ON      0
> diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> new file mode 100644
> index 0000000000..fbaa414f0c
> --- /dev/null
> +++ b/xen/include/asm-arm/suspend.h
> @@ -0,0 +1,17 @@
> +#ifndef __ASM_ARM_SUSPEND_H__
> +#define __ASM_ARM_SUSPEND_H__
> +
> +int32_t domain_suspend(register_t epoint, register_t cid);
> +void vcpu_resume(struct vcpu *v);
> +
> +#endif
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> index 3b4ed3a2ab..b2f6d1af28 100644
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -903,6 +903,9 @@ static inline struct domain *next_domain_in_cpupool(
>  /* VCPU is parked. */
>  #define _VPF_parked          8
>  #define VPF_parked           (1UL<<_VPF_parked)
> +/* VCPU is suspended */
> +#define _VPF_suspended       9
> +#define VPF_suspended        (1UL<<_VPF_suspended)
>  
>  static inline bool vcpu_runnable(const struct vcpu *v)
>  {


-- 
Volodymyr Babchuk at EPAM

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 01/19] xen/arm: Implement PSCI system suspend
  2022-10-08 22:00   ` Volodymyr Babchuk
@ 2022-10-09  9:30     ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-10-09  9:30 UTC (permalink / raw)
  To: Volodymyr Babchuk, Mykyta Poturai
  Cc: xen-devel, Mirela Simonovic, Stefano Stabellini,
	Bertrand Marquis, Andrew Cooper, George Dunlap, Ian Jackson,
	Jan Beulich, Wei Liu, Dario Faggioli, Saeed Nowshadi,
	Mykyta Poturai

Hi,

On 08/10/2022 23:00, Volodymyr Babchuk wrote:
>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>> new file mode 100644
>> index 0000000000..987ba6ac11
>> --- /dev/null
>> +++ b/xen/arch/arm/suspend.c
>> @@ -0,0 +1,182 @@
>> +/*
>> + * Copyright (C) 2022 EPAM Systems Inc.

Not related to Volodymyr's answer but I will reply here as I noticed it.

The code was written in 2018 by Aggios. So shouldn't this be a 2018 
copyright from Aggios? You can add yours on top if you want but it is 
not super clear what has been modified (a changelog after --- in the 
commit message would have been useful).

>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU Lesser General Public License as published
>> + * by the Free Software Foundation; version 2.1 only. with the special
>> + * exception on linking described in file LICENSE.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU Lesser General Public License for more details.
>> + */

We are not allowing LGPL* license in the hypervisor. This has to be 
GPLv2-only (this was the original contribution even if there were no 
copyright).

But it would be better to use an SPDX tag as it makes clearer which 
license is used (see [1]).

[...]

>> diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
>> index 8f4b1ca10d..4e1ea62c44 100644
>> --- a/xen/common/sched/core.c
>> +++ b/xen/common/sched/core.c
>> @@ -38,6 +38,10 @@
>>   #include <xsm/xsm.h>
>>   #include <xen/err.h>
>>   
>> +#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
>> +#include <asm/suspend.h>
>> +#endif
>> +
>>   #include "private.h"
>>   
>>   #ifdef CONFIG_XEN_GUEST
>> @@ -957,6 +961,10 @@ void vcpu_unblock(struct vcpu *v)
>>   {
>>       if ( !test_and_clear_bit(_VPF_blocked, &v->pause_flags) )
>>           return;
>> +#if defined(CONFIG_ARM) || defined(CONFIG_ARM64)
>> +    if ( test_bit(_VPF_suspended, &v->pause_flags) )
>> +        vcpu_resume(v);
>> +#endif
> 
> This does not look good. I do understant that that was I, who suggested
> to add this flag, but I didn't expected that it will get into common code.

AFAIU, you are using the flag to indicate whether the vCPU should be 
reset. If I am not mistaken, this should only happen for vCPU0 (other 
vCPUs will be brought up using PSCI CPU ON). So you could reset the vCPU 
before blocking it.

With that, the flag should not be necessary (at least in this situation).

> 
> Also, this is not justified in the commit message. I remeber that there was
> some discussion about why vcpu_block()/vcpu_unblock() could not be used, and
> I'd love to see its summary in the commit message.

Are you referring to the discussion in [2]?

Cheers,

[1] 20221008001544.78302-2-sstabellini@kernel.org
[2] 1542022244-22977-3-git-send-email-mirela.simonovic@aggios.com

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 17/19] xen: don't free percpu areas during suspend
  2022-10-07 11:17   ` Juergen Gross
@ 2022-10-21  9:46     ` Dario Faggioli
  0 siblings, 0 replies; 37+ messages in thread
From: Dario Faggioli @ 2022-10-21  9:46 UTC (permalink / raw)
  To: Juergen Gross, Mykyta_Poturai, xen-devel
  Cc: sstabellini, bertrand.marquis, Volodymyr_Babchuk, julien

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

On Fri, 2022-10-07 at 13:17 +0200, Juergen Gross wrote:
> On 07.10.22 12:32, Mykyta Poturai wrote:
> > 
> > Signed-off-by: Juergen Gross <jgross@suse.com>
> 
> I can't remember having written this patch. The one I remember was
> for
> x86.
> 
> > Reviewed-by: Dario Faggioli <dfaggioli@suse.com>
> 
> I doubt that, reasoning see above.
> 
Right. In fact, I can't find any records of having sent an email with
such a tag... Or to have ever even replied to any patch sent from this
email address, for what matters.

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 05/19] xen/x86: Move freeze/thaw_domains into common files
  2022-10-07 10:32 ` [PATCH 05/19] xen/x86: Move freeze/thaw_domains into common files Mykyta Poturai
@ 2022-12-06 20:36   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 20:36 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Andrew Cooper, George Dunlap, Ian Jackson,
	Jan Beulich, Stefano Stabellini, Wei Liu, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> These functions will be reused by suspend/resume support for ARM.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

Your Signed-off-by is missing.

> ---
>   xen/common/domain.c     | 29 +++++++++++++++++++++++++++++

The title suggests that there will be code movement. However... I only 
see addition here. Did you intend to remove the functions from 
arch/x86/acpi/power.c?

>   xen/include/xen/sched.h |  3 +++
>   2 files changed, 32 insertions(+)
> 
> diff --git a/xen/common/domain.c b/xen/common/domain.c
> index 56d47dd664..5e5894c468 100644
> --- a/xen/common/domain.c
> +++ b/xen/common/domain.c
> @@ -1884,6 +1884,35 @@ int continue_hypercall_on_cpu(
>       return 0;
>   }
>   
> +
> +void freeze_domains(void)
> +{
> +    struct domain *d;
> +
> +    rcu_read_lock(&domlist_read_lock);
> +    /*
> +     * Note that we iterate in order of domain-id. Hence we will pause dom0
> +     * first which is required for correctness (as only dom0 can add domains to
> +     * the domain list). Otherwise we could miss concurrently-created domains.
> +     */
> +    for_each_domain ( d )
> +        domain_pause(d);
> +    rcu_read_unlock(&domlist_read_lock);
> +}
> +
> +void thaw_domains(void)
> +{
> +    struct domain *d;
> +
> +    rcu_read_lock(&domlist_read_lock);
> +    for_each_domain ( d )
> +    {
> +        restore_vcpu_affinity(d);
> +        domain_unpause(d);
> +    }
> +    rcu_read_unlock(&domlist_read_lock);
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> index 00a939aa01..c8ddfdd51c 100644
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -978,6 +978,9 @@ static inline struct vcpu *domain_vcpu(const struct domain *d,
>       return vcpu_id >= d->max_vcpus ? NULL : d->vcpu[idx];
>   }
>   
> +void freeze_domains(void);
> +void thaw_domains(void);
> +
>   void cpu_init(void);
>   
>   /*

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 06/19] xen/arm: Freeze domains on suspend and thaw them on resume
  2022-10-07 10:32 ` [PATCH 06/19] xen/arm: Freeze domains on suspend and thaw them " Mykyta Poturai
@ 2022-12-06 20:47   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 20:47 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> Freeze and thaw of domains is reused as implemented for x86. In
> addition, system_state variable is updated to represent the actual
> state of the system.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

Your signed-off-by is missing.

> ---
>   xen/arch/arm/suspend.c | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index b09bf319d0..2b94816b63 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -137,6 +137,14 @@ static long system_suspend(void *data)

So this is something I already hinted in the original series. But I will 
repeat here.

I find quite difficult to review the code in system_suspend() when this 
introduced in piece meal. For instance, patch #8 is fixing a bug 
introduced by patch #7.

Patch #7 adds barrier after code your added here...

So I would much prefer if we introduce the helpers in a few patches and 
then have one patch that will in system_suspend().

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 07/19] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume
  2022-10-07 10:32 ` [PATCH 07/19] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume Mykyta Poturai
@ 2022-12-06 21:00   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 21:00 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> Non-boot CPUs have to be disabled on suspend and enabled on resume
> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
> platform capabilities, this may lead to the physical powering down of
> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
> each non-boot CPU).
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c | 15 ++++++++++++++-
>   1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index 2b94816b63..0784979e4f 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -13,6 +13,7 @@
>    */
>   
>   #include <xen/sched.h>
> +#include <xen/cpu.h>
>   #include <asm/cpufeature.h>
>   #include <asm/event.h>
>   #include <asm/psci.h>
> @@ -135,17 +136,29 @@ void vcpu_resume(struct vcpu *v)
>   /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>   static long system_suspend(void *data)
>   {
> +    int status;
> +
>       BUG_ON(system_state != SYS_STATE_active);
>   
>       system_state = SYS_STATE_suspend;
>       freeze_domains();
>   
> +    status = disable_nonboot_cpus();
> +    if ( status )
> +    {
> +        system_state = SYS_STATE_resume;
> +        goto resume_nonboot_cpus;
> +    }
> +
>       system_state = SYS_STATE_resume;
>   
> +resume_nonboot_cpus:
> +    enable_nonboot_cpus();
>       thaw_domains();
>       system_state = SYS_STATE_active;
> +    dsb(sy);

Please document what is this dsb() for.

>   
> -    return -ENOSYS;
> +    return status;
>   }
>   
>   int32_t domain_suspend(register_t epoint, register_t cid)

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 09/19] xen/arm: Implement GIC suspend/resume functions (gicv2 only)
  2022-10-07 10:32 ` [PATCH 09/19] xen/arm: Implement GIC suspend/resume functions (gicv2 only) Mykyta Poturai
@ 2022-12-06 21:18   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 21:18 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> System suspend may lead to a state where GIC would be powered down.
> Therefore, Xen should save/restore the context of GIC on suspend/resume.
> Note that the context consists of states of registers which are
> controlled by the hypervisor. Other GIC registers which are accessible
> by guests are saved/restored on context switch.
> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
> the GIC.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

Your signed-off-by is missing.

> ---
>   xen/arch/arm/gic-v2.c     | 138 +++++++++++++++++++++++++++++++++++++-
>   xen/arch/arm/gic.c        |  27 ++++++++
>   xen/include/asm-arm/gic.h |   8 +++
>   3 files changed, 172 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index b2adc8ec9a..b077b66d92 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -123,6 +123,23 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
>   /* Maximum cpu interface per GIC */
>   #define NR_GIC_CPU_IF 8
>   
> +/* GICv2 registers to be saved/restored on system suspend/resume */
> +struct gicv2_context {
> +    /* GICC context */
> +    uint32_t gicc_ctlr;
> +    uint32_t gicc_pmr;
> +    uint32_t gicc_bpr;
> +    /* GICD context */
> +    uint32_t gicd_ctlr;
> +    uint32_t *gicd_isenabler;
> +    uint32_t *gicd_isactiver;
> +    uint32_t *gicd_ipriorityr;
> +    uint32_t *gicd_itargetsr;
> +    uint32_t *gicd_icfgr;
> +};
> +
> +static struct gicv2_context gicv2_context;
> +
>   static inline void writeb_gicd(uint8_t val, unsigned int offset)
>   {
>       writeb_relaxed(val, gicv2.map_dbase + offset);
> @@ -1251,6 +1268,40 @@ static void __init gicv2_acpi_init(void)
>   static void __init gicv2_acpi_init(void) { }
>   #endif
>   
> +static int gicv2_alloc_context(struct gicv2_context *gc)
> +{
> +    uint32_t n = gicv2_info.nr_lines;
> +
> +    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> +    if ( !gc->gicd_isenabler )
> +        goto err_free;
> +
> +    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> +    if ( !gc->gicd_isactiver )
> +        goto err_free;
> +
> +    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> +    if ( !gc->gicd_itargetsr )
> +        goto err_free;
> +
> +    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> +    if ( !gc->gicd_ipriorityr )
> +        goto err_free;
> +
> +    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
> +    if ( !gc->gicd_icfgr )
> +        goto err_free;
> +
> +    return 0;
> +err_free:
> +    xfree(gc->gicd_icfgr);
> +    xfree(gc->gicd_ipriorityr);
> +    xfree(gc->gicd_itargetsr);
> +    xfree(gc->gicd_isactiver);
> +    xfree(gc->gicd_isenabler);

Please add a newline.

> +    return -ENOMEM;
> +}
> +
>   static int __init gicv2_init(void)
>   {
>       uint32_t aliased_offset = 0;
> @@ -1318,7 +1369,8 @@ static int __init gicv2_init(void)
>   
>       spin_unlock(&gicv2.lock);
>   
> -    return 0;
> +    /* Allocate memory to be used for saving GIC context during the suspend */
> +    return gicv2_alloc_context(&gicv2_context);
As I pointed out in the previous version, suspend/resume is not going to 
be widely used. So I don't think we should prevent booting if we can't 
allocate the memory.

In addition to that, I think we should consider to have:
   * a command line option to avoid wasting memory if the feature is not 
going to be used
   * ifdef the suspend/resume code to allow an integrator to compile out 
any related code.

>   }
>   
>   static void gicv2_do_LPI(unsigned int lpi)
> @@ -1327,6 +1379,88 @@ static void gicv2_do_LPI(unsigned int lpi)
>       BUG();
>   }
>   
> +static int gicv2_suspend(void)
> +{
> +    int i;

This should be unsigned int.

> +
> +    ASSERT(gicv2_context.gicd_isenabler);
> +    ASSERT(gicv2_context.gicd_isactiver);
> +    ASSERT(gicv2_context.gicd_ipriorityr);
> +    ASSERT(gicv2_context.gicd_itargetsr);
> +    ASSERT(gicv2_context.gicd_icfgr);
> +
> +    /* Save GICC configuration */
> +    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
> +    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
> +    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
> +
> +    /* Save GICD configuration */
> +    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> +        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
> +
> +    return 0;
> +}
> +
> +static void gicv2_resume(void)
> +{
> +    int i;

Ditto.

> +
> +    ASSERT(gicv2_context.gicd_isenabler);
> +    ASSERT(gicv2_context.gicd_isactiver);
> +    ASSERT(gicv2_context.gicd_ipriorityr);
> +    ASSERT(gicv2_context.gicd_itargetsr);
> +    ASSERT(gicv2_context.gicd_icfgr);
> +
> +    /* Disable CPU interface and distributor */
> +    writel_gicc(0, GICC_CTLR);
> +    writel_gicd(0, GICD_CTLR);

We don't need those two lines during boot. So why do you need them 
during resume?

> +
> +    /* Restore GICD configuration */
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ ) {
> +        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
> +        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
> +    }
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ ) {
> +        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
> +        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);
> +    }
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> +        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
> +
> +    /* Make sure all registers are restored and enable distributor */

The first part reads like there is a missing barrier.

> +    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
> +
> +    /* Restore GIC CPU interface configuration */
> +    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
> +    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
> +
> +    /* Enable GIC CPU interface */
> +    writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | GICC_CTL_EOI,
> +                GICC_CTLR);
> +}
> +
>   const static struct gic_hw_operations gicv2_ops = {
>       .info                = &gicv2_info,
>       .init                = gicv2_init,
> @@ -1361,6 +1495,8 @@ const static struct gic_hw_operations gicv2_ops = {
>       .map_hwdom_extra_mappings = gicv2_map_hwdown_extra_mappings,
>       .iomem_deny_access   = gicv2_iomem_deny_access,
>       .do_LPI              = gicv2_do_LPI,
> +    .suspend             = gicv2_suspend,
> +    .resume              = gicv2_resume,
>   };
>   
>   /* Set up the GIC */
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 3b0331b538..e9feb1fd3b 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -467,6 +467,33 @@ int gic_iomem_deny_access(const struct domain *d)
>       return gic_hw_ops->iomem_deny_access(d);
>   }
>   
> +int gic_suspend(void)
> +{
> +    /* Must be called by boot CPU#0 with interrupts disabled */
> +    ASSERT(!local_irq_is_enabled());
> +    ASSERT(!smp_processor_id());
> +
> +    if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
> +        return -ENOSYS;
> +
> +    gic_hw_ops->suspend();
> +
> +    return 0;
> +}
> +
> +void gic_resume(void)
> +{
> +    /*
> +     * Must be called by boot CPU#0 with interrupts disabled after gic_suspend
> +     * has returned successfully.
> +     */
> +    ASSERT(!local_irq_is_enabled());
> +    ASSERT(!smp_processor_id());
> +    ASSERT(gic_hw_ops->resume);
> +
> +    gic_hw_ops->resume();
> +}
> +
>   static int cpu_gic_callback(struct notifier_block *nfb,
>                               unsigned long action,
>                               void *hcpu)
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index c7f0c343d1..113e39460d 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -275,6 +275,10 @@ extern int gicv_setup(struct domain *d);
>   extern void gic_save_state(struct vcpu *v);
>   extern void gic_restore_state(struct vcpu *v);
>   
> +/* Suspend/resume */
> +extern int gic_suspend(void);
> +extern void gic_resume(void);
> +
>   /* SGI (AKA IPIs) */
>   enum gic_sgi {
>       GIC_SGI_EVENT_CHECK,
> @@ -390,6 +394,10 @@ struct gic_hw_operations {
>       int (*iomem_deny_access)(const struct domain *d);
>       /* Handle LPIs, which require special handling */
>       void (*do_LPI)(unsigned int lpi);
> +    /* Save GIC configuration due to the system suspend */
> +    int (*suspend)(void);
> +    /* Restore GIC configuration due to the system resume */
> +    void (*resume)(void);
>   };
>   
>   extern const struct gic_hw_operations *gic_hw_ops;

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 10/19] xen/arm: Suspend/resume GIC on system suspend/resume
  2022-10-07 10:32 ` [PATCH 10/19] xen/arm: Suspend/resume GIC on system suspend/resume Mykyta Poturai
@ 2022-12-06 21:20   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 21:20 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> GIC state is saved on system suspend by calling gic_suspend
> (this function does not change current state of the GIC but only
> saves the values of configuration registers).
> The state of GIC has to be restored by calling gic_resume, but only
> if the gic_suspend has succeeded. If gic_suspend fails, we'll just
> restore interrupts configuration and abort suspend.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/gic.c     |  4 +---
>   xen/arch/arm/suspend.c | 14 ++++++++++++++
>   2 files changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index e9feb1fd3b..ef90664929 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -476,9 +476,7 @@ int gic_suspend(void)
>       if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
>           return -ENOSYS;
>   
> -    gic_hw_ops->suspend();
> -
> -    return 0;
> +    return gic_hw_ops->suspend();

This should be part of the previous patch.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 11/19] xen/arm: Suspend/resume timer interrupt generation
  2022-10-07 10:32 ` [PATCH 11/19] xen/arm: Suspend/resume timer interrupt generation Mykyta Poturai
@ 2022-12-06 21:29   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 21:29 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> Timer interrupts have to be disabled while the system is in suspend.
> Otherwise, a timer interrupt would fire and wake-up the system.
> Suspending the timer interrupts consists of disabling physical EL1
> and EL2 timers. The resume consists only of raising timer softirq,
> which will trigger the generic timer code to reprogram the EL2 timer
> as needed. Enabling of EL1 physical timer will be triggered by an
> entity which uses it.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c     |  4 ++++
>   xen/arch/arm/time.c        | 22 ++++++++++++++++++++++
>   xen/include/asm-arm/time.h |  3 +++
>   3 files changed, 29 insertions(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index b94a6df86d..05c43ce502 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -151,6 +151,8 @@ static long system_suspend(void *data)
>           goto resume_nonboot_cpus;
>       }
>   
> +    time_suspend();
> +
>       local_irq_save(flags);
>       status = gic_suspend();
>       if ( status )
> @@ -166,6 +168,8 @@ static long system_suspend(void *data)
>   resume_irqs:
>       local_irq_restore(flags);
>   
> +    time_resume();
> +
>   resume_nonboot_cpus:
>       rcu_barrier();
>       enable_nonboot_cpus();
> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
> index dec53b5f7d..ca54bcfe68 100644
> --- a/xen/arch/arm/time.c
> +++ b/xen/arch/arm/time.c
> @@ -363,6 +363,28 @@ void domain_set_time_offset(struct domain *d, int64_t time_offset_seconds)
>       /* XXX update guest visible wallclock time */
>   }
>   
> +void time_suspend(void)
> +{
> +    /* Disable physical EL1 timer */
> +    WRITE_SYSREG(0, CNTP_CTL_EL0);
> +
> +    /* Disable hypervisor's timer */
> +    WRITE_SYSREG(0, CNTHP_CTL_EL2);
> +    isb();
> +}
> +
> +void time_resume(void)
> +{
> +    /*
> +     * Raising timer softirq will trigger generic timer code to reprogram_timer
> +     * with the correct timeout value (which is not known here). There is no
> +     * need to do anything else in order to recover the time keeping from power
> +     * down, because the system counter is not affected by the power down (it
> +     * resides out of the ARM's cluster in an always-on part of the SoC).
> +     */

Can you point me to the section in the Arm Arm stating the system 
counter is in an always-on part of the SoC?

> +    raise_softirq(TIMER_SOFTIRQ);

My expectation is that the time_resume() would closely match the x86 
version (aside the arch specific part). I can't see the raise_softirq(). 
So why do we need it here?

Also, there is a missing call to do_settime() and update_vcpu_system_time().

> +}
> +
>   static int cpu_time_callback(struct notifier_block *nfb,
>                                unsigned long action,
>                                void *hcpu)
> diff --git a/xen/include/asm-arm/time.h b/xen/include/asm-arm/time.h
> index 4b401c1110..ded93b490a 100644
> --- a/xen/include/asm-arm/time.h
> +++ b/xen/include/asm-arm/time.h
> @@ -107,6 +107,9 @@ void preinit_xen_time(void);
>   
>   void force_update_vcpu_system_time(struct vcpu *v);
>   
> +void time_suspend(void);
> +void time_resume(void);
> +
>   #endif /* __ARM_TIME_H__ */
>   /*
>    * Local variables:

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 12/19] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)
  2022-10-07 10:32 ` [PATCH 12/19] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface) Mykyta Poturai
@ 2022-12-06 21:34   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 21:34 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> PSCI system suspend function shall be invoked to finalize Xen suspend
> procedure. Resume entry point, which needs to be passed via 1st argument
> of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
> is just a placeholder that will be implemented in assembly. Context ID,
> which is 2nd argument of system suspend PSCI call, is unused, as in Linux.
> 
> Update: moved hyp_resume to head.S to place it near the rest of the
> start code
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
>   xen/arch/arm/arm64/head.S     |  3 +++
>   xen/arch/arm/psci.c           | 16 ++++++++++++++++
>   xen/arch/arm/suspend.c        |  4 ++++
>   xen/include/asm-arm/psci.h    |  1 +
>   xen/include/asm-arm/suspend.h |  1 +
>   5 files changed, 25 insertions(+)
> 
> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> index aa1f88c764..8857955699 100644
> --- a/xen/arch/arm/arm64/head.S
> +++ b/xen/arch/arm/arm64/head.S
> @@ -957,6 +957,9 @@ ENTRY(efi_xen_start)
>           b     real_start_efi
>   ENDPROC(efi_xen_start)
>   
> +ENTRY(hyp_resume)
> +        b .

This needs to be part of the idmap. At the moment, this will only cover 
up to _end_boot.

> +
>   /*
>    * Local variables:
>    * mode: ASM
> diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
> index 0c90c2305c..43a67eb345 100644
> --- a/xen/arch/arm/psci.c
> +++ b/xen/arch/arm/psci.c
> @@ -25,6 +25,7 @@
>   #include <asm/cpufeature.h>
>   #include <asm/psci.h>
>   #include <asm/acpi.h>
> +#include <asm/suspend.h>
>   
>   /*
>    * While a 64-bit OS can make calls with SMC32 calling conventions, for
> @@ -68,6 +69,21 @@ void call_psci_cpu_off(void)
>       }
>   }
>   
> +int call_psci_system_suspend(void)
> +{
> +#ifdef CONFIG_ARM_64
> +    struct arm_smccc_res res;
> +
> +    /* 2nd argument (context ID) is not used */
> +    arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), &res);
> +
> +    return PSCI_RET(res);
> +#else
> +    /* not supported */
> +    return 1;
> +#endif
> +}
> +
>   void call_psci_system_off(void)
>   {
>       if ( psci_ver > PSCI_VERSION(0, 1) )
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index 05c43ce502..a0258befc9 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -161,6 +161,10 @@ static long system_suspend(void *data)
>           goto resume_irqs;
>       }
>   
> +    status = call_psci_system_suspend();
> +    if ( status )
> +        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
> +
>       system_state = SYS_STATE_resume;
>   
>       gic_resume();
> diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> index 26462d0c47..9f6116a224 100644
> --- a/xen/include/asm-arm/psci.h
> +++ b/xen/include/asm-arm/psci.h
> @@ -20,6 +20,7 @@ extern uint32_t psci_ver;
>   
>   int psci_init(void);
>   int call_psci_cpu_on(int cpu);
> +int call_psci_system_suspend(void);
>   void call_psci_cpu_off(void);
>   void call_psci_system_off(void);
>   void call_psci_system_reset(void);
> diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> index fbaa414f0c..29420e27fa 100644
> --- a/xen/include/asm-arm/suspend.h
> +++ b/xen/include/asm-arm/suspend.h
> @@ -3,6 +3,7 @@
>   
>   int32_t domain_suspend(register_t epoint, register_t cid);
>   void vcpu_resume(struct vcpu *v);
> +void hyp_resume(void);
>   
>   #endif
>   

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 13/19] xen/arm: Resume memory management on Xen resume
  2022-10-07 10:32 ` [PATCH 13/19] xen/arm: Resume memory management on Xen resume Mykyta Poturai
@ 2022-12-06 21:52   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 21:52 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> The MMU needs to be enabled in the resume flow before the context
> can be restored (we need to be able to access the context data by
> virtual address in order to restore it). The configuration of system
> registers prior to branching to the routine that sets up the page
> tables is copied from xen/arch/arm/arm64/head.S.
> After the MMU is enabled, the content of TTBR0_EL2 is changed to
> point to init_ttbr (page tables used at runtime).

This is not Arm Arm compliant. Please look at the series [1] to see how 
you can safely switch the MMU on and use the page tables.

> 
> At boot the init_ttbr variable is updated when a secondary CPU is
> hotplugged. In the scenario where there is only one physical CPU in
> the system, the init_ttbr would not be initialized for the use in
> resume flow. To get the variable initialized in all scenarios in this
> patch we add that the boot CPU updates init_ttbr after it sets the
> page tables for runtime.
> 
> After the memory management is resumed, the SCTLR_WXN in SCTLR_EL2
> has to be set in order to configure that a mapping cannot be both
> writable and executable (this was configured prior to suspend).
> This is done using an existing function (mmu_init_secondary_cpu).
> 
> Update: moved hyp_resume to head.S to place it near the rest of the
> start code
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
>   xen/arch/arm/arm64/entry.S      |  2 ++
>   xen/arch/arm/arm64/head.S       | 30 ++++++++++++++++++++++++++++++
>   xen/arch/arm/mm.c               |  1 +
>   xen/arch/arm/suspend.c          |  6 ++++++
>   xen/include/asm-arm/processor.h | 22 ++++++++++++++++++++++
>   5 files changed, 61 insertions(+)
> 
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index fc3811ad0a..f49f1daf46 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -1,4 +1,6 @@
>   #include <asm/current.h>
> +#include <asm/macros.h>
> +#include <asm/page.h>
>   #include <asm/regs.h>
>   #include <asm/alternative.h>
>   #include <asm/smccc.h>
> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> index 8857955699..82d83214dc 100644
> --- a/xen/arch/arm/arm64/head.S
> +++ b/xen/arch/arm/arm64/head.S
> @@ -958,6 +958,36 @@ ENTRY(efi_xen_start)
>   ENDPROC(efi_xen_start)
>   
>   ENTRY(hyp_resume)
> +        msr   DAIFSet, 0xf           /* Disable all interrupts */
> +
> +        tlbi  alle2
> +        dsb   sy                     /* Ensure completion of TLB flush */
> +        isb

Please explain what this TLB is for.

> +
> +        ldr   x0, =start
> +        adr   x19, start             /* x19 := paddr (start) */
> +        sub   x20, x19, x0           /* x20 := phys-offset */
> +
> +        mov   x22, #0                /* x22 := is_secondary_cpu */

x22 is not hold the is_secondary_cpu anymore.

> +
> +        bl    check_cpu_mode
> +        bl    cpu_init
> +        bl    create_page_tables
> +        bl    enable_mmu
> +
> +        ldr   x0, =mmu_resumed      /* Explicit vaddr, not RIP-relative */
> +        br    x0                    /* Get a proper vaddr into PC */
> +
> +mmu_resumed:
> +        ldr   x4, =init_ttbr         /* VA of TTBR0_EL2 stashed by CPU 0 */
> +        ldr   x4, [x4]               /* Actual value */
> +        dsb   sy
> +        msr   TTBR0_EL2, x4
> +        dsb   sy
> +        isb
> +        tlbi  alle2
> +        dsb   sy                     /* Ensure completion of TLB flush */
> +        isb
>           b .
>   
>   /*
> diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> index eea926d823..29cdaff3bf 100644
> --- a/xen/arch/arm/mm.c
> +++ b/xen/arch/arm/mm.c
> @@ -708,6 +708,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
>       switch_ttbr(ttbr);
>   
>       xen_pt_enforce_wnx();
> +    init_secondary_pagetables(0);

This function is used to prepare the page tables for the secondary CPUs. 
This may allocate memory. So this is incorrect to call for CPU0.

In this case, I think it would be better if the code to suspend sets 
init_ttbr and clear the boot pages tables. This could be done by split 
init_secondary_pagetables() in two:
  1) Allocate memory for the page tables
  2) Clear the boot page tables and the init_ttbr

>   
>   #ifdef CONFIG_ARM_32
>       per_cpu(xen_pgtable, 0) = cpu0_pgtable;
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index a0258befc9..aa5ee4714b 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -167,6 +167,12 @@ static long system_suspend(void *data)
>   
>       system_state = SYS_STATE_resume;
>   
> +    /*
> +     * SCTLR_WXN needs to be set to configure that a mapping cannot be both
> +     * writable and executable. This is done by mmu_init_secondary_cpu.
> +     */

Let's avoid to describe what a function does in the caller. This can be 
easily rot.

> +    mmu_init_secondary_cpu();

I dislike the idea of using this function here. It is meant to be used 
by secondary CPUs, not CPU0.

If you want to use it here, then it should be renamed to reflect how the 
function is used.

> +
>       gic_resume();
>   
>   resume_irqs:
> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
> index 8ab2940f68..ecf97f1ab4 100644
> --- a/xen/include/asm-arm/processor.h
> +++ b/xen/include/asm-arm/processor.h
> @@ -133,6 +133,28 @@
>   #define TTBCR_PD1       (_AC(1,U)<<5)
>   
>   /* SCTLR System Control Register. */
> +/* HSCTLR is a subset of this. */
> +#define SCTLR_TE        (_AC(1,U)<<30)
> +#define SCTLR_AFE       (_AC(1,U)<<29)
> +#define SCTLR_TRE       (_AC(1,U)<<28)
> +#define SCTLR_NMFI      (_AC(1,U)<<27)
> +#define SCTLR_EE        (_AC(1,U)<<25)
> +#define SCTLR_VE        (_AC(1,U)<<24)
> +#define SCTLR_U         (_AC(1,U)<<22)
> +#define SCTLR_FI        (_AC(1,U)<<21)
> +#define SCTLR_WXN       (_AC(1,U)<<19)
> +#define SCTLR_HA        (_AC(1,U)<<17)
> +#define SCTLR_RR        (_AC(1,U)<<14)
> +#define SCTLR_V         (_AC(1,U)<<13)
> +#define SCTLR_I         (_AC(1,U)<<12)
> +#define SCTLR_Z         (_AC(1,U)<<11)
> +#define SCTLR_SW        (_AC(1,U)<<10)
> +#define SCTLR_B         (_AC(1,U)<<7)
> +#define SCTLR_C         (_AC(1,U)<<2)
> +#define SCTLR_A         (_AC(1,U)<<1)
> +#define SCTLR_M         (_AC(1,U)<<0)
> +
> +#define HSCTLR_BASE     _AC(0x30c51878,U)

I don't see any use of SCTLR_* and HSCTLR_* here. What why do you need 
to define them?

>   
>   /* Bits specific to SCTLR_EL1 for Arm32 */
>   

Cheers,

[2] https://lore.kernel.org/all/20221022150422.17707-1-julien@xen.org/

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 14/19] xen/arm: Save/restore context on suspend/resume
  2022-10-07 10:32 ` [PATCH 14/19] xen/arm: Save/restore context on suspend/resume Mykyta Poturai
@ 2022-12-06 22:09   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 22:09 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> The context of CPU general purpose and system control registers
> has to be saved on suspend and restored on resume. This is
> implemented in hyp_suspend and before the return from hyp_resume
> function. The hyp_suspend is invoked just before the PSCI system > suspend call is issued to the ATF.
The software handling the PSCI call may not be ATF. It could be another 
vendor specific firmware or even another hypervisor (if running in 
nested environment). So let's avoid mention it.

> The hyp_suspend has to return a
> non-zero value so that the calling 'if' statement evaluates to true,
> causing the system suspend to be invoked. > Upon the resume, context
> saved on suspend will be restored, including the link register.
> Therefore, after restoring the context the control flow will
> return to the address pointed by the saved link register, which
> is the place from which the hyp_suspend was called. To ensure
> that the calling 'if' statement doesn't again evaluate to true
> and initiate system suspend, hyp_resume has to return a zero value
> after restoring the context.
> Note that the order of saving register context into cpu_context
> structure has to match the order of restoring.
> Since the suspend/resume is supported only for arm64, we define
> a null cpu_context structure so arm32 could compile.
> 
> Update: moved hyp_resume to head.S to place it near the rest of the
> start code

hyp_resume() is not moved in this patch. But this looks like more a 
changelog rather than something that should be part of the commit message.

> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
>   xen/arch/arm/arm64/head.S     | 90 ++++++++++++++++++++++++++++++++++-
>   xen/arch/arm/suspend.c        | 25 ++++++++--
>   xen/include/asm-arm/suspend.h | 22 +++++++++
>   3 files changed, 133 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
> index 82d83214dc..e2c46a864c 100644
> --- a/xen/arch/arm/arm64/head.S
> +++ b/xen/arch/arm/arm64/head.S
> @@ -957,6 +957,53 @@ ENTRY(efi_xen_start)
>           b     real_start_efi
>   ENDPROC(efi_xen_start)
>   
> +/*
> + * void hyp_suspend(struct cpu_context *ptr)

This prototype doesn't match the code nor the commit message.

> + *
> + * x0 - pointer to the storage where callee's context will be saved
> + *
> + * CPU context saved here will be restored on resume in hyp_resume function.
> + * hyp_suspend shall return a non-zero value. Upon restoring context
> + * hyp_resume shall return value zero instead. From C code that invokes
> + * hyp_suspend, the return value is interpreted to determine whether the context
> + * is saved (hyp_suspend) or restored (hyp_resume).
> + */
> +ENTRY(hyp_suspend)
> +        /* Store callee-saved registers */
> +        stp     x19, x20, [x0], #16
> +        stp     x21, x22, [x0], #16
> +        stp     x23, x24, [x0], #16
> +        stp     x25, x26, [x0], #16
> +        stp     x27, x28, [x0], #16
> +        stp     x29, lr, [x0], #16
> +
> +        /* Store stack-pointer */
> +        mov     x2, sp
> +        str     x2, [x0], #8
> +
> +        /* Store system control registers */
> +        mrs     x2, VBAR_EL2
> +        str     x2, [x0], #8
> +        mrs     x2, VTCR_EL2
> +        str     x2, [x0], #8
> +        mrs     x2, VTTBR_EL2
> +        str     x2, [x0], #8
> +        mrs     x2, TPIDR_EL2
> +        str     x2, [x0], #8
> +        mrs     x2, MDCR_EL2
> +        str     x2, [x0], #8
> +        mrs     x2, HSTR_EL2
> +        str     x2, [x0], #8
> +        mrs     x2, CPTR_EL2
> +        str     x2, [x0], #8
> +        mrs     x2, HCR_EL2
> +        str     x2, [x0], #8
> +
> +        /* hyp_suspend must return a non-zero value */
> +        mov     x0, #1
> +        ret
> +
> +
>   ENTRY(hyp_resume)
>           msr   DAIFSet, 0xf           /* Disable all interrupts */
>   
> @@ -988,7 +1035,48 @@ mmu_resumed:
>           tlbi  alle2
>           dsb   sy                     /* Ensure completion of TLB flush */
>           isb
> -        b .
> +
> +        /* Now we can access the cpu_context, so restore the context here */
> +        ldr     x0, =cpu_context
> +
> +        /* Restore callee-saved registers */
> +        ldp     x19, x20, [x0], #16
> +        ldp     x21, x22, [x0], #16
> +        ldp     x23, x24, [x0], #16
> +        ldp     x25, x26, [x0], #16
> +        ldp     x27, x28, [x0], #16
> +        ldp     x29, lr, [x0], #16
> +
> +        /* Restore stack pointer */
> +        ldr     x2, [x0], #8
> +        mov     sp, x2
> +
> +        /* Restore system control registers */
> +        ldr     x2, [x0], #8
> +        msr     VBAR_EL2, x2
> +        ldr     x2, [x0], #8
> +        msr     VTCR_EL2, x2
> +        ldr     x2, [x0], #8
> +        msr     VTTBR_EL2, x2
> +        ldr     x2, [x0], #8
> +        msr     TPIDR_EL2, x2
> +        ldr     x2, [x0], #8
> +        msr     MDCR_EL2, x2
> +        ldr     x2, [x0], #8
> +        msr     HSTR_EL2, x2
> +        ldr     x2, [x0], #8
> +        msr     CPTR_EL2, x2
> +        ldr     x2, [x0], #8
> +        msr     HCR_EL2, x2
> +        isb
> +
> +        /* Since context is restored return from this function will appear as

I can't parse it.

> +         * return from hyp_suspend. To distinguish a return from hyp_suspend
> +         * which is called upon finalizing the suspend, as opposed to return
> +         * from this function which executes on resume, we need to return zero
> +         * value here. */
Coding style. Multi-line comments looks like:

/*
  * Foo
  * Bar
  */

> +        mov x0, #0
> +        ret
>   
>   /*
>    * Local variables:
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index aa5ee4714b..13d1aba658 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -133,6 +133,11 @@ void vcpu_resume(struct vcpu *v)
>       clear_bit(_VPF_suspended, &v->pause_flags);
>   }
>   
> +#ifndef CONFIG_ARM_64
> +/* not supported on ARM_32 */
> +int32_t hyp_suspend(struct cpu_context *ptr) { return 1; }
> +#endif
> +

It would be better if suspend.c is compiled out. So we avoid introduce 
hyp_suspend().

>   /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>   static long system_suspend(void *data)
>   {
> @@ -161,9 +166,23 @@ static long system_suspend(void *data)
>           goto resume_irqs;
>       }
>   
> -    status = call_psci_system_suspend();
> -    if ( status )
> -        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
> +    if ( hyp_suspend(&cpu_context) )
> +    {
> +        status = call_psci_system_suspend();
> +        /*
> +         * If suspend is finalized properly by above system suspend PSCI call,
> +         * the code below in this 'if' branch will never execute. Execution
> +         * will continue from hyp_resume which is the hypervisor's resume point.
> +         * In hyp_resume CPU context will be restored and since link-register is
> +         * restored as well, it will appear to return from hyp_suspend. The
> +         * difference in returning from hyp_suspend on system suspend versus
> +         * resume is in function's return value: on suspend, the return value is
> +         * a non-zero value, on resume it is zero. That is why the control flow
> +         * will not re-enter this 'if' branch on resume.
> +         */
> +        if ( status )
> +            dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
> +    }
>   
>       system_state = SYS_STATE_resume;
>   
> diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> index 29420e27fa..70dbf4e208 100644
> --- a/xen/include/asm-arm/suspend.h
> +++ b/xen/include/asm-arm/suspend.h
> @@ -1,9 +1,31 @@
>   #ifndef __ASM_ARM_SUSPEND_H__
>   #define __ASM_ARM_SUSPEND_H__
>   
> +#ifdef CONFIG_ARM_64
> +struct cpu_context {
> +    uint64_t callee_regs[12];
> +    uint64_t sp;
> +    uint64_t vbar_el2;
> +    uint64_t vtcr_el2;
> +    uint64_t vttbr_el2;
> +    uint64_t tpidr_el2;
> +    uint64_t mdcr_el2;
> +    uint64_t hstr_el2;
> +    uint64_t cptr_el2;
> +    uint64_t hcr_el2;
> +} __aligned(16);
> +#else
> +struct cpu_context {
> +    uint8_t pad;
> +};
> +#endif
> +
> +extern struct cpu_context cpu_context;
> +
>   int32_t domain_suspend(register_t epoint, register_t cid);
>   void vcpu_resume(struct vcpu *v);
>   void hyp_resume(void);
> +int32_t hyp_suspend(struct cpu_context *ptr);
>   
>   #endif
>   

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 16/19] xen/arm: Suspend/resume console on Xen suspend/resume
  2022-10-07 10:32 ` [PATCH 16/19] xen/arm: Suspend/resume console on Xen suspend/resume Mykyta Poturai
@ 2022-12-06 22:12   ` Julien Grall
  2022-12-06 22:14     ` Julien Grall
  0 siblings, 1 reply; 37+ messages in thread
From: Julien Grall @ 2022-12-06 22:12 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Mirela Simonovic <mirela.simonovic@aggios.com>
> 
> This is done using generic console_suspend/resume functions that cause
> uart driver specific suspend/resume handlers to be called for each
> initialized port (if the port has suspend/resume driver handlers
> implemented).

Looking at the UART driver for Arm, most of them (if not all) implement 
suspend/resume with BUG(). So don't you need to properly implement them?

> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

Your signed-off-by is missing.

> ---
>   xen/arch/arm/suspend.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index 4a690eac3b..cf3aab0099 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -14,6 +14,7 @@
>   
>   #include <xen/sched.h>
>   #include <xen/cpu.h>
> +#include <xen/console.h>
>   #include <asm/cpufeature.h>
>   #include <asm/event.h>
>   #include <asm/psci.h>
> @@ -166,6 +167,15 @@ static long system_suspend(void *data)
>           goto resume_irqs;
>       }
>   
> +    dprintk(XENLOG_DEBUG, "Suspend\n");
> +    status = console_suspend();
> +    if ( status )
> +    {
> +        dprintk(XENLOG_ERR, "Failed to suspend the console, err=%d\n", status);
> +        system_state = SYS_STATE_resume;
> +        goto resume_console;
> +    }
> +
>       if ( hyp_suspend(&cpu_context) )
>       {
>           status = call_psci_system_suspend();
> @@ -192,6 +202,10 @@ static long system_suspend(void *data)
>        */
>       mmu_init_secondary_cpu();
>   
> +resume_console:
> +    console_resume();
> +    dprintk(XENLOG_DEBUG, "Resume\n");
> +
>       gic_resume();
>   
>   resume_irqs:

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 16/19] xen/arm: Suspend/resume console on Xen suspend/resume
  2022-12-06 22:12   ` Julien Grall
@ 2022-12-06 22:14     ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 22:14 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Mirela Simonovic, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis, Saeed Nowshadi



On 06/12/2022 22:12, Julien Grall wrote:
> Hi,
> 
> On 07/10/2022 11:32, Mykyta Poturai wrote:
>> From: Mirela Simonovic <mirela.simonovic@aggios.com>
>>
>> This is done using generic console_suspend/resume functions that cause
>> uart driver specific suspend/resume handlers to be called for each
>> initialized port (if the port has suspend/resume driver handlers
>> implemented).
> 
> Looking at the UART driver for Arm, most of them (if not all) implement 
> suspend/resume with BUG(). So don't you need to properly implement them?

I forgot to clarify. No need to implement for every driver. I am just 
puzzled how this was actually tested. What UART are you using on your 
platform?

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 19/19] Fix misleading indentation gcc warning
  2022-10-07 10:32 ` [PATCH 19/19] Fix misleading indentation gcc warning Mykyta Poturai
@ 2022-12-06 22:17   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 22:17 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Oleksandr Andrushchenko, Stefano Stabellini, Volodymyr Babchuk,
	Bertrand Marquis

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> 
> percpu.c: In function 'cpu_percpu_callback':
> percpu.c:61:7: error: this 'if' clause does not guard... [-Werror=misleading-indentation]
>         if ( system_state != SYS_STATE_resume )
>         ^~
> percpu.c:63:9: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
>           break;
>           ^~~~~
> percpu.c:67:7: error: this 'if' clause does not guard... [-Werror=misleading-indentation]
>         if ( system_state != SYS_STATE_suspend )
>         ^~
> percpu.c:69:9: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
>           break;
>           ^~~~~
> 
> Fixes: c3109b76d967 ("xen: don't free percpu areas during suspend")

This commit hash doesn't seem to exist upstream. But it sounds like you 
are fixing a bug that was introduced by this series.

In general, patch within a series should be able to compile without any 
follow-up requirement. Similarly, we should not bug fix a patch within 
the same series.

Instead, this should be folded in the patch that introduced the problem.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 18/19] timers: Don't migrate timers during suspend
  2022-10-07 10:32 ` [PATCH 18/19] timers: Don't migrate timers during suspend Mykyta Poturai
@ 2022-12-06 22:19   ` Julien Grall
  0 siblings, 0 replies; 37+ messages in thread
From: Julien Grall @ 2022-12-06 22:19 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel
  Cc: Andrew Cooper, George Dunlap, Ian Jackson, Jan Beulich,
	Stefano Stabellini, Wei Liu

Hi,

On 07/10/2022 11:32, Mykyta Poturai wrote:
> Migrating timers during suspend causes Dom0 to freeze after resume.

This wants a bit more details such as why dom0 will freeze. But looking 
at upstream, there might be some patches that (e.g. 37f82facd62f 
"xen/sched: migrate timers to correct cpus after suspend") could have 
solved a similar problem.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 37+ messages in thread

end of thread, other threads:[~2022-12-06 22:19 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <cover.1665137247.git.mykyta_poturai@epam.com>
2022-10-07 10:32 ` [PATCH 01/19] xen/arm: Implement PSCI system suspend Mykyta Poturai
2022-10-07 10:59   ` Julien Grall
2022-10-08 22:00   ` Volodymyr Babchuk
2022-10-09  9:30     ` Julien Grall
2022-10-07 10:32 ` [PATCH 02/19] watchdog: Introduce a separate struct for watchdog timers Mykyta Poturai
2022-10-07 10:32 ` [PATCH 03/19] xen/arm: While a domain is suspended put its watchdogs on pause Mykyta Poturai
2022-10-07 10:32 ` [PATCH 04/19] xen/arm: Trigger Xen suspend when Dom0 completes suspend Mykyta Poturai
2022-10-07 10:32 ` [PATCH 05/19] xen/x86: Move freeze/thaw_domains into common files Mykyta Poturai
2022-12-06 20:36   ` Julien Grall
2022-10-07 10:32 ` [PATCH 09/19] xen/arm: Implement GIC suspend/resume functions (gicv2 only) Mykyta Poturai
2022-12-06 21:18   ` Julien Grall
2022-10-07 10:32 ` [PATCH 08/19] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume Mykyta Poturai
2022-10-07 10:32 ` [PATCH 06/19] xen/arm: Freeze domains on suspend and thaw them " Mykyta Poturai
2022-12-06 20:47   ` Julien Grall
2022-10-07 10:32 ` [PATCH 07/19] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume Mykyta Poturai
2022-12-06 21:00   ` Julien Grall
2022-10-07 10:32 ` [PATCH 10/19] xen/arm: Suspend/resume GIC on system suspend/resume Mykyta Poturai
2022-12-06 21:20   ` Julien Grall
2022-10-07 10:32 ` [PATCH 12/19] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface) Mykyta Poturai
2022-12-06 21:34   ` Julien Grall
2022-10-07 10:32 ` [PATCH 11/19] xen/arm: Suspend/resume timer interrupt generation Mykyta Poturai
2022-12-06 21:29   ` Julien Grall
2022-10-07 10:32 ` [PATCH 15/19] xen/arm: Resume Dom0 after Xen resumes Mykyta Poturai
2022-10-07 10:32 ` [PATCH 13/19] xen/arm: Resume memory management on Xen resume Mykyta Poturai
2022-12-06 21:52   ` Julien Grall
2022-10-07 10:32 ` [PATCH 16/19] xen/arm: Suspend/resume console on Xen suspend/resume Mykyta Poturai
2022-12-06 22:12   ` Julien Grall
2022-12-06 22:14     ` Julien Grall
2022-10-07 10:32 ` [PATCH 14/19] xen/arm: Save/restore context on suspend/resume Mykyta Poturai
2022-12-06 22:09   ` Julien Grall
2022-10-07 10:32 ` [PATCH 17/19] xen: don't free percpu areas during suspend Mykyta Poturai
2022-10-07 11:17   ` Juergen Gross
2022-10-21  9:46     ` Dario Faggioli
2022-10-07 10:32 ` [PATCH 19/19] Fix misleading indentation gcc warning Mykyta Poturai
2022-12-06 22:17   ` Julien Grall
2022-10-07 10:32 ` [PATCH 18/19] timers: Don't migrate timers during suspend Mykyta Poturai
2022-12-06 22:19   ` Julien Grall

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.