[PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

This series contains support for suspend to RAM (in the following text just
'suspend') for Xen on arm64. The implementation is aligned with the design
specification that has been proposed on xen-devel list:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html

At a high-level the series contains:
1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
2) Support for resuming a guest on an interrupt targeted to that guest
3) Support for suspending Xen after dom0 finalizes the suspend
4) Support for resuming Xen on an interrupt that is targeted to a guest

--------------------------------------------------------------------------------
In more details:

*** About suspending/resuming guests

The patches included in this series allow PSCI compliant guests that have
support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
suspend and resume on top of Xen without any EL1 code changes.

During their suspend procedure guests will hot-unplug their secondary CPUs,
triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
Guests will save/restore their own EL1 context on suspend/resume.

A guest is expected to leave enabled interrupts that are considered to be its
wake-up sources. Those interrupts will be able to wake up the guest. This holds
regardless of the state of the underlying software layers, i.e. whether Xen gets
suspended or not doesn't affect the ability of the guest to wake up.

First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from which
the guest assumes to start on resume. On resume, guests assume to be running in
an environment whose state matches the CPU state after reset, e.g. with reset
register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
VCPU context and save the resume entry point into program counter before the
guest's VCPU gets scheduled in on resume. This is done when the guest finalizes
its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure that
the resume-ready VCPU's context does not get overwritten later upon the context
switch when the VCPU is scheduled out.
Xen also needs to take care that the guest's view of GIC and timer gets saved.
Also, while a guest is suspended its watchdogs are paused, in order to avoid
watchdog triggered shutdown of a guest that has been asleep for a period of time
that is longer than the watchdog period.

After this point, from Xen's point of view a suspended guest has one VCPU
blocked, waiting for an interrupt. When such an interrupt comes, Xen will
unblock the VCPU of the suspended domain, which results in the guest resuming.

*** About suspending/resuming Xen

Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
considered to be the decision maker for EL1 and EL2.
On suspend, Xen will first freeze all domains. Then, Xen disables physical
secondary CPUs, which leads to physical CPU_OFF to be called by each secondary
CPU. After that Xen finalizes the suspend from the boot CPU.

This consists of suspending the timer, i.e. suppressing its interrupts (we don't
want to be woken up by a timer, there is no VCPU ready to be scheduled). Then
the state of GIC is saved, console is suspended, and CPU context is saved. The
saved context tells where Xen needs to continue execution on resume.
Since Xen will resume with MMU disabled, the first thing to do in resume is to
resume memory management in order to be able to access the context that needs to
be restored (we know virtual address of the context data). Finally Xen calls
SYSTEM_SUSPEND PSCI to the EL3.

When resuming, all the steps done in suspend need to be reverted. This is
completed by unblocking dom0's VCPU, because we always want the dom0 to resume,
regardless of the target domain whose interrupt woke up Xen.

*** Handling of unprivileged guests during Xen suspend/resume

Any domU that is not suspended when dom0 suspends will be frozen, domUs that are
already suspended remain suspended. On resume the suspended domUs still remain
suspended (unless their wake interrupt caused Xen to wake) while the others will
be thawed.

For more details please refer to patches or the design specification:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html

--------------------------------------------------------------------------------
The series does not include:
a) UART driver-specific suspend/resume that gets called when console suspends
b) SMMU suspend/resume
c) Suspend coordination support that would allow dom0 to request domUs to
suspend
These will be submitted in the following series.

Mirela Simonovic (16):
  xen/arm: Implement PSCI system suspend call (virtual interface)
  xen/arm: Save GIC and virtual timer context when the domain suspends
  xen/arm: While a domain is suspended put its watchdogs on pause
  xen/arm: Trigger Xen suspend when Dom0 completes suspend
  xen/x86: Move freeze/thaw_domains into common files
  xen/arm: Freeze domains on suspend and thaw them on resume
  xen/arm: Disable/enable non-boot physical CPUs on suspend/resume
  xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume
  xen/arm: Implement GIC suspend/resume functions (gicv2 only)
  xen/arm: Suspend/resume GIC on system suspend/resume
  xen/arm: Suspend/resume timer interrupt generation
  xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)
  xen/arm: Resume memory management on Xen resume
  xen/arm: Save/restore context on suspend/resume
  xen/arm: Resume Dom0 after Xen resumes
  xen/arm: Suspend/resume console on Xen suspend/resume

Saeed Nowshadi (2):
  xen/arm: Move code that initializes VCPU context into a separate
    function
  xen/arm: Convert setting MMU page tables code into a routine

 xen/arch/arm/Makefile            |   1 +
 xen/arch/arm/arm64/entry.S       | 178 ++++++++++++++++++++++++
 xen/arch/arm/arm64/head.S        | 265 ++++++++++++++++++-----------------
 xen/arch/arm/domain.c            |  62 ++++++---
 xen/arch/arm/gic-v2.c            | 147 ++++++++++++++++++++
 xen/arch/arm/gic.c               |  27 ++++
 xen/arch/arm/mm.c                |   1 +
 xen/arch/arm/psci.c              |  16 +++
 xen/arch/arm/suspend.c           | 292 +++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/time.c              |  22 +++
 xen/arch/arm/vpsci.c             |  19 +++
 xen/arch/x86/acpi/power.c        |  28 ----
 xen/common/domain.c              |  29 ++++
 xen/common/schedule.c            |  38 +++++
 xen/include/asm-arm/gic.h        |   8 ++
 xen/include/asm-arm/perfc_defn.h |   1 +
 xen/include/asm-arm/psci.h       |   3 +
 xen/include/asm-arm/suspend.h    |  39 ++++++
 xen/include/asm-arm/time.h       |   3 +
 xen/include/xen/domain.h         |   1 +
 xen/include/xen/sched.h          |  11 ++
 xen/include/xen/timer.h          |   3 +
 22 files changed, 1019 insertions(+), 175 deletions(-)
 create mode 100644 xen/arch/arm/suspend.c
 create mode 100644 xen/include/asm-arm/suspend.h

-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 01/18] xen/arm: Move code that initializes VCPU context into a separate function

[Mirela Simonovic]

From: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

The arch_set_info_guest() has code to initialize the context of a VCPU.
When a VCPU is resumed it needs to go through the same context
initialization excluding all the validations that this routine does.
We move the actual VCPU context setting into a function so that it can be
shared with the resume path.

Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
---
 xen/arch/arm/domain.c    | 34 +++++++++++++++++++++-------------
 xen/include/xen/domain.h |  1 +
 2 files changed, 22 insertions(+), 13 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 80432872d6..e594b48d81 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -781,6 +781,26 @@ static int is_guest_pv64_psr(uint32_t psr)
 #endif
 
 /*
+ * The actual VCPU initialization after all validations are passed.
+ */
+void _arch_set_info_guest(struct vcpu *v, struct vcpu_guest_context *ctxt)
+{
+    vcpu_regs_user_to_hyp(v, &ctxt->user_regs);
+
+    v->arch.sctlr = ctxt->sctlr;
+    v->arch.ttbr0 = ctxt->ttbr0;
+    v->arch.ttbr1 = ctxt->ttbr1;
+    v->arch.ttbcr = ctxt->ttbcr;
+
+    v->is_initialised = 1;
+
+    if ( ctxt->flags & VGCF_online )
+        clear_bit(_VPF_down, &v->pause_flags);
+    else
+        set_bit(_VPF_down, &v->pause_flags);
+}
+
+/*
  * Initialise VCPU state. The context can be supplied by either the
  * toolstack (XEN_DOMCTL_setvcpucontext) or the guest
  * (VCPUOP_initialise) and therefore must be properly validated.
@@ -818,19 +838,7 @@ int arch_set_info_guest(
     }
 #endif
 
-    vcpu_regs_user_to_hyp(v, regs);
-
-    v->arch.sctlr = ctxt->sctlr;
-    v->arch.ttbr0 = ctxt->ttbr0;
-    v->arch.ttbr1 = ctxt->ttbr1;
-    v->arch.ttbcr = ctxt->ttbcr;
-
-    v->is_initialised = 1;
-
-    if ( ctxt->flags & VGCF_online )
-        clear_bit(_VPF_down, &v->pause_flags);
-    else
-        set_bit(_VPF_down, &v->pause_flags);
+    _arch_set_info_guest(v, ctxt);
 
     return 0;
 }
diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index 33e41486cb..904624e070 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -73,6 +73,7 @@ int arch_domain_soft_reset(struct domain *d);
 void arch_p2m_set_access_required(struct domain *d, bool access_required);
 
 int arch_set_info_guest(struct vcpu *, vcpu_guest_context_u);
+void _arch_set_info_guest(struct vcpu *, struct vcpu_guest_context *);
 void arch_get_info_guest(struct vcpu *, vcpu_guest_context_u);
 
 int arch_initialise_vcpu(struct vcpu *v, XEN_GUEST_HANDLE_PARAM(void) arg);
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

The implementation consists of:
-Adding PSCI system suspend call as new PSCI function
-Trapping PSCI system_suspend HVC
-Implementing PSCI system suspend call (virtual interface that allows
 guests to suspend themselves)

The PSCI system suspend should be called by a guest from its boot
VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
are left enabled by the guest are assumed to be its wake-up interrupts.
Therefore, a wake-up interrupt triggers the resume of the guest. Guest
should resume regardless of the state of Xen (suspended or not).

When a guest calls PSCI system suspend the respective domain will be
suspended if the following conditions are met:
1) Given resume entry point is not invalid
2) Other (if any) VCPUs of the calling guest are hot-unplugged

If the conditions above are met the calling domain is labeled as
suspended and the calling VCPU is blocked. If nothing else wouldn't
be done the suspended domain would resume from the place where it
called PSCI system suspend. This is expected if processing of the PSCI
system suspend call fails. However, in the case of success the calling
guest should resume (continue execution after the wake-up) from the entry
point which is given as the first argument of the PSCI system suspend
call. In addition to the entry point, the guest expects to start within
the environment whose state matches the state after reset. This means
that the guest should find reset register values, MMU disabled, etc.
Thereby, the context of VCPU should be 'reset' (as if the system is
comming out of reset), the program counter should contain entry point,
which is 1st argument, and r0/x0 should contain context ID which is 2nd
argument of PSCI system suspend call. The context of VCPU is set
accordingly when the PSCI system suspend is processed, so that nothing
needs to be done on resume/wake-up path. However, in order to ensure that
this context doesn't get overwritten by the scheduler when scheduling out
this VCPU (would normally happen after the calling CPU is blocked), we need
to check whether to return early from ctxt_switch_from().

There are variables in domain structure to keep track of domain shutdown.
One of existing shutdown reason is 'suspend' which this patch is using to
track the suspend state of a domain. Those variables are used to determine
whether to early return from ctxt_switch_from() or not.

A suspended domain will resume after the Xen receives an interrupt which is
targeted to the domain, unblocks the domain's VCPU, and schedules it in.
When the VCPU is scheduled in, the VCPU context is already reset, and
contains the right resume entry point in program counter that will be
restored in ctxt_switch_to(). The only thing that needs to be done at this
point is to clear the variables that marked the domain state as suspended.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

---
Changes in v2:

-Fix print to compile for arm32 and to align with Xen coding style
---
 xen/arch/arm/Makefile            |   1 +
 xen/arch/arm/domain.c            |  13 +++
 xen/arch/arm/suspend.c           | 166 +++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vpsci.c             |  19 +++++
 xen/include/asm-arm/perfc_defn.h |   1 +
 xen/include/asm-arm/psci.h       |   2 +
 xen/include/asm-arm/suspend.h    |  16 ++++
 xen/include/xen/sched.h          |   1 +
 8 files changed, 219 insertions(+)
 create mode 100644 xen/arch/arm/suspend.c
 create mode 100644 xen/include/asm-arm/suspend.h

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 23c5d9adbc..744b1a4dc8 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -43,6 +43,7 @@ obj-y += setup.o
 obj-y += shutdown.o
 obj-y += smp.o
 obj-y += smpboot.o
+obj-y += suspend.o
 obj-y += sysctl.o
 obj-y += time.o
 obj-y += traps.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index e594b48d81..7f8105465c 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
     if ( is_idle_vcpu(p) )
         return;
 
+    /* VCPU's context should not be saved if its domain is suspended */
+    if ( p->domain->is_shut_down &&
+        (p->domain->shutdown_code == SHUTDOWN_suspend) )
+        return;
+
     p2m_save_state(p);
 
     /* CP 15 */
@@ -181,6 +186,14 @@ static void ctxt_switch_to(struct vcpu *n)
     if ( is_idle_vcpu(n) )
         return;
 
+    /* If the domain was suspended, it is resuming now */
+    if ( n->domain->is_shut_down &&
+        (n->domain->shutdown_code == SHUTDOWN_suspend) )
+    {
+        n->domain->is_shut_down = 0;
+        n->domain->shutdown_code = SHUTDOWN_CODE_INVALID;
+    }
+
     p2m_restore_state(n);
 
     vpidr = READ_SYSREG32(MIDR_EL1);
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
new file mode 100644
index 0000000000..9eea9214e1
--- /dev/null
+++ b/xen/arch/arm/suspend.c
@@ -0,0 +1,166 @@
+#include <xen/sched.h>
+#include <asm/cpufeature.h>
+#include <asm/event.h>
+#include <asm/psci.h>
+
+/* Reset values of VCPU architecture specific registers */
+static void vcpu_arch_reset(struct vcpu *v)
+{
+    v->arch.ttbr0 = 0;
+    v->arch.ttbr1 = 0;
+    v->arch.ttbcr = 0;
+
+    v->arch.csselr = 0;
+    v->arch.cpacr = 0;
+    v->arch.contextidr = 0;
+    v->arch.tpidr_el0 = 0;
+    v->arch.tpidrro_el0 = 0;
+    v->arch.tpidr_el1 = 0;
+    v->arch.vbar = 0;
+    if ( is_32bit_domain(v->domain) )
+        v->arch.dacr = 0;
+    v->arch.par = 0;
+#if defined(CONFIG_ARM_32)
+    v->arch.mair0 = 0;
+    v->arch.mair1 = 0;
+    v->arch.amair0 = 0;
+    v->arch.amair1 = 0;
+#else
+    v->arch.mair = 0;
+    v->arch.amair = 0;
+#endif
+    /* Fault Status */
+#if defined(CONFIG_ARM_32)
+    v->arch.dfar = 0;
+    v->arch.ifar = 0;
+    v->arch.dfsr = 0;
+#elif defined(CONFIG_ARM_64)
+    v->arch.far = 0;
+    v->arch.esr = 0;
+#endif
+
+    if ( is_32bit_domain(v->domain) )
+        v->arch.ifsr  = 0;
+    v->arch.afsr0 = 0;
+    v->arch.afsr1 = 0;
+
+#ifdef CONFIG_ARM_32
+    v->arch.joscr = 0;
+    v->arch.jmcr = 0;
+#endif
+
+    if ( is_32bit_domain(v->domain) && cpu_has_thumbee )
+    {
+        v->arch.teecr = 0;
+        v->arch.teehbr = 0;
+    }
+}
+
+/*
+ * This function sets the context of current VCPU to the state which is expected
+ * by the guest on resume. The expected VCPU state is:
+ * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
+ * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
+ * 3) All other general purpose and system registers should have reset values
+ *
+ * Note: this function has to return void because it has to always succeed. In
+ * other words, this function is called from virtual PSCI SYSTEM_SUSPEND
+ * implementation, which can return only a limited number of possible errors,
+ * none of which could represent the fact that an error occurred when preparing
+ * the domain for suspend.
+ * Consequently, dynamic memory allocation cannot be done within this function,
+ * because if malloc fails the error has nowhere to propagate.
+ */
+static void vcpu_suspend(register_t epoint, register_t cid)
+{
+    /* Static allocation because dynamic would need a non-void return */
+    static struct vcpu_guest_context ctxt;
+    struct vcpu *v = current;
+
+    /* Make sure that VCPU guest regs are zeroied */
+    memset(&ctxt, 0, sizeof(ctxt));
+
+    /* Set non-zero values to the registers prior to copying */
+    ctxt.user_regs.pc64 = (u64)epoint;
+
+    if ( is_32bit_domain(current->domain) )
+    {
+        ctxt.user_regs.r0_usr = cid;
+        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
+
+        /* Thumb set is allowed only for 32-bit domain */
+        if ( epoint & 1 )
+        {
+            ctxt.user_regs.cpsr |= PSR_THUMB;
+            ctxt.user_regs.pc64 &= ~(u64)1;
+        }
+    }
+#ifdef CONFIG_ARM_64
+    else
+    {
+        ctxt.user_regs.x0 = cid;
+        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;
+    }
+#endif
+    ctxt.sctlr = SCTLR_GUEST_INIT;
+    ctxt.flags = VGCF_online;
+
+    /* Reset architecture specific registers */
+    vcpu_arch_reset(v);
+
+    /* Initialize VCPU registers */
+    _arch_set_info_guest(v, &ctxt);
+}
+
+int32_t domain_suspend(register_t epoint, register_t cid)
+{
+    struct vcpu *v;
+    struct domain *d = current->domain;
+    bool is_thumb = epoint & 1;
+
+    dprintk(XENLOG_DEBUG,
+            "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
+            d->domain_id, epoint, cid);
+
+    /* THUMB set is not allowed with 64-bit domain */
+    if ( is_64bit_domain(d) && is_thumb )
+        return PSCI_INVALID_ADDRESS;
+
+    /* Ensure that all CPUs other than the calling one are offline */
+    for_each_vcpu ( d, v )
+    {
+        if ( v != current && is_vcpu_online(v) )
+            return PSCI_DENIED;
+    }
+
+    /*
+     * Prepare the calling VCPU for suspend (reset its context, save entry point
+     * into pc and context ID into r0/x0 as specified by PSCI SYSTEM_SUSPEND)
+     */
+    vcpu_suspend(epoint, cid);
+
+    /*
+     * Set the domain state to suspended (will be cleared when the domain
+     * resumes, i.e. VCPU of this domain gets scheduled in).
+     */
+    d->is_shut_down = 1;
+    d->shutdown_code = SHUTDOWN_suspend;
+
+    /*
+     * The calling domain is suspended by blocking its last running VCPU. If an
+     * event is pending the domain will resume right away (VCPU will not block,
+     * but when scheduled in it will resume from the given entry point).
+     */
+    vcpu_block_unless_event_pending(current);
+
+    return PSCI_SUCCESS;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vpsci.c b/xen/arch/arm/vpsci.c
index 9f4e5b8844..f7922be0c5 100644
--- a/xen/arch/arm/vpsci.c
+++ b/xen/arch/arm/vpsci.c
@@ -18,6 +18,7 @@
 #include <asm/vgic.h>
 #include <asm/vpsci.h>
 #include <asm/event.h>
+#include <asm/suspend.h>
 
 #include <public/sched.h>
 
@@ -210,6 +211,11 @@ static void do_psci_0_2_system_reset(void)
     domain_shutdown(d,SHUTDOWN_reboot);
 }
 
+static int32_t do_psci_1_0_system_suspend(register_t epoint, register_t cid)
+{
+    return domain_suspend(epoint, cid);
+}
+
 static int32_t do_psci_1_0_features(uint32_t psci_func_id)
 {
     /* /!\ Ordered by function ID and not name */
@@ -227,6 +233,8 @@ static int32_t do_psci_1_0_features(uint32_t psci_func_id)
     case PSCI_0_2_FN32_SYSTEM_OFF:
     case PSCI_0_2_FN32_SYSTEM_RESET:
     case PSCI_1_0_FN32_PSCI_FEATURES:
+    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
+    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
     case ARM_SMCCC_VERSION_FID:
         return 0;
     default:
@@ -357,6 +365,17 @@ bool do_vpsci_0_2_call(struct cpu_user_regs *regs, uint32_t fid)
         return true;
     }
 
+    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
+    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
+    {
+        register_t epoint = PSCI_ARG(regs,1);
+        register_t cid = PSCI_ARG(regs,2);
+
+        perfc_incr(vpsci_system_suspend);
+        PSCI_SET_RESULT(regs, do_psci_1_0_system_suspend(epoint, cid));
+        return true;
+    }
+
     default:
         return false;
     }
diff --git a/xen/include/asm-arm/perfc_defn.h b/xen/include/asm-arm/perfc_defn.h
index 8922e9525a..a02d0adea8 100644
--- a/xen/include/asm-arm/perfc_defn.h
+++ b/xen/include/asm-arm/perfc_defn.h
@@ -32,6 +32,7 @@ PERFCOUNTER(vpsci_system_reset,        "vpsci: system_reset")
 PERFCOUNTER(vpsci_cpu_suspend,         "vpsci: cpu_suspend")
 PERFCOUNTER(vpsci_cpu_affinity_info,   "vpsci: cpu_affinity_info")
 PERFCOUNTER(vpsci_features,            "vpsci: features")
+PERFCOUNTER(vpsci_system_suspend,      "vpsci: system_suspend")
 
 PERFCOUNTER(vcpu_kick,                 "vcpu: notify other vcpu")
 
diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
index 832f77afff..26462d0c47 100644
--- a/xen/include/asm-arm/psci.h
+++ b/xen/include/asm-arm/psci.h
@@ -43,10 +43,12 @@ void call_psci_system_reset(void);
 #define PSCI_0_2_FN32_SYSTEM_OFF          PSCI_0_2_FN32(8)
 #define PSCI_0_2_FN32_SYSTEM_RESET        PSCI_0_2_FN32(9)
 #define PSCI_1_0_FN32_PSCI_FEATURES       PSCI_0_2_FN32(10)
+#define PSCI_1_0_FN32_SYSTEM_SUSPEND      PSCI_0_2_FN32(14)
 
 #define PSCI_0_2_FN64_CPU_SUSPEND         PSCI_0_2_FN64(1)
 #define PSCI_0_2_FN64_CPU_ON              PSCI_0_2_FN64(3)
 #define PSCI_0_2_FN64_AFFINITY_INFO       PSCI_0_2_FN64(4)
+#define PSCI_1_0_FN64_SYSTEM_SUSPEND      PSCI_0_2_FN64(14)
 
 /* PSCI v0.2 affinity level state returned by AFFINITY_INFO */
 #define PSCI_0_2_AFFINITY_LEVEL_ON      0
diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
new file mode 100644
index 0000000000..de787d296a
--- /dev/null
+++ b/xen/include/asm-arm/suspend.h
@@ -0,0 +1,16 @@
+#ifndef __ASM_ARM_SUSPEND_H__
+#define __ASM_ARM_SUSPEND_H__
+
+int32_t domain_suspend(register_t epoint, register_t cid);
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 3171eabfd6..1f4e86524f 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -24,6 +24,7 @@
 #include <xen/wait.h>
 #include <public/xen.h>
 #include <public/domctl.h>
+#include <public/sched.h>
 #include <public/sysctl.h>
 #include <public/vcpu.h>
 #include <public/vm_event.h>
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Mirela Simonovic]

GIC and virtual timer context must be saved when the domain suspends.
This is done by moving the respective code in ctxt_switch_from()
before the return that happens if the domain suspended.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/domain.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 7f8105465c..bebe3238e8 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
     if ( is_idle_vcpu(p) )
         return;
 
+    /* VGIC */
+    gic_save_state(p);
+
+    /* Arch timer */
+    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
+    virt_timer_save(p);
+
     /* VCPU's context should not be saved if its domain is suspended */
     if ( p->domain->is_shut_down &&
         (p->domain->shutdown_code == SHUTDOWN_suspend) )
@@ -115,10 +122,6 @@ static void ctxt_switch_from(struct vcpu *p)
     p->arch.tpidrro_el0 = READ_SYSREG(TPIDRRO_EL0);
     p->arch.tpidr_el1 = READ_SYSREG(TPIDR_EL1);
 
-    /* Arch timer */
-    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
-    virt_timer_save(p);
-
     if ( is_32bit_domain(p->domain) && cpu_has_thumbee )
     {
         p->arch.teecr = READ_SYSREG32(TEECR32_EL1);
@@ -170,9 +173,6 @@ static void ctxt_switch_from(struct vcpu *p)
     /* VFP */
     vfp_save_state(p);
 
-    /* VGIC */
-    gic_save_state(p);
-
     isb();
 }
 
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Mirela Simonovic]

While a domain is suspended its watchdogs must be paused. Otherwise,
if the domain stays in the suspend state for a longer period of time
compared to the watchdog period, the domain would be shutdown on resume.
Proper solution to this problem is to stop (suspend) the watchdog timers
after the domain suspends and to restart (resume) the watchdog timers
before the domain resumes. The suspend/resume of watchdog timers is done
in Xen and is invisible to the guests.
Just before the domain starts resuming the watchdog timers are programmed
with a new expire value. The new expire value is equal to the expire
value prior to suspend plus the period of time for which the watchdog
was on pause (the domain was suspended). In order to save the suspend
timestamp and afterwards calculate for how long the domain was suspended,
a 'suspended' variable is added into the generic timer structure.
Programming of the timers is triggered when a VCPU of the suspended
domain is scheduled in on resume.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

---
Changes in v2:

-Fixed typo in commit message
---
 xen/arch/arm/domain.c   |  1 +
 xen/arch/arm/suspend.c  |  3 +++
 xen/common/schedule.c   | 38 ++++++++++++++++++++++++++++++++++++++
 xen/include/xen/sched.h |  7 +++++++
 xen/include/xen/timer.h |  3 +++
 5 files changed, 52 insertions(+)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index bebe3238e8..68f038458f 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -192,6 +192,7 @@ static void ctxt_switch_to(struct vcpu *n)
     {
         n->domain->is_shut_down = 0;
         n->domain->shutdown_code = SHUTDOWN_CODE_INVALID;
+        watchdog_domain_resume(n->domain);
     }
 
     p2m_restore_state(n);
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 9eea9214e1..f2338e41db 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -146,6 +146,9 @@ int32_t domain_suspend(register_t epoint, register_t cid)
     d->is_shut_down = 1;
     d->shutdown_code = SHUTDOWN_suspend;
 
+    /* Disable watchdogs of this domain */
+    watchdog_domain_suspend(d);
+
     /*
      * The calling domain is suspended by blocking its last running VCPU. If an
      * event is pending the domain will resume right away (VCPU will not block,
diff --git a/xen/common/schedule.c b/xen/common/schedule.c
index a957c5e57c..9b2882a168 100644
--- a/xen/common/schedule.c
+++ b/xen/common/schedule.c
@@ -1172,6 +1172,44 @@ void watchdog_domain_destroy(struct domain *d)
         kill_timer(&d->watchdog_timer[i]);
 }
 
+void watchdog_domain_suspend(struct domain *d)
+{
+    unsigned int i;
+
+    spin_lock(&d->watchdog_lock);
+
+    for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
+    {
+        if ( test_bit(i, &d->watchdog_inuse_map) )
+        {
+            struct timer *timer = &d->watchdog_timer[i];
+            timer->suspended = NOW();
+            stop_timer(timer);
+        }
+    }
+
+    spin_unlock(&d->watchdog_lock);
+}
+
+void watchdog_domain_resume(struct domain *d)
+{
+    unsigned int i;
+
+    spin_lock(&d->watchdog_lock);
+
+    for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
+    {
+        if ( test_bit(i, &d->watchdog_inuse_map) )
+        {
+            struct timer *timer = &d->watchdog_timer[i];
+            s_time_t sleep_interval = NOW() - timer->suspended;
+            set_timer(timer, timer->expires + sleep_interval);
+        }
+    }
+
+    spin_unlock(&d->watchdog_lock);
+}
+
 int vcpu_pin_override(struct vcpu *v, int cpu)
 {
     spinlock_t *lock;
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 1f4e86524f..366acaf69a 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -862,6 +862,13 @@ void watchdog_domain_init(struct domain *d);
 void watchdog_domain_destroy(struct domain *d);
 
 /*
+ * Suspend/resume watchdogs of domain (while the domain is suspended its
+ * watchdogs should be on pause)
+ */
+void watchdog_domain_suspend(struct domain *d);
+void watchdog_domain_resume(struct domain *d);
+
+/*
  * Use this check when the following are both true:
  *  - Using this feature or interface requires full access to the hardware
  *    (that is, this would not be suitable for a driver domain)
diff --git a/xen/include/xen/timer.h b/xen/include/xen/timer.h
index 4513260b0d..4a88af7de0 100644
--- a/xen/include/xen/timer.h
+++ b/xen/include/xen/timer.h
@@ -18,6 +18,9 @@ struct timer {
     /* System time expiry value (nanoseconds since boot). */
     s_time_t expires;
 
+    /* Suspend timestamp value (nanoseconds since boot). */
+    s_time_t suspended;
+
     /* Position in active-timer data structure. */
     union {
         /* Timer-heap offset (TIMER_STATUS_in_heap). */
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

When Dom0 finalizes its suspend procedure the suspend of Xen is
triggered by calling system_suspend(). Dom0 finalizes the suspend from
its boot core (VCPU#0), which could be mapped to any physical CPU,
i.e. the system_suspend() function could be executed by any physical
CPU. Since Xen suspend procedure has to be run by the boot CPU
(non-boot CPUs will be disabled at some point in suspend procedure),
system_suspend() execution has to continue on CPU#0.

When the system_suspend() returns 0, it means that the system was
suspended and it is coming out of the resume procedure. Regardless
of the system_suspend() return value, after this function returns
Xen is fully functional, and its state, including all devices and data
structures, matches the state prior to calling system_suspend().
The status is returned by system_suspend() for debugging/logging
purposes and function prototype compatibility.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index f2338e41db..21b45f8248 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
     _arch_set_info_guest(v, &ctxt);
 }
 
+/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
+static long system_suspend(void *data)
+{
+    BUG_ON(system_state != SYS_STATE_active);
+
+    return -ENOSYS;
+}
+
 int32_t domain_suspend(register_t epoint, register_t cid)
 {
     struct vcpu *v;
     struct domain *d = current->domain;
     bool is_thumb = epoint & 1;
+    int status;
 
     dprintk(XENLOG_DEBUG,
             "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
@@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
      */
     vcpu_block_unless_event_pending(current);
 
+    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
+    if ( is_hardware_domain(d) )
+    {
+        /*
+         * system_suspend should be called when Dom0 finalizes the suspend
+         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
+         * be mapped to any PCPU (this function could be executed by any PCPU).
+         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
+         * PCPUs will be disabled during the suspend).
+         */
+        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
+        /*
+         * If an error happened, there is nothing that needs to be done here
+         * because the system_suspend always returns in fully functional state
+         * no matter what the outcome of suspend procedure is. If the system
+         * suspended successfully the function will return 0 after the resume.
+         * Otherwise, if an error is returned it means Xen did not suspended,
+         * but it is still in the same state as if the system_suspend was never
+         * called. We dump a debug message in case of an error for debugging/
+         * logging purpose.
+         */
+        if ( status )
+            dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
+    }
+
     return PSCI_SUCCESS;
 }
 
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 06/18] xen/x86: Move freeze/thaw_domains into common files

[Mirela Simonovic]

These functions will be reused by suspend/resume support for ARM.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/x86/acpi/power.c | 28 ----------------------------
 xen/common/domain.c       | 29 +++++++++++++++++++++++++++++
 xen/include/xen/sched.h   |  3 +++
 3 files changed, 32 insertions(+), 28 deletions(-)

diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c
index 93e967fe8f..794750e45b 100644
--- a/xen/arch/x86/acpi/power.c
+++ b/xen/arch/x86/acpi/power.c
@@ -109,34 +109,6 @@ static void device_power_up(enum dev_power_saved saved)
     }
 }
 
-static void freeze_domains(void)
-{
-    struct domain *d;
-
-    rcu_read_lock(&domlist_read_lock);
-    /*
-     * Note that we iterate in order of domain-id. Hence we will pause dom0
-     * first which is required for correctness (as only dom0 can add domains to
-     * the domain list). Otherwise we could miss concurrently-created domains.
-     */
-    for_each_domain ( d )
-        domain_pause(d);
-    rcu_read_unlock(&domlist_read_lock);
-}
-
-static void thaw_domains(void)
-{
-    struct domain *d;
-
-    rcu_read_lock(&domlist_read_lock);
-    for_each_domain ( d )
-    {
-        restore_vcpu_affinity(d);
-        domain_unpause(d);
-    }
-    rcu_read_unlock(&domlist_read_lock);
-}
-
 static void acpi_sleep_prepare(u32 state)
 {
     void *wakeup_vector_va;
diff --git a/xen/common/domain.c b/xen/common/domain.c
index d6650f0656..fdd00dc661 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -1666,6 +1666,35 @@ int continue_hypercall_on_cpu(
     return 0;
 }
 
+
+void freeze_domains(void)
+{
+    struct domain *d;
+
+    rcu_read_lock(&domlist_read_lock);
+    /*
+     * Note that we iterate in order of domain-id. Hence we will pause dom0
+     * first which is required for correctness (as only dom0 can add domains to
+     * the domain list). Otherwise we could miss concurrently-created domains.
+     */
+    for_each_domain ( d )
+        domain_pause(d);
+    rcu_read_unlock(&domlist_read_lock);
+}
+
+void thaw_domains(void)
+{
+    struct domain *d;
+
+    rcu_read_lock(&domlist_read_lock);
+    for_each_domain ( d )
+    {
+        restore_vcpu_affinity(d);
+        domain_unpause(d);
+    }
+    rcu_read_unlock(&domlist_read_lock);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 366acaf69a..c7a6d9504a 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -821,6 +821,9 @@ static inline int domain_pause_by_systemcontroller_nosync(struct domain *d)
 void domain_pause_except_self(struct domain *d);
 void domain_unpause_except_self(struct domain *d);
 
+void freeze_domains(void);
+void thaw_domains(void);
+
 void cpu_init(void);
 
 struct scheduler;
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 07/18] xen/arm: Freeze domains on suspend and thaw them on resume

[Mirela Simonovic]

Freeze and thaw of domains is reused as implemented for x86. In
addition, system_state variable is updated to represent the actual
state of the system.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 21b45f8248..575afd5eb8 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -117,6 +117,14 @@ static long system_suspend(void *data)
 {
     BUG_ON(system_state != SYS_STATE_active);
 
+    system_state = SYS_STATE_suspend;
+    freeze_domains();
+
+    system_state = SYS_STATE_resume;
+
+    thaw_domains();
+    system_state = SYS_STATE_active;
+
     return -ENOSYS;
 }
 
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Mirela Simonovic]

Non-boot CPUs have to be disabled on suspend and enabled on resume
(hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
CPU_OFF to be called by each non-boot CPU. Depending on the underlying
platform capabilities, this may lead to the physical powering down of
CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
each non-boot CPU).

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 575afd5eb8..dae1b1f7d6 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -1,4 +1,5 @@
 #include <xen/sched.h>
+#include <xen/cpu.h>
 #include <asm/cpufeature.h>
 #include <asm/event.h>
 #include <asm/psci.h>
@@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, register_t cid)
 /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
 static long system_suspend(void *data)
 {
+    int status;
+
     BUG_ON(system_state != SYS_STATE_active);
 
     system_state = SYS_STATE_suspend;
     freeze_domains();
 
+    status = disable_nonboot_cpus();
+    if ( status )
+    {
+        system_state = SYS_STATE_resume;
+        goto resume_nonboot_cpus;
+    }
+
     system_state = SYS_STATE_resume;
 
+resume_nonboot_cpus:
+    enable_nonboot_cpus();
     thaw_domains();
     system_state = SYS_STATE_active;
+    dsb(sy);
 
-    return -ENOSYS;
+    return status;
 }
 
 int32_t domain_suspend(register_t epoint, register_t cid)
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 09/18] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume

[Mirela Simonovic]

The rcu_barrier() has to be added to ensure that the per cpu area is
freed before a non-boot CPU tries to initialize it (_free_percpu_area()
has to be called before the init_percpu_area()). This scenario occurs
when non-boot CPUs are hot-unplugged on suspend and hotplugged on resume.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index dae1b1f7d6..8e8e531d61 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -133,6 +133,7 @@ static long system_suspend(void *data)
     system_state = SYS_STATE_resume;
 
 resume_nonboot_cpus:
+    rcu_barrier();
     enable_nonboot_cpus();
     thaw_domains();
     system_state = SYS_STATE_active;
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Mirela Simonovic]

System suspend may lead to a state where GIC would be powered down.
Therefore, Xen should save/restore the context of GIC on suspend/resume.
Note that the context consists of states of registers which are
controlled by the hypervisor. Other GIC registers which are accessible
by guests are saved/restored on context switch.
Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
the GIC.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/gic-v2.c     | 147 ++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic.c        |  27 +++++++++
 xen/include/asm-arm/gic.h |   8 +++
 3 files changed, 182 insertions(+)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index e7eb01f30a..bb52b64ecb 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
 /* Maximum cpu interface per GIC */
 #define NR_GIC_CPU_IF 8
 
+/* GICv2 registers to be saved/restored on system suspend/resume */
+struct gicv2_context {
+    /* GICC context */
+    uint32_t gicc_ctlr;
+    uint32_t gicc_pmr;
+    uint32_t gicc_bpr;
+    /* GICD context */
+    uint32_t gicd_ctlr;
+    uint32_t *gicd_isenabler;
+    uint32_t *gicd_isactiver;
+    uint32_t *gicd_ipriorityr;
+    uint32_t *gicd_itargetsr;
+    uint32_t *gicd_icfgr;
+};
+
+static struct gicv2_context gicv2_context;
+
+static void gicv2_alloc_context(struct gicv2_context *gc);
+
 static inline void writeb_gicd(uint8_t val, unsigned int offset)
 {
     writeb_relaxed(val, gicv2.map_dbase + offset);
@@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
 
     spin_unlock(&gicv2.lock);
 
+    /* Allocate memory to be used for saving GIC context during the suspend */
+    gicv2_alloc_context(&gicv2_context);
+
     return 0;
 }
 
@@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
     BUG();
 }
 
+static void gicv2_alloc_context(struct gicv2_context *gc)
+{
+    uint32_t n = gicv2_info.nr_lines;
+
+    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+    if ( !gc->gicd_isenabler )
+        return;
+
+    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+    if ( !gc->gicd_isactiver )
+        goto free_gicd_isenabler;
+
+    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+    if ( !gc->gicd_itargetsr )
+        goto free_gicd_isactiver;
+
+    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+    if ( !gc->gicd_ipriorityr )
+        goto free_gicd_itargetsr;
+
+    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
+    if ( gc->gicd_icfgr )
+        return;
+
+    xfree(gc->gicd_ipriorityr);
+
+free_gicd_itargetsr:
+    xfree(gc->gicd_itargetsr);
+
+free_gicd_isactiver:
+    xfree(gc->gicd_isactiver);
+
+free_gicd_isenabler:
+    xfree(gc->gicd_isenabler);
+    gc->gicd_isenabler = NULL;
+}
+
+static int gicv2_suspend(void)
+{
+    int i;
+
+    /* Save GICC configuration */
+    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
+    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
+    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
+
+    /* If gicv2_alloc_context() hasn't allocated memory, return */
+    if ( !gicv2_context.gicd_isenabler )
+        return -ENOMEM;
+
+    /* Save GICD configuration */
+    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
+        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
+
+    return 0;
+}
+
+static void gicv2_resume(void)
+{
+    int i;
+
+    ASSERT(gicv2_context.gicd_isenabler);
+    ASSERT(gicv2_context.gicd_isactiver);
+    ASSERT(gicv2_context.gicd_ipriorityr);
+    ASSERT(gicv2_context.gicd_itargetsr);
+    ASSERT(gicv2_context.gicd_icfgr);
+
+    /* Disable CPU interface and distributor */
+    writel_gicc(0, GICC_CTLR);
+    writel_gicd(0, GICD_CTLR);
+    isb();
+
+    /* Restore GICD configuration */
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
+
+    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
+        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
+
+    /* Make sure all registers are restored and enable distributor */
+    isb();
+    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
+
+    /* Restore GIC CPU interface configuration */
+    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
+    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
+    isb();
+
+    /* Enable GIC CPU interface */
+    writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | GICC_CTL_EOI,
+                GICC_CTLR);
+    isb();
+}
+
 const static struct gic_hw_operations gicv2_ops = {
     .info                = &gicv2_info,
     .init                = gicv2_init,
@@ -1351,6 +1496,8 @@ const static struct gic_hw_operations gicv2_ops = {
     .map_hwdom_extra_mappings = gicv2_map_hwdown_extra_mappings,
     .iomem_deny_access   = gicv2_iomem_deny_access,
     .do_LPI              = gicv2_do_LPI,
+    .suspend             = gicv2_suspend,
+    .resume              = gicv2_resume,
 };
 
 /* Set up the GIC */
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index e524ad583d..6e98f43691 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -464,6 +464,33 @@ int gic_iomem_deny_access(const struct domain *d)
     return gic_hw_ops->iomem_deny_access(d);
 }
 
+int gic_suspend(void)
+{
+    /* Must be called by boot CPU#0 with interrupts disabled */
+    ASSERT(!local_irq_is_enabled());
+    ASSERT(!smp_processor_id());
+
+    if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
+        return -ENOSYS;
+
+    gic_hw_ops->suspend();
+
+    return 0;
+}
+
+void gic_resume(void)
+{
+    /*
+     * Must be called by boot CPU#0 with interrupts disabled after gic_suspend
+     * has returned successfully.
+     */
+    ASSERT(!local_irq_is_enabled());
+    ASSERT(!smp_processor_id());
+    ASSERT(gic_hw_ops->resume);
+
+    gic_hw_ops->resume();
+}
+
 static int cpu_gic_callback(struct notifier_block *nfb,
                             unsigned long action,
                             void *hcpu)
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 22fa122e52..46066caac8 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -277,6 +277,10 @@ extern int gicv_setup(struct domain *d);
 extern void gic_save_state(struct vcpu *v);
 extern void gic_restore_state(struct vcpu *v);
 
+/* Suspend/resume */
+extern int gic_suspend(void);
+extern void gic_resume(void);
+
 /* SGI (AKA IPIs) */
 enum gic_sgi {
     GIC_SGI_EVENT_CHECK = 0,
@@ -390,6 +394,10 @@ struct gic_hw_operations {
     int (*iomem_deny_access)(const struct domain *d);
     /* Handle LPIs, which require special handling */
     void (*do_LPI)(unsigned int lpi);
+    /* Save GIC configuration due to the system suspend */
+    int (*suspend)(void);
+    /* Restore GIC configuration due to the system resume */
+    void (*resume)(void);
 };
 
 extern const struct gic_hw_operations *gic_hw_ops;
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 11/18] xen/arm: Suspend/resume GIC on system suspend/resume

[Mirela Simonovic]

GIC state is saved on system suspend by calling gic_suspend
(this function does not change current state of the GIC but only
saves the values of configuration registers).
The state of GIC has to be restored by calling gic_resume, but only
if the gic_suspend has succeeded. If gic_suspend fails, we'll just
restore interrupts configuration and abort suspend.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 8e8e531d61..b7940fe03f 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -117,6 +117,7 @@ static void vcpu_suspend(register_t epoint, register_t cid)
 static long system_suspend(void *data)
 {
     int status;
+    unsigned long flags;
 
     BUG_ON(system_state != SYS_STATE_active);
 
@@ -130,8 +131,21 @@ static long system_suspend(void *data)
         goto resume_nonboot_cpus;
     }
 
+    local_irq_save(flags);
+    status = gic_suspend();
+    if ( status )
+    {
+        system_state = SYS_STATE_resume;
+        goto resume_irqs;
+    }
+
     system_state = SYS_STATE_resume;
 
+    gic_resume();
+
+resume_irqs:
+    local_irq_restore(flags);
+
 resume_nonboot_cpus:
     rcu_barrier();
     enable_nonboot_cpus();
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 12/18] xen/arm: Suspend/resume timer interrupt generation

[Mirela Simonovic]

Timer interrupts have to be disabled while the system is in suspend.
Otherwise, a timer interrupt would fire and wake-up the system.
Suspending the timer interrupts consists of disabling physical EL1
and EL2 timers. The resume consists only of raising timer softirq,
which will trigger the generic timer code to reprogram the EL2 timer
as needed. Enabling of EL1 physical timer will be triggered by an
entity which uses it.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c     |  4 ++++
 xen/arch/arm/time.c        | 22 ++++++++++++++++++++++
 xen/include/asm-arm/time.h |  3 +++
 3 files changed, 29 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index b7940fe03f..d1b48c339a 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -131,6 +131,8 @@ static long system_suspend(void *data)
         goto resume_nonboot_cpus;
     }
 
+    time_suspend();
+
     local_irq_save(flags);
     status = gic_suspend();
     if ( status )
@@ -146,6 +148,8 @@ static long system_suspend(void *data)
 resume_irqs:
     local_irq_restore(flags);
 
+    time_resume();
+
 resume_nonboot_cpus:
     rcu_barrier();
     enable_nonboot_cpus();
diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index bbccee742e..aaefcb6441 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -357,6 +357,28 @@ void domain_set_time_offset(struct domain *d, int64_t time_offset_seconds)
     /* XXX update guest visible wallclock time */
 }
 
+void time_suspend(void)
+{
+    /* Disable physical EL1 timer */
+    WRITE_SYSREG32(0, CNTP_CTL_EL0);
+
+    /* Disable hypervisor's timer */
+    WRITE_SYSREG32(0, CNTHP_CTL_EL2);
+    isb();
+}
+
+void time_resume(void)
+{
+    /*
+     * Raising timer softirq will trigger generic timer code to reprogram_timer
+     * with the correct timeout value (which is not known here). There is no
+     * need to do anything else in order to recover the time keeping from power
+     * down, because the system counter is not affected by the power down (it
+     * resides out of the ARM's cluster in an always-on part of the SoC).
+     */
+    raise_softirq(TIMER_SOFTIRQ);
+}
+
 static int cpu_time_callback(struct notifier_block *nfb,
                              unsigned long action,
                              void *hcpu)
diff --git a/xen/include/asm-arm/time.h b/xen/include/asm-arm/time.h
index 19a4515e72..9275b39a4a 100644
--- a/xen/include/asm-arm/time.h
+++ b/xen/include/asm-arm/time.h
@@ -42,6 +42,9 @@ extern uint64_t ns_to_ticks(s_time_t ns);
 
 void preinit_xen_time(void);
 
+void time_suspend(void);
+void time_resume(void);
+
 #endif /* __ARM_TIME_H__ */
 /*
  * Local variables:
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 13/18] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)

[Mirela Simonovic]

PSCI system suspend function shall be invoked to finalize Xen suspend
procedure. Resume entry point, which needs to be passed via 1st argument
of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
is just a placeholder that will be implemented in assembly. Context ID,
which is 2nd argument of system suspend PSCI call, is unused, as in Linux.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

---
Changes in v2:

-The commit message was stale - referring to the do_suspend function
that has been renamed long time ago. Fixed commit message
---
 xen/arch/arm/arm64/entry.S    |  3 +++
 xen/arch/arm/psci.c           | 16 ++++++++++++++++
 xen/arch/arm/suspend.c        |  4 ++++
 xen/include/asm-arm/psci.h    |  1 +
 xen/include/asm-arm/suspend.h |  1 +
 5 files changed, 25 insertions(+)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 97b05f53ea..dbc4717903 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -533,6 +533,9 @@ ENTRY(__context_switch)
         mov     sp, x9
         ret
 
+ENTRY(hyp_resume)
+        b .
+
 /*
  * Local variables:
  * mode: ASM
diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
index a93121f43b..b100bd8ad2 100644
--- a/xen/arch/arm/psci.c
+++ b/xen/arch/arm/psci.c
@@ -24,6 +24,7 @@
 #include <asm/cpufeature.h>
 #include <asm/psci.h>
 #include <asm/acpi.h>
+#include <asm/suspend.h>
 
 /*
  * While a 64-bit OS can make calls with SMC32 calling conventions, for
@@ -67,6 +68,21 @@ void call_psci_cpu_off(void)
     }
 }
 
+int call_psci_system_suspend(void)
+{
+#ifdef CONFIG_ARM_64
+    struct arm_smccc_res res;
+
+    /* 2nd argument (context ID) is not used */
+    arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), &res);
+
+    return PSCI_RET(res);
+#else
+    /* not supported */
+    return 1;
+#endif
+}
+
 void call_psci_system_off(void)
 {
     if ( psci_ver > PSCI_VERSION(0, 1) )
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index d1b48c339a..37926374bc 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -141,6 +141,10 @@ static long system_suspend(void *data)
         goto resume_irqs;
     }
 
+    status = call_psci_system_suspend();
+    if ( status )
+        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
+
     system_state = SYS_STATE_resume;
 
     gic_resume();
diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
index 26462d0c47..9f6116a224 100644
--- a/xen/include/asm-arm/psci.h
+++ b/xen/include/asm-arm/psci.h
@@ -20,6 +20,7 @@ extern uint32_t psci_ver;
 
 int psci_init(void);
 int call_psci_cpu_on(int cpu);
+int call_psci_system_suspend(void);
 void call_psci_cpu_off(void);
 void call_psci_system_off(void);
 void call_psci_system_reset(void);
diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
index de787d296a..7604e2e2e2 100644
--- a/xen/include/asm-arm/suspend.h
+++ b/xen/include/asm-arm/suspend.h
@@ -2,6 +2,7 @@
 #define __ASM_ARM_SUSPEND_H__
 
 int32_t domain_suspend(register_t epoint, register_t cid);
+void hyp_resume(void);
 
 #endif
 
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 14/18] xen/arm: Convert setting MMU page tables code into a routine

[Mirela Simonovic]

From: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

The code that sets up MMU page tables during the boot is also
needed when the system resumes. Convert that code in head.S
into a routine so the resume code can use it as well. This patch
does not include any functional change.

Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
---
 xen/arch/arm/arm64/head.S | 265 ++++++++++++++++++++++++----------------------
 1 file changed, 138 insertions(+), 127 deletions(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index ef87b5c254..f95390dcfe 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -379,134 +379,10 @@ skip_bss:
          * than SP_EL0. */
         msr spsel, #1
 
-        /* Rebuild the boot pagetable's first-level entries. The structure
-         * is described in mm.c.
-         *
-         * After the CPU enables paging it will add the fixmap mapping
-         * to these page tables, however this may clash with the 1:1
-         * mapping. So each CPU must rebuild the page tables here with
-         * the 1:1 in place. */
+        /* If setting up page_tables are not successful, fail to boot */
+        bl    setup_page_tables
+        cbz   x25, fail
 
-        /* If Xen is loaded at exactly XEN_VIRT_START then we don't
-         * need an additional 1:1 mapping, the virtual mapping will
-         * suffice.
-         */
-        cmp   x19, #XEN_VIRT_START
-        cset  x25, eq                /* x25 := identity map in place, or not */
-
-        /* Write Xen's PT's paddr into TTBR0_EL2 */
-        load_paddr x4, boot_pgtable
-        msr   TTBR0_EL2, x4
-
-        /* Setup boot_pgtable: */
-        load_paddr x1, boot_first
-
-        /* ... map boot_first in boot_pgtable[0] */
-        mov   x3, #PT_PT             /* x2 := table map of boot_first */
-        orr   x2, x1, x3             /*       + rights for linear PT */
-        str   x2, [x4, #0]           /* Map it in slot 0 */
-
-        /* ... map of paddr(start) in boot_pgtable+boot_first_id */
-        lsr   x1, x19, #ZEROETH_SHIFT/* Offset of base paddr in boot_pgtable */
-        cbz   x1, 1f                 /* It's in slot 0, map in boot_first
-                                      * or boot_second later on */
-
-        /* Level zero does not support superpage mappings, so we have
-         * to use an extra first level page in which we create a 1GB mapping.
-         */
-        load_paddr x2, boot_first_id
-
-        mov   x3, #PT_PT             /* x2 := table map of boot_first_id */
-        orr   x2, x2, x3             /*       + rights for linear PT */
-        lsl   x1, x1, #3             /* x1 := Slot offset */
-        str   x2, [x4, x1]
-
-        load_paddr x4, boot_first_id
-
-        lsr   x1, x19, #FIRST_SHIFT  /* x1 := Offset of base paddr in boot_first_id */
-        lsl   x2, x1, #FIRST_SHIFT   /* x2 := Base address for 1GB mapping */
-        mov   x3, #PT_MEM            /* x2 := Section map */
-        orr   x2, x2, x3
-        and   x1, x1, #LPAE_ENTRY_MASK /* x1 := Slot offset */
-        lsl   x1, x1, #3
-        str   x2, [x4, x1]           /* Mapping of paddr(start) */
-        mov   x25, #1                /* x25 := identity map now in place */
-
-1:      /* Setup boot_first: */
-        load_paddr x4, boot_first   /* Next level into boot_first */
-
-        /* ... map boot_second in boot_first[0] */
-        load_paddr x1, boot_second
-        mov   x3, #PT_PT             /* x2 := table map of boot_second */
-        orr   x2, x1, x3             /*       + rights for linear PT */
-        str   x2, [x4, #0]           /* Map it in slot 0 */
-
-        /* ... map of paddr(start) in boot_first */
-        cbnz  x25, 1f                /* x25 is set if already created */
-        lsr   x2, x19, #FIRST_SHIFT  /* x2 := Offset of base paddr in boot_first */
-        and   x1, x2, #LPAE_ENTRY_MASK /* x1 := Slot to use */
-        cbz   x1, 1f                 /* It's in slot 0, map in boot_second */
-
-        lsl   x2, x2, #FIRST_SHIFT   /* Base address for 1GB mapping */
-        mov   x3, #PT_MEM            /* x2 := Section map */
-        orr   x2, x2, x3
-        lsl   x1, x1, #3             /* x1 := Slot offset */
-        str   x2, [x4, x1]           /* Create mapping of paddr(start)*/
-        mov   x25, #1                /* x25 := identity map now in place */
-
-1:      /* Setup boot_second: */
-        load_paddr x4, boot_second
-
-        /* ... map boot_third in boot_second[1] */
-        load_paddr x1, boot_third
-        mov   x3, #PT_PT             /* x2 := table map of boot_third */
-        orr   x2, x1, x3             /*       + rights for linear PT */
-        str   x2, [x4, #8]           /* Map it in slot 1 */
-
-        /* ... map of paddr(start) in boot_second */
-        cbnz  x25, 1f                /* x25 is set if already created */
-        lsr   x2, x19, #SECOND_SHIFT /* x2 := Offset of base paddr in boot_second */
-        and   x1, x2, #LPAE_ENTRY_MASK /* x1 := Slot to use */
-        cmp   x1, #1
-        b.eq  virtphys_clash         /* It's in slot 1, which we cannot handle */
-
-        lsl   x2, x2, #SECOND_SHIFT  /* Base address for 2MB mapping */
-        mov   x3, #PT_MEM            /* x2 := Section map */
-        orr   x2, x2, x3
-        lsl   x1, x1, #3             /* x1 := Slot offset */
-        str   x2, [x4, x1]           /* Create mapping of paddr(start)*/
-        mov   x25, #1                /* x25 := identity map now in place */
-
-1:      /* Setup boot_third: */
-        load_paddr x4, boot_third
-
-        lsr   x2, x19, #THIRD_SHIFT  /* Base address for 4K mapping */
-        lsl   x2, x2, #THIRD_SHIFT
-        mov   x3, #PT_MEM_L3         /* x2 := Section map */
-        orr   x2, x2, x3
-
-        /* ... map of vaddr(start) in boot_third */
-        mov   x1, xzr
-1:      str   x2, [x4, x1]           /* Map vaddr(start) */
-        add   x2, x2, #PAGE_SIZE     /* Next page */
-        add   x1, x1, #8             /* Next slot */
-        cmp   x1, #(LPAE_ENTRIES<<3) /* 512 entries per page */
-        b.lt  1b
-
-        /* Defer fixmap and dtb mapping until after paging enabled, to
-         * avoid them clashing with the 1:1 mapping. */
-
-        /* boot pagetable setup complete */
-
-        cbnz  x25, 1f                /* Did we manage to create an identity mapping ? */
-        PRINT("Unable to build boot page tables - Failed to identity map Xen.\r\n")
-        b     fail
-virtphys_clash:
-        /* Identity map clashes with boot_third, which we cannot handle yet */
-        PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
-        b     fail
-
-1:
         PRINT("- Turning on paging -\r\n")
 
         /*
@@ -797,6 +673,141 @@ ENTRY(efi_xen_start)
         b     real_start_efi
 ENDPROC(efi_xen_start)
 
+ENTRY(setup_page_tables)
+        ldr   x0, =start
+        adr   x19, start             /* x19 := paddr (start) */
+        sub   x20, x19, x0           /* x20 := phys-offset */
+
+        /* Rebuild the boot pagetable's first-level entries. The structure
+         * is described in mm.c.
+         *
+         * After the CPU enables paging it will add the fixmap mapping
+         * to these page tables, however this may clash with the 1:1
+         * mapping. So each CPU must rebuild the page tables here with
+         * the 1:1 in place. */
+
+        /* If Xen is loaded at exactly XEN_VIRT_START then we don't
+         * need an additional 1:1 mapping, the virtual mapping will
+         * suffice.
+         */
+        cmp   x19, #XEN_VIRT_START
+        cset  x25, eq                /* x25 := identity map in place, or not */
+
+        /* Write Xen's PT's paddr into TTBR0_EL2 */
+        load_paddr x4, boot_pgtable
+        msr   TTBR0_EL2, x4
+
+        /* Setup boot_pgtable: */
+        load_paddr x1, boot_first
+
+        /* ... map boot_first in boot_pgtable[0] */
+        mov   x3, #PT_PT             /* x2 := table map of boot_first */
+        orr   x2, x1, x3             /*       + rights for linear PT */
+        str   x2, [x4, #0]           /* Map it in slot 0 */
+
+        /* ... map of paddr(start) in boot_pgtable+boot_first_id */
+        lsr   x1, x19, #ZEROETH_SHIFT/* Offset of base paddr in boot_pgtable */
+        cbz   x1, 1f                 /* It's in slot 0, map in boot_first
+                                      * or boot_second later on */
+
+        /* Level zero does not support superpage mappings, so we have
+         * to use an extra first level page in which we create a 1GB mapping.
+         */
+        load_paddr x2, boot_first_id
+
+        mov   x3, #PT_PT             /* x2 := table map of boot_first_id */
+        orr   x2, x2, x3             /*       + rights for linear PT */
+        lsl   x1, x1, #3             /* x1 := Slot offset */
+        str   x2, [x4, x1]
+
+        load_paddr x4, boot_first_id
+
+        lsr   x1, x19, #FIRST_SHIFT  /* x1 := Offset of base paddr in boot_first_id */
+        lsl   x2, x1, #FIRST_SHIFT   /* x2 := Base address for 1GB mapping */
+        mov   x3, #PT_MEM            /* x2 := Section map */
+        orr   x2, x2, x3
+        and   x1, x1, #LPAE_ENTRY_MASK /* x1 := Slot offset */
+        lsl   x1, x1, #3
+        str   x2, [x4, x1]           /* Mapping of paddr(start) */
+        mov   x25, #1                /* x25 := identity map now in place */
+
+1:      /* Setup boot_first: */
+        load_paddr x4, boot_first   /* Next level into boot_first */
+
+        /* ... map boot_second in boot_first[0] */
+        load_paddr x1, boot_second
+        mov   x3, #PT_PT             /* x2 := table map of boot_second */
+        orr   x2, x1, x3             /*       + rights for linear PT */
+        str   x2, [x4, #0]           /* Map it in slot 0 */
+
+        /* ... map of paddr(start) in boot_first */
+        cbnz  x25, 1f                /* x25 is set if already created */
+        lsr   x2, x19, #FIRST_SHIFT  /* x2 := Offset of base paddr in boot_first */
+        and   x1, x2, #LPAE_ENTRY_MASK /* x1 := Slot to use */
+        cbz   x1, 1f                 /* It's in slot 0, map in boot_second */
+
+        lsl   x2, x2, #FIRST_SHIFT   /* Base address for 1GB mapping */
+        mov   x3, #PT_MEM            /* x2 := Section map */
+        orr   x2, x2, x3
+        lsl   x1, x1, #3             /* x1 := Slot offset */
+        str   x2, [x4, x1]           /* Create mapping of paddr(start)*/
+        mov   x25, #1                /* x25 := identity map now in place */
+
+1:      /* Setup boot_second: */
+        load_paddr x4, boot_second
+
+        /* ... map boot_third in boot_second[1] */
+        load_paddr x1, boot_third
+        mov   x3, #PT_PT             /* x2 := table map of boot_third */
+        orr   x2, x1, x3             /*       + rights for linear PT */
+        str   x2, [x4, #8]           /* Map it in slot 1 */
+
+        /* ... map of paddr(start) in boot_second */
+        cbnz  x25, 1f                /* x25 is set if already created */
+        lsr   x2, x19, #SECOND_SHIFT /* x2 := Offset of base paddr in boot_second */
+        and   x1, x2, #LPAE_ENTRY_MASK /* x1 := Slot to use */
+        cmp   x1, #1
+        b.eq  virtphys_clash         /* It's in slot 1, which we cannot handle */
+
+        lsl   x2, x2, #SECOND_SHIFT  /* Base address for 2MB mapping */
+        mov   x3, #PT_MEM            /* x2 := Section map */
+        orr   x2, x2, x3
+        lsl   x1, x1, #3             /* x1 := Slot offset */
+        str   x2, [x4, x1]           /* Create mapping of paddr(start)*/
+        mov   x25, #1                /* x25 := identity map now in place */
+
+1:      /* Setup boot_third: */
+        load_paddr x4, boot_third
+
+        lsr   x2, x19, #THIRD_SHIFT  /* Base address for 4K mapping */
+        lsl   x2, x2, #THIRD_SHIFT
+        mov   x3, #PT_MEM_L3         /* x2 := Section map */
+        orr   x2, x2, x3
+
+        /* ... map of vaddr(start) in boot_third */
+        mov   x1, xzr
+1:      str   x2, [x4, x1]           /* Map vaddr(start) */
+        add   x2, x2, #PAGE_SIZE     /* Next page */
+        add   x1, x1, #8             /* Next slot */
+        cmp   x1, #(LPAE_ENTRIES<<3) /* 512 entries per page */
+        b.lt  1b
+
+        /* Defer fixmap and dtb mapping until after paging enabled, to
+         * avoid them clashing with the 1:1 mapping. */
+
+        /* boot pagetable setup complete */
+
+        cbnz  x25, 1f                /* Did we manage to create an identity mapping ? */
+        PRINT("Unable to build boot page tables - Failed to identity map Xen.\r\n")
+        b    1f
+virtphys_clash:
+        /* Identity map clashes with boot_third, which we cannot handle yet */
+        PRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
+1:
+        ret
+
+ENDPROC(setup_page_tables)
+
 /*
  * Local variables:
  * mode: ASM
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 15/18] xen/arm: Resume memory management on Xen resume

[Mirela Simonovic]

The MMU needs to be enabled in the resume flow before the context
can be restored (we need to be able to access the context data by
virtual address in order to restore it). The configuration of system
registers prior to branching to the routine that sets up the page
tables is copied from xen/arch/arm/arm64/head.S.
After the MMU is enabled, the content of TTBR0_EL2 is changed to
point to init_ttbr (page tables used at runtime).

At boot the init_ttbr variable is updated when a secondary CPU is
hotplugged. In the scenario where there is only one physical CPU in
the system, the init_ttbr would not be initialized for the use in
resume flow. To get the variable initialized in all scenarios in this
patch we add that the boot CPU updates init_ttbr after it sets the
page tables for runtime.

After the memory management is resumed, the SCTLR_WXN in SCTLR_EL2
has to be set in order to configure that a mapping cannot be both
writable and executable (this was configured prior to suspend).
This is done using an existing function (mmu_init_secondary_cpu).

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

---
Changes in v2:

- Patch from v1:
"[XEN PATCH 17/21] xen/arm: Set SCTLR_WXN in SCTLR_EL2 on resume"
is squashed with this patch, because it is indeed related to resuming
the memory management
- Since the original patch was named 'Enable the MMU', and this is
not only enabling anymore, but the full resume of functionality, the
commit title and message is fixed
---
 xen/arch/arm/arm64/entry.S | 88 ++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/mm.c          |  1 +
 xen/arch/arm/suspend.c     |  6 ++++
 3 files changed, 95 insertions(+)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index dbc4717903..5efa30e8ee 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -1,6 +1,7 @@
 #include <asm/asm_defns.h>
 #include <asm/current.h>
 #include <asm/macros.h>
+#include <asm/page.h>
 #include <asm/regs.h>
 #include <asm/alternative.h>
 #include <asm/smccc.h>
@@ -534,6 +535,93 @@ ENTRY(__context_switch)
         ret
 
 ENTRY(hyp_resume)
+        msr   DAIFSet, 0xf           /* Disable all interrupts */
+
+        tlbi  alle2
+        dsb   sy                     /* Ensure completion of TLB flush */
+        isb
+
+        ldr   x0, =start
+        adr   x19, start             /* x19 := paddr (start) */
+        sub   x20, x19, x0           /* x20 := phys-offset */
+
+        /* XXXX call PROCINFO_cpu_init here */
+
+        /* Set up memory attribute type tables */
+        ldr   x0, =MAIRVAL
+        msr   mair_el2, x0
+
+        /* Set up TCR_EL2:
+         * PS -- Based on ID_AA64MMFR0_EL1.PARange
+         * Top byte is used
+         * PT walks use Inner-Shareable accesses,
+         * PT walks are write-back, write-allocate in both cache levels,
+         * 48-bit virtual address space goes through this table. */
+        ldr   x0, =(TCR_RES1|TCR_SH0_IS|TCR_ORGN0_WBWA|TCR_IRGN0_WBWA|TCR_T0SZ(64-48))
+        /* ID_AA64MMFR0_EL1[3:0] (PARange) corresponds to TCR_EL2[18:16] (PS) */
+        mrs   x1, ID_AA64MMFR0_EL1
+        bfi   x0, x1, #16, #3
+
+        msr   tcr_el2, x0
+
+        /* Set up the SCTLR_EL2:
+         * Exceptions in LE ARM,
+         * Low-latency IRQs disabled,
+         * Write-implies-XN disabled (for now),
+         * D-cache disabled (for now),
+         * I-cache enabled,
+         * Alignment checking disabled,
+         * MMU translation disabled (for now). */
+        ldr   x0, =(HSCTLR_BASE)
+        msr   SCTLR_EL2, x0
+
+        /* Ensure that any exceptions encountered at EL2
+         * are handled using the EL2 stack pointer, rather
+         * than SP_EL0. */
+        msr spsel, #1
+
+        /* Rebuild the boot pagetable's first-level entries. The structure
+         * is described in mm.c.
+         *
+         * After the CPU enables paging it will add the fixmap mapping
+         * to these page tables, however this may clash with the 1:1
+         * mapping. So each CPU must rebuild the page tables here with
+         * the 1:1 in place. */
+
+        /* If Xen is loaded at exactly XEN_VIRT_START then we don't
+         * need an additional 1:1 mapping, the virtual mapping will
+         * suffice.
+         */
+        cmp   x19, #XEN_VIRT_START
+        cset  x25, eq                /* x25 := identity map in place, or not */
+
+        /* Write Xen's PT's paddr into TTBR0_EL2 */
+        ldr   x4, =boot_pgtable     /* xen_pgtable    */
+        add   x4, x4, x20           /* x4 := paddr (boot_pagetable) */
+        msr   TTBR0_EL2, x4
+
+        /* Set up page tables */
+        bl    setup_page_tables
+
+        ldr   x1, =mmu_resumed      /* Explicit vaddr, not RIP-relative */
+        mrs   x0, SCTLR_EL2
+        orr   x0, x0, #SCTLR_M      /* Enable MMU */
+        orr   x0, x0, #SCTLR_C      /* Enable D-cache */
+        dsb   sy                    /* Flush PTE writes and finish reads */
+        msr   SCTLR_EL2, x0         /* now paging is enabled */
+        isb                         /* Now, flush the icache */
+        br    x1                    /* Get a proper vaddr into PC */
+
+mmu_resumed:
+        ldr   x4, =init_ttbr         /* VA of TTBR0_EL2 stashed by CPU 0 */
+        ldr   x4, [x4]               /* Actual value */
+        dsb   sy
+        msr   TTBR0_EL2, x4
+        dsb   sy
+        isb
+        tlbi  alle2
+        dsb   sy                     /* Ensure completion of TLB flush */
+        isb
         b .
 
 /*
diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 7a06a33e21..7ad0ee1947 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -722,6 +722,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
     WRITE_SYSREG32(READ_SYSREG32(SCTLR_EL2) | SCTLR_WXN, SCTLR_EL2);
     /* Flush everything after setting WXN bit. */
     flush_xen_text_tlb_local();
+    init_secondary_pagetables(0);
 
 #ifdef CONFIG_ARM_32
     per_cpu(xen_pgtable, 0) = cpu0_pgtable;
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 37926374bc..365c32ec3c 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -147,6 +147,12 @@ static long system_suspend(void *data)
 
     system_state = SYS_STATE_resume;
 
+    /*
+     * SCTLR_WXN needs to be set to configure that a mapping cannot be both
+     * writable and executable. This is done by mmu_init_secondary_cpu.
+     */
+    mmu_init_secondary_cpu();
+
     gic_resume();
 
 resume_irqs:
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 16/18] xen/arm: Save/restore context on suspend/resume

[Mirela Simonovic]

The context of CPU general purpose and system control registers
has to be saved on suspend and restored on resume. This is
implemented in hyp_suspend and before the return from hyp_resume
function. The hyp_suspend is invoked just before the PSCI system
suspend call is issued to the ATF. The hyp_suspend has to return a
non-zero value so that the calling 'if' statement evaluates to true,
causing the system suspend to be invoked. Upon the resume, context
saved on suspend will be restored, including the link register.
Therefore, after restoring the context the control flow will
return to the address pointed by the saved link register, which
is the place from which the hyp_suspend was called. To ensure
that the calling 'if' statement doesn't again evaluate to true
and initiate system suspend, hyp_resume has to return a zero value
after restoring the context.
Note that the order of saving register context into cpu_context
structure has to match the order of restoring.
Since the suspend/resume is supported only for arm64, we define
a null cpu_context structure so arm32 could compile.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/arm64/entry.S    | 89 ++++++++++++++++++++++++++++++++++++++++++-
 xen/arch/arm/suspend.c        | 28 ++++++++++++--
 xen/include/asm-arm/suspend.h | 22 +++++++++++
 3 files changed, 135 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 5efa30e8ee..861a401b18 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -534,6 +534,52 @@ ENTRY(__context_switch)
         mov     sp, x9
         ret
 
+/*
+ * void hyp_suspend(struct cpu_context *ptr)
+ *
+ * x0 - pointer to the storage where callee's context will be saved
+ *
+ * CPU context saved here will be restored on resume in hyp_resume function.
+ * hyp_suspend shall return a non-zero value. Upon restoring context
+ * hyp_resume shall return value zero instead. From C code that invokes
+ * hyp_suspend, the return value is interpreted to determine whether the context
+ * is saved (hyp_suspend) or restored (hyp_resume).
+ */
+ENTRY(hyp_suspend)
+        /* Store callee-saved registers */
+        stp     x19, x20, [x0], #16
+        stp     x21, x22, [x0], #16
+        stp     x23, x24, [x0], #16
+        stp     x25, x26, [x0], #16
+        stp     x27, x28, [x0], #16
+        stp     x29, lr, [x0], #16
+
+        /* Store stack-pointer */
+        mov     x2, sp
+        str     x2, [x0], #8
+
+        /* Store system control registers */
+        mrs     x2, VBAR_EL2
+        str     x2, [x0], #8
+        mrs     x2, VTCR_EL2
+        str     x2, [x0], #8
+        mrs     x2, VTTBR_EL2
+        str     x2, [x0], #8
+        mrs     x2, TPIDR_EL2
+        str     x2, [x0], #8
+        mrs     x2, MDCR_EL2
+        str     x2, [x0], #8
+        mrs     x2, HSTR_EL2
+        str     x2, [x0], #8
+        mrs     x2, CPTR_EL2
+        str     x2, [x0], #8
+        mrs     x2, HCR_EL2
+        str     x2, [x0], #8
+
+        /* hyp_suspend must return a non-zero value */
+        mov     x0, #1
+        ret
+
 ENTRY(hyp_resume)
         msr   DAIFSet, 0xf           /* Disable all interrupts */
 
@@ -622,7 +668,48 @@ mmu_resumed:
         tlbi  alle2
         dsb   sy                     /* Ensure completion of TLB flush */
         isb
-        b .
+
+        /* Now we can access the cpu_context, so restore the context here */
+        ldr     x0, =cpu_context
+
+        /* Restore callee-saved registers */
+        ldp     x19, x20, [x0], #16
+        ldp     x21, x22, [x0], #16
+        ldp     x23, x24, [x0], #16
+        ldp     x25, x26, [x0], #16
+        ldp     x27, x28, [x0], #16
+        ldp     x29, lr, [x0], #16
+
+        /* Restore stack pointer */
+        ldr     x2, [x0], #8
+        mov     sp, x2
+
+        /* Restore system control registers */
+        ldr     x2, [x0], #8
+        msr     VBAR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     VTCR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     VTTBR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     TPIDR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     MDCR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     HSTR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     CPTR_EL2, x2
+        ldr     x2, [x0], #8
+        msr     HCR_EL2, x2
+        isb
+
+        /* Since context is restored return from this function will appear as
+         * return from hyp_suspend. To distinguish a return from hyp_suspend
+         * which is called upon finalizing the suspend, as opposed to return
+         * from this function which executes on resume, we need to return zero
+         * value here. */
+        mov x0, #0
+        ret
 
 /*
  * Local variables:
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 365c32ec3c..423d9c0e90 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -3,6 +3,9 @@
 #include <asm/cpufeature.h>
 #include <asm/event.h>
 #include <asm/psci.h>
+#include <asm/suspend.h>
+
+struct cpu_context cpu_context;
 
 /* Reset values of VCPU architecture specific registers */
 static void vcpu_arch_reset(struct vcpu *v)
@@ -113,6 +116,11 @@ static void vcpu_suspend(register_t epoint, register_t cid)
     _arch_set_info_guest(v, &ctxt);
 }
 
+#ifndef CONFIG_ARM_64
+/* not supported on ARM_32 */
+int32_t hyp_suspend(struct cpu_context *ptr) { return 1; }
+#endif
+
 /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
 static long system_suspend(void *data)
 {
@@ -141,9 +149,23 @@ static long system_suspend(void *data)
         goto resume_irqs;
     }
 
-    status = call_psci_system_suspend();
-    if ( status )
-        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
+    if ( hyp_suspend(&cpu_context) )
+    {
+        status = call_psci_system_suspend();
+        /*
+         * If suspend is finalized properly by above system suspend PSCI call,
+         * the code below in this 'if' branch will never execute. Execution
+         * will continue from hyp_resume which is the hypervisor's resume point.
+         * In hyp_resume CPU context will be restored and since link-register is
+         * restored as well, it will appear to return from hyp_suspend. The
+         * difference in returning from hyp_suspend on system suspend versus
+         * resume is in function's return value: on suspend, the return value is
+         * a non-zero value, on resume it is zero. That is why the control flow
+         * will not re-enter this 'if' branch on resume.
+         */
+        if ( status )
+            dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
+    }
 
     system_state = SYS_STATE_resume;
 
diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
index 7604e2e2e2..a39071566a 100644
--- a/xen/include/asm-arm/suspend.h
+++ b/xen/include/asm-arm/suspend.h
@@ -1,8 +1,30 @@
 #ifndef __ASM_ARM_SUSPEND_H__
 #define __ASM_ARM_SUSPEND_H__
 
+#ifdef CONFIG_ARM_64
+struct cpu_context {
+    uint64_t callee_regs[12];
+    uint64_t sp;
+    uint64_t vbar_el2;
+    uint64_t vtcr_el2;
+    uint64_t vttbr_el2;
+    uint64_t tpidr_el2;
+    uint64_t mdcr_el2;
+    uint64_t hstr_el2;
+    uint64_t cptr_el2;
+    uint64_t hcr_el2;
+} __aligned(16);
+#else
+struct cpu_context {
+    uint8_t pad;
+};
+#endif
+
+extern struct cpu_context cpu_context;
+
 int32_t domain_suspend(register_t epoint, register_t cid);
 void hyp_resume(void);
+int32_t hyp_suspend(struct cpu_context *ptr);
 
 #endif
 
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

The resume of Dom0 should always follow Xen's resume. This is
done by unblocking the first vCPU of Dom0.

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 423d9c0e90..a05aea9c25 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -189,6 +189,9 @@ resume_nonboot_cpus:
     system_state = SYS_STATE_active;
     dsb(sy);
 
+    /* Wake-up hardware domain (should always resume after Xen) */
+    vcpu_unblock(hardware_domain->vcpu[0]);
+
     return status;
 }
 
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 18/18] xen/arm: Suspend/resume console on Xen suspend/resume

[Mirela Simonovic]

This is done using generic console_suspend/resume functions that cause
uart driver specific suspend/resume handlers to be called for each
initialized port (if the port has suspend/resume driver handlers
implemented).

Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
---
 xen/arch/arm/suspend.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index a05aea9c25..6d7d69539b 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -1,5 +1,6 @@
 #include <xen/sched.h>
 #include <xen/cpu.h>
+#include <xen/console.h>
 #include <asm/cpufeature.h>
 #include <asm/event.h>
 #include <asm/psci.h>
@@ -149,6 +150,15 @@ static long system_suspend(void *data)
         goto resume_irqs;
     }
 
+    dprintk(XENLOG_DEBUG, "Suspend\n");
+    status = console_suspend();
+    if ( status )
+    {
+        dprintk(XENLOG_ERR, "Failed to suspend the console, err=%d\n", status);
+        system_state = SYS_STATE_resume;
+        goto resume_console;
+    }
+
     if ( hyp_suspend(&cpu_context) )
     {
         status = call_psci_system_suspend();
@@ -175,6 +185,10 @@ static long system_suspend(void *data)
      */
     mmu_init_secondary_cpu();
 
+resume_console:
+    console_resume();
+    dprintk(XENLOG_DEBUG, "Resume\n");
+
     gic_resume();
 
 resume_irqs:
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

[-- Attachment #1.1: Type: text/plain, Size: 7629 bytes --]

Hi,

One thing I screwed - I forgot to remove changes log from an internal
review, so please ignore it. This is officially the first version.

Thanks,
Mirela

On Mon, Nov 12, 2018 at 12:31 PM Mirela Simonovic <
mirela.simonovic@aggios.com> wrote:

> This series contains support for suspend to RAM (in the following text just
> 'suspend') for Xen on arm64. The implementation is aligned with the design
> specification that has been proposed on xen-devel list:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>
> At a high-level the series contains:
> 1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
> 2) Support for resuming a guest on an interrupt targeted to that guest
> 3) Support for suspending Xen after dom0 finalizes the suspend
> 4) Support for resuming Xen on an interrupt that is targeted to a guest
>
>
> --------------------------------------------------------------------------------
> In more details:
>
> *** About suspending/resuming guests
>
> The patches included in this series allow PSCI compliant guests that have
> support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
> suspend and resume on top of Xen without any EL1 code changes.
>
> During their suspend procedure guests will hot-unplug their secondary CPUs,
> triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
> suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
> Guests will save/restore their own EL1 context on suspend/resume.
>
> A guest is expected to leave enabled interrupts that are considered to be
> its
> wake-up sources. Those interrupts will be able to wake up the guest. This
> holds
> regardless of the state of the underlying software layers, i.e. whether
> Xen gets
> suspended or not doesn't affect the ability of the guest to wake up.
>
> First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from
> which
> the guest assumes to start on resume. On resume, guests assume to be
> running in
> an environment whose state matches the CPU state after reset, e.g. with
> reset
> register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
> VCPU context and save the resume entry point into program counter before
> the
> guest's VCPU gets scheduled in on resume. This is done when the guest
> finalizes
> its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure
> that
> the resume-ready VCPU's context does not get overwritten later upon the
> context
> switch when the VCPU is scheduled out.
> Xen also needs to take care that the guest's view of GIC and timer gets
> saved.
> Also, while a guest is suspended its watchdogs are paused, in order to
> avoid
> watchdog triggered shutdown of a guest that has been asleep for a period
> of time
> that is longer than the watchdog period.
>
> After this point, from Xen's point of view a suspended guest has one VCPU
> blocked, waiting for an interrupt. When such an interrupt comes, Xen will
> unblock the VCPU of the suspended domain, which results in the guest
> resuming.
>
> *** About suspending/resuming Xen
>
> Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
> considered to be the decision maker for EL1 and EL2.
> On suspend, Xen will first freeze all domains. Then, Xen disables physical
> secondary CPUs, which leads to physical CPU_OFF to be called by each
> secondary
> CPU. After that Xen finalizes the suspend from the boot CPU.
>
> This consists of suspending the timer, i.e. suppressing its interrupts (we
> don't
> want to be woken up by a timer, there is no VCPU ready to be scheduled).
> Then
> the state of GIC is saved, console is suspended, and CPU context is saved.
> The
> saved context tells where Xen needs to continue execution on resume.
> Since Xen will resume with MMU disabled, the first thing to do in resume
> is to
> resume memory management in order to be able to access the context that
> needs to
> be restored (we know virtual address of the context data). Finally Xen
> calls
> SYSTEM_SUSPEND PSCI to the EL3.
>
> When resuming, all the steps done in suspend need to be reverted. This is
> completed by unblocking dom0's VCPU, because we always want the dom0 to
> resume,
> regardless of the target domain whose interrupt woke up Xen.
>
> *** Handling of unprivileged guests during Xen suspend/resume
>
> Any domU that is not suspended when dom0 suspends will be frozen, domUs
> that are
> already suspended remain suspended. On resume the suspended domUs still
> remain
> suspended (unless their wake interrupt caused Xen to wake) while the
> others will
> be thawed.
>
> For more details please refer to patches or the design specification:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>
>
> --------------------------------------------------------------------------------
> The series does not include:
> a) UART driver-specific suspend/resume that gets called when console
> suspends
> b) SMMU suspend/resume
> c) Suspend coordination support that would allow dom0 to request domUs to
> suspend
> These will be submitted in the following series.
>
> Mirela Simonovic (16):
>   xen/arm: Implement PSCI system suspend call (virtual interface)
>   xen/arm: Save GIC and virtual timer context when the domain suspends
>   xen/arm: While a domain is suspended put its watchdogs on pause
>   xen/arm: Trigger Xen suspend when Dom0 completes suspend
>   xen/x86: Move freeze/thaw_domains into common files
>   xen/arm: Freeze domains on suspend and thaw them on resume
>   xen/arm: Disable/enable non-boot physical CPUs on suspend/resume
>   xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume
>   xen/arm: Implement GIC suspend/resume functions (gicv2 only)
>   xen/arm: Suspend/resume GIC on system suspend/resume
>   xen/arm: Suspend/resume timer interrupt generation
>   xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)
>   xen/arm: Resume memory management on Xen resume
>   xen/arm: Save/restore context on suspend/resume
>   xen/arm: Resume Dom0 after Xen resumes
>   xen/arm: Suspend/resume console on Xen suspend/resume
>
> Saeed Nowshadi (2):
>   xen/arm: Move code that initializes VCPU context into a separate
>     function
>   xen/arm: Convert setting MMU page tables code into a routine
>
>  xen/arch/arm/Makefile            |   1 +
>  xen/arch/arm/arm64/entry.S       | 178 ++++++++++++++++++++++++
>  xen/arch/arm/arm64/head.S        | 265 ++++++++++++++++++-----------------
>  xen/arch/arm/domain.c            |  62 ++++++---
>  xen/arch/arm/gic-v2.c            | 147 ++++++++++++++++++++
>  xen/arch/arm/gic.c               |  27 ++++
>  xen/arch/arm/mm.c                |   1 +
>  xen/arch/arm/psci.c              |  16 +++
>  xen/arch/arm/suspend.c           | 292
> +++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/time.c              |  22 +++
>  xen/arch/arm/vpsci.c             |  19 +++
>  xen/arch/x86/acpi/power.c        |  28 ----
>  xen/common/domain.c              |  29 ++++
>  xen/common/schedule.c            |  38 +++++
>  xen/include/asm-arm/gic.h        |   8 ++
>  xen/include/asm-arm/perfc_defn.h |   1 +
>  xen/include/asm-arm/psci.h       |   3 +
>  xen/include/asm-arm/suspend.h    |  39 ++++++
>  xen/include/asm-arm/time.h       |   3 +
>  xen/include/xen/domain.h         |   1 +
>  xen/include/xen/sched.h          |  11 ++
>  xen/include/xen/timer.h          |   3 +
>  22 files changed, 1019 insertions(+), 175 deletions(-)
>  create mode 100644 xen/arch/arm/suspend.c
>  create mode 100644 xen/include/asm-arm/suspend.h
>
> --
> 2.13.0
>
>

[-- Attachment #1.2: Type: text/html, Size: 8799 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 01/18] xen/arm: Move code that initializes VCPU context into a separate function

[Jan Beulich]

>>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
> --- a/xen/include/xen/domain.h
> +++ b/xen/include/xen/domain.h
> @@ -73,6 +73,7 @@ int arch_domain_soft_reset(struct domain *d);
>  void arch_p2m_set_access_required(struct domain *d, bool access_required);
>  
>  int arch_set_info_guest(struct vcpu *, vcpu_guest_context_u);
> +void _arch_set_info_guest(struct vcpu *, struct vcpu_guest_context *);

If the Arm maintainers are happy with such a name space violation
I won't object, but if you place this in a common header (without
strict need afaict) please respect the C spec reserving names like
this one for file scope symbols.

Also please don't post patches to xen-devel@ twice (using different
domain names).

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Jan Beulich]

>>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -24,6 +24,7 @@
>  #include <xen/wait.h>
>  #include <public/xen.h>
>  #include <public/domctl.h>
> +#include <public/sched.h>
>  #include <public/sysctl.h>
>  #include <public/vcpu.h>
>  #include <public/vm_event.h>

Why?

Jan.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 01/18] xen/arm: Move code that initializes VCPU context into a separate function

[Wei Liu]

First thing first, thanks for posting this series.  I'm not an ARM
maintainer but I do have a few comments.

On Mon, Nov 12, 2018 at 12:30:27PM +0100, Mirela Simonovic wrote:
> From: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> 
> The arch_set_info_guest() has code to initialize the context of a VCPU.
> When a VCPU is resumed it needs to go through the same context
> initialization excluding all the validations that this routine does.

OOI why does it not need validation?

> We move the actual VCPU context setting into a function so that it can be
> shared with the resume path.
> 
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> ---
>  xen/arch/arm/domain.c    | 34 +++++++++++++++++++++-------------
>  xen/include/xen/domain.h |  1 +
>  2 files changed, 22 insertions(+), 13 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 80432872d6..e594b48d81 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -781,6 +781,26 @@ static int is_guest_pv64_psr(uint32_t psr)
>  #endif
>  
>  /*
> + * The actual VCPU initialization after all validations are passed.
> + */
> +void _arch_set_info_guest(struct vcpu *v, struct vcpu_guest_context *ctxt)

A better name is needed, maybe arch_set_info_guest_no_validation?

> +{
> +    vcpu_regs_user_to_hyp(v, &ctxt->user_regs);
> +
> +    v->arch.sctlr = ctxt->sctlr;
> +    v->arch.ttbr0 = ctxt->ttbr0;
> +    v->arch.ttbr1 = ctxt->ttbr1;
> +    v->arch.ttbcr = ctxt->ttbcr;
> +
> +    v->is_initialised = 1;
> +
> +    if ( ctxt->flags & VGCF_online )
> +        clear_bit(_VPF_down, &v->pause_flags);
> +    else
> +        set_bit(_VPF_down, &v->pause_flags);
> +}
> +
> +/*
>   * Initialise VCPU state. The context can be supplied by either the
>   * toolstack (XEN_DOMCTL_setvcpucontext) or the guest
>   * (VCPUOP_initialise) and therefore must be properly validated.
> @@ -818,19 +838,7 @@ int arch_set_info_guest(
>      }
>  #endif
>  
> -    vcpu_regs_user_to_hyp(v, regs);
> -
> -    v->arch.sctlr = ctxt->sctlr;
> -    v->arch.ttbr0 = ctxt->ttbr0;
> -    v->arch.ttbr1 = ctxt->ttbr1;
> -    v->arch.ttbcr = ctxt->ttbcr;
> -
> -    v->is_initialised = 1;
> -
> -    if ( ctxt->flags & VGCF_online )
> -        clear_bit(_VPF_down, &v->pause_flags);
> -    else
> -        set_bit(_VPF_down, &v->pause_flags);
> +    _arch_set_info_guest(v, ctxt);
>  
>      return 0;
>  }
> diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
> index 33e41486cb..904624e070 100644
> --- a/xen/include/xen/domain.h
> +++ b/xen/include/xen/domain.h
> @@ -73,6 +73,7 @@ int arch_domain_soft_reset(struct domain *d);
>  void arch_p2m_set_access_required(struct domain *d, bool access_required);
>  
>  int arch_set_info_guest(struct vcpu *, vcpu_guest_context_u);
> +void _arch_set_info_guest(struct vcpu *, struct vcpu_guest_context *);

Please put the new function's declaration into asm-arm/domain.h.

Normally we only introduce declaration when the function is needed
externally. Please at least say in the commit message that the
introduction of the declaration is because it will be needed in later
patches.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Jan Beulich]

>>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
> --- a/xen/include/xen/timer.h
> +++ b/xen/include/xen/timer.h
> @@ -18,6 +18,9 @@ struct timer {
>      /* System time expiry value (nanoseconds since boot). */
>      s_time_t expires;
>  
> +    /* Suspend timestamp value (nanoseconds since boot). */
> +    s_time_t suspended;

I can't see how this becomes a universally useful field, so I
don't think it belongs here. For your purpose deriving a new
structure containing struct timer and this one field should
work.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Julien Grall]

Hi Mirela,

Thank you for posting the series. Could you provide a branch with the 
patch applied?

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> --------------------------------------------------------------------------------
> The series does not include:
> a) UART driver-specific suspend/resume that gets called when console suspends

I feel it will be difficult to test this series without UART driver 
support. Would it be possible to integrate them in that series?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

[-- Attachment #1.1: Type: text/plain, Size: 474 bytes --]

What?

On Mon, Nov 12, 2018 at 12:42 PM Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
> > --- a/xen/include/xen/sched.h
> > +++ b/xen/include/xen/sched.h
> > @@ -24,6 +24,7 @@
> >  #include <xen/wait.h>
> >  #include <public/xen.h>
> >  #include <public/domctl.h>
> > +#include <public/sched.h>
> >  #include <public/sysctl.h>
> >  #include <public/vcpu.h>
> >  #include <public/vm_event.h>
>
> Why?
>
> Jan.
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 924 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

[-- Attachment #1.1: Type: text/plain, Size: 1128 bytes --]

Hi Julien,

On Mon, Nov 12, 2018 at 12:50 PM Julien Grall <julien.grall@arm.com> wrote:

> Hi Mirela,
>
> Thank you for posting the series. Could you provide a branch with the
> patch applied?
>
>
I have applied patches on top of upstream/staging.


> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >
> --------------------------------------------------------------------------------
> > The series does not include:
> > a) UART driver-specific suspend/resume that gets called when console
> suspends
>
> I feel it will be difficult to test this series without UART driver
> support. Would it be possible to integrate them in that series?
>
>
Yes that's possible, but you'll need more than a UART driver's
suspend/resume to test this. E.g. for testing the series on Xilinx
Ultrascale+ MPSoC we need a set of patches that provides the basic platform
support needed to boot, which is not yet upstreamed. Luckily, the suspend
support is independent (ARM specific).
Please let me check with Xilinx people and Stefano how to deal with this,
and I'll come back with a real answer.

Thanks,
Mirela


> Cheers,
>
> --
> Julien Grall
>

[-- Attachment #1.2: Type: text/html, Size: 1801 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Julien Grall]

On 11/12/18 12:01 PM, Mirela Simonovic wrote:
> Hi Julien,

Hi Mirela,

Please configure your e-mail client to avoid quoting using "space". 
Otherwise, this is going to make difficult to follow the discussions.

> On Mon, Nov 12, 2018 at 12:50 PM Julien Grall <julien.grall@arm.com 
> <mailto:julien.grall@arm.com>> wrote:
> 
>     Hi Mirela,
> 
>     Thank you for posting the series. Could you provide a branch with the
>     patch applied?
> 
> 
> I have applied patches on top of upstream/staging.

patches are applied every day to staging. So can you please provide the 
exact commit?

Also, it is common for big series to provide a branch with the patch 
series applied. This makes easier for people to try your series.

> 
>     On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>      >
>     --------------------------------------------------------------------------------
>      > The series does not include:
>      > a) UART driver-specific suspend/resume that gets called when
>     console suspends
> 
>     I feel it will be difficult to test this series without UART driver
>     support. Would it be possible to integrate them in that series?
> 
> 
> Yes that's possible, but you'll need more than a UART driver's 
> suspend/resume to test this. E.g. for testing the series on Xilinx 
> Ultrascale+ MPSoC we need a set of patches that provides the basic 
> platform support needed to boot, which is not yet upstreamed. Luckily, 
> the suspend support is independent (ARM specific).

I am a bit confused. I thought it was possible to boot Xen out-of-box on 
the Zynq MP. What is missing?

> Please let me check with Xilinx people and Stefano how to deal with 
> this, and I'll come back with a real answer.

If that's too difficult to provide in short term. Then I would 
appreciate if testing is done on another platform (e.g Foundation 
Model). So I can at least verify the series before we merge it.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

Hi Julien,

On Mon, Nov 12, 2018 at 1:08 PM Julien Grall <julien.grall@arm.com> wrote:
>
> On 11/12/18 12:01 PM, Mirela Simonovic wrote:
> > Hi Julien,
>
> Hi Mirela,
>
> Please configure your e-mail client to avoid quoting using "space".
> Otherwise, this is going to make difficult to follow the discussions.
>

Sure, sorry.

> > On Mon, Nov 12, 2018 at 12:50 PM Julien Grall <julien.grall@arm.com
> > <mailto:julien.grall@arm.com>> wrote:
> >
> >     Hi Mirela,
> >
> >     Thank you for posting the series. Could you provide a branch with the
> >     patch applied?
> >
> >
> > I have applied patches on top of upstream/staging.
>
> patches are applied every day to staging. So can you please provide the
> exact commit?
>

Oh yes, the staging already moved. It was this one: 388c55b
tools/misc: fix hard tabs in xen-hvmctx.c

> Also, it is common for big series to provide a branch with the patch
> series applied. This makes easier for people to try your series.
>
> >
> >     On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >      >
> >     --------------------------------------------------------------------------------
> >      > The series does not include:
> >      > a) UART driver-specific suspend/resume that gets called when
> >     console suspends
> >
> >     I feel it will be difficult to test this series without UART driver
> >     support. Would it be possible to integrate them in that series?
> >
> >
> > Yes that's possible, but you'll need more than a UART driver's
> > suspend/resume to test this. E.g. for testing the series on Xilinx
> > Ultrascale+ MPSoC we need a set of patches that provides the basic
> > platform support needed to boot, which is not yet upstreamed. Luckily,
> > the suspend support is independent (ARM specific).
>
> I am a bit confused. I thought it was possible to boot Xen out-of-box on
> the Zynq MP. What is missing?
>
> > Please let me check with Xilinx people and Stefano how to deal with
> > this, and I'll come back with a real answer.
>
> If that's too difficult to provide in short term. Then I would
> appreciate if testing is done on another platform (e.g Foundation
> Model). So I can at least verify the series before we merge it.
>

No, it's not difficult at all, I just need Xilinx support on this.

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Jan Beulich]

>>> On 12.11.18 at 12:50, <mirela.simonovic@aggios.com> wrote:
> What?

Please don't top post.

> On Mon, Nov 12, 2018 at 12:42 PM Jan Beulich <JBeulich@suse.com> wrote:
> 
>> >>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
>> > --- a/xen/include/xen/sched.h
>> > +++ b/xen/include/xen/sched.h
>> > @@ -24,6 +24,7 @@
>> >  #include <xen/wait.h>
>> >  #include <public/xen.h>
>> >  #include <public/domctl.h>
>> > +#include <public/sched.h>
>> >  #include <public/sysctl.h>
>> >  #include <public/vcpu.h>
>> >  #include <public/vm_event.h>
>>
>> Why?

I think given the context the question, even if terse, is pretty
clear: Why do you add this #include as the only change in this
header? Nothing in the header previously required it, and if
you have a sudden new requirement elsewhere, then please
include the extra header just there.

Also please accept that given the amount of time needed to
deal with incoming patches, it is not generally scalable to
always give long explanations to what seems pretty obvious
anyway. (I do realize that "obvious" is a rather subjective
attribute to things.)

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 01/18] xen/arm: Move code that initializes VCPU context into a separate function

[Julien Grall]

Hi Mirela,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> From: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> 
> The arch_set_info_guest() has code to initialize the context of a VCPU.
> When a VCPU is resumed it needs to go through the same context
> initialization excluding all the validations that this routine does.
> We move the actual VCPU context setting into a function so that it can be
> shared with the resume path.

I would rather not introduce a function that skip validation. They are 
here to catch error when setting up vCPU registers. If those errors are 
not caught, then the hypervisor will likely crash on return to the guest.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Mirela Simonovic]

Hi Jan,

On Mon, Nov 12, 2018 at 12:47 PM Jan Beulich <JBeulich@suse.com> wrote:
>
> >>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
> > --- a/xen/include/xen/timer.h
> > +++ b/xen/include/xen/timer.h
> > @@ -18,6 +18,9 @@ struct timer {
> >      /* System time expiry value (nanoseconds since boot). */
> >      s_time_t expires;
> >
> > +    /* Suspend timestamp value (nanoseconds since boot). */
> > +    s_time_t suspended;
>
> I can't see how this becomes a universally useful field, so I
> don't think it belongs here. For your purpose deriving a new
> structure containing struct timer and this one field should
> work.

Each 'expires' value needs to be coupled with 'suspended' value to be
able to deal with watchdogs when suspending a domain. This is specific
to a watchdog.
If we add a derived structure to contain this field we still need to
associate such a structure with each watchdog of a domain, which is
also domain generic info and requires a lot more (in my opinion
unnecessary) changes. Please let me know if I misinterpreted your
proposal.

Thanks,
Mirela

>
> Jan
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Jan Beulich]

>>> On 12.11.18 at 16:17, <mirela.simonovic@aggios.com> wrote:
> Hi Jan,
> 
> On Mon, Nov 12, 2018 at 12:47 PM Jan Beulich <JBeulich@suse.com> wrote:
>>
>> >>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
>> > --- a/xen/include/xen/timer.h
>> > +++ b/xen/include/xen/timer.h
>> > @@ -18,6 +18,9 @@ struct timer {
>> >      /* System time expiry value (nanoseconds since boot). */
>> >      s_time_t expires;
>> >
>> > +    /* Suspend timestamp value (nanoseconds since boot). */
>> > +    s_time_t suspended;
>>
>> I can't see how this becomes a universally useful field, so I
>> don't think it belongs here. For your purpose deriving a new
>> structure containing struct timer and this one field should
>> work.
> 
> Each 'expires' value needs to be coupled with 'suspended' value to be
> able to deal with watchdogs when suspending a domain. This is specific
> to a watchdog.
> If we add a derived structure to contain this field we still need to
> associate such a structure with each watchdog of a domain, which is
> also domain generic info and requires a lot more (in my opinion
> unnecessary) changes. Please let me know if I misinterpreted your
> proposal.

You add the field to all timer instances, yet you need it only for
very few. Why not have a struct watchdog_timer, and use it
for struct domain's watchdog_timer[]? I'm afraid I don't see the
difficulties you appear to see.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Mirela,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> The implementation consists of:
> -Adding PSCI system suspend call as new PSCI function
> -Trapping PSCI system_suspend HVC
> -Implementing PSCI system suspend call (virtual interface that allows
>   guests to suspend themselves)
> 
> The PSCI system suspend should be called by a guest from its boot
> VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
> CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
> are left enabled by the guest are assumed to be its wake-up interrupts.
> Therefore, a wake-up interrupt triggers the resume of the guest. Guest
> should resume regardless of the state of Xen (suspended or not).
> 
> When a guest calls PSCI system suspend the respective domain will be
> suspended if the following conditions are met:
> 1) Given resume entry point is not invalid
> 2) Other (if any) VCPUs of the calling guest are hot-unplugged
> 
> If the conditions above are met the calling domain is labeled as
> suspended and the calling VCPU is blocked. If nothing else wouldn't
> be done the suspended domain would resume from the place where it
> called PSCI system suspend. This is expected if processing of the PSCI
> system suspend call fails. However, in the case of success the calling
> guest should resume (continue execution after the wake-up) from the entry
> point which is given as the first argument of the PSCI system suspend
> call. In addition to the entry point, the guest expects to start within
> the environment whose state matches the state after reset. This means
> that the guest should find reset register values, MMU disabled, etc.
> Thereby, the context of VCPU should be 'reset' (as if the system is
> comming out of reset), the program counter should contain entry point,
> which is 1st argument, and r0/x0 should contain context ID which is 2nd
> argument of PSCI system suspend call. The context of VCPU is set
> accordingly when the PSCI system suspend is processed, so that nothing
> needs to be done on resume/wake-up path. However, in order to ensure that
> this context doesn't get overwritten by the scheduler when scheduling out
> this VCPU (would normally happen after the calling CPU is blocked), we need
> to check whether to return early from ctxt_switch_from().
> 
> There are variables in domain structure to keep track of domain shutdown.
> One of existing shutdown reason is 'suspend' which this patch is using to
> track the suspend state of a domain. Those variables are used to determine
> whether to early return from ctxt_switch_from() or not.
> 
> A suspended domain will resume after the Xen receives an interrupt which is
> targeted to the domain, unblocks the domain's VCPU, and schedules it in.
> When the VCPU is scheduled in, the VCPU context is already reset, and
> contains the right resume entry point in program counter that will be
> restored in ctxt_switch_to(). The only thing that needs to be done at this
> point is to clear the variables that marked the domain state as suspended.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> 
> ---
> Changes in v2:
> 
> -Fix print to compile for arm32 and to align with Xen coding style
> ---
>   xen/arch/arm/Makefile            |   1 +
>   xen/arch/arm/domain.c            |  13 +++
>   xen/arch/arm/suspend.c           | 166 +++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vpsci.c             |  19 +++++
>   xen/include/asm-arm/perfc_defn.h |   1 +
>   xen/include/asm-arm/psci.h       |   2 +
>   xen/include/asm-arm/suspend.h    |  16 ++++
>   xen/include/xen/sched.h          |   1 +
>   8 files changed, 219 insertions(+)
>   create mode 100644 xen/arch/arm/suspend.c
>   create mode 100644 xen/include/asm-arm/suspend.h
> 
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 23c5d9adbc..744b1a4dc8 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -43,6 +43,7 @@ obj-y += setup.o
>   obj-y += shutdown.o
>   obj-y += smp.o
>   obj-y += smpboot.o
> +obj-y += suspend.o
>   obj-y += sysctl.o
>   obj-y += time.o
>   obj-y += traps.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index e594b48d81..7f8105465c 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>       if ( is_idle_vcpu(p) )
>           return;
>   
> +    /* VCPU's context should not be saved if its domain is suspended */
> +    if ( p->domain->is_shut_down &&
> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> +        return;
SHUTDOWN_suspend is used in Xen for other purpose (see 
SCHEDOP_shutdown). The other user of that code relies on all the state 
to be saved on suspend.

However, what is the issue with saving all the registers here?

> +
>       p2m_save_state(p);
>   
>       /* CP 15 */
> @@ -181,6 +186,14 @@ static void ctxt_switch_to(struct vcpu *n)
>       if ( is_idle_vcpu(n) )
>           return;
>   
> +    /* If the domain was suspended, it is resuming now */
> +    if ( n->domain->is_shut_down &&
> +        (n->domain->shutdown_code == SHUTDOWN_suspend) )
> +    {
> +        n->domain->is_shut_down = 0;
> +        n->domain->shutdown_code = SHUTDOWN_CODE_INVALID;
> +    }

This looks like a hack. Why not calling domain_resume when receiving the 
interrupt?

> +
>       p2m_restore_state(n);
>   
>       vpidr = READ_SYSREG32(MIDR_EL1);
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> new file mode 100644
> index 0000000000..9eea9214e1
> --- /dev/null
> +++ b/xen/arch/arm/suspend.c

I would prefer if we don't mix guest and host suspend in the same file.

> @@ -0,0 +1,166 @@

Missing copyright headers here.

> +#include <xen/sched.h>
> +#include <asm/cpufeature.h>
> +#include <asm/event.h>
> +#include <asm/psci.h>
> +
> +/* Reset values of VCPU architecture specific registers */

Technically this is not requires as most of the registers are unknown. I 
understand this helps for debugging an OS.

I would introduce it in a separate patch and directly in 
arch_set_info_guest as I would like the behavior to be the same 
everywhere we need to reset a vCPU.

> +static void vcpu_arch_reset(struct vcpu *v)
> +{
> +    v->arch.ttbr0 = 0;
> +    v->arch.ttbr1 = 0;
> +    v->arch.ttbcr = 0;
> +
> +    v->arch.csselr = 0;
> +    v->arch.cpacr = 0;
> +    v->arch.contextidr = 0;
> +    v->arch.tpidr_el0 = 0;
> +    v->arch.tpidrro_el0 = 0;
> +    v->arch.tpidr_el1 = 0;
> +    v->arch.vbar = 0;
> +    if ( is_32bit_domain(v->domain) )

This is not necessary

> +        v->arch.dacr = 0;
> +    v->arch.par = 0;
> +#if defined(CONFIG_ARM_32)
> +    v->arch.mair0 = 0;
> +    v->arch.mair1 = 0;
> +    v->arch.amair0 = 0;
> +    v->arch.amair1 = 0;
> +#else
> +    v->arch.mair = 0;
> +    v->arch.amair = 0;
> +#endif
> +    /* Fault Status */
> +#if defined(CONFIG_ARM_32)
> +    v->arch.dfar = 0;
> +    v->arch.ifar = 0;
> +    v->arch.dfsr = 0;
> +#elif defined(CONFIG_ARM_64)
> +    v->arch.far = 0;
> +    v->arch.esr = 0;
> +#endif
> +
> +    if ( is_32bit_domain(v->domain) )

Same here.

> +        v->arch.ifsr  = 0;
> +    v->arch.afsr0 = 0;
> +    v->arch.afsr1 = 0;
> +
> +#ifdef CONFIG_ARM_32
> +    v->arch.joscr = 0;
> +    v->arch.jmcr = 0;
> +#endif
> +
> +    if ( is_32bit_domain(v->domain) && cpu_has_thumbee )

Same here.

> +    {
> +        v->arch.teecr = 0;
> +        v->arch.teehbr = 0;
> +    }
> +}
> +
> +/*
> + * This function sets the context of current VCPU to the state which is expected
> + * by the guest on resume. The expected VCPU state is:
> + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
> + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
> + * 3) All other general purpose and system registers should have reset values
> + *
> + * Note: this function has to return void because it has to always succeed. In
> + * other words, this function is called from virtual PSCI SYSTEM_SUSPEND
> + * implementation, which can return only a limited number of possible errors,
> + * none of which could represent the fact that an error occurred when preparing
> + * the domain for suspend.
> + * Consequently, dynamic memory allocation cannot be done within this function,
> + * because if malloc fails the error has nowhere to propagate.

You could crash the domain if you are not able to resume it. In the 
current context...

> + */
> +static void vcpu_suspend(register_t epoint, register_t cid)
> +{
> +    /* Static allocation because dynamic would need a non-void return */
> +    static struct vcpu_guest_context ctxt;

... this is not right. This function can be called concurrently, so a 
lot of funny things can happen (i.e corruption).

The vCPU context does not look too big. So I would just allocate it on 
the stack directly.

> +    struct vcpu *v = current;
> +
> +    /* Make sure that VCPU guest regs are zeroied */

s/zeroied/zeroed/

> +    memset(&ctxt, 0, sizeof(ctxt));
> +
> +    /* Set non-zero values to the registers prior to copying */
> +    ctxt.user_regs.pc64 = (u64)epoint;
> +
> +    if ( is_32bit_domain(current->domain) )
> +    {
> +        ctxt.user_regs.r0_usr = cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
> +
> +        /* Thumb set is allowed only for 32-bit domain */
> +        if ( epoint & 1 )
> +        {
> +            ctxt.user_regs.cpsr |= PSR_THUMB;
> +            ctxt.user_regs.pc64 &= ~(u64)1;
> +        }
> +    }
> +#ifdef CONFIG_ARM_64
> +    else
> +    {
> +        ctxt.user_regs.x0 = cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;
> +    }
> +#endif
> +    ctxt.sctlr = SCTLR_GUEST_INIT;
> +    ctxt.flags = VGCF_online;
> +
> +    /* Reset architecture specific registers */
> +    vcpu_arch_reset(v); > +
> +    /* Initialize VCPU registers */
> +    _arch_set_info_guest(v, &ctxt);

AFAICT, this is expected to be called with the domain lock taken as this 
can be called by various path.

Also, most of the function is the same as CPU_on. So I would like to see 
the code factored in the separate function and used in both place.

> +}
> +
> +int32_t domain_suspend(register_t epoint, register_t cid)
> +{
> +    struct vcpu *v;
> +    struct domain *d = current->domain;
> +    bool is_thumb = epoint & 1;
> +
> +    dprintk(XENLOG_DEBUG,
> +            "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> +            d->domain_id, epoint, cid);
> +
> +    /* THUMB set is not allowed with 64-bit domain */
> +    if ( is_64bit_domain(d) && is_thumb )
> +        return PSCI_INVALID_ADDRESS;
> +
> +    /* Ensure that all CPUs other than the calling one are offline */
> +    for_each_vcpu ( d, v )
> +    {
> +        if ( v != current && is_vcpu_online(v) )
> +            return PSCI_DENIED;
> +    }

What does prevent a vCPU to not come online while doing the loop?

> +
> +    /*
> +     * Prepare the calling VCPU for suspend (reset its context, save entry point
> +     * into pc and context ID into r0/x0 as specified by PSCI SYSTEM_SUSPEND)
> +     */
> +    vcpu_suspend(epoint, cid);
> +
> +    /*
> +     * Set the domain state to suspended (will be cleared when the domain
> +     * resumes, i.e. VCPU of this domain gets scheduled in).
> +     */
> +    d->is_shut_down = 1;
> +    d->shutdown_code = SHUTDOWN_suspend;

If you look at the other usage, you will notice that they are protected 
with a lock. Why is it not necessary here?

I am also not entirely sure why we could not re-use code that already 
exist in common code. Surely suspend/resume should work in a similar way?

> +
> +    /*
> +     * The calling domain is suspended by blocking its last running VCPU. If an
> +     * event is pending the domain will resume right away (VCPU will not block,
> +     * but when scheduled in it will resume from the given entry point).
> +     */
> +    vcpu_block_unless_event_pending(current);
> +
> +    return PSCI_SUCCESS;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vpsci.c b/xen/arch/arm/vpsci.c
> index 9f4e5b8844..f7922be0c5 100644
> --- a/xen/arch/arm/vpsci.c
> +++ b/xen/arch/arm/vpsci.c
> @@ -18,6 +18,7 @@
>   #include <asm/vgic.h>
>   #include <asm/vpsci.h>
>   #include <asm/event.h>
> +#include <asm/suspend.h>
>   
>   #include <public/sched.h>
>   
> @@ -210,6 +211,11 @@ static void do_psci_0_2_system_reset(void)
>       domain_shutdown(d,SHUTDOWN_reboot);
>   }
>   
> +static int32_t do_psci_1_0_system_suspend(register_t epoint, register_t cid)
> +{
> +    return domain_suspend(epoint, cid);
> +}
> +
>   static int32_t do_psci_1_0_features(uint32_t psci_func_id)
>   {
>       /* /!\ Ordered by function ID and not name */
> @@ -227,6 +233,8 @@ static int32_t do_psci_1_0_features(uint32_t psci_func_id)
>       case PSCI_0_2_FN32_SYSTEM_OFF:
>       case PSCI_0_2_FN32_SYSTEM_RESET:
>       case PSCI_1_0_FN32_PSCI_FEATURES:
> +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
>       case ARM_SMCCC_VERSION_FID:
>           return 0;
>       default:
> @@ -357,6 +365,17 @@ bool do_vpsci_0_2_call(struct cpu_user_regs *regs, uint32_t fid)
>           return true;
>       }
>   
> +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
> +    {
> +        register_t epoint = PSCI_ARG(regs,1);
> +        register_t cid = PSCI_ARG(regs,2);
> +
> +        perfc_incr(vpsci_system_suspend);
> +        PSCI_SET_RESULT(regs, do_psci_1_0_system_suspend(epoint, cid));
> +        return true;
> +    }
> +
>       default:
>           return false;
>       }
> diff --git a/xen/include/asm-arm/perfc_defn.h b/xen/include/asm-arm/perfc_defn.h
> index 8922e9525a..a02d0adea8 100644
> --- a/xen/include/asm-arm/perfc_defn.h
> +++ b/xen/include/asm-arm/perfc_defn.h
> @@ -32,6 +32,7 @@ PERFCOUNTER(vpsci_system_reset,        "vpsci: system_reset")
>   PERFCOUNTER(vpsci_cpu_suspend,         "vpsci: cpu_suspend")
>   PERFCOUNTER(vpsci_cpu_affinity_info,   "vpsci: cpu_affinity_info")
>   PERFCOUNTER(vpsci_features,            "vpsci: features")
> +PERFCOUNTER(vpsci_system_suspend,      "vpsci: system_suspend")
>   
>   PERFCOUNTER(vcpu_kick,                 "vcpu: notify other vcpu")
>   
> diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> index 832f77afff..26462d0c47 100644
> --- a/xen/include/asm-arm/psci.h
> +++ b/xen/include/asm-arm/psci.h
> @@ -43,10 +43,12 @@ void call_psci_system_reset(void);
>   #define PSCI_0_2_FN32_SYSTEM_OFF          PSCI_0_2_FN32(8)
>   #define PSCI_0_2_FN32_SYSTEM_RESET        PSCI_0_2_FN32(9)
>   #define PSCI_1_0_FN32_PSCI_FEATURES       PSCI_0_2_FN32(10)
> +#define PSCI_1_0_FN32_SYSTEM_SUSPEND      PSCI_0_2_FN32(14)
>   
>   #define PSCI_0_2_FN64_CPU_SUSPEND         PSCI_0_2_FN64(1)
>   #define PSCI_0_2_FN64_CPU_ON              PSCI_0_2_FN64(3)
>   #define PSCI_0_2_FN64_AFFINITY_INFO       PSCI_0_2_FN64(4)
> +#define PSCI_1_0_FN64_SYSTEM_SUSPEND      PSCI_0_2_FN64(14)
>   
>   /* PSCI v0.2 affinity level state returned by AFFINITY_INFO */
>   #define PSCI_0_2_AFFINITY_LEVEL_ON      0
> diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> new file mode 100644
> index 0000000000..de787d296a
> --- /dev/null
> +++ b/xen/include/asm-arm/suspend.h
> @@ -0,0 +1,16 @@
> +#ifndef __ASM_ARM_SUSPEND_H__
> +#define __ASM_ARM_SUSPEND_H__
> +
> +int32_t domain_suspend(register_t epoint, register_t cid);
> +
> +#endif
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> index 3171eabfd6..1f4e86524f 100644
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -24,6 +24,7 @@
>   #include <xen/wait.h>
>   #include <public/xen.h>
>   #include <public/domctl.h>
> +#include <public/sched.h>
>   #include <public/sysctl.h>
>   #include <public/vcpu.h>
>   #include <public/vm_event.h>
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Mirela Simonovic]

Hi Jan,

On Mon, Nov 12, 2018 at 4:23 PM Jan Beulich <JBeulich@suse.com> wrote:
>
> >>> On 12.11.18 at 16:17, <mirela.simonovic@aggios.com> wrote:
> > Hi Jan,
> >
> > On Mon, Nov 12, 2018 at 12:47 PM Jan Beulich <JBeulich@suse.com> wrote:
> >>
> >> >>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
> >> > --- a/xen/include/xen/timer.h
> >> > +++ b/xen/include/xen/timer.h
> >> > @@ -18,6 +18,9 @@ struct timer {
> >> >      /* System time expiry value (nanoseconds since boot). */
> >> >      s_time_t expires;
> >> >
> >> > +    /* Suspend timestamp value (nanoseconds since boot). */
> >> > +    s_time_t suspended;
> >>
> >> I can't see how this becomes a universally useful field, so I
> >> don't think it belongs here. For your purpose deriving a new
> >> structure containing struct timer and this one field should
> >> work.
> >
> > Each 'expires' value needs to be coupled with 'suspended' value to be
> > able to deal with watchdogs when suspending a domain. This is specific
> > to a watchdog.
> > If we add a derived structure to contain this field we still need to
> > associate such a structure with each watchdog of a domain, which is
> > also domain generic info and requires a lot more (in my opinion
> > unnecessary) changes. Please let me know if I misinterpreted your
> > proposal.
>
> You add the field to all timer instances, yet you need it only for
> very few. Why not have a struct watchdog_timer, and use it
> for struct domain's watchdog_timer[]? I'm afraid I don't see the
> difficulties you appear to see.
>

Thanks, now it's clear. We need to change the type for watchdog_timer
to be the derived structure of timer that additionally contains
'suspended' variable. That makes sense, we'll do so.

Thanks,
Mirela

> Jan
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Andrew Cooper]

On 12/11/18 15:31, Mirela Simonovic wrote:
> Hi Jan,
>
> On Mon, Nov 12, 2018 at 4:23 PM Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 12.11.18 at 16:17, <mirela.simonovic@aggios.com> wrote:
>>> Hi Jan,
>>>
>>> On Mon, Nov 12, 2018 at 12:47 PM Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 12.11.18 at 12:30, <mirela.simonovic@aggios.com> wrote:
>>>>> --- a/xen/include/xen/timer.h
>>>>> +++ b/xen/include/xen/timer.h
>>>>> @@ -18,6 +18,9 @@ struct timer {
>>>>>      /* System time expiry value (nanoseconds since boot). */
>>>>>      s_time_t expires;
>>>>>
>>>>> +    /* Suspend timestamp value (nanoseconds since boot). */
>>>>> +    s_time_t suspended;
>>>> I can't see how this becomes a universally useful field, so I
>>>> don't think it belongs here. For your purpose deriving a new
>>>> structure containing struct timer and this one field should
>>>> work.
>>> Each 'expires' value needs to be coupled with 'suspended' value to be
>>> able to deal with watchdogs when suspending a domain. This is specific
>>> to a watchdog.
>>> If we add a derived structure to contain this field we still need to
>>> associate such a structure with each watchdog of a domain, which is
>>> also domain generic info and requires a lot more (in my opinion
>>> unnecessary) changes. Please let me know if I misinterpreted your
>>> proposal.
>> You add the field to all timer instances, yet you need it only for
>> very few. Why not have a struct watchdog_timer, and use it
>> for struct domain's watchdog_timer[]? I'm afraid I don't see the
>> difficulties you appear to see.
>>
> Thanks, now it's clear. We need to change the type for watchdog_timer
> to be the derived structure of timer that additionally contains
> 'suspended' variable. That makes sense, we'll do so.

Why do you need a suspended variable?  Stash the frequency the guest
requested on the last hypercall, and fully kill/re-create the timer on
suspend/resume.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Julien Grall]

Hi Mirela,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> GIC and virtual timer context must be saved when the domain suspends.

Please provide the rationale for this. Is it required by the spec?

> This is done by moving the respective code in ctxt_switch_from()
> before the return that happens if the domain suspended.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/domain.c | 14 +++++++-------
>   1 file changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 7f8105465c..bebe3238e8 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
>       if ( is_idle_vcpu(p) )
>           return;
>   
> +    /* VGIC */
> +    gic_save_state(p);
> +
> +    /* Arch timer */
> +    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> +    virt_timer_save(p);
> +
>       /* VCPU's context should not be saved if its domain is suspended */

The GIC and timer are part of the vCPU context. So this comment looks a 
bit stale.

>       if ( p->domain->is_shut_down &&
>           (p->domain->shutdown_code == SHUTDOWN_suspend) )
> @@ -115,10 +122,6 @@ static void ctxt_switch_from(struct vcpu *p)
>       p->arch.tpidrro_el0 = READ_SYSREG(TPIDRRO_EL0);
>       p->arch.tpidr_el1 = READ_SYSREG(TPIDR_EL1);
>   
> -    /* Arch timer */
> -    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> -    virt_timer_save(p);
> -
>       if ( is_32bit_domain(p->domain) && cpu_has_thumbee )
>       {
>           p->arch.teecr = READ_SYSREG32(TEECR32_EL1);
> @@ -170,9 +173,6 @@ static void ctxt_switch_from(struct vcpu *p)
>       /* VFP */
>       vfp_save_state(p);
>   
> -    /* VGIC */
> -    gic_save_state(p);
> -
>       isb();
>   }
>   
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

Hi,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> When Dom0 finalizes its suspend procedure the suspend of Xen is
> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> its boot core (VCPU#0), which could be mapped to any physical CPU,
> i.e. the system_suspend() function could be executed by any physical
> CPU. Since Xen suspend procedure has to be run by the boot CPU
> (non-boot CPUs will be disabled at some point in suspend procedure),
> system_suspend() execution has to continue on CPU#0.

Nothing in the domain_suspend code checks that domain_suspend is called 
by vCPU0. I also can't find any restriction in the PSCI spec to run on 
pCPU0. Can you provide more details why this required?

> 
> When the system_suspend() returns 0, it means that the system was
> suspended and it is coming out of the resume procedure. Regardless
> of the system_suspend() return value, after this function returns
> Xen is fully functional, and its state, including all devices and data
> structures, matches the state prior to calling system_suspend().
> The status is returned by system_suspend() for debugging/logging
> purposes and function prototype compatibility.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
>   1 file changed, 34 insertions(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index f2338e41db..21b45f8248 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>       _arch_set_info_guest(v, &ctxt);
>   }
>   
> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> +static long system_suspend(void *data)
> +{
> +    BUG_ON(system_state != SYS_STATE_active);
> +
> +    return -ENOSYS;
> +}
> +
>   int32_t domain_suspend(register_t epoint, register_t cid)
>   {
>       struct vcpu *v;
>       struct domain *d = current->domain;
>       bool is_thumb = epoint & 1;
> +    int status;
>   
>       dprintk(XENLOG_DEBUG,
>               "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
>        */
>       vcpu_block_unless_event_pending(current);
>   
> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
> +    if ( is_hardware_domain(d) )
> +    {
> +        /*
> +         * system_suspend should be called when Dom0 finalizes the suspend
> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
> +         * be mapped to any PCPU (this function could be executed by any PCPU).
> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
> +         * PCPUs will be disabled during the suspend).
> +         */
> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> +        /*
> +         * If an error happened, there is nothing that needs to be done here
> +         * because the system_suspend always returns in fully functional state
> +         * no matter what the outcome of suspend procedure is. If the system
> +         * suspended successfully the function will return 0 after the resume.
> +         * Otherwise, if an error is returned it means Xen did not suspended,
> +         * but it is still in the same state as if the system_suspend was never
> +         * called. We dump a debug message in case of an error for debugging/
> +         * logging purpose.
> +         */
> +        if ( status )
> +            dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
> +    }
> +
>       return PSCI_SUCCESS;
>   }
>   
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 15/18] xen/arm: Resume memory management on Xen resume

[Julien Grall]

Hi,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> The MMU needs to be enabled in the resume flow before the context
> can be restored (we need to be able to access the context data by
> virtual address in order to restore it). The configuration of system
> registers prior to branching to the routine that sets up the page
> tables is copied from xen/arch/arm/arm64/head.S.
> After the MMU is enabled, the content of TTBR0_EL2 is changed to
> point to init_ttbr (page tables used at runtime).
> 
> At boot the init_ttbr variable is updated when a secondary CPU is
> hotplugged. In the scenario where there is only one physical CPU in
> the system, the init_ttbr would not be initialized for the use in
> resume flow. To get the variable initialized in all scenarios in this
> patch we add that the boot CPU updates init_ttbr after it sets the
> page tables for runtime.
> 
> After the memory management is resumed, the SCTLR_WXN in SCTLR_EL2
> has to be set in order to configure that a mapping cannot be both
> writable and executable (this was configured prior to suspend).
> This is done using an existing function (mmu_init_secondary_cpu).
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> 
> ---
> Changes in v2:
> 
> - Patch from v1:
> "[XEN PATCH 17/21] xen/arm: Set SCTLR_WXN in SCTLR_EL2 on resume"
> is squashed with this patch, because it is indeed related to resuming
> the memory management
> - Since the original patch was named 'Enable the MMU', and this is
> not only enabling anymore, but the full resume of functionality, the
> commit title and message is fixed
> ---
>   xen/arch/arm/arm64/entry.S | 88 ++++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/mm.c          |  1 +
>   xen/arch/arm/suspend.c     |  6 ++++
>   3 files changed, 95 insertions(+)
> 
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index dbc4717903..5efa30e8ee 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -1,6 +1,7 @@
>   #include <asm/asm_defns.h>
>   #include <asm/current.h>
>   #include <asm/macros.h>
> +#include <asm/page.h>
>   #include <asm/regs.h>
>   #include <asm/alternative.h>
>   #include <asm/smccc.h>
> @@ -534,6 +535,93 @@ ENTRY(__context_switch)
>           ret
>   
>   ENTRY(hyp_resume)
> +        msr   DAIFSet, 0xf           /* Disable all interrupts */
> +
> +        tlbi  alle2
> +        dsb   sy                     /* Ensure completion of TLB flush */
> +        isb
> +
> +        ldr   x0, =start
> +        adr   x19, start             /* x19 := paddr (start) */
> +        sub   x20, x19, x0           /* x20 := phys-offset */
> +
> +        /* XXXX call PROCINFO_cpu_init here */
> +
> +        /* Set up memory attribute type tables */
> +        ldr   x0, =MAIRVAL
> +        msr   mair_el2, x0
> +
> +        /* Set up TCR_EL2:
> +         * PS -- Based on ID_AA64MMFR0_EL1.PARange
> +         * Top byte is used
> +         * PT walks use Inner-Shareable accesses,
> +         * PT walks are write-back, write-allocate in both cache levels,
> +         * 48-bit virtual address space goes through this table. */
> +        ldr   x0, =(TCR_RES1|TCR_SH0_IS|TCR_ORGN0_WBWA|TCR_IRGN0_WBWA|TCR_T0SZ(64-48))
> +        /* ID_AA64MMFR0_EL1[3:0] (PARange) corresponds to TCR_EL2[18:16] (PS) */
> +        mrs   x1, ID_AA64MMFR0_EL1
> +        bfi   x0, x1, #16, #3
> +
> +        msr   tcr_el2, x0
> +
> +        /* Set up the SCTLR_EL2:
> +         * Exceptions in LE ARM,
> +         * Low-latency IRQs disabled,
> +         * Write-implies-XN disabled (for now),
> +         * D-cache disabled (for now),
> +         * I-cache enabled,
> +         * Alignment checking disabled,
> +         * MMU translation disabled (for now). */
> +        ldr   x0, =(HSCTLR_BASE)
> +        msr   SCTLR_EL2, x0
> +
> +        /* Ensure that any exceptions encountered at EL2
> +         * are handled using the EL2 stack pointer, rather
> +         * than SP_EL0. */
> +        msr spsel, #1
> +
> +        /* Rebuild the boot pagetable's first-level entries. The structure
> +         * is described in mm.c.
> +         *
> +         * After the CPU enables paging it will add the fixmap mapping
> +         * to these page tables, however this may clash with the 1:1
> +         * mapping. So each CPU must rebuild the page tables here with
> +         * the 1:1 in place. */
> +
> +        /* If Xen is loaded at exactly XEN_VIRT_START then we don't
> +         * need an additional 1:1 mapping, the virtual mapping will
> +         * suffice.
> +         */
> +        cmp   x19, #XEN_VIRT_START
> +        cset  x25, eq                /* x25 := identity map in place, or not */
> +
> +        /* Write Xen's PT's paddr into TTBR0_EL2 */
> +        ldr   x4, =boot_pgtable     /* xen_pgtable    */
> +        add   x4, x4, x20           /* x4 := paddr (boot_pagetable) */
> +        msr   TTBR0_EL2, x4
> +
> +        /* Set up page tables */
> +        bl    setup_page_tables
> +
> +        ldr   x1, =mmu_resumed      /* Explicit vaddr, not RIP-relative */
> +        mrs   x0, SCTLR_EL2
> +        orr   x0, x0, #SCTLR_M      /* Enable MMU */
> +        orr   x0, x0, #SCTLR_C      /* Enable D-cache */
> +        dsb   sy                    /* Flush PTE writes and finish reads */
> +        msr   SCTLR_EL2, x0         /* now paging is enabled */
> +        isb                         /* Now, flush the icache */
> +        br    x1                    /* Get a proper vaddr into PC */
> +
> +mmu_resumed:
> +        ldr   x4, =init_ttbr         /* VA of TTBR0_EL2 stashed by CPU 0 */

While I know that Xen is switching between page-tables in some place, 
this should not be done.  Indeed, such sequence may break coherency of 
the TLBs. In order to avoid that, you want to apply the 
break-before-sequence. I haven't fully think how to solve it in Xen 
properly today.

I am quite worried to introduce simlar sequence in Xen before fixing the 
current sequences.

> +        ldr   x4, [x4]               /* Actual value */
> +        dsb   sy
> +        msr   TTBR0_EL2, x4
> +        dsb   sy
> +        isb
> +        tlbi  alle2
> +        dsb   sy                     /* Ensure completion of TLB flush */
> +        isb
>           b .
>   
>   /*
> diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> index 7a06a33e21..7ad0ee1947 100644
> --- a/xen/arch/arm/mm.c
> +++ b/xen/arch/arm/mm.c
> @@ -722,6 +722,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
>       WRITE_SYSREG32(READ_SYSREG32(SCTLR_EL2) | SCTLR_WXN, SCTLR_EL2);
>       /* Flush everything after setting WXN bit. */
>       flush_xen_text_tlb_local();
> +    init_secondary_pagetables(0);
>   
>   #ifdef CONFIG_ARM_32
>       per_cpu(xen_pgtable, 0) = cpu0_pgtable;
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index 37926374bc..365c32ec3c 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -147,6 +147,12 @@ static long system_suspend(void *data)
>   
>       system_state = SYS_STATE_resume;
>   
> +    /*
> +     * SCTLR_WXN needs to be set to configure that a mapping cannot be both
> +     * writable and executable. This is done by mmu_init_secondary_cpu.
> +     */
> +    mmu_init_secondary_cpu();

This is a bit odd to call mmu_init_secondary_cpu() on boot CPU. You 
probably want a rename here.

> +
>       gic_resume();
>   
>   resume_irqs:
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

Thanks for your feedback, I'll need to answer in iterations.

On Mon, Nov 12, 2018 at 4:27 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > The implementation consists of:
> > -Adding PSCI system suspend call as new PSCI function
> > -Trapping PSCI system_suspend HVC
> > -Implementing PSCI system suspend call (virtual interface that allows
> >   guests to suspend themselves)
> >
> > The PSCI system suspend should be called by a guest from its boot
> > VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
> > CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
> > are left enabled by the guest are assumed to be its wake-up interrupts.
> > Therefore, a wake-up interrupt triggers the resume of the guest. Guest
> > should resume regardless of the state of Xen (suspended or not).
> >
> > When a guest calls PSCI system suspend the respective domain will be
> > suspended if the following conditions are met:
> > 1) Given resume entry point is not invalid
> > 2) Other (if any) VCPUs of the calling guest are hot-unplugged
> >
> > If the conditions above are met the calling domain is labeled as
> > suspended and the calling VCPU is blocked. If nothing else wouldn't
> > be done the suspended domain would resume from the place where it
> > called PSCI system suspend. This is expected if processing of the PSCI
> > system suspend call fails. However, in the case of success the calling
> > guest should resume (continue execution after the wake-up) from the entry
> > point which is given as the first argument of the PSCI system suspend
> > call. In addition to the entry point, the guest expects to start within
> > the environment whose state matches the state after reset. This means
> > that the guest should find reset register values, MMU disabled, etc.
> > Thereby, the context of VCPU should be 'reset' (as if the system is
> > comming out of reset), the program counter should contain entry point,
> > which is 1st argument, and r0/x0 should contain context ID which is 2nd
> > argument of PSCI system suspend call. The context of VCPU is set
> > accordingly when the PSCI system suspend is processed, so that nothing
> > needs to be done on resume/wake-up path. However, in order to ensure that
> > this context doesn't get overwritten by the scheduler when scheduling out
> > this VCPU (would normally happen after the calling CPU is blocked), we need
> > to check whether to return early from ctxt_switch_from().
> >
> > There are variables in domain structure to keep track of domain shutdown.
> > One of existing shutdown reason is 'suspend' which this patch is using to
> > track the suspend state of a domain. Those variables are used to determine
> > whether to early return from ctxt_switch_from() or not.
> >
> > A suspended domain will resume after the Xen receives an interrupt which is
> > targeted to the domain, unblocks the domain's VCPU, and schedules it in.
> > When the VCPU is scheduled in, the VCPU context is already reset, and
> > contains the right resume entry point in program counter that will be
> > restored in ctxt_switch_to(). The only thing that needs to be done at this
> > point is to clear the variables that marked the domain state as suspended.
> >
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >
> > ---
> > Changes in v2:
> >
> > -Fix print to compile for arm32 and to align with Xen coding style
> > ---
> >   xen/arch/arm/Makefile            |   1 +
> >   xen/arch/arm/domain.c            |  13 +++
> >   xen/arch/arm/suspend.c           | 166 +++++++++++++++++++++++++++++++++++++++
> >   xen/arch/arm/vpsci.c             |  19 +++++
> >   xen/include/asm-arm/perfc_defn.h |   1 +
> >   xen/include/asm-arm/psci.h       |   2 +
> >   xen/include/asm-arm/suspend.h    |  16 ++++
> >   xen/include/xen/sched.h          |   1 +
> >   8 files changed, 219 insertions(+)
> >   create mode 100644 xen/arch/arm/suspend.c
> >   create mode 100644 xen/include/asm-arm/suspend.h
> >
> > diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> > index 23c5d9adbc..744b1a4dc8 100644
> > --- a/xen/arch/arm/Makefile
> > +++ b/xen/arch/arm/Makefile
> > @@ -43,6 +43,7 @@ obj-y += setup.o
> >   obj-y += shutdown.o
> >   obj-y += smp.o
> >   obj-y += smpboot.o
> > +obj-y += suspend.o
> >   obj-y += sysctl.o
> >   obj-y += time.o
> >   obj-y += traps.o
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > index e594b48d81..7f8105465c 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >       if ( is_idle_vcpu(p) )
> >           return;
> >
> > +    /* VCPU's context should not be saved if its domain is suspended */
> > +    if ( p->domain->is_shut_down &&
> > +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> > +        return;
> SHUTDOWN_suspend is used in Xen for other purpose (see
> SCHEDOP_shutdown). The other user of that code relies on all the state
> to be saved on suspend.
>

We just need a flag to mark a domain as suspended, and I do believe
SHUTDOWN_suspend is not used anywhere else.
Let's come back on this.

> However, what is the issue with saving all the registers here?
>

We need to save arguments that are provided by a guest with system
suspend PSCI call. These arguments are the entry point that needs to
be saved in program counter and context ID that needs to be saved in
x0/r0. We don't have these arguments here. Context switch happens
after processing the system suspend PSCI call, so it's too late.

> > +
> >       p2m_save_state(p);
> >
> >       /* CP 15 */
> > @@ -181,6 +186,14 @@ static void ctxt_switch_to(struct vcpu *n)
> >       if ( is_idle_vcpu(n) )
> >           return;
> >
> > +    /* If the domain was suspended, it is resuming now */
> > +    if ( n->domain->is_shut_down &&
> > +        (n->domain->shutdown_code == SHUTDOWN_suspend) )
> > +    {
> > +        n->domain->is_shut_down = 0;
> > +        n->domain->shutdown_code = SHUTDOWN_CODE_INVALID;
> > +    }
>
> This looks like a hack. Why not calling domain_resume when receiving the
> interrupt?
>

Good point, I need to double check and come back on this.

> > +
> >       p2m_restore_state(n);
> >
> >       vpidr = READ_SYSREG32(MIDR_EL1);
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > new file mode 100644
> > index 0000000000..9eea9214e1
> > --- /dev/null
> > +++ b/xen/arch/arm/suspend.c
>
> I would prefer if we don't mix guest and host suspend in the same file.

Sure, we can move guest suspend code into an another file, e.g.
xen/arch/arm/vsuspend.c

>
> > @@ -0,0 +1,166 @@
>
> Missing copyright headers here.

Thanks

>
> > +#include <xen/sched.h>
> > +#include <asm/cpufeature.h>
> > +#include <asm/event.h>
> > +#include <asm/psci.h>
> > +
> > +/* Reset values of VCPU architecture specific registers */
>
> Technically this is not requires as most of the registers are unknown. I
> understand this helps for debugging an OS.
>
> I would introduce it in a separate patch and directly in
> arch_set_info_guest as I would like the behavior to be the same
> everywhere we need to reset a vCPU.

I agree. Please just consider that resetting a vCPU context is done in
2 scenarios: one where a vCPU is just created, and another one when
the vCPU already exists but the context has to be cleared. Could you
please provide some guidance on how to do this, because we struggled
for a while and didn't really find a nice way?

>
> > +static void vcpu_arch_reset(struct vcpu *v)
> > +{
> > +    v->arch.ttbr0 = 0;
> > +    v->arch.ttbr1 = 0;
> > +    v->arch.ttbcr = 0;
> > +
> > +    v->arch.csselr = 0;
> > +    v->arch.cpacr = 0;
> > +    v->arch.contextidr = 0;
> > +    v->arch.tpidr_el0 = 0;
> > +    v->arch.tpidrro_el0 = 0;
> > +    v->arch.tpidr_el1 = 0;
> > +    v->arch.vbar = 0;
> > +    if ( is_32bit_domain(v->domain) )
>
> This is not necessary
>
> > +        v->arch.dacr = 0;
> > +    v->arch.par = 0;
> > +#if defined(CONFIG_ARM_32)
> > +    v->arch.mair0 = 0;
> > +    v->arch.mair1 = 0;
> > +    v->arch.amair0 = 0;
> > +    v->arch.amair1 = 0;
> > +#else
> > +    v->arch.mair = 0;
> > +    v->arch.amair = 0;
> > +#endif
> > +    /* Fault Status */
> > +#if defined(CONFIG_ARM_32)
> > +    v->arch.dfar = 0;
> > +    v->arch.ifar = 0;
> > +    v->arch.dfsr = 0;
> > +#elif defined(CONFIG_ARM_64)
> > +    v->arch.far = 0;
> > +    v->arch.esr = 0;
> > +#endif
> > +
> > +    if ( is_32bit_domain(v->domain) )
>
> Same here.
>
> > +        v->arch.ifsr  = 0;
> > +    v->arch.afsr0 = 0;
> > +    v->arch.afsr1 = 0;
> > +
> > +#ifdef CONFIG_ARM_32
> > +    v->arch.joscr = 0;
> > +    v->arch.jmcr = 0;
> > +#endif
> > +
> > +    if ( is_32bit_domain(v->domain) && cpu_has_thumbee )
>
> Same here.
>
> > +    {
> > +        v->arch.teecr = 0;
> > +        v->arch.teehbr = 0;
> > +    }
> > +}
> > +
> > +/*
> > + * This function sets the context of current VCPU to the state which is expected
> > + * by the guest on resume. The expected VCPU state is:
> > + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
> > + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
> > + * 3) All other general purpose and system registers should have reset values
> > + *
> > + * Note: this function has to return void because it has to always succeed. In
> > + * other words, this function is called from virtual PSCI SYSTEM_SUSPEND
> > + * implementation, which can return only a limited number of possible errors,
> > + * none of which could represent the fact that an error occurred when preparing
> > + * the domain for suspend.
> > + * Consequently, dynamic memory allocation cannot be done within this function,
> > + * because if malloc fails the error has nowhere to propagate.
>
> You could crash the domain if you are not able to resume it. In the
> current context...
>
> > + */
> > +static void vcpu_suspend(register_t epoint, register_t cid)
> > +{
> > +    /* Static allocation because dynamic would need a non-void return */
> > +    static struct vcpu_guest_context ctxt;
>
> ... this is not right. This function can be called concurrently, so a
> lot of funny things can happen (i.e corruption).
>
> The vCPU context does not look too big. So I would just allocate it on
> the stack directly.
>

Agreed, 'static' should be removed to address all these issues.

> > +    struct vcpu *v = current;
> > +
> > +    /* Make sure that VCPU guest regs are zeroied */
>
> s/zeroied/zeroed/

Thanks

>
> > +    memset(&ctxt, 0, sizeof(ctxt));
> > +
> > +    /* Set non-zero values to the registers prior to copying */
> > +    ctxt.user_regs.pc64 = (u64)epoint;
> > +
> > +    if ( is_32bit_domain(current->domain) )
> > +    {
> > +        ctxt.user_regs.r0_usr = cid;
> > +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
> > +
> > +        /* Thumb set is allowed only for 32-bit domain */
> > +        if ( epoint & 1 )
> > +        {
> > +            ctxt.user_regs.cpsr |= PSR_THUMB;
> > +            ctxt.user_regs.pc64 &= ~(u64)1;
> > +        }
> > +    }
> > +#ifdef CONFIG_ARM_64
> > +    else
> > +    {
> > +        ctxt.user_regs.x0 = cid;
> > +        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;
> > +    }
> > +#endif
> > +    ctxt.sctlr = SCTLR_GUEST_INIT;
> > +    ctxt.flags = VGCF_online;
> > +
> > +    /* Reset architecture specific registers */
> > +    vcpu_arch_reset(v); > +
> > +    /* Initialize VCPU registers */
> > +    _arch_set_info_guest(v, &ctxt);
>
> AFAICT, this is expected to be called with the domain lock taken as this
> can be called by various path.
>
> Also, most of the function is the same as CPU_on. So I would like to see
> the code factored in the separate function and used in both place.
>

I agree, but the 2 scenarios (VCPU allocation versus clearing VCPU
context) made it a bit difficult to share. Please let me know if you
have some additional hint on how to exactly structure the code.

> > +}
> > +
> > +int32_t domain_suspend(register_t epoint, register_t cid)
> > +{
> > +    struct vcpu *v;
> > +    struct domain *d = current->domain;
> > +    bool is_thumb = epoint & 1;
> > +
> > +    dprintk(XENLOG_DEBUG,
> > +            "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> > +            d->domain_id, epoint, cid);
> > +
> > +    /* THUMB set is not allowed with 64-bit domain */
> > +    if ( is_64bit_domain(d) && is_thumb )
> > +        return PSCI_INVALID_ADDRESS;
> > +
> > +    /* Ensure that all CPUs other than the calling one are offline */
> > +    for_each_vcpu ( d, v )
> > +    {
> > +        if ( v != current && is_vcpu_online(v) )
> > +            return PSCI_DENIED;
> > +    }
>
> What does prevent a vCPU to not come online while doing the loop?

As you suggested probably nothing if there is a bug in the guest,
which we want to check for. Is the domain_lock right thing to use
here?

>
> > +
> > +    /*
> > +     * Prepare the calling VCPU for suspend (reset its context, save entry point
> > +     * into pc and context ID into r0/x0 as specified by PSCI SYSTEM_SUSPEND)
> > +     */
> > +    vcpu_suspend(epoint, cid);
> > +
> > +    /*
> > +     * Set the domain state to suspended (will be cleared when the domain
> > +     * resumes, i.e. VCPU of this domain gets scheduled in).
> > +     */
> > +    d->is_shut_down = 1;
> > +    d->shutdown_code = SHUTDOWN_suspend;
>
> If you look at the other usage, you will notice that they are protected
> with a lock. Why is it not necessary here?
>

I think it is necessary here too

> I am also not entirely sure why we could not re-use code that already
> exist in common code. Surely suspend/resume should work in a similar way?
>

Could you please be more specific (which common code)?

> > +
> > +    /*
> > +     * The calling domain is suspended by blocking its last running VCPU. If an
> > +     * event is pending the domain will resume right away (VCPU will not block,
> > +     * but when scheduled in it will resume from the given entry point).
> > +     */
> > +    vcpu_block_unless_event_pending(current);
> > +
> > +    return PSCI_SUCCESS;
> > +}
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/xen/arch/arm/vpsci.c b/xen/arch/arm/vpsci.c
> > index 9f4e5b8844..f7922be0c5 100644
> > --- a/xen/arch/arm/vpsci.c
> > +++ b/xen/arch/arm/vpsci.c
> > @@ -18,6 +18,7 @@
> >   #include <asm/vgic.h>
> >   #include <asm/vpsci.h>
> >   #include <asm/event.h>
> > +#include <asm/suspend.h>
> >
> >   #include <public/sched.h>
> >
> > @@ -210,6 +211,11 @@ static void do_psci_0_2_system_reset(void)
> >       domain_shutdown(d,SHUTDOWN_reboot);
> >   }
> >
> > +static int32_t do_psci_1_0_system_suspend(register_t epoint, register_t cid)
> > +{
> > +    return domain_suspend(epoint, cid);
> > +}
> > +
> >   static int32_t do_psci_1_0_features(uint32_t psci_func_id)
> >   {
> >       /* /!\ Ordered by function ID and not name */
> > @@ -227,6 +233,8 @@ static int32_t do_psci_1_0_features(uint32_t psci_func_id)
> >       case PSCI_0_2_FN32_SYSTEM_OFF:
> >       case PSCI_0_2_FN32_SYSTEM_RESET:
> >       case PSCI_1_0_FN32_PSCI_FEATURES:
> > +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> > +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
> >       case ARM_SMCCC_VERSION_FID:
> >           return 0;
> >       default:
> > @@ -357,6 +365,17 @@ bool do_vpsci_0_2_call(struct cpu_user_regs *regs, uint32_t fid)
> >           return true;
> >       }
> >
> > +    case PSCI_1_0_FN32_SYSTEM_SUSPEND:
> > +    case PSCI_1_0_FN64_SYSTEM_SUSPEND:
> > +    {
> > +        register_t epoint = PSCI_ARG(regs,1);
> > +        register_t cid = PSCI_ARG(regs,2);
> > +
> > +        perfc_incr(vpsci_system_suspend);
> > +        PSCI_SET_RESULT(regs, do_psci_1_0_system_suspend(epoint, cid));
> > +        return true;
> > +    }
> > +
> >       default:
> >           return false;
> >       }
> > diff --git a/xen/include/asm-arm/perfc_defn.h b/xen/include/asm-arm/perfc_defn.h
> > index 8922e9525a..a02d0adea8 100644
> > --- a/xen/include/asm-arm/perfc_defn.h
> > +++ b/xen/include/asm-arm/perfc_defn.h
> > @@ -32,6 +32,7 @@ PERFCOUNTER(vpsci_system_reset,        "vpsci: system_reset")
> >   PERFCOUNTER(vpsci_cpu_suspend,         "vpsci: cpu_suspend")
> >   PERFCOUNTER(vpsci_cpu_affinity_info,   "vpsci: cpu_affinity_info")
> >   PERFCOUNTER(vpsci_features,            "vpsci: features")
> > +PERFCOUNTER(vpsci_system_suspend,      "vpsci: system_suspend")
> >
> >   PERFCOUNTER(vcpu_kick,                 "vcpu: notify other vcpu")
> >
> > diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> > index 832f77afff..26462d0c47 100644
> > --- a/xen/include/asm-arm/psci.h
> > +++ b/xen/include/asm-arm/psci.h
> > @@ -43,10 +43,12 @@ void call_psci_system_reset(void);
> >   #define PSCI_0_2_FN32_SYSTEM_OFF          PSCI_0_2_FN32(8)
> >   #define PSCI_0_2_FN32_SYSTEM_RESET        PSCI_0_2_FN32(9)
> >   #define PSCI_1_0_FN32_PSCI_FEATURES       PSCI_0_2_FN32(10)
> > +#define PSCI_1_0_FN32_SYSTEM_SUSPEND      PSCI_0_2_FN32(14)
> >
> >   #define PSCI_0_2_FN64_CPU_SUSPEND         PSCI_0_2_FN64(1)
> >   #define PSCI_0_2_FN64_CPU_ON              PSCI_0_2_FN64(3)
> >   #define PSCI_0_2_FN64_AFFINITY_INFO       PSCI_0_2_FN64(4)
> > +#define PSCI_1_0_FN64_SYSTEM_SUSPEND      PSCI_0_2_FN64(14)
> >
> >   /* PSCI v0.2 affinity level state returned by AFFINITY_INFO */
> >   #define PSCI_0_2_AFFINITY_LEVEL_ON      0
> > diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> > new file mode 100644
> > index 0000000000..de787d296a
> > --- /dev/null
> > +++ b/xen/include/asm-arm/suspend.h
> > @@ -0,0 +1,16 @@
> > +#ifndef __ASM_ARM_SUSPEND_H__
> > +#define __ASM_ARM_SUSPEND_H__
> > +
> > +int32_t domain_suspend(register_t epoint, register_t cid);
> > +
> > +#endif
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * tab-width: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> > index 3171eabfd6..1f4e86524f 100644
> > --- a/xen/include/xen/sched.h
> > +++ b/xen/include/xen/sched.h
> > @@ -24,6 +24,7 @@
> >   #include <xen/wait.h>
> >   #include <public/xen.h>
> >   #include <public/domctl.h>
> > +#include <public/sched.h>
> >   #include <public/sysctl.h>
> >   #include <public/vcpu.h>
> >   #include <public/vm_event.h>
> >
>
> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Andrew Cooper]

On 12/11/18 16:35, Mirela Simonovic wrote:
> Hi Julien,
>
> Thanks for your feedback, I'll need to answer in iterations.
>
> On Mon, Nov 12, 2018 at 4:27 PM Julien Grall <julien.grall@arm.com> wrote:
>> Hi Mirela,
>>
>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>> The implementation consists of:
>>> -Adding PSCI system suspend call as new PSCI function
>>> -Trapping PSCI system_suspend HVC
>>> -Implementing PSCI system suspend call (virtual interface that allows
>>>   guests to suspend themselves)
>>>
>>> The PSCI system suspend should be called by a guest from its boot
>>> VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
>>> CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
>>> are left enabled by the guest are assumed to be its wake-up interrupts.
>>> Therefore, a wake-up interrupt triggers the resume of the guest. Guest
>>> should resume regardless of the state of Xen (suspended or not).
>>>
>>> When a guest calls PSCI system suspend the respective domain will be
>>> suspended if the following conditions are met:
>>> 1) Given resume entry point is not invalid
>>> 2) Other (if any) VCPUs of the calling guest are hot-unplugged
>>>
>>> If the conditions above are met the calling domain is labeled as
>>> suspended and the calling VCPU is blocked. If nothing else wouldn't
>>> be done the suspended domain would resume from the place where it
>>> called PSCI system suspend. This is expected if processing of the PSCI
>>> system suspend call fails. However, in the case of success the calling
>>> guest should resume (continue execution after the wake-up) from the entry
>>> point which is given as the first argument of the PSCI system suspend
>>> call. In addition to the entry point, the guest expects to start within
>>> the environment whose state matches the state after reset. This means
>>> that the guest should find reset register values, MMU disabled, etc.
>>> Thereby, the context of VCPU should be 'reset' (as if the system is
>>> comming out of reset), the program counter should contain entry point,
>>> which is 1st argument, and r0/x0 should contain context ID which is 2nd
>>> argument of PSCI system suspend call. The context of VCPU is set
>>> accordingly when the PSCI system suspend is processed, so that nothing
>>> needs to be done on resume/wake-up path. However, in order to ensure that
>>> this context doesn't get overwritten by the scheduler when scheduling out
>>> this VCPU (would normally happen after the calling CPU is blocked), we need
>>> to check whether to return early from ctxt_switch_from().
>>>
>>> There are variables in domain structure to keep track of domain shutdown.
>>> One of existing shutdown reason is 'suspend' which this patch is using to
>>> track the suspend state of a domain. Those variables are used to determine
>>> whether to early return from ctxt_switch_from() or not.
>>>
>>> A suspended domain will resume after the Xen receives an interrupt which is
>>> targeted to the domain, unblocks the domain's VCPU, and schedules it in.
>>> When the VCPU is scheduled in, the VCPU context is already reset, and
>>> contains the right resume entry point in program counter that will be
>>> restored in ctxt_switch_to(). The only thing that needs to be done at this
>>> point is to clear the variables that marked the domain state as suspended.
>>>
>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>>
>>> ---
>>> Changes in v2:
>>>
>>> -Fix print to compile for arm32 and to align with Xen coding style
>>> ---
>>>   xen/arch/arm/Makefile            |   1 +
>>>   xen/arch/arm/domain.c            |  13 +++
>>>   xen/arch/arm/suspend.c           | 166 +++++++++++++++++++++++++++++++++++++++
>>>   xen/arch/arm/vpsci.c             |  19 +++++
>>>   xen/include/asm-arm/perfc_defn.h |   1 +
>>>   xen/include/asm-arm/psci.h       |   2 +
>>>   xen/include/asm-arm/suspend.h    |  16 ++++
>>>   xen/include/xen/sched.h          |   1 +
>>>   8 files changed, 219 insertions(+)
>>>   create mode 100644 xen/arch/arm/suspend.c
>>>   create mode 100644 xen/include/asm-arm/suspend.h
>>>
>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>> index 23c5d9adbc..744b1a4dc8 100644
>>> --- a/xen/arch/arm/Makefile
>>> +++ b/xen/arch/arm/Makefile
>>> @@ -43,6 +43,7 @@ obj-y += setup.o
>>>   obj-y += shutdown.o
>>>   obj-y += smp.o
>>>   obj-y += smpboot.o
>>> +obj-y += suspend.o
>>>   obj-y += sysctl.o
>>>   obj-y += time.o
>>>   obj-y += traps.o
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index e594b48d81..7f8105465c 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>       if ( is_idle_vcpu(p) )
>>>           return;
>>>
>>> +    /* VCPU's context should not be saved if its domain is suspended */
>>> +    if ( p->domain->is_shut_down &&
>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>> +        return;
>> SHUTDOWN_suspend is used in Xen for other purpose (see
>> SCHEDOP_shutdown). The other user of that code relies on all the state
>> to be saved on suspend.
>>
> We just need a flag to mark a domain as suspended, and I do believe
> SHUTDOWN_suspend is not used anywhere else.
> Let's come back on this.

SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
tree and you'll find several pieces of documentation, including the
description of what this shutdown code means.

What you are introducing here is not a shutdown - it is a suspend with
the intent to resume executing later.  As such, it shouldn't use Xen's
shutdown infrastructure, which exists mainly to communicate with the
toolstack.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Mirela Simonovic]

Hi Julien,

Thanks for the feedback.

On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > GIC and virtual timer context must be saved when the domain suspends.
>
> Please provide the rationale for this. Is it required by the spec?
>

This is required for GIC because a guest leaves enabled interrupts in
the GIC that could wake it up, and on resume it should be able to
detect which interrupt woke it up. Without saving/restoring the state
of GIC this would not be possible.
For timer, my understanding is that vtimer accounts for how long the
guest was executing and this tracking should not be affected by the
suspend/resume sequence. Without saving/restoring the state of vtimer,
the time tracking would be affected by the suspend/resume.
Hope this is ok, I'll add it to the commit message.

> > This is done by moving the respective code in ctxt_switch_from()
> > before the return that happens if the domain suspended.
> >
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> > ---
> >   xen/arch/arm/domain.c | 14 +++++++-------
> >   1 file changed, 7 insertions(+), 7 deletions(-)
> >
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > index 7f8105465c..bebe3238e8 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
> >       if ( is_idle_vcpu(p) )
> >           return;
> >
> > +    /* VGIC */
> > +    gic_save_state(p);
> > +
> > +    /* Arch timer */
> > +    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> > +    virt_timer_save(p);
> > +
> >       /* VCPU's context should not be saved if its domain is suspended */
>
> The GIC and timer are part of the vCPU context. So this comment looks a
> bit stale.

It's not stale, but incorrect in that perspective. The comment should
be updated to "VCPU architecture specific context should ..."

>
> >       if ( p->domain->is_shut_down &&
> >           (p->domain->shutdown_code == SHUTDOWN_suspend) )
> > @@ -115,10 +122,6 @@ static void ctxt_switch_from(struct vcpu *p)
> >       p->arch.tpidrro_el0 = READ_SYSREG(TPIDRRO_EL0);
> >       p->arch.tpidr_el1 = READ_SYSREG(TPIDR_EL1);
> >
> > -    /* Arch timer */
> > -    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> > -    virt_timer_save(p);
> > -
> >       if ( is_32bit_domain(p->domain) && cpu_has_thumbee )
> >       {
> >           p->arch.teecr = READ_SYSREG32(TEECR32_EL1);
> > @@ -170,9 +173,6 @@ static void ctxt_switch_from(struct vcpu *p)
> >       /* VFP */
> >       vfp_save_state(p);
> >
> > -    /* VGIC */
> > -    gic_save_state(p);
> > -
> >       isb();
> >   }
> >
> >
>
> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Julien Grall]

On 11/12/18 4:52 PM, Mirela Simonovic wrote:
> Hi Julien,

Hi,

> Thanks for the feedback.
> 
> On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com> wrote:
>>
>> Hi Mirela,
>>
>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>> GIC and virtual timer context must be saved when the domain suspends.
>>
>> Please provide the rationale for this. Is it required by the spec?
>>
> 
> This is required for GIC because a guest leaves enabled interrupts in
> the GIC that could wake it up, and on resume it should be able to
> detect which interrupt woke it up. Without saving/restoring the state
> of GIC this would not be possible.

I am confused. In patch #10, you save the GIC host because the GIC can 
be powered-down. Linux is also saving the GIC state. So how the 
interrupt can come up if the GIC is powered down?

> For timer, my understanding is that vtimer accounts for how long the
> guest was executing and this tracking should not be affected by the
> suspend/resume sequence. Without saving/restoring the state of vtimer,
> the time tracking would be affected by the suspend/resume.

Currently, vtimer does not take such things into account. It only setup 
a timer in Xen to wake up the guest later on.

> Hope this is ok, I'll add it to the commit message.
> 
>>> This is done by moving the respective code in ctxt_switch_from()
>>> before the return that happens if the domain suspended.
>>>
>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>> ---
>>>    xen/arch/arm/domain.c | 14 +++++++-------
>>>    1 file changed, 7 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index 7f8105465c..bebe3238e8 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
>>>        if ( is_idle_vcpu(p) )
>>>            return;
>>>
>>> +    /* VGIC */
>>> +    gic_save_state(p);
>>> +
>>> +    /* Arch timer */
>>> +    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
>>> +    virt_timer_save(p);
>>> +
>>>        /* VCPU's context should not be saved if its domain is suspended */
>>
>> The GIC and timer are part of the vCPU context. So this comment looks a
>> bit stale.
> 
> It's not stale, but incorrect in that perspective. The comment should
> be updated to "VCPU architecture specific context should ..."

But the timer is part of the architecture...

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Mirela Simonovic]

Hi Julien,

On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com> wrote:
>
>
>
> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
> > Hi Julien,
>
> Hi,
>
> > Thanks for the feedback.
> >
> > On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com> wrote:
> >>
> >> Hi Mirela,
> >>
> >> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>> GIC and virtual timer context must be saved when the domain suspends.
> >>
> >> Please provide the rationale for this. Is it required by the spec?
> >>
> >
> > This is required for GIC because a guest leaves enabled interrupts in
> > the GIC that could wake it up, and on resume it should be able to
> > detect which interrupt woke it up. Without saving/restoring the state
> > of GIC this would not be possible.
>
> I am confused. In patch #10, you save the GIC host because the GIC can
> be powered-down. Linux is also saving the GIC state. So how the
> interrupt can come up if the GIC is powered down?
>

After Xen (or Linux in the config without Xen) hands over the control
to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
enabled interrupts that are its wake-up sources. Saving a GIC state
only means that its current configuration will be remembered somewhere
in data structures, but the configuration is not changed on suspend.
Everything that happens with interrupt configuration from this point
on is platform specific. Now there are 2 options: 1) EL3 will never
allow CPU0 to be powered down and the wake-up interrupt will indeed
propagate via GIC; or 2) CPU0 will be powered down and the GIC may be
powered down as well, so an external help is needed to deal with
wake-up and resume (e.g. in order to react to a wake-up interrupt
while the GIC is down, and power up CPU0). This external help is
provided by some low-power processor outside of the Cortex-A cluster.

So the platform firmware is responsible for properly configuring a
wake-up path if GIC goes down. This is commonly handled by EL3
communicating with low-power processor. When the GIC power comes up,
the interrupt triggered by a peripheral is still active and the
software on Cortex-A cluster should be able to observe it once the GIC
state is restored, i.e. interrupts get enabled at GIC.

> > For timer, my understanding is that vtimer accounts for how long the
> > guest was executing and this tracking should not be affected by the
> > suspend/resume sequence. Without saving/restoring the state of vtimer,
> > the time tracking would be affected by the suspend/resume.
>
> Currently, vtimer does not take such things into account. It only setup
> a timer in Xen to wake up the guest later on.
>
> > Hope this is ok, I'll add it to the commit message.
> >
> >>> This is done by moving the respective code in ctxt_switch_from()
> >>> before the return that happens if the domain suspended.
> >>>
> >>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> >>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >>> ---
> >>>    xen/arch/arm/domain.c | 14 +++++++-------
> >>>    1 file changed, 7 insertions(+), 7 deletions(-)
> >>>
> >>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>> index 7f8105465c..bebe3238e8 100644
> >>> --- a/xen/arch/arm/domain.c
> >>> +++ b/xen/arch/arm/domain.c
> >>> @@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>        if ( is_idle_vcpu(p) )
> >>>            return;
> >>>
> >>> +    /* VGIC */
> >>> +    gic_save_state(p);
> >>> +
> >>> +    /* Arch timer */
> >>> +    p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> >>> +    virt_timer_save(p);
> >>> +
> >>>        /* VCPU's context should not be saved if its domain is suspended */
> >>
> >> The GIC and timer are part of the vCPU context. So this comment looks a
> >> bit stale.
> >
> > It's not stale, but incorrect in that perspective. The comment should
> > be updated to "VCPU architecture specific context should ..."
>
> But the timer is part of the architecture...
>
> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Julien Grall]

(+ Andre)

On 11/12/18 5:42 PM, Mirela Simonovic wrote:
> Hi Julien,
> 
> On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com> wrote:
>>
>>
>>
>> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
>>> Hi Julien,
>>
>> Hi,
>>
>>> Thanks for the feedback.
>>>
>>> On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com> wrote:
>>>>
>>>> Hi Mirela,
>>>>
>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>> GIC and virtual timer context must be saved when the domain suspends.
>>>>
>>>> Please provide the rationale for this. Is it required by the spec?
>>>>
>>>
>>> This is required for GIC because a guest leaves enabled interrupts in
>>> the GIC that could wake it up, and on resume it should be able to
>>> detect which interrupt woke it up. Without saving/restoring the state
>>> of GIC this would not be possible.
>>
>> I am confused. In patch #10, you save the GIC host because the GIC can
>> be powered-down. Linux is also saving the GIC state. So how the
>> interrupt can come up if the GIC is powered down?
>>
> 
> After Xen (or Linux in the config without Xen) hands over the control
> to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
> enabled interrupts that are its wake-up sources. Saving a GIC state
> only means that its current configuration will be remembered somewhere
> in data structures, but the configuration is not changed on suspend.
> Everything that happens with interrupt configuration from this point
> on is platform specific. Now there are 2 options: 1) EL3 will never
> allow CPU0 to be powered down and the wake-up interrupt will indeed
> propagate via GIC;
> or 2) CPU0 will be powered down and the GIC may be
> powered down as well, so an external help is needed to deal with
> wake-up and resume (e.g. in order to react to a wake-up interrupt
> while the GIC is down, and power up CPU0). This external help is
> provided by some low-power processor outside of the Cortex-A cluster.
> 
> So the platform firmware is responsible for properly configuring a
> wake-up path if GIC goes down. This is commonly handled by EL3
> communicating with low-power processor. When the GIC power comes up,
> the interrupt triggered by a peripheral is still active and the
> software on Cortex-A cluster should be able to observe it once the GIC
> state is restored, i.e. interrupts get enabled at GIC.

Thank you for the explanation.  Now the question is why can't we reset 
at least the GIC CPU interface?

AFAIU, the guest should restore them in any case. The only things we 
need to know is the interrupt was received for a given guest. We can 
then queue it and wake-up the domain.

This seems to fit with the description on top of gic_dist_save() in 
Linux GICv2 driver.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Andrew,

On 11/12/18 4:41 PM, Andrew Cooper wrote:
> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>> index e594b48d81..7f8105465c 100644
>>>> --- a/xen/arch/arm/domain.c
>>>> +++ b/xen/arch/arm/domain.c
>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>        if ( is_idle_vcpu(p) )
>>>>            return;
>>>>
>>>> +    /* VCPU's context should not be saved if its domain is suspended */
>>>> +    if ( p->domain->is_shut_down &&
>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>> +        return;
>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>> SCHEDOP_shutdown). The other user of that code relies on all the state
>>> to be saved on suspend.
>>>
>> We just need a flag to mark a domain as suspended, and I do believe
>> SHUTDOWN_suspend is not used anywhere else.
>> Let's come back on this.
> 
> SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
> tree and you'll find several pieces of documentation, including the
> description of what this shutdown code means.
> 
> What you are introducing here is not a shutdown - it is a suspend with
> the intent to resume executing later.  As such, it shouldn't use Xen's
> shutdown infrastructure, which exists mainly to communicate with the
> toolstack.

Would domain pause/unpause be a better solution?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

On 11/12/18 4:35 PM, Mirela Simonovic wrote:
> Hi Julien,
> 
> Thanks for your feedback, I'll need to answer in iterations.
> 
> On Mon, Nov 12, 2018 at 4:27 PM Julien Grall <julien.grall@arm.com> wrote:
>>
>> Hi Mirela,
>>
>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>> The implementation consists of:
>>> -Adding PSCI system suspend call as new PSCI function
>>> -Trapping PSCI system_suspend HVC
>>> -Implementing PSCI system suspend call (virtual interface that allows
>>>    guests to suspend themselves)
>>>
>>> The PSCI system suspend should be called by a guest from its boot
>>> VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
>>> CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
>>> are left enabled by the guest are assumed to be its wake-up interrupts.
>>> Therefore, a wake-up interrupt triggers the resume of the guest. Guest
>>> should resume regardless of the state of Xen (suspended or not).
>>>
>>> When a guest calls PSCI system suspend the respective domain will be
>>> suspended if the following conditions are met:
>>> 1) Given resume entry point is not invalid
>>> 2) Other (if any) VCPUs of the calling guest are hot-unplugged
>>>
>>> If the conditions above are met the calling domain is labeled as
>>> suspended and the calling VCPU is blocked. If nothing else wouldn't
>>> be done the suspended domain would resume from the place where it
>>> called PSCI system suspend. This is expected if processing of the PSCI
>>> system suspend call fails. However, in the case of success the calling
>>> guest should resume (continue execution after the wake-up) from the entry
>>> point which is given as the first argument of the PSCI system suspend
>>> call. In addition to the entry point, the guest expects to start within
>>> the environment whose state matches the state after reset. This means
>>> that the guest should find reset register values, MMU disabled, etc.
>>> Thereby, the context of VCPU should be 'reset' (as if the system is
>>> comming out of reset), the program counter should contain entry point,
>>> which is 1st argument, and r0/x0 should contain context ID which is 2nd
>>> argument of PSCI system suspend call. The context of VCPU is set
>>> accordingly when the PSCI system suspend is processed, so that nothing
>>> needs to be done on resume/wake-up path. However, in order to ensure that
>>> this context doesn't get overwritten by the scheduler when scheduling out
>>> this VCPU (would normally happen after the calling CPU is blocked), we need
>>> to check whether to return early from ctxt_switch_from().
>>>
>>> There are variables in domain structure to keep track of domain shutdown.
>>> One of existing shutdown reason is 'suspend' which this patch is using to
>>> track the suspend state of a domain. Those variables are used to determine
>>> whether to early return from ctxt_switch_from() or not.
>>>
>>> A suspended domain will resume after the Xen receives an interrupt which is
>>> targeted to the domain, unblocks the domain's VCPU, and schedules it in.
>>> When the VCPU is scheduled in, the VCPU context is already reset, and
>>> contains the right resume entry point in program counter that will be
>>> restored in ctxt_switch_to(). The only thing that needs to be done at this
>>> point is to clear the variables that marked the domain state as suspended.
>>>
>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>>
>>> ---
>>> Changes in v2:
>>>
>>> -Fix print to compile for arm32 and to align with Xen coding style
>>> ---
>>>    xen/arch/arm/Makefile            |   1 +
>>>    xen/arch/arm/domain.c            |  13 +++
>>>    xen/arch/arm/suspend.c           | 166 +++++++++++++++++++++++++++++++++++++++
>>>    xen/arch/arm/vpsci.c             |  19 +++++
>>>    xen/include/asm-arm/perfc_defn.h |   1 +
>>>    xen/include/asm-arm/psci.h       |   2 +
>>>    xen/include/asm-arm/suspend.h    |  16 ++++
>>>    xen/include/xen/sched.h          |   1 +
>>>    8 files changed, 219 insertions(+)
>>>    create mode 100644 xen/arch/arm/suspend.c
>>>    create mode 100644 xen/include/asm-arm/suspend.h
>>>
>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>> index 23c5d9adbc..744b1a4dc8 100644
>>> --- a/xen/arch/arm/Makefile
>>> +++ b/xen/arch/arm/Makefile
>>> @@ -43,6 +43,7 @@ obj-y += setup.o
>>>    obj-y += shutdown.o
>>>    obj-y += smp.o
>>>    obj-y += smpboot.o
>>> +obj-y += suspend.o
>>>    obj-y += sysctl.o
>>>    obj-y += time.o
>>>    obj-y += traps.o
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index e594b48d81..7f8105465c 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>        if ( is_idle_vcpu(p) )
>>>            return;
>>>
>>> +    /* VCPU's context should not be saved if its domain is suspended */
>>> +    if ( p->domain->is_shut_down &&
>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>> +        return;
>> SHUTDOWN_suspend is used in Xen for other purpose (see
>> SCHEDOP_shutdown). The other user of that code relies on all the state
>> to be saved on suspend.
>>
> 
> We just need a flag to mark a domain as suspended, and I do believe
> SHUTDOWN_suspend is not used anywhere else.
> Let's come back on this.

See Andrew's comment here.

> 
>> However, what is the issue with saving all the registers here?
>>
> 
> We need to save arguments that are provided by a guest with system
> suspend PSCI call. These arguments are the entry point that needs to
> be saved in program counter and context ID that needs to be saved in
> x0/r0. We don't have these arguments here. Context switch happens
> after processing the system suspend PSCI call, so it's too late.

It does not feel right to modify ctxt_switch{from,to} for 
suspend/resume. If you want to reset the vCPU state before blocking the 
vCPU, then you should instead

Another way would be to reset the vCPU once you receive the interrupt.

>>> +
>>>        p2m_save_state(p);
>>>
>>>        /* CP 15 */
>>> @@ -181,6 +186,14 @@ static void ctxt_switch_to(struct vcpu *n)
>>>        if ( is_idle_vcpu(n) )
>>>            return;
>>>
>>> +    /* If the domain was suspended, it is resuming now */
>>> +    if ( n->domain->is_shut_down &&
>>> +        (n->domain->shutdown_code == SHUTDOWN_suspend) )
>>> +    {
>>> +        n->domain->is_shut_down = 0;
>>> +        n->domain->shutdown_code = SHUTDOWN_CODE_INVALID;
>>> +    }
>>
>> This looks like a hack. Why not calling domain_resume when receiving the
>> interrupt?
>>
> 
> Good point, I need to double check and come back on this.

It looks like domain_resume may not be the right solution (see Andrew's 
comment). Another solution would be to use domain_pause/domain_unpause.

> 
>>> +
>>>        p2m_restore_state(n);
>>>
>>>        vpidr = READ_SYSREG32(MIDR_EL1);
>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>>> new file mode 100644
>>> index 0000000000..9eea9214e1
>>> --- /dev/null
>>> +++ b/xen/arch/arm/suspend.c
>>
>> I would prefer if we don't mix guest and host suspend in the same file.
> 
> Sure, we can move guest suspend code into an another file, e.g.
> xen/arch/arm/vsuspend.c

Do we really need a separate file? The code does not look too big.

> 
>>
>>> @@ -0,0 +1,166 @@
>>
>> Missing copyright headers here.
> 
> Thanks
> 
>>
>>> +#include <xen/sched.h>
>>> +#include <asm/cpufeature.h>
>>> +#include <asm/event.h>
>>> +#include <asm/psci.h>
>>> +
>>> +/* Reset values of VCPU architecture specific registers */
>>
>> Technically this is not requires as most of the registers are unknown. I
>> understand this helps for debugging an OS.
>>
>> I would introduce it in a separate patch and directly in
>> arch_set_info_guest as I would like the behavior to be the same
>> everywhere we need to reset a vCPU.
> 
> I agree. Please just consider that resetting a vCPU context is done in
> 2 scenarios: one where a vCPU is just created, and another one when
> the vCPU already exists but the context has to be cleared. Could you
> please provide some guidance on how to do this, because we struggled
> for a while and didn't really find a nice way?

Why do you think the 2 scenarios requires different path? In both case 
the CPU is off and should be reset. It is just the current PSCI CPU on 
took some shortcut that likely needs to be fixed.

> 
>>
>>> +static void vcpu_arch_reset(struct vcpu *v)
>>> +{
>>> +    v->arch.ttbr0 = 0;
>>> +    v->arch.ttbr1 = 0;
>>> +    v->arch.ttbcr = 0;
>>> +
>>> +    v->arch.csselr = 0;
>>> +    v->arch.cpacr = 0;
>>> +    v->arch.contextidr = 0;
>>> +    v->arch.tpidr_el0 = 0;
>>> +    v->arch.tpidrro_el0 = 0;
>>> +    v->arch.tpidr_el1 = 0;
>>> +    v->arch.vbar = 0;
>>> +    if ( is_32bit_domain(v->domain) )
>>
>> This is not necessary
>>
>>> +        v->arch.dacr = 0;
>>> +    v->arch.par = 0;
>>> +#if defined(CONFIG_ARM_32)
>>> +    v->arch.mair0 = 0;
>>> +    v->arch.mair1 = 0;
>>> +    v->arch.amair0 = 0;
>>> +    v->arch.amair1 = 0;
>>> +#else
>>> +    v->arch.mair = 0;
>>> +    v->arch.amair = 0;
>>> +#endif
>>> +    /* Fault Status */
>>> +#if defined(CONFIG_ARM_32)
>>> +    v->arch.dfar = 0;
>>> +    v->arch.ifar = 0;
>>> +    v->arch.dfsr = 0;
>>> +#elif defined(CONFIG_ARM_64)
>>> +    v->arch.far = 0;
>>> +    v->arch.esr = 0;
>>> +#endif
>>> +
>>> +    if ( is_32bit_domain(v->domain) )
>>
>> Same here.
>>
>>> +        v->arch.ifsr  = 0;
>>> +    v->arch.afsr0 = 0;
>>> +    v->arch.afsr1 = 0;
>>> +
>>> +#ifdef CONFIG_ARM_32
>>> +    v->arch.joscr = 0;
>>> +    v->arch.jmcr = 0;
>>> +#endif
>>> +
>>> +    if ( is_32bit_domain(v->domain) && cpu_has_thumbee )
>>
>> Same here.
>>
>>> +    {
>>> +        v->arch.teecr = 0;
>>> +        v->arch.teehbr = 0;
>>> +    }
>>> +}
>>> +
>>> +/*
>>> + * This function sets the context of current VCPU to the state which is expected
>>> + * by the guest on resume. The expected VCPU state is:
>>> + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
>>> + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
>>> + * 3) All other general purpose and system registers should have reset values
>>> + *
>>> + * Note: this function has to return void because it has to always succeed. In
>>> + * other words, this function is called from virtual PSCI SYSTEM_SUSPEND
>>> + * implementation, which can return only a limited number of possible errors,
>>> + * none of which could represent the fact that an error occurred when preparing
>>> + * the domain for suspend.
>>> + * Consequently, dynamic memory allocation cannot be done within this function,
>>> + * because if malloc fails the error has nowhere to propagate.
>>
>> You could crash the domain if you are not able to resume it. In the
>> current context...
>>
>>> + */
>>> +static void vcpu_suspend(register_t epoint, register_t cid)
>>> +{
>>> +    /* Static allocation because dynamic would need a non-void return */
>>> +    static struct vcpu_guest_context ctxt;
>>
>> ... this is not right. This function can be called concurrently, so a
>> lot of funny things can happen (i.e corruption).
>>
>> The vCPU context does not look too big. So I would just allocate it on
>> the stack directly.
>>
> 
> Agreed, 'static' should be removed to address all these issues.
> 
>>> +    struct vcpu *v = current;
>>> +
>>> +    /* Make sure that VCPU guest regs are zeroied */
>>
>> s/zeroied/zeroed/
> 
> Thanks
> 
>>
>>> +    memset(&ctxt, 0, sizeof(ctxt));
>>> +
>>> +    /* Set non-zero values to the registers prior to copying */
>>> +    ctxt.user_regs.pc64 = (u64)epoint;
>>> +
>>> +    if ( is_32bit_domain(current->domain) )
>>> +    {
>>> +        ctxt.user_regs.r0_usr = cid;
>>> +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
>>> +
>>> +        /* Thumb set is allowed only for 32-bit domain */
>>> +        if ( epoint & 1 )
>>> +        {
>>> +            ctxt.user_regs.cpsr |= PSR_THUMB;
>>> +            ctxt.user_regs.pc64 &= ~(u64)1;
>>> +        }
>>> +    }
>>> +#ifdef CONFIG_ARM_64
>>> +    else
>>> +    {
>>> +        ctxt.user_regs.x0 = cid;
>>> +        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;
>>> +    }
>>> +#endif
>>> +    ctxt.sctlr = SCTLR_GUEST_INIT;
>>> +    ctxt.flags = VGCF_online;
>>> +
>>> +    /* Reset architecture specific registers */
>>> +    vcpu_arch_reset(v); > +
>>> +    /* Initialize VCPU registers */
>>> +    _arch_set_info_guest(v, &ctxt);
>>
>> AFAICT, this is expected to be called with the domain lock taken as this
>> can be called by various path.
>>
>> Also, most of the function is the same as CPU_on. So I would like to see
>> the code factored in the separate function and used in both place.
>>
> 
> I agree, but the 2 scenarios (VCPU allocation versus clearing VCPU
> context) made it a bit difficult to share. Please let me know if you
> have some additional hint on how to exactly structure the code.

See above. I would be interested to know why you think they are different.

> 
>>> +}
>>> +
>>> +int32_t domain_suspend(register_t epoint, register_t cid)
>>> +{
>>> +    struct vcpu *v;
>>> +    struct domain *d = current->domain;
>>> +    bool is_thumb = epoint & 1;
>>> +
>>> +    dprintk(XENLOG_DEBUG,
>>> +            "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
>>> +            d->domain_id, epoint, cid);
>>> +
>>> +    /* THUMB set is not allowed with 64-bit domain */
>>> +    if ( is_64bit_domain(d) && is_thumb )
>>> +        return PSCI_INVALID_ADDRESS;
>>> +
>>> +    /* Ensure that all CPUs other than the calling one are offline */
>>> +    for_each_vcpu ( d, v )
>>> +    {
>>> +        if ( v != current && is_vcpu_online(v) )
>>> +            return PSCI_DENIED;
>>> +    }
>>
>> What does prevent a vCPU to not come online while doing the loop?
> 
> As you suggested probably nothing if there is a bug in the guest,
> which we want to check for. Is the domain_lock right thing to use
> here?

The domain_lock is probably the best solution here. You want CPU_ON and 
SYSTEM_OFF to race. So CPU_ON may need some modification as well.

> 
>>
>>> +
>>> +    /*
>>> +     * Prepare the calling VCPU for suspend (reset its context, save entry point
>>> +     * into pc and context ID into r0/x0 as specified by PSCI SYSTEM_SUSPEND)
>>> +     */
>>> +    vcpu_suspend(epoint, cid);
>>> +
>>> +    /*
>>> +     * Set the domain state to suspended (will be cleared when the domain
>>> +     * resumes, i.e. VCPU of this domain gets scheduled in).
>>> +     */
>>> +    d->is_shut_down = 1;
>>> +    d->shutdown_code = SHUTDOWN_suspend;
>>
>> If you look at the other usage, you will notice that they are protected
>> with a lock. Why is it not necessary here?
>>
> 
> I think it is necessary here too
> 
>> I am also not entirely sure why we could not re-use code that already
>> exist in common code. Surely suspend/resume should work in a similar way?
>>
> 
> Could you please be more specific (which common code)

You can ignore this for now as shutdown_code may not be the right 
solution here.

Cheers,


-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Stefano Stabellini]

On Mon, 12 Nov 2018, Julien Grall wrote:
> Hi,
> 
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > When Dom0 finalizes its suspend procedure the suspend of Xen is
> > triggered by calling system_suspend(). Dom0 finalizes the suspend from
> > its boot core (VCPU#0), which could be mapped to any physical CPU,
> > i.e. the system_suspend() function could be executed by any physical
> > CPU. Since Xen suspend procedure has to be run by the boot CPU
> > (non-boot CPUs will be disabled at some point in suspend procedure),
> > system_suspend() execution has to continue on CPU#0.
> 
> Nothing in the domain_suspend code checks that domain_suspend is called by
> vCPU0. I also can't find any restriction in the PSCI spec to run on pCPU0. Can
> you provide more details why this required?

The spec says that "to use this API, a calling OS must power down all
but one core through calls to CPU_OFF". It is natural to think that the
remaining core would be (physical or virtual) cpu0, but actually it is
not clearly stated by the spec. For dom0, we could simply check that
only 1 vcpu is left ON.  For Xen and the physical system_suspend call,
it makes sense to make the call on pcpu0.

 
> > When the system_suspend() returns 0, it means that the system was
> > suspended and it is coming out of the resume procedure. Regardless
> > of the system_suspend() return value, after this function returns
> > Xen is fully functional, and its state, including all devices and data
> > structures, matches the state prior to calling system_suspend().
> > The status is returned by system_suspend() for debugging/logging
> > purposes and function prototype compatibility.
> > 
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> > ---
> >   xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
> >   1 file changed, 34 insertions(+)
> > 
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index f2338e41db..21b45f8248 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t
> > cid)
> >       _arch_set_info_guest(v, &ctxt);
> >   }
> >   +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> > +static long system_suspend(void *data)
> > +{
> > +    BUG_ON(system_state != SYS_STATE_active);
> > +
> > +    return -ENOSYS;
> > +}
> > +
> >   int32_t domain_suspend(register_t epoint, register_t cid)
> >   {
> >       struct vcpu *v;
> >       struct domain *d = current->domain;
> >       bool is_thumb = epoint & 1;
> > +    int status;
> >         dprintk(XENLOG_DEBUG,
> >               "Dom%d suspend: epoint=0x%"PRIregister",
> > cid=0x%"PRIregister"\n",
> > @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t
> > cid)
> >        */
> >       vcpu_block_unless_event_pending(current);
> >   +    /* If this was dom0 the whole system should suspend: trigger Xen
> > suspend */
> > +    if ( is_hardware_domain(d) )
> > +    {
> > +        /*
> > +         * system_suspend should be called when Dom0 finalizes the suspend
> > +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0
> > could
> > +         * be mapped to any PCPU (this function could be executed by any
> > PCPU).
> > +         * The suspend procedure has to be finalized by the PCPU#0
> > (non-boot
> > +         * PCPUs will be disabled during the suspend).
> > +         */
> > +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> > +        /*
> > +         * If an error happened, there is nothing that needs to be done
> > here
> > +         * because the system_suspend always returns in fully functional
> > state
> > +         * no matter what the outcome of suspend procedure is. If the
> > system
> > +         * suspended successfully the function will return 0 after the
> > resume.
> > +         * Otherwise, if an error is returned it means Xen did not
> > suspended,
> > +         * but it is still in the same state as if the system_suspend was
> > never
> > +         * called. We dump a debug message in case of an error for
> > debugging/
> > +         * logging purpose.
> > +         */
> > +        if ( status )
> > +            dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
> > +    }
> > +
> >       return PSCI_SUCCESS;
> >   }
> >   
> 
> -- 
> Julien Grall
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 15/18] xen/arm: Resume memory management on Xen resume

[Stefano Stabellini]

On Mon, 12 Nov 2018, Julien Grall wrote:
> Hi,
> 
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > The MMU needs to be enabled in the resume flow before the context
> > can be restored (we need to be able to access the context data by
> > virtual address in order to restore it). The configuration of system
> > registers prior to branching to the routine that sets up the page
> > tables is copied from xen/arch/arm/arm64/head.S.
> > After the MMU is enabled, the content of TTBR0_EL2 is changed to
> > point to init_ttbr (page tables used at runtime).
> > 
> > At boot the init_ttbr variable is updated when a secondary CPU is
> > hotplugged. In the scenario where there is only one physical CPU in
> > the system, the init_ttbr would not be initialized for the use in
> > resume flow. To get the variable initialized in all scenarios in this
> > patch we add that the boot CPU updates init_ttbr after it sets the
> > page tables for runtime.
> > 
> > After the memory management is resumed, the SCTLR_WXN in SCTLR_EL2
> > has to be set in order to configure that a mapping cannot be both
> > writable and executable (this was configured prior to suspend).
> > This is done using an existing function (mmu_init_secondary_cpu).
> > 
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> > 
> > ---
> > Changes in v2:
> > 
> > - Patch from v1:
> > "[XEN PATCH 17/21] xen/arm: Set SCTLR_WXN in SCTLR_EL2 on resume"
> > is squashed with this patch, because it is indeed related to resuming
> > the memory management
> > - Since the original patch was named 'Enable the MMU', and this is
> > not only enabling anymore, but the full resume of functionality, the
> > commit title and message is fixed
> > ---
> >   xen/arch/arm/arm64/entry.S | 88
> > ++++++++++++++++++++++++++++++++++++++++++++++
> >   xen/arch/arm/mm.c          |  1 +
> >   xen/arch/arm/suspend.c     |  6 ++++
> >   3 files changed, 95 insertions(+)
> > 
> > diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> > index dbc4717903..5efa30e8ee 100644
> > --- a/xen/arch/arm/arm64/entry.S
> > +++ b/xen/arch/arm/arm64/entry.S
> > @@ -1,6 +1,7 @@
> >   #include <asm/asm_defns.h>
> >   #include <asm/current.h>
> >   #include <asm/macros.h>
> > +#include <asm/page.h>
> >   #include <asm/regs.h>
> >   #include <asm/alternative.h>
> >   #include <asm/smccc.h>
> > @@ -534,6 +535,93 @@ ENTRY(__context_switch)
> >           ret
> >     ENTRY(hyp_resume)
> > +        msr   DAIFSet, 0xf           /* Disable all interrupts */
> > +
> > +        tlbi  alle2
> > +        dsb   sy                     /* Ensure completion of TLB flush */
> > +        isb
> > +
> > +        ldr   x0, =start
> > +        adr   x19, start             /* x19 := paddr (start) */
> > +        sub   x20, x19, x0           /* x20 := phys-offset */
> > +
> > +        /* XXXX call PROCINFO_cpu_init here */
> > +
> > +        /* Set up memory attribute type tables */
> > +        ldr   x0, =MAIRVAL
> > +        msr   mair_el2, x0
> > +
> > +        /* Set up TCR_EL2:
> > +         * PS -- Based on ID_AA64MMFR0_EL1.PARange
> > +         * Top byte is used
> > +         * PT walks use Inner-Shareable accesses,
> > +         * PT walks are write-back, write-allocate in both cache levels,
> > +         * 48-bit virtual address space goes through this table. */
> > +        ldr   x0,
> > =(TCR_RES1|TCR_SH0_IS|TCR_ORGN0_WBWA|TCR_IRGN0_WBWA|TCR_T0SZ(64-48))
> > +        /* ID_AA64MMFR0_EL1[3:0] (PARange) corresponds to TCR_EL2[18:16]
> > (PS) */
> > +        mrs   x1, ID_AA64MMFR0_EL1
> > +        bfi   x0, x1, #16, #3
> > +
> > +        msr   tcr_el2, x0
> > +
> > +        /* Set up the SCTLR_EL2:
> > +         * Exceptions in LE ARM,
> > +         * Low-latency IRQs disabled,
> > +         * Write-implies-XN disabled (for now),
> > +         * D-cache disabled (for now),
> > +         * I-cache enabled,
> > +         * Alignment checking disabled,
> > +         * MMU translation disabled (for now). */
> > +        ldr   x0, =(HSCTLR_BASE)
> > +        msr   SCTLR_EL2, x0
> > +
> > +        /* Ensure that any exceptions encountered at EL2
> > +         * are handled using the EL2 stack pointer, rather
> > +         * than SP_EL0. */
> > +        msr spsel, #1
> > +
> > +        /* Rebuild the boot pagetable's first-level entries. The structure
> > +         * is described in mm.c.
> > +         *
> > +         * After the CPU enables paging it will add the fixmap mapping
> > +         * to these page tables, however this may clash with the 1:1
> > +         * mapping. So each CPU must rebuild the page tables here with
> > +         * the 1:1 in place. */
> > +
> > +        /* If Xen is loaded at exactly XEN_VIRT_START then we don't
> > +         * need an additional 1:1 mapping, the virtual mapping will
> > +         * suffice.
> > +         */
> > +        cmp   x19, #XEN_VIRT_START
> > +        cset  x25, eq                /* x25 := identity map in place, or
> > not */
> > +
> > +        /* Write Xen's PT's paddr into TTBR0_EL2 */
> > +        ldr   x4, =boot_pgtable     /* xen_pgtable    */
> > +        add   x4, x4, x20           /* x4 := paddr (boot_pagetable) */
> > +        msr   TTBR0_EL2, x4
> > +
> > +        /* Set up page tables */
> > +        bl    setup_page_tables
> > +
> > +        ldr   x1, =mmu_resumed      /* Explicit vaddr, not RIP-relative */
> > +        mrs   x0, SCTLR_EL2
> > +        orr   x0, x0, #SCTLR_M      /* Enable MMU */
> > +        orr   x0, x0, #SCTLR_C      /* Enable D-cache */
> > +        dsb   sy                    /* Flush PTE writes and finish reads */
> > +        msr   SCTLR_EL2, x0         /* now paging is enabled */
> > +        isb                         /* Now, flush the icache */
> > +        br    x1                    /* Get a proper vaddr into PC */
> > +
> > +mmu_resumed:
> > +        ldr   x4, =init_ttbr         /* VA of TTBR0_EL2 stashed by CPU 0 */
> 
> While I know that Xen is switching between page-tables in some place, this
> should not be done.  Indeed, such sequence may break coherency of the TLBs. In
> order to avoid that, you want to apply the break-before-sequence. I haven't
> fully think how to solve it in Xen properly today.
> 
> I am quite worried to introduce simlar sequence in Xen before fixing the
> current sequences.

We can turn this assembly code into a .macro, so that at least when we
get around to it, we just need to fix it one place. In fact, I noticed
that the code sequence right before "mmu_resumed" and the one right
after are both indentical to the ones head.S, it would be good to avoid
the duplication if possible.


> > +        ldr   x4, [x4]               /* Actual value */
> > +        dsb   sy
> > +        msr   TTBR0_EL2, x4
> > +        dsb   sy
> > +        isb
> > +        tlbi  alle2
> > +        dsb   sy                     /* Ensure completion of TLB flush */
> > +        isb
> >           b .
> >     /*

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 6837 bytes --]

On Mon, 12 Nov 2018, Andrew Cooper wrote:
> On 12/11/18 16:35, Mirela Simonovic wrote:
> > Hi Julien,
> >
> > Thanks for your feedback, I'll need to answer in iterations.
> >
> > On Mon, Nov 12, 2018 at 4:27 PM Julien Grall <julien.grall@arm.com> wrote:
> >> Hi Mirela,
> >>
> >> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>> The implementation consists of:
> >>> -Adding PSCI system suspend call as new PSCI function
> >>> -Trapping PSCI system_suspend HVC
> >>> -Implementing PSCI system suspend call (virtual interface that allows
> >>>   guests to suspend themselves)
> >>>
> >>> The PSCI system suspend should be called by a guest from its boot
> >>> VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
> >>> CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
> >>> are left enabled by the guest are assumed to be its wake-up interrupts.
> >>> Therefore, a wake-up interrupt triggers the resume of the guest. Guest
> >>> should resume regardless of the state of Xen (suspended or not).
> >>>
> >>> When a guest calls PSCI system suspend the respective domain will be
> >>> suspended if the following conditions are met:
> >>> 1) Given resume entry point is not invalid
> >>> 2) Other (if any) VCPUs of the calling guest are hot-unplugged
> >>>
> >>> If the conditions above are met the calling domain is labeled as
> >>> suspended and the calling VCPU is blocked. If nothing else wouldn't
> >>> be done the suspended domain would resume from the place where it
> >>> called PSCI system suspend. This is expected if processing of the PSCI
> >>> system suspend call fails. However, in the case of success the calling
> >>> guest should resume (continue execution after the wake-up) from the entry
> >>> point which is given as the first argument of the PSCI system suspend
> >>> call. In addition to the entry point, the guest expects to start within
> >>> the environment whose state matches the state after reset. This means
> >>> that the guest should find reset register values, MMU disabled, etc.
> >>> Thereby, the context of VCPU should be 'reset' (as if the system is
> >>> comming out of reset), the program counter should contain entry point,
> >>> which is 1st argument, and r0/x0 should contain context ID which is 2nd
> >>> argument of PSCI system suspend call. The context of VCPU is set
> >>> accordingly when the PSCI system suspend is processed, so that nothing
> >>> needs to be done on resume/wake-up path. However, in order to ensure that
> >>> this context doesn't get overwritten by the scheduler when scheduling out
> >>> this VCPU (would normally happen after the calling CPU is blocked), we need
> >>> to check whether to return early from ctxt_switch_from().
> >>>
> >>> There are variables in domain structure to keep track of domain shutdown.
> >>> One of existing shutdown reason is 'suspend' which this patch is using to
> >>> track the suspend state of a domain. Those variables are used to determine
> >>> whether to early return from ctxt_switch_from() or not.
> >>>
> >>> A suspended domain will resume after the Xen receives an interrupt which is
> >>> targeted to the domain, unblocks the domain's VCPU, and schedules it in.
> >>> When the VCPU is scheduled in, the VCPU context is already reset, and
> >>> contains the right resume entry point in program counter that will be
> >>> restored in ctxt_switch_to(). The only thing that needs to be done at this
> >>> point is to clear the variables that marked the domain state as suspended.
> >>>
> >>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> >>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >>>
> >>> ---
> >>> Changes in v2:
> >>>
> >>> -Fix print to compile for arm32 and to align with Xen coding style
> >>> ---
> >>>   xen/arch/arm/Makefile            |   1 +
> >>>   xen/arch/arm/domain.c            |  13 +++
> >>>   xen/arch/arm/suspend.c           | 166 +++++++++++++++++++++++++++++++++++++++
> >>>   xen/arch/arm/vpsci.c             |  19 +++++
> >>>   xen/include/asm-arm/perfc_defn.h |   1 +
> >>>   xen/include/asm-arm/psci.h       |   2 +
> >>>   xen/include/asm-arm/suspend.h    |  16 ++++
> >>>   xen/include/xen/sched.h          |   1 +
> >>>   8 files changed, 219 insertions(+)
> >>>   create mode 100644 xen/arch/arm/suspend.c
> >>>   create mode 100644 xen/include/asm-arm/suspend.h
> >>>
> >>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> >>> index 23c5d9adbc..744b1a4dc8 100644
> >>> --- a/xen/arch/arm/Makefile
> >>> +++ b/xen/arch/arm/Makefile
> >>> @@ -43,6 +43,7 @@ obj-y += setup.o
> >>>   obj-y += shutdown.o
> >>>   obj-y += smp.o
> >>>   obj-y += smpboot.o
> >>> +obj-y += suspend.o
> >>>   obj-y += sysctl.o
> >>>   obj-y += time.o
> >>>   obj-y += traps.o
> >>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>> index e594b48d81..7f8105465c 100644
> >>> --- a/xen/arch/arm/domain.c
> >>> +++ b/xen/arch/arm/domain.c
> >>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>       if ( is_idle_vcpu(p) )
> >>>           return;
> >>>
> >>> +    /* VCPU's context should not be saved if its domain is suspended */
> >>> +    if ( p->domain->is_shut_down &&
> >>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> >>> +        return;
> >> SHUTDOWN_suspend is used in Xen for other purpose (see
> >> SCHEDOP_shutdown). The other user of that code relies on all the state
> >> to be saved on suspend.
> >>
> > We just need a flag to mark a domain as suspended, and I do believe
> > SHUTDOWN_suspend is not used anywhere else.
> > Let's come back on this.
> 
> SHUTDOWN_suspend is used for migration.

That is true, but it is not only used for migration. It is also used
for suspending a guest and saving its state to file with the intention
of resuming it later from file.


> Grep for it through the Xen
> tree and you'll find several pieces of documentation, including the
> description of what this shutdown code means.
> 
> What you are introducing here is not a shutdown - it is a suspend with
> the intent to resume executing later.  As such, it shouldn't use Xen's
> shutdown infrastructure, which exists mainly to communicate with the
> toolstack.

Future work will need toolstack support for suspending/resuming guests.
SHUTDOWN_suspend is the most natural fit today; we don't want to hijack
domain pause, because if we do, then we can't normally pause a domain
anymore. From the Xen side of thing, there isn't a huge difference
between saving the state of a domain and writing it to file, or saving
the state of a domain in memory. However, I agree that there is a
difference.

If we don't want to reuse SHUTDOWN_suspend, then the only other option I
can think of is to introduce a new ARM specific suspend code for this
(and new xl commands and hypercalls in the future).

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Stefano Stabellini]

On Mon, 12 Nov 2018, Julien Grall wrote:
> Hi Mirela,
> 
> Thank you for posting the series. Could you provide a branch with the patch
> applied?
> 
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > 
> > --------------------------------------------------------------------------------
> > The series does not include:
> > a) UART driver-specific suspend/resume that gets called when console
> > suspends
> 
> I feel it will be difficult to test this series without UART driver support.
> Would it be possible to integrate them in that series?

Actually, you can test this series simply using upstream Linux and
upstream Xen + this series. You can use echo mem > /sys/power/state in
dom0 to go into suspend, and for example setup an RTC wakeup event to
wakeup after a certain amount of time. For instance echo +30 >
/sys/class/rtc/rtc0/wakealarm.

For the Xilinx MPSoC, Linux 4.19 works fine with defconfig. Linux 4.20
(current master) gained EEMI support, but I tested it and it seems to
suspend successfully even if Xen doesn't know how to handle the EEMI
calls that Linux makes when suspending. 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Andrew Cooper]

On 12/11/2018 19:56, Julien Grall wrote:
> Hi Andrew,
>
> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>> index e594b48d81..7f8105465c 100644
>>>>> --- a/xen/arch/arm/domain.c
>>>>> +++ b/xen/arch/arm/domain.c
>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>        if ( is_idle_vcpu(p) )
>>>>>            return;
>>>>>
>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>> suspended */
>>>>> +    if ( p->domain->is_shut_down &&
>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>> +        return;
>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>> SCHEDOP_shutdown). The other user of that code relies on all the state
>>>> to be saved on suspend.
>>>>
>>> We just need a flag to mark a domain as suspended, and I do believe
>>> SHUTDOWN_suspend is not used anywhere else.
>>> Let's come back on this.
>>
>> SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
>> tree and you'll find several pieces of documentation, including the
>> description of what this shutdown code means.
>>
>> What you are introducing here is not a shutdown - it is a suspend with
>> the intent to resume executing later.  As such, it shouldn't use Xen's
>> shutdown infrastructure, which exists mainly to communicate with the
>> toolstack.
>
> Would domain pause/unpause be a better solution?

Actually yes - that sounds like a very neat solution.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Stefano,

On 13/11/2018 01:53, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Andrew Cooper wrote:
>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>> Hi Julien,
>>>
>>> Thanks for your feedback, I'll need to answer in iterations.
>>>
>>> On Mon, Nov 12, 2018 at 4:27 PM Julien Grall <julien.grall@arm.com> wrote:
>>>> Hi Mirela,
>>>>
>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>> The implementation consists of:
>>>>> -Adding PSCI system suspend call as new PSCI function
>>>>> -Trapping PSCI system_suspend HVC
>>>>> -Implementing PSCI system suspend call (virtual interface that allows
>>>>>    guests to suspend themselves)
>>>>>
>>>>> The PSCI system suspend should be called by a guest from its boot
>>>>> VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
>>>>> CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
>>>>> are left enabled by the guest are assumed to be its wake-up interrupts.
>>>>> Therefore, a wake-up interrupt triggers the resume of the guest. Guest
>>>>> should resume regardless of the state of Xen (suspended or not).
>>>>>
>>>>> When a guest calls PSCI system suspend the respective domain will be
>>>>> suspended if the following conditions are met:
>>>>> 1) Given resume entry point is not invalid
>>>>> 2) Other (if any) VCPUs of the calling guest are hot-unplugged
>>>>>
>>>>> If the conditions above are met the calling domain is labeled as
>>>>> suspended and the calling VCPU is blocked. If nothing else wouldn't
>>>>> be done the suspended domain would resume from the place where it
>>>>> called PSCI system suspend. This is expected if processing of the PSCI
>>>>> system suspend call fails. However, in the case of success the calling
>>>>> guest should resume (continue execution after the wake-up) from the entry
>>>>> point which is given as the first argument of the PSCI system suspend
>>>>> call. In addition to the entry point, the guest expects to start within
>>>>> the environment whose state matches the state after reset. This means
>>>>> that the guest should find reset register values, MMU disabled, etc.
>>>>> Thereby, the context of VCPU should be 'reset' (as if the system is
>>>>> comming out of reset), the program counter should contain entry point,
>>>>> which is 1st argument, and r0/x0 should contain context ID which is 2nd
>>>>> argument of PSCI system suspend call. The context of VCPU is set
>>>>> accordingly when the PSCI system suspend is processed, so that nothing
>>>>> needs to be done on resume/wake-up path. However, in order to ensure that
>>>>> this context doesn't get overwritten by the scheduler when scheduling out
>>>>> this VCPU (would normally happen after the calling CPU is blocked), we need
>>>>> to check whether to return early from ctxt_switch_from().
>>>>>
>>>>> There are variables in domain structure to keep track of domain shutdown.
>>>>> One of existing shutdown reason is 'suspend' which this patch is using to
>>>>> track the suspend state of a domain. Those variables are used to determine
>>>>> whether to early return from ctxt_switch_from() or not.
>>>>>
>>>>> A suspended domain will resume after the Xen receives an interrupt which is
>>>>> targeted to the domain, unblocks the domain's VCPU, and schedules it in.
>>>>> When the VCPU is scheduled in, the VCPU context is already reset, and
>>>>> contains the right resume entry point in program counter that will be
>>>>> restored in ctxt_switch_to(). The only thing that needs to be done at this
>>>>> point is to clear the variables that marked the domain state as suspended.
>>>>>
>>>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>>>>
>>>>> ---
>>>>> Changes in v2:
>>>>>
>>>>> -Fix print to compile for arm32 and to align with Xen coding style
>>>>> ---
>>>>>    xen/arch/arm/Makefile            |   1 +
>>>>>    xen/arch/arm/domain.c            |  13 +++
>>>>>    xen/arch/arm/suspend.c           | 166 +++++++++++++++++++++++++++++++++++++++
>>>>>    xen/arch/arm/vpsci.c             |  19 +++++
>>>>>    xen/include/asm-arm/perfc_defn.h |   1 +
>>>>>    xen/include/asm-arm/psci.h       |   2 +
>>>>>    xen/include/asm-arm/suspend.h    |  16 ++++
>>>>>    xen/include/xen/sched.h          |   1 +
>>>>>    8 files changed, 219 insertions(+)
>>>>>    create mode 100644 xen/arch/arm/suspend.c
>>>>>    create mode 100644 xen/include/asm-arm/suspend.h
>>>>>
>>>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>>>> index 23c5d9adbc..744b1a4dc8 100644
>>>>> --- a/xen/arch/arm/Makefile
>>>>> +++ b/xen/arch/arm/Makefile
>>>>> @@ -43,6 +43,7 @@ obj-y += setup.o
>>>>>    obj-y += shutdown.o
>>>>>    obj-y += smp.o
>>>>>    obj-y += smpboot.o
>>>>> +obj-y += suspend.o
>>>>>    obj-y += sysctl.o
>>>>>    obj-y += time.o
>>>>>    obj-y += traps.o
>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>> index e594b48d81..7f8105465c 100644
>>>>> --- a/xen/arch/arm/domain.c
>>>>> +++ b/xen/arch/arm/domain.c
>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>        if ( is_idle_vcpu(p) )
>>>>>            return;
>>>>>
>>>>> +    /* VCPU's context should not be saved if its domain is suspended */
>>>>> +    if ( p->domain->is_shut_down &&
>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>> +        return;
>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>> SCHEDOP_shutdown). The other user of that code relies on all the state
>>>> to be saved on suspend.
>>>>
>>> We just need a flag to mark a domain as suspended, and I do believe
>>> SHUTDOWN_suspend is not used anywhere else.
>>> Let's come back on this.
>>
>> SHUTDOWN_suspend is used for migration.
> 
> That is true, but it is not only used for migration. It is also used
> for suspending a guest and saving its state to file with the intention
> of resuming it later from file.

Which is some sort of migration at the end. However, they don't have the same 
semantics as suspend/resume regarding the state of the vCPU.

> 
> 
>> Grep for it through the Xen
>> tree and you'll find several pieces of documentation, including the
>> description of what this shutdown code means.
>>
>> What you are introducing here is not a shutdown - it is a suspend with
>> the intent to resume executing later.  As such, it shouldn't use Xen's
>> shutdown infrastructure, which exists mainly to communicate with the
>> toolstack.
> 
> Future work will need toolstack support for suspending/resuming guests.
> SHUTDOWN_suspend is the most natural fit today; we don't want to hijack
> domain pause, because if we do, then we can't normally pause a domain
> anymore. 

Why? suspend/resume is like pausing the domain but will be resumed on event (e.g 
interrupts) rather than user request.

> From the Xen side of thing, there isn't a huge difference
> between saving the state of a domain and writing it to file, or saving
> the state of a domain in memory. However, I agree that there is a
> difference.
> 
> If we don't want to reuse SHUTDOWN_suspend, then the only other option I
> can think of is to introduce a new ARM specific suspend code for this
> (and new xl commands and hypercalls in the future).

Why would you need a specific xl commands and hypercalls for it?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

Hi Stefano,

On 12/11/2018 23:46, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Julien Grall wrote:
>> Hi,
>>
>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>> When Dom0 finalizes its suspend procedure the suspend of Xen is
>>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
>>> its boot core (VCPU#0), which could be mapped to any physical CPU,
>>> i.e. the system_suspend() function could be executed by any physical
>>> CPU. Since Xen suspend procedure has to be run by the boot CPU
>>> (non-boot CPUs will be disabled at some point in suspend procedure),
>>> system_suspend() execution has to continue on CPU#0.
>>
>> Nothing in the domain_suspend code checks that domain_suspend is called by
>> vCPU0. I also can't find any restriction in the PSCI spec to run on pCPU0. Can
>> you provide more details why this required?
> 
> The spec says that "to use this API, a calling OS must power down all
> but one core through calls to CPU_OFF". It is natural to think that the
> remaining core would be (physical or virtual) cpu0, but actually it is
> not clearly stated by the spec. For dom0, we could simply check that
> only 1 vcpu is left ON.

It is what we already do in the code. The comment and commit message does not 
match the code.

> For Xen and the physical system_suspend call,
> it makes sense to make the call on pcpu0.

This needs some rationale. What's wrong with issue the suspend from the current 
pCPU?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 15/18] xen/arm: Resume memory management on Xen resume

[Julien Grall]

Hi,

On 13/11/2018 01:36, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Julien Grall wrote:
>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> We can turn this assembly code into a .macro, so that at least when we
> get around to it, we just need to fix it one place. In fact, I noticed
> that the code sequence right before "mmu_resumed" and the one right
> after are both indentical to the ones head.S, it would be good to avoid
> the duplication if possible.

This does not address my concern. We want to have suspend/resume compliant to 
the Arm architecture (at least to our best knowledge). Hiding it in a 
macro/function is not going to make it compliant.

Nobody wants its platform randomly crashing on suspend/resume because the Arm 
Arm was knowingly not followed. This is more critical than the boot one as we 
are going to call that more often. So what is the action plan to correctly 
resume memory management in this series?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Julien Grall]

Hi Stefano,

On 13/11/2018 02:22, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Julien Grall wrote:
>> Hi Mirela,
>>
>> Thank you for posting the series. Could you provide a branch with the patch
>> applied?
>>
>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>
>>> --------------------------------------------------------------------------------
>>> The series does not include:
>>> a) UART driver-specific suspend/resume that gets called when console
>>> suspends
>>
>> I feel it will be difficult to test this series without UART driver support.
>> Would it be possible to integrate them in that series?
> 
> Actually, you can test this series simply using upstream Linux and
> upstream Xen + this series. You can use echo mem > /sys/power/state in
> dom0 to go into suspend, and for example setup an RTC wakeup event to
> wakeup after a certain amount of time. For instance echo +30 >
> /sys/class/rtc/rtc0/wakealarm.

I am quite surprised this series is enough given there are a BUG() in the 
suspend path for the console.

So what is your xen command line?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi,

On 12/11/2018 11:30, Mirela Simonovic wrote:
> +/*
> + * This function sets the context of current VCPU to the state which is expected
> + * by the guest on resume. The expected VCPU state is:
> + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
> + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
> + * 3) All other general purpose and system registers should have reset values
> + *
> + * Note: this function has to return void because it has to always succeed. In
> + * other words, this function is called from virtual PSCI SYSTEM_SUSPEND
> + * implementation, which can return only a limited number of possible errors,
> + * none of which could represent the fact that an error occurred when preparing
> + * the domain for suspend.
> + * Consequently, dynamic memory allocation cannot be done within this function,
> + * because if malloc fails the error has nowhere to propagate.
> + */
> +static void vcpu_suspend(register_t epoint, register_t cid)
> +{
> +    /* Static allocation because dynamic would need a non-void return */
> +    static struct vcpu_guest_context ctxt;
> +    struct vcpu *v = current;
> +
> +    /* Make sure that VCPU guest regs are zeroied */
> +    memset(&ctxt, 0, sizeof(ctxt));
> +
> +    /* Set non-zero values to the registers prior to copying */
> +    ctxt.user_regs.pc64 = (u64)epoint;
> +
> +    if ( is_32bit_domain(current->domain) )
> +    {
> +        ctxt.user_regs.r0_usr = cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;

This is going to disable the MMU and Cache as requested by the PSCI spec. As the 
guest is not required to clean the cache when turning off the CPU/suspending, 
the data may not have reached the main memory.

So do you need to perform cache maintenance to avoid stale information?

> +
> +        /* Thumb set is allowed only for 32-bit domain */
> +        if ( epoint & 1 )
> +        {
> +            ctxt.user_regs.cpsr |= PSR_THUMB;
> +            ctxt.user_regs.pc64 &= ~(u64)1;
> +        }
> +    }
> +#ifdef CONFIG_ARM_64
> +    else
> +    {
> +        ctxt.user_regs.x0 = cid;
> +        ctxt.user_regs.cpsr = PSR_GUEST64_INIT;

Same here.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi,

On Tue, Nov 13, 2018 at 10:43 AM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Stefano,
>
> On 12/11/2018 23:46, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Julien Grall wrote:
> >> Hi,
> >>
> >> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>> When Dom0 finalizes its suspend procedure the suspend of Xen is
> >>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> >>> its boot core (VCPU#0), which could be mapped to any physical CPU,
> >>> i.e. the system_suspend() function could be executed by any physical
> >>> CPU. Since Xen suspend procedure has to be run by the boot CPU
> >>> (non-boot CPUs will be disabled at some point in suspend procedure),
> >>> system_suspend() execution has to continue on CPU#0.
> >>
> >> Nothing in the domain_suspend code checks that domain_suspend is called by
> >> vCPU0. I also can't find any restriction in the PSCI spec to run on pCPU0. Can
> >> you provide more details why this required?
> >
> > The spec says that "to use this API, a calling OS must power down all
> > but one core through calls to CPU_OFF". It is natural to think that the
> > remaining core would be (physical or virtual) cpu0, but actually it is
> > not clearly stated by the spec. For dom0, we could simply check that
> > only 1 vcpu is left ON.
>
> It is what we already do in the code. The comment and commit message does not
> match the code.
>
> > For Xen and the physical system_suspend call,
> > it makes sense to make the call on pcpu0.
>
> This needs some rationale. What's wrong with issue the suspend from the current
> pCPU?
>

The guest doesn't have to finalize its suspend procedure from VCPU0.
That is not required by the spec, although it is commonly the case in
implementations. We don't make any assumptions on CPU ID until Dom0
suspends. In the context of Dom0 and comments in this patch, linux
will finalize the suspend procedure from its boot core/CPU0, so the
comment is correct.

When it comes to suspending Xen, we use generic, existing functions to
disable secondary cpus (disable_nonboot_cpus) later in the suspend
procedure. This function makes the assumption that non-boot CPUs are
the ones whose index is not zero, and the pCPU0 will not be disabled
in this process. So we have to switch the execution to physical CPU0
before we move on with suspend. The same is done for x86 suspend.

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

On 13/11/2018 11:26, Mirela Simonovic wrote:
> Hi,
> 
> On Tue, Nov 13, 2018 at 10:43 AM Julien Grall <julien.grall@arm.com> wrote:
>>
>> Hi Stefano,
>>
>> On 12/11/2018 23:46, Stefano Stabellini wrote:
>>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>> When Dom0 finalizes its suspend procedure the suspend of Xen is
>>>>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
>>>>> its boot core (VCPU#0), which could be mapped to any physical CPU,
>>>>> i.e. the system_suspend() function could be executed by any physical
>>>>> CPU. Since Xen suspend procedure has to be run by the boot CPU
>>>>> (non-boot CPUs will be disabled at some point in suspend procedure),
>>>>> system_suspend() execution has to continue on CPU#0.
>>>>
>>>> Nothing in the domain_suspend code checks that domain_suspend is called by
>>>> vCPU0. I also can't find any restriction in the PSCI spec to run on pCPU0. Can
>>>> you provide more details why this required?
>>>
>>> The spec says that "to use this API, a calling OS must power down all
>>> but one core through calls to CPU_OFF". It is natural to think that the
>>> remaining core would be (physical or virtual) cpu0, but actually it is
>>> not clearly stated by the spec. For dom0, we could simply check that
>>> only 1 vcpu is left ON.
>>
>> It is what we already do in the code. The comment and commit message does not
>> match the code.
>>
>>> For Xen and the physical system_suspend call,
>>> it makes sense to make the call on pcpu0.
>>
>> This needs some rationale. What's wrong with issue the suspend from the current
>> pCPU?
>>
> 
> The guest doesn't have to finalize its suspend procedure from VCPU0.
> That is not required by the spec, although it is commonly the case in
> implementations. We don't make any assumptions on CPU ID until Dom0
> suspends. In the context of Dom0 and comments in this patch, linux
> will finalize the suspend procedure from its boot core/CPU0, so the
> comment is correct.

The hypervisor is not tie to a specific Operating System. So comment should 
*not* assume that Dom0 is always Linux.

The comment should be as generic as possible and match the code. If you say 
"Dom0 will suspend from vCPU#0", then I expect a check "if vcpu_id != 0 then 
return PSCI_DENIED".

Example are always fine, but this should be treated as such. In that context, we 
don't really care that Dom0 called from CPU0 or CPU1. What matters is you have 
only one vCPU left.

> 
> When it comes to suspending Xen, we use generic, existing functions to
> disable secondary cpus (disable_nonboot_cpus) later in the suspend
> procedure. This function makes the assumption that non-boot CPUs are
> the ones whose index is not zero, and the pCPU0 will not be disabled
> in this process. So we have to switch the execution to physical CPU0
> before we move on with suspend. The same is done for x86 suspend.

Please clarify it in the commit message.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

On 13/11/2018 10:23, Julien Grall wrote:
> Hi,
> 
> On 12/11/2018 11:30, Mirela Simonovic wrote:
>> +/*
>> + * This function sets the context of current VCPU to the state which is expected
>> + * by the guest on resume. The expected VCPU state is:
>> + * 1) pc to contain resume entry point (1st argument of PSCI SYSTEM_SUSPEND)
>> + * 2) r0/x0 to contain context ID (2nd argument of PSCI SYSTEM_SUSPEND)
>> + * 3) All other general purpose and system registers should have reset values
>> + *
>> + * Note: this function has to return void because it has to always succeed. In
>> + * other words, this function is called from virtual PSCI SYSTEM_SUSPEND
>> + * implementation, which can return only a limited number of possible errors,
>> + * none of which could represent the fact that an error occurred when preparing
>> + * the domain for suspend.
>> + * Consequently, dynamic memory allocation cannot be done within this function,
>> + * because if malloc fails the error has nowhere to propagate.
>> + */
>> +static void vcpu_suspend(register_t epoint, register_t cid)
>> +{
>> +    /* Static allocation because dynamic would need a non-void return */
>> +    static struct vcpu_guest_context ctxt;
>> +    struct vcpu *v = current;
>> +
>> +    /* Make sure that VCPU guest regs are zeroied */
>> +    memset(&ctxt, 0, sizeof(ctxt));
>> +
>> +    /* Set non-zero values to the registers prior to copying */
>> +    ctxt.user_regs.pc64 = (u64)epoint;
>> +
>> +    if ( is_32bit_domain(current->domain) )
>> +    {
>> +        ctxt.user_regs.r0_usr = cid;
>> +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
> 
> This is going to disable the MMU and Cache as requested by the PSCI spec. As the 
> guest is not required to clean the cache when turning off the CPU/suspending, 
> the data may not have reached the main memory.
> 
> So do you need to perform cache maintenance to avoid stale information?

Answering to myself, I have discussed about the cache with others Arm folks 
today. SYSTEM_SUSPEND may exit early due to a pending event (BTW you don't seem 
to handle it) and could jump to the specified entry point address.

In that case, the only guarantee is the data would not be lost, yet can still be 
in the cache. This means that any data used before the MMU & cache are enabled 
in the entry point should have been cleaned to PoC and if you modify data make 
sure the cache is invalidated.

Linux is doing that properly. Looking at Xen, we don't really properly handle 
the cache in the boot path. So you may randomly crash.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Stefano Stabellini]

On Tue, 13 Nov 2018, Julien Grall wrote:
> Hi Stefano,
> 
> On 13/11/2018 02:22, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Julien Grall wrote:
> > > Hi Mirela,
> > > 
> > > Thank you for posting the series. Could you provide a branch with the
> > > patch
> > > applied?
> > > 
> > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > 
> > > > 
> > > > --------------------------------------------------------------------------------
> > > > The series does not include:
> > > > a) UART driver-specific suspend/resume that gets called when console
> > > > suspends
> > > 
> > > I feel it will be difficult to test this series without UART driver
> > > support.
> > > Would it be possible to integrate them in that series?
> > 
> > Actually, you can test this series simply using upstream Linux and
> > upstream Xen + this series. You can use echo mem > /sys/power/state in
> > dom0 to go into suspend, and for example setup an RTC wakeup event to
> > wakeup after a certain amount of time. For instance echo +30 >
> > /sys/class/rtc/rtc0/wakealarm.
> 
> I am quite surprised this series is enough given there are a BUG() in the
> suspend path for the console.
> 
> So what is your xen command line?

No, I made a mistake in rebasing Mirela's branch: I tested a branch
which included the two unsent patches that removed exactly the BUG() you
are talking about. Without those two patches, Xen is unable to resume
correctly on the MPSoC due to the UART.

Mirela, it is better to include the two patches in your next version of
the series. If you think they are not up to quality for being merged
upstream, i.e. they are just hacks for testing, that's OK, tag them with
[PATCH DO-NOT-APPLY] to make it clear. It is also useful to post a git
branch in your 0 email when the patch series is large.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Stefano Stabellini]

On Mon, 12 Nov 2018, Julien Grall wrote:
> > > However, what is the issue with saving all the registers here?
> > > 
> > 
> > We need to save arguments that are provided by a guest with system
> > suspend PSCI call. These arguments are the entry point that needs to
> > be saved in program counter and context ID that needs to be saved in
> > x0/r0. We don't have these arguments here. Context switch happens
> > after processing the system suspend PSCI call, so it's too late.
> 
> It does not feel right to modify ctxt_switch{from,to} for suspend/resume. If
> you want to reset the vCPU state before blocking the vCPU, then you should
> instead

You missed the end of the suggestion here


> Another way would be to reset the vCPU once you receive the interrupt.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3953 bytes --]

On Tue, 13 Nov 2018, Julien Grall wrote:
> > > > > > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > > > > > index e594b48d81..7f8105465c 100644
> > > > > > --- a/xen/arch/arm/domain.c
> > > > > > +++ b/xen/arch/arm/domain.c
> > > > > > @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> > > > > >        if ( is_idle_vcpu(p) )
> > > > > >            return;
> > > > > > 
> > > > > > +    /* VCPU's context should not be saved if its domain is
> > > > > > suspended */
> > > > > > +    if ( p->domain->is_shut_down &&
> > > > > > +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> > > > > > +        return;
> > > > > SHUTDOWN_suspend is used in Xen for other purpose (see
> > > > > SCHEDOP_shutdown). The other user of that code relies on all the state
> > > > > to be saved on suspend.
> > > > > 
> > > > We just need a flag to mark a domain as suspended, and I do believe
> > > > SHUTDOWN_suspend is not used anywhere else.
> > > > Let's come back on this.
> > > 
> > > SHUTDOWN_suspend is used for migration.
> > 
> > That is true, but it is not only used for migration. It is also used
> > for suspending a guest and saving its state to file with the intention
> > of resuming it later from file.
> 
> Which is some sort of migration at the end. However, they don't have the same
> semantics as suspend/resume regarding the state of the vCPU.
 
Right

 
> > > Grep for it through the Xen
> > > tree and you'll find several pieces of documentation, including the
> > > description of what this shutdown code means.
> > > 
> > > What you are introducing here is not a shutdown - it is a suspend with
> > > the intent to resume executing later.  As such, it shouldn't use Xen's
> > > shutdown infrastructure, which exists mainly to communicate with the
> > > toolstack.
> > 
> > Future work will need toolstack support for suspending/resuming guests.
> > SHUTDOWN_suspend is the most natural fit today; we don't want to hijack
> > domain pause, because if we do, then we can't normally pause a domain
> > anymore. 
> 
> Why? suspend/resume is like pausing the domain but will be resumed on event
> (e.g interrupts) rather than user request.

I meant to say two things.

1) Which domain state should we use for suspended guests?

This patch is using SHUTDOWN_suspend. Taken on its own, a regular "pause
state" sounds like an option, in fact it is necessary to set the domain
as paused otherwise the scheduler will schedule it. But in addition we
need to distinguish a normal paused state from a PSCI system suspend
state. We need to know that the domain has been system-suspended with
PSCI, not just paused.


2) This ties into the discussion of what xl commands we want to use to
request a domU to suspend. We don't need to talk about it now, but at
some point we'll want something equivalent to "xl save" or "xl trigger
sleep" and "xl restore" or "xl trigger s3resume" for this suspended
state.

If we mark the domU simply as "paused" it will be difficult to implement
correctly "xl restore" / "xl trigger s3resume". We should be able to
distinguish a domain which is simply not running or paused (as in "xl
pause") from one that has been put to sleep.  Otherwise we won't be able
to use "xl pause" in its original sense anymore. "xl pause" will become
effectively "xl trigger sleep" on ARM. Which is something to consider
but I don't think that is what we want.

The most similar feature is ACPI S3 in x86-land but the current calls
are so ACPI/x86 specific that don't make much sense on a ACPI-less ARM
implementation. If I am not mistaken, on x86 there is a special struct
domain flag for this: d->arch.hvm.is_s3_suspended. 


Let's leave aside the "which commands should we use" discussion for now
because it doesn't related to this patch series. It seems to me that the
best option is to introduce a new ARM specific struct domain flag,
something akin to d->arch.hvm.is_s3_suspended but ARM PSCI specific.

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Stefano Stabellini]

On Mon, 12 Nov 2018, Julien Grall wrote:
> (+ Andre)
> 
> On 11/12/18 5:42 PM, Mirela Simonovic wrote:
> > Hi Julien,
> > 
> > On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com> wrote:
> > > 
> > > 
> > > 
> > > On 11/12/18 4:52 PM, Mirela Simonovic wrote:
> > > > Hi Julien,
> > > 
> > > Hi,
> > > 
> > > > Thanks for the feedback.
> > > > 
> > > > On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com>
> > > > wrote:
> > > > > 
> > > > > Hi Mirela,
> > > > > 
> > > > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > > > GIC and virtual timer context must be saved when the domain
> > > > > > suspends.
> > > > > 
> > > > > Please provide the rationale for this. Is it required by the spec?
> > > > > 
> > > > 
> > > > This is required for GIC because a guest leaves enabled interrupts in
> > > > the GIC that could wake it up, and on resume it should be able to
> > > > detect which interrupt woke it up. Without saving/restoring the state
> > > > of GIC this would not be possible.
> > > 
> > > I am confused. In patch #10, you save the GIC host because the GIC can
> > > be powered-down. Linux is also saving the GIC state. So how the
> > > interrupt can come up if the GIC is powered down?
> > > 
> > 
> > After Xen (or Linux in the config without Xen) hands over the control
> > to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
> > enabled interrupts that are its wake-up sources. Saving a GIC state
> > only means that its current configuration will be remembered somewhere
> > in data structures, but the configuration is not changed on suspend.
> > Everything that happens with interrupt configuration from this point
> > on is platform specific. Now there are 2 options: 1) EL3 will never
> > allow CPU0 to be powered down and the wake-up interrupt will indeed
> > propagate via GIC;
> > or 2) CPU0 will be powered down and the GIC may be
> > powered down as well, so an external help is needed to deal with
> > wake-up and resume (e.g. in order to react to a wake-up interrupt
> > while the GIC is down, and power up CPU0). This external help is
> > provided by some low-power processor outside of the Cortex-A cluster.
> > 
> > So the platform firmware is responsible for properly configuring a
> > wake-up path if GIC goes down. This is commonly handled by EL3
> > communicating with low-power processor. When the GIC power comes up,
> > the interrupt triggered by a peripheral is still active and the
> > software on Cortex-A cluster should be able to observe it once the GIC
> > state is restored, i.e. interrupts get enabled at GIC.
> 
> Thank you for the explanation.  Now the question is why can't we reset at
> least the GIC CPU interface?
> 
> AFAIU, the guest should restore them in any case. The only things we need to
> know is the interrupt was received for a given guest. We can then queue it and
> wake-up the domain.
> 
> This seems to fit with the description on top of gic_dist_save() in Linux
> GICv2 driver.

Can we rely on all PSCI compliant OSes to restore their own GIC again at
resume? The PSCI spec is not very clear in that regard (at the the
version I am looking at...) I am just asking so that we don't come up
with a solution that only works with Linux.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 15/18] xen/arm: Resume memory management on Xen resume

[Stefano Stabellini]

On Tue, 13 Nov 2018, Julien Grall wrote:
> Hi,
> 
> On 13/11/2018 01:36, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Julien Grall wrote:
> > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > We can turn this assembly code into a .macro, so that at least when we
> > get around to it, we just need to fix it one place. In fact, I noticed
> > that the code sequence right before "mmu_resumed" and the one right
> > after are both indentical to the ones head.S, it would be good to avoid
> > the duplication if possible.
> 
> This does not address my concern. We want to have suspend/resume compliant to
> the Arm architecture (at least to our best knowledge). Hiding it in a
> macro/function is not going to make it compliant.
> 
> Nobody wants its platform randomly crashing on suspend/resume because the Arm
> Arm was knowingly not followed. This is more critical than the boot one as we
> are going to call that more often. So what is the action plan to correctly
> resume memory management in this series?

My suggestion does not address your concern, but at least it limits the
spreading of errors code-wise.

It is only natural for a contributor to start from existing code to
write new code. It just happens that the existing code is broken, thus,
the new code is broken too. We can't really expect any different :)

If we want the contributor to fix this, given that the change is a
difficult modification to delicate assembly code, we need to provide
detailed documentation and/or sample code. Otherwise I can't see how it
can go well.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 15/18] xen/arm: Resume memory management on Xen resume

[Julien Grall]

Hi,

On 11/13/18 9:35 PM, Stefano Stabellini wrote:
> On Tue, 13 Nov 2018, Julien Grall wrote:
>> On 13/11/2018 01:36, Stefano Stabellini wrote:
>>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>> We can turn this assembly code into a .macro, so that at least when we
>>> get around to it, we just need to fix it one place. In fact, I noticed
>>> that the code sequence right before "mmu_resumed" and the one right
>>> after are both indentical to the ones head.S, it would be good to avoid
>>> the duplication if possible.
>>
>> This does not address my concern. We want to have suspend/resume compliant to
>> the Arm architecture (at least to our best knowledge). Hiding it in a
>> macro/function is not going to make it compliant.
>>
>> Nobody wants its platform randomly crashing on suspend/resume because the Arm
>> Arm was knowingly not followed. This is more critical than the boot one as we
>> are going to call that more often. So what is the action plan to correctly
>> resume memory management in this series?
> 
> My suggestion does not address your concern, but at least it limits the
> spreading of errors code-wise.

You can't build a great building on a weak foundation.

> It is only natural for a contributor to start from existing code to
> write new code. It just happens that the existing code is broken, thus, > the new code is broken too. We can't really expect any different :)

I didn't expect anything different. In fact, I explained why the code is 
wrong and asked what is the plan to fix it. I would have expect you or 
the contributor to come back with suggestion how to fix this.

Instead you suggested to keep wrong code in Xen with no plan to resolve it.

> 
> If we want the contributor to fix this, given that the change is a
> difficult modification to delicate assembly code, we need to provide
> detailed documentation and/or sample code. Otherwise I can't see how it
> can go well.

I don't think it is that difficult at least compare to implementing 
suspend/resume. There are far more corner case in the latter.

The rules are fairly simple here:
	1) You cannot switch between TTBR without disabling the MMU and 
therefore use an ID map
	2) We always need to keep an ID map in place to allow turning off the MMU
	3) Replacing a valid mapping should follow the break-before make
	4) Memory written with MMU/cache off should be precede by clean and 
followed by invalidate

The best example might be KVM boot code yet the page-tables are very 
simplistic compare to Xen. Linux is not a good example because EL1 
supports 2 set of page-tables.

I am happy to provide more details if someone is going to look at it.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Stefano Stabellini]

On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> Non-boot CPUs have to be disabled on suspend and enabled on resume
> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
> platform capabilities, this may lead to the physical powering down of
> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
> each non-boot CPU).
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>  xen/arch/arm/suspend.c | 15 ++++++++++++++-
>  1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index 575afd5eb8..dae1b1f7d6 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -1,4 +1,5 @@
>  #include <xen/sched.h>
> +#include <xen/cpu.h>
>  #include <asm/cpufeature.h>
>  #include <asm/event.h>
>  #include <asm/psci.h>
> @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>  /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>  static long system_suspend(void *data)
>  {
> +    int status;
> +
>      BUG_ON(system_state != SYS_STATE_active);
>  
>      system_state = SYS_STATE_suspend;
>      freeze_domains();
>  
> +    status = disable_nonboot_cpus();
> +    if ( status )
> +    {
> +        system_state = SYS_STATE_resume;
> +        goto resume_nonboot_cpus;
> +    }
> +
>      system_state = SYS_STATE_resume;
>  
> +resume_nonboot_cpus:
> +    enable_nonboot_cpus();
>      thaw_domains();
>      system_state = SYS_STATE_active;
> +    dsb(sy);
>  
> -    return -ENOSYS;
> +    return status;
>  }

I think we need a spin_lock to protect system_suspend from concurrent
calls, or (better) we need to make sure that the caller is only allowed
to call system_suspend if there is just one vcpu active in the system,
and that vcpu is blocked on this PSCI system suspend call.


>  int32_t domain_suspend(register_t epoint, register_t cid)
> -- 
> 2.13.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 06/18] xen/x86: Move freeze/thaw_domains into common files

[Stefano Stabellini]

On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> These functions will be reused by suspend/resume support for ARM.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>  xen/arch/x86/acpi/power.c | 28 ----------------------------
>  xen/common/domain.c       | 29 +++++++++++++++++++++++++++++
>  xen/include/xen/sched.h   |  3 +++
>  3 files changed, 32 insertions(+), 28 deletions(-)
> 
> diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c
> index 93e967fe8f..794750e45b 100644
> --- a/xen/arch/x86/acpi/power.c
> +++ b/xen/arch/x86/acpi/power.c
> @@ -109,34 +109,6 @@ static void device_power_up(enum dev_power_saved saved)
>      }
>  }
>  
> -static void freeze_domains(void)
> -{
> -    struct domain *d;
> -
> -    rcu_read_lock(&domlist_read_lock);
> -    /*
> -     * Note that we iterate in order of domain-id. Hence we will pause dom0
> -     * first which is required for correctness (as only dom0 can add domains to
> -     * the domain list). Otherwise we could miss concurrently-created domains.
> -     */
> -    for_each_domain ( d )
> -        domain_pause(d);
> -    rcu_read_unlock(&domlist_read_lock);
> -}
> -
> -static void thaw_domains(void)
> -{
> -    struct domain *d;
> -
> -    rcu_read_lock(&domlist_read_lock);
> -    for_each_domain ( d )
> -    {
> -        restore_vcpu_affinity(d);
> -        domain_unpause(d);
> -    }
> -    rcu_read_unlock(&domlist_read_lock);
> -}
> -
>  static void acpi_sleep_prepare(u32 state)
>  {
>      void *wakeup_vector_va;
> diff --git a/xen/common/domain.c b/xen/common/domain.c
> index d6650f0656..fdd00dc661 100644
> --- a/xen/common/domain.c
> +++ b/xen/common/domain.c
> @@ -1666,6 +1666,35 @@ int continue_hypercall_on_cpu(
>      return 0;
>  }
>  
> +

NIT: spurious newline

Aside from that:

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> +void freeze_domains(void)
> +{
> +    struct domain *d;
> +
> +    rcu_read_lock(&domlist_read_lock);
> +    /*
> +     * Note that we iterate in order of domain-id. Hence we will pause dom0
> +     * first which is required for correctness (as only dom0 can add domains to
> +     * the domain list). Otherwise we could miss concurrently-created domains.
> +     */
> +    for_each_domain ( d )
> +        domain_pause(d);
> +    rcu_read_unlock(&domlist_read_lock);
> +}
> +
> +void thaw_domains(void)
> +{
> +    struct domain *d;
> +
> +    rcu_read_lock(&domlist_read_lock);
> +    for_each_domain ( d )
> +    {
> +        restore_vcpu_affinity(d);
> +        domain_unpause(d);
> +    }
> +    rcu_read_unlock(&domlist_read_lock);
> +}
> +
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> index 366acaf69a..c7a6d9504a 100644
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -821,6 +821,9 @@ static inline int domain_pause_by_systemcontroller_nosync(struct domain *d)
>  void domain_pause_except_self(struct domain *d);
>  void domain_unpause_except_self(struct domain *d);
>  
> +void freeze_domains(void);
> +void thaw_domains(void);
> +
>  void cpu_init(void);
>  
>  struct scheduler;
> -- 
> 2.13.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 09/18] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume

[Stefano Stabellini]

On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> The rcu_barrier() has to be added to ensure that the per cpu area is
> freed before a non-boot CPU tries to initialize it (_free_percpu_area()
> has to be called before the init_percpu_area()). This scenario occurs
> when non-boot CPUs are hot-unplugged on suspend and hotplugged on resume.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  xen/arch/arm/suspend.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index dae1b1f7d6..8e8e531d61 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -133,6 +133,7 @@ static long system_suspend(void *data)
>      system_state = SYS_STATE_resume;
>  
>  resume_nonboot_cpus:
> +    rcu_barrier();
>      enable_nonboot_cpus();
>      thaw_domains();
>      system_state = SYS_STATE_active;
> -- 
> 2.13.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Stefano Stabellini]

On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> System suspend may lead to a state where GIC would be powered down.
> Therefore, Xen should save/restore the context of GIC on suspend/resume.
> Note that the context consists of states of registers which are
> controlled by the hypervisor. Other GIC registers which are accessible
> by guests are saved/restored on context switch.
> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
> the GIC.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>  xen/arch/arm/gic-v2.c     | 147 ++++++++++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/gic.c        |  27 +++++++++
>  xen/include/asm-arm/gic.h |   8 +++
>  3 files changed, 182 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index e7eb01f30a..bb52b64ecb 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
>  /* Maximum cpu interface per GIC */
>  #define NR_GIC_CPU_IF 8
>  
> +/* GICv2 registers to be saved/restored on system suspend/resume */
> +struct gicv2_context {
> +    /* GICC context */
> +    uint32_t gicc_ctlr;
> +    uint32_t gicc_pmr;
> +    uint32_t gicc_bpr;
> +    /* GICD context */
> +    uint32_t gicd_ctlr;
> +    uint32_t *gicd_isenabler;
> +    uint32_t *gicd_isactiver;
> +    uint32_t *gicd_ipriorityr;
> +    uint32_t *gicd_itargetsr;
> +    uint32_t *gicd_icfgr;
> +};
> +
> +static struct gicv2_context gicv2_context;
> +
> +static void gicv2_alloc_context(struct gicv2_context *gc);
> +
>  static inline void writeb_gicd(uint8_t val, unsigned int offset)
>  {
>      writeb_relaxed(val, gicv2.map_dbase + offset);
> @@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
>  
>      spin_unlock(&gicv2.lock);
>  
> +    /* Allocate memory to be used for saving GIC context during the suspend */
> +    gicv2_alloc_context(&gicv2_context);

Please check for the return of gicv2_alloc_context and return error
accordingly.


>      return 0;
>  }
>  
> @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
>      BUG();
>  }
>  
> +static void gicv2_alloc_context(struct gicv2_context *gc)
> +{
> +    uint32_t n = gicv2_info.nr_lines;
> +
> +    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> +    if ( !gc->gicd_isenabler )
> +        return;

I would return error and return error also below for all the other
similar cases.


> +    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> +    if ( !gc->gicd_isactiver )
> +        goto free_gicd_isenabler;
> +
> +    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> +    if ( !gc->gicd_itargetsr )
> +        goto free_gicd_isactiver;
> +
> +    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> +    if ( !gc->gicd_ipriorityr )
> +        goto free_gicd_itargetsr;
> +
> +    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
> +    if ( gc->gicd_icfgr )
> +        return;
> +
> +    xfree(gc->gicd_ipriorityr);
> +
> +free_gicd_itargetsr:

You can have just one label that frees everything, as you can rely on
xfree working fine (doing nothing) for NULL pointers.


> +    xfree(gc->gicd_itargetsr);
> +
> +free_gicd_isactiver:
> +    xfree(gc->gicd_isactiver);
> +
> +free_gicd_isenabler:
> +    xfree(gc->gicd_isenabler);
> +    gc->gicd_isenabler = NULL;
> +}
> +
> +static int gicv2_suspend(void)
> +{
> +    int i;
> +
> +    /* Save GICC configuration */
> +    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
> +    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
> +    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
> +
> +    /* If gicv2_alloc_context() hasn't allocated memory, return */
> +    if ( !gicv2_context.gicd_isenabler )
> +        return -ENOMEM;

If you are going to check for this, then please check for all the others
as well (gicd_isactiver, gicd_ipriorityr, etc.) But if you follow my
other suggestion to return error if we fail the memory allocation at
init, then this can become an ASSERT. Also, ASSERTS or checks should be
at the very beginning of this function.


> +    /* Save GICD configuration */
> +    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> +        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);

Technically, GICD_ICFGR doesn't need to be saved because it could be
entirely reconstructed from the informations we have, but I imagine it
could be difficult to call the right set of route_irq_to_guest/xen calls
at resume time, so I think it is OK.


> +    return 0;
> +}
> +
> +static void gicv2_resume(void)
> +{
> +    int i;
> +
> +    ASSERT(gicv2_context.gicd_isenabler);
> +    ASSERT(gicv2_context.gicd_isactiver);
> +    ASSERT(gicv2_context.gicd_ipriorityr);
> +    ASSERT(gicv2_context.gicd_itargetsr);
> +    ASSERT(gicv2_context.gicd_icfgr);
> +
> +    /* Disable CPU interface and distributor */
> +    writel_gicc(0, GICC_CTLR);
> +    writel_gicd(0, GICD_CTLR);
> +    isb();
> +
> +    /* Restore GICD configuration */
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> +        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
> +
> +    /* Make sure all registers are restored and enable distributor */
> +    isb();
> +    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
> +
> +    /* Restore GIC CPU interface configuration */
> +    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
> +    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
> +    isb();

I don't think we need all these isb()'s in this function. Maybe only one
at the end, but probably not even that. Julien, what do you think?


> +    /* Enable GIC CPU interface */
> +    writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | GICC_CTL_EOI,
> +                GICC_CTLR);
> +    isb();
> +}
> +
>  const static struct gic_hw_operations gicv2_ops = {
>      .info                = &gicv2_info,
>      .init                = gicv2_init,
> @@ -1351,6 +1496,8 @@ const static struct gic_hw_operations gicv2_ops = {
>      .map_hwdom_extra_mappings = gicv2_map_hwdown_extra_mappings,
>      .iomem_deny_access   = gicv2_iomem_deny_access,
>      .do_LPI              = gicv2_do_LPI,
> +    .suspend             = gicv2_suspend,
> +    .resume              = gicv2_resume,
>  };
>  
>  /* Set up the GIC */
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index e524ad583d..6e98f43691 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -464,6 +464,33 @@ int gic_iomem_deny_access(const struct domain *d)
>      return gic_hw_ops->iomem_deny_access(d);
>  }
>  
> +int gic_suspend(void)
> +{
> +    /* Must be called by boot CPU#0 with interrupts disabled */
> +    ASSERT(!local_irq_is_enabled());
> +    ASSERT(!smp_processor_id());
> +
> +    if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
> +        return -ENOSYS;
> +
> +    gic_hw_ops->suspend();

Should return the return value of this suspend() call.


> +    return 0;
> +}
> +
> +void gic_resume(void)
> +{
> +    /*
> +     * Must be called by boot CPU#0 with interrupts disabled after gic_suspend
> +     * has returned successfully.
> +     */
> +    ASSERT(!local_irq_is_enabled());
> +    ASSERT(!smp_processor_id());
> +    ASSERT(gic_hw_ops->resume);
> +
> +    gic_hw_ops->resume();
> +}
> +
>  static int cpu_gic_callback(struct notifier_block *nfb,
>                              unsigned long action,
>                              void *hcpu)
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 22fa122e52..46066caac8 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -277,6 +277,10 @@ extern int gicv_setup(struct domain *d);
>  extern void gic_save_state(struct vcpu *v);
>  extern void gic_restore_state(struct vcpu *v);
>  
> +/* Suspend/resume */
> +extern int gic_suspend(void);
> +extern void gic_resume(void);
> +
>  /* SGI (AKA IPIs) */
>  enum gic_sgi {
>      GIC_SGI_EVENT_CHECK = 0,
> @@ -390,6 +394,10 @@ struct gic_hw_operations {
>      int (*iomem_deny_access)(const struct domain *d);
>      /* Handle LPIs, which require special handling */
>      void (*do_LPI)(unsigned int lpi);
> +    /* Save GIC configuration due to the system suspend */
> +    int (*suspend)(void);
> +    /* Restore GIC configuration due to the system resume */
> +    void (*resume)(void);
>  };
>  
>  extern const struct gic_hw_operations *gic_hw_ops;
> -- 
> 2.13.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 13/18] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)

[Stefano Stabellini]

On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> PSCI system suspend function shall be invoked to finalize Xen suspend
> procedure. Resume entry point, which needs to be passed via 1st argument
> of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
> is just a placeholder that will be implemented in assembly. Context ID,
> which is 2nd argument of system suspend PSCI call, is unused, as in Linux.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> 
> ---
> Changes in v2:
> 
> -The commit message was stale - referring to the do_suspend function
> that has been renamed long time ago. Fixed commit message
> ---
>  xen/arch/arm/arm64/entry.S    |  3 +++
>  xen/arch/arm/psci.c           | 16 ++++++++++++++++
>  xen/arch/arm/suspend.c        |  4 ++++
>  xen/include/asm-arm/psci.h    |  1 +
>  xen/include/asm-arm/suspend.h |  1 +
>  5 files changed, 25 insertions(+)
> 
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index 97b05f53ea..dbc4717903 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -533,6 +533,9 @@ ENTRY(__context_switch)
>          mov     sp, x9
>          ret
>  
> +ENTRY(hyp_resume)
> +        b .
> +
>  /*
>   * Local variables:
>   * mode: ASM
> diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
> index a93121f43b..b100bd8ad2 100644
> --- a/xen/arch/arm/psci.c
> +++ b/xen/arch/arm/psci.c
> @@ -24,6 +24,7 @@
>  #include <asm/cpufeature.h>
>  #include <asm/psci.h>
>  #include <asm/acpi.h>
> +#include <asm/suspend.h>
>  
>  /*
>   * While a 64-bit OS can make calls with SMC32 calling conventions, for
> @@ -67,6 +68,21 @@ void call_psci_cpu_off(void)
>      }
>  }
>  
> +int call_psci_system_suspend(void)
> +{
> +#ifdef CONFIG_ARM_64
> +    struct arm_smccc_res res;
> +
> +    /* 2nd argument (context ID) is not used */
> +    arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), &res);
> +
> +    return PSCI_RET(res);
> +#else
> +    /* not supported */
> +    return 1;
> +#endif
> +}

Given that suspend is unimplemented on arm32, the #ifdef is OK. But
in that case return PSCI_NOT_SUPPORTED for arm32.

Otherwise you should be able to remove this #ifdef by introducing
something similar to PSCI_0_2_FN_NATIVE, but for PSCI_1_0 calls.


>  void call_psci_system_off(void)
>  {
>      if ( psci_ver > PSCI_VERSION(0, 1) )
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index d1b48c339a..37926374bc 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -141,6 +141,10 @@ static long system_suspend(void *data)
>          goto resume_irqs;
>      }
>  
> +    status = call_psci_system_suspend();
> +    if ( status )
> +        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
> +
>      system_state = SYS_STATE_resume;
>  
>      gic_resume();
> diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> index 26462d0c47..9f6116a224 100644
> --- a/xen/include/asm-arm/psci.h
> +++ b/xen/include/asm-arm/psci.h
> @@ -20,6 +20,7 @@ extern uint32_t psci_ver;
>  
>  int psci_init(void);
>  int call_psci_cpu_on(int cpu);
> +int call_psci_system_suspend(void);
>  void call_psci_cpu_off(void);
>  void call_psci_system_off(void);
>  void call_psci_system_reset(void);
> diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> index de787d296a..7604e2e2e2 100644
> --- a/xen/include/asm-arm/suspend.h
> +++ b/xen/include/asm-arm/suspend.h
> @@ -2,6 +2,7 @@
>  #define __ASM_ARM_SUSPEND_H__
>  
>  int32_t domain_suspend(register_t epoint, register_t cid);
> +void hyp_resume(void);

I think it would be better to separate physical suspend from virtual
suspend, like Julien suggested. As you separate the C implementation,
please also introduce separate header files (for instance vsuspend.h and
suspend.h).


>  #endif
>  
> -- 
> 2.13.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Julien Grall]

Hi Stefano,

On 11/13/18 11:41 PM, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Mirela Simonovic wrote:
>> System suspend may lead to a state where GIC would be powered down.
>> Therefore, Xen should save/restore the context of GIC on suspend/resume.
>> Note that the context consists of states of registers which are
>> controlled by the hypervisor. Other GIC registers which are accessible
>> by guests are saved/restored on context switch.
>> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
>> the GIC.
>>
>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>> ---
>>   xen/arch/arm/gic-v2.c     | 147 ++++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/gic.c        |  27 +++++++++
>>   xen/include/asm-arm/gic.h |   8 +++
>>   3 files changed, 182 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>> index e7eb01f30a..bb52b64ecb 100644
>> --- a/xen/arch/arm/gic-v2.c
>> +++ b/xen/arch/arm/gic-v2.c
>> @@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
>>   /* Maximum cpu interface per GIC */
>>   #define NR_GIC_CPU_IF 8
>>   
>> +/* GICv2 registers to be saved/restored on system suspend/resume */
>> +struct gicv2_context {
>> +    /* GICC context */
>> +    uint32_t gicc_ctlr;
>> +    uint32_t gicc_pmr;
>> +    uint32_t gicc_bpr;
>> +    /* GICD context */
>> +    uint32_t gicd_ctlr;
>> +    uint32_t *gicd_isenabler;
>> +    uint32_t *gicd_isactiver;
>> +    uint32_t *gicd_ipriorityr;
>> +    uint32_t *gicd_itargetsr;
>> +    uint32_t *gicd_icfgr;
>> +};
>> +
>> +static struct gicv2_context gicv2_context;
>> +
>> +static void gicv2_alloc_context(struct gicv2_context *gc);
>> +
>>   static inline void writeb_gicd(uint8_t val, unsigned int offset)
>>   {
>>       writeb_relaxed(val, gicv2.map_dbase + offset);
>> @@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
>>   
>>       spin_unlock(&gicv2.lock);
>>   
>> +    /* Allocate memory to be used for saving GIC context during the suspend */
>> +    gicv2_alloc_context(&gicv2_context);
> 
> Please check for the return of gicv2_alloc_context and return error
> accordingly.

Suspend/resume is not a critical feature in common case. So I would 
prefer if we disable it when we can't alloc memory.

I would also be tempt to #ifdef all related suspend/resume code to allow 
the integrator disabling the feature when they don't want it.

> 
> 
>>       return 0;
>>   }
>>   
>> @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
>>       BUG();
>>   }
>>   
>> +static void gicv2_alloc_context(struct gicv2_context *gc)
>> +{
>> +    uint32_t n = gicv2_info.nr_lines;
>> +
>> +    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
>> +    if ( !gc->gicd_isenabler )
>> +        return;
> 
> I would return error and return error also below for all the other
> similar cases.
> 
> 
>> +    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
>> +    if ( !gc->gicd_isactiver )
>> +        goto free_gicd_isenabler;
>> +
>> +    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
>> +    if ( !gc->gicd_itargetsr )
>> +        goto free_gicd_isactiver;
>> +
>> +    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
>> +    if ( !gc->gicd_ipriorityr )
>> +        goto free_gicd_itargetsr;
>> +
>> +    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
>> +    if ( gc->gicd_icfgr )
>> +        return;
>> +
>> +    xfree(gc->gicd_ipriorityr);
>> +
>> +free_gicd_itargetsr:
> 
> You can have just one label that frees everything, as you can rely on
> xfree working fine (doing nothing) for NULL pointers.
> 
> 
>> +    xfree(gc->gicd_itargetsr);
>> +
>> +free_gicd_isactiver:
>> +    xfree(gc->gicd_isactiver);
>> +
>> +free_gicd_isenabler:
>> +    xfree(gc->gicd_isenabler);
>> +    gc->gicd_isenabler = NULL;
>> +}
>> +
>> +static int gicv2_suspend(void)
>> +{
>> +    int i;
>> +
>> +    /* Save GICC configuration */
>> +    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
>> +    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
>> +    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
>> +
>> +    /* If gicv2_alloc_context() hasn't allocated memory, return */
>> +    if ( !gicv2_context.gicd_isenabler )
>> +        return -ENOMEM;
> 
> If you are going to check for this, then please check for all the others
> as well (gicd_isactiver, gicd_ipriorityr, etc.) But if you follow my
> other suggestion to return error if we fail the memory allocation at
> init, then this can become an ASSERT. Also, ASSERTS or checks should be
> at the very beginning of this function.
> 
> 
>> +    /* Save GICD configuration */
>> +    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
>> +        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
> 
> Technically, GICD_ICFGR doesn't need to be saved because it could be
> entirely reconstructed from the informations we have, but I imagine it
> could be difficult to call the right set of route_irq_to_guest/xen calls
> at resume time, so I think it is OK.
> 
> 
>> +    return 0;
>> +}
>> +
>> +static void gicv2_resume(void)
>> +{
>> +    int i;
>> +
>> +    ASSERT(gicv2_context.gicd_isenabler);
>> +    ASSERT(gicv2_context.gicd_isactiver);
>> +    ASSERT(gicv2_context.gicd_ipriorityr);
>> +    ASSERT(gicv2_context.gicd_itargetsr);
>> +    ASSERT(gicv2_context.gicd_icfgr);
>> +
>> +    /* Disable CPU interface and distributor */
>> +    writel_gicc(0, GICC_CTLR);
>> +    writel_gicd(0, GICD_CTLR);
>> +    isb();
>> +
>> +    /* Restore GICD configuration */
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
>> +        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
>> +
>> +    /* Make sure all registers are restored and enable distributor */
>> +    isb();
>> +    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
>> +
>> +    /* Restore GIC CPU interface configuration */
>> +    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
>> +    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
>> +    isb();
> 
> I don't think we need all these isb()'s in this function. Maybe only one
> at the end, but probably not even that. Julien, what do you think?

I don't think any of the isb() in the code are necessary. What are we 
trying to prevent with them?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi,

On 13/11/2018 20:39, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>> However, what is the issue with saving all the registers here?
>>>>
>>>
>>> We need to save arguments that are provided by a guest with system
>>> suspend PSCI call. These arguments are the entry point that needs to
>>> be saved in program counter and context ID that needs to be saved in
>>> x0/r0. We don't have these arguments here. Context switch happens
>>> after processing the system suspend PSCI call, so it's too late.
>>
>> It does not feel right to modify ctxt_switch{from,to} for suspend/resume. If
>> you want to reset the vCPU state before blocking the vCPU, then you should
>> instead
> 
> You missed the end of the suggestion here

Whoops. I meant that instead you should save the context of the vCPU in advance 
or reset the vCPU using the system registers directly.

But my preference is to reset the vCPU when you receive the wake-up interrupt.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Julien Grall]

Hi,

On 13/11/2018 20:44, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Julien Grall wrote:
>> (+ Andre)
>>
>> On 11/12/18 5:42 PM, Mirela Simonovic wrote:
>>> Hi Julien,
>>>
>>> On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com> wrote:
>>>>
>>>>
>>>>
>>>> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
>>>>> Hi Julien,
>>>>
>>>> Hi,
>>>>
>>>>> Thanks for the feedback.
>>>>>
>>>>> On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi Mirela,
>>>>>>
>>>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>>>> GIC and virtual timer context must be saved when the domain
>>>>>>> suspends.
>>>>>>
>>>>>> Please provide the rationale for this. Is it required by the spec?
>>>>>>
>>>>>
>>>>> This is required for GIC because a guest leaves enabled interrupts in
>>>>> the GIC that could wake it up, and on resume it should be able to
>>>>> detect which interrupt woke it up. Without saving/restoring the state
>>>>> of GIC this would not be possible.
>>>>
>>>> I am confused. In patch #10, you save the GIC host because the GIC can
>>>> be powered-down. Linux is also saving the GIC state. So how the
>>>> interrupt can come up if the GIC is powered down?
>>>>
>>>
>>> After Xen (or Linux in the config without Xen) hands over the control
>>> to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
>>> enabled interrupts that are its wake-up sources. Saving a GIC state
>>> only means that its current configuration will be remembered somewhere
>>> in data structures, but the configuration is not changed on suspend.
>>> Everything that happens with interrupt configuration from this point
>>> on is platform specific. Now there are 2 options: 1) EL3 will never
>>> allow CPU0 to be powered down and the wake-up interrupt will indeed
>>> propagate via GIC;
>>> or 2) CPU0 will be powered down and the GIC may be
>>> powered down as well, so an external help is needed to deal with
>>> wake-up and resume (e.g. in order to react to a wake-up interrupt
>>> while the GIC is down, and power up CPU0). This external help is
>>> provided by some low-power processor outside of the Cortex-A cluster.
>>>
>>> So the platform firmware is responsible for properly configuring a
>>> wake-up path if GIC goes down. This is commonly handled by EL3
>>> communicating with low-power processor. When the GIC power comes up,
>>> the interrupt triggered by a peripheral is still active and the
>>> software on Cortex-A cluster should be able to observe it once the GIC
>>> state is restored, i.e. interrupts get enabled at GIC.
>>
>> Thank you for the explanation.  Now the question is why can't we reset at
>> least the GIC CPU interface?
>>
>> AFAIU, the guest should restore them in any case. The only things we need to
>> know is the interrupt was received for a given guest. We can then queue it and
>> wake-up the domain.
>>
>> This seems to fit with the description on top of gic_dist_save() in Linux
>> GICv2 driver.
> 
> Can we rely on all PSCI compliant OSes to restore their own GIC again at
> resume? The PSCI spec is not very clear in that regard (at the the
> version I am looking at...) I am just asking so that we don't come up
> with a solution that only works with Linux.
See PSCI 1.1 (DEN0022D) section 6.8. Each level should save its own context 
because the PSCI implementations is allowed to shutdown the GIC.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Julien Grall]

Hi Mirela,

On 12/11/2018 11:30, Mirela Simonovic wrote:
> Non-boot CPUs have to be disabled on suspend and enabled on resume
> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
> platform capabilities, this may lead to the physical powering down of
> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
> each non-boot CPU).
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c | 15 ++++++++++++++-
>   1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index 575afd5eb8..dae1b1f7d6 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -1,4 +1,5 @@
>   #include <xen/sched.h>
> +#include <xen/cpu.h>
>   #include <asm/cpufeature.h>
>   #include <asm/event.h>
>   #include <asm/psci.h>
> @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>   /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>   static long system_suspend(void *data)
>   {
> +    int status;
> +
>       BUG_ON(system_state != SYS_STATE_active);
>   
>       system_state = SYS_STATE_suspend;
>       freeze_domains();
>   
> +    status = disable_nonboot_cpus();
> +    if ( status )
> +    {
> +        system_state = SYS_STATE_resume;
> +        goto resume_nonboot_cpus;
> +    }
> +
>       system_state = SYS_STATE_resume;
>   
> +resume_nonboot_cpus:
> +    enable_nonboot_cpus();
>       thaw_domains();
>       system_state = SYS_STATE_active;
> +    dsb(sy);

Why do you need a dsb(sy) here?

>   
> -    return -ENOSYS;

Why do you return -ENOSYS until this patch? Should not have it be 0?

> +    return status;
>   }
>   
>   int32_t domain_suspend(register_t epoint, register_t cid)
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 01/18] xen/arm: Move code that initializes VCPU context into a separate function

[Mirela Simonovic]

On Mon, Nov 12, 2018 at 2:15 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > From: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >
> > The arch_set_info_guest() has code to initialize the context of a VCPU.
> > When a VCPU is resumed it needs to go through the same context
> > initialization excluding all the validations that this routine does.
> > We move the actual VCPU context setting into a function so that it can be
> > shared with the resume path.
>
> I would rather not introduce a function that skip validation. They are
> here to catch error when setting up vCPU registers. If those errors are
> not caught, then the hypervisor will likely crash on return to the guest.
>

Ok, then we should just use arch_set_info_guest.

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Mirela Simonovic]

On Wed, Nov 14, 2018 at 10:13 AM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Stefano,
>
> On 11/13/18 11:41 PM, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> >> System suspend may lead to a state where GIC would be powered down.
> >> Therefore, Xen should save/restore the context of GIC on suspend/resume.
> >> Note that the context consists of states of registers which are
> >> controlled by the hypervisor. Other GIC registers which are accessible
> >> by guests are saved/restored on context switch.
> >> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
> >> the GIC.
> >>
> >> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> >> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >> ---
> >>   xen/arch/arm/gic-v2.c     | 147 ++++++++++++++++++++++++++++++++++++++++++++++
> >>   xen/arch/arm/gic.c        |  27 +++++++++
> >>   xen/include/asm-arm/gic.h |   8 +++
> >>   3 files changed, 182 insertions(+)
> >>
> >> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> >> index e7eb01f30a..bb52b64ecb 100644
> >> --- a/xen/arch/arm/gic-v2.c
> >> +++ b/xen/arch/arm/gic-v2.c
> >> @@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
> >>   /* Maximum cpu interface per GIC */
> >>   #define NR_GIC_CPU_IF 8
> >>
> >> +/* GICv2 registers to be saved/restored on system suspend/resume */
> >> +struct gicv2_context {
> >> +    /* GICC context */
> >> +    uint32_t gicc_ctlr;
> >> +    uint32_t gicc_pmr;
> >> +    uint32_t gicc_bpr;
> >> +    /* GICD context */
> >> +    uint32_t gicd_ctlr;
> >> +    uint32_t *gicd_isenabler;
> >> +    uint32_t *gicd_isactiver;
> >> +    uint32_t *gicd_ipriorityr;
> >> +    uint32_t *gicd_itargetsr;
> >> +    uint32_t *gicd_icfgr;
> >> +};
> >> +
> >> +static struct gicv2_context gicv2_context;
> >> +
> >> +static void gicv2_alloc_context(struct gicv2_context *gc);
> >> +
> >>   static inline void writeb_gicd(uint8_t val, unsigned int offset)
> >>   {
> >>       writeb_relaxed(val, gicv2.map_dbase + offset);
> >> @@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
> >>
> >>       spin_unlock(&gicv2.lock);
> >>
> >> +    /* Allocate memory to be used for saving GIC context during the suspend */
> >> +    gicv2_alloc_context(&gicv2_context);
> >
> > Please check for the return of gicv2_alloc_context and return error
> > accordingly.
>
> Suspend/resume is not a critical feature in common case. So I would
> prefer if we disable it when we can't alloc memory.
>
> I would also be tempt to #ifdef all related suspend/resume code to allow
> the integrator disabling the feature when they don't want it.
>
> >
> >
> >>       return 0;
> >>   }
> >>
> >> @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
> >>       BUG();
> >>   }
> >>
> >> +static void gicv2_alloc_context(struct gicv2_context *gc)
> >> +{
> >> +    uint32_t n = gicv2_info.nr_lines;
> >> +
> >> +    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> >> +    if ( !gc->gicd_isenabler )
> >> +        return;
> >
> > I would return error and return error also below for all the other
> > similar cases.
> >
> >
> >> +    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> >> +    if ( !gc->gicd_isactiver )
> >> +        goto free_gicd_isenabler;
> >> +
> >> +    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> >> +    if ( !gc->gicd_itargetsr )
> >> +        goto free_gicd_isactiver;
> >> +
> >> +    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> >> +    if ( !gc->gicd_ipriorityr )
> >> +        goto free_gicd_itargetsr;
> >> +
> >> +    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
> >> +    if ( gc->gicd_icfgr )
> >> +        return;
> >> +
> >> +    xfree(gc->gicd_ipriorityr);
> >> +
> >> +free_gicd_itargetsr:
> >
> > You can have just one label that frees everything, as you can rely on
> > xfree working fine (doing nothing) for NULL pointers.
> >
> >
> >> +    xfree(gc->gicd_itargetsr);
> >> +
> >> +free_gicd_isactiver:
> >> +    xfree(gc->gicd_isactiver);
> >> +
> >> +free_gicd_isenabler:
> >> +    xfree(gc->gicd_isenabler);
> >> +    gc->gicd_isenabler = NULL;
> >> +}
> >> +
> >> +static int gicv2_suspend(void)
> >> +{
> >> +    int i;
> >> +
> >> +    /* Save GICC configuration */
> >> +    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
> >> +    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
> >> +    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
> >> +
> >> +    /* If gicv2_alloc_context() hasn't allocated memory, return */
> >> +    if ( !gicv2_context.gicd_isenabler )
> >> +        return -ENOMEM;
> >
> > If you are going to check for this, then please check for all the others
> > as well (gicd_isactiver, gicd_ipriorityr, etc.) But if you follow my
> > other suggestion to return error if we fail the memory allocation at
> > init, then this can become an ASSERT. Also, ASSERTS or checks should be
> > at the very beginning of this function.
> >
> >
> >> +    /* Save GICD configuration */
> >> +    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> >> +        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> >> +        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> >> +        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> >> +        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> >> +        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
> >
> > Technically, GICD_ICFGR doesn't need to be saved because it could be
> > entirely reconstructed from the informations we have, but I imagine it
> > could be difficult to call the right set of route_irq_to_guest/xen calls
> > at resume time, so I think it is OK.
> >
> >
> >> +    return 0;
> >> +}
> >> +
> >> +static void gicv2_resume(void)
> >> +{
> >> +    int i;
> >> +
> >> +    ASSERT(gicv2_context.gicd_isenabler);
> >> +    ASSERT(gicv2_context.gicd_isactiver);
> >> +    ASSERT(gicv2_context.gicd_ipriorityr);
> >> +    ASSERT(gicv2_context.gicd_itargetsr);
> >> +    ASSERT(gicv2_context.gicd_icfgr);
> >> +
> >> +    /* Disable CPU interface and distributor */
> >> +    writel_gicc(0, GICC_CTLR);
> >> +    writel_gicd(0, GICD_CTLR);
> >> +    isb();
> >> +
> >> +    /* Restore GICD configuration */
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> >> +        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> >> +        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> >> +        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> >> +        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> >> +        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> >> +        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
> >> +
> >> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> >> +        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
> >> +
> >> +    /* Make sure all registers are restored and enable distributor */
> >> +    isb();
> >> +    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
> >> +
> >> +    /* Restore GIC CPU interface configuration */
> >> +    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
> >> +    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
> >> +    isb();
> >
> > I don't think we need all these isb()'s in this function. Maybe only one
> > at the end, but probably not even that. Julien, what do you think?
>
> I don't think any of the isb() in the code are necessary. What are we
> trying to prevent with them?
>

I also think it's not needed - anywhere

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 13/18] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)

[Mirela Simonovic]

On Wed, Nov 14, 2018 at 1:14 AM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> > PSCI system suspend function shall be invoked to finalize Xen suspend
> > procedure. Resume entry point, which needs to be passed via 1st argument
> > of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
> > is just a placeholder that will be implemented in assembly. Context ID,
> > which is 2nd argument of system suspend PSCI call, is unused, as in Linux.
> >
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >
> > ---
> > Changes in v2:
> >
> > -The commit message was stale - referring to the do_suspend function
> > that has been renamed long time ago. Fixed commit message
> > ---
> >  xen/arch/arm/arm64/entry.S    |  3 +++
> >  xen/arch/arm/psci.c           | 16 ++++++++++++++++
> >  xen/arch/arm/suspend.c        |  4 ++++
> >  xen/include/asm-arm/psci.h    |  1 +
> >  xen/include/asm-arm/suspend.h |  1 +
> >  5 files changed, 25 insertions(+)
> >
> > diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> > index 97b05f53ea..dbc4717903 100644
> > --- a/xen/arch/arm/arm64/entry.S
> > +++ b/xen/arch/arm/arm64/entry.S
> > @@ -533,6 +533,9 @@ ENTRY(__context_switch)
> >          mov     sp, x9
> >          ret
> >
> > +ENTRY(hyp_resume)
> > +        b .
> > +
> >  /*
> >   * Local variables:
> >   * mode: ASM
> > diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
> > index a93121f43b..b100bd8ad2 100644
> > --- a/xen/arch/arm/psci.c
> > +++ b/xen/arch/arm/psci.c
> > @@ -24,6 +24,7 @@
> >  #include <asm/cpufeature.h>
> >  #include <asm/psci.h>
> >  #include <asm/acpi.h>
> > +#include <asm/suspend.h>
> >
> >  /*
> >   * While a 64-bit OS can make calls with SMC32 calling conventions, for
> > @@ -67,6 +68,21 @@ void call_psci_cpu_off(void)
> >      }
> >  }
> >
> > +int call_psci_system_suspend(void)
> > +{
> > +#ifdef CONFIG_ARM_64
> > +    struct arm_smccc_res res;
> > +
> > +    /* 2nd argument (context ID) is not used */
> > +    arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), &res);
> > +
> > +    return PSCI_RET(res);
> > +#else
> > +    /* not supported */
> > +    return 1;
> > +#endif
> > +}
>
> Given that suspend is unimplemented on arm32, the #ifdef is OK. But
> in that case return PSCI_NOT_SUPPORTED for arm32.
>
> Otherwise you should be able to remove this #ifdef by introducing
> something similar to PSCI_0_2_FN_NATIVE, but for PSCI_1_0 calls.
>
>
> >  void call_psci_system_off(void)
> >  {
> >      if ( psci_ver > PSCI_VERSION(0, 1) )
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index d1b48c339a..37926374bc 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -141,6 +141,10 @@ static long system_suspend(void *data)
> >          goto resume_irqs;
> >      }
> >
> > +    status = call_psci_system_suspend();
> > +    if ( status )
> > +        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
> > +
> >      system_state = SYS_STATE_resume;
> >
> >      gic_resume();
> > diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> > index 26462d0c47..9f6116a224 100644
> > --- a/xen/include/asm-arm/psci.h
> > +++ b/xen/include/asm-arm/psci.h
> > @@ -20,6 +20,7 @@ extern uint32_t psci_ver;
> >
> >  int psci_init(void);
> >  int call_psci_cpu_on(int cpu);
> > +int call_psci_system_suspend(void);
> >  void call_psci_cpu_off(void);
> >  void call_psci_system_off(void);
> >  void call_psci_system_reset(void);
> > diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> > index de787d296a..7604e2e2e2 100644
> > --- a/xen/include/asm-arm/suspend.h
> > +++ b/xen/include/asm-arm/suspend.h
> > @@ -2,6 +2,7 @@
> >  #define __ASM_ARM_SUSPEND_H__
> >
> >  int32_t domain_suspend(register_t epoint, register_t cid);
> > +void hyp_resume(void);
>
> I think it would be better to separate physical suspend from virtual
> suspend, like Julien suggested. As you separate the C implementation,
> please also introduce separate header files (for instance vsuspend.h and
> suspend.h).
>

AFAIU Julien came back with Andrew's feedback suggesting that this
should rather not be done. Please see discussion "[PATCH 02/18]
xen/arm: Implement PSCI system suspend call (virtual interface)"

>
> >  #endif
> >
> > --
> > 2.13.0
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Julien Grall]

Hi,

On 13/11/2018 22:35, Stefano Stabellini wrote:
> On Mon, 12 Nov 2018, Mirela Simonovic wrote:
>> Non-boot CPUs have to be disabled on suspend and enabled on resume
>> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
>> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
>> platform capabilities, this may lead to the physical powering down of
>> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
>> each non-boot CPU).
>>
>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>> ---
>>   xen/arch/arm/suspend.c | 15 ++++++++++++++-
>>   1 file changed, 14 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>> index 575afd5eb8..dae1b1f7d6 100644
>> --- a/xen/arch/arm/suspend.c
>> +++ b/xen/arch/arm/suspend.c
>> @@ -1,4 +1,5 @@
>>   #include <xen/sched.h>
>> +#include <xen/cpu.h>
>>   #include <asm/cpufeature.h>
>>   #include <asm/event.h>
>>   #include <asm/psci.h>
>> @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>>   /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>>   static long system_suspend(void *data)
>>   {
>> +    int status;
>> +
>>       BUG_ON(system_state != SYS_STATE_active);
>>   
>>       system_state = SYS_STATE_suspend;
>>       freeze_domains();
>>   
>> +    status = disable_nonboot_cpus();
>> +    if ( status )
>> +    {
>> +        system_state = SYS_STATE_resume;
>> +        goto resume_nonboot_cpus;
>> +    }
>> +
>>       system_state = SYS_STATE_resume;
>>   
>> +resume_nonboot_cpus:
>> +    enable_nonboot_cpus();
>>       thaw_domains();
>>       system_state = SYS_STATE_active;
>> +    dsb(sy);
>>   
>> -    return -ENOSYS;
>> +    return status;
>>   }
> 
> I think we need a spin_lock to protect system_suspend from concurrent
> calls, or (better) we need to make sure that the caller is only allowed
> to call system_suspend if there is just one vcpu active in the system,
> and that vcpu is blocked on this PSCI system suspend call.

I don't think this is correct. It is valid to have more than on vCPU running 
when calling system_suspend. An example would be Dom0 suspending while the other 
domain are still running.

This is handled properly via freeze_domains()/thaw_domains().

So a spinlock would be a better solution here. If we can't acquire the lock then 
the function would return -EBUSY.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> On 12/11/2018 19:56, Julien Grall wrote:
>> Hi Andrew,
>>
>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>> index e594b48d81..7f8105465c 100644
>>>>>> --- a/xen/arch/arm/domain.c
>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>         if ( is_idle_vcpu(p) )
>>>>>>             return;
>>>>>>
>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>> suspended */
>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>> +        return;
>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>> SCHEDOP_shutdown). The other user of that code relies on all the state
>>>>> to be saved on suspend.
>>>>>
>>>> We just need a flag to mark a domain as suspended, and I do believe
>>>> SHUTDOWN_suspend is not used anywhere else.
>>>> Let's come back on this.
>>> SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
>>> tree and you'll find several pieces of documentation, including the
>>> description of what this shutdown code means.
>>>
>>> What you are introducing here is not a shutdown - it is a suspend with
>>> the intent to resume executing later.  As such, it shouldn't use Xen's
>>> shutdown infrastructure, which exists mainly to communicate with the
>>> toolstack.
>> Would domain pause/unpause be a better solution?
> Actually yes - that sounds like a very neat solution.

I believe domain pause will not work here - a domain cannot pause 
itself, i.e. current domain cannot be paused. This functionality seems 
to assume that a domain is pausing another domain. Or I missed the point.

> ~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 09/18] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume

[Julien Grall]

Hi,

On 12/11/2018 11:30, Mirela Simonovic wrote:
> The rcu_barrier() has to be added to ensure that the per cpu area is
> freed before a non-boot CPU tries to initialize it (_free_percpu_area()
> has to be called before the init_percpu_area()). This scenario occurs
> when non-boot CPUs are hot-unplugged on suspend and hotplugged on resume.

 From the description, this is a bug introduced by the previous patch. So should 
merged in it.

The commit message will also need to explain why the rcu_barrier() is required.

Cheers,

> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index dae1b1f7d6..8e8e531d61 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -133,6 +133,7 @@ static long system_suspend(void *data)
>       system_state = SYS_STATE_resume;
>   
>   resume_nonboot_cpus:
> +    rcu_barrier();
>       enable_nonboot_cpus();
>       thaw_domains();
>       system_state = SYS_STATE_active;
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,


On 11/14/2018 11:45 AM, Julien Grall wrote:
> Hi,
>
> On 13/11/2018 20:39, Stefano Stabellini wrote:
>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>>> However, what is the issue with saving all the registers here?
>>>>>
>>>>
>>>> We need to save arguments that are provided by a guest with system
>>>> suspend PSCI call. These arguments are the entry point that needs to
>>>> be saved in program counter and context ID that needs to be saved in
>>>> x0/r0. We don't have these arguments here. Context switch happens
>>>> after processing the system suspend PSCI call, so it's too late.
>>>
>>> It does not feel right to modify ctxt_switch{from,to} for 
>>> suspend/resume. If
>>> you want to reset the vCPU state before blocking the vCPU, then you 
>>> should
>>> instead
>>

I think it's not clear what problem are we discussing here, at least 
it's not to me. So I'll make an assumption, and please correct me if I'm 
wrong.
In the patches we submitted, the VCPU context is not reset in 
ctxt_switch{from,to}. My understanding is that you suggested/asked to 
reset the VCPU context when switch happens, and I explained why is that 
not possible - at least not without additional code changes, that may 
not be so small. I agree with Andrew's comment in this perspective - 
reset of VCPU should not (and right now cannot) be done when the context 
is switched.

>> You missed the end of the suggestion here
>
> Whoops. I meant that instead you should save the context of the vCPU 
> in advance or reset the vCPU using the system registers directly.
>
> But my preference is to reset the vCPU when you receive the wake-up 
> interrupt.
>

Without you presenting more details how would that work I cannot really 
provide any comment, nor say that your preference could work or be 
better compared to what is in this series. Honestly, I don't understand 
what exactly you're proposing, because more things needs to be 
think-through beyond the place to put a code.
We submitted a code that works, which is very elegant and nice in my 
opinion (fair to say we may not share opinions here), and does not 
require lots of code changes. So there's the reference.
Could you please clarify why do you think the proposed solution is not good?
And why do you think that what you're proposing is better? Lets be more 
clear here - how exactly you propose to implement that?

I haven't understood so far why do you think that the proposed approach 
is not good. Maybe the whole discussion drifted a bit for no reason.

Thanks,
Mirela

> Cheers,
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Julien Grall]

Hi,

I am not entirely sure where to ask it, so I will do it here. From my 
understanding of the PSCI spec, all the interrupts should have been migrated 
away from turned off CPU. Where do you ensure that in the suspend path?


On 12/11/2018 11:30, Mirela Simonovic wrote:
> System suspend may lead to a state where GIC would be powered down.
> Therefore, Xen should save/restore the context of GIC on suspend/resume.
> Note that the context consists of states of registers which are
> controlled by the hypervisor. Other GIC registers which are accessible
> by guests are saved/restored on context switch.
> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
> the GIC.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/gic-v2.c     | 147 ++++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/gic.c        |  27 +++++++++
>   xen/include/asm-arm/gic.h |   8 +++
>   3 files changed, 182 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index e7eb01f30a..bb52b64ecb 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
>   /* Maximum cpu interface per GIC */
>   #define NR_GIC_CPU_IF 8
>   
> +/* GICv2 registers to be saved/restored on system suspend/resume */
> +struct gicv2_context {
> +    /* GICC context */
> +    uint32_t gicc_ctlr;
> +    uint32_t gicc_pmr;
> +    uint32_t gicc_bpr;
> +    /* GICD context */
> +    uint32_t gicd_ctlr;
> +    uint32_t *gicd_isenabler;
> +    uint32_t *gicd_isactiver;
> +    uint32_t *gicd_ipriorityr;
> +    uint32_t *gicd_itargetsr;
> +    uint32_t *gicd_icfgr;
> +};

It took me a long time to understand that you will only save the context for the 
CPU0 and Distributor.

I would prefer if we keep separate per-CPU context and common context. This 
would keep the logic very similar to the rest of the GIC drivers.

> +
> +static struct gicv2_context gicv2_context;
> +
> +static void gicv2_alloc_context(struct gicv2_context *gc);

Please don't do forward declaration. The code should be added in the correct place.

> +
>   static inline void writeb_gicd(uint8_t val, unsigned int offset)
>   {
>       writeb_relaxed(val, gicv2.map_dbase + offset);
> @@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
>   
>       spin_unlock(&gicv2.lock);
>   
> +    /* Allocate memory to be used for saving GIC context during the suspend */
> +    gicv2_alloc_context(&gicv2_context);
> +
>       return 0;
>   }
>   
> @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
>       BUG();
>   }
>   
> +static void gicv2_alloc_context(struct gicv2_context *gc)
> +{

Is it necessary to allocate them at boot? Can we make them static or allocate 
them when we suspend?

> +    uint32_t n = gicv2_info.nr_lines;
> +
> +    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> +    if ( !gc->gicd_isenabler )
> +        return;
> +
> +    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> +    if ( !gc->gicd_isactiver )
> +        goto free_gicd_isenabler;
> +
> +    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> +    if ( !gc->gicd_itargetsr )
> +        goto free_gicd_isactiver;
> +
> +    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> +    if ( !gc->gicd_ipriorityr )
> +        goto free_gicd_itargetsr;
> +
> +    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
> +    if ( gc->gicd_icfgr )
> +        return;
> +
> +    xfree(gc->gicd_ipriorityr);
> +
> +free_gicd_itargetsr:
> +    xfree(gc->gicd_itargetsr);
> +
> +free_gicd_isactiver:
> +    xfree(gc->gicd_isactiver);
> +
> +free_gicd_isenabler:
> +    xfree(gc->gicd_isenabler);
> +    gc->gicd_isenabler = NULL;
> +}
> +
> +static int gicv2_suspend(void)
> +{
> +    int i;
> +
> +    /* Save GICC configuration */
> +    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
> +    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
> +    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);

AFAICT, those values should always be the same. So I would just restore them 
with the hardcoded value.

This could likely be factored in a separate function that could be used in 
gicv2_cpu_init as well.

> +
> +    /* If gicv2_alloc_context() hasn't allocated memory, return */
> +    if ( !gicv2_context.gicd_isenabler )
> +        return -ENOMEM;
> +
> +    /* Save GICD configuration */
> +    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);

Same here.

> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> +        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
> +
> +    return 0;
> +}
> +
> +static void gicv2_resume(void)
> +{
> +    int i;
> +
> +    ASSERT(gicv2_context.gicd_isenabler);
> +    ASSERT(gicv2_context.gicd_isactiver);
> +    ASSERT(gicv2_context.gicd_ipriorityr);
> +    ASSERT(gicv2_context.gicd_itargetsr);
> +    ASSERT(gicv2_context.gicd_icfgr);
> +
> +    /* Disable CPU interface and distributor */
> +    writel_gicc(0, GICC_CTLR);
> +    writel_gicd(0, GICD_CTLR);
> +    isb();
> +
> +    /* Restore GICD configuration */
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);

I would prefer if there is only one for with 2 write. This would make more 
obvious why you need to clear all enabled bit first.

> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
> +        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);

Same here.

> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
> +        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
> +
> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
> +        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
> +
> +    /* Make sure all registers are restored and enable distributor */
> +    isb();
> +    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
> +
> +    /* Restore GIC CPU interface configuration */
> +    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
> +    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
> +    isb();
> +
> +    /* Enable GIC CPU interface */
> +    writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | GICC_CTL_EOI,
> +                GICC_CTLR);
> +    isb();
> +}
> +
>   const static struct gic_hw_operations gicv2_ops = {
>       .info                = &gicv2_info,
>       .init                = gicv2_init,
> @@ -1351,6 +1496,8 @@ const static struct gic_hw_operations gicv2_ops = {
>       .map_hwdom_extra_mappings = gicv2_map_hwdown_extra_mappings,
>       .iomem_deny_access   = gicv2_iomem_deny_access,
>       .do_LPI              = gicv2_do_LPI,
> +    .suspend             = gicv2_suspend,
> +    .resume              = gicv2_resume,
>   };
>   
>   /* Set up the GIC */
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index e524ad583d..6e98f43691 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -464,6 +464,33 @@ int gic_iomem_deny_access(const struct domain *d)
>       return gic_hw_ops->iomem_deny_access(d);
>   }
>   
> +int gic_suspend(void)
> +{
> +    /* Must be called by boot CPU#0 with interrupts disabled */
> +    ASSERT(!local_irq_is_enabled());
> +    ASSERT(!smp_processor_id());
> +
> +    if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
> +        return -ENOSYS;
> +
> +    gic_hw_ops->suspend();
> +
> +    return 0;
> +}
> +
> +void gic_resume(void)
> +{
> +    /*
> +     * Must be called by boot CPU#0 with interrupts disabled after gic_suspend
> +     * has returned successfully.
> +     */
> +    ASSERT(!local_irq_is_enabled());
> +    ASSERT(!smp_processor_id());
> +    ASSERT(gic_hw_ops->resume);
> +
> +    gic_hw_ops->resume();
> +}
> +
>   static int cpu_gic_callback(struct notifier_block *nfb,
>                               unsigned long action,
>                               void *hcpu)
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 22fa122e52..46066caac8 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -277,6 +277,10 @@ extern int gicv_setup(struct domain *d);
>   extern void gic_save_state(struct vcpu *v);
>   extern void gic_restore_state(struct vcpu *v);
>   
> +/* Suspend/resume */
> +extern int gic_suspend(void);
> +extern void gic_resume(void);
> +
>   /* SGI (AKA IPIs) */
>   enum gic_sgi {
>       GIC_SGI_EVENT_CHECK = 0,
> @@ -390,6 +394,10 @@ struct gic_hw_operations {
>       int (*iomem_deny_access)(const struct domain *d);
>       /* Handle LPIs, which require special handling */
>       void (*do_LPI)(unsigned int lpi);
> +    /* Save GIC configuration due to the system suspend */
> +    int (*suspend)(void);
> +    /* Restore GIC configuration due to the system resume */
> +    void (*resume)(void);

The comments on tops of suspend/resume suggest that this should be called 
save/restore.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On 11/13/2018 04:09 PM, Julien Grall wrote:
>
>
> On 13/11/2018 10:23, Julien Grall wrote:
>> Hi,
>>
>> On 12/11/2018 11:30, Mirela Simonovic wrote:
>>> +/*
>>> + * This function sets the context of current VCPU to the state 
>>> which is expected
>>> + * by the guest on resume. The expected VCPU state is:
>>> + * 1) pc to contain resume entry point (1st argument of PSCI 
>>> SYSTEM_SUSPEND)
>>> + * 2) r0/x0 to contain context ID (2nd argument of PSCI 
>>> SYSTEM_SUSPEND)
>>> + * 3) All other general purpose and system registers should have 
>>> reset values
>>> + *
>>> + * Note: this function has to return void because it has to always 
>>> succeed. In
>>> + * other words, this function is called from virtual PSCI 
>>> SYSTEM_SUSPEND
>>> + * implementation, which can return only a limited number of 
>>> possible errors,
>>> + * none of which could represent the fact that an error occurred 
>>> when preparing
>>> + * the domain for suspend.
>>> + * Consequently, dynamic memory allocation cannot be done within 
>>> this function,
>>> + * because if malloc fails the error has nowhere to propagate.
>>> + */
>>> +static void vcpu_suspend(register_t epoint, register_t cid)
>>> +{
>>> +    /* Static allocation because dynamic would need a non-void 
>>> return */
>>> +    static struct vcpu_guest_context ctxt;
>>> +    struct vcpu *v = current;
>>> +
>>> +    /* Make sure that VCPU guest regs are zeroied */
>>> +    memset(&ctxt, 0, sizeof(ctxt));
>>> +
>>> +    /* Set non-zero values to the registers prior to copying */
>>> +    ctxt.user_regs.pc64 = (u64)epoint;
>>> +
>>> +    if ( is_32bit_domain(current->domain) )
>>> +    {
>>> +        ctxt.user_regs.r0_usr = cid;
>>> +        ctxt.user_regs.cpsr = PSR_GUEST32_INIT;
>>
>> This is going to disable the MMU and Cache as requested by the PSCI 
>> spec. As the guest is not required to clean the cache when turning 
>> off the CPU/suspending, the data may not have reached the main memory.
>>
>> So do you need to perform cache maintenance to avoid stale information?
>
> Answering to myself, I have discussed about the cache with others Arm 
> folks today. SYSTEM_SUSPEND may exit early due to a pending event (BTW 
> you don't seem to handle it) and could jump to the specified entry 
> point address.
>

I believe it is handled - with the vcpu_block_unless_event_pending in 
domain_suspend

> In that case, the only guarantee is the data would not be lost, yet 
> can still be in the cache. This means that any data used before the 
> MMU & cache are enabled in the entry point should have been cleaned to 
> PoC and if you modify data make sure the cache is invalidated.
>
> Linux is doing that properly. Looking at Xen, we don't really properly 
> handle the cache in the boot path. So you may randomly crash.
>
> Cheers,
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Mirela,

On 14/11/2018 12:08, Mirela Simonovic wrote:
> 
> 
> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>> On 12/11/2018 19:56, Julien Grall wrote:
>>> Hi Andrew,
>>>
>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>         if ( is_idle_vcpu(p) )
>>>>>>>             return;
>>>>>>>
>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>> suspended */
>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>> +        return;
>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the state
>>>>>> to be saved on suspend.
>>>>>>
>>>>> We just need a flag to mark a domain as suspended, and I do believe
>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>> Let's come back on this.
>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
>>>> tree and you'll find several pieces of documentation, including the
>>>> description of what this shutdown code means.
>>>>
>>>> What you are introducing here is not a shutdown - it is a suspend with
>>>> the intent to resume executing later.  As such, it shouldn't use Xen's
>>>> shutdown infrastructure, which exists mainly to communicate with the
>>>> toolstack.
>>> Would domain pause/unpause be a better solution?
>> Actually yes - that sounds like a very neat solution.
> 
> I believe domain pause will not work here - a domain cannot pause itself, i.e. 
> current domain cannot be paused. This functionality seems to assume that a 
> domain is pausing another domain. Or I missed the point.

Yes domain pause/unpause will not work. However, you can introduce a boolean to 
tell you whether the domain was suspend.

I actually quite like how suspend work for x86 HVM. This is based on 
pause/unpause. Have a look at hvm_s3_{suspend/resume}.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Mirela Simonovic]

Thanks for the review


On 11/14/2018 01:41 PM, Julien Grall wrote:
> Hi,
>
> I am not entirely sure where to ask it, so I will do it here. From my 
> understanding of the PSCI spec, all the interrupts should have been 
> migrated away from turned off CPU. Where do you ensure that in the 
> suspend path?
>

disable_nonboot_cpus will lead to CPU_OFF PSCI to be called. That is 
orthogonal to suspend support in this series

>
> On 12/11/2018 11:30, Mirela Simonovic wrote:
>> System suspend may lead to a state where GIC would be powered down.
>> Therefore, Xen should save/restore the context of GIC on suspend/resume.
>> Note that the context consists of states of registers which are
>> controlled by the hypervisor. Other GIC registers which are accessible
>> by guests are saved/restored on context switch.
>> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
>> the GIC.
>>
>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>> ---
>>   xen/arch/arm/gic-v2.c     | 147 
>> ++++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/gic.c        |  27 +++++++++
>>   xen/include/asm-arm/gic.h |   8 +++
>>   3 files changed, 182 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>> index e7eb01f30a..bb52b64ecb 100644
>> --- a/xen/arch/arm/gic-v2.c
>> +++ b/xen/arch/arm/gic-v2.c
>> @@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
>>   /* Maximum cpu interface per GIC */
>>   #define NR_GIC_CPU_IF 8
>>   +/* GICv2 registers to be saved/restored on system suspend/resume */
>> +struct gicv2_context {
>> +    /* GICC context */
>> +    uint32_t gicc_ctlr;
>> +    uint32_t gicc_pmr;
>> +    uint32_t gicc_bpr;
>> +    /* GICD context */
>> +    uint32_t gicd_ctlr;
>> +    uint32_t *gicd_isenabler;
>> +    uint32_t *gicd_isactiver;
>> +    uint32_t *gicd_ipriorityr;
>> +    uint32_t *gicd_itargetsr;
>> +    uint32_t *gicd_icfgr;
>> +};
>
> It took me a long time to understand that you will only save the 
> context for the CPU0 and Distributor.

Yes, other physical CPUs are hot-unplugged at this point.

>
> I would prefer if we keep separate per-CPU context and common context. 
> This would keep the logic very similar to the rest of the GIC drivers.
>
>> +
>> +static struct gicv2_context gicv2_context;
>> +
>> +static void gicv2_alloc_context(struct gicv2_context *gc);
>
> Please don't do forward declaration. The code should be added in the 
> correct place.
>

Agreed

>> +
>>   static inline void writeb_gicd(uint8_t val, unsigned int offset)
>>   {
>>       writeb_relaxed(val, gicv2.map_dbase + offset);
>> @@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
>>         spin_unlock(&gicv2.lock);
>>   +    /* Allocate memory to be used for saving GIC context during 
>> the suspend */
>> +    gicv2_alloc_context(&gicv2_context);
>> +
>>       return 0;
>>   }
>>   @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
>>       BUG();
>>   }
>>   +static void gicv2_alloc_context(struct gicv2_context *gc)
>> +{
>
> Is it necessary to allocate them at boot? Can we make them static or 
> allocate them when we suspend?
>

We need to allocate dynamically because the size of allocated data 
depends on the number of irq lines, which is not known at the compile time.
Alternative is to allocate on suspend, but I believe it is better to do 
this when the system boots.

>> +    uint32_t n = gicv2_info.nr_lines;
>> +
>> +    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
>> +    if ( !gc->gicd_isenabler )
>> +        return;
>> +
>> +    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
>> +    if ( !gc->gicd_isactiver )
>> +        goto free_gicd_isenabler;
>> +
>> +    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
>> +    if ( !gc->gicd_itargetsr )
>> +        goto free_gicd_isactiver;
>> +
>> +    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
>> +    if ( !gc->gicd_ipriorityr )
>> +        goto free_gicd_itargetsr;
>> +
>> +    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
>> +    if ( gc->gicd_icfgr )
>> +        return;
>> +
>> +    xfree(gc->gicd_ipriorityr);
>> +
>> +free_gicd_itargetsr:
>> +    xfree(gc->gicd_itargetsr);
>> +
>> +free_gicd_isactiver:
>> +    xfree(gc->gicd_isactiver);
>> +
>> +free_gicd_isenabler:
>> +    xfree(gc->gicd_isenabler);
>> +    gc->gicd_isenabler = NULL;
>> +}
>> +
>> +static int gicv2_suspend(void)
>> +{
>> +    int i;
>> +
>> +    /* Save GICC configuration */
>> +    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
>> +    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
>> +    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
>
> AFAICT, those values should always be the same. So I would just 
> restore them with the hardcoded value.
>
> This could likely be factored in a separate function that could be 
> used in gicv2_cpu_init as well.
>
>> +
>> +    /* If gicv2_alloc_context() hasn't allocated memory, return */
>> +    if ( !gicv2_context.gicd_isenabler )
>> +        return -ENOMEM;
>> +
>> +    /* Save GICD configuration */
>> +    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
>
> Same here.
>
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER 
>> + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER 
>> + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        gicv2_context.gicd_ipriorityr[i] = 
>> readl_gicd(GICD_IPRIORITYR + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR 
>> + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
>> +        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
>> +
>> +    return 0;
>> +}
>> +
>> +static void gicv2_resume(void)
>> +{
>> +    int i;
>> +
>> +    ASSERT(gicv2_context.gicd_isenabler);
>> +    ASSERT(gicv2_context.gicd_isactiver);
>> +    ASSERT(gicv2_context.gicd_ipriorityr);
>> +    ASSERT(gicv2_context.gicd_itargetsr);
>> +    ASSERT(gicv2_context.gicd_icfgr);
>> +
>> +    /* Disable CPU interface and distributor */
>> +    writel_gicc(0, GICC_CTLR);
>> +    writel_gicd(0, GICD_CTLR);
>> +    isb();
>> +
>> +    /* Restore GICD configuration */
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER 
>> + i * 4);
>
> I would prefer if there is only one for with 2 write. This would make 
> more obvious why you need to clear all enabled bit first.
>
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>> +        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER 
>> + i * 4);
>
> Same here.
>
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        writel_gicd(gicv2_context.gicd_ipriorityr[i], 
>> GICD_IPRIORITYR + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>> +        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR 
>> + i * 4);
>> +
>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
>> +        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
>> +
>> +    /* Make sure all registers are restored and enable distributor */
>> +    isb();
>> +    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
>> +
>> +    /* Restore GIC CPU interface configuration */
>> +    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
>> +    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
>> +    isb();
>> +
>> +    /* Enable GIC CPU interface */
>> +    writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | 
>> GICC_CTL_EOI,
>> +                GICC_CTLR);
>> +    isb();
>> +}
>> +
>>   const static struct gic_hw_operations gicv2_ops = {
>>       .info                = &gicv2_info,
>>       .init                = gicv2_init,
>> @@ -1351,6 +1496,8 @@ const static struct gic_hw_operations gicv2_ops 
>> = {
>>       .map_hwdom_extra_mappings = gicv2_map_hwdown_extra_mappings,
>>       .iomem_deny_access   = gicv2_iomem_deny_access,
>>       .do_LPI              = gicv2_do_LPI,
>> +    .suspend             = gicv2_suspend,
>> +    .resume              = gicv2_resume,
>>   };
>>     /* Set up the GIC */
>> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
>> index e524ad583d..6e98f43691 100644
>> --- a/xen/arch/arm/gic.c
>> +++ b/xen/arch/arm/gic.c
>> @@ -464,6 +464,33 @@ int gic_iomem_deny_access(const struct domain *d)
>>       return gic_hw_ops->iomem_deny_access(d);
>>   }
>>   +int gic_suspend(void)
>> +{
>> +    /* Must be called by boot CPU#0 with interrupts disabled */
>> +    ASSERT(!local_irq_is_enabled());
>> +    ASSERT(!smp_processor_id());
>> +
>> +    if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
>> +        return -ENOSYS;
>> +
>> +    gic_hw_ops->suspend();
>> +
>> +    return 0;
>> +}
>> +
>> +void gic_resume(void)
>> +{
>> +    /*
>> +     * Must be called by boot CPU#0 with interrupts disabled after 
>> gic_suspend
>> +     * has returned successfully.
>> +     */
>> +    ASSERT(!local_irq_is_enabled());
>> +    ASSERT(!smp_processor_id());
>> +    ASSERT(gic_hw_ops->resume);
>> +
>> +    gic_hw_ops->resume();
>> +}
>> +
>>   static int cpu_gic_callback(struct notifier_block *nfb,
>>                               unsigned long action,
>>                               void *hcpu)
>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>> index 22fa122e52..46066caac8 100644
>> --- a/xen/include/asm-arm/gic.h
>> +++ b/xen/include/asm-arm/gic.h
>> @@ -277,6 +277,10 @@ extern int gicv_setup(struct domain *d);
>>   extern void gic_save_state(struct vcpu *v);
>>   extern void gic_restore_state(struct vcpu *v);
>>   +/* Suspend/resume */
>> +extern int gic_suspend(void);
>> +extern void gic_resume(void);
>> +
>>   /* SGI (AKA IPIs) */
>>   enum gic_sgi {
>>       GIC_SGI_EVENT_CHECK = 0,
>> @@ -390,6 +394,10 @@ struct gic_hw_operations {
>>       int (*iomem_deny_access)(const struct domain *d);
>>       /* Handle LPIs, which require special handling */
>>       void (*do_LPI)(unsigned int lpi);
>> +    /* Save GIC configuration due to the system suspend */
>> +    int (*suspend)(void);
>> +    /* Restore GIC configuration due to the system resume */
>> +    void (*resume)(void);
>
> The comments on tops of suspend/resume suggest that this should be 
> called save/restore.

I thought that too, but save/restore already exist and are used for 
other purposes.

>
> Cheers,
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Mirela Simonovic]

On 11/14/2018 11:52 AM, Julien Grall wrote:
> Hi Mirela,
>
> On 12/11/2018 11:30, Mirela Simonovic wrote:
>> Non-boot CPUs have to be disabled on suspend and enabled on resume
>> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
>> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
>> platform capabilities, this may lead to the physical powering down of
>> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
>> each non-boot CPU).
>>
>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>> ---
>>   xen/arch/arm/suspend.c | 15 ++++++++++++++-
>>   1 file changed, 14 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>> index 575afd5eb8..dae1b1f7d6 100644
>> --- a/xen/arch/arm/suspend.c
>> +++ b/xen/arch/arm/suspend.c
>> @@ -1,4 +1,5 @@
>>   #include <xen/sched.h>
>> +#include <xen/cpu.h>
>>   #include <asm/cpufeature.h>
>>   #include <asm/event.h>
>>   #include <asm/psci.h>
>> @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, 
>> register_t cid)
>>   /* Xen suspend. Note: data is not used (suspend is the suspend to 
>> RAM) */
>>   static long system_suspend(void *data)
>>   {
>> +    int status;
>> +
>>       BUG_ON(system_state != SYS_STATE_active);
>>         system_state = SYS_STATE_suspend;
>>       freeze_domains();
>>   +    status = disable_nonboot_cpus();
>> +    if ( status )
>> +    {
>> +        system_state = SYS_STATE_resume;
>> +        goto resume_nonboot_cpus;
>> +    }
>> +
>>       system_state = SYS_STATE_resume;
>>   +resume_nonboot_cpus:
>> +    enable_nonboot_cpus();
>>       thaw_domains();
>>       system_state = SYS_STATE_active;
>> +    dsb(sy);
>
> Why do you need a dsb(sy) here?
>

Updated value of system_state variable needs to be visible to other CPUs 
before we move on

>>   -    return -ENOSYS;
>
> Why do you return -ENOSYS until this patch? Should not have it be 0?
>

To distinguish that Xen suspend wasn't supported until we at least do 
something useful in suspend procedure. This is not so important, we can 
return 0 but needs to be fixed in previous patches.

>> +    return status;
>>   }
>>     int32_t domain_suspend(register_t epoint, register_t cid)
>>
>
> Cheers,
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

On 14/11/2018 12:35, Mirela Simonovic wrote:
> Hi Julien,

Hi,

> 
> On 11/14/2018 11:45 AM, Julien Grall wrote:
>> Hi,
>>
>> On 13/11/2018 20:39, Stefano Stabellini wrote:
>>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>>>> However, what is the issue with saving all the registers here?
>>>>>>
>>>>>
>>>>> We need to save arguments that are provided by a guest with system
>>>>> suspend PSCI call. These arguments are the entry point that needs to
>>>>> be saved in program counter and context ID that needs to be saved in
>>>>> x0/r0. We don't have these arguments here. Context switch happens
>>>>> after processing the system suspend PSCI call, so it's too late.
>>>>
>>>> It does not feel right to modify ctxt_switch{from,to} for suspend/resume. If
>>>> you want to reset the vCPU state before blocking the vCPU, then you should
>>>> instead
>>>
> 
> I think it's not clear what problem are we discussing here, at least it's not to 
> me. So I'll make an assumption, and please correct me if I'm wrong.
> In the patches we submitted, the VCPU context is not reset in 
> ctxt_switch{from,to}. My understanding is that you suggested/asked to reset the 
> VCPU context when switch happens, and I explained why is that not possible - at 
> least not without additional code changes, that may not be so small. I agree 
> with Andrew's comment in this perspective - reset of VCPU should not (and right 
> now cannot) be done when the context is switched.

I didn't ask to reset the vCPU context in the switch. Instead we should make 
sure the vCPU context is synced to memory before modifying it.

It seems that solution works on x86 using domain_pause (see 
hvm_s3_{resume,suspend}). So I am not sure why it cannot be use on Arm. Note 
that it may require more work.

> 
>>> You missed the end of the suggestion here
>>
>> Whoops. I meant that instead you should save the context of the vCPU in 
>> advance or reset the vCPU using the system registers directly.
>>
>> But my preference is to reset the vCPU when you receive the wake-up interrupt.
>>
> 
> Without you presenting more details how would that work I cannot really provide 
> any comment, nor say that your preference could work or be better compared to 
> what is in this series. Honestly, I don't understand what exactly you're 
> proposing, because more things needs to be think-through beyond the place to put 
> a code.
> We submitted a code that works, which is very elegant and nice in my opinion 
> (fair to say we may not share opinions here), and does not require lots of code 
> changes. So there's the reference.
> Could you please clarify why do you think the proposed solution is not good?

The context switch is about saving/restore the context from the hardware. We can 
decide to optimize it in the suspend case (though it might be premature), but it 
is clearly the wrong place to decide to resume a domain.

If saving the context happens to late, then we should look at making sure it 
will happen earlier on (see my comment above).

> And why do you think that what you're proposing is better? Lets be more clear 
> here - how exactly you propose to implement that?

The same way as hvm_s3_{suspend/resume} works on x86.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

On 13/11/2018 20:39, Stefano Stabellini wrote:
> On Tue, 13 Nov 2018, Julien Grall wrote:
> 
> If we mark the domU simply as "paused" it will be difficult to implement
> correctly "xl restore" / "xl trigger s3resume". We should be able to
> distinguish a domain which is simply not running or paused (as in "xl
> pause") from one that has been put to sleep.  Otherwise we won't be able
> to use "xl pause" in its original sense anymore. "xl pause" will become
> effectively "xl trigger sleep" on ARM. Which is something to consider
> but I don't think that is what we want.

I didn't suggested to only use those 2 helpers. But you can build suspend/resume 
on top of it. AFAICT, this is how s3 suspend/resume works on x86 HVM.

> 
> The most similar feature is ACPI S3 in x86-land but the current calls
> are so ACPI/x86 specific that don't make much sense on a ACPI-less ARM
> implementation. If I am not mistaken, on x86 there is a special struct
> domain flag for this: d->arch.hvm.is_s3_suspended.
> 
> 
> Let's leave aside the "which commands should we use" discussion for now
> because it doesn't related to this patch series. It seems to me that the
> best option is to introduce a new ARM specific struct domain flag,
> something akin to d->arch.hvm.is_s3_suspended but ARM PSCI specific.

See above.

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Julien Grall]

Hi Mirela,

On 14/11/2018 13:00, Mirela Simonovic wrote:
> 
> 
> On 11/14/2018 11:52 AM, Julien Grall wrote:
>> Hi Mirela,
>>
>> On 12/11/2018 11:30, Mirela Simonovic wrote:
>>> Non-boot CPUs have to be disabled on suspend and enabled on resume
>>> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
>>> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
>>> platform capabilities, this may lead to the physical powering down of
>>> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
>>> each non-boot CPU).
>>>
>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>> ---
>>>   xen/arch/arm/suspend.c | 15 ++++++++++++++-
>>>   1 file changed, 14 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>>> index 575afd5eb8..dae1b1f7d6 100644
>>> --- a/xen/arch/arm/suspend.c
>>> +++ b/xen/arch/arm/suspend.c
>>> @@ -1,4 +1,5 @@
>>>   #include <xen/sched.h>
>>> +#include <xen/cpu.h>
>>>   #include <asm/cpufeature.h>
>>>   #include <asm/event.h>
>>>   #include <asm/psci.h>
>>> @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, register_t 
>>> cid)
>>>   /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>>>   static long system_suspend(void *data)
>>>   {
>>> +    int status;
>>> +
>>>       BUG_ON(system_state != SYS_STATE_active);
>>>         system_state = SYS_STATE_suspend;
>>>       freeze_domains();
>>>   +    status = disable_nonboot_cpus();
>>> +    if ( status )
>>> +    {
>>> +        system_state = SYS_STATE_resume;
>>> +        goto resume_nonboot_cpus;
>>> +    }
>>> +
>>>       system_state = SYS_STATE_resume;
>>>   +resume_nonboot_cpus:
>>> +    enable_nonboot_cpus();
>>>       thaw_domains();
>>>       system_state = SYS_STATE_active;
>>> +    dsb(sy);
>>
>> Why do you need a dsb(sy) here?
>>
> 
> Updated value of system_state variable needs to be visible to other CPUs before 
> we move on

We tend to write the reason on top of barrier why they are necessary. But I am 
still unsure to understand why this is important. What would happen if move on 
without it?

> 
>>>   -    return -ENOSYS;
>>
>> Why do you return -ENOSYS until this patch? Should not have it be 0?
>>
> 
> To distinguish that Xen suspend wasn't supported until we at least do something 
> useful in suspend procedure. This is not so important, we can return 0 but needs 
> to be fixed in previous patches.

If you return 0 before hand you can more easily bisect this series and know 
where it suspend/resume breaks.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Julien Grall]

Hi Mirela,

On 14/11/2018 12:52, Mirela Simonovic wrote:
> Thanks for the review
> 
> 
> On 11/14/2018 01:41 PM, Julien Grall wrote:
>> Hi,
>>
>> I am not entirely sure where to ask it, so I will do it here. From my 
>> understanding of the PSCI spec, all the interrupts should have been migrated 
>> away from turned off CPU. Where do you ensure that in the suspend path?
>>
> 
> disable_nonboot_cpus will lead to CPU_OFF PSCI to be called. That is orthogonal 
> to suspend support in this series

AFAICT, none of the PSCI CPU_OFF code will migrate the interrupt. I already 
pointed that when you send your series to fix the CPU_OFF. But this still does 
not seem to be addressed. So my question is still open.

> 
>>
>> On 12/11/2018 11:30, Mirela Simonovic wrote:
>>> System suspend may lead to a state where GIC would be powered down.
>>> Therefore, Xen should save/restore the context of GIC on suspend/resume.
>>> Note that the context consists of states of registers which are
>>> controlled by the hypervisor. Other GIC registers which are accessible
>>> by guests are saved/restored on context switch.
>>> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
>>> the GIC.
>>>
>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>> ---
>>>   xen/arch/arm/gic-v2.c     | 147 ++++++++++++++++++++++++++++++++++++++++++++++
>>>   xen/arch/arm/gic.c        |  27 +++++++++
>>>   xen/include/asm-arm/gic.h |   8 +++
>>>   3 files changed, 182 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>>> index e7eb01f30a..bb52b64ecb 100644
>>> --- a/xen/arch/arm/gic-v2.c
>>> +++ b/xen/arch/arm/gic-v2.c
>>> @@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
>>>   /* Maximum cpu interface per GIC */
>>>   #define NR_GIC_CPU_IF 8
>>>   +/* GICv2 registers to be saved/restored on system suspend/resume */
>>> +struct gicv2_context {
>>> +    /* GICC context */
>>> +    uint32_t gicc_ctlr;
>>> +    uint32_t gicc_pmr;
>>> +    uint32_t gicc_bpr;
>>> +    /* GICD context */
>>> +    uint32_t gicd_ctlr;
>>> +    uint32_t *gicd_isenabler;
>>> +    uint32_t *gicd_isactiver;
>>> +    uint32_t *gicd_ipriorityr;
>>> +    uint32_t *gicd_itargetsr;
>>> +    uint32_t *gicd_icfgr;
>>> +};
>>
>> It took me a long time to understand that you will only save the context for 
>> the CPU0 and Distributor.
> 
> Yes, other physical CPUs are hot-unplugged at this point.
> 
>>
>> I would prefer if we keep separate per-CPU context and common context. This 
>> would keep the logic very similar to the rest of the GIC drivers.
>>
>>> +
>>> +static struct gicv2_context gicv2_context;
>>> +
>>> +static void gicv2_alloc_context(struct gicv2_context *gc);
>>
>> Please don't do forward declaration. The code should be added in the correct 
>> place.
>>
> 
> Agreed
> 
>>> +
>>>   static inline void writeb_gicd(uint8_t val, unsigned int offset)
>>>   {
>>>       writeb_relaxed(val, gicv2.map_dbase + offset);
>>> @@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
>>>         spin_unlock(&gicv2.lock);
>>>   +    /* Allocate memory to be used for saving GIC context during the 
>>> suspend */
>>> +    gicv2_alloc_context(&gicv2_context);
>>> +
>>>       return 0;
>>>   }
>>>   @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
>>>       BUG();
>>>   }
>>>   +static void gicv2_alloc_context(struct gicv2_context *gc)
>>> +{
>>
>> Is it necessary to allocate them at boot? Can we make them static or allocate 
>> them when we suspend?
>>
> 
> We need to allocate dynamically because the size of allocated data depends on 
> the number of irq lines, which is not known at the compile time.

Well you know the upper bound. Why can't you use the upper bound?

> Alternative is to allocate on suspend, but I believe it is better to do this 
> when the system boots.

Why is it better?

> 
>>> +    uint32_t n = gicv2_info.nr_lines;
>>> +
>>> +    gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
>>> +    if ( !gc->gicd_isenabler )
>>> +        return;
>>> +
>>> +    gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
>>> +    if ( !gc->gicd_isactiver )
>>> +        goto free_gicd_isenabler;
>>> +
>>> +    gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
>>> +    if ( !gc->gicd_itargetsr )
>>> +        goto free_gicd_isactiver;
>>> +
>>> +    gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
>>> +    if ( !gc->gicd_ipriorityr )
>>> +        goto free_gicd_itargetsr;
>>> +
>>> +    gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
>>> +    if ( gc->gicd_icfgr )
>>> +        return;
>>> +
>>> +    xfree(gc->gicd_ipriorityr);
>>> +
>>> +free_gicd_itargetsr:
>>> +    xfree(gc->gicd_itargetsr);
>>> +
>>> +free_gicd_isactiver:
>>> +    xfree(gc->gicd_isactiver);
>>> +
>>> +free_gicd_isenabler:
>>> +    xfree(gc->gicd_isenabler);
>>> +    gc->gicd_isenabler = NULL;
>>> +}
>>> +
>>> +static int gicv2_suspend(void)
>>> +{
>>> +    int i;
>>> +
>>> +    /* Save GICC configuration */
>>> +    gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
>>> +    gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
>>> +    gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
>>
>> AFAICT, those values should always be the same. So I would just restore them 
>> with the hardcoded value.
>>
>> This could likely be factored in a separate function that could be used in 
>> gicv2_cpu_init as well.
>>
>>> +
>>> +    /* If gicv2_alloc_context() hasn't allocated memory, return */
>>> +    if ( !gicv2_context.gicd_isenabler )
>>> +        return -ENOMEM;
>>> +
>>> +    /* Save GICD configuration */
>>> +    gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
>>
>> Same here.
>>
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>>> +        gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>>> +        gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>>> +        gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>>> +        gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
>>> +        gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void gicv2_resume(void)
>>> +{
>>> +    int i;
>>> +
>>> +    ASSERT(gicv2_context.gicd_isenabler);
>>> +    ASSERT(gicv2_context.gicd_isactiver);
>>> +    ASSERT(gicv2_context.gicd_ipriorityr);
>>> +    ASSERT(gicv2_context.gicd_itargetsr);
>>> +    ASSERT(gicv2_context.gicd_icfgr);
>>> +
>>> +    /* Disable CPU interface and distributor */
>>> +    writel_gicc(0, GICC_CTLR);
>>> +    writel_gicd(0, GICD_CTLR);
>>> +    isb();
>>> +
>>> +    /* Restore GICD configuration */
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>>> +        writel_gicd(0xffffffff, GICD_ICENABLER + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>>> +        writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
>>
>> I would prefer if there is only one for with 2 write. This would make more 
>> obvious why you need to clear all enabled bit first.
>>
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>>> +        writel_gicd(0xffffffff, GICD_ICACTIVER + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
>>> +        writel_gicd(gicv2_context.gicd_isactiver[i], GICD_ISACTIVER + i * 4);
>>
>> Same here.
>>
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>>> +        writel_gicd(gicv2_context.gicd_ipriorityr[i], GICD_IPRIORITYR + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
>>> +        writel_gicd(gicv2_context.gicd_itargetsr[i], GICD_ITARGETSR + i * 4);
>>> +
>>> +    for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
>>> +        writel_gicd(gicv2_context.gicd_icfgr[i], GICD_ICFGR + i * 4);
>>> +
>>> +    /* Make sure all registers are restored and enable distributor */
>>> +    isb();
>>> +    writel_gicd(gicv2_context.gicd_ctlr | GICD_CTL_ENABLE, GICD_CTLR);
>>> +
>>> +    /* Restore GIC CPU interface configuration */
>>> +    writel_gicc(gicv2_context.gicc_pmr, GICC_PMR);
>>> +    writel_gicc(gicv2_context.gicc_bpr, GICC_BPR);
>>> +    isb();
>>> +
>>> +    /* Enable GIC CPU interface */
>>> +    writel_gicc(gicv2_context.gicc_ctlr | GICC_CTL_ENABLE | GICC_CTL_EOI,
>>> +                GICC_CTLR);
>>> +    isb();
>>> +}
>>> +
>>>   const static struct gic_hw_operations gicv2_ops = {
>>>       .info                = &gicv2_info,
>>>       .init                = gicv2_init,
>>> @@ -1351,6 +1496,8 @@ const static struct gic_hw_operations gicv2_ops = {
>>>       .map_hwdom_extra_mappings = gicv2_map_hwdown_extra_mappings,
>>>       .iomem_deny_access   = gicv2_iomem_deny_access,
>>>       .do_LPI              = gicv2_do_LPI,
>>> +    .suspend             = gicv2_suspend,
>>> +    .resume              = gicv2_resume,
>>>   };
>>>     /* Set up the GIC */
>>> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
>>> index e524ad583d..6e98f43691 100644
>>> --- a/xen/arch/arm/gic.c
>>> +++ b/xen/arch/arm/gic.c
>>> @@ -464,6 +464,33 @@ int gic_iomem_deny_access(const struct domain *d)
>>>       return gic_hw_ops->iomem_deny_access(d);
>>>   }
>>>   +int gic_suspend(void)
>>> +{
>>> +    /* Must be called by boot CPU#0 with interrupts disabled */
>>> +    ASSERT(!local_irq_is_enabled());
>>> +    ASSERT(!smp_processor_id());
>>> +
>>> +    if ( !gic_hw_ops->suspend || !gic_hw_ops->resume )
>>> +        return -ENOSYS;
>>> +
>>> +    gic_hw_ops->suspend();
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +void gic_resume(void)
>>> +{
>>> +    /*
>>> +     * Must be called by boot CPU#0 with interrupts disabled after gic_suspend
>>> +     * has returned successfully.
>>> +     */
>>> +    ASSERT(!local_irq_is_enabled());
>>> +    ASSERT(!smp_processor_id());
>>> +    ASSERT(gic_hw_ops->resume);
>>> +
>>> +    gic_hw_ops->resume();
>>> +}
>>> +
>>>   static int cpu_gic_callback(struct notifier_block *nfb,
>>>                               unsigned long action,
>>>                               void *hcpu)
>>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>>> index 22fa122e52..46066caac8 100644
>>> --- a/xen/include/asm-arm/gic.h
>>> +++ b/xen/include/asm-arm/gic.h
>>> @@ -277,6 +277,10 @@ extern int gicv_setup(struct domain *d);
>>>   extern void gic_save_state(struct vcpu *v);
>>>   extern void gic_restore_state(struct vcpu *v);
>>>   +/* Suspend/resume */
>>> +extern int gic_suspend(void);
>>> +extern void gic_resume(void);
>>> +
>>>   /* SGI (AKA IPIs) */
>>>   enum gic_sgi {
>>>       GIC_SGI_EVENT_CHECK = 0,
>>> @@ -390,6 +394,10 @@ struct gic_hw_operations {
>>>       int (*iomem_deny_access)(const struct domain *d);
>>>       /* Handle LPIs, which require special handling */
>>>       void (*do_LPI)(unsigned int lpi);
>>> +    /* Save GIC configuration due to the system suspend */
>>> +    int (*suspend)(void);
>>> +    /* Restore GIC configuration due to the system resume */
>>> +    void (*resume)(void);
>>
>> The comments on tops of suspend/resume suggest that this should be called 
>> save/restore.
> 
> I thought that too, but save/restore already exist and are used for other purposes.

How about renaming the other functions to save_vcpu_state/restore_vcpu_state?

Cheers,

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

On 14/11/2018 13:05, Julien Grall wrote:
> 
> 
> On 14/11/2018 12:35, Mirela Simonovic wrote:
>> Hi Julien,
> 
> Hi,
> 
>>
>> On 11/14/2018 11:45 AM, Julien Grall wrote:
>>> Hi,
>>>
>>> On 13/11/2018 20:39, Stefano Stabellini wrote:
>>>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>>>>> However, what is the issue with saving all the registers here?
>>>>>>>
>>>>>>
>>>>>> We need to save arguments that are provided by a guest with system
>>>>>> suspend PSCI call. These arguments are the entry point that needs to
>>>>>> be saved in program counter and context ID that needs to be saved in
>>>>>> x0/r0. We don't have these arguments here. Context switch happens
>>>>>> after processing the system suspend PSCI call, so it's too late.
>>>>>
>>>>> It does not feel right to modify ctxt_switch{from,to} for suspend/resume. If
>>>>> you want to reset the vCPU state before blocking the vCPU, then you should
>>>>> instead
>>>>
>>
>> I think it's not clear what problem are we discussing here, at least it's not 
>> to me. So I'll make an assumption, and please correct me if I'm wrong.
>> In the patches we submitted, the VCPU context is not reset in 
>> ctxt_switch{from,to}. My understanding is that you suggested/asked to reset 
>> the VCPU context when switch happens, and I explained why is that not possible 
>> - at least not without additional code changes, that may not be so small. I 
>> agree with Andrew's comment in this perspective - reset of VCPU should not 
>> (and right now cannot) be done when the context is switched.
> 
> I didn't ask to reset the vCPU context in the switch. Instead we should make 
> sure the vCPU context is synced to memory before modifying it.
> 
> It seems that solution works on x86 using domain_pause (see 
> hvm_s3_{resume,suspend}). So I am not sure why it cannot be use on Arm. Note 
> that it may require more work.
> 
>>
>>>> You missed the end of the suggestion here
>>>
>>> Whoops. I meant that instead you should save the context of the vCPU in 
>>> advance or reset the vCPU using the system registers directly.
>>>
>>> But my preference is to reset the vCPU when you receive the wake-up interrupt.
>>>
>>
>> Without you presenting more details how would that work I cannot really 
>> provide any comment, nor say that your preference could work or be better 
>> compared to what is in this series. Honestly, I don't understand what exactly 
>> you're proposing, because more things needs to be think-through beyond the 
>> place to put a code.
>> We submitted a code that works, which is very elegant and nice in my opinion 
>> (fair to say we may not share opinions here), and does not require lots of 
>> code changes. So there's the reference.
>> Could you please clarify why do you think the proposed solution is not good?
> 
> The context switch is about saving/restore the context from the hardware. We can 
> decide to optimize it in the suspend case (though it might be premature), but it 
> is clearly the wrong place to decide to resume a domain.

Actually, I just found a good example of why I think it is wrong and broken. You 
rely on a context switch to always happen after suspending and on resuming the 
guest. There are path where context/switch will not happen. An example is if you 
have interrupt pending, you may return to the guest directly if the scheduler 
does not have anything else to schedule.

The problem is the variable is_shut_down and shutdown_code are only be reset on 
restoring the vCPU. This means the next context switch will skip the saving part 
and result to vCPU corruption.

In general, you cannot rely on the context/switch code as it may or may not 
happen (that's up to the scheduler).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

Hi,

On 12/11/2018 11:30, Mirela Simonovic wrote:
> When Dom0 finalizes its suspend procedure the suspend of Xen is
> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> its boot core (VCPU#0), which could be mapped to any physical CPU,
> i.e. the system_suspend() function could be executed by any physical
> CPU. Since Xen suspend procedure has to be run by the boot CPU
> (non-boot CPUs will be disabled at some point in suspend procedure),
> system_suspend() execution has to continue on CPU#0.
> 
> When the system_suspend() returns 0, it means that the system was
> suspended and it is coming out of the resume procedure. Regardless
> of the system_suspend() return value, after this function returns
> Xen is fully functional, and its state, including all devices and data
> structures, matches the state prior to calling system_suspend().
> The status is returned by system_suspend() for debugging/logging
> purposes and function prototype compatibility.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
>   1 file changed, 34 insertions(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index f2338e41db..21b45f8248 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>       _arch_set_info_guest(v, &ctxt);
>   }
>   
> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> +static long system_suspend(void *data)
> +{
> +    BUG_ON(system_state != SYS_STATE_active);
> +
> +    return -ENOSYS;
> +}
> +
>   int32_t domain_suspend(register_t epoint, register_t cid)
>   {
>       struct vcpu *v;
>       struct domain *d = current->domain;
>       bool is_thumb = epoint & 1;
> +    int status;
>   
>       dprintk(XENLOG_DEBUG,
>               "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
>        */
>       vcpu_block_unless_event_pending(current);
>   
> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
> +    if ( is_hardware_domain(d) )
> +    {
> +        /*
> +         * system_suspend should be called when Dom0 finalizes the suspend
> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
> +         * be mapped to any PCPU (this function could be executed by any PCPU).
> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
> +         * PCPUs will be disabled during the suspend).
> +         */
> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);

Based on my comment in patch #2, I don't think this will do the correct thing on 
Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
do_vpsci_0_2_call.

As upper layer may modify the vCPU context, I think it would be best to run 
system_suspend in a tasklet bound to CPU#0.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 3:49 PM Julien Grall <julien.grall@arm.com> wrote:
>
>
>
> On 14/11/2018 13:05, Julien Grall wrote:
> >
> >
> > On 14/11/2018 12:35, Mirela Simonovic wrote:
> >> Hi Julien,
> >
> > Hi,
> >
> >>
> >> On 11/14/2018 11:45 AM, Julien Grall wrote:
> >>> Hi,
> >>>
> >>> On 13/11/2018 20:39, Stefano Stabellini wrote:
> >>>> On Mon, 12 Nov 2018, Julien Grall wrote:
> >>>>>>> However, what is the issue with saving all the registers here?
> >>>>>>>
> >>>>>>
> >>>>>> We need to save arguments that are provided by a guest with system
> >>>>>> suspend PSCI call. These arguments are the entry point that needs to
> >>>>>> be saved in program counter and context ID that needs to be saved in
> >>>>>> x0/r0. We don't have these arguments here. Context switch happens
> >>>>>> after processing the system suspend PSCI call, so it's too late.
> >>>>>
> >>>>> It does not feel right to modify ctxt_switch{from,to} for suspend/resume. If
> >>>>> you want to reset the vCPU state before blocking the vCPU, then you should
> >>>>> instead
> >>>>
> >>
> >> I think it's not clear what problem are we discussing here, at least it's not
> >> to me. So I'll make an assumption, and please correct me if I'm wrong.
> >> In the patches we submitted, the VCPU context is not reset in
> >> ctxt_switch{from,to}. My understanding is that you suggested/asked to reset
> >> the VCPU context when switch happens, and I explained why is that not possible
> >> - at least not without additional code changes, that may not be so small. I
> >> agree with Andrew's comment in this perspective - reset of VCPU should not
> >> (and right now cannot) be done when the context is switched.
> >
> > I didn't ask to reset the vCPU context in the switch. Instead we should make
> > sure the vCPU context is synced to memory before modifying it.
> >
> > It seems that solution works on x86 using domain_pause (see
> > hvm_s3_{resume,suspend}). So I am not sure why it cannot be use on Arm. Note
> > that it may require more work.
> >
> >>
> >>>> You missed the end of the suggestion here
> >>>
> >>> Whoops. I meant that instead you should save the context of the vCPU in
> >>> advance or reset the vCPU using the system registers directly.
> >>>
> >>> But my preference is to reset the vCPU when you receive the wake-up interrupt.
> >>>
> >>
> >> Without you presenting more details how would that work I cannot really
> >> provide any comment, nor say that your preference could work or be better
> >> compared to what is in this series. Honestly, I don't understand what exactly
> >> you're proposing, because more things needs to be think-through beyond the
> >> place to put a code.
> >> We submitted a code that works, which is very elegant and nice in my opinion
> >> (fair to say we may not share opinions here), and does not require lots of
> >> code changes. So there's the reference.
> >> Could you please clarify why do you think the proposed solution is not good?
> >
> > The context switch is about saving/restore the context from the hardware. We can
> > decide to optimize it in the suspend case (though it might be premature), but it
> > is clearly the wrong place to decide to resume a domain.
>
> Actually, I just found a good example of why I think it is wrong and broken. You
> rely on a context switch to always happen after suspending and on resuming the
> guest. There are path where context/switch will not happen. An example is if you
> have interrupt pending, you may return to the guest directly if the scheduler
> does not have anything else to schedule.
>

Can we check whether the vcpu blocked, right? Let me be more specific
- after calling vcpu_block_unless_event_pending we can check whether
the vcpu is indeed blocked?

> The problem is the variable is_shut_down and shutdown_code are only be reset on
> restoring the vCPU. This means the next context switch will skip the saving part
> and result to vCPU corruption.
>
> In general, you cannot rely on the context/switch code as it may or may not
> happen (that's up to the scheduler).
>



> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On Wed, Nov 14, 2018 at 4:36 PM Mirela Simonovic
<mirela.simonovic@aggios.com> wrote:
>
> Hi Julien,
>
> On Wed, Nov 14, 2018 at 3:49 PM Julien Grall <julien.grall@arm.com> wrote:
> >
> >
> >
> > On 14/11/2018 13:05, Julien Grall wrote:
> > >
> > >
> > > On 14/11/2018 12:35, Mirela Simonovic wrote:
> > >> Hi Julien,
> > >
> > > Hi,
> > >
> > >>
> > >> On 11/14/2018 11:45 AM, Julien Grall wrote:
> > >>> Hi,
> > >>>
> > >>> On 13/11/2018 20:39, Stefano Stabellini wrote:
> > >>>> On Mon, 12 Nov 2018, Julien Grall wrote:
> > >>>>>>> However, what is the issue with saving all the registers here?
> > >>>>>>>
> > >>>>>>
> > >>>>>> We need to save arguments that are provided by a guest with system
> > >>>>>> suspend PSCI call. These arguments are the entry point that needs to
> > >>>>>> be saved in program counter and context ID that needs to be saved in
> > >>>>>> x0/r0. We don't have these arguments here. Context switch happens
> > >>>>>> after processing the system suspend PSCI call, so it's too late.
> > >>>>>
> > >>>>> It does not feel right to modify ctxt_switch{from,to} for suspend/resume. If
> > >>>>> you want to reset the vCPU state before blocking the vCPU, then you should
> > >>>>> instead
> > >>>>
> > >>
> > >> I think it's not clear what problem are we discussing here, at least it's not
> > >> to me. So I'll make an assumption, and please correct me if I'm wrong.
> > >> In the patches we submitted, the VCPU context is not reset in
> > >> ctxt_switch{from,to}. My understanding is that you suggested/asked to reset
> > >> the VCPU context when switch happens, and I explained why is that not possible
> > >> - at least not without additional code changes, that may not be so small. I
> > >> agree with Andrew's comment in this perspective - reset of VCPU should not
> > >> (and right now cannot) be done when the context is switched.
> > >
> > > I didn't ask to reset the vCPU context in the switch. Instead we should make
> > > sure the vCPU context is synced to memory before modifying it.
> > >
> > > It seems that solution works on x86 using domain_pause (see
> > > hvm_s3_{resume,suspend}). So I am not sure why it cannot be use on Arm. Note
> > > that it may require more work.
> > >
> > >>
> > >>>> You missed the end of the suggestion here
> > >>>
> > >>> Whoops. I meant that instead you should save the context of the vCPU in
> > >>> advance or reset the vCPU using the system registers directly.
> > >>>
> > >>> But my preference is to reset the vCPU when you receive the wake-up interrupt.
> > >>>
> > >>
> > >> Without you presenting more details how would that work I cannot really
> > >> provide any comment, nor say that your preference could work or be better
> > >> compared to what is in this series. Honestly, I don't understand what exactly
> > >> you're proposing, because more things needs to be think-through beyond the
> > >> place to put a code.
> > >> We submitted a code that works, which is very elegant and nice in my opinion
> > >> (fair to say we may not share opinions here), and does not require lots of
> > >> code changes. So there's the reference.
> > >> Could you please clarify why do you think the proposed solution is not good?
> > >
> > > The context switch is about saving/restore the context from the hardware. We can
> > > decide to optimize it in the suspend case (though it might be premature), but it
> > > is clearly the wrong place to decide to resume a domain.
> >
> > Actually, I just found a good example of why I think it is wrong and broken. You
> > rely on a context switch to always happen after suspending and on resuming the
> > guest. There are path where context/switch will not happen. An example is if you
> > have interrupt pending, you may return to the guest directly if the scheduler
> > does not have anything else to schedule.
> >
>
> Can we check whether the vcpu blocked, right? Let me be more specific - after calling vcpu_block_unless_event_pending we can check whether the vcpu is indeed blocked?
>
> > The problem is the variable is_shut_down and shutdown_code are only be reset on
> > restoring the vCPU. This means the next context switch will skip the saving part
> > and result to vCPU corruption.
> >
> > In general, you cannot rely on the context/switch code as it may or may not
> > happen (that's up to the scheduler).
> >
>

I wanted to write also that this is a very good example, thank you for
putting it together

>
>
> > Cheers,
> >
> > --
> > Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 4:07 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi,
>
> On 12/11/2018 11:30, Mirela Simonovic wrote:
> > When Dom0 finalizes its suspend procedure the suspend of Xen is
> > triggered by calling system_suspend(). Dom0 finalizes the suspend from
> > its boot core (VCPU#0), which could be mapped to any physical CPU,
> > i.e. the system_suspend() function could be executed by any physical
> > CPU. Since Xen suspend procedure has to be run by the boot CPU
> > (non-boot CPUs will be disabled at some point in suspend procedure),
> > system_suspend() execution has to continue on CPU#0.
> >
> > When the system_suspend() returns 0, it means that the system was
> > suspended and it is coming out of the resume procedure. Regardless
> > of the system_suspend() return value, after this function returns
> > Xen is fully functional, and its state, including all devices and data
> > structures, matches the state prior to calling system_suspend().
> > The status is returned by system_suspend() for debugging/logging
> > purposes and function prototype compatibility.
> >
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> > ---
> >   xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
> >   1 file changed, 34 insertions(+)
> >
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index f2338e41db..21b45f8248 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
> >       _arch_set_info_guest(v, &ctxt);
> >   }
> >
> > +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> > +static long system_suspend(void *data)
> > +{
> > +    BUG_ON(system_state != SYS_STATE_active);
> > +
> > +    return -ENOSYS;
> > +}
> > +
> >   int32_t domain_suspend(register_t epoint, register_t cid)
> >   {
> >       struct vcpu *v;
> >       struct domain *d = current->domain;
> >       bool is_thumb = epoint & 1;
> > +    int status;
> >
> >       dprintk(XENLOG_DEBUG,
> >               "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> > @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
> >        */
> >       vcpu_block_unless_event_pending(current);
> >
> > +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
> > +    if ( is_hardware_domain(d) )
> > +    {
> > +        /*
> > +         * system_suspend should be called when Dom0 finalizes the suspend
> > +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
> > +         * be mapped to any PCPU (this function could be executed by any PCPU).
> > +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
> > +         * PCPUs will be disabled during the suspend).
> > +         */
> > +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
>
> Based on my comment in patch #2, I don't think this will do the correct thing on
> Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
> do_vpsci_0_2_call.
>

Could you please explain? I can't follow

> As upper layer may modify the vCPU context, I think it would be best to run
> system_suspend in a tasklet bound to CPU#0.
>

I'm not following this too, please explain

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Mirela,

On 14/11/2018 15:36, Mirela Simonovic wrote:
> On Wed, Nov 14, 2018 at 3:49 PM Julien Grall <julien.grall@arm.com> wrote:
>> On 14/11/2018 13:05, Julien Grall wrote:
>>> On 14/11/2018 12:35, Mirela Simonovic wrote:
>>>> On 11/14/2018 11:45 AM, Julien Grall wrote:
>>>>> Hi,
>>>>>
>>>>> On 13/11/2018 20:39, Stefano Stabellini wrote:
>>>>>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>>>>>>> However, what is the issue with saving all the registers here?
>>>>>>>>>
>>>>>>>>
>>>>>>>> We need to save arguments that are provided by a guest with system
>>>>>>>> suspend PSCI call. These arguments are the entry point that needs to
>>>>>>>> be saved in program counter and context ID that needs to be saved in
>>>>>>>> x0/r0. We don't have these arguments here. Context switch happens
>>>>>>>> after processing the system suspend PSCI call, so it's too late.
>>>>>>>
>>>>>>> It does not feel right to modify ctxt_switch{from,to} for suspend/resume. If
>>>>>>> you want to reset the vCPU state before blocking the vCPU, then you should
>>>>>>> instead
>>>>>>
>>>>
>>>> I think it's not clear what problem are we discussing here, at least it's not
>>>> to me. So I'll make an assumption, and please correct me if I'm wrong.
>>>> In the patches we submitted, the VCPU context is not reset in
>>>> ctxt_switch{from,to}. My understanding is that you suggested/asked to reset
>>>> the VCPU context when switch happens, and I explained why is that not possible
>>>> - at least not without additional code changes, that may not be so small. I
>>>> agree with Andrew's comment in this perspective - reset of VCPU should not
>>>> (and right now cannot) be done when the context is switched.
>>>
>>> I didn't ask to reset the vCPU context in the switch. Instead we should make
>>> sure the vCPU context is synced to memory before modifying it.
>>>
>>> It seems that solution works on x86 using domain_pause (see
>>> hvm_s3_{resume,suspend}). So I am not sure why it cannot be use on Arm. Note
>>> that it may require more work.
>>>
>>>>
>>>>>> You missed the end of the suggestion here
>>>>>
>>>>> Whoops. I meant that instead you should save the context of the vCPU in
>>>>> advance or reset the vCPU using the system registers directly.
>>>>>
>>>>> But my preference is to reset the vCPU when you receive the wake-up interrupt.
>>>>>
>>>>
>>>> Without you presenting more details how would that work I cannot really
>>>> provide any comment, nor say that your preference could work or be better
>>>> compared to what is in this series. Honestly, I don't understand what exactly
>>>> you're proposing, because more things needs to be think-through beyond the
>>>> place to put a code.
>>>> We submitted a code that works, which is very elegant and nice in my opinion
>>>> (fair to say we may not share opinions here), and does not require lots of
>>>> code changes. So there's the reference.
>>>> Could you please clarify why do you think the proposed solution is not good?
>>>
>>> The context switch is about saving/restore the context from the hardware. We can
>>> decide to optimize it in the suspend case (though it might be premature), but it
>>> is clearly the wrong place to decide to resume a domain.
>>
>> Actually, I just found a good example of why I think it is wrong and broken. You
>> rely on a context switch to always happen after suspending and on resuming the
>> guest. There are path where context/switch will not happen. An example is if you
>> have interrupt pending, you may return to the guest directly if the scheduler
>> does not have anything else to schedule.
>>
> 
> Can we check whether the vcpu blocked, right? Let me be more specific
> - after calling vcpu_block_unless_event_pending we can check whether
> the vcpu is indeed blocked?
This would be racy. The vCPU might be blocked when you check it. However, it can 
  get unblocked before you reached the schedule softirq (where schedule is done).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

Hi Mirela,

On 14/11/2018 15:40, Mirela Simonovic wrote:
> On Wed, Nov 14, 2018 at 4:07 PM Julien Grall <julien.grall@arm.com> wrote:
>> On 12/11/2018 11:30, Mirela Simonovic wrote:
>>> When Dom0 finalizes its suspend procedure the suspend of Xen is
>>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
>>> its boot core (VCPU#0), which could be mapped to any physical CPU,
>>> i.e. the system_suspend() function could be executed by any physical
>>> CPU. Since Xen suspend procedure has to be run by the boot CPU
>>> (non-boot CPUs will be disabled at some point in suspend procedure),
>>> system_suspend() execution has to continue on CPU#0.
>>>
>>> When the system_suspend() returns 0, it means that the system was
>>> suspended and it is coming out of the resume procedure. Regardless
>>> of the system_suspend() return value, after this function returns
>>> Xen is fully functional, and its state, including all devices and data
>>> structures, matches the state prior to calling system_suspend().
>>> The status is returned by system_suspend() for debugging/logging
>>> purposes and function prototype compatibility.
>>>
>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>> ---
>>>    xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
>>>    1 file changed, 34 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>>> index f2338e41db..21b45f8248 100644
>>> --- a/xen/arch/arm/suspend.c
>>> +++ b/xen/arch/arm/suspend.c
>>> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>>>        _arch_set_info_guest(v, &ctxt);
>>>    }
>>>
>>> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>>> +static long system_suspend(void *data)
>>> +{
>>> +    BUG_ON(system_state != SYS_STATE_active);
>>> +
>>> +    return -ENOSYS;
>>> +}
>>> +
>>>    int32_t domain_suspend(register_t epoint, register_t cid)
>>>    {
>>>        struct vcpu *v;
>>>        struct domain *d = current->domain;
>>>        bool is_thumb = epoint & 1;
>>> +    int status;
>>>
>>>        dprintk(XENLOG_DEBUG,
>>>                "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
>>> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
>>>         */
>>>        vcpu_block_unless_event_pending(current);
>>>
>>> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
>>> +    if ( is_hardware_domain(d) )
>>> +    {
>>> +        /*
>>> +         * system_suspend should be called when Dom0 finalizes the suspend
>>> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
>>> +         * be mapped to any PCPU (this function could be executed by any PCPU).
>>> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
>>> +         * PCPUs will be disabled during the suspend).
>>> +         */
>>> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
>>
>> Based on my comment in patch #2, I don't think this will do the correct thing on
>> Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
>> do_vpsci_0_2_call.
>>
> 
> Could you please explain? I can't follow

General purpose (e.g xN, pc) registers live at the bottom of the vCPU stack. The 
function vcpu_suspend will reset all of them to 0 but x0 (Context ID) and pc 
(entry point).

You rely on those registers to be untouched in the return path. However, this is 
not the case. If you look at do_vpsci_0_2_call, x0 will be set to whatever is 
the return value of domain_suspend (e.g PSCI_SUSPEND). So x0 will not contain 
anymore the Context ID as expected by the guest.

You probably haven't noticed it because Linux is currently using 0 for the 
context ID. This is the same value as PSCI_SUCCESS.

In the case of Dom0, this is a bit different to what I explained in my previous 
e-mail because I got confused. The function continue_hypercall_on_cpu is not 
doing what you expect. The function will pause the vCPU, schedule the tasklet 
and then return directly.

At some point the tasklet will get scheduled on CPU#0 and system_suspend will be 
called. The return value of that function will be copied to x0. The vCPU will 
then get unpaused and continue to run.

So x0 will not contain the Context ID but whatever system_suspend return.

The more I look at the code, the more I think that domain_suspend should be 
executed in a tasklet (e.g continue_hypercall_on_cpu) because the vCPU would not 
be running anymore. So you don't have to worry of anyone modifying the registers 
and therefore can access them safely as they would all be synced to the 
structure vCPU.

The tasklet code would be very similar to hvm_s3_suspend.

> 
>> As upper layer may modify the vCPU context, I think it would be best to run
>> system_suspend in a tasklet bound to CPU#0.
>>
> 
> I'm not following this too, please explain
I tried to cover it in my explanation above. Let me know if it is unclear.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 6:10 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Mirela,
>
> On 14/11/2018 15:40, Mirela Simonovic wrote:
> > On Wed, Nov 14, 2018 at 4:07 PM Julien Grall <julien.grall@arm.com> wrote:
> >> On 12/11/2018 11:30, Mirela Simonovic wrote:
> >>> When Dom0 finalizes its suspend procedure the suspend of Xen is
> >>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> >>> its boot core (VCPU#0), which could be mapped to any physical CPU,
> >>> i.e. the system_suspend() function could be executed by any physical
> >>> CPU. Since Xen suspend procedure has to be run by the boot CPU
> >>> (non-boot CPUs will be disabled at some point in suspend procedure),
> >>> system_suspend() execution has to continue on CPU#0.
> >>>
> >>> When the system_suspend() returns 0, it means that the system was
> >>> suspended and it is coming out of the resume procedure. Regardless
> >>> of the system_suspend() return value, after this function returns
> >>> Xen is fully functional, and its state, including all devices and data
> >>> structures, matches the state prior to calling system_suspend().
> >>> The status is returned by system_suspend() for debugging/logging
> >>> purposes and function prototype compatibility.
> >>>
> >>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> >>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >>> ---
> >>>    xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
> >>>    1 file changed, 34 insertions(+)
> >>>
> >>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> >>> index f2338e41db..21b45f8248 100644
> >>> --- a/xen/arch/arm/suspend.c
> >>> +++ b/xen/arch/arm/suspend.c
> >>> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
> >>>        _arch_set_info_guest(v, &ctxt);
> >>>    }
> >>>
> >>> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> >>> +static long system_suspend(void *data)
> >>> +{
> >>> +    BUG_ON(system_state != SYS_STATE_active);
> >>> +
> >>> +    return -ENOSYS;
> >>> +}
> >>> +
> >>>    int32_t domain_suspend(register_t epoint, register_t cid)
> >>>    {
> >>>        struct vcpu *v;
> >>>        struct domain *d = current->domain;
> >>>        bool is_thumb = epoint & 1;
> >>> +    int status;
> >>>
> >>>        dprintk(XENLOG_DEBUG,
> >>>                "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> >>> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
> >>>         */
> >>>        vcpu_block_unless_event_pending(current);
> >>>
> >>> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
> >>> +    if ( is_hardware_domain(d) )
> >>> +    {
> >>> +        /*
> >>> +         * system_suspend should be called when Dom0 finalizes the suspend
> >>> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
> >>> +         * be mapped to any PCPU (this function could be executed by any PCPU).
> >>> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
> >>> +         * PCPUs will be disabled during the suspend).
> >>> +         */
> >>> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> >>
> >> Based on my comment in patch #2, I don't think this will do the correct thing on
> >> Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
> >> do_vpsci_0_2_call.
> >>
> >
> > Could you please explain? I can't follow
>
> General purpose (e.g xN, pc) registers live at the bottom of the vCPU stack. The
> function vcpu_suspend will reset all of them to 0 but x0 (Context ID) and pc
> (entry point).
>
> You rely on those registers to be untouched in the return path. However, this is
> not the case. If you look at do_vpsci_0_2_call, x0 will be set to whatever is
> the return value of domain_suspend (e.g PSCI_SUSPEND). So x0 will not contain
> anymore the Context ID as expected by the guest.

This is expected, the system should behave that way. If the guest
managed to suspend, i.e. domain_suspend has returned PSCI_SUCCESS, the
return value is meaningless to the guest because it won't return to
it. The guest has suspended, just the fact the it will start from an
another entry point implicitly carries the information that the
suspend was successful.
However, if the return value from domain_suspend is an error then the
error will be returned to the guest because the suspend has failed.
Then the x0 register should contain error code, not the context ID.
The PC should be untouched, i.e. it should not contain the resume
entry point, but whatever was in there once the hvc/system_suspend was
issued by the guest. Guests handle errors right below the code from
which they tried to suspend.

>
> You probably haven't noticed it because Linux is currently using 0 for the
> context ID. This is the same value as PSCI_SUCCESS.
>
> In the case of Dom0, this is a bit different to what I explained in my previous
> e-mail because I got confused. The function continue_hypercall_on_cpu is not
> doing what you expect. The function will pause the vCPU, schedule the tasklet
> and then return directly.
>
> At some point the tasklet will get scheduled on CPU#0 and system_suspend will be
> called. The return value of that function will be copied to x0. The vCPU will
> then get unpaused and continue to run.
>
> So x0 will not contain the Context ID but whatever system_suspend return.
>

I agree with all you described above, but that is intended - the
system should behave that way. I believe the cause of misunderstanding
could be in how the return value versus context ID is handled. Those
are different paths, and only one of the following executes: 1) either
the suspend is successful and nothing will be returned to the guest
because it is suspended; or 2) the suspend of the guest has failed so
context ID is useless.

> The more I look at the code, the more I think that domain_suspend should be
> executed in a tasklet (e.g continue_hypercall_on_cpu) because the vCPU would not
> be running anymore. So you don't have to worry of anyone modifying the registers
> and therefore can access them safely as they would all be synced to the
> structure vCPU.
>

Lets revisit this once we align on the points above. Please let me
know if there is anything else I need to clarify, or if I
misunderstood your point.

Thanks,
Mirela

> The tasklet code would be very similar to hvm_s3_suspend.
>
> >
> >> As upper layer may modify the vCPU context, I think it would be best to run
> >> system_suspend in a tasklet bound to CPU#0.
> >>
> >
> > I'm not following this too, please explain
> I tried to cover it in my explanation above. Let me know if it is unclear.
>
> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

Hi,

On 14/11/2018 17:35, Mirela Simonovic wrote:
> On Wed, Nov 14, 2018 at 6:10 PM Julien Grall <julien.grall@arm.com> wrote:
>> On 14/11/2018 15:40, Mirela Simonovic wrote:
>>> On Wed, Nov 14, 2018 at 4:07 PM Julien Grall <julien.grall@arm.com> wrote:
>>>> On 12/11/2018 11:30, Mirela Simonovic wrote:
>>>>> When Dom0 finalizes its suspend procedure the suspend of Xen is
>>>>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
>>>>> its boot core (VCPU#0), which could be mapped to any physical CPU,
>>>>> i.e. the system_suspend() function could be executed by any physical
>>>>> CPU. Since Xen suspend procedure has to be run by the boot CPU
>>>>> (non-boot CPUs will be disabled at some point in suspend procedure),
>>>>> system_suspend() execution has to continue on CPU#0.
>>>>>
>>>>> When the system_suspend() returns 0, it means that the system was
>>>>> suspended and it is coming out of the resume procedure. Regardless
>>>>> of the system_suspend() return value, after this function returns
>>>>> Xen is fully functional, and its state, including all devices and data
>>>>> structures, matches the state prior to calling system_suspend().
>>>>> The status is returned by system_suspend() for debugging/logging
>>>>> purposes and function prototype compatibility.
>>>>>
>>>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>>>> ---
>>>>>     xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
>>>>>     1 file changed, 34 insertions(+)
>>>>>
>>>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>>>>> index f2338e41db..21b45f8248 100644
>>>>> --- a/xen/arch/arm/suspend.c
>>>>> +++ b/xen/arch/arm/suspend.c
>>>>> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>>>>>         _arch_set_info_guest(v, &ctxt);
>>>>>     }
>>>>>
>>>>> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>>>>> +static long system_suspend(void *data)
>>>>> +{
>>>>> +    BUG_ON(system_state != SYS_STATE_active);
>>>>> +
>>>>> +    return -ENOSYS;
>>>>> +}
>>>>> +
>>>>>     int32_t domain_suspend(register_t epoint, register_t cid)
>>>>>     {
>>>>>         struct vcpu *v;
>>>>>         struct domain *d = current->domain;
>>>>>         bool is_thumb = epoint & 1;
>>>>> +    int status;
>>>>>
>>>>>         dprintk(XENLOG_DEBUG,
>>>>>                 "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
>>>>> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
>>>>>          */
>>>>>         vcpu_block_unless_event_pending(current);
>>>>>
>>>>> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
>>>>> +    if ( is_hardware_domain(d) )
>>>>> +    {
>>>>> +        /*
>>>>> +         * system_suspend should be called when Dom0 finalizes the suspend
>>>>> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
>>>>> +         * be mapped to any PCPU (this function could be executed by any PCPU).
>>>>> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
>>>>> +         * PCPUs will be disabled during the suspend).
>>>>> +         */
>>>>> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
>>>>
>>>> Based on my comment in patch #2, I don't think this will do the correct thing on
>>>> Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
>>>> do_vpsci_0_2_call.
>>>>
>>>
>>> Could you please explain? I can't follow
>>
>> General purpose (e.g xN, pc) registers live at the bottom of the vCPU stack. The
>> function vcpu_suspend will reset all of them to 0 but x0 (Context ID) and pc
>> (entry point).
>>
>> You rely on those registers to be untouched in the return path. However, this is
>> not the case. If you look at do_vpsci_0_2_call, x0 will be set to whatever is
>> the return value of domain_suspend (e.g PSCI_SUSPEND). So x0 will not contain
>> anymore the Context ID as expected by the guest.
> 
> This is expected, the system should behave that way. If the guest
> managed to suspend, i.e. domain_suspend has returned PSCI_SUCCESS, the
> return value is meaningless to the guest because it won't return to
> it. The guest has suspended, just the fact the it will start from an
> another entry point implicitly carries the information that the
> suspend was successful.
> However, if the return value from domain_suspend is an error then the
> error will be returned to the guest because the suspend has failed.
> Then the x0 register should contain error code, not the context ID.
> The PC should be untouched, i.e. it should not contain the resume
> entry point, but whatever was in there once the hvc/system_suspend was
> issued by the guest. Guests handle errors right below the code from
> which they tried to suspend.

I think you misunderstood my comment. I am not speaking about the failure case 
but the success case where the domain is actually suspended for some time.

When the domain is resuming, the "boot" vCPU should see Context ID and not 
PSCI_SUCCESS.

However, this is not the case because of the following path:

	-> do_vpsci_0_2_call
		-> domain_suspend
			-> vcpu_suspend
				-> x0 = Context ID
			-> return PSCI_SUCCESS
		-> x0 = PSCI_SUCCESS

At some point in the future your guest will resume. Instead of seen Context ID, 
it will actually see PSCI_SUCCESS. You can easily test that by modifying the 
SYSTEM_SUSPEND code in Linux to use a different context ID and add hvc #0xffe0 
to dump register x0.

> 
>>
>> You probably haven't noticed it because Linux is currently using 0 for the
>> context ID. This is the same value as PSCI_SUCCESS.
>>
>> In the case of Dom0, this is a bit different to what I explained in my previous
>> e-mail because I got confused. The function continue_hypercall_on_cpu is not
>> doing what you expect. The function will pause the vCPU, schedule the tasklet
>> and then return directly.
>>
>> At some point the tasklet will get scheduled on CPU#0 and system_suspend will be
>> called. The return value of that function will be copied to x0. The vCPU will
>> then get unpaused and continue to run.
>>
>> So x0 will not contain the Context ID but whatever system_suspend return.
>>
> 
> I agree with all you described above, but that is intended - the
> system should behave that way. I believe the cause of misunderstanding
> could be in how the return value versus context ID is handled. Those
> are different paths, and only one of the following executes: 1) either
> the suspend is successful and nothing will be returned to the guest
> because it is suspended; or 2) the suspend of the guest has failed so
> context ID is useless.

You missed my point here, your guest will resume at some point. As you reset the 
vCPU and set x0 in vcpu_suspend. Then anything after can overwrite the registers 
you set in vcpu_suspend.

Please explain why you think your code behave differently that I wrote when 
resuming.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1850 bytes --]

On Wed, 14 Nov 2018, Julien Grall wrote:
> > > >   @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
> > > >       BUG();
> > > >   }
> > > >   +static void gicv2_alloc_context(struct gicv2_context *gc)
> > > > +{
> > > 
> > > Is it necessary to allocate them at boot? Can we make them static or
> > > allocate them when we suspend?
> > > 
> > 
> > We need to allocate dynamically because the size of allocated data depends
> > on the number of irq lines, which is not known at the compile time.
> 
> Well you know the upper bound. Why can't you use the upper bound?
> 
> > Alternative is to allocate on suspend, but I believe it is better to do this
> > when the system boots.
> 
> Why is it better?

I'll reply here also to your other point because they are related:

> Suspend/resume is not a critical feature in common case. So I would
> prefer if we disable it when we can't alloc memory.


It is true that suspend/resume is not a critical feature for the common
case, but proceeding as "normal" when a memory allocation fails is not a
good idea: if the hypervisor is so low in memory as to fail in an
allocation like this one, it is not going to be able to work right. In
no other cases in Xen we continue on memory allocation failures, even
for less-critical features.

I suggest that we either allocate statically using the upper bound as
you suggested, although it leads to some memory being wasted. Or, and
this is my favorite option, we allocate it dynamically but we return a
proper error on memory allocation failures. We should at the very
least print an error.

I would prefer if it is done at boot time so that the user can figure
out that their configuration is wrong and fix it straight away, but it
could also be done at suspend time. Better to fail reliably early,
rather than failing unpredictably later.

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 09/18] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume

[Stefano Stabellini]

On Wed, 14 Nov 2018, Julien Grall wrote:
> Hi,
> 
> On 12/11/2018 11:30, Mirela Simonovic wrote:
> > The rcu_barrier() has to be added to ensure that the per cpu area is
> > freed before a non-boot CPU tries to initialize it (_free_percpu_area()
> > has to be called before the init_percpu_area()). This scenario occurs
> > when non-boot CPUs are hot-unplugged on suspend and hotplugged on resume.
> 
> From the description, this is a bug introduced by the previous patch. So
> should merged in it.
 
It makes sense to squash this patch together with patch #8.


> The commit message will also need to explain why the rcu_barrier() is
> required.
>
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> > ---
> >   xen/arch/arm/suspend.c | 1 +
> >   1 file changed, 1 insertion(+)
> > 
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index dae1b1f7d6..8e8e531d61 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -133,6 +133,7 @@ static long system_suspend(void *data)
> >       system_state = SYS_STATE_resume;
> >     resume_nonboot_cpus:
> > +    rcu_barrier();
> >       enable_nonboot_cpus();
> >       thaw_domains();
> >       system_state = SYS_STATE_active;
> > 
> 
> -- 
> Julien Grall
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Stefano Stabellini]

On Wed, 14 Nov 2018, Julien Grall wrote:
> Hi,
> 
> On 13/11/2018 20:44, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Julien Grall wrote:
> > > (+ Andre)
> > > 
> > > On 11/12/18 5:42 PM, Mirela Simonovic wrote:
> > > > Hi Julien,
> > > > 
> > > > On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com>
> > > > wrote:
> > > > > 
> > > > > 
> > > > > 
> > > > > On 11/12/18 4:52 PM, Mirela Simonovic wrote:
> > > > > > Hi Julien,
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > > Thanks for the feedback.
> > > > > > 
> > > > > > On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com>
> > > > > > wrote:
> > > > > > > 
> > > > > > > Hi Mirela,
> > > > > > > 
> > > > > > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > > > > > GIC and virtual timer context must be saved when the domain
> > > > > > > > suspends.
> > > > > > > 
> > > > > > > Please provide the rationale for this. Is it required by the spec?
> > > > > > > 
> > > > > > 
> > > > > > This is required for GIC because a guest leaves enabled interrupts
> > > > > > in
> > > > > > the GIC that could wake it up, and on resume it should be able to
> > > > > > detect which interrupt woke it up. Without saving/restoring the
> > > > > > state
> > > > > > of GIC this would not be possible.
> > > > > 
> > > > > I am confused. In patch #10, you save the GIC host because the GIC can
> > > > > be powered-down. Linux is also saving the GIC state. So how the
> > > > > interrupt can come up if the GIC is powered down?
> > > > > 
> > > > 
> > > > After Xen (or Linux in the config without Xen) hands over the control
> > > > to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
> > > > enabled interrupts that are its wake-up sources. Saving a GIC state
> > > > only means that its current configuration will be remembered somewhere
> > > > in data structures, but the configuration is not changed on suspend.
> > > > Everything that happens with interrupt configuration from this point
> > > > on is platform specific. Now there are 2 options: 1) EL3 will never
> > > > allow CPU0 to be powered down and the wake-up interrupt will indeed
> > > > propagate via GIC;
> > > > or 2) CPU0 will be powered down and the GIC may be
> > > > powered down as well, so an external help is needed to deal with
> > > > wake-up and resume (e.g. in order to react to a wake-up interrupt
> > > > while the GIC is down, and power up CPU0). This external help is
> > > > provided by some low-power processor outside of the Cortex-A cluster.
> > > > 
> > > > So the platform firmware is responsible for properly configuring a
> > > > wake-up path if GIC goes down. This is commonly handled by EL3
> > > > communicating with low-power processor. When the GIC power comes up,
> > > > the interrupt triggered by a peripheral is still active and the
> > > > software on Cortex-A cluster should be able to observe it once the GIC
> > > > state is restored, i.e. interrupts get enabled at GIC.
> > > 
> > > Thank you for the explanation.  Now the question is why can't we reset at
> > > least the GIC CPU interface?
> > > 
> > > AFAIU, the guest should restore them in any case. The only things we need
> > > to
> > > know is the interrupt was received for a given guest. We can then queue it
> > > and
> > > wake-up the domain.
> > > 
> > > This seems to fit with the description on top of gic_dist_save() in Linux
> > > GICv2 driver.
> > 
> > Can we rely on all PSCI compliant OSes to restore their own GIC again at
> > resume? The PSCI spec is not very clear in that regard (at the the
> > version I am looking at...) I am just asking so that we don't come up
> > with a solution that only works with Linux.
> See PSCI 1.1 (DEN0022D) section 6.8. Each level should save its own context
> because the PSCI implementations is allowed to shutdown the GIC.

Great, in that case we should be able to skip saving some of the GICD
registers too. We do need to save GICD_ISACTIVER, and GICD_ICFGR,
but we should be able to skip the others (GICD_ISENABLER,
GICD_IPRIORITYR, GICD_ITARGETSR). If we do, we still need to
re-initialize them as we do in gicv2_dist_init.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3593 bytes --]

On Wed, 14 Nov 2018, Julien Grall wrote:
> Hi Mirela,
> 
> On 14/11/2018 13:00, Mirela Simonovic wrote:
> > 
> > 
> > On 11/14/2018 11:52 AM, Julien Grall wrote:
> > > Hi Mirela,
> > > 
> > > On 12/11/2018 11:30, Mirela Simonovic wrote:
> > > > Non-boot CPUs have to be disabled on suspend and enabled on resume
> > > > (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
> > > > CPU_OFF to be called by each non-boot CPU. Depending on the underlying
> > > > platform capabilities, this may lead to the physical powering down of
> > > > CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
> > > > each non-boot CPU).
> > > > 
> > > > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > > > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> > > > ---
> > > >   xen/arch/arm/suspend.c | 15 ++++++++++++++-
> > > >   1 file changed, 14 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > > > index 575afd5eb8..dae1b1f7d6 100644
> > > > --- a/xen/arch/arm/suspend.c
> > > > +++ b/xen/arch/arm/suspend.c
> > > > @@ -1,4 +1,5 @@
> > > >   #include <xen/sched.h>
> > > > +#include <xen/cpu.h>
> > > >   #include <asm/cpufeature.h>
> > > >   #include <asm/event.h>
> > > >   #include <asm/psci.h>
> > > > @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint,
> > > > register_t cid)
> > > >   /* Xen suspend. Note: data is not used (suspend is the suspend to RAM)
> > > > */
> > > >   static long system_suspend(void *data)
> > > >   {
> > > > +    int status;
> > > > +
> > > >       BUG_ON(system_state != SYS_STATE_active);
> > > >         system_state = SYS_STATE_suspend;
> > > >       freeze_domains();
> > > >   +    status = disable_nonboot_cpus();
> > > > +    if ( status )
> > > > +    {
> > > > +        system_state = SYS_STATE_resume;
> > > > +        goto resume_nonboot_cpus;
> > > > +    }
> > > > +
> > > >       system_state = SYS_STATE_resume;
> > > >   +resume_nonboot_cpus:
> > > > +    enable_nonboot_cpus();
> > > >       thaw_domains();
> > > >       system_state = SYS_STATE_active;
> > > > +    dsb(sy);
> > > 
> > > Why do you need a dsb(sy) here?
> > > 
> > 
> > Updated value of system_state variable needs to be visible to other CPUs
> > before we move on
> 
> We tend to write the reason on top of barrier why they are necessary. But I am
> still unsure to understand why this is important. What would happen if move on
> without it?

That is a good question. I went through the code and it seems that the
only effect could be potentially taking the wrong path in
cpupool_cpu_add, but since that's called from a .notifier_call I don't
think it can happen in practice. It is always difficult to prove that
we don't need a barrier, it is easier to prove when we need a barrier,
but in this case it does look like we do not need it after all.

 
> > > >   -    return -ENOSYS;
> > > 
> > > Why do you return -ENOSYS until this patch? Should not have it be 0?
> > > 
> > 
> > To distinguish that Xen suspend wasn't supported until we at least do
> > something useful in suspend procedure. This is not so important, we can
> > return 0 but needs to be fixed in previous patches.
> 
> If you return 0 before hand you can more easily bisect this series and know
> where it suspend/resume breaks.

Why 0 improves bisectability? 0 prevents other checks from figuring out
that there was an error. Also, the feature is not bisectable until the
full series is applied.

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Stefano Stabellini]

On Wed, 14 Nov 2018, Julien Grall wrote:
> On 13/11/2018 20:39, Stefano Stabellini wrote:
> > On Tue, 13 Nov 2018, Julien Grall wrote:
> > 
> > If we mark the domU simply as "paused" it will be difficult to implement
> > correctly "xl restore" / "xl trigger s3resume". We should be able to
> > distinguish a domain which is simply not running or paused (as in "xl
> > pause") from one that has been put to sleep.  Otherwise we won't be able
> > to use "xl pause" in its original sense anymore. "xl pause" will become
> > effectively "xl trigger sleep" on ARM. Which is something to consider
> > but I don't think that is what we want.
> 
> I didn't suggested to only use those 2 helpers. But you can build
> suspend/resume on top of it. AFAICT, this is how s3 suspend/resume works on
> x86 HVM.

Sounds like we are saying the same thing.


> > The most similar feature is ACPI S3 in x86-land but the current calls
> > are so ACPI/x86 specific that don't make much sense on a ACPI-less ARM
> > implementation. If I am not mistaken, on x86 there is a special struct
> > domain flag for this: d->arch.hvm.is_s3_suspended.
> > 
> > 
> > Let's leave aside the "which commands should we use" discussion for now
> > because it doesn't related to this patch series. It seems to me that the
> > best option is to introduce a new ARM specific struct domain flag,
> > something akin to d->arch.hvm.is_s3_suspended but ARM PSCI specific.
> 
> See above.
> 
> -- 
> Julien Grall
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Julien Grall]

On 14/11/2018 22:45, Stefano Stabellini wrote:
> On Wed, 14 Nov 2018, Julien Grall wrote:
>> Hi,
>>
>> On 13/11/2018 20:44, Stefano Stabellini wrote:
>>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>> (+ Andre)
>>>>
>>>> On 11/12/18 5:42 PM, Mirela Simonovic wrote:
>>>>> Hi Julien,
>>>>>
>>>>> On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
>>>>>>> Hi Julien,
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>> Thanks for the feedback.
>>>>>>>
>>>>>>> On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Mirela,
>>>>>>>>
>>>>>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>>>>>> GIC and virtual timer context must be saved when the domain
>>>>>>>>> suspends.
>>>>>>>>
>>>>>>>> Please provide the rationale for this. Is it required by the spec?
>>>>>>>>
>>>>>>>
>>>>>>> This is required for GIC because a guest leaves enabled interrupts
>>>>>>> in
>>>>>>> the GIC that could wake it up, and on resume it should be able to
>>>>>>> detect which interrupt woke it up. Without saving/restoring the
>>>>>>> state
>>>>>>> of GIC this would not be possible.
>>>>>>
>>>>>> I am confused. In patch #10, you save the GIC host because the GIC can
>>>>>> be powered-down. Linux is also saving the GIC state. So how the
>>>>>> interrupt can come up if the GIC is powered down?
>>>>>>
>>>>>
>>>>> After Xen (or Linux in the config without Xen) hands over the control
>>>>> to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
>>>>> enabled interrupts that are its wake-up sources. Saving a GIC state
>>>>> only means that its current configuration will be remembered somewhere
>>>>> in data structures, but the configuration is not changed on suspend.
>>>>> Everything that happens with interrupt configuration from this point
>>>>> on is platform specific. Now there are 2 options: 1) EL3 will never
>>>>> allow CPU0 to be powered down and the wake-up interrupt will indeed
>>>>> propagate via GIC;
>>>>> or 2) CPU0 will be powered down and the GIC may be
>>>>> powered down as well, so an external help is needed to deal with
>>>>> wake-up and resume (e.g. in order to react to a wake-up interrupt
>>>>> while the GIC is down, and power up CPU0). This external help is
>>>>> provided by some low-power processor outside of the Cortex-A cluster.
>>>>>
>>>>> So the platform firmware is responsible for properly configuring a
>>>>> wake-up path if GIC goes down. This is commonly handled by EL3
>>>>> communicating with low-power processor. When the GIC power comes up,
>>>>> the interrupt triggered by a peripheral is still active and the
>>>>> software on Cortex-A cluster should be able to observe it once the GIC
>>>>> state is restored, i.e. interrupts get enabled at GIC.
>>>>
>>>> Thank you for the explanation.  Now the question is why can't we reset at
>>>> least the GIC CPU interface?
>>>>
>>>> AFAIU, the guest should restore them in any case. The only things we need
>>>> to
>>>> know is the interrupt was received for a given guest. We can then queue it
>>>> and
>>>> wake-up the domain.
>>>>
>>>> This seems to fit with the description on top of gic_dist_save() in Linux
>>>> GICv2 driver.
>>>
>>> Can we rely on all PSCI compliant OSes to restore their own GIC again at
>>> resume? The PSCI spec is not very clear in that regard (at the the
>>> version I am looking at...) I am just asking so that we don't come up
>>> with a solution that only works with Linux.
>> See PSCI 1.1 (DEN0022D) section 6.8. Each level should save its own context
>> because the PSCI implementations is allowed to shutdown the GIC.
> 
> Great, in that case we should be able to skip saving some of the GICD
> registers too. We do need to save GICD_ISACTIVER, and GICD_ICFGR,
> but we should be able to skip the others (GICD_ISENABLER,
> GICD_IPRIORITYR, GICD_ITARGETSR). If we do, we still need to
> re-initialize them as we do in gicv2_dist_init.

You are assuming a domain will handle properly the suspend/resume. I 
don't think we can promise that as we call freeze/thaw.

Furthermore, we still have to suspend/resume other drivers in Xen. I 
think this is a bit painful to have to rely on every drivers to deal 
with their interrupts.

Cheers,

-- 
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Julien Grall]

On 14/11/2018 23:04, Stefano Stabellini wrote:
> On Wed, 14 Nov 2018, Julien Grall wrote:
>> Hi Mirela,
>>
>> On 14/11/2018 13:00, Mirela Simonovic wrote:
>>>
>>>
>>> On 11/14/2018 11:52 AM, Julien Grall wrote:
>>>> Hi Mirela,
>>>>
>>>> On 12/11/2018 11:30, Mirela Simonovic wrote:
>>>>> Non-boot CPUs have to be disabled on suspend and enabled on resume
>>>>> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
>>>>> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
>>>>> platform capabilities, this may lead to the physical powering down of
>>>>> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
>>>>> each non-boot CPU).
>>>>>
>>>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>>>> ---
>>>>>    xen/arch/arm/suspend.c | 15 ++++++++++++++-
>>>>>    1 file changed, 14 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>>>>> index 575afd5eb8..dae1b1f7d6 100644
>>>>> --- a/xen/arch/arm/suspend.c
>>>>> +++ b/xen/arch/arm/suspend.c
>>>>> @@ -1,4 +1,5 @@
>>>>>    #include <xen/sched.h>
>>>>> +#include <xen/cpu.h>
>>>>>    #include <asm/cpufeature.h>
>>>>>    #include <asm/event.h>
>>>>>    #include <asm/psci.h>
>>>>> @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint,
>>>>> register_t cid)
>>>>>    /* Xen suspend. Note: data is not used (suspend is the suspend to RAM)
>>>>> */
>>>>>    static long system_suspend(void *data)
>>>>>    {
>>>>> +    int status;
>>>>> +
>>>>>        BUG_ON(system_state != SYS_STATE_active);
>>>>>          system_state = SYS_STATE_suspend;
>>>>>        freeze_domains();
>>>>>    +    status = disable_nonboot_cpus();
>>>>> +    if ( status )
>>>>> +    {
>>>>> +        system_state = SYS_STATE_resume;
>>>>> +        goto resume_nonboot_cpus;
>>>>> +    }
>>>>> +
>>>>>        system_state = SYS_STATE_resume;
>>>>>    +resume_nonboot_cpus:
>>>>> +    enable_nonboot_cpus();
>>>>>        thaw_domains();
>>>>>        system_state = SYS_STATE_active;
>>>>> +    dsb(sy);
>>>>
>>>> Why do you need a dsb(sy) here?
>>>>
>>>
>>> Updated value of system_state variable needs to be visible to other CPUs
>>> before we move on
>>
>> We tend to write the reason on top of barrier why they are necessary. But I am
>> still unsure to understand why this is important. What would happen if move on
>> without it?
> 
> That is a good question. I went through the code and it seems that the
> only effect could be potentially taking the wrong path in
> cpupool_cpu_add, but since that's called from a .notifier_call I don't
> think it can happen in practice. It is always difficult to prove that
> we don't need a barrier, it is easier to prove when we need a barrier,
> but in this case it does look like we do not need it after all.

It is also very easy to add barrier everywhere so we are sure what to do 
;). If you need a barrier, then you need to give plausible explanation.

In that case, if you need barrier here for system_state. Then what 
wouldn't you need it in other places where system_state is updated/read?
> 
>   
>>>>>    -    return -ENOSYS;
>>>>
>>>> Why do you return -ENOSYS until this patch? Should not have it be 0?
>>>>
>>>
>>> To distinguish that Xen suspend wasn't supported until we at least do
>>> something useful in suspend procedure. This is not so important, we can
>>> return 0 but needs to be fixed in previous patches.
>>
>> If you return 0 before hand you can more easily bisect this series and know
>> where it suspend/resume breaks.
> 
> Why 0 improves bisectability? 0 prevents other checks from figuring out
> that there was an error.

But this code is exactly replacing -ENOSYS by state (that would be 0 in 
success. So what's wrong to return 0 rather than -ENOSYS in that patch 
implement the dummy system_state?

> Also, the feature is not bisectable until the
> full series is applied.

Really? I was under the impression you can still do some sort of 
suspend/resume patch by patch. Although, you would not do a full 
suspend/resume.

Cheers,

-- 
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Julien Grall]

Hi,

On 14/11/2018 22:18, Stefano Stabellini wrote:
> On Wed, 14 Nov 2018, Julien Grall wrote:
>>>>>    @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
>>>>>        BUG();
>>>>>    }
>>>>>    +static void gicv2_alloc_context(struct gicv2_context *gc)
>>>>> +{
>>>>
>>>> Is it necessary to allocate them at boot? Can we make them static or
>>>> allocate them when we suspend?
>>>>
>>>
>>> We need to allocate dynamically because the size of allocated data depends
>>> on the number of irq lines, which is not known at the compile time.
>>
>> Well you know the upper bound. Why can't you use the upper bound?
>>
>>> Alternative is to allocate on suspend, but I believe it is better to do this
>>> when the system boots.
>>
>> Why is it better?
> 
> I'll reply here also to your other point because they are related:
> 
>> Suspend/resume is not a critical feature in common case. So I would
>> prefer if we disable it when we can't alloc memory.
> 
> 
> It is true that suspend/resume is not a critical feature for the common
> case, but proceeding as "normal" when a memory allocation fails is not a
> good idea: if the hypervisor is so low in memory as to fail in an
> allocation like this one, it is not going to be able to work right. In
> no other cases in Xen we continue on memory allocation failures, even
> for less-critical features.
> 
> I suggest that we either allocate statically using the upper bound as
> you suggested, although it leads to some memory being wasted.

We are speaking of at most 2KB of memory. I don't think it is going to 
be waste given of the number of interrupts GIC usually supports.

The more that we already statically allocate irq_desc.

Cheers,

-- 
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 5116 bytes --]

Sorry for the formatting.

On Wed, 14 Nov 2018, 23:41 Julien Grall, <Julien.Grall@arm.com> wrote:

>
>
> On 14/11/2018 22:45, Stefano Stabellini wrote:
> > On Wed, 14 Nov 2018, Julien Grall wrote:
> >> Hi,
> >>
> >> On 13/11/2018 20:44, Stefano Stabellini wrote:
> >>> On Mon, 12 Nov 2018, Julien Grall wrote:
> >>>> (+ Andre)
> >>>>
> >>>> On 11/12/18 5:42 PM, Mirela Simonovic wrote:
> >>>>> Hi Julien,
> >>>>>
> >>>>> On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
> >>>>>>> Hi Julien,
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>> Thanks for the feedback.
> >>>>>>>
> >>>>>>> On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com
> >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Mirela,
> >>>>>>>>
> >>>>>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>>>>>>>> GIC and virtual timer context must be saved when the domain
> >>>>>>>>> suspends.
> >>>>>>>>
> >>>>>>>> Please provide the rationale for this. Is it required by the spec?
> >>>>>>>>
> >>>>>>>
> >>>>>>> This is required for GIC because a guest leaves enabled interrupts
> >>>>>>> in
> >>>>>>> the GIC that could wake it up, and on resume it should be able to
> >>>>>>> detect which interrupt woke it up. Without saving/restoring the
> >>>>>>> state
> >>>>>>> of GIC this would not be possible.
> >>>>>>
> >>>>>> I am confused. In patch #10, you save the GIC host because the GIC
> can
> >>>>>> be powered-down. Linux is also saving the GIC state. So how the
> >>>>>> interrupt can come up if the GIC is powered down?
> >>>>>>
> >>>>>
> >>>>> After Xen (or Linux in the config without Xen) hands over the control
> >>>>> to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
> >>>>> enabled interrupts that are its wake-up sources. Saving a GIC state
> >>>>> only means that its current configuration will be remembered
> somewhere
> >>>>> in data structures, but the configuration is not changed on suspend.
> >>>>> Everything that happens with interrupt configuration from this point
> >>>>> on is platform specific. Now there are 2 options: 1) EL3 will never
> >>>>> allow CPU0 to be powered down and the wake-up interrupt will indeed
> >>>>> propagate via GIC;
> >>>>> or 2) CPU0 will be powered down and the GIC may be
> >>>>> powered down as well, so an external help is needed to deal with
> >>>>> wake-up and resume (e.g. in order to react to a wake-up interrupt
> >>>>> while the GIC is down, and power up CPU0). This external help is
> >>>>> provided by some low-power processor outside of the Cortex-A cluster.
> >>>>>
> >>>>> So the platform firmware is responsible for properly configuring a
> >>>>> wake-up path if GIC goes down. This is commonly handled by EL3
> >>>>> communicating with low-power processor. When the GIC power comes up,
> >>>>> the interrupt triggered by a peripheral is still active and the
> >>>>> software on Cortex-A cluster should be able to observe it once the
> GIC
> >>>>> state is restored, i.e. interrupts get enabled at GIC.
> >>>>
> >>>> Thank you for the explanation.  Now the question is why can't we
> reset at
> >>>> least the GIC CPU interface?
> >>>>
> >>>> AFAIU, the guest should restore them in any case. The only things we
> need
> >>>> to
> >>>> know is the interrupt was received for a given guest. We can then
> queue it
> >>>> and
> >>>> wake-up the domain.
> >>>>
> >>>> This seems to fit with the description on top of gic_dist_save() in
> Linux
> >>>> GICv2 driver.
> >>>
> >>> Can we rely on all PSCI compliant OSes to restore their own GIC again
> at
> >>> resume? The PSCI spec is not very clear in that regard (at the the
> >>> version I am looking at...) I am just asking so that we don't come up
> >>> with a solution that only works with Linux.
> >> See PSCI 1.1 (DEN0022D) section 6.8. Each level should save its own
> context
> >> because the PSCI implementations is allowed to shutdown the GIC.
> >
> > Great, in that case we should be able to skip saving some of the GICD
> > registers too. We do need to save GICD_ISACTIVER, and GICD_ICFGR,
> > but we should be able to skip the others (GICD_ISENABLER,
> > GICD_IPRIORITYR, GICD_ITARGETSR). If we do, we still need to
> > re-initialize them as we do in gicv2_dist_init.
>
> You are assuming a domain will handle properly the suspend/resume. I
> don't think we can promise that as we call freeze/thaw.
>

To clarify what I meant by "handle properly" is any domain that will not
call SYSTEM_SUSPEND before the host is suspending. That may happen if the
domain is not aware of suspend/resume.

If you wonder, it is not Xen to decide whether we should stop suspending
but the control domain to not issue the suspend.

Cheers,


> Furthermore, we still have to suspend/resume other drivers in Xen. I
> think this is a bit painful to have to rely on every drivers to deal
> with their interrupts.
>
> Cheers,
>
> --
> Julien Grall
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 7576 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Stefano Stabellini]

On Wed, 14 Nov 2018, Julien Grall wrote:
> On 14/11/2018 22:45, Stefano Stabellini wrote:
> > On Wed, 14 Nov 2018, Julien Grall wrote:
> >> Hi,
> >>
> >> On 13/11/2018 20:44, Stefano Stabellini wrote:
> >>> On Mon, 12 Nov 2018, Julien Grall wrote:
> >>>> (+ Andre)
> >>>>
> >>>> On 11/12/18 5:42 PM, Mirela Simonovic wrote:
> >>>>> Hi Julien,
> >>>>>
> >>>>> On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
> >>>>>>> Hi Julien,
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>> Thanks for the feedback.
> >>>>>>>
> >>>>>>> On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Mirela,
> >>>>>>>>
> >>>>>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>>>>>>>> GIC and virtual timer context must be saved when the domain
> >>>>>>>>> suspends.
> >>>>>>>>
> >>>>>>>> Please provide the rationale for this. Is it required by the spec?
> >>>>>>>>
> >>>>>>>
> >>>>>>> This is required for GIC because a guest leaves enabled interrupts
> >>>>>>> in
> >>>>>>> the GIC that could wake it up, and on resume it should be able to
> >>>>>>> detect which interrupt woke it up. Without saving/restoring the
> >>>>>>> state
> >>>>>>> of GIC this would not be possible.
> >>>>>>
> >>>>>> I am confused. In patch #10, you save the GIC host because the GIC can
> >>>>>> be powered-down. Linux is also saving the GIC state. So how the
> >>>>>> interrupt can come up if the GIC is powered down?
> >>>>>>
> >>>>>
> >>>>> After Xen (or Linux in the config without Xen) hands over the control
> >>>>> to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
> >>>>> enabled interrupts that are its wake-up sources. Saving a GIC state
> >>>>> only means that its current configuration will be remembered somewhere
> >>>>> in data structures, but the configuration is not changed on suspend.
> >>>>> Everything that happens with interrupt configuration from this point
> >>>>> on is platform specific. Now there are 2 options: 1) EL3 will never
> >>>>> allow CPU0 to be powered down and the wake-up interrupt will indeed
> >>>>> propagate via GIC;
> >>>>> or 2) CPU0 will be powered down and the GIC may be
> >>>>> powered down as well, so an external help is needed to deal with
> >>>>> wake-up and resume (e.g. in order to react to a wake-up interrupt
> >>>>> while the GIC is down, and power up CPU0). This external help is
> >>>>> provided by some low-power processor outside of the Cortex-A cluster.
> >>>>>
> >>>>> So the platform firmware is responsible for properly configuring a
> >>>>> wake-up path if GIC goes down. This is commonly handled by EL3
> >>>>> communicating with low-power processor. When the GIC power comes up,
> >>>>> the interrupt triggered by a peripheral is still active and the
> >>>>> software on Cortex-A cluster should be able to observe it once the GIC
> >>>>> state is restored, i.e. interrupts get enabled at GIC.
> >>>>
> >>>> Thank you for the explanation.  Now the question is why can't we reset at
> >>>> least the GIC CPU interface?
> >>>>
> >>>> AFAIU, the guest should restore them in any case. The only things we need
> >>>> to
> >>>> know is the interrupt was received for a given guest. We can then queue it
> >>>> and
> >>>> wake-up the domain.
> >>>>
> >>>> This seems to fit with the description on top of gic_dist_save() in Linux
> >>>> GICv2 driver.
> >>>
> >>> Can we rely on all PSCI compliant OSes to restore their own GIC again at
> >>> resume? The PSCI spec is not very clear in that regard (at the the
> >>> version I am looking at...) I am just asking so that we don't come up
> >>> with a solution that only works with Linux.
> >> See PSCI 1.1 (DEN0022D) section 6.8. Each level should save its own context
> >> because the PSCI implementations is allowed to shutdown the GIC.
> > 
> > Great, in that case we should be able to skip saving some of the GICD
> > registers too. We do need to save GICD_ISACTIVER, and GICD_ICFGR,
> > but we should be able to skip the others (GICD_ISENABLER,
> > GICD_IPRIORITYR, GICD_ITARGETSR). If we do, we still need to
> > re-initialize them as we do in gicv2_dist_init.
> 
> You are assuming a domain will handle properly the suspend/resume. I 
> don't think we can promise that as we call freeze/thaw.

Dho! That would break every single guest that has been forcefully
suspended :-/

Right, we can't do that (FYI I tested today the series with an unaware
domU and it all resumed correctly.)

But given that we only suspend/resume GICC_CTLR, GICC_PMR, GICC_BPR of
the GICC interface, it should be fine to re-initialize that. We do need
to be careful because the current implementation of gicv2_cpu_init
touches a bunch of GICD registers that we'll have to save separately for
suspend/resume.


> Furthermore, we still have to suspend/resume other drivers in Xen. I 
> think this is a bit painful to have to rely on every drivers to deal 
> with their interrupts.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Andrew Cooper]

On 14/11/2018 12:49, Julien Grall wrote:
> Hi Mirela,
>
> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>
>>
>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>> Hi Andrew,
>>>>
>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>         if ( is_idle_vcpu(p) )
>>>>>>>>             return;
>>>>>>>>
>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>> suspended */
>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>> +        return;
>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>> state
>>>>>>> to be saved on suspend.
>>>>>>>
>>>>>> We just need a flag to mark a domain as suspended, and I do believe
>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>> Let's come back on this.
>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
>>>>> tree and you'll find several pieces of documentation, including the
>>>>> description of what this shutdown code means.
>>>>>
>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>> with
>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>> Xen's
>>>>> shutdown infrastructure, which exists mainly to communicate with the
>>>>> toolstack.
>>>> Would domain pause/unpause be a better solution?
>>> Actually yes - that sounds like a very neat solution.
>>
>> I believe domain pause will not work here - a domain cannot pause
>> itself, i.e. current domain cannot be paused. This functionality
>> seems to assume that a domain is pausing another domain. Or I missed
>> the point.
>
> Yes domain pause/unpause will not work. However, you can introduce a
> boolean to tell you whether the domain was suspend.
>
> I actually quite like how suspend work for x86 HVM. This is based on
> pause/unpause. Have a look at hvm_s3_{suspend/resume}.

That only exists because the ACPI controller is/was implemented in
QEMU.  I wouldn't recommend copying it.

Having spent some more time thinking about this problem, what properties
does the PSCI call have?

I gather from other parts of this thread that there may be a partial
reset of state?  Beyond that, what else?

I ask, because conceptually, domU suspend to RAM is literally just
"de-schedule until we want to wake up", and in this case, it appears to
be "wake up on any of the still-active interrupts".  We've already got a
scheduler API for that, and its called vcpu_block().  This already
exists with "wait until a new event occurs" semantics.

Is there anything else I've missed?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

(+ Andre)

On 11/15/18 12:47 AM, Andrew Cooper wrote:
> On 14/11/2018 12:49, Julien Grall wrote:
>> Hi Mirela,
>>
>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>
>>>
>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>> Hi Andrew,
>>>>>
>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>>          if ( is_idle_vcpu(p) )
>>>>>>>>>              return;
>>>>>>>>>
>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>> suspended */
>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>> +        return;
>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>>> state
>>>>>>>> to be saved on suspend.
>>>>>>>>
>>>>>>> We just need a flag to mark a domain as suspended, and I do believe
>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>> Let's come back on this.
>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
>>>>>> tree and you'll find several pieces of documentation, including the
>>>>>> description of what this shutdown code means.
>>>>>>
>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>> with
>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>> Xen's
>>>>>> shutdown infrastructure, which exists mainly to communicate with the
>>>>>> toolstack.
>>>>> Would domain pause/unpause be a better solution?
>>>> Actually yes - that sounds like a very neat solution.
>>>
>>> I believe domain pause will not work here - a domain cannot pause
>>> itself, i.e. current domain cannot be paused. This functionality
>>> seems to assume that a domain is pausing another domain. Or I missed
>>> the point.
>>
>> Yes domain pause/unpause will not work. However, you can introduce a
>> boolean to tell you whether the domain was suspend.
>>
>> I actually quite like how suspend work for x86 HVM. This is based on
>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
> 
> That only exists because the ACPI controller is/was implemented in
> QEMU.  I wouldn't recommend copying it.

May I ask why? I don't think the properties are very different from Arm 
here.

> 
> Having spent some more time thinking about this problem, what properties
> does the PSCI call have?
> 
> I gather from other parts of this thread that there may be a partial
> reset of state?  Beyond that, what else?
> 
> I ask, because conceptually, domU suspend to RAM is literally just
> "de-schedule until we want to wake up", and in this case, it appears to
> be "wake up on any of the still-active interrupts".  We've already got a
> scheduler API for that, and its called vcpu_block().  This already
> exists with "wait until a new event occurs" semantics.
> 
> Is there anything else I've missed?

All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also, only 
events on that vCPU can trigger a resume. All the other event should not 
be taken into account.

My worry with vcpu_block() is we don't prevent the other vCPUs to run. 
Technically they should be off, but I would like some safety to avoid 
any potential corner case (i.e other way to turn a vCPU on).

That's why I think domain_pause() is more suitable in that case. We 
could unpause the domain either when receiving an interrupt or when 
requested by the toolstack).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Andrew Cooper]

On 15/11/2018 10:13, Julien Grall wrote:
> (+ Andre)
>
> On 11/15/18 12:47 AM, Andrew Cooper wrote:
>> On 14/11/2018 12:49, Julien Grall wrote:
>>> Hi Mirela,
>>>
>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>>
>>>>
>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>>> Hi Andrew,
>>>>>>
>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>>>          if ( is_idle_vcpu(p) )
>>>>>>>>>>              return;
>>>>>>>>>>
>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>>> suspended */
>>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>>> +        return;
>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>>>> state
>>>>>>>>> to be saved on suspend.
>>>>>>>>>
>>>>>>>> We just need a flag to mark a domain as suspended, and I do
>>>>>>>> believe
>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>>> Let's come back on this.
>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
>>>>>>> Xen
>>>>>>> tree and you'll find several pieces of documentation, including the
>>>>>>> description of what this shutdown code means.
>>>>>>>
>>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>>> with
>>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>>> Xen's
>>>>>>> shutdown infrastructure, which exists mainly to communicate with
>>>>>>> the
>>>>>>> toolstack.
>>>>>> Would domain pause/unpause be a better solution?
>>>>> Actually yes - that sounds like a very neat solution.
>>>>
>>>> I believe domain pause will not work here - a domain cannot pause
>>>> itself, i.e. current domain cannot be paused. This functionality
>>>> seems to assume that a domain is pausing another domain. Or I missed
>>>> the point.
>>>
>>> Yes domain pause/unpause will not work. However, you can introduce a
>>> boolean to tell you whether the domain was suspend.
>>>
>>> I actually quite like how suspend work for x86 HVM. This is based on
>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
>>
>> That only exists because the ACPI controller is/was implemented in
>> QEMU.  I wouldn't recommend copying it.
>
> May I ask why? I don't think the properties are very different from
> Arm here.

If you observe, that can only be actioned by a hypercall from qemu.  It
can't be actioned by an ACPI controller emulated by Xen.

>
>>
>> Having spent some more time thinking about this problem, what properties
>> does the PSCI call have?
>>
>> I gather from other parts of this thread that there may be a partial
>> reset of state?  Beyond that, what else?
>>
>> I ask, because conceptually, domU suspend to RAM is literally just
>> "de-schedule until we want to wake up", and in this case, it appears to
>> be "wake up on any of the still-active interrupts".  We've already got a
>> scheduler API for that, and its called vcpu_block().  This already
>> exists with "wait until a new event occurs" semantics.
>>
>> Is there anything else I've missed?
>
> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
> only events on that vCPU can trigger a resume. All the other event
> should not be taken into account.
>
> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
> Technically they should be off, but I would like some safety to avoid
> any potential corner case (i.e other way to turn a vCPU on).

Why not have the SYSTEM_SUSPEND check that all other vCPUs are DOWN
first, and fail the call if not?

If instead of waiting for any event, you need to wait for a specific
event, there is also vcpu_poll() which is a related scheduler API.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>
> On 15/11/2018 10:13, Julien Grall wrote:
> > (+ Andre)
> >
> > On 11/15/18 12:47 AM, Andrew Cooper wrote:
> >> On 14/11/2018 12:49, Julien Grall wrote:
> >>> Hi Mirela,
> >>>
> >>> On 14/11/2018 12:08, Mirela Simonovic wrote:
> >>>>
> >>>>
> >>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> >>>>> On 12/11/2018 19:56, Julien Grall wrote:
> >>>>>> Hi Andrew,
> >>>>>>
> >>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
> >>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
> >>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>>>>>>>> index e594b48d81..7f8105465c 100644
> >>>>>>>>>> --- a/xen/arch/arm/domain.c
> >>>>>>>>>> +++ b/xen/arch/arm/domain.c
> >>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>>>>>>>>          if ( is_idle_vcpu(p) )
> >>>>>>>>>>              return;
> >>>>>>>>>>
> >>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
> >>>>>>>>>> suspended */
> >>>>>>>>>> +    if ( p->domain->is_shut_down &&
> >>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> >>>>>>>>>> +        return;
> >>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
> >>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
> >>>>>>>>> state
> >>>>>>>>> to be saved on suspend.
> >>>>>>>>>
> >>>>>>>> We just need a flag to mark a domain as suspended, and I do
> >>>>>>>> believe
> >>>>>>>> SHUTDOWN_suspend is not used anywhere else.
> >>>>>>>> Let's come back on this.
> >>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
> >>>>>>> Xen
> >>>>>>> tree and you'll find several pieces of documentation, including the
> >>>>>>> description of what this shutdown code means.
> >>>>>>>
> >>>>>>> What you are introducing here is not a shutdown - it is a suspend
> >>>>>>> with
> >>>>>>> the intent to resume executing later.  As such, it shouldn't use
> >>>>>>> Xen's
> >>>>>>> shutdown infrastructure, which exists mainly to communicate with
> >>>>>>> the
> >>>>>>> toolstack.
> >>>>>> Would domain pause/unpause be a better solution?
> >>>>> Actually yes - that sounds like a very neat solution.
> >>>>
> >>>> I believe domain pause will not work here - a domain cannot pause
> >>>> itself, i.e. current domain cannot be paused. This functionality
> >>>> seems to assume that a domain is pausing another domain. Or I missed
> >>>> the point.
> >>>
> >>> Yes domain pause/unpause will not work. However, you can introduce a
> >>> boolean to tell you whether the domain was suspend.
> >>>
> >>> I actually quite like how suspend work for x86 HVM. This is based on
> >>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
> >>
> >> That only exists because the ACPI controller is/was implemented in
> >> QEMU.  I wouldn't recommend copying it.
> >
> > May I ask why? I don't think the properties are very different from
> > Arm here.
>
> If you observe, that can only be actioned by a hypercall from qemu.  It
> can't be actioned by an ACPI controller emulated by Xen.
>
> >
> >>
> >> Having spent some more time thinking about this problem, what properties
> >> does the PSCI call have?
> >>
> >> I gather from other parts of this thread that there may be a partial
> >> reset of state?  Beyond that, what else?
> >>
> >> I ask, because conceptually, domU suspend to RAM is literally just
> >> "de-schedule until we want to wake up", and in this case, it appears to
> >> be "wake up on any of the still-active interrupts".  We've already got a
> >> scheduler API for that, and its called vcpu_block().  This already
> >> exists with "wait until a new event occurs" semantics.
> >>
> >> Is there anything else I've missed?
> >

That's correct, and I agree. BTW that is what's implemented in this series.

> > All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
> > only events on that vCPU can trigger a resume. All the other event
> > should not be taken into account.
> >

What other events are talking about here?

> > My worry with vcpu_block() is we don't prevent the other vCPUs to run.
> > Technically they should be off, but I would like some safety to avoid
> > any potential corner case (i.e other way to turn a vCPU on).
>

Other vCPUs are hot-unplugged (offlined) by the guest. If that is not
the case, SYSTEM_SUSPEND will return an error.
Could you please clarify what a potential corner case would be?

> Why not have the SYSTEM_SUSPEND check that all other vCPUs are DOWN
> first, and fail the call if not?
>

This is also already done.

> If instead of waiting for any event, you need to wait for a specific
> event, there is also vcpu_poll() which is a related scheduler API.
>
> ~Andrew

Some content disappeared, so I'll write here to avoid thread branching.

The semantic of vCPU block and domain_pause is not the same. When
guest suspends the domain is not (and should not be) paused, instead
its last online vCPU is blocked waiting on an interrupt. That's it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Andrew,

On 11/15/18 10:26 AM, Andrew Cooper wrote:
> On 15/11/2018 10:13, Julien Grall wrote:
>> (+ Andre)
>>
>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
>>> On 14/11/2018 12:49, Julien Grall wrote:
>>>> Hi Mirela,
>>>>
>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>>>
>>>>>
>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>>>>           if ( is_idle_vcpu(p) )
>>>>>>>>>>>               return;
>>>>>>>>>>>
>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>>>> suspended */
>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>>>> +        return;
>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>>>>> state
>>>>>>>>>> to be saved on suspend.
>>>>>>>>>>
>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
>>>>>>>>> believe
>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>>>> Let's come back on this.
>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
>>>>>>>> Xen
>>>>>>>> tree and you'll find several pieces of documentation, including the
>>>>>>>> description of what this shutdown code means.
>>>>>>>>
>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>>>> with
>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>>>> Xen's
>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
>>>>>>>> the
>>>>>>>> toolstack.
>>>>>>> Would domain pause/unpause be a better solution?
>>>>>> Actually yes - that sounds like a very neat solution.
>>>>>
>>>>> I believe domain pause will not work here - a domain cannot pause
>>>>> itself, i.e. current domain cannot be paused. This functionality
>>>>> seems to assume that a domain is pausing another domain. Or I missed vC
>>>>> the point.
>>>>
>>>> Yes domain pause/unpause will not work. However, you can introduce a
>>>> boolean to tell you whether the domain was suspend.
>>>>
>>>> I actually quite like how suspend work for x86 HVM. This is based on
>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
>>>
>>> That only exists because the ACPI controller is/was implemented in
>>> QEMU.  I wouldn't recommend copying it.
>>
>> May I ask why? I don't think the properties are very different from
>> Arm here.
> 
> If you observe, that can only be actioned by a hypercall from qemu.  It
> can't be actioned by an ACPI controller emulated by Xen.
> 
>>
>>>
>>> Having spent some more time thinking about this problem, what properties
>>> does the PSCI call have?
>>>
>>> I gather from other parts of this thread that there may be a partial
>>> reset of state?  Beyond that, what else?
>>>
>>> I ask, because conceptually, domU suspend to RAM is literally just
>>> "de-schedule until we want to wake up", and in this case, it appears to
>>> be "wake up on any of the still-active interrupts".  We've already got a
>>> scheduler API for that, and its called vcpu_block().  This already
>>> exists with "wait until a new event occurs" semantics.
>>>
>>> Is there anything else I've missed?
>>
>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
>> only events on that vCPU can trigger a resume. All the other event
>> should not be taken into account.
>>
>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
>> Technically they should be off, but I would like some safety to avoid
>> any potential corner case (i.e other way to turn a vCPU on).
> 
> Why not have the SYSTEM_SUSPEND check that all other vCPUs are DOWN
> first, and fail the call if not?

The code already check for that. My concern is there might (today or in 
the future) other bits of Xen that can potentially turn on the vCPU (e.g 
the toolstack).

> 
> If instead of waiting for any event, you need to wait for a specific
> event, there is also vcpu_poll() which is a related scheduler API.
I guess we need to agree how what kind of event can resume the guest 
from suspend/resume. I am not convinced that all events should be equal 
here.

By vcpu_poll, do you mean do_poll?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Andrew,

On 11/15/18 10:26 AM, Andrew Cooper wrote:
> On 15/11/2018 10:13, Julien Grall wrote:
>> (+ Andre)
>>
>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
>>> On 14/11/2018 12:49, Julien Grall wrote:
>>>> Hi Mirela,
>>>>
>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>>>
>>>>>
>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>>>>           if ( is_idle_vcpu(p) )
>>>>>>>>>>>               return;
>>>>>>>>>>>
>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>>>> suspended */
>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>>>> +        return;
>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>>>>> state
>>>>>>>>>> to be saved on suspend.
>>>>>>>>>>
>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
>>>>>>>>> believe
>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>>>> Let's come back on this.
>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
>>>>>>>> Xen
>>>>>>>> tree and you'll find several pieces of documentation, including the
>>>>>>>> description of what this shutdown code means.
>>>>>>>>
>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>>>> with
>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>>>> Xen's
>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
>>>>>>>> the
>>>>>>>> toolstack.
>>>>>>> Would domain pause/unpause be a better solution?
>>>>>> Actually yes - that sounds like a very neat solution.
>>>>>
>>>>> I believe domain pause will not work here - a domain cannot pause
>>>>> itself, i.e. current domain cannot be paused. This functionality
>>>>> seems to assume that a domain is pausing another domain. Or I missed vC
>>>>> the point.
>>>>
>>>> Yes domain pause/unpause will not work. However, you can introduce a
>>>> boolean to tell you whether the domain was suspend.
>>>>
>>>> I actually quite like how suspend work for x86 HVM. This is based on
>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
>>>
>>> That only exists because the ACPI controller is/was implemented in
>>> QEMU.  I wouldn't recommend copying it.
>>
>> May I ask why? I don't think the properties are very different from
>> Arm here.
> 
> If you observe, that can only be actioned by a hypercall from qemu.  It
> can't be actioned by an ACPI controller emulated by Xen.

How does the ACPI controller emulated by Xen deal with suspend/resume? 
Do you have any pointer?

> 
>>
>>>
>>> Having spent some more time thinking about this problem, what properties
>>> does the PSCI call have?
>>>
>>> I gather from other parts of this thread that there may be a partial
>>> reset of state?  Beyond that, what else?
>>>
>>> I ask, because conceptually, domU suspend to RAM is literally just
>>> "de-schedule until we want to wake up", and in this case, it appears to
>>> be "wake up on any of the still-active interrupts".  We've already got a
>>> scheduler API for that, and its called vcpu_block().  This already
>>> exists with "wait until a new event occurs" semantics.
>>>
>>> Is there anything else I've missed?
>>
>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
>> only events on that vCPU can trigger a resume. All the other event
>> should not be taken into account.
>>
>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
>> Technically they should be off, but I would like some safety to avoid
>> any potential corner case (i.e other way to turn a vCPU on).
> 
> Why not have the SYSTEM_SUSPEND check that all other vCPUs are DOWN
> first, and fail the call if not?

The code already check for that. My concern is there might (today or in 
the future) other bits of Xen that can potentially turn on the vCPU (e.g 
the toolstack).

> 
> If instead of waiting for any event, you need to wait for a specific
> event, there is also vcpu_poll() which is a related scheduler API.
I guess we need to agree how what kind of event can resume the guest 
from suspend/resume. I am not convinced that all events should be equal 
here.

By vcpu_poll, do you mean do_poll?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Andrew Cooper]

On 15/11/2018 10:36, Julien Grall wrote:
> Hi Andrew,
>
> On 11/15/18 10:26 AM, Andrew Cooper wrote:
>> On 15/11/2018 10:13, Julien Grall wrote:
>>> (+ Andre)
>>>
>>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
>>>> On 14/11/2018 12:49, Julien Grall wrote:
>>>>> Hi Mirela,
>>>>>
>>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>>>>
>>>>>>
>>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu
>>>>>>>>>>>> *p)
>>>>>>>>>>>>           if ( is_idle_vcpu(p) )
>>>>>>>>>>>>               return;
>>>>>>>>>>>>
>>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>>>>> suspended */
>>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>>>>> +        return;
>>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all
>>>>>>>>>>> the
>>>>>>>>>>> state
>>>>>>>>>>> to be saved on suspend.
>>>>>>>>>>>
>>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
>>>>>>>>>> believe
>>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>>>>> Let's come back on this.
>>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
>>>>>>>>> Xen
>>>>>>>>> tree and you'll find several pieces of documentation,
>>>>>>>>> including the
>>>>>>>>> description of what this shutdown code means.
>>>>>>>>>
>>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>>>>> with
>>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>>>>> Xen's
>>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
>>>>>>>>> the
>>>>>>>>> toolstack.
>>>>>>>> Would domain pause/unpause be a better solution?
>>>>>>> Actually yes - that sounds like a very neat solution.
>>>>>>
>>>>>> I believe domain pause will not work here - a domain cannot pause
>>>>>> itself, i.e. current domain cannot be paused. This functionality
>>>>>> seems to assume that a domain is pausing another domain. Or I
>>>>>> missed vC
>>>>>> the point.
>>>>>
>>>>> Yes domain pause/unpause will not work. However, you can introduce a
>>>>> boolean to tell you whether the domain was suspend.
>>>>>
>>>>> I actually quite like how suspend work for x86 HVM. This is based on
>>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
>>>>
>>>> That only exists because the ACPI controller is/was implemented in
>>>> QEMU.  I wouldn't recommend copying it.
>>>
>>> May I ask why? I don't think the properties are very different from
>>> Arm here.
>>
>> If you observe, that can only be actioned by a hypercall from qemu.  It
>> can't be actioned by an ACPI controller emulated by Xen.
>
> How does the ACPI controller emulated by Xen deal with suspend/resume?
> Do you have any pointer?

It doesn't, and that is one of the outstanding issues for trying to make
it work sensibly.  In the end it wasn't merged, and is still on
someone’s TODO list.

>
>>
>>>
>>>>
>>>> Having spent some more time thinking about this problem, what
>>>> properties
>>>> does the PSCI call have?
>>>>
>>>> I gather from other parts of this thread that there may be a partial
>>>> reset of state?  Beyond that, what else?
>>>>
>>>> I ask, because conceptually, domU suspend to RAM is literally just
>>>> "de-schedule until we want to wake up", and in this case, it
>>>> appears to
>>>> be "wake up on any of the still-active interrupts".  We've already
>>>> got a
>>>> scheduler API for that, and its called vcpu_block().  This already
>>>> exists with "wait until a new event occurs" semantics.
>>>>
>>>> Is there anything else I've missed?
>>>
>>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
>>> only events on that vCPU can trigger a resume. All the other event
>>> should not be taken into account.
>>>
>>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
>>> Technically they should be off, but I would like some safety to avoid
>>> any potential corner case (i.e other way to turn a vCPU on).
>>
>> Why not have the SYSTEM_SUSPEND check that all other vCPUs are DOWN
>> first, and fail the call if not?
>
> The code already check for that. My concern is there might (today or
> in the future) other bits of Xen that can potentially turn on the vCPU
> (e.g the toolstack).

I wouldn't be concerned - there is nothing you can do about it.

Because of things like domain construction / migration / emulation /
etc, the toolstack(/control domain) has all the low level hooks into Xen
to do everything.  Your concern here about the toolstack onlining other
vcpus is just as problematic as the toolstack issuing an unpause
hypercall, or mapping part of the guest and splatting /dev/random
everywhere.

Any state you can arrange in Xen can be undone/altered by the toolstack,
and you've got to accept that the toolstack can do this, and won't,
because it is a trusted piece of the system and has a vested interest in
the guest not crashing.

Beyond that, you'll want to use the most appropriate interfaces within
Xen to implement this, and it very much sounds like vcpu_block() has the
semantics that you want.

>
>>
>> If instead of waiting for any event, you need to wait for a specific
>> event, there is also vcpu_poll() which is a related scheduler API.
> I guess we need to agree how what kind of event can resume the guest
> from suspend/resume. I am not convinced that all events should be
> equal here.
>
> By vcpu_poll, do you mean do_poll?

That is the hypercall, but there is a variation of it for internal use.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Mirela,

On 11/15/18 10:33 AM, Mirela Simonovic wrote:
> On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
> <andrew.cooper3@citrix.com> wrote:
>>
>> On 15/11/2018 10:13, Julien Grall wrote:
>>> (+ Andre)
>>>
>>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
>>>> On 14/11/2018 12:49, Julien Grall wrote:
>>>>> Hi Mirela,
>>>>>
>>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>>>>
>>>>>>
>>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>>>>>           if ( is_idle_vcpu(p) )
>>>>>>>>>>>>               return;
>>>>>>>>>>>>
>>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>>>>> suspended */
>>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>>>>> +        return;
>>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>>>>>> state
>>>>>>>>>>> to be saved on suspend.
>>>>>>>>>>>
>>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
>>>>>>>>>> believe
>>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>>>>> Let's come back on this.
>>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
>>>>>>>>> Xen
>>>>>>>>> tree and you'll find several pieces of documentation, including the
>>>>>>>>> description of what this shutdown code means.
>>>>>>>>>
>>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>>>>> with
>>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>>>>> Xen's
>>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
>>>>>>>>> the
>>>>>>>>> toolstack.
>>>>>>>> Would domain pause/unpause be a better solution?
>>>>>>> Actually yes - that sounds like a very neat solution.
>>>>>>
>>>>>> I believe domain pause will not work here - a domain cannot pause
>>>>>> itself, i.e. current domain cannot be paused. This functionality
>>>>>> seems to assume that a domain is pausing another domain. Or I missed
>>>>>> the point.
>>>>>
>>>>> Yes domain pause/unpause will not work. However, you can introduce a
>>>>> boolean to tell you whether the domain was suspend.
>>>>>
>>>>> I actually quite like how suspend work for x86 HVM. This is based on
>>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
>>>>
>>>> That only exists because the ACPI controller is/was implemented in
>>>> QEMU.  I wouldn't recommend copying it.
>>>
>>> May I ask why? I don't think the properties are very different from
>>> Arm here.
>>
>> If you observe, that can only be actioned by a hypercall from qemu.  It
>> can't be actioned by an ACPI controller emulated by Xen.
>>
>>>
>>>>
>>>> Having spent some more time thinking about this problem, what properties
>>>> does the PSCI call have?
>>>>
>>>> I gather from other parts of this thread that there may be a partial
>>>> reset of state?  Beyond that, what else?
>>>>
>>>> I ask, because conceptually, domU suspend to RAM is literally just
>>>> "de-schedule until we want to wake up", and in this case, it appears to
>>>> be "wake up on any of the still-active interrupts".  We've already got a
>>>> scheduler API for that, and its called vcpu_block().  This already
>>>> exists with "wait until a new event occurs" semantics.
>>>>
>>>> Is there anything else I've missed?
>>>
> 
> That's correct, and I agree. BTW that is what's implemented in this series.
> 
>>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
>>> only events on that vCPU can trigger a resume. All the other event
>>> should not be taken into account.
>>>
> 
> What other events are talking about here?

vcpu_unblock is not only called when you receive interrupts. It can be 
called in other place when you receive an events.

 From the Arm Arm, an event can be anything. So do we really want to 
wake-up on any events?

> 
>>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
>>> Technically they should be off, but I would like some safety to avoid
>>> any potential corner case (i.e other way to turn a vCPU on).
>>
> 
> Other vCPUs are hot-unplugged (offlined) by the guest. If that is not
> the case, SYSTEM_SUSPEND will return an error.
> Could you please clarify what a potential corner case would be?

PSCI CPU_ON is not the only way to online a vCPU. I merely want to 
prevent other path to play with the vCPU when it is not necessary.

[...]

>> If instead of waiting for any event, you need to wait for a specific
>> event, there is also vcpu_poll() which is a related scheduler API.
>>
>> ~Andrew
> 
> Some content disappeared, so I'll write here to avoid thread branching.
> 
> The semantic of vCPU block and domain_pause is not the same. When
> guest suspends the domain is not (and should not be) paused, instead
> its last online vCPU is blocked waiting on an interrupt. That's it.
Well no, you will block until you receive an event. Interrupts are one 
of them.

Do we want to consider all events as wakeup event?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 11:59 AM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Mirela,
>
> On 11/15/18 10:33 AM, Mirela Simonovic wrote:
> > On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
> > <andrew.cooper3@citrix.com> wrote:
> >>
> >> On 15/11/2018 10:13, Julien Grall wrote:
> >>> (+ Andre)
> >>>
> >>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
> >>>> On 14/11/2018 12:49, Julien Grall wrote:
> >>>>> Hi Mirela,
> >>>>>
> >>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> >>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
> >>>>>>>> Hi Andrew,
> >>>>>>>>
> >>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
> >>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
> >>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>>>>>>>>>> index e594b48d81..7f8105465c 100644
> >>>>>>>>>>>> --- a/xen/arch/arm/domain.c
> >>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
> >>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>>>>>>>>>>           if ( is_idle_vcpu(p) )
> >>>>>>>>>>>>               return;
> >>>>>>>>>>>>
> >>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
> >>>>>>>>>>>> suspended */
> >>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
> >>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> >>>>>>>>>>>> +        return;
> >>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
> >>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
> >>>>>>>>>>> state
> >>>>>>>>>>> to be saved on suspend.
> >>>>>>>>>>>
> >>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
> >>>>>>>>>> believe
> >>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
> >>>>>>>>>> Let's come back on this.
> >>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
> >>>>>>>>> Xen
> >>>>>>>>> tree and you'll find several pieces of documentation, including the
> >>>>>>>>> description of what this shutdown code means.
> >>>>>>>>>
> >>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
> >>>>>>>>> with
> >>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
> >>>>>>>>> Xen's
> >>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
> >>>>>>>>> the
> >>>>>>>>> toolstack.
> >>>>>>>> Would domain pause/unpause be a better solution?
> >>>>>>> Actually yes - that sounds like a very neat solution.
> >>>>>>
> >>>>>> I believe domain pause will not work here - a domain cannot pause
> >>>>>> itself, i.e. current domain cannot be paused. This functionality
> >>>>>> seems to assume that a domain is pausing another domain. Or I missed
> >>>>>> the point.
> >>>>>
> >>>>> Yes domain pause/unpause will not work. However, you can introduce a
> >>>>> boolean to tell you whether the domain was suspend.
> >>>>>
> >>>>> I actually quite like how suspend work for x86 HVM. This is based on
> >>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
> >>>>
> >>>> That only exists because the ACPI controller is/was implemented in
> >>>> QEMU.  I wouldn't recommend copying it.
> >>>
> >>> May I ask why? I don't think the properties are very different from
> >>> Arm here.
> >>
> >> If you observe, that can only be actioned by a hypercall from qemu.  It
> >> can't be actioned by an ACPI controller emulated by Xen.
> >>
> >>>
> >>>>
> >>>> Having spent some more time thinking about this problem, what properties
> >>>> does the PSCI call have?
> >>>>
> >>>> I gather from other parts of this thread that there may be a partial
> >>>> reset of state?  Beyond that, what else?
> >>>>
> >>>> I ask, because conceptually, domU suspend to RAM is literally just
> >>>> "de-schedule until we want to wake up", and in this case, it appears to
> >>>> be "wake up on any of the still-active interrupts".  We've already got a
> >>>> scheduler API for that, and its called vcpu_block().  This already
> >>>> exists with "wait until a new event occurs" semantics.
> >>>>
> >>>> Is there anything else I've missed?
> >>>
> >
> > That's correct, and I agree. BTW that is what's implemented in this series.
> >
> >>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
> >>> only events on that vCPU can trigger a resume. All the other event
> >>> should not be taken into account.
> >>>
> >
> > What other events are talking about here?
>
> vcpu_unblock is not only called when you receive interrupts. It can be
> called in other place when you receive an events.
>
>  From the Arm Arm, an event can be anything. So do we really want to
> wake-up on any events?
>
> >
> >>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
> >>> Technically they should be off, but I would like some safety to avoid
> >>> any potential corner case (i.e other way to turn a vCPU on).
> >>
> >
> > Other vCPUs are hot-unplugged (offlined) by the guest. If that is not
> > the case, SYSTEM_SUSPEND will return an error.
> > Could you please clarify what a potential corner case would be?
>
> PSCI CPU_ON is not the only way to online a vCPU. I merely want to
> prevent other path to play with the vCPU when it is not necessary.
>
> [...]
>
> >> If instead of waiting for any event, you need to wait for a specific
> >> event, there is also vcpu_poll() which is a related scheduler API.
> >>
> >> ~Andrew
> >
> > Some content disappeared, so I'll write here to avoid thread branching.
> >
> > The semantic of vCPU block and domain_pause is not the same. When
> > guest suspends the domain is not (and should not be) paused, instead
> > its last online vCPU is blocked waiting on an interrupt. That's it.
> Well no, you will block until you receive an event. Interrupts are one
> of them.
>
> Do we want to consider all events as wakeup event?
>

I think we need to assume that events are not triggered via toolstack,
Andrew made a good point.
Given the assumption, my understanding is that Xen itself will not
unblock vCPU, except due to an interrupt targeted to the guest.
Am I missing something? An example would be appreciated. Then we can
say whether those events should wake-up the guest or not.

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi,

On 11/15/18 11:10 AM, Mirela Simonovic wrote:
> Hi Julien,
> 
> On Thu, Nov 15, 2018 at 11:59 AM Julien Grall <julien.grall@arm.com> wrote:
>>
>> Hi Mirela,
>>
>> On 11/15/18 10:33 AM, Mirela Simonovic wrote:
>>> On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
>>> <andrew.cooper3@citrix.com> wrote:
>>>>
>>>> On 15/11/2018 10:13, Julien Grall wrote:
>>>>> (+ Andre)
>>>>>
>>>>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
>>>>>> On 14/11/2018 12:49, Julien Grall wrote:
>>>>>>> Hi Mirela,
>>>>>>>
>>>>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>>>>>>> Hi Andrew,
>>>>>>>>>>
>>>>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>>>>>>>            if ( is_idle_vcpu(p) )
>>>>>>>>>>>>>>                return;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>>>>>>> suspended */
>>>>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>>>>>>> +        return;
>>>>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>>>>>>>> state
>>>>>>>>>>>>> to be saved on suspend.
>>>>>>>>>>>>>
>>>>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
>>>>>>>>>>>> believe
>>>>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>>>>>>> Let's come back on this.
>>>>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
>>>>>>>>>>> Xen
>>>>>>>>>>> tree and you'll find several pieces of documentation, including the
>>>>>>>>>>> description of what this shutdown code means>>>>>>>>>>>
>>>>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>>>>>>> with
>>>>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>>>>>>> Xen's
>>>>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
>>>>>>>>>>> the
>>>>>>>>>>> toolstack.
>>>>>>>>>> Would domain pause/unpause be a better solution?
>>>>>>>>> Actually yes - that sounds like a very neat solution.
>>>>>>>>
>>>>>>>> I believe domain pause will not work here - a domain cannot pause
>>>>>>>> itself, i.e. current domain cannot be paused. This functionality
>>>>>>>> seems to assume that a domain is pausing another domain. Or I missed
>>>>>>>> the point.
>>>>>>>
>>>>>>> Yes domain pause/unpause will not work. However, you can introduce a
>>>>>>> boolean to tell you whether the domain was suspend.
>>>>>>>
>>>>>>> I actually quite like how suspend work for x86 HVM. This is based on
>>>>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
>>>>>>
>>>>>> That only exists because the ACPI controller is/was implemented in
>>>>>> QEMU.  I wouldn't recommend copying it.
>>>>>
>>>>> May I ask why? I don't think the properties are very different from
>>>>> Arm here.
>>>>
>>>> If you observe, that can only be actioned by a hypercall from qemu.  It
>>>> can't be actioned by an ACPI controller emulated by Xen.
>>>>
>>>>>
>>>>>>
>>>>>> Having spent some more time thinking about this problem, what properties
>>>>>> does the PSCI call have?
>>>>>>
>>>>>> I gather from other parts of this thread that there may be a partial
>>>>>> reset of state?  Beyond that, what else?
>>>>>>
>>>>>> I ask, because conceptually, domU suspend to RAM is literally just
>>>>>> "de-schedule until we want to wake up", and in this case, it appears to
>>>>>> be "wake up on any of the still-active interrupts".  We've already got a
>>>>>> scheduler API for that, and its called vcpu_block().  This already
>>>>>> exists with "wait until a new event occurs" semantics.
>>>>>>
>>>>>> Is there anything else I've missed?
>>>>>
>>>
>>> That's correct, and I agree. BTW that is what's implemented in this series.
>>>
>>>>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
>>>>> only events on that vCPU can trigger a resume. All the other event
>>>>> should not be taken into account.
>>>>>
>>>
>>> What other events are talking about here?
>>
>> vcpu_unblock is not only called when you receive interrupts. It can be
>> called in other place when you receive an events.
>>
>>   From the Arm Arm, an event can be anything. So do we really want to
>> wake-up on any events?
>>
>>>
>>>>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
>>>>> Technically they should be off, but I would like some safety to avoid
>>>>> any potential corner case (i.e other way to turn a vCPU on).
>>>>
>>>
>>> Other vCPUs are hot-unplugged (offlined) by the guest. If that is not
>>> the case, SYSTEM_SUSPEND will return an error.
>>> Could you please clarify what a potential corner case would be?
>>
>> PSCI CPU_ON is not the only way to online a vCPU. I merely want to
>> prevent other path to play with the vCPU when it is not necessary.
>>
>> [...]
>>
>>>> If instead of waiting for any event, you need to wait for a specific
>>>> event, there is also vcpu_poll() which is a related scheduler API.
>>>>
>>>> ~Andrew
>>>
>>> Some content disappeared, so I'll write here to avoid thread branching.
>>>
>>> The semantic of vCPU block and domain_pause is not the same. When
>>> guest suspends the domain is not (and should not be) paused, instead
>>> its last online vCPU is blocked waiting on an interrupt. That's it.
>> Well no, you will block until you receive an event. Interrupts are one
>> of them.
>>
>> Do we want to consider all events as wakeup event?
>>
> 
> I think we need to assume that events are not triggered via toolstack,
> Andrew made a good point.

I don't think we are discussing the same thing. The discussion was 
around other vCPUs, not the vCPU calling SYSTEM_SUSPEND.

Most likely in the future, we would want to allow the toolstack to 
request resuming the domain. This can be considered as an event.

> Given the assumption, my understanding is that Xen itself will not
> unblock vCPU, except due to an interrupt targeted to the guest.
> Am I missing something? An example would be appreciated.

At least on Arm, the current semantics of vcpu_block/vcpu_unblock is to 
block until you receive an events.

I don't much want to restrict the definition of events to only 
interrupts.  To clarify my point, if you want to wake-up for any events 
then fine. But this needs to be understood that it may not be only 
interrupts.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 12:38 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi,
>
> On 11/15/18 11:10 AM, Mirela Simonovic wrote:
> > Hi Julien,
> >
> > On Thu, Nov 15, 2018 at 11:59 AM Julien Grall <julien.grall@arm.com> wrote:
> >>
> >> Hi Mirela,
> >>
> >> On 11/15/18 10:33 AM, Mirela Simonovic wrote:
> >>> On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
> >>> <andrew.cooper3@citrix.com> wrote:
> >>>>
> >>>> On 15/11/2018 10:13, Julien Grall wrote:
> >>>>> (+ Andre)
> >>>>>
> >>>>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
> >>>>>> On 14/11/2018 12:49, Julien Grall wrote:
> >>>>>>> Hi Mirela,
> >>>>>>>
> >>>>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> >>>>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
> >>>>>>>>>> Hi Andrew,
> >>>>>>>>>>
> >>>>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
> >>>>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
> >>>>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>>>>>>>>>>>> index e594b48d81..7f8105465c 100644
> >>>>>>>>>>>>>> --- a/xen/arch/arm/domain.c
> >>>>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
> >>>>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>>>>>>>>>>>>            if ( is_idle_vcpu(p) )
> >>>>>>>>>>>>>>                return;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
> >>>>>>>>>>>>>> suspended */
> >>>>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
> >>>>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> >>>>>>>>>>>>>> +        return;
> >>>>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
> >>>>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
> >>>>>>>>>>>>> state
> >>>>>>>>>>>>> to be saved on suspend.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
> >>>>>>>>>>>> believe
> >>>>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
> >>>>>>>>>>>> Let's come back on this.
> >>>>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
> >>>>>>>>>>> Xen
> >>>>>>>>>>> tree and you'll find several pieces of documentation, including the
> >>>>>>>>>>> description of what this shutdown code means>>>>>>>>>>>
> >>>>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
> >>>>>>>>>>> with
> >>>>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
> >>>>>>>>>>> Xen's
> >>>>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
> >>>>>>>>>>> the
> >>>>>>>>>>> toolstack.
> >>>>>>>>>> Would domain pause/unpause be a better solution?
> >>>>>>>>> Actually yes - that sounds like a very neat solution.
> >>>>>>>>
> >>>>>>>> I believe domain pause will not work here - a domain cannot pause
> >>>>>>>> itself, i.e. current domain cannot be paused. This functionality
> >>>>>>>> seems to assume that a domain is pausing another domain. Or I missed
> >>>>>>>> the point.
> >>>>>>>
> >>>>>>> Yes domain pause/unpause will not work. However, you can introduce a
> >>>>>>> boolean to tell you whether the domain was suspend.
> >>>>>>>
> >>>>>>> I actually quite like how suspend work for x86 HVM. This is based on
> >>>>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
> >>>>>>
> >>>>>> That only exists because the ACPI controller is/was implemented in
> >>>>>> QEMU.  I wouldn't recommend copying it.
> >>>>>
> >>>>> May I ask why? I don't think the properties are very different from
> >>>>> Arm here.
> >>>>
> >>>> If you observe, that can only be actioned by a hypercall from qemu.  It
> >>>> can't be actioned by an ACPI controller emulated by Xen.
> >>>>
> >>>>>
> >>>>>>
> >>>>>> Having spent some more time thinking about this problem, what properties
> >>>>>> does the PSCI call have?
> >>>>>>
> >>>>>> I gather from other parts of this thread that there may be a partial
> >>>>>> reset of state?  Beyond that, what else?
> >>>>>>
> >>>>>> I ask, because conceptually, domU suspend to RAM is literally just
> >>>>>> "de-schedule until we want to wake up", and in this case, it appears to
> >>>>>> be "wake up on any of the still-active interrupts".  We've already got a
> >>>>>> scheduler API for that, and its called vcpu_block().  This already
> >>>>>> exists with "wait until a new event occurs" semantics.
> >>>>>>
> >>>>>> Is there anything else I've missed?
> >>>>>
> >>>
> >>> That's correct, and I agree. BTW that is what's implemented in this series.
> >>>
> >>>>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
> >>>>> only events on that vCPU can trigger a resume. All the other event
> >>>>> should not be taken into account.
> >>>>>
> >>>
> >>> What other events are talking about here?
> >>
> >> vcpu_unblock is not only called when you receive interrupts. It can be
> >> called in other place when you receive an events.
> >>
> >>   From the Arm Arm, an event can be anything. So do we really want to
> >> wake-up on any events?
> >>
> >>>
> >>>>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
> >>>>> Technically they should be off, but I would like some safety to avoid
> >>>>> any potential corner case (i.e other way to turn a vCPU on).
> >>>>
> >>>
> >>> Other vCPUs are hot-unplugged (offlined) by the guest. If that is not
> >>> the case, SYSTEM_SUSPEND will return an error.
> >>> Could you please clarify what a potential corner case would be?
> >>
> >> PSCI CPU_ON is not the only way to online a vCPU. I merely want to
> >> prevent other path to play with the vCPU when it is not necessary.
> >>
> >> [...]
> >>
> >>>> If instead of waiting for any event, you need to wait for a specific
> >>>> event, there is also vcpu_poll() which is a related scheduler API.
> >>>>
> >>>> ~Andrew
> >>>
> >>> Some content disappeared, so I'll write here to avoid thread branching.
> >>>
> >>> The semantic of vCPU block and domain_pause is not the same. When
> >>> guest suspends the domain is not (and should not be) paused, instead
> >>> its last online vCPU is blocked waiting on an interrupt. That's it.
> >> Well no, you will block until you receive an event. Interrupts are one
> >> of them.
> >>
> >> Do we want to consider all events as wakeup event?
> >>
> >
> > I think we need to assume that events are not triggered via toolstack,
> > Andrew made a good point.
>
> I don't think we are discussing the same thing. The discussion was
> around other vCPUs, not the vCPU calling SYSTEM_SUSPEND.
>
> Most likely in the future, we would want to allow the toolstack to
> request resuming the domain. This can be considered as an event.
>

Yes, such an event will unblock the vcpu and cause the domain to
resume. So from this perspective it's not only an interrupt targeted
to the guest.

> > Given the assumption, my understanding is that Xen itself will not
> > unblock vCPU, except due to an interrupt targeted to the guest.
> > Am I missing something? An example would be appreciated.
>
> At least on Arm, the current semantics of vcpu_block/vcpu_unblock is to
> block until you receive an events.
>
> I don't much want to restrict the definition of events to only
> interrupts.  To clarify my point, if you want to wake-up for any events
> then fine. But this needs to be understood that it may not be only
> interrupts.
>

In the context described above this is fine - events are not only interrupts.

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 7:48 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi,
>
> On 14/11/2018 17:35, Mirela Simonovic wrote:
> > On Wed, Nov 14, 2018 at 6:10 PM Julien Grall <julien.grall@arm.com> wrote:
> >> On 14/11/2018 15:40, Mirela Simonovic wrote:
> >>> On Wed, Nov 14, 2018 at 4:07 PM Julien Grall <julien.grall@arm.com> wrote:
> >>>> On 12/11/2018 11:30, Mirela Simonovic wrote:
> >>>>> When Dom0 finalizes its suspend procedure the suspend of Xen is
> >>>>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> >>>>> its boot core (VCPU#0), which could be mapped to any physical CPU,
> >>>>> i.e. the system_suspend() function could be executed by any physical
> >>>>> CPU. Since Xen suspend procedure has to be run by the boot CPU
> >>>>> (non-boot CPUs will be disabled at some point in suspend procedure),
> >>>>> system_suspend() execution has to continue on CPU#0.
> >>>>>
> >>>>> When the system_suspend() returns 0, it means that the system was
> >>>>> suspended and it is coming out of the resume procedure. Regardless
> >>>>> of the system_suspend() return value, after this function returns
> >>>>> Xen is fully functional, and its state, including all devices and data
> >>>>> structures, matches the state prior to calling system_suspend().
> >>>>> The status is returned by system_suspend() for debugging/logging
> >>>>> purposes and function prototype compatibility.
> >>>>>
> >>>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> >>>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >>>>> ---
> >>>>>     xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
> >>>>>     1 file changed, 34 insertions(+)
> >>>>>
> >>>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> >>>>> index f2338e41db..21b45f8248 100644
> >>>>> --- a/xen/arch/arm/suspend.c
> >>>>> +++ b/xen/arch/arm/suspend.c
> >>>>> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
> >>>>>         _arch_set_info_guest(v, &ctxt);
> >>>>>     }
> >>>>>
> >>>>> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> >>>>> +static long system_suspend(void *data)
> >>>>> +{
> >>>>> +    BUG_ON(system_state != SYS_STATE_active);
> >>>>> +
> >>>>> +    return -ENOSYS;
> >>>>> +}
> >>>>> +
> >>>>>     int32_t domain_suspend(register_t epoint, register_t cid)
> >>>>>     {
> >>>>>         struct vcpu *v;
> >>>>>         struct domain *d = current->domain;
> >>>>>         bool is_thumb = epoint & 1;
> >>>>> +    int status;
> >>>>>
> >>>>>         dprintk(XENLOG_DEBUG,
> >>>>>                 "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> >>>>> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
> >>>>>          */
> >>>>>         vcpu_block_unless_event_pending(current);
> >>>>>
> >>>>> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
> >>>>> +    if ( is_hardware_domain(d) )
> >>>>> +    {
> >>>>> +        /*
> >>>>> +         * system_suspend should be called when Dom0 finalizes the suspend
> >>>>> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
> >>>>> +         * be mapped to any PCPU (this function could be executed by any PCPU).
> >>>>> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
> >>>>> +         * PCPUs will be disabled during the suspend).
> >>>>> +         */
> >>>>> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> >>>>
> >>>> Based on my comment in patch #2, I don't think this will do the correct thing on
> >>>> Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
> >>>> do_vpsci_0_2_call.
> >>>>
> >>>
> >>> Could you please explain? I can't follow
> >>
> >> General purpose (e.g xN, pc) registers live at the bottom of the vCPU stack. The
> >> function vcpu_suspend will reset all of them to 0 but x0 (Context ID) and pc
> >> (entry point).
> >>
> >> You rely on those registers to be untouched in the return path. However, this is
> >> not the case. If you look at do_vpsci_0_2_call, x0 will be set to whatever is
> >> the return value of domain_suspend (e.g PSCI_SUSPEND). So x0 will not contain
> >> anymore the Context ID as expected by the guest.
> >
> > This is expected, the system should behave that way. If the guest
> > managed to suspend, i.e. domain_suspend has returned PSCI_SUCCESS, the
> > return value is meaningless to the guest because it won't return to
> > it. The guest has suspended, just the fact the it will start from an
> > another entry point implicitly carries the information that the
> > suspend was successful.
> > However, if the return value from domain_suspend is an error then the
> > error will be returned to the guest because the suspend has failed.
> > Then the x0 register should contain error code, not the context ID.
> > The PC should be untouched, i.e. it should not contain the resume
> > entry point, but whatever was in there once the hvc/system_suspend was
> > issued by the guest. Guests handle errors right below the code from
> > which they tried to suspend.
>
> I think you misunderstood my comment. I am not speaking about the failure case
> but the success case where the domain is actually suspended for some time.
>
> When the domain is resuming, the "boot" vCPU should see Context ID and not
> PSCI_SUCCESS.
>
> However, this is not the case because of the following path:
>
>         -> do_vpsci_0_2_call
>                 -> domain_suspend
>                         -> vcpu_suspend
>                                 -> x0 = Context ID
>                         -> return PSCI_SUCCESS
>                 -> x0 = PSCI_SUCCESS
>
> At some point in the future your guest will resume. Instead of seen Context ID,
> it will actually see PSCI_SUCCESS. You can easily test that by modifying the
> SYSTEM_SUSPEND code in Linux to use a different context ID and add hvc #0xffe0
> to dump register x0.
>

You're right, x0 get overwritten by set_user_reg after the
do_psci_1_0_system_suspend returns. This needs to be fixed, thanks.

> >
> >>
> >> You probably haven't noticed it because Linux is currently using 0 for the
> >> context ID. This is the same value as PSCI_SUCCESS.
> >>
> >> In the case of Dom0, this is a bit different to what I explained in my previous
> >> e-mail because I got confused. The function continue_hypercall_on_cpu is not
> >> doing what you expect. The function will pause the vCPU, schedule the tasklet
> >> and then return directly.
> >>
> >> At some point the tasklet will get scheduled on CPU#0 and system_suspend will be
> >> called. The return value of that function will be copied to x0. The vCPU will
> >> then get unpaused and continue to run.
> >>
> >> So x0 will not contain the Context ID but whatever system_suspend return.
> >>
> >
> > I agree with all you described above, but that is intended - the
> > system should behave that way. I believe the cause of misunderstanding
> > could be in how the return value versus context ID is handled. Those
> > are different paths, and only one of the following executes: 1) either
> > the suspend is successful and nothing will be returned to the guest
> > because it is suspended; or 2) the suspend of the guest has failed so
> > context ID is useless.
>
> You missed my point here, your guest will resume at some point. As you reset the
> vCPU and set x0 in vcpu_suspend. Then anything after can overwrite the registers
> you set in vcpu_suspend.
>
> Please explain why you think your code behave differently that I wrote when
> resuming.
>
> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

Hi Mirela,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> When Dom0 finalizes its suspend procedure the suspend of Xen is
> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> its boot core (VCPU#0), which could be mapped to any physical CPU,
> i.e. the system_suspend() function could be executed by any physical
> CPU. Since Xen suspend procedure has to be run by the boot CPU
> (non-boot CPUs will be disabled at some point in suspend procedure),
> system_suspend() execution has to continue on CPU#0.
> 
> When the system_suspend() returns 0, it means that the system was
> suspended and it is coming out of the resume procedure. Regardless
> of the system_suspend() return value, after this function returns
> Xen is fully functional, and its state, including all devices and data
> structures, matches the state prior to calling system_suspend().
> The status is returned by system_suspend() for debugging/logging
> purposes and function prototype compatibility.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
>   1 file changed, 34 insertions(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index f2338e41db..21b45f8248 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>       _arch_set_info_guest(v, &ctxt);
>   }
>   
> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> +static long system_suspend(void *data)
> +{
> +    BUG_ON(system_state != SYS_STATE_active);
> +
> +    return -ENOSYS;
> +}
> +
>   int32_t domain_suspend(register_t epoint, register_t cid)
>   {
>       struct vcpu *v;
>       struct domain *d = current->domain;
>       bool is_thumb = epoint & 1;
> +    int status;
>   
>       dprintk(XENLOG_DEBUG,
>               "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
>        */
>       vcpu_block_unless_event_pending(current);
>   
> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
> +    if ( is_hardware_domain(d) )
> +    {
> +        /*
> +         * system_suspend should be called when Dom0 finalizes the suspend
> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
> +         * be mapped to any PCPU (this function could be executed by any PCPU).
> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
> +         * PCPUs will be disabled during the suspend).
> +         */
> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> +        /*
> +         * If an error happened, there is nothing that needs to be done here
> +         * because the system_suspend always returns in fully functional state
> +         * no matter what the outcome of suspend procedure is. If the system
> +         * suspended successfully the function will return 0 after the resume.
> +         * Otherwise, if an error is returned it means Xen did not suspended,
> +         * but it is still in the same state as if the system_suspend was never
> +         * called. We dump a debug message in case of an error for debugging/
> +         * logging purpose.
> +         */
> +        if ( status )
> +            dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
> +    }

After discussing we Stefano today, I have one more question regarding 
Dom0 suspend.

 From my understanding, the host may resume because of an event 
targeting a guest. This means that Dom0 would still be blocked. As Dom0 
would contain PV backend, how do you expect this to work?

Is there any potential dependency between frontend and backend? Or would 
Dom0 be resume when the PV frontend probe the backend?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1833 bytes --]

On Wed, 14 Nov 2018, Julien Grall wrote:
> Hi,
> 
> On 14/11/2018 22:18, Stefano Stabellini wrote:
> > On Wed, 14 Nov 2018, Julien Grall wrote:
> >>>>>    @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
> >>>>>        BUG();
> >>>>>    }
> >>>>>    +static void gicv2_alloc_context(struct gicv2_context *gc)
> >>>>> +{
> >>>>
> >>>> Is it necessary to allocate them at boot? Can we make them static or
> >>>> allocate them when we suspend?
> >>>>
> >>>
> >>> We need to allocate dynamically because the size of allocated data depends
> >>> on the number of irq lines, which is not known at the compile time.
> >>
> >> Well you know the upper bound. Why can't you use the upper bound?
> >>
> >>> Alternative is to allocate on suspend, but I believe it is better to do this
> >>> when the system boots.
> >>
> >> Why is it better?
> > 
> > I'll reply here also to your other point because they are related:
> > 
> >> Suspend/resume is not a critical feature in common case. So I would
> >> prefer if we disable it when we can't alloc memory.
> > 
> > 
> > It is true that suspend/resume is not a critical feature for the common
> > case, but proceeding as "normal" when a memory allocation fails is not a
> > good idea: if the hypervisor is so low in memory as to fail in an
> > allocation like this one, it is not going to be able to work right. In
> > no other cases in Xen we continue on memory allocation failures, even
> > for less-critical features.
> > 
> > I suggest that we either allocate statically using the upper bound as
> > you suggested, although it leads to some memory being wasted.
> 
> We are speaking of at most 2KB of memory. I don't think it is going to 
> be waste given of the number of interrupts GIC usually supports.
> 
> The more that we already statically allocate irq_desc.

OK

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5095 bytes --]

On Wed, 14 Nov 2018, Julien Grall wrote:
> On 14/11/2018 23:04, Stefano Stabellini wrote:
> > On Wed, 14 Nov 2018, Julien Grall wrote:
> >> Hi Mirela,
> >>
> >> On 14/11/2018 13:00, Mirela Simonovic wrote:
> >>>
> >>>
> >>> On 11/14/2018 11:52 AM, Julien Grall wrote:
> >>>> Hi Mirela,
> >>>>
> >>>> On 12/11/2018 11:30, Mirela Simonovic wrote:
> >>>>> Non-boot CPUs have to be disabled on suspend and enabled on resume
> >>>>> (hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
> >>>>> CPU_OFF to be called by each non-boot CPU. Depending on the underlying
> >>>>> platform capabilities, this may lead to the physical powering down of
> >>>>> CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
> >>>>> each non-boot CPU).
> >>>>>
> >>>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> >>>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> >>>>> ---
> >>>>>    xen/arch/arm/suspend.c | 15 ++++++++++++++-
> >>>>>    1 file changed, 14 insertions(+), 1 deletion(-)
> >>>>>
> >>>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> >>>>> index 575afd5eb8..dae1b1f7d6 100644
> >>>>> --- a/xen/arch/arm/suspend.c
> >>>>> +++ b/xen/arch/arm/suspend.c
> >>>>> @@ -1,4 +1,5 @@
> >>>>>    #include <xen/sched.h>
> >>>>> +#include <xen/cpu.h>
> >>>>>    #include <asm/cpufeature.h>
> >>>>>    #include <asm/event.h>
> >>>>>    #include <asm/psci.h>
> >>>>> @@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint,
> >>>>> register_t cid)
> >>>>>    /* Xen suspend. Note: data is not used (suspend is the suspend to RAM)
> >>>>> */
> >>>>>    static long system_suspend(void *data)
> >>>>>    {
> >>>>> +    int status;
> >>>>> +
> >>>>>        BUG_ON(system_state != SYS_STATE_active);
> >>>>>          system_state = SYS_STATE_suspend;
> >>>>>        freeze_domains();
> >>>>>    +    status = disable_nonboot_cpus();
> >>>>> +    if ( status )
> >>>>> +    {
> >>>>> +        system_state = SYS_STATE_resume;
> >>>>> +        goto resume_nonboot_cpus;
> >>>>> +    }
> >>>>> +
> >>>>>        system_state = SYS_STATE_resume;
> >>>>>    +resume_nonboot_cpus:
> >>>>> +    enable_nonboot_cpus();
> >>>>>        thaw_domains();
> >>>>>        system_state = SYS_STATE_active;
> >>>>> +    dsb(sy);
> >>>>
> >>>> Why do you need a dsb(sy) here?
> >>>>
> >>>
> >>> Updated value of system_state variable needs to be visible to other CPUs
> >>> before we move on
> >>
> >> We tend to write the reason on top of barrier why they are necessary. But I am
> >> still unsure to understand why this is important. What would happen if move on
> >> without it?
> > 
> > That is a good question. I went through the code and it seems that the
> > only effect could be potentially taking the wrong path in
> > cpupool_cpu_add, but since that's called from a .notifier_call I don't
> > think it can happen in practice. It is always difficult to prove that
> > we don't need a barrier, it is easier to prove when we need a barrier,
> > but in this case it does look like we do not need it after all.
> 
> It is also very easy to add barrier everywhere so we are sure what to do 
> ;). If you need a barrier, then you need to give plausible explanation.
> 
> In that case, if you need barrier here for system_state. Then what 
> wouldn't you need it in other places where system_state is updated/read?

Right, no plausible explanation here, so no barrier.


> >   
> >>>>>    -    return -ENOSYS;
> >>>>
> >>>> Why do you return -ENOSYS until this patch? Should not have it be 0?
> >>>>
> >>>
> >>> To distinguish that Xen suspend wasn't supported until we at least do
> >>> something useful in suspend procedure. This is not so important, we can
> >>> return 0 but needs to be fixed in previous patches.
> >>
> >> If you return 0 before hand you can more easily bisect this series and know
> >> where it suspend/resume breaks.
> > 
> > Why 0 improves bisectability? 0 prevents other checks from figuring out
> > that there was an error.
> 
> But this code is exactly replacing -ENOSYS by state (that would be 0 in 
> success. So what's wrong to return 0 rather than -ENOSYS in that patch 
> implement the dummy system_state?
> 
> > Also, the feature is not bisectable until the
> > full series is applied.
> 
> Really? I was under the impression you can still do some sort of 
> suspend/resume patch by patch. Although, you would not do a full 
> suspend/resume.

You are saying that we could call the function and return successfully
even if the function does nothing, simply by returning 0. That would
make suspend bisectable within this series, patch by patch.

I think it's impressive that Mirela managed to write the series this
way, and if suspend is actually bisectable patch by patch simply by
returning 0 here, it would be amazing, and certainly worth doing.
However, if it is not the case, I wouldn't ask Mirela to make the effort
to make suspend bisectable patch by patch beyond returning 0 here, it
would be good to have but not required.

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 7:23 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > When Dom0 finalizes its suspend procedure the suspend of Xen is
> > triggered by calling system_suspend(). Dom0 finalizes the suspend from
> > its boot core (VCPU#0), which could be mapped to any physical CPU,
> > i.e. the system_suspend() function could be executed by any physical
> > CPU. Since Xen suspend procedure has to be run by the boot CPU
> > (non-boot CPUs will be disabled at some point in suspend procedure),
> > system_suspend() execution has to continue on CPU#0.
> >
> > When the system_suspend() returns 0, it means that the system was
> > suspended and it is coming out of the resume procedure. Regardless
> > of the system_suspend() return value, after this function returns
> > Xen is fully functional, and its state, including all devices and data
> > structures, matches the state prior to calling system_suspend().
> > The status is returned by system_suspend() for debugging/logging
> > purposes and function prototype compatibility.
> >
> > Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> > Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> > ---
> >   xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
> >   1 file changed, 34 insertions(+)
> >
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index f2338e41db..21b45f8248 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
> >       _arch_set_info_guest(v, &ctxt);
> >   }
> >
> > +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> > +static long system_suspend(void *data)
> > +{
> > +    BUG_ON(system_state != SYS_STATE_active);
> > +
> > +    return -ENOSYS;
> > +}
> > +
> >   int32_t domain_suspend(register_t epoint, register_t cid)
> >   {
> >       struct vcpu *v;
> >       struct domain *d = current->domain;
> >       bool is_thumb = epoint & 1;
> > +    int status;
> >
> >       dprintk(XENLOG_DEBUG,
> >               "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
> > @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
> >        */
> >       vcpu_block_unless_event_pending(current);
> >
> > +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
> > +    if ( is_hardware_domain(d) )
> > +    {
> > +        /*
> > +         * system_suspend should be called when Dom0 finalizes the suspend
> > +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
> > +         * be mapped to any PCPU (this function could be executed by any PCPU).
> > +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
> > +         * PCPUs will be disabled during the suspend).
> > +         */
> > +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> > +        /*
> > +         * If an error happened, there is nothing that needs to be done here
> > +         * because the system_suspend always returns in fully functional state
> > +         * no matter what the outcome of suspend procedure is. If the system
> > +         * suspended successfully the function will return 0 after the resume.
> > +         * Otherwise, if an error is returned it means Xen did not suspended,
> > +         * but it is still in the same state as if the system_suspend was never
> > +         * called. We dump a debug message in case of an error for debugging/
> > +         * logging purpose.
> > +         */
> > +        if ( status )
> > +            dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
> > +    }
>
> After discussing we Stefano today, I have one more question regarding
> Dom0 suspend.
>
>  From my understanding, the host may resume because of an event
> targeting a guest. This means that Dom0 would still be blocked. As Dom0
> would contain PV backend, how do you expect this to work?
>
> Is there any potential dependency between frontend and backend? Or would
> Dom0 be resume when the PV frontend probe the backend?
>

We have assumed that Dom0 has to resume whenever Xen resume. So if the
wake-up interrupt was targeted to DomU, the Dom0 would resume
regarless. Vice versa does not hold.
This is done in patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
resumes" - the Dom0 is simply unblocked at the end of Xen's resume.

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Stefano Stabellini]

On Thu, 15 Nov 2018, Mirela Simonovic wrote:
> Hi Julien,
> 
> On Thu, Nov 15, 2018 at 12:38 PM Julien Grall <julien.grall@arm.com> wrote:
> >
> > Hi,
> >
> > On 11/15/18 11:10 AM, Mirela Simonovic wrote:
> > > Hi Julien,
> > >
> > > On Thu, Nov 15, 2018 at 11:59 AM Julien Grall <julien.grall@arm.com> wrote:
> > >>
> > >> Hi Mirela,
> > >>
> > >> On 11/15/18 10:33 AM, Mirela Simonovic wrote:
> > >>> On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
> > >>> <andrew.cooper3@citrix.com> wrote:
> > >>>>
> > >>>> On 15/11/2018 10:13, Julien Grall wrote:
> > >>>>> (+ Andre)
> > >>>>>
> > >>>>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
> > >>>>>> On 14/11/2018 12:49, Julien Grall wrote:
> > >>>>>>> Hi Mirela,
> > >>>>>>>
> > >>>>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> > >>>>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
> > >>>>>>>>>> Hi Andrew,
> > >>>>>>>>>>
> > >>>>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
> > >>>>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
> > >>>>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > >>>>>>>>>>>>>> index e594b48d81..7f8105465c 100644
> > >>>>>>>>>>>>>> --- a/xen/arch/arm/domain.c
> > >>>>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
> > >>>>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> > >>>>>>>>>>>>>>            if ( is_idle_vcpu(p) )
> > >>>>>>>>>>>>>>                return;
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
> > >>>>>>>>>>>>>> suspended */
> > >>>>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
> > >>>>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
> > >>>>>>>>>>>>>> +        return;
> > >>>>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
> > >>>>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
> > >>>>>>>>>>>>> state
> > >>>>>>>>>>>>> to be saved on suspend.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
> > >>>>>>>>>>>> believe
> > >>>>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
> > >>>>>>>>>>>> Let's come back on this.
> > >>>>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
> > >>>>>>>>>>> Xen
> > >>>>>>>>>>> tree and you'll find several pieces of documentation, including the
> > >>>>>>>>>>> description of what this shutdown code means>>>>>>>>>>>
> > >>>>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
> > >>>>>>>>>>> with
> > >>>>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
> > >>>>>>>>>>> Xen's
> > >>>>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
> > >>>>>>>>>>> the
> > >>>>>>>>>>> toolstack.
> > >>>>>>>>>> Would domain pause/unpause be a better solution?
> > >>>>>>>>> Actually yes - that sounds like a very neat solution.
> > >>>>>>>>
> > >>>>>>>> I believe domain pause will not work here - a domain cannot pause
> > >>>>>>>> itself, i.e. current domain cannot be paused. This functionality
> > >>>>>>>> seems to assume that a domain is pausing another domain. Or I missed
> > >>>>>>>> the point.
> > >>>>>>>
> > >>>>>>> Yes domain pause/unpause will not work. However, you can introduce a
> > >>>>>>> boolean to tell you whether the domain was suspend.
> > >>>>>>>
> > >>>>>>> I actually quite like how suspend work for x86 HVM. This is based on
> > >>>>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
> > >>>>>>
> > >>>>>> That only exists because the ACPI controller is/was implemented in
> > >>>>>> QEMU.  I wouldn't recommend copying it.
> > >>>>>
> > >>>>> May I ask why? I don't think the properties are very different from
> > >>>>> Arm here.
> > >>>>
> > >>>> If you observe, that can only be actioned by a hypercall from qemu.  It
> > >>>> can't be actioned by an ACPI controller emulated by Xen.
> > >>>>
> > >>>>>
> > >>>>>>
> > >>>>>> Having spent some more time thinking about this problem, what properties
> > >>>>>> does the PSCI call have?
> > >>>>>>
> > >>>>>> I gather from other parts of this thread that there may be a partial
> > >>>>>> reset of state?  Beyond that, what else?
> > >>>>>>
> > >>>>>> I ask, because conceptually, domU suspend to RAM is literally just
> > >>>>>> "de-schedule until we want to wake up", and in this case, it appears to
> > >>>>>> be "wake up on any of the still-active interrupts".  We've already got a
> > >>>>>> scheduler API for that, and its called vcpu_block().  This already
> > >>>>>> exists with "wait until a new event occurs" semantics.
> > >>>>>>
> > >>>>>> Is there anything else I've missed?
> > >>>>>
> > >>>
> > >>> That's correct, and I agree. BTW that is what's implemented in this series.
> > >>>
> > >>>>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
> > >>>>> only events on that vCPU can trigger a resume. All the other event
> > >>>>> should not be taken into account.
> > >>>>>
> > >>>
> > >>> What other events are talking about here?
> > >>
> > >> vcpu_unblock is not only called when you receive interrupts. It can be
> > >> called in other place when you receive an events.
> > >>
> > >>   From the Arm Arm, an event can be anything. So do we really want to
> > >> wake-up on any events?
> > >>
> > >>>
> > >>>>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
> > >>>>> Technically they should be off, but I would like some safety to avoid
> > >>>>> any potential corner case (i.e other way to turn a vCPU on).
> > >>>>
> > >>>
> > >>> Other vCPUs are hot-unplugged (offlined) by the guest. If that is not
> > >>> the case, SYSTEM_SUSPEND will return an error.
> > >>> Could you please clarify what a potential corner case would be?
> > >>
> > >> PSCI CPU_ON is not the only way to online a vCPU. I merely want to
> > >> prevent other path to play with the vCPU when it is not necessary.
> > >>
> > >> [...]
> > >>
> > >>>> If instead of waiting for any event, you need to wait for a specific
> > >>>> event, there is also vcpu_poll() which is a related scheduler API.
> > >>>>
> > >>>> ~Andrew
> > >>>
> > >>> Some content disappeared, so I'll write here to avoid thread branching.
> > >>>
> > >>> The semantic of vCPU block and domain_pause is not the same. When
> > >>> guest suspends the domain is not (and should not be) paused, instead
> > >>> its last online vCPU is blocked waiting on an interrupt. That's it.

From the scheduler's point of view, a suspended domain is not running,
so it is basically "paused". We are "pausing the scheduling" of it. Also
looking at the implementation of hvm_s3_suspend, I think it is fine to
call domain_pause() on ARM too.

However, also like hvm_s3_suspend, we need to set an additional special
flag (such as d->arch.hvm.is_s3_suspended) to make sure we know how to
differentiate a paused domain from a suspended domain: a user should not
be able to resume a domain with "xl unpause", they should use something
like xl trigger s3resume. We should not lose the differentiation between
suspend and pause in our internal state tracking.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Julien Grall]

Hi Mirela,

On 11/15/18 11:42 AM, Mirela Simonovic wrote:
> Hi Julien,
> 
> On Thu, Nov 15, 2018 at 12:38 PM Julien Grall <julien.grall@arm.com> wrote:
>>
>> Hi,
>>
>> On 11/15/18 11:10 AM, Mirela Simonovic wrote:
>>> Hi Julien,
>>>
>>> On Thu, Nov 15, 2018 at 11:59 AM Julien Grall <julien.grall@arm.com> wrote:
>>>>
>>>> Hi Mirela,
>>>>
>>>> On 11/15/18 10:33 AM, Mirela Simonovic wrote:
>>>>> On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
>>>>> <andrew.cooper3@citrix.com> wrote:
>>>>>>
>>>>>> On 15/11/2018 10:13, Julien Grall wrote:
>>>>>>> (+ Andre)
>>>>>>>
>>>>>>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
>>>>>>>> On 14/11/2018 12:49, Julien Grall wrote:
>>>>>>>>> Hi Mirela,
>>>>>>>>>
>>>>>>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
>>>>>>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
>>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
>>>>>>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
>>>>>>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>>>>>>>>>> index e594b48d81..7f8105465c 100644
>>>>>>>>>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
>>>>>>>>>>>>>>>>             if ( is_idle_vcpu(p) )
>>>>>>>>>>>>>>>>                 return;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +    /* VCPU's context should not be saved if its domain is
>>>>>>>>>>>>>>>> suspended */
>>>>>>>>>>>>>>>> +    if ( p->domain->is_shut_down &&
>>>>>>>>>>>>>>>> +        (p->domain->shutdown_code == SHUTDOWN_suspend) )
>>>>>>>>>>>>>>>> +        return;
>>>>>>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
>>>>>>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
>>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>> to be saved on suspend.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
>>>>>>>>>>>>>> believe
>>>>>>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
>>>>>>>>>>>>>> Let's come back on this.
>>>>>>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
>>>>>>>>>>>>> Xen
>>>>>>>>>>>>> tree and you'll find several pieces of documentation, including the
>>>>>>>>>>>>> description of what this shutdown code means>>>>>>>>>>>
>>>>>>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
>>>>>>>>>>>>> with
>>>>>>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
>>>>>>>>>>>>> Xen's
>>>>>>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
>>>>>>>>>>>>> the
>>>>>>>>>>>>> toolstack.
>>>>>>>>>>>> Would domain pause/unpause be a better solution?
>>>>>>>>>>> Actually yes - that sounds like a very neat solution.
>>>>>>>>>>
>>>>>>>>>> I believe domain pause will not work here - a domain cannot pause
>>>>>>>>>> itself, i.e. current domain cannot be paused. This functionality
>>>>>>>>>> seems to assume that a domain is pausing another domain. Or I missed
>>>>>>>>>> the point.
>>>>>>>>>
>>>>>>>>> Yes domain pause/unpause will not work. However, you can introduce a
>>>>>>>>> boolean to tell you whether the domain was suspend.
>>>>>>>>>
>>>>>>>>> I actually quite like how suspend work for x86 HVM. This is based on
>>>>>>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
>>>>>>>>
>>>>>>>> That only exists because the ACPI controller is/was implemented in
>>>>>>>> QEMU.  I wouldn't recommend copying it.
>>>>>>>
>>>>>>> May I ask why? I don't think the properties are very different from
>>>>>>> Arm here.
>>>>>>
>>>>>> If you observe, that can only be actioned by a hypercall from qemu.  It
>>>>>> can't be actioned by an ACPI controller emulated by Xen.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Having spent some more time thinking about this problem, what properties
>>>>>>>> does the PSCI call have?
>>>>>>>>
>>>>>>>> I gather from other parts of this thread that there may be a partial
>>>>>>>> reset of state?  Beyond that, what else?
>>>>>>>>
>>>>>>>> I ask, because conceptually, domU suspend to RAM is literally just
>>>>>>>> "de-schedule until we want to wake up", and in this case, it appears to
>>>>>>>> be "wake up on any of the still-active interrupts".  We've already got a
>>>>>>>> scheduler API for that, and its called vcpu_block().  This already
>>>>>>>> exists with "wait until a new event occurs" semantics.
>>>>>>>>
>>>>>>>> Is there anything else I've missed?
>>>>>>>
>>>>>
>>>>> That's correct, and I agree. BTW that is what's implemented in this series.
>>>>>
>>>>>>> All vCPUs but the vCPU calling SYSTEM_SUSPEND should be off. Also,
>>>>>>> only events on that vCPU can trigger a resume. All the other event
>>>>>>> should not be taken into account.
>>>>>>>
>>>>>
>>>>> What other events are talking about here?
>>>>
>>>> vcpu_unblock is not only called when you receive interrupts. It can be
>>>> called in other place when you receive an events.
>>>>
>>>>    From the Arm Arm, an event can be anything. So do we really want to
>>>> wake-up on any events?
>>>>
>>>>>
>>>>>>> My worry with vcpu_block() is we don't prevent the other vCPUs to run.
>>>>>>> Technically they should be off, but I would like some safety to avoid
>>>>>>> any potential corner case (i.e other way to turn a vCPU on).
>>>>>>
>>>>>
>>>>> Other vCPUs are hot-unplugged (offlined) by the guest. If that is not
>>>>> the case, SYSTEM_SUSPEND will return an error.
>>>>> Could you please clarify what a potential corner case would be?
>>>>
>>>> PSCI CPU_ON is not the only way to online a vCPU. I merely want to
>>>> prevent other path to play with the vCPU when it is not necessary.
>>>>
>>>> [...]
>>>>
>>>>>> If instead of waiting for any event, you need to wait for a specific
>>>>>> event, there is also vcpu_poll() which is a related scheduler API.
>>>>>>
>>>>>> ~Andrew
>>>>>
>>>>> Some content disappeared, so I'll write here to avoid thread branching.
>>>>>
>>>>> The semantic of vCPU block and domain_pause is not the same. When
>>>>> guest suspends the domain is not (and should not be) paused, instead
>>>>> its last online vCPU is blocked waiting on an interrupt. That's it.
>>>> Well no, you will block until you receive an event. Interrupts are one
>>>> of them.
>>>>
>>>> Do we want to consider all events as wakeup event?
>>>>
>>>
>>> I think we need to assume that events are not triggered via toolstack,
>>> Andrew made a good point.
>>
>> I don't think we are discussing the same thing. The discussion was
>> around other vCPUs, not the vCPU calling SYSTEM_SUSPEND.
>>
>> Most likely in the future, we would want to allow the toolstack to
>> request resuming the domain. This can be considered as an event.
>>
> 
> Yes, such an event will unblock the vcpu and cause the domain to
> resume. So from this perspective it's not only an interrupt targeted
> to the guest.
> 
>>> Given the assumption, my understanding is that Xen itself will not
>>> unblock vCPU, except due to an interrupt targeted to the guest.
>>> Am I missing something? An example would be appreciated.
>>
>> At least on Arm, the current semantics of vcpu_block/vcpu_unblock is to
>> block until you receive an events.
>>
>> I don't much want to restrict the definition of events to only
>> interrupts.  To clarify my point, if you want to wake-up for any events
>> then fine. But this needs to be understood that it may not be only
>> interrupts.
>>
> 
> In the context described above this is fine - events are not only interrupts.

So I guess we have a way forward here. This could be implemented with 
vcpu_block(). This would need to be done in combination with as tasklet 
as discussed in patch #5.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Julien Grall]

Hi,

On 11/15/18 12:35 AM, Stefano Stabellini wrote:
> On Wed, 14 Nov 2018, Julien Grall wrote:
>> On 14/11/2018 22:45, Stefano Stabellini wrote:
>>> On Wed, 14 Nov 2018, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 13/11/2018 20:44, Stefano Stabellini wrote:
>>>>> On Mon, 12 Nov 2018, Julien Grall wrote:
>>>>>> (+ Andre)
>>>>>>
>>>>>> On 11/12/18 5:42 PM, Mirela Simonovic wrote:
>>>>>>> Hi Julien,
>>>>>>>
>>>>>>> On Mon, Nov 12, 2018 at 6:00 PM Julien Grall <julien.grall@arm.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
>>>>>>>>> Hi Julien,
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>> Thanks for the feedback.
>>>>>>>>>
>>>>>>>>> On Mon, Nov 12, 2018 at 4:36 PM Julien Grall <julien.grall@arm.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Mirela,
>>>>>>>>>>
>>>>>>>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>>>>>>>> GIC and virtual timer context must be saved when the domain
>>>>>>>>>>> suspends.
>>>>>>>>>>
>>>>>>>>>> Please provide the rationale for this. Is it required by the spec?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is required for GIC because a guest leaves enabled interrupts
>>>>>>>>> in
>>>>>>>>> the GIC that could wake it up, and on resume it should be able to
>>>>>>>>> detect which interrupt woke it up. Without saving/restoring the
>>>>>>>>> state
>>>>>>>>> of GIC this would not be possible.
>>>>>>>>
>>>>>>>> I am confused. In patch #10, you save the GIC host because the GIC can
>>>>>>>> be powered-down. Linux is also saving the GIC state. So how the
>>>>>>>> interrupt can come up if the GIC is powered down?
>>>>>>>>
>>>>>>>
>>>>>>> After Xen (or Linux in the config without Xen) hands over the control
>>>>>>> to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
>>>>>>> enabled interrupts that are its wake-up sources. Saving a GIC state
>>>>>>> only means that its current configuration will be remembered somewhere
>>>>>>> in data structures, but the configuration is not changed on suspend.
>>>>>>> Everything that happens with interrupt configuration from this point
>>>>>>> on is platform specific. Now there are 2 options: 1) EL3 will never
>>>>>>> allow CPU0 to be powered down and the wake-up interrupt will indeed
>>>>>>> propagate via GIC;
>>>>>>> or 2) CPU0 will be powered down and the GIC may be
>>>>>>> powered down as well, so an external help is needed to deal with
>>>>>>> wake-up and resume (e.g. in order to react to a wake-up interrupt
>>>>>>> while the GIC is down, and power up CPU0). This external help is
>>>>>>> provided by some low-power processor outside of the Cortex-A cluster.
>>>>>>>
>>>>>>> So the platform firmware is responsible for properly configuring a
>>>>>>> wake-up path if GIC goes down. This is commonly handled by EL3
>>>>>>> communicating with low-power processor. When the GIC power comes up,
>>>>>>> the interrupt triggered by a peripheral is still active and the
>>>>>>> software on Cortex-A cluster should be able to observe it once the GIC
>>>>>>> state is restored, i.e. interrupts get enabled at GIC.
>>>>>>
>>>>>> Thank you for the explanation.  Now the question is why can't we reset at
>>>>>> least the GIC CPU interface?
>>>>>>
>>>>>> AFAIU, the guest should restore them in any case. The only things we need
>>>>>> to
>>>>>> know is the interrupt was received for a given guest. We can then queue it
>>>>>> and
>>>>>> wake-up the domain.
>>>>>>
>>>>>> This seems to fit with the description on top of gic_dist_save() in Linux
>>>>>> GICv2 driver.
>>>>>
>>>>> Can we rely on all PSCI compliant OSes to restore their own GIC again at
>>>>> resume? The PSCI spec is not very clear in that regard (at the the
>>>>> version I am looking at...) I am just asking so that we don't come up
>>>>> with a solution that only works with Linux.
>>>> See PSCI 1.1 (DEN0022D) section 6.8. Each level should save its own context
>>>> because the PSCI implementations is allowed to shutdown the GIC.
>>>
>>> Great, in that case we should be able to skip saving some of the GICD
>>> registers too. We do need to save GICD_ISACTIVER, and GICD_ICFGR,
>>> but we should be able to skip the others (GICD_ISENABLER,
>>> GICD_IPRIORITYR, GICD_ITARGETSR). If we do, we still need to
>>> re-initialize them as we do in gicv2_dist_init.
>>
>> You are assuming a domain will handle properly the suspend/resume. I
>> don't think we can promise that as we call freeze/thaw.
> 
> Dho! That would break every single guest that has been forcefully
> suspended :-/
> 
> Right, we can't do that (FYI I tested today the series with an unaware
> domU and it all resumed correctly.)
> 
> But given that we only suspend/resume GICC_CTLR, GICC_PMR, GICC_BPR of
> the GICC interface, it should be fine to re-initialize that. We do need
> to be careful because the current implementation of gicv2_cpu_init
> touches a bunch of GICD registers that we'll have to save separately for
> suspend/resume.

See my review on patch #10. I suggested to move out GICC_CTLR, GICC_PMR, 
GICC_BPR in a separate helpers that could be called by gicv2_cpu_init 
and the new function.

A good name for that helper would be gicv2_cpu_if_up().

Cheers,


-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Julien Grall]

Hi,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> The resume of Dom0 should always follow Xen's resume. This is
> done by unblocking the first vCPU of Dom0.

Please explain why you always need to resume Dom0 afterwards.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Julien Grall]

On 11/15/18 7:17 PM, Mirela Simonovic wrote:
> Hi Julien,
> 
> On Thu, Nov 15, 2018 at 7:23 PM Julien Grall <julien.grall@arm.com> wrote:
>>
>> Hi Mirela,
>>
>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>> When Dom0 finalizes its suspend procedure the suspend of Xen is
>>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
>>> its boot core (VCPU#0), which could be mapped to any physical CPU,
>>> i.e. the system_suspend() function could be executed by any physical
>>> CPU. Since Xen suspend procedure has to be run by the boot CPU
>>> (non-boot CPUs will be disabled at some point in suspend procedure),
>>> system_suspend() execution has to continue on CPU#0.
>>>
>>> When the system_suspend() returns 0, it means that the system was
>>> suspended and it is coming out of the resume procedure. Regardless
>>> of the system_suspend() return value, after this function returns
>>> Xen is fully functional, and its state, including all devices and data
>>> structures, matches the state prior to calling system_suspend().
>>> The status is returned by system_suspend() for debugging/logging
>>> purposes and function prototype compatibility.
>>>
>>> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
>>> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
>>> ---
>>>    xen/arch/arm/suspend.c | 34 ++++++++++++++++++++++++++++++++++
>>>    1 file changed, 34 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
>>> index f2338e41db..21b45f8248 100644
>>> --- a/xen/arch/arm/suspend.c
>>> +++ b/xen/arch/arm/suspend.c
>>> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t cid)
>>>        _arch_set_info_guest(v, &ctxt);
>>>    }
>>>
>>> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
>>> +static long system_suspend(void *data)
>>> +{
>>> +    BUG_ON(system_state != SYS_STATE_active);
>>> +
>>> +    return -ENOSYS;
>>> +}
>>> +
>>>    int32_t domain_suspend(register_t epoint, register_t cid)
>>>    {
>>>        struct vcpu *v;
>>>        struct domain *d = current->domain;
>>>        bool is_thumb = epoint & 1;
>>> +    int status;
>>>
>>>        dprintk(XENLOG_DEBUG,
>>>                "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
>>> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
>>>         */
>>>        vcpu_block_unless_event_pending(current);
>>>
>>> +    /* If this was dom0 the whole system should suspend: trigger Xen suspend */
>>> +    if ( is_hardware_domain(d) )
>>> +    {
>>> +        /*
>>> +         * system_suspend should be called when Dom0 finalizes the suspend
>>> +         * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
>>> +         * be mapped to any PCPU (this function could be executed by any PCPU).
>>> +         * The suspend procedure has to be finalized by the PCPU#0 (non-boot
>>> +         * PCPUs will be disabled during the suspend).
>>> +         */
>>> +        status = continue_hypercall_on_cpu(0, system_suspend, NULL);
>>> +        /*
>>> +         * If an error happened, there is nothing that needs to be done here
>>> +         * because the system_suspend always returns in fully functional state
>>> +         * no matter what the outcome of suspend procedure is. If the system
>>> +         * suspended successfully the function will return 0 after the resume.
>>> +         * Otherwise, if an error is returned it means Xen did not suspended,
>>> +         * but it is still in the same state as if the system_suspend was never
>>> +         * called. We dump a debug message in case of an error for debugging/
>>> +         * logging purpose.
>>> +         */
>>> +        if ( status )
>>> +            dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
>>> +    }
>>
>> After discussing we Stefano today, I have one more question regarding
>> Dom0 suspend.
>>
>>   From my understanding, the host may resume because of an event
>> targeting a guest. This means that Dom0 would still be blocked. As Dom0
>> would contain PV backend, how do you expect this to work?
>>
>> Is there any potential dependency between frontend and backend? Or would
>> Dom0 be resume when the PV frontend probe the backend?
>>
> 
> We have assumed that Dom0 has to resume whenever Xen resume. So if the
> wake-up interrupt was targeted to DomU, the Dom0 would resume
> regarless. Vice versa does not hold.
> This is done in patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> resumes" - the Dom0 is simply unblocked at the end of Xen's resume.
How about the disaggregated case where backend may live in a different 
guest. How is this going to happen?

I have heard Stefano suggesting to resume all the domains but that seem 
to be a massive hammer to solve it.

I am wondering whether we can rely on the frontend sending an event (i.e 
event channel) in part of resuming the PV protocols.

Did you test it the PV protocols in the suspend/resume? I know that 
Linux does not have everything for suspend/resume Xen Arm today. There 
are some patches inflight (see [1]).

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg00823.html

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Julien Grall]

On 11/15/18 6:57 PM, Stefano Stabellini wrote:
> On Wed, 14 Nov 2018, Julien Grall wrote:
>> On 14/11/2018 23:04, Stefano Stabellini wrote:
>>> On Wed, 14 Nov 2018, Julien Grall wrote:
>>>>>>>     -    return -ENOSYS;
>>>>>>
>>>>>> Why do you return -ENOSYS until this patch? Should not have it be 0?
>>>>>>
>>>>>
>>>>> To distinguish that Xen suspend wasn't supported until we at least do
>>>>> something useful in suspend procedure. This is not so important, we can
>>>>> return 0 but needs to be fixed in previous patches.
>>>>
>>>> If you return 0 before hand you can more easily bisect this series and know
>>>> where it suspend/resume breaks.
>>>
>>> Why 0 improves bisectability? 0 prevents other checks from figuring out
>>> that there was an error.
>>
>> But this code is exactly replacing -ENOSYS by state (that would be 0 in
>> success. So what's wrong to return 0 rather than -ENOSYS in that patch
>> implement the dummy system_state?
>>
>>> Also, the feature is not bisectable until the
>>> full series is applied.
>>
>> Really? I was under the impression you can still do some sort of
>> suspend/resume patch by patch. Although, you would not do a full
>> suspend/resume.
> 
> You are saying that we could call the function and return successfully
> even if the function does nothing, simply by returning 0. That would
> make suspend bisectable within this series, patch by patch.
> 
> I think it's impressive that Mirela managed to write the series this
> way, and if suspend is actually bisectable patch by patch simply by
> returning 0 here, it would be amazing, and certainly worth doing.
> However, if it is not the case, I wouldn't ask Mirela to make the effort
> to make suspend bisectable patch by patch beyond returning 0 here, it
> would be good to have but not required.

This wasn't my intention :).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 13/18] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)

[Julien Grall]

Hi,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> PSCI system suspend function shall be invoked to finalize Xen suspend
> procedure. Resume entry point, which needs to be passed via 1st argument
> of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
> is just a placeholder that will be implemented in assembly. Context ID,
> which is 2nd argument of system suspend PSCI call, is unused, as in Linux.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> 
> ---
> Changes in v2:
> 
> -The commit message was stale - referring to the do_suspend function
> that has been renamed long time ago. Fixed commit message
> ---
>   xen/arch/arm/arm64/entry.S    |  3 +++
>   xen/arch/arm/psci.c           | 16 ++++++++++++++++
>   xen/arch/arm/suspend.c        |  4 ++++
>   xen/include/asm-arm/psci.h    |  1 +
>   xen/include/asm-arm/suspend.h |  1 +
>   5 files changed, 25 insertions(+)
> 
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index 97b05f53ea..dbc4717903 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -533,6 +533,9 @@ ENTRY(__context_switch)
>           mov     sp, x9
>           ret
>   
> +ENTRY(hyp_resume)
> +        b .
> +
>   /*
>    * Local variables:
>    * mode: ASM
> diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
> index a93121f43b..b100bd8ad2 100644
> --- a/xen/arch/arm/psci.c
> +++ b/xen/arch/arm/psci.c
> @@ -24,6 +24,7 @@
>   #include <asm/cpufeature.h>
>   #include <asm/psci.h>
>   #include <asm/acpi.h>
> +#include <asm/suspend.h>
>   
>   /*
>    * While a 64-bit OS can make calls with SMC32 calling conventions, for
> @@ -67,6 +68,21 @@ void call_psci_cpu_off(void)
>       }
>   }
>   
> +int call_psci_system_suspend(void)
> +{

SYSTEM_SUSPEND was introduced by PSCI 1.0 and optional. So you need to 
check the PSCI version and use PSCI_FEATURES to check if it was implemented.

> +#ifdef CONFIG_ARM_64
> +    struct arm_smccc_res res;
> +
> +    /* 2nd argument (context ID) is not used */

It still needs to be defined to some known values rather than whatever 
is in x2 at that time.

But I would suggest to make good use of it to catch implementation not 
doing the right thing. We could define it to 0xdeadbeef and shout at 
anyone not preserving the value.

> +    arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), &res);
> +
> +    return PSCI_RET(res);
> +#else
> +    /* not supported */
> +    return 1;
> +#endif
> +}
> +
>   void call_psci_system_off(void)
>   {
>       if ( psci_ver > PSCI_VERSION(0, 1) )
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index d1b48c339a..37926374bc 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -141,6 +141,10 @@ static long system_suspend(void *data)
>           goto resume_irqs;
>       }
>   
> +    status = call_psci_system_suspend();

Some platform may not support PSCI at all. So this need to be check.

> +    if ( status )
> +        dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
> +
>       system_state = SYS_STATE_resume;
>   
>       gic_resume();
> diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> index 26462d0c47..9f6116a224 100644
> --- a/xen/include/asm-arm/psci.h
> +++ b/xen/include/asm-arm/psci.h
> @@ -20,6 +20,7 @@ extern uint32_t psci_ver;
>   
>   int psci_init(void);
>   int call_psci_cpu_on(int cpu);
> +int call_psci_system_suspend(void);
>   void call_psci_cpu_off(void);
>   void call_psci_system_off(void);
>   void call_psci_system_reset(void);
> diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> index de787d296a..7604e2e2e2 100644
> --- a/xen/include/asm-arm/suspend.h
> +++ b/xen/include/asm-arm/suspend.h
> @@ -2,6 +2,7 @@
>   #define __ASM_ARM_SUSPEND_H__
>   
>   int32_t domain_suspend(register_t epoint, register_t cid);
> +void hyp_resume(void);
>   
>   #endif
>   
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com> wrote:
>
> Hi,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > The resume of Dom0 should always follow Xen's resume. This is
> > done by unblocking the first vCPU of Dom0.
>
> Please explain why you always need to resume Dom0 afterwards.
>

We don't need to, but that is what is promised in the design spec.

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
<mirela.simonovic@aggios.com> wrote:
>
> Hi Julien,
>
> On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com> wrote:
> >
> > Hi,
> >
> > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > The resume of Dom0 should always follow Xen's resume. This is
> > > done by unblocking the first vCPU of Dom0.
> >
> > Please explain why you always need to resume Dom0 afterwards.
> >
>
> We don't need to, but that is what is promised in the design spec.
>

To be more specific - a domU that doesn't depend on dom0 can resume
and work happily without dom0 being resumed, i.e. just Xen and domU
resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
resumes" is not a must (when there are no PV drivers involved).

> > Cheers,
> >
> > --
> > Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Julien Grall]

On 16/11/2018 11:29, Mirela Simonovic wrote:
> On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
> <mirela.simonovic@aggios.com> wrote:
>>
>> Hi Julien,
>>
>> On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com> wrote:
>>>
>>> Hi,
>>>
>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>> The resume of Dom0 should always follow Xen's resume. This is
>>>> done by unblocking the first vCPU of Dom0.
>>>
>>> Please explain why you always need to resume Dom0 afterwards.
>>>
>>
>> We don't need to, but that is what is promised in the design spec.

You surely had some rationale when writing the promise in the design document, 
right?

So what is the reason behind it? I don't want to resume a domain if that's not 
necessary.

>>
> 
> To be more specific - a domU that doesn't depend on dom0 can resume
> and work happily without dom0 being resumed, i.e. just Xen and domU
> resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> resumes" is not a must (when there are no PV drivers involved).

PV backends don't necessarily reside in the hardware domain. So how is this 
going to work for the other case?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Julien,

On Fri, Nov 16, 2018 at 12:44 PM Julien Grall <julien.grall@arm.com> wrote:
>
>
>
> On 16/11/2018 11:29, Mirela Simonovic wrote:
> > On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
> > <mirela.simonovic@aggios.com> wrote:
> >>
> >> Hi Julien,
> >>
> >> On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>>> The resume of Dom0 should always follow Xen's resume. This is
> >>>> done by unblocking the first vCPU of Dom0.
> >>>
> >>> Please explain why you always need to resume Dom0 afterwards.
> >>>
> >>
> >> We don't need to, but that is what is promised in the design spec.
>
> You surely had some rationale when writing the promise in the design document,
> right?
>
> So what is the reason behind it? I don't want to resume a domain if that's not
> necessary.
>
> >>
> >
> > To be more specific - a domU that doesn't depend on dom0 can resume
> > and work happily without dom0 being resumed, i.e. just Xen and domU
> > resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> > resumes" is not a must (when there are no PV drivers involved).
>
> PV backends don't necessarily reside in the hardware domain. So how is this
> going to work for the other case?
>

I honestly believe that this is not necessary, and is sub-optimal. It
relies on an assumption that dom0 contains all the PV drivers, which
is not always correct.

I would prefer if someone can tell us that any frontend will trigger
an event to the backend, and the event would go through Xen. That way,
this event would cause a domain containing the backend driver to
resume. I think this is the best possible solution, but it relies on
an assumption that the event will go through Xen, and I'm not
knowledgeable enough to claim that this is indeed the case.
If this assumption is correct, I strongly suggest to remove this patch
and let the wake-up for all domains happen automatically after the Xen
resumes.

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Julien Grall]

On 16/11/2018 12:34, Mirela Simonovic wrote:
> Hi Julien,

Hi Mirela,

> 
> On Fri, Nov 16, 2018 at 12:44 PM Julien Grall <julien.grall@arm.com> wrote:
>>
>>
>>
>> On 16/11/2018 11:29, Mirela Simonovic wrote:
>>> On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
>>> <mirela.simonovic@aggios.com> wrote:
>>>>
>>>> Hi Julien,
>>>>
>>>> On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>>> The resume of Dom0 should always follow Xen's resume. This is
>>>>>> done by unblocking the first vCPU of Dom0.
>>>>>
>>>>> Please explain why you always need to resume Dom0 afterwards.
>>>>>
>>>>
>>>> We don't need to, but that is what is promised in the design spec.
>>
>> You surely had some rationale when writing the promise in the design document,
>> right?
>>
>> So what is the reason behind it? I don't want to resume a domain if that's not
>> necessary.
>>
>>>>
>>>
>>> To be more specific - a domU that doesn't depend on dom0 can resume
>>> and work happily without dom0 being resumed, i.e. just Xen and domU
>>> resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
>>> resumes" is not a must (when there are no PV drivers involved).
>>
>> PV backends don't necessarily reside in the hardware domain. So how is this
>> going to work for the other case?
>>
> 
> I honestly believe that this is not necessary, and is sub-optimal. It
> relies on an assumption that dom0 contains all the PV drivers, which
> is not always correct.

Well, there are other reasons to resume the hardware domain. The hardware domain 
owns most the devices and may be part of the suspend/resume path.

As you tie the host suspend to the hardware domain suspend, it may makes sense 
to resume at the same time. It is the kind of rationale I would expect in the 
commit message.

> 
> I would prefer if someone can tell us that any frontend will trigger
> an event to the backend, and the event would go through Xen. That way,
> this event would cause a domain containing the backend driver to
> resume. I think this is the best possible solution, but it relies on
> an assumption that the event will go through Xen, and I'm not
> knowledgeable enough to claim that this is indeed the case.

I think it is should work, the best way to find out if building a test case for it.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Stefano Stabellini]

On Fri, 16 Nov 2018, Julien Grall wrote:
> On 16/11/2018 12:34, Mirela Simonovic wrote:
> > Hi Julien,
> 
> Hi Mirela,
> 
> > 
> > On Fri, Nov 16, 2018 at 12:44 PM Julien Grall <julien.grall@arm.com> wrote:
> > > 
> > > 
> > > 
> > > On 16/11/2018 11:29, Mirela Simonovic wrote:
> > > > On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
> > > > <mirela.simonovic@aggios.com> wrote:
> > > > > 
> > > > > Hi Julien,
> > > > > 
> > > > > On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com>
> > > > > wrote:
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > > > > The resume of Dom0 should always follow Xen's resume. This is
> > > > > > > done by unblocking the first vCPU of Dom0.
> > > > > > 
> > > > > > Please explain why you always need to resume Dom0 afterwards.
> > > > > > 
> > > > > 
> > > > > We don't need to, but that is what is promised in the design spec.
> > > 
> > > You surely had some rationale when writing the promise in the design
> > > document,
> > > right?
> > > 
> > > So what is the reason behind it? I don't want to resume a domain if that's
> > > not
> > > necessary.
> > > 
> > > > > 
> > > > 
> > > > To be more specific - a domU that doesn't depend on dom0 can resume
> > > > and work happily without dom0 being resumed, i.e. just Xen and domU
> > > > resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> > > > resumes" is not a must (when there are no PV drivers involved).
> > > 
> > > PV backends don't necessarily reside in the hardware domain. So how is
> > > this
> > > going to work for the other case?
> > > 
> > 
> > I honestly believe that this is not necessary, and is sub-optimal. It
> > relies on an assumption that dom0 contains all the PV drivers, which
> > is not always correct.
> 
> Well, there are other reasons to resume the hardware domain. The hardware
> domain owns most the devices and may be part of the suspend/resume path.
> 
> As you tie the host suspend to the hardware domain suspend, it may makes sense
> to resume at the same time. It is the kind of rationale I would expect in the
> commit message.
>
> > I would prefer if someone can tell us that any frontend will trigger
> > an event to the backend, and the event would go through Xen. That way,
> > this event would cause a domain containing the backend driver to
> > resume. I think this is the best possible solution, but it relies on
> > an assumption that the event will go through Xen, and I'm not
> > knowledgeable enough to claim that this is indeed the case.
> 
> I think it is should work, the best way to find out if building a test case
> for it.

Yes, PV protocols use hypercalls to send notifications to the other end.
Specifically, the function at the Linux side is
include/xen/events.h:notify_remote_via_evtchn. The Xen implementation
for the hypercall is xen/common/event_channel.c:evtchn_send, where 'rd'
is the destination domain.

It should be possible to figure out which domain needs to awaken from
there.

It would fantastic to have that in this patch series, and might not be
too difficult to do. However, I also think that always waking up the
hardware domain could be a decent way to start and could be OK for this
patch series which is the very first to introduce suspend/resume
functionalities on Xen on ARM. But the limitation should be well
explained in the commit message, as Julien wrote, and possibly even as
an in-code comment with a TODO.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Stefano Stabellini]

On Fri, 16 Nov 2018, Stefano Stabellini wrote:
> On Fri, 16 Nov 2018, Julien Grall wrote:
> > On 16/11/2018 12:34, Mirela Simonovic wrote:
> > > Hi Julien,
> > 
> > Hi Mirela,
> > 
> > > 
> > > On Fri, Nov 16, 2018 at 12:44 PM Julien Grall <julien.grall@arm.com> wrote:
> > > > 
> > > > 
> > > > 
> > > > On 16/11/2018 11:29, Mirela Simonovic wrote:
> > > > > On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
> > > > > <mirela.simonovic@aggios.com> wrote:
> > > > > > 
> > > > > > Hi Julien,
> > > > > > 
> > > > > > On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com>
> > > > > > wrote:
> > > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > > > > > The resume of Dom0 should always follow Xen's resume. This is
> > > > > > > > done by unblocking the first vCPU of Dom0.
> > > > > > > 
> > > > > > > Please explain why you always need to resume Dom0 afterwards.
> > > > > > > 
> > > > > > 
> > > > > > We don't need to, but that is what is promised in the design spec.
> > > > 
> > > > You surely had some rationale when writing the promise in the design
> > > > document,
> > > > right?
> > > > 
> > > > So what is the reason behind it? I don't want to resume a domain if that's
> > > > not
> > > > necessary.
> > > > 
> > > > > > 
> > > > > 
> > > > > To be more specific - a domU that doesn't depend on dom0 can resume
> > > > > and work happily without dom0 being resumed, i.e. just Xen and domU
> > > > > resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> > > > > resumes" is not a must (when there are no PV drivers involved).
> > > > 
> > > > PV backends don't necessarily reside in the hardware domain. So how is
> > > > this
> > > > going to work for the other case?
> > > > 
> > > 
> > > I honestly believe that this is not necessary, and is sub-optimal. It
> > > relies on an assumption that dom0 contains all the PV drivers, which
> > > is not always correct.
> > 
> > Well, there are other reasons to resume the hardware domain. The hardware
> > domain owns most the devices and may be part of the suspend/resume path.
> > 
> > As you tie the host suspend to the hardware domain suspend, it may makes sense
> > to resume at the same time. It is the kind of rationale I would expect in the
> > commit message.
> >
> > > I would prefer if someone can tell us that any frontend will trigger
> > > an event to the backend, and the event would go through Xen. That way,
> > > this event would cause a domain containing the backend driver to
> > > resume. I think this is the best possible solution, but it relies on
> > > an assumption that the event will go through Xen, and I'm not
> > > knowledgeable enough to claim that this is indeed the case.
> > 
> > I think it is should work, the best way to find out if building a test case
> > for it.
> 
> Yes, PV protocols use hypercalls to send notifications to the other end.
> Specifically, the function at the Linux side is
> include/xen/events.h:notify_remote_via_evtchn. The Xen implementation
> for the hypercall is xen/common/event_channel.c:evtchn_send, where 'rd'
> is the destination domain.
> 
> It should be possible to figure out which domain needs to awaken from
> there.

Actually, evtchn_send eventually will trigger a proper interrupt
injection into the domain (xen/arch/arm/vgic.c:arch_evtchn_inject),
which will necessarely wake it up. So it is possible that it will
already work without any need for additional changes?


> It would fantastic to have that in this patch series, and might not be
> too difficult to do. However, I also think that always waking up the
> hardware domain could be a decent way to start and could be OK for this
> patch series which is the very first to introduce suspend/resume
> functionalities on Xen on ARM. But the limitation should be well
> explained in the commit message, as Julien wrote, and possibly even as
> an in-code comment with a TODO.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Stefano,

On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Fri, 16 Nov 2018, Stefano Stabellini wrote:
> > On Fri, 16 Nov 2018, Julien Grall wrote:
> > > On 16/11/2018 12:34, Mirela Simonovic wrote:
> > > > Hi Julien,
> > >
> > > Hi Mirela,
> > >
> > > >
> > > > On Fri, Nov 16, 2018 at 12:44 PM Julien Grall <julien.grall@arm.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On 16/11/2018 11:29, Mirela Simonovic wrote:
> > > > > > On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
> > > > > > <mirela.simonovic@aggios.com> wrote:
> > > > > > >
> > > > > > > Hi Julien,
> > > > > > >
> > > > > > > On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > > > > > > The resume of Dom0 should always follow Xen's resume. This is
> > > > > > > > > done by unblocking the first vCPU of Dom0.
> > > > > > > >
> > > > > > > > Please explain why you always need to resume Dom0 afterwards.
> > > > > > > >
> > > > > > >
> > > > > > > We don't need to, but that is what is promised in the design spec.
> > > > >
> > > > > You surely had some rationale when writing the promise in the design
> > > > > document,
> > > > > right?
> > > > >
> > > > > So what is the reason behind it? I don't want to resume a domain if that's
> > > > > not
> > > > > necessary.
> > > > >
> > > > > > >
> > > > > >
> > > > > > To be more specific - a domU that doesn't depend on dom0 can resume
> > > > > > and work happily without dom0 being resumed, i.e. just Xen and domU
> > > > > > resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> > > > > > resumes" is not a must (when there are no PV drivers involved).
> > > > >
> > > > > PV backends don't necessarily reside in the hardware domain. So how is
> > > > > this
> > > > > going to work for the other case?
> > > > >
> > > >
> > > > I honestly believe that this is not necessary, and is sub-optimal. It
> > > > relies on an assumption that dom0 contains all the PV drivers, which
> > > > is not always correct.
> > >
> > > Well, there are other reasons to resume the hardware domain. The hardware
> > > domain owns most the devices and may be part of the suspend/resume path.
> > >
> > > As you tie the host suspend to the hardware domain suspend, it may makes sense
> > > to resume at the same time. It is the kind of rationale I would expect in the
> > > commit message.
> > >
> > > > I would prefer if someone can tell us that any frontend will trigger
> > > > an event to the backend, and the event would go through Xen. That way,
> > > > this event would cause a domain containing the backend driver to
> > > > resume. I think this is the best possible solution, but it relies on
> > > > an assumption that the event will go through Xen, and I'm not
> > > > knowledgeable enough to claim that this is indeed the case.
> > >
> > > I think it is should work, the best way to find out if building a test case
> > > for it.
> >
> > Yes, PV protocols use hypercalls to send notifications to the other end.
> > Specifically, the function at the Linux side is
> > include/xen/events.h:notify_remote_via_evtchn. The Xen implementation
> > for the hypercall is xen/common/event_channel.c:evtchn_send, where 'rd'
> > is the destination domain.
> >
> > It should be possible to figure out which domain needs to awaken from
> > there.
>
> Actually, evtchn_send eventually will trigger a proper interrupt
> injection into the domain (xen/arch/arm/vgic.c:arch_evtchn_inject),
> which will necessarely wake it up. So it is possible that it will
> already work without any need for additional changes?
>

Absolutely, that sounds great :) Then we could just drop this patch.
Of course, we need to test to confirm

>
> > It would fantastic to have that in this patch series, and might not be
> > too difficult to do. However, I also think that always waking up the
> > hardware domain could be a decent way to start and could be OK for this
> > patch series which is the very first to introduce suspend/resume
> > functionalities on Xen on ARM. But the limitation should be well
> > explained in the commit message, as Julien wrote, and possibly even as
> > an in-code comment with a TODO.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Julien Grall]

On 16/11/2018 21:41, Mirela Simonovic wrote:
> Hi Stefano,
> 
> On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
>>
>> On Fri, 16 Nov 2018, Stefano Stabellini wrote:
>>> On Fri, 16 Nov 2018, Julien Grall wrote:
>>>> On 16/11/2018 12:34, Mirela Simonovic wrote:
>>>>> Hi Julien,
>>>>
>>>> Hi Mirela,
>>>>
>>>>>
>>>>> On Fri, Nov 16, 2018 at 12:44 PM Julien Grall <julien.grall@arm.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 16/11/2018 11:29, Mirela Simonovic wrote:
>>>>>>> On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
>>>>>>> <mirela.simonovic@aggios.com> wrote:
>>>>>>>>
>>>>>>>> Hi Julien,
>>>>>>>>
>>>>>>>> On Thu, Nov 15, 2018 at 9:31 PM Julien Grall <julien.grall@arm.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
>>>>>>>>>> The resume of Dom0 should always follow Xen's resume. This is
>>>>>>>>>> done by unblocking the first vCPU of Dom0.
>>>>>>>>>
>>>>>>>>> Please explain why you always need to resume Dom0 afterwards.
>>>>>>>>>
>>>>>>>>
>>>>>>>> We don't need to, but that is what is promised in the design spec.
>>>>>>
>>>>>> You surely had some rationale when writing the promise in the design
>>>>>> document,
>>>>>> right?
>>>>>>
>>>>>> So what is the reason behind it? I don't want to resume a domain if that's
>>>>>> not
>>>>>> necessary.
>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> To be more specific - a domU that doesn't depend on dom0 can resume
>>>>>>> and work happily without dom0 being resumed, i.e. just Xen and domU
>>>>>>> resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
>>>>>>> resumes" is not a must (when there are no PV drivers involved).
>>>>>>
>>>>>> PV backends don't necessarily reside in the hardware domain. So how is
>>>>>> this
>>>>>> going to work for the other case?
>>>>>>
>>>>>
>>>>> I honestly believe that this is not necessary, and is sub-optimal. It
>>>>> relies on an assumption that dom0 contains all the PV drivers, which
>>>>> is not always correct.
>>>>
>>>> Well, there are other reasons to resume the hardware domain. The hardware
>>>> domain owns most the devices and may be part of the suspend/resume path.
>>>>
>>>> As you tie the host suspend to the hardware domain suspend, it may makes sense
>>>> to resume at the same time. It is the kind of rationale I would expect in the
>>>> commit message.
>>>>
>>>>> I would prefer if someone can tell us that any frontend will trigger
>>>>> an event to the backend, and the event would go through Xen. That way,
>>>>> this event would cause a domain containing the backend driver to
>>>>> resume. I think this is the best possible solution, but it relies on
>>>>> an assumption that the event will go through Xen, and I'm not
>>>>> knowledgeable enough to claim that this is indeed the case.
>>>>
>>>> I think it is should work, the best way to find out if building a test case
>>>> for it.
>>>
>>> Yes, PV protocols use hypercalls to send notifications to the other end.
>>> Specifically, the function at the Linux side is
>>> include/xen/events.h:notify_remote_via_evtchn. The Xen implementation
>>> for the hypercall is xen/common/event_channel.c:evtchn_send, where 'rd'
>>> is the destination domain.
>>>
>>> It should be possible to figure out which domain needs to awaken from
>>> there.
>>
>> Actually, evtchn_send eventually will trigger a proper interrupt
>> injection into the domain (xen/arch/arm/vgic.c:arch_evtchn_inject),
>> which will necessarely wake it up. So it is possible that it will
>> already work without any need for additional changes?
>>
> 
> Absolutely, that sounds great :) Then we could just drop this patch.

I don't think you can drop this patch... As you tie the host suspend to 
the hardware domain suspend, it may makes sense to resume at the same time.

Otherwise we should provide a separate hypercall to suspend/resume the host.

The whole point of the thread is we need to document why the decision 
was made in one way or another. So when someone look at it in 2 years 
time, we know why it has been done like that.

Cheers,

-- 
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2066 bytes --]

On Thu, 2018-11-15 at 11:38 +0000, Julien Grall wrote:
> On 11/15/18 11:10 AM, Mirela Simonovic wrote:
> I don't think we are discussing the same thing. The discussion was 
> around other vCPUs, not the vCPU calling SYSTEM_SUSPEND.
> 
> Most likely in the future, we would want to allow the toolstack to 
> request resuming the domain. This can be considered as an event.
> 
> > Given the assumption, my understanding is that Xen itself will not
> > unblock vCPU, except due to an interrupt targeted to the guest.
> > Am I missing something? An example would be appreciated.
> 
> At least on Arm, the current semantics of vcpu_block/vcpu_unblock is
> to 
> block until you receive an events.
> 
> I don't much want to restrict the definition of events to only 
> interrupts.  To clarify my point, if you want to wake-up for any
> events 
> then fine. 
>
So, I certainly lack deep knowledge of PSCI, as well as other aspects
of how this suspend/resume logic will work, but just to clarify on
this.

vcpu_unblock() may indeed be called from a number of places, but it
actually wakes-up the vcpu _iff_ the vcpu is runnable. If it is not
--e.g., because pause_count is not zero, or any vcpu or domain
pause_flags are set-- the vcpu stays blocked (check the implementation
of vcpu_wake() and of vcpu_runnable()).

So, it looks to me that what you want is to be sure that when an event
arrives, but the vcpus need to remain suspended, vcpu_runnable()
returns false for them. OTOH, when the events that you want to wake
them up arrives, you want it to return true.

Whether that is better done by using pause_count (plus some other flag,
as Stefano is saying) or with either an existing or new vcpu or domain
pause flag, I don't know, but it looks like it could work to me.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1186 bytes --]

On Mon, 2018-11-12 at 15:33 +0000, Andrew Cooper wrote:
> On 12/11/18 15:31, Mirela Simonovic wrote:
> > 
> > Thanks, now it's clear. We need to change the type for
> > watchdog_timer
> > to be the derived structure of timer that additionally contains
> > 'suspended' variable. That makes sense, we'll do so.
> 
> Why do you need a suspended variable?  Stash the frequency the guest
> requested on the last hypercall, and fully kill/re-create the timer
> on
> suspend/resume.
> 
I *think* what they want is something like: when, at time t1, a domain
is suspended, it has a (watchdog) timer armed, which was going to
expire and fire in 13.2 ms (i.e., at t1 + 13.2ms); when, at time t2 >>
t1, the domain is resumed, that (watchdog) timer should fire in 13.2
ms, i.e. at t2 + 13.2ms.

This being said, I don't see a behavior like this terribly useful for a
whatchdog either (but I may be missing something).

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1369 bytes --]

On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
> On 16/11/2018 21:41, Mirela Simonovic wrote:
> > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > <sstabellini@kernel.org> wrote:
> > > > It should be possible to figure out which domain needs to
> > > > awaken from
> > > > there.
> > > 
> > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > injection into the domain
> > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > which will necessarely wake it up. So it is possible that it will
> > > already work without any need for additional changes?
> > > 
> > 
> > Absolutely, that sounds great :) Then we could just drop this
> > patch.
> 
> I don't think you can drop this patch... As you tie the host suspend
> to 
> the hardware domain suspend, it may makes sense to resume at the same
> time.
> 
FWIW, I think that too.

In fact, let's assume a *fully* disaggregated setup, where dom0 only
has the toolstack, while it has no hardware, no PV backend, etc... If
we don't resume it explicitly together with Xen, who is going to resume
it? :-O

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Stefano Stabellini]

On Sat, 17 Nov 2018, Dario Faggioli wrote:
> On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
> > On 16/11/2018 21:41, Mirela Simonovic wrote:
> > > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > > <sstabellini@kernel.org> wrote:
> > > > > It should be possible to figure out which domain needs to
> > > > > awaken from
> > > > > there.
> > > > 
> > > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > > injection into the domain
> > > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > > which will necessarely wake it up. So it is possible that it will
> > > > already work without any need for additional changes?
> > > > 
> > > 
> > > Absolutely, that sounds great :) Then we could just drop this
> > > patch.
> > 
> > I don't think you can drop this patch... As you tie the host suspend
> > to 
> > the hardware domain suspend, it may makes sense to resume at the same
> > time.
> > 
> FWIW, I think that too.
> 
> In fact, let's assume a *fully* disaggregated setup, where dom0 only
> has the toolstack, while it has no hardware, no PV backend, etc... If
> we don't resume it explicitly together with Xen, who is going to resume
> it? :-O

Yes, that's right. However, it should work for driver domains: there is
no need to wake up driver domains explicitly because they will be
woken up by the frontends?


> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Software Engineer @ SUSE https://www.suse.com/
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi,

On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Sat, 17 Nov 2018, Dario Faggioli wrote:
> > On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
> > > On 16/11/2018 21:41, Mirela Simonovic wrote:
> > > > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > > > <sstabellini@kernel.org> wrote:
> > > > > > It should be possible to figure out which domain needs to
> > > > > > awaken from
> > > > > > there.
> > > > >
> > > > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > > > injection into the domain
> > > > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > > > which will necessarely wake it up. So it is possible that it will
> > > > > already work without any need for additional changes?
> > > > >
> > > >
> > > > Absolutely, that sounds great :) Then we could just drop this
> > > > patch.
> > >
> > > I don't think you can drop this patch... As you tie the host suspend
> > > to
> > > the hardware domain suspend, it may makes sense to resume at the same
> > > time.
> > >
> > FWIW, I think that too.
> >
> > In fact, let's assume a *fully* disaggregated setup, where dom0 only
> > has the toolstack, while it has no hardware, no PV backend, etc... If
> > we don't resume it explicitly together with Xen, who is going to resume
> > it? :-O
>
> Yes, that's right. However, it should work for driver domains: there is
> no need to wake up driver domains explicitly because they will be
> woken up by the frontends?
>

I think we all agree, except some of us weren't so clear about it :)
For now, dom0 issues suspend and should resume as well when Xen
suspends. This is done in the series, resume is covered by this patch,
and commit message should be clarified.

If a domU has a backend, we should verify that it can be woken-up by
an event triggered by a frontend driver in another domain.

One day, this patch could be dropped/reverted if one come up with a
different logic for triggering Xen suspend. This should be of the
table for now, but a good option to remember for future.

>
> > <<This happens because I choose it to happen!>> (Raistlin Majere)
> > -----------------------------------------------------------------
> > Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> > Software Engineer @ SUSE https://www.suse.com/
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

On Sat, Nov 17, 2018 at 5:01 PM Mirela Simonovic
<mirela.simonovic@aggios.com> wrote:
>
> Hi,
>
> On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
> >
> > On Sat, 17 Nov 2018, Dario Faggioli wrote:
> > > On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
> > > > On 16/11/2018 21:41, Mirela Simonovic wrote:
> > > > > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > > > > <sstabellini@kernel.org> wrote:
> > > > > > > It should be possible to figure out which domain needs to
> > > > > > > awaken from
> > > > > > > there.
> > > > > >
> > > > > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > > > > injection into the domain
> > > > > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > > > > which will necessarely wake it up. So it is possible that it will
> > > > > > already work without any need for additional changes?
> > > > > >
> > > > >
> > > > > Absolutely, that sounds great :) Then we could just drop this
> > > > > patch.
> > > >
> > > > I don't think you can drop this patch... As you tie the host suspend
> > > > to
> > > > the hardware domain suspend, it may makes sense to resume at the same
> > > > time.
> > > >
> > > FWIW, I think that too.
> > >
> > > In fact, let's assume a *fully* disaggregated setup, where dom0 only
> > > has the toolstack, while it has no hardware, no PV backend, etc... If
> > > we don't resume it explicitly together with Xen, who is going to resume
> > > it? :-O
> >
> > Yes, that's right. However, it should work for driver domains: there is
> > no need to wake up driver domains explicitly because they will be
> > woken up by the frontends?
> >
>
> I think we all agree, except some of us weren't so clear about it :)
> For now, dom0 issues suspend and should resume as well when Xen
> suspends. This is done in the series, resume is covered by this patch,
resumes

> and commit message should be clarified.
>
> If a domU has a backend, we should verify that it can be woken-up by
> an event triggered by a frontend driver in another domain.
>
> One day, this patch could be dropped/reverted if one come up with a
> different logic for triggering Xen suspend. This should be of the
> table for now, but a good option to remember for future.
>
> >
> > > <<This happens because I choose it to happen!>> (Raistlin Majere)
> > > -----------------------------------------------------------------
> > > Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> > > Software Engineer @ SUSE https://www.suse.com/
> > >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi,

On Sat, Nov 17, 2018 at 5:02 PM Mirela Simonovic
<mirela.simonovic@aggios.com> wrote:
>
> On Sat, Nov 17, 2018 at 5:01 PM Mirela Simonovic
> <mirela.simonovic@aggios.com> wrote:
> >
> > Hi,
> >
> > On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
> > <sstabellini@kernel.org> wrote:
> > >
> > > On Sat, 17 Nov 2018, Dario Faggioli wrote:
> > > > On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
> > > > > On 16/11/2018 21:41, Mirela Simonovic wrote:
> > > > > > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > > > > > <sstabellini@kernel.org> wrote:
> > > > > > > > It should be possible to figure out which domain needs to
> > > > > > > > awaken from
> > > > > > > > there.
> > > > > > >
> > > > > > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > > > > > injection into the domain
> > > > > > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > > > > > which will necessarely wake it up. So it is possible that it will
> > > > > > > already work without any need for additional changes?
> > > > > > >
> > > > > >
> > > > > > Absolutely, that sounds great :) Then we could just drop this
> > > > > > patch.
> > > > >
> > > > > I don't think you can drop this patch... As you tie the host suspend
> > > > > to
> > > > > the hardware domain suspend, it may makes sense to resume at the same
> > > > > time.
> > > > >
> > > > FWIW, I think that too.
> > > >
> > > > In fact, let's assume a *fully* disaggregated setup, where dom0 only
> > > > has the toolstack, while it has no hardware, no PV backend, etc... If
> > > > we don't resume it explicitly together with Xen, who is going to resume
> > > > it? :-O

Forgot to answer this - when someone needs a toolstack, it will try to
type. That will raise a UART interrupt (UART used by Xen console). It
could be implemented that a UART interrupt by default wakes up dom0 if
it's asleep.
But I would keep anything like that out of this series.

> > >
> > > Yes, that's right. However, it should work for driver domains: there is
> > > no need to wake up driver domains explicitly because they will be
> > > woken up by the frontends?
> > >
> >
> > I think we all agree, except some of us weren't so clear about it :)
> > For now, dom0 issues suspend and should resume as well when Xen
> > suspends. This is done in the series, resume is covered by this patch,
> resumes
>
> > and commit message should be clarified.
> >
> > If a domU has a backend, we should verify that it can be woken-up by
> > an event triggered by a frontend driver in another domain.
> >
> > One day, this patch could be dropped/reverted if one come up with a
> > different logic for triggering Xen suspend. This should be of the
> > table for now, but a good option to remember for future.
> >
> > >
> > > > <<This happens because I choose it to happen!>> (Raistlin Majere)
> > > > -----------------------------------------------------------------
> > > > Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> > > > Software Engineer @ SUSE https://www.suse.com/
> > > >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Julien Grall]

On 11/17/18 4:01 PM, Mirela Simonovic wrote:
> Hi,

Hi Mirela,

> 
> On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
>>
>> On Sat, 17 Nov 2018, Dario Faggioli wrote:
>>> On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
>>>> On 16/11/2018 21:41, Mirela Simonovic wrote:
>>>>> On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
>>>>> <sstabellini@kernel.org> wrote:
>>>>>>> It should be possible to figure out which domain needs to
>>>>>>> awaken from
>>>>>>> there.
>>>>>>
>>>>>> Actually, evtchn_send eventually will trigger a proper interrupt
>>>>>> injection into the domain
>>>>>> (xen/arch/arm/vgic.c:arch_evtchn_inject),
>>>>>> which will necessarely wake it up. So it is possible that it will
>>>>>> already work without any need for additional changes?
>>>>>>
>>>>>
>>>>> Absolutely, that sounds great :) Then we could just drop this
>>>>> patch.
>>>>
>>>> I don't think you can drop this patch... As you tie the host suspend
>>>> to
>>>> the hardware domain suspend, it may makes sense to resume at the same
>>>> time.
>>>>
>>> FWIW, I think that too.
>>>
>>> In fact, let's assume a *fully* disaggregated setup, where dom0 only
>>> has the toolstack, while it has no hardware, no PV backend, etc... If
>>> we don't resume it explicitly together with Xen, who is going to resume
>>> it? :-O
>>
>> Yes, that's right. However, it should work for driver domains: there is
>> no need to wake up driver domains explicitly because they will be
>> woken up by the frontends?
>>
> 
> I think we all agree, except some of us weren't so clear about it :)
> For now, dom0 issues suspend and should resume as well when Xen
> suspends. This is done in the series, resume is covered by this patch,
> and commit message should be clarified.
> 
> If a domU has a backend, we should verify that it can be woken-up by
> an event triggered by a frontend driver in another domain.
> 
> One day, this patch could be dropped/reverted if one come up with a
> different logic for triggering Xen suspend. This should be of the
> table for now, but a good option to remember for future.

Such change cannot be easily dropped because some hardware domain OS may 
rely on that behavior.

I am also interested to see how this is going to fit with the Dom0less 
use case. The end goal is to have no Dom0/Hardware domain. So how do you 
expect suspend/resume to work in that case? Note that I am not asking to 
implement it :).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 18/18] xen/arm: Suspend/resume console on Xen suspend/resume

[Julien Grall]

Hi Mirela,

On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> This is done using generic console_suspend/resume functions that cause
> uart driver specific suspend/resume handlers to be called for each
> initialized port (if the port has suspend/resume driver handlers
> implemented).
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@aggios.com>
> Signed-off-by: Saeed Nowshadi <saeed.nowshadi@xilinx.com>
> ---
>   xen/arch/arm/suspend.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> index a05aea9c25..6d7d69539b 100644
> --- a/xen/arch/arm/suspend.c
> +++ b/xen/arch/arm/suspend.c
> @@ -1,5 +1,6 @@
>   #include <xen/sched.h>
>   #include <xen/cpu.h>
> +#include <xen/console.h>
>   #include <asm/cpufeature.h>
>   #include <asm/event.h>
>   #include <asm/psci.h>
> @@ -149,6 +150,15 @@ static long system_suspend(void *data)
>           goto resume_irqs;
>       }
>   
> +    dprintk(XENLOG_DEBUG, "Suspend\n");

This message may not appear on the console unless it is synchronized 
(see console_start_sync). Also, it might be a useful message in 
non-debug build.

Ideally the suspend/resume logic should not be much different than x86 
(see arch/x86/acpi/power.c).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Julien,

On Tue, Nov 27, 2018 at 7:36 PM Julien Grall <julien.grall@arm.com> wrote:
>
>
>
> On 11/17/18 4:01 PM, Mirela Simonovic wrote:
> > Hi,
>
> Hi Mirela,
>
> >
> > On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
> > <sstabellini@kernel.org> wrote:
> >>
> >> On Sat, 17 Nov 2018, Dario Faggioli wrote:
> >>> On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
> >>>> On 16/11/2018 21:41, Mirela Simonovic wrote:
> >>>>> On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> >>>>> <sstabellini@kernel.org> wrote:
> >>>>>>> It should be possible to figure out which domain needs to
> >>>>>>> awaken from
> >>>>>>> there.
> >>>>>>
> >>>>>> Actually, evtchn_send eventually will trigger a proper interrupt
> >>>>>> injection into the domain
> >>>>>> (xen/arch/arm/vgic.c:arch_evtchn_inject),
> >>>>>> which will necessarely wake it up. So it is possible that it will
> >>>>>> already work without any need for additional changes?
> >>>>>>
> >>>>>
> >>>>> Absolutely, that sounds great :) Then we could just drop this
> >>>>> patch.
> >>>>
> >>>> I don't think you can drop this patch... As you tie the host suspend
> >>>> to
> >>>> the hardware domain suspend, it may makes sense to resume at the same
> >>>> time.
> >>>>
> >>> FWIW, I think that too.
> >>>
> >>> In fact, let's assume a *fully* disaggregated setup, where dom0 only
> >>> has the toolstack, while it has no hardware, no PV backend, etc... If
> >>> we don't resume it explicitly together with Xen, who is going to resume
> >>> it? :-O
> >>
> >> Yes, that's right. However, it should work for driver domains: there is
> >> no need to wake up driver domains explicitly because they will be
> >> woken up by the frontends?
> >>
> >
> > I think we all agree, except some of us weren't so clear about it :)
> > For now, dom0 issues suspend and should resume as well when Xen
> > suspends. This is done in the series, resume is covered by this patch,
> > and commit message should be clarified.
> >
> > If a domU has a backend, we should verify that it can be woken-up by
> > an event triggered by a frontend driver in another domain.
> >
> > One day, this patch could be dropped/reverted if one come up with a
> > different logic for triggering Xen suspend. This should be of the
> > table for now, but a good option to remember for future.
>
> Such change cannot be easily dropped because some hardware domain OS may
> rely on that behavior.
>
> I am also interested to see how this is going to fit with the Dom0less
> use case. The end goal is to have no Dom0/Hardware domain. So how do you
> expect suspend/resume to work in that case? Note that I am not asking to
> implement it :).
>

From the implementation perspective and future changes which would be
required in this series it's not going to be the problem to support
dom0less approach - just one line of code (the if statement that
checks whether the hardware domain suspended) has to be replaced with
some other check. That other check would be a new condition to suspend
Xen that needs to be implemented. What that check would do depends on
the system architecture and target use cases that are specific to
dom0less architecture. I'm not so familiar with dom0less work, so
can't really say what would be the best approach to suspend condition
rules.
Do you have some whitepaper or anything similar that describes an
example of a target system architecture and/or use cases for dom0less
setup?
In dom0less world, who creates other domains? Is some domain still
more privileged than the others? E.g. is there a domain which knows
about other domains in the system so that it could do the coordination
in EL1?

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Julien Grall]

Hi Mirela,

On 11/29/18 2:02 PM, Mirela Simonovic wrote:
> Hi Julien,
> 
> On Tue, Nov 27, 2018 at 7:36 PM Julien Grall <julien.grall@arm.com> wrote:
>>
>>
>>
>> On 11/17/18 4:01 PM, Mirela Simonovic wrote:
>>> Hi,
>>
>> Hi Mirela,
>>
>>>
>>> On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
>>> <sstabellini@kernel.org> wrote:
>>>>
>>>> On Sat, 17 Nov 2018, Dario Faggioli wrote:
>>>>> On Fri, 2018-11-16 at 21:58 +0000, Julien Grall wrote:
>>>>>> On 16/11/2018 21:41, Mirela Simonovic wrote:
>>>>>>> On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
>>>>>>> <sstabellini@kernel.org> wrote:
>>>>>>>>> It should be possible to figure out which domain needs to
>>>>>>>>> awaken from
>>>>>>>>> there.
>>>>>>>>
>>>>>>>> Actually, evtchn_send eventually will trigger a proper interrupt
>>>>>>>> injection into the domain
>>>>>>>> (xen/arch/arm/vgic.c:arch_evtchn_inject),
>>>>>>>> which will necessarely wake it up. So it is possible that it will
>>>>>>>> already work without any need for additional changes?
>>>>>>>>
>>>>>>>
>>>>>>> Absolutely, that sounds great :) Then we could just drop this
>>>>>>> patch.
>>>>>>
>>>>>> I don't think you can drop this patch... As you tie the host suspend
>>>>>> to
>>>>>> the hardware domain suspend, it may makes sense to resume at the same
>>>>>> time.
>>>>>>
>>>>> FWIW, I think that too.
>>>>>
>>>>> In fact, let's assume a *fully* disaggregated setup, where dom0 only
>>>>> has the toolstack, while it has no hardware, no PV backend, etc... If
>>>>> we don't resume it explicitly together with Xen, who is going to resume
>>>>> it? :-O
>>>>
>>>> Yes, that's right. However, it should work for driver domains: there is
>>>> no need to wake up driver domains explicitly because they will be
>>>> woken up by the frontends?
>>>>
>>>
>>> I think we all agree, except some of us weren't so clear about it :)
>>> For now, dom0 issues suspend and should resume as well when Xen
>>> suspends. This is done in the series, resume is covered by this patch,
>>> and commit message should be clarified.
>>>
>>> If a domU has a backend, we should verify that it can be woken-up by
>>> an event triggered by a frontend driver in another domain.
>>>
>>> One day, this patch could be dropped/reverted if one come up with a
>>> different logic for triggering Xen suspend. This should be of the
>>> table for now, but a good option to remember for future.
>>
>> Such change cannot be easily dropped because some hardware domain OS may
>> rely on that behavior.
>>
>> I am also interested to see how this is going to fit with the Dom0less
>> use case. The end goal is to have no Dom0/Hardware domain. So how do you
>> expect suspend/resume to work in that case? Note that I am not asking to
>> implement it :).
>>
> 
>  From the implementation perspective and future changes which would be
> required in this series it's not going to be the problem to support
> dom0less approach - just one line of code (the if statement that
> checks whether the hardware domain suspended) has to be replaced with
> some other check. That other check would be a new condition to suspend
> Xen that needs to be implemented. What that check would do depends on
> the system architecture and target use cases that are specific to
> dom0less architecture. I'm not so familiar with dom0less work, so
> can't really say what would be the best approach to suspend condition
> rules.
> Do you have some whitepaper or anything similar that describes an
> example of a target system architecture and/or use cases for dom0less
> setup?

Have a look at docs/features/dom0less.markdown

> In dom0less world, who creates other domains? Is some domain still
> more privileged than the others? E.g. is there a domain which knows
> about other domains in the system so that it could do the coordination
> in EL1?

It depends on your setup. In some setup, all the domains could be 
created by Xen and therefore there are no guest more privileged than 
one. But in other setup, you may still have a privilege domain (i.e 
hwdom/dom0).

What I want to avoid is if we tie ourself to an ABI than can't be 
changed anymore. In the guest context, I guess considering that Xen will 
suspend/resuming when Xen is suspending/resuming is probably a safe bet.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel