Re: [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic <mirela.simonovic@aggios.com>]

[-- Attachment #1.1: Type: text/plain, Size: 7629 bytes --]

Hi,

One thing I screwed - I forgot to remove changes log from an internal
review, so please ignore it. This is officially the first version.

Thanks,
Mirela

On Mon, Nov 12, 2018 at 12:31 PM Mirela Simonovic <
mirela.simonovic@aggios.com> wrote:

> This series contains support for suspend to RAM (in the following text just
> 'suspend') for Xen on arm64. The implementation is aligned with the design
> specification that has been proposed on xen-devel list:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>
> At a high-level the series contains:
> 1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
> 2) Support for resuming a guest on an interrupt targeted to that guest
> 3) Support for suspending Xen after dom0 finalizes the suspend
> 4) Support for resuming Xen on an interrupt that is targeted to a guest
>
>
> --------------------------------------------------------------------------------
> In more details:
>
> *** About suspending/resuming guests
>
> The patches included in this series allow PSCI compliant guests that have
> support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
> suspend and resume on top of Xen without any EL1 code changes.
>
> During their suspend procedure guests will hot-unplug their secondary CPUs,
> triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
> suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
> Guests will save/restore their own EL1 context on suspend/resume.
>
> A guest is expected to leave enabled interrupts that are considered to be
> its
> wake-up sources. Those interrupts will be able to wake up the guest. This
> holds
> regardless of the state of the underlying software layers, i.e. whether
> Xen gets
> suspended or not doesn't affect the ability of the guest to wake up.
>
> First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from
> which
> the guest assumes to start on resume. On resume, guests assume to be
> running in
> an environment whose state matches the CPU state after reset, e.g. with
> reset
> register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
> VCPU context and save the resume entry point into program counter before
> the
> guest's VCPU gets scheduled in on resume. This is done when the guest
> finalizes
> its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure
> that
> the resume-ready VCPU's context does not get overwritten later upon the
> context
> switch when the VCPU is scheduled out.
> Xen also needs to take care that the guest's view of GIC and timer gets
> saved.
> Also, while a guest is suspended its watchdogs are paused, in order to
> avoid
> watchdog triggered shutdown of a guest that has been asleep for a period
> of time
> that is longer than the watchdog period.
>
> After this point, from Xen's point of view a suspended guest has one VCPU
> blocked, waiting for an interrupt. When such an interrupt comes, Xen will
> unblock the VCPU of the suspended domain, which results in the guest
> resuming.
>
> *** About suspending/resuming Xen
>
> Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
> considered to be the decision maker for EL1 and EL2.
> On suspend, Xen will first freeze all domains. Then, Xen disables physical
> secondary CPUs, which leads to physical CPU_OFF to be called by each
> secondary
> CPU. After that Xen finalizes the suspend from the boot CPU.
>
> This consists of suspending the timer, i.e. suppressing its interrupts (we
> don't
> want to be woken up by a timer, there is no VCPU ready to be scheduled).
> Then
> the state of GIC is saved, console is suspended, and CPU context is saved.
> The
> saved context tells where Xen needs to continue execution on resume.
> Since Xen will resume with MMU disabled, the first thing to do in resume
> is to
> resume memory management in order to be able to access the context that
> needs to
> be restored (we know virtual address of the context data). Finally Xen
> calls
> SYSTEM_SUSPEND PSCI to the EL3.
>
> When resuming, all the steps done in suspend need to be reverted. This is
> completed by unblocking dom0's VCPU, because we always want the dom0 to
> resume,
> regardless of the target domain whose interrupt woke up Xen.
>
> *** Handling of unprivileged guests during Xen suspend/resume
>
> Any domU that is not suspended when dom0 suspends will be frozen, domUs
> that are
> already suspended remain suspended. On resume the suspended domUs still
> remain
> suspended (unless their wake interrupt caused Xen to wake) while the
> others will
> be thawed.
>
> For more details please refer to patches or the design specification:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>
>
> --------------------------------------------------------------------------------
> The series does not include:
> a) UART driver-specific suspend/resume that gets called when console
> suspends
> b) SMMU suspend/resume
> c) Suspend coordination support that would allow dom0 to request domUs to
> suspend
> These will be submitted in the following series.
>
> Mirela Simonovic (16):
>   xen/arm: Implement PSCI system suspend call (virtual interface)
>   xen/arm: Save GIC and virtual timer context when the domain suspends
>   xen/arm: While a domain is suspended put its watchdogs on pause
>   xen/arm: Trigger Xen suspend when Dom0 completes suspend
>   xen/x86: Move freeze/thaw_domains into common files
>   xen/arm: Freeze domains on suspend and thaw them on resume
>   xen/arm: Disable/enable non-boot physical CPUs on suspend/resume
>   xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume
>   xen/arm: Implement GIC suspend/resume functions (gicv2 only)
>   xen/arm: Suspend/resume GIC on system suspend/resume
>   xen/arm: Suspend/resume timer interrupt generation
>   xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)
>   xen/arm: Resume memory management on Xen resume
>   xen/arm: Save/restore context on suspend/resume
>   xen/arm: Resume Dom0 after Xen resumes
>   xen/arm: Suspend/resume console on Xen suspend/resume
>
> Saeed Nowshadi (2):
>   xen/arm: Move code that initializes VCPU context into a separate
>     function
>   xen/arm: Convert setting MMU page tables code into a routine
>
>  xen/arch/arm/Makefile            |   1 +
>  xen/arch/arm/arm64/entry.S       | 178 ++++++++++++++++++++++++
>  xen/arch/arm/arm64/head.S        | 265 ++++++++++++++++++-----------------
>  xen/arch/arm/domain.c            |  62 ++++++---
>  xen/arch/arm/gic-v2.c            | 147 ++++++++++++++++++++
>  xen/arch/arm/gic.c               |  27 ++++
>  xen/arch/arm/mm.c                |   1 +
>  xen/arch/arm/psci.c              |  16 +++
>  xen/arch/arm/suspend.c           | 292
> +++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/time.c              |  22 +++
>  xen/arch/arm/vpsci.c             |  19 +++
>  xen/arch/x86/acpi/power.c        |  28 ----
>  xen/common/domain.c              |  29 ++++
>  xen/common/schedule.c            |  38 +++++
>  xen/include/asm-arm/gic.h        |   8 ++
>  xen/include/asm-arm/perfc_defn.h |   1 +
>  xen/include/asm-arm/psci.h       |   3 +
>  xen/include/asm-arm/suspend.h    |  39 ++++++
>  xen/include/asm-arm/time.h       |   3 +
>  xen/include/xen/domain.h         |   1 +
>  xen/include/xen/sched.h          |  11 ++
>  xen/include/xen/timer.h          |   3 +
>  22 files changed, 1019 insertions(+), 175 deletions(-)
>  create mode 100644 xen/arch/arm/suspend.c
>  create mode 100644 xen/include/asm-arm/suspend.h
>
> --
> 2.13.0
>
>

[-- Attachment #1.2: Type: text/html, Size: 8799 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel