From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 157B8C07E94 for ; Fri, 4 Jun 2021 07:25:42 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D1F2661415 for ; Fri, 4 Jun 2021 07:25:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D1F2661415 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7FpzAOl3tO8IabVYT2WT0pkvfe8V7tx1N3V7jYsnDJ4=; b=3H+9a48CyTrFZA U9MgLHKJNmlQEJxLKBDJxRZsAyK4OROJxe40K+8e4G7uJpMWbXuGigNr1cIWMyxcTim+AVBIjujo+ eHrZBRqlNNfVqa4FIQC9ktYp1VDMpAHYGlBSB7njUPX84sFijrQ4HDSHWsgJ3HF3MJmwDM3cUEgm8 lI7xAMuEPh5cEm3ngbCfic70bQtiBxur5pcZV2Ilayyh1hZ0ENLO8d99Cdcc+yT65EkJMDB/JMMUf myCPdR6EptWd1ptCupzogoiRjxooaDOs4JzbsYDYDyf2DpRnYabumygKufSA0gP9pHCOSg36gm7fU Ki+K7DMMCQXTYuxCopeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lp4BO-00C2xh-Lx; Fri, 04 Jun 2021 07:23:54 +0000 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lp4B9-00C2ql-4j for linux-arm-kernel@lists.infradead.org; Fri, 04 Jun 2021 07:23:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622791416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=orYq01Q31Yc39qB3ORg0nxtfa/WHWL4qWvUfOGPRsP0=; b=Dx8S5xsCHvf+cKjydq/HbgnwprTpC+YpINu9i+bWFgXpWJxRORoP6XQaHHWHDpzXZ1XBII WCkNF0O2N5wUdFNPu0wrPwqBD4uX/EGPyrRzfNpuD7EyX8v46D9OSGcw3DcFILKxU5kLXK ELRWb3R9Gq6tpX/wfBZuLYWEAJw/Ajg= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-589-CrVxt9H8MpClU2_M5VtQ0A-1; Fri, 04 Jun 2021 03:22:00 -0400 X-MC-Unique: CrVxt9H8MpClU2_M5VtQ0A-1 Received: by mail-ed1-f72.google.com with SMTP id h18-20020a05640250d2b029038cc3938914so4541406edb.17 for ; Fri, 04 Jun 2021 00:22:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=orYq01Q31Yc39qB3ORg0nxtfa/WHWL4qWvUfOGPRsP0=; b=nosQlwUJWzVNeJB6izzP4iDJ82nhFns7h/VmMdBFSAvrKPkSvEldfm3hR8tIODSuJr q02G3RqY1xPFFbNg9UeCKKJlnl6ig2R2PifWyXRn/doxNkUkelnuNvtt6xmzNG3xAagE MJGkaVM3EfChyAXzcl2En0MEXT4zMiJKIuH0AceXwNhVbT+g/OLUzcPfuC2PqpczelvU t2g5wmohQPBJt3cdQmIJi+L6ns11hGnhhEnukobRzgbwe+xjjSvjVCAAROW/UaZ2BE0Q HhynJOlIm1EMsAONGhcfwypEEl5hI41QYWIVqQW7yYCxC/QMbH7zBPJACDk/GywkWx+I dewA== X-Gm-Message-State: AOAM530XH82WDNXGe0WlzcxBHonwZEudyTT5/vo0dun7p+rsgG6tyk0q eatCfZPpovg3oLAWbnLsZvm4qbg7xouuxZTbXubHay8VoRd4NJ0hmV18fIs4qymuptF8Qq5AcKt Ea3raLpIo+e0tHE8UEeUdg/plE5X6hpn41AM= X-Received: by 2002:a17:906:3042:: with SMTP id d2mr2962220ejd.234.1622791319623; Fri, 04 Jun 2021 00:21:59 -0700 (PDT) X-Google-Smtp-Source: ABdhPJye48vq4jhWt3rfSGy/IZIfVkJe+CPcNZKJdo8nz+4VSYXPjlnXO0JZDmfdCwTwFNTDbvESog== X-Received: by 2002:a17:906:3042:: with SMTP id d2mr2962196ejd.234.1622791319297; Fri, 04 Jun 2021 00:21:59 -0700 (PDT) Received: from vitty.brq.redhat.com (g-server-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id gx23sm2456681ejb.125.2021.06.04.00.21.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Jun 2021 00:21:58 -0700 (PDT) From: Vitaly Kuznetsov To: Sergey Senozhatsky Cc: Marc Zyngier , Huacai Chen , Paul Mackerras , Christian Borntraeger , Suleiman Souhlal , x86@kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, linux-s390@vger.kernel.org, Sergey Senozhatsky , Paolo Bonzini , Sean Christopherson , Jim Mattson Subject: Re: [RFC][PATCH] kvm: add suspend pm-notifier In-Reply-To: <20210603164315.682994-1-senozhatsky@chromium.org> References: <20210603164315.682994-1-senozhatsky@chromium.org> Date: Fri, 04 Jun 2021 09:21:56 +0200 Message-ID: <87a6o614dn.fsf@vitty.brq.redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vkuznets@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210604_002339_311651_E5335626 X-CRM114-Status: GOOD ( 23.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Sergey Senozhatsky writes: > Add KVM suspend/hibernate PM-notifier which lets architectures > to implement arch-specific VM suspend code. For instance, on x86 > this sets PVCLOCK_GUEST_STOPPED on all the VCPUs. > > Our case is that user puts the host system into sleep multiple > times a day (e.g. closes the laptop's lid) so we need a reliable > way to suspend VMs properly. > > Signed-off-by: Sergey Senozhatsky > --- > arch/arm64/kvm/arm.c | 4 ++++ > arch/mips/kvm/mips.c | 4 ++++ > arch/powerpc/kvm/powerpc.c | 4 ++++ > arch/s390/kvm/kvm-s390.c | 4 ++++ > arch/x86/kvm/x86.c | 21 ++++++++++++++++++++ > include/linux/kvm_host.h | 8 ++++++++ > virt/kvm/kvm_main.c | 40 ++++++++++++++++++++++++++++++++++++++ > 7 files changed, 85 insertions(+) > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c > index 1126eae27400..547dbe44d039 100644 > --- a/arch/arm64/kvm/arm.c > +++ b/arch/arm64/kvm/arm.c > @@ -1311,6 +1311,10 @@ static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, > } > } > > +void kvm_arch_pm_notifier(struct kvm *kvm) > +{ > +} > + > long kvm_arch_vm_ioctl(struct file *filp, > unsigned int ioctl, unsigned long arg) > { > diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c > index 4d4af97dcc88..d4408acd2be6 100644 > --- a/arch/mips/kvm/mips.c > +++ b/arch/mips/kvm/mips.c > @@ -980,6 +980,10 @@ void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, > kvm_flush_remote_tlbs(kvm); > } > > +void kvm_arch_pm_notifier(struct kvm *kvm) > +{ > +} > + > long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) > { > long r; > diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c > index a2a68a958fa0..96e8a7b6fcf0 100644 > --- a/arch/powerpc/kvm/powerpc.c > +++ b/arch/powerpc/kvm/powerpc.c > @@ -2334,6 +2334,10 @@ static int kvmppc_get_cpu_char(struct kvm_ppc_cpu_char *cp) > } > #endif > > +void kvm_arch_pm_notifier(struct kvm *kvm) > +{ > +} > + > long kvm_arch_vm_ioctl(struct file *filp, > unsigned int ioctl, unsigned long arg) > { > diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c > index 1296fc10f80c..c5f86fc1e497 100644 > --- a/arch/s390/kvm/kvm-s390.c > +++ b/arch/s390/kvm/kvm-s390.c > @@ -2367,6 +2367,10 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) > return r; > } > > +void kvm_arch_pm_notifier(struct kvm *kvm) > +{ > +} > + > long kvm_arch_vm_ioctl(struct file *filp, > unsigned int ioctl, unsigned long arg) > { > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c > index bbc4e04e67ad..3f3d6497593f 100644 > --- a/arch/x86/kvm/x86.c > +++ b/arch/x86/kvm/x86.c > @@ -5613,6 +5613,27 @@ static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, void __user *argp) > return 0; > } > > +void kvm_arch_pm_notifier(struct kvm *kvm) > +{ > +#ifdef CONFIG_PM > + int c; > + > + mutex_lock(&kvm->lock); > + for (c = 0; c < kvm->created_vcpus; c++) { > + struct kvm_vcpu *vcpu = kvm->vcpus[c]; > + int r; > + > + if (!vcpu) > + continue; > + r = kvm_set_guest_paused(vcpu); > + if (!r) > + continue; > + pr_err("Failed to suspend VCPU-%d: %d\n", vcpu->vcpu_id, r); > + } > + mutex_unlock(&kvm->lock); > +#endif > +} > + > long kvm_arch_vm_ioctl(struct file *filp, > unsigned int ioctl, unsigned long arg) > { > diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h > index 2f34487e21f2..86695320a6b7 100644 > --- a/include/linux/kvm_host.h > +++ b/include/linux/kvm_host.h > @@ -37,6 +37,8 @@ > #include > #include > > +#include > + > #ifndef KVM_MAX_VCPU_ID > #define KVM_MAX_VCPU_ID KVM_MAX_VCPUS > #endif > @@ -579,6 +581,10 @@ struct kvm { > pid_t userspace_pid; > unsigned int max_halt_poll_ns; > u32 dirty_ring_size; > + > +#ifdef CONFIG_PM > + struct notifier_block pm_notifier; > +#endif > }; > > #define kvm_err(fmt, ...) \ > @@ -992,6 +998,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu); > void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu); > void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu); > > +void kvm_arch_pm_notifier(struct kvm *kvm); > + > #ifdef __KVM_HAVE_ARCH_VCPU_DEBUGFS > void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_dentry); > #endif > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c > index 6b4feb92dc79..86925ab7d162 100644 > --- a/virt/kvm/kvm_main.c > +++ b/virt/kvm/kvm_main.c > @@ -51,6 +51,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -779,6 +780,43 @@ static int kvm_init_mmu_notifier(struct kvm *kvm) > > #endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */ > > +#ifdef CONFIG_PM > +static int kvm_pm_notifier_call(struct notifier_block *bl, > + unsigned long state, > + void *unused) > +{ > + struct kvm *kvm = container_of(bl, struct kvm, pm_notifier); > + > + switch (state) { > + case PM_HIBERNATION_PREPARE: > + case PM_SUSPEND_PREPARE: > + kvm_arch_pm_notifier(kvm); > + break; > + } > + return NOTIFY_DONE; > +} > + > +static void kvm_init_pm_notifier(struct kvm *kvm) > +{ > + kvm->pm_notifier.notifier_call = kvm_pm_notifier_call; > + kvm->pm_notifier.priority = INT_MAX; > + register_pm_notifier(&kvm->pm_notifier); > +} > + > +static void kvm_destroy_pm_notifier(struct kvm *kvm) > +{ > + unregister_pm_notifier(&kvm->pm_notifier); > +} > +#else > +static void kvm_init_pm_notifier(struct kvm *kvm) > +{ > +} > + > +static void kvm_destroy_pm_notifier(struct kvm *kvm) > +{ > +} > +#endif /* CONFIG_PM */ > + > static struct kvm_memslots *kvm_alloc_memslots(void) > { > int i; > @@ -962,6 +1000,7 @@ static struct kvm *kvm_create_vm(unsigned long type) > mutex_unlock(&kvm_lock); > > preempt_notifier_inc(); > + kvm_init_pm_notifier(kvm); > You've probably thought it through and I didn't but wouldn't it be easier to have one global pm_notifier call for KVM which would go through the list of VMs instead of registering/deregistering a pm_notifier call for every created/destroyed VM? > return kvm; > > @@ -1009,6 +1048,7 @@ static void kvm_destroy_vm(struct kvm *kvm) > int i; > struct mm_struct *mm = kvm->mm; > > + kvm_destroy_pm_notifier(kvm); > kvm_uevent_notify_change(KVM_EVENT_DESTROY_VM, kvm); > kvm_destroy_vm_debugfs(kvm); > kvm_arch_sync_events(kvm); -- Vitaly _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel