From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 566EDC433EF for ; Tue, 8 Feb 2022 09:48:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5KcbbsyZ0vswg6HY0lXKj0tsBkGJRt+w1QqXTXUdd/c=; b=H6HvPF/R/rLRCr 9bHJ84Hq+6Rye3rHCpttLdlR/WisQW4nWvEaqUXbqly7+uXpM5cQGKUW3z1mXNS18IhteZHOw4eOH eNXzImyCFG8ntAf1jIvwn02KDJmbHQ8sNZU0NxAbS++A3RuS0tvE8YDc9mJFZOsIlOCgKD2/GZk8I KTnIOH2MxAr0K0O3TTFTUW+CA3I6F2XBTL/Ss28Mji9oiCBdqy8pKLhLKm9Ik332OeIuvg2Z/h5Xp G3ytZWPohgM7u/3zDDMqwSpRHZnXKmUtpg9t/MWzaGmESFxKBnhiChidN37okgB+Vyp1N8kfB/2zK NlChXWK8vZDgAc9OKqvg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHN4y-00DIY8-BE; Tue, 08 Feb 2022 09:46:32 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHN4t-00DIXP-OJ for linux-arm-kernel@lists.infradead.org; Tue, 08 Feb 2022 09:46:29 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D9718B817D0; Tue, 8 Feb 2022 09:46:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AA0B9C004E1; Tue, 8 Feb 2022 09:46:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644313584; bh=N2iwFt+9nu2+miw7Yd5yWcMLN1r1UbYEZrqOpFnZfVY=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=Tg7xQX2aFayXpgxebMRXN32qDsQLMRc5MalU2t91q1q7VDMd0YBdTLKrltFONY6qS vHvy7FENHlTVkIwCJKTeBpfkBIMuOBD/J+l0bsgxjigeFXZMJuF+6HZt8l3TeyQvp0 wpxLQJmW7BPc9NRxx5cgd1Gk3jlHEjMrfuBdXg9WZXUz1JPIzaKnW26BTrd9vU2TTw CGT2xxVM9XFgxO7Uf4hXcXKx7I9rN08k3pftUMqhM1FGr3icweYCN9hvvBvluB/s0P JJQyhxXmkbHSBLbAbz0IhxMZyYnr5e+ejYopvsBH0DtySSmMQZtu4FQJn/YLGR7WWE Yfs6vDtHrfysQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=billy-the-mountain.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nHN4o-006DrU-IX; Tue, 08 Feb 2022 09:46:22 +0000 Date: Tue, 08 Feb 2022 09:46:16 +0000 Message-ID: <878ruld72v.wl-maz@kernel.org> From: Marc Zyngier To: Oliver Upton Cc: Raghavendra Rao Ananta , Andrew Jones , kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, Peter Maydell , Sean Christopherson Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe In-Reply-To: References: <87mtp5q3gx.wl-maz@kernel.org> <87fsuxq049.wl-maz@kernel.org> <20210825150713.5rpwzm4grfn7akcw@gator.home> <877dg8ppnt.wl-maz@kernel.org> <20210827074011.ci2kzo4cnlp3qz7h@gator.home> <87ilyitt6e.wl-maz@kernel.org> <87lf3drmvp.wl-maz@kernel.org> <875yq88app.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: oupton@google.com, rananta@google.com, drjones@redhat.com, kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, peter.maydell@linaro.org, seanjc@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220208_014628_112429_46B54B51 X-CRM114-Status: GOOD ( 39.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Huh, somewhat missed that email, apologies for the delay. On Tue, 25 Jan 2022 17:29:13 +0000, Oliver Upton wrote: > > Hi Marc, > > On Tue, Jan 25, 2022 at 12:46 AM Marc Zyngier wrote: > > > If I understand correctly, the original motivation for going with > > > pseudo-registers was to comply with QEMU, which uses KVM_GET_REG_LIST > > > and KVM_[GET|SET]_ONE_REG interface, but I'm guessing the VMMs doing > > > save/restore across migration might write the same values for every > > > vCPU. > > > > KVM currently restricts the vcpu features to be unified across vcpus, > > but that's only an implementation choice. > > But that implementation choice has become ABI, no? How could support > for asymmetry be added without requiring userspace opt-in or breaking > existing VMMs that depend on feature unification? Of course, you'd need some sort of advertising of this new behaviour. One thing I would like to add to the current state of thing is an indication of whether the effects of a sysreg being written from userspace are global or local to a vcpu. You'd need a new capability, and an extra flag added to the encoding of each register. > > > The ARM architecture doesn't > > mandate that these registers are all the same, and it isn't impossible > > that we'd allow for the feature set to become per-vcpu at some point > > in time. So this argument doesn't really hold. > > Accessing per-VM state N times is bound to increase VM blackout time > during migrations ~linearly as the number of vCPUs in a VM increases, > since a VM scoped lock is necessary to serialize guest accesses. It > could be tolerable at present scale, but seems like in the future it > could become a real problem. I don't disagree. But I doubt switching to a different API altogether is the solution to this. > > > Furthermore, compatibility with QEMU's save/restore model is > > essential, and AFAICT, there is no open source alternative. > > Agree fundamentally, but I believe it is entirely reasonable to > require a userspace change to adopt a new KVM feature. Otherwise, we > may be trying to shoehorn new features into existing UAPI that may not > be a precise fit.. But the very purpose of this API is to support discoverability. If we can't support it, then we might as well declare the whole API deprecated and restart from scratch. No, I'm not seriously suggesting this :-/. > In order to cure the serialization mentioned above, two options are > top of mind: accessing the VM state with the VM FD or informing > userspace that a set of registers need only be written once for an > entire VM. If we add support for asymmetry later down the road, that > would become an opt-in such that userspace will do the access > per-vCPU. It is the latter that I'm suggesting. > > > A device means yet another configuration and migration API. Don't you > > think we have enough of those? The complexity of KVM/arm64 userspace > > API is already insane, and extremely fragile. Adding to it will be a > > validation nightmare (it already is, and I don't see anyone actively > > helping with it). > > It seems equally fragile to introduce VM-wide serialization to vCPU > UAPI that we know is in the live migration critical path for _any_ > VMM. Without requiring userspace changes for all the new widgets under > discussion we're effectively forcing VMMs to do something suboptimal. I'm perfectly happy with suboptimal to start with, and find ways to make it better once we have a clear path forward. I just don't want to conflate the two. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel