From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 063C7C433F5 for ; Wed, 29 Sep 2021 18:24:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B502461465 for ; Wed, 29 Sep 2021 18:24:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B502461465 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fBeJODcehcLkSpEGS5zgfW0Jbg4zACrVhZ6PKzNGk/A=; b=pQbUWOxBNgn7wU 7QQfncFMJdm+jUpr8dNiSoR9eClYpgAZzwJ6ETvT47UjWbRhGu8AKhc3C442IfPt0bha8M3iU4MVU 8l5dv+n/8Pc8+XsFEaBR0iG/3k0iK4MRrdoRRDenjPD8g3CzESOIDatiuImtQxdwSoZu+vMbG9gpe bNJlWhsbyUm9wtzr/NKF0c32uAx4S41REzE6B5JbgyGtVi1HbiGDBdU8hZ6HgbpqK8sdMIfYdtWn8 xVCZR98Fqe0g+TQTxfYo0gY2Jqayf+qkI4f4lQ7LB++BGR/jjCOkGBrkLQjlMOUuNO1Si4nyQNrJS vHMPPbQtxjSxBMg9h5Ng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mVeDo-00C3bF-P7; Wed, 29 Sep 2021 18:22:24 +0000 Received: from mail-lf1-x129.google.com ([2a00:1450:4864:20::129]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mVeDk-00C3aQ-Kp for linux-arm-kernel@lists.infradead.org; Wed, 29 Sep 2021 18:22:22 +0000 Received: by mail-lf1-x129.google.com with SMTP id b20so14668342lfv.3 for ; Wed, 29 Sep 2021 11:22:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Qtfpl4QD2zskjNpgRMOv1rx5s1LAJUZnW7UP8LdwoyI=; b=jNO8noc6wU0vBuBuA5SgzKDJSdjcJtQzSu6NU2uxa8ZSMdSU0rBu5zmLhm+/yKCjoy ef+9UZO7RcYlfw89pWUi7lY4xnn8me2aZdS3wY5HXVu5P3KcRiAQQ6flwJa5W7D9nNWE +vtTRXVJPaeLOhyI/rzF4EWfzApnPvO8UWRG2sY1OU74NZvn/66Z2/yMf3454os4jWBw iQT+Gwy5fTV072sCo8419n5yroRDCEbVmOY6QLwxGPPPnFv8scUz0XuRBI51kwyt2WTW 3VWE5RJiCfA+/3QrFCWXNZWBAWYucDnltzpZp8V3SPSuPmWyMavydbnEeT8F6fDR5ZBi midQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Qtfpl4QD2zskjNpgRMOv1rx5s1LAJUZnW7UP8LdwoyI=; b=hdmmkI1NLezGHSmvo9qGfSqmnaFOgi7nbyWwZ4NYwqYUK8Ura5f+f0XLhy/PNos4eD 1N0WjCC5eLUMiQ3fS9evcyGSE2/SWPwtXS3hPIIHoX2qjwyZO449AHPXnh4bw9ZDyndv iqRHeE7jl4+0C5b7CsmA96p+H8MPgcZMfs8nkmSw1If3/W5105aQZfR93YFtu6W5qaxL +SFpyFyzDzLalwgPPwc+XUqm0zzMJQEjVDtQf3fwkLKI4ofhyV9YNNqLvCkqlcdB3rds QuT9xasrMahfshlx7KfyvgE3vUTjdt5ij0LApjFhKJUlhC1zqk8ovMT5FuM5YwR4EsoY yCpQ== X-Gm-Message-State: AOAM533h6TeOO/tlvelGlrlyQ/ml6u17B6nqk2ApQ7eb0Vq2kaNOgbiX nkjYkWEOQXtBYN1V4+2fcV4K9zjIISUOCMK6H4UyDg== X-Google-Smtp-Source: ABdhPJzaUBqKkCyaK9Fbh1Rz2z0TQHkOtkLi8JOjPmgJpuI5lcxIdv+YXlRfj2naCQ46/v6CbxB5cMwrUmzNFTnMYlI= X-Received: by 2002:ac2:4217:: with SMTP id y23mr1086804lfh.361.1632939736699; Wed, 29 Sep 2021 11:22:16 -0700 (PDT) MIME-Version: 1.0 References: <87mtp5q3gx.wl-maz@kernel.org> <87fsuxq049.wl-maz@kernel.org> <20210825150713.5rpwzm4grfn7akcw@gator.home> <877dg8ppnt.wl-maz@kernel.org> <20210827074011.ci2kzo4cnlp3qz7h@gator.home> In-Reply-To: <20210827074011.ci2kzo4cnlp3qz7h@gator.home> From: Oliver Upton Date: Wed, 29 Sep 2021 11:22:05 -0700 Message-ID: Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe To: Andrew Jones Cc: Marc Zyngier , kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, rananta@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, Peter Maydell X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210929_112220_728729_3F0BC9F3 X-CRM114-Status: GOOD ( 48.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Aug 27, 2021 at 12:40 AM Andrew Jones wrote: > > On Thu, Aug 26, 2021 at 06:49:27PM +0000, Oliver Upton wrote: > > On Thu, Aug 26, 2021 at 09:37:42AM +0100, Marc Zyngier wrote: > > > On Wed, 25 Aug 2021 19:14:59 +0100, > > > Oliver Upton wrote: > > > > > > > > On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones wrote: > > > > > > [...] > > > > > > > > Thanks for including me Marc. I think you've mentioned all the examples > > > > > of why we don't generally expect N+1 -> N migrations to work that I > > > > > can think of. While some of the examples like get-reg-list could > > > > > eventually be eliminated if we had CPU models to tighten our machine type > > > > > state, I think N+1 -> N migrations will always be best effort at most. > > > > > > > > > > I agree with giving userspace control over the exposer of the hypercalls > > > > > though. Using pseudo-registers for that purpose rather than a pile of > > > > > CAPs also seems reasonable to me. > > > > > > > > > > And, while I don't think this patch is going to proceed, I thought I'd > > > > > point out that the opt-out approach doesn't help much with expanding > > > > > our migration support unless we require the VMM to be upgraded first. > > > > > > > > > > And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't > > > > > work as expected, since the source enforce opt-out, but the destination > > > > > won't. > > > > > > > > Right, there's going to need to be a fence in both kernel and VMM > > > > versions. Before the fence, you can't rollback with either component. > > > > Once on the other side of the fence, the user may freely migrate > > > > between kernel + VMM combinations. > > > > > > > > > Also, since the VMM doesn't key off the kernel version, for the > > > > > most part N+1 VMMs won't know when they're supposed to opt-out or not, > > > > > leaving it to the user to ensure they consider everything. opt-in > > > > > usually only needs the user to consider what machine type they want to > > > > > launch. > > > > > > > > Going the register route will implicitly require opt-out for all old > > > > hypercalls. We exposed them unconditionally to the guest before, and > > > > we must uphold that behavior. The default value for the bitmap will > > > > have those features set. Any hypercalls added after that register > > > > interface will then require explicit opt-in from userspace. > > > > > > I disagree here. This makes the ABI inconsistent, and means that no > > > feature can be implemented without changing userspace. If you can deal > > > with the existing features, you should be able to deal with the next > > > lot. > > > > > > > With regards to the pseudoregister interface, how would a VMM discover > > > > new bits? From my perspective, you need to have two bitmaps that the > > > > VMM can get at: the set of supported feature bits and the active > > > > bitmap of features for a running guest. > > > > > > My proposal is that we have a single pseudo-register exposing the list > > > of implemented by the kernel. Clear the bits you don't want, and write > > > back the result. As long as you haven't written anything, you have the > > > full feature set. That's pretty similar to the virtio feature > > > negotiation. > > > > Ah, yes I agree. Thinking about it more we will not need something > > similar to KVM_GET_SUPPORTED_CPUID. > > > > So then, for any register where userspace/KVM need to negotiate > > features, the default value will return the maximum feature set that is > > supported. If userspace wants to constrain features, read out the > > register, make sure everything you want is there, and write it back > > blowing away the superfluous bits. Given this should we enforce ordering > > on feature registers, such that a VMM can only write to the registers > > before a VM is started? > > That's a good idea. KVM_REG_ARM64_SVE_VLS has this type of constraint so > we can model the feature register control off that. > > > > > Also, Reiji is working on making the identity registers writable for the > > sake of feature restriction. The suggested negotiation interface would > > be applicable there too, IMO. > > This this interesting news. I'll look forward to the posting. > > > > > Many thanks to both you and Drew for working this out with me. > > > > Thanks, > drew > Hey folks, I have some lingering thoughts on this subject since we last spoke and wanted to discuss. I'm having a hard time figuring out how a VMM should handle a new hypercall identity register introduced on a newer kernel. In order to maintain guest ABI, the VMM would need to know about that register and zero it when restoring an older guest. Perhaps instead we could reserve a range of firmware registers as the 'hypercall identity' registers. Implement all of them as RAZ/WI by default, encouraging userspace to zero these registers away for older VMs but still allowing an old userspace to pick up new KVM features. Doing so would align the hypercall identity registers with the feature ID registers from the architecture. Thoughts? -- Thanks, Oliver _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel