From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C48CAC432BE for ; Thu, 26 Aug 2021 18:49:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AE19061004 for ; Thu, 26 Aug 2021 18:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243349AbhHZSuU (ORCPT ); Thu, 26 Aug 2021 14:50:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231453AbhHZSuT (ORCPT ); Thu, 26 Aug 2021 14:50:19 -0400 Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D6C0C061757 for ; Thu, 26 Aug 2021 11:49:32 -0700 (PDT) Received: by mail-io1-xd2c.google.com with SMTP id z1so4999812ioh.7 for ; Thu, 26 Aug 2021 11:49:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=RVUTQwfXwIWJ8KSHNLLKmgxbQdeuJteYiwLVXKrHs2E=; b=RlH7+pWHb6AHxbU3UBBJJ37bcoBxFJ+OtQc00AVbSygJknZ/usx11UO93s13G1kXX9 CcZJrRyf0UdTbKCwUaojenkX5SVnXL9NHwvsqqAY/zQ//p0C+YM4cQD7FneQjJ3tB8Aa cMrJUpb9BQzx2jUWBC1kV31Q8634J4jt38JtglF1vnnYuD+EgmbxuiAdKNe0yIPy5viO tbjYlVlNGUw9dtSd01/0cTaV2aGVOlCz1pzGl7CAV/zNmDWjdq1bz6VMh9cU5i4GYnxj 4F3MA1cOOAlb0UtfQ1G0jAbM/WTqp+4jaVkgoa7ZNZ7B4nBPcn1mEfwDWFAId+KT8PYw UaHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=RVUTQwfXwIWJ8KSHNLLKmgxbQdeuJteYiwLVXKrHs2E=; b=HcZgTJtKVw4c5huz58v6OMo7ckyBeuP2qdK3+9MfwcKiaFikm9m69f1JiJ06qav9sq 53kSc/Bz0iXJNj4+GbG4iPv/GkGNIWWw8SnZwCD1Qs/eWD7Sln3/kRZl3BTf7pSI/L74 UM2Gp9RcYt1VHhTND3fMVVFcnpE9rLGCAtiYcj3JGJtJF39lI+rdzxXGpHUryfILrpie ImnCN0mfbpp1Es3D/ahPWlV6CkwY40I01TlMtbNCFsqZGU89U+NQB4DdnAE/MwKiN//v uYLaocICPjzEX64UqGRazeccFwolUfSYGv+s0BPVVC1p5gN+nbGxgaaCfOXdhJCJt7FL tp/g== X-Gm-Message-State: AOAM531EEkDBd6Iw7dkHwWbMRaKFjBQY4h9N8twGRd3gLjIbLvbo/bt4 1eON7v6R/3nmTRm2robsPLarZA== X-Google-Smtp-Source: ABdhPJx5cxoJqj5jt4PGdlUhbRpwDMVgyY3ef2R2HYspFPIJb/ORtT336DtZE2lfPru1xakwQbOqEA== X-Received: by 2002:a6b:3c16:: with SMTP id k22mr4252523iob.130.1630003771571; Thu, 26 Aug 2021 11:49:31 -0700 (PDT) Received: from google.com (194.225.68.34.bc.googleusercontent.com. [34.68.225.194]) by smtp.gmail.com with ESMTPSA id a11sm2067749ilf.79.2021.08.26.11.49.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 11:49:30 -0700 (PDT) Date: Thu, 26 Aug 2021 18:49:27 +0000 From: Oliver Upton To: Marc Zyngier Cc: Andrew Jones , kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, rananta@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, Peter Maydell Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe Message-ID: References: <87mtp5q3gx.wl-maz@kernel.org> <87fsuxq049.wl-maz@kernel.org> <20210825150713.5rpwzm4grfn7akcw@gator.home> <877dg8ppnt.wl-maz@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <877dg8ppnt.wl-maz@kernel.org> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Thu, Aug 26, 2021 at 09:37:42AM +0100, Marc Zyngier wrote: > On Wed, 25 Aug 2021 19:14:59 +0100, > Oliver Upton wrote: > > > > On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones wrote: > > [...] > > > > Thanks for including me Marc. I think you've mentioned all the examples > > > of why we don't generally expect N+1 -> N migrations to work that I > > > can think of. While some of the examples like get-reg-list could > > > eventually be eliminated if we had CPU models to tighten our machine type > > > state, I think N+1 -> N migrations will always be best effort at most. > > > > > > I agree with giving userspace control over the exposer of the hypercalls > > > though. Using pseudo-registers for that purpose rather than a pile of > > > CAPs also seems reasonable to me. > > > > > > And, while I don't think this patch is going to proceed, I thought I'd > > > point out that the opt-out approach doesn't help much with expanding > > > our migration support unless we require the VMM to be upgraded first. > > > > > > And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't > > > work as expected, since the source enforce opt-out, but the destination > > > won't. > > > > Right, there's going to need to be a fence in both kernel and VMM > > versions. Before the fence, you can't rollback with either component. > > Once on the other side of the fence, the user may freely migrate > > between kernel + VMM combinations. > > > > > Also, since the VMM doesn't key off the kernel version, for the > > > most part N+1 VMMs won't know when they're supposed to opt-out or not, > > > leaving it to the user to ensure they consider everything. opt-in > > > usually only needs the user to consider what machine type they want to > > > launch. > > > > Going the register route will implicitly require opt-out for all old > > hypercalls. We exposed them unconditionally to the guest before, and > > we must uphold that behavior. The default value for the bitmap will > > have those features set. Any hypercalls added after that register > > interface will then require explicit opt-in from userspace. > > I disagree here. This makes the ABI inconsistent, and means that no > feature can be implemented without changing userspace. If you can deal > with the existing features, you should be able to deal with the next > lot. > > > With regards to the pseudoregister interface, how would a VMM discover > > new bits? From my perspective, you need to have two bitmaps that the > > VMM can get at: the set of supported feature bits and the active > > bitmap of features for a running guest. > > My proposal is that we have a single pseudo-register exposing the list > of implemented by the kernel. Clear the bits you don't want, and write > back the result. As long as you haven't written anything, you have the > full feature set. That's pretty similar to the virtio feature > negotiation. Ah, yes I agree. Thinking about it more we will not need something similar to KVM_GET_SUPPORTED_CPUID. So then, for any register where userspace/KVM need to negotiate features, the default value will return the maximum feature set that is supported. If userspace wants to constrain features, read out the register, make sure everything you want is there, and write it back blowing away the superfluous bits. Given this should we enforce ordering on feature registers, such that a VMM can only write to the registers before a VM is started? Also, Reiji is working on making the identity registers writable for the sake of feature restriction. The suggested negotiation interface would be applicable there too, IMO. Many thanks to both you and Drew for working this out with me. -- Best, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D44FEC432BE for ; Thu, 26 Aug 2021 18:49:40 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 3F60A61038 for ; Thu, 26 Aug 2021 18:49:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3F60A61038 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 9F1444B102; Thu, 26 Aug 2021 14:49:39 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qJaHaJyViQYc; Thu, 26 Aug 2021 14:49:35 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id BB78A4B0C3; Thu, 26 Aug 2021 14:49:35 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id A24C94B0C3 for ; Thu, 26 Aug 2021 14:49:33 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 72qXBKlC0yt3 for ; Thu, 26 Aug 2021 14:49:32 -0400 (EDT) Received: from mail-io1-f48.google.com (mail-io1-f48.google.com [209.85.166.48]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 7016E4B0BA for ; Thu, 26 Aug 2021 14:49:32 -0400 (EDT) Received: by mail-io1-f48.google.com with SMTP id a21so5037793ioq.6 for ; Thu, 26 Aug 2021 11:49:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=RVUTQwfXwIWJ8KSHNLLKmgxbQdeuJteYiwLVXKrHs2E=; b=RlH7+pWHb6AHxbU3UBBJJ37bcoBxFJ+OtQc00AVbSygJknZ/usx11UO93s13G1kXX9 CcZJrRyf0UdTbKCwUaojenkX5SVnXL9NHwvsqqAY/zQ//p0C+YM4cQD7FneQjJ3tB8Aa cMrJUpb9BQzx2jUWBC1kV31Q8634J4jt38JtglF1vnnYuD+EgmbxuiAdKNe0yIPy5viO tbjYlVlNGUw9dtSd01/0cTaV2aGVOlCz1pzGl7CAV/zNmDWjdq1bz6VMh9cU5i4GYnxj 4F3MA1cOOAlb0UtfQ1G0jAbM/WTqp+4jaVkgoa7ZNZ7B4nBPcn1mEfwDWFAId+KT8PYw UaHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=RVUTQwfXwIWJ8KSHNLLKmgxbQdeuJteYiwLVXKrHs2E=; b=dF1gpkP+ZN3yCZvPHTIpaumNKsMcAj1NboE7nO0hV3l2Pgc/fVEQWgNoClnzA6IWdT fcAi1a8scDO43HeCZvnXPqCWUjfRM9T1KJe9w4EGs05w8uKSQJaXi73fEmFNCJ2lNw7N bI1+q+XpvVpeCo8e+GHqK98wl94vg8GpkWffIQWm+TqcGcNG3IhwBCpTYwXYWquJrH/H BGE1SybRSaUa45y2kwmmlEqcE/0qq8YxxKGgQzdqRW9QyD21ovpb5etcKrHJcj3TGZOM TusWrXia8fPfUOv9fqMW4VelzyL+bnuwHZSjdIGnbTB5AuZ7/deCdkBHDLZo1Zh/pPiO GiVw== X-Gm-Message-State: AOAM5309L+LC0+LdZHe62cpOMaxSgRC9PZvROzZsu4+DQj4tEMDZ6lIv VqQcM3/XyCTt4xAynguixOUfrw== X-Google-Smtp-Source: ABdhPJx5cxoJqj5jt4PGdlUhbRpwDMVgyY3ef2R2HYspFPIJb/ORtT336DtZE2lfPru1xakwQbOqEA== X-Received: by 2002:a6b:3c16:: with SMTP id k22mr4252523iob.130.1630003771571; Thu, 26 Aug 2021 11:49:31 -0700 (PDT) Received: from google.com (194.225.68.34.bc.googleusercontent.com. [34.68.225.194]) by smtp.gmail.com with ESMTPSA id a11sm2067749ilf.79.2021.08.26.11.49.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 11:49:30 -0700 (PDT) Date: Thu, 26 Aug 2021 18:49:27 +0000 From: Oliver Upton To: Marc Zyngier Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe Message-ID: References: <87mtp5q3gx.wl-maz@kernel.org> <87fsuxq049.wl-maz@kernel.org> <20210825150713.5rpwzm4grfn7akcw@gator.home> <877dg8ppnt.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <877dg8ppnt.wl-maz@kernel.org> Cc: kvm@vger.kernel.org, pshier@google.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Thu, Aug 26, 2021 at 09:37:42AM +0100, Marc Zyngier wrote: > On Wed, 25 Aug 2021 19:14:59 +0100, > Oliver Upton wrote: > > > > On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones wrote: > > [...] > > > > Thanks for including me Marc. I think you've mentioned all the examples > > > of why we don't generally expect N+1 -> N migrations to work that I > > > can think of. While some of the examples like get-reg-list could > > > eventually be eliminated if we had CPU models to tighten our machine type > > > state, I think N+1 -> N migrations will always be best effort at most. > > > > > > I agree with giving userspace control over the exposer of the hypercalls > > > though. Using pseudo-registers for that purpose rather than a pile of > > > CAPs also seems reasonable to me. > > > > > > And, while I don't think this patch is going to proceed, I thought I'd > > > point out that the opt-out approach doesn't help much with expanding > > > our migration support unless we require the VMM to be upgraded first. > > > > > > And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't > > > work as expected, since the source enforce opt-out, but the destination > > > won't. > > > > Right, there's going to need to be a fence in both kernel and VMM > > versions. Before the fence, you can't rollback with either component. > > Once on the other side of the fence, the user may freely migrate > > between kernel + VMM combinations. > > > > > Also, since the VMM doesn't key off the kernel version, for the > > > most part N+1 VMMs won't know when they're supposed to opt-out or not, > > > leaving it to the user to ensure they consider everything. opt-in > > > usually only needs the user to consider what machine type they want to > > > launch. > > > > Going the register route will implicitly require opt-out for all old > > hypercalls. We exposed them unconditionally to the guest before, and > > we must uphold that behavior. The default value for the bitmap will > > have those features set. Any hypercalls added after that register > > interface will then require explicit opt-in from userspace. > > I disagree here. This makes the ABI inconsistent, and means that no > feature can be implemented without changing userspace. If you can deal > with the existing features, you should be able to deal with the next > lot. > > > With regards to the pseudoregister interface, how would a VMM discover > > new bits? From my perspective, you need to have two bitmaps that the > > VMM can get at: the set of supported feature bits and the active > > bitmap of features for a running guest. > > My proposal is that we have a single pseudo-register exposing the list > of implemented by the kernel. Clear the bits you don't want, and write > back the result. As long as you haven't written anything, you have the > full feature set. That's pretty similar to the virtio feature > negotiation. Ah, yes I agree. Thinking about it more we will not need something similar to KVM_GET_SUPPORTED_CPUID. So then, for any register where userspace/KVM need to negotiate features, the default value will return the maximum feature set that is supported. If userspace wants to constrain features, read out the register, make sure everything you want is there, and write it back blowing away the superfluous bits. Given this should we enforce ordering on feature registers, such that a VMM can only write to the registers before a VM is started? Also, Reiji is working on making the identity registers writable for the sake of feature restriction. The suggested negotiation interface would be applicable there too, IMO. Many thanks to both you and Drew for working this out with me. -- Best, Oliver _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3015C432BE for ; Thu, 26 Aug 2021 18:51:46 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A1A486103A for ; Thu, 26 Aug 2021 18:51:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A1A486103A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Hk5Z5hywGW0z1m3s5ZCoSeYCz2jIJ4voXM+U5YWq15M=; b=SlNCE5yK7PvZUr 1Kt+lzDVgRW7uzvDwyOfm47OcoQFUPDtbWfD8+M4PRIu9lT+lRJvisrKLVEg3YagZHTFcFQzKl7Op 5utF2yXAIeK8v8OF/w/7USa2jjH/SG1AvbZ0eV5ciQfjvJz9Wq9PPp95FWZOW+QQFmQkrv1+Sk/e8 LBxo4lMUeKB2ybAyp9m9q+XNsPzF4KkFxMRJiWgBYrwGEoRPCgf5Mg31ikpmpuFS7L85xm0yh2wSW hybgp8OEp5R354KNivg52gjltvjKDvUaCfp+PhFeTTYol+iL2R564RJAOJuYdxJAlrlQG7ojdEXqC tbrqmZWu5YlSh+3pjUUQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mJKRX-00AlTQ-Ig; Thu, 26 Aug 2021 18:49:39 +0000 Received: from mail-io1-xd2c.google.com ([2607:f8b0:4864:20::d2c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mJKRU-00AlST-Ee for linux-arm-kernel@lists.infradead.org; Thu, 26 Aug 2021 18:49:37 +0000 Received: by mail-io1-xd2c.google.com with SMTP id n24so4987775ion.10 for ; Thu, 26 Aug 2021 11:49:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=RVUTQwfXwIWJ8KSHNLLKmgxbQdeuJteYiwLVXKrHs2E=; b=RlH7+pWHb6AHxbU3UBBJJ37bcoBxFJ+OtQc00AVbSygJknZ/usx11UO93s13G1kXX9 CcZJrRyf0UdTbKCwUaojenkX5SVnXL9NHwvsqqAY/zQ//p0C+YM4cQD7FneQjJ3tB8Aa cMrJUpb9BQzx2jUWBC1kV31Q8634J4jt38JtglF1vnnYuD+EgmbxuiAdKNe0yIPy5viO tbjYlVlNGUw9dtSd01/0cTaV2aGVOlCz1pzGl7CAV/zNmDWjdq1bz6VMh9cU5i4GYnxj 4F3MA1cOOAlb0UtfQ1G0jAbM/WTqp+4jaVkgoa7ZNZ7B4nBPcn1mEfwDWFAId+KT8PYw UaHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=RVUTQwfXwIWJ8KSHNLLKmgxbQdeuJteYiwLVXKrHs2E=; b=Aqu77EoDIhlGX6cCjoH5I5kN/S5E6NPtZPm1GD3FZkmy4YNVthXdkvEUinpSWLScGr niBeo0+GYOop7pUM0o78aWKPTE/MSZHpOy5CLA3A0zF4hcYnC6u9Zsgc1RyW4cr4wOIB oG1aA1BmNj7lvet2dojqKDH8mSDORVkmwNihsPnuiAkXXsyeN2XKudrN9HRTa15Rv9Oi BcQLj8wnsHWgmBjMQznIdGl0CuH7ZKCI4PCAsBgomJbr0G/Ncgci8JuR8ZLr+rx0Rg+K Svf7OFF1XVHl5D5OSv0w8IEBI4qBfpasXQOzlri5fjRt59qvKLddR1SiKwuCj1SZwsuc A8ig== X-Gm-Message-State: AOAM53026mjK1qj9gIXFsWbR8uAljgTQ7I1GXMsteJyJK/sGCaKOuek1 y7IMwlHRSkWm54mOibbtsBuI9A== X-Google-Smtp-Source: ABdhPJx5cxoJqj5jt4PGdlUhbRpwDMVgyY3ef2R2HYspFPIJb/ORtT336DtZE2lfPru1xakwQbOqEA== X-Received: by 2002:a6b:3c16:: with SMTP id k22mr4252523iob.130.1630003771571; Thu, 26 Aug 2021 11:49:31 -0700 (PDT) Received: from google.com (194.225.68.34.bc.googleusercontent.com. [34.68.225.194]) by smtp.gmail.com with ESMTPSA id a11sm2067749ilf.79.2021.08.26.11.49.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 11:49:30 -0700 (PDT) Date: Thu, 26 Aug 2021 18:49:27 +0000 From: Oliver Upton To: Marc Zyngier Cc: Andrew Jones , kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, rananta@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, Peter Maydell Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe Message-ID: References: <87mtp5q3gx.wl-maz@kernel.org> <87fsuxq049.wl-maz@kernel.org> <20210825150713.5rpwzm4grfn7akcw@gator.home> <877dg8ppnt.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <877dg8ppnt.wl-maz@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210826_114936_542125_6A9E415A X-CRM114-Status: GOOD ( 36.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Aug 26, 2021 at 09:37:42AM +0100, Marc Zyngier wrote: > On Wed, 25 Aug 2021 19:14:59 +0100, > Oliver Upton wrote: > > > > On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones wrote: > > [...] > > > > Thanks for including me Marc. I think you've mentioned all the examples > > > of why we don't generally expect N+1 -> N migrations to work that I > > > can think of. While some of the examples like get-reg-list could > > > eventually be eliminated if we had CPU models to tighten our machine type > > > state, I think N+1 -> N migrations will always be best effort at most. > > > > > > I agree with giving userspace control over the exposer of the hypercalls > > > though. Using pseudo-registers for that purpose rather than a pile of > > > CAPs also seems reasonable to me. > > > > > > And, while I don't think this patch is going to proceed, I thought I'd > > > point out that the opt-out approach doesn't help much with expanding > > > our migration support unless we require the VMM to be upgraded first. > > > > > > And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't > > > work as expected, since the source enforce opt-out, but the destination > > > won't. > > > > Right, there's going to need to be a fence in both kernel and VMM > > versions. Before the fence, you can't rollback with either component. > > Once on the other side of the fence, the user may freely migrate > > between kernel + VMM combinations. > > > > > Also, since the VMM doesn't key off the kernel version, for the > > > most part N+1 VMMs won't know when they're supposed to opt-out or not, > > > leaving it to the user to ensure they consider everything. opt-in > > > usually only needs the user to consider what machine type they want to > > > launch. > > > > Going the register route will implicitly require opt-out for all old > > hypercalls. We exposed them unconditionally to the guest before, and > > we must uphold that behavior. The default value for the bitmap will > > have those features set. Any hypercalls added after that register > > interface will then require explicit opt-in from userspace. > > I disagree here. This makes the ABI inconsistent, and means that no > feature can be implemented without changing userspace. If you can deal > with the existing features, you should be able to deal with the next > lot. > > > With regards to the pseudoregister interface, how would a VMM discover > > new bits? From my perspective, you need to have two bitmaps that the > > VMM can get at: the set of supported feature bits and the active > > bitmap of features for a running guest. > > My proposal is that we have a single pseudo-register exposing the list > of implemented by the kernel. Clear the bits you don't want, and write > back the result. As long as you haven't written anything, you have the > full feature set. That's pretty similar to the virtio feature > negotiation. Ah, yes I agree. Thinking about it more we will not need something similar to KVM_GET_SUPPORTED_CPUID. So then, for any register where userspace/KVM need to negotiate features, the default value will return the maximum feature set that is supported. If userspace wants to constrain features, read out the register, make sure everything you want is there, and write it back blowing away the superfluous bits. Given this should we enforce ordering on feature registers, such that a VMM can only write to the registers before a VM is started? Also, Reiji is working on making the identity registers writable for the sake of feature restriction. The suggested negotiation interface would be applicable there too, IMO. Many thanks to both you and Drew for working this out with me. -- Best, Oliver _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel