From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 736E8C433E2 for ; Mon, 7 Sep 2020 02:48:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A12420796 for ; Mon, 7 Sep 2020 02:48:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726357AbgIGCsj (ORCPT ); Sun, 6 Sep 2020 22:48:39 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:60770 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726213AbgIGCsh (ORCPT ); Sun, 6 Sep 2020 22:48:37 -0400 Received: from DGGEMS408-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 58D119CDACDCB4858C94; Mon, 7 Sep 2020 10:48:35 +0800 (CST) Received: from [10.174.187.22] (10.174.187.22) by DGGEMS408-HUB.china.huawei.com (10.3.19.208) with Microsoft SMTP Server id 14.3.487.0; Mon, 7 Sep 2020 10:48:29 +0800 Subject: Re: [RFC PATCH 0/5] KVM: arm64: Add pvtime LPT support To: Steven Price , Marc Zyngier References: <20200817084110.2672-1-zhukeqian1@huawei.com> <8308f52e4c906cad710575724f9e3855@kernel.org> <87h7svm0o5.wl-maz@kernel.org> <75ce4c12-f0e3-32c4-604f-9745980022e0@arm.com> CC: , , , , Catalin Marinas , Will Deacon , James Morse , Suzuki K Poulose , , From: zhukeqian Message-ID: Date: Mon, 7 Sep 2020 10:48:28 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <75ce4c12-f0e3-32c4-604f-9745980022e0@arm.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Marc and Steven, On 2020/9/2 18:09, Steven Price wrote: > Hi Marc, > > Sorry for the slow response, I've been on holiday. > > On 22/08/2020 11:31, Marc Zyngier wrote: >> Hi Steven, >> >> On Wed, 19 Aug 2020 09:54:40 +0100, >> Steven Price wrote: >>> >>> On 18/08/2020 15:41, Marc Zyngier wrote: >>>> On 2020-08-17 09:41, Keqian Zhu wrote: > [...] >>>>> >>>>> Things need concern: >>>>> 1. https://developer.arm.com/docs/den0057/a needs update. >>>> >>>> LPT was explicitly removed from the spec because it doesn't really >>>> solve the problem, specially for the firmware: EFI knows >>>> nothing about this, for example. How is it going to work? >>>> Also, nobody was ever able to explain how this would work for >>>> nested virt. >>>> >>>> ARMv8.4 and ARMv8.6 have the feature set that is required to solve >>>> this problem without adding more PV to the kernel. >>> >>> Hi Marc, >>> >>> These are good points, however we do still have the situation that >>> CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I >>> presume the use-case Keqian is looking at predates the necessary >>> support in the CPU - Keqian if you can provide more details on the >>> architecture(s) involved that would be helpful. >> >> My take on this is that it is a fictional use case. In my experience, >> migration happens across *identical* systems, and *any* difference >> visible to guests will cause things to go wrong. Errata management >> gets in the way, as usual (name *one* integration that isn't broken >> one way or another!). > > Keqian appears to have a use case - but obviously I don't know the details. I guess Keqian needs to convince you of that. Sure, there is use case, but I'm very sorry that it's inconvenient to show the detail. Maybe cross-chip migration will be supported by arm64 eventually, so I think use case is not a key problem. > >> Allowing migration across heterogeneous hosts requires a solution to >> the errata management problem, which everyone (including me) has >> decided to ignore so far (and I claim that not having a constant timer >> frequency exposed to guests is an architecture bug). > > I agree - errata management needs to be solved before LPT. Between restricted subsets of hosts this doesn't seem impossible, but I guess we should stall LPT until a credible solution is proposed. I'm certainly not proposing one at the moment. > >>> Nested virt is indeed more of an issue - we did have some ideas around >>> using SDEI that never made it to the spec. >> >> SDEI? Sigh... Why would SDEI be useful for NV and not for !NV? > > SDEI provides a way of injecting a synchronous exception on migration - although that certainly isn't the only possible mechanism. For NV we have the problem that a guest-guest may be running at the point of migration. However it's not practical for the host hypervisor to provide the necessary table directly to the guest-guest which means the guest-hypervisor must update the tables before the guest-guest is allowed to run on the new host. The only plausible route I could see for this is injecting a synchronous exception into the guest (per VCPU) to ensure any guest-guests running are exited at migration time. > > !NV is easier because we don't have to worry about multiple levels of para-virtualisation. > >>> However I would argue that the most pragmatic approach would be to >>> not support the combination of nested virt and LPT. Hopefully that >>> can wait until the counter scaling support is available and not >>> require PV. >> >> And have yet another set of band aids that paper over the fact that we >> can't get a consistent story on virtualization? No, thank you. >> >> NV is (IMHO) much more important than LPT as it has a chance of >> getting used. LPT is just another tick box, and the fact that ARM is >> ready to ignore sideline a decent portion of the architecture is a >> clear sign that it hasn't been thought out. > > Different people have different priorities. NV is definitely important for many people. LPT may also be important if you've already got a bunch of VMs running on machines and you want to be able to (gradually) replace them with newer hosts which happen to have a different clock frequency. Those VMs running now clearly aren't using NV. > > However, I have to admit it's not me that has the use-case, so I'll leave it for others who might actually know the specifics to explain the details. > >>> We are discussing (re-)releasing the spec with the LPT parts added. If >>> you have fundamental objections then please me know. >> >> I do, see above. I'm stating that the use case doesn't really exist >> given the state of the available HW and the fragmentation of the >> architecture, and that ignoring the most important innovation in the >> virtualization architecture since ARMv7 is at best short-sighted. >> >> Time scaling is just an instance of the errata management problem, and >> that is the issue that needs solving. Papering over part of the >> problem is not helping. > > I fully agree - errata management is definitely the first step that needs solving. This is why I abandoned LPT originally because I don't have a generic solution and the testing I did involved really ugly hacks just to make the migration possible. > > For now I propose we (again) park LPT until some progress has been made on errata management. > > Thanks, > > Steve > . As we have discussed, to support the vtimer part of cross-chip migration, we still face many problems. Firstly, we have no complete solution to realize the basic functionality. For PV solution, LPT just handles Linux kernel, other SW agents are not involved. For non-PV solution, ARMv8.4 ext and ARMv8.6 ext is not enough. Besides the basic functionality, we should concern errata management and NV (I think this is not urgent). Giving above, I agree with Steven that re-park LPT. Thanks, Keqian From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEBFFC10DAA for ; Mon, 7 Sep 2020 02:48:45 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id B908020796 for ; Mon, 7 Sep 2020 02:48:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B908020796 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 13C164B3D2; Sun, 6 Sep 2020 22:48:44 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jroTItihNSDj; Sun, 6 Sep 2020 22:48:42 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 9274D4B49E; Sun, 6 Sep 2020 22:48:42 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 22A824B449 for ; Sun, 6 Sep 2020 22:48:41 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f97lDmYMpPLb for ; Sun, 6 Sep 2020 22:48:39 -0400 (EDT) Received: from huawei.com (szxga06-in.huawei.com [45.249.212.32]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id B9CFB4B3D2 for ; Sun, 6 Sep 2020 22:48:38 -0400 (EDT) Received: from DGGEMS408-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 58D119CDACDCB4858C94; Mon, 7 Sep 2020 10:48:35 +0800 (CST) Received: from [10.174.187.22] (10.174.187.22) by DGGEMS408-HUB.china.huawei.com (10.3.19.208) with Microsoft SMTP Server id 14.3.487.0; Mon, 7 Sep 2020 10:48:29 +0800 Subject: Re: [RFC PATCH 0/5] KVM: arm64: Add pvtime LPT support To: Steven Price , Marc Zyngier References: <20200817084110.2672-1-zhukeqian1@huawei.com> <8308f52e4c906cad710575724f9e3855@kernel.org> <87h7svm0o5.wl-maz@kernel.org> <75ce4c12-f0e3-32c4-604f-9745980022e0@arm.com> From: zhukeqian Message-ID: Date: Mon, 7 Sep 2020 10:48:28 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <75ce4c12-f0e3-32c4-604f-9745980022e0@arm.com> X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected Cc: kvm@vger.kernel.org, Catalin Marinas , linux-kernel@vger.kernel.org, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Hi Marc and Steven, On 2020/9/2 18:09, Steven Price wrote: > Hi Marc, > > Sorry for the slow response, I've been on holiday. > > On 22/08/2020 11:31, Marc Zyngier wrote: >> Hi Steven, >> >> On Wed, 19 Aug 2020 09:54:40 +0100, >> Steven Price wrote: >>> >>> On 18/08/2020 15:41, Marc Zyngier wrote: >>>> On 2020-08-17 09:41, Keqian Zhu wrote: > [...] >>>>> >>>>> Things need concern: >>>>> 1. https://developer.arm.com/docs/den0057/a needs update. >>>> >>>> LPT was explicitly removed from the spec because it doesn't really >>>> solve the problem, specially for the firmware: EFI knows >>>> nothing about this, for example. How is it going to work? >>>> Also, nobody was ever able to explain how this would work for >>>> nested virt. >>>> >>>> ARMv8.4 and ARMv8.6 have the feature set that is required to solve >>>> this problem without adding more PV to the kernel. >>> >>> Hi Marc, >>> >>> These are good points, however we do still have the situation that >>> CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I >>> presume the use-case Keqian is looking at predates the necessary >>> support in the CPU - Keqian if you can provide more details on the >>> architecture(s) involved that would be helpful. >> >> My take on this is that it is a fictional use case. In my experience, >> migration happens across *identical* systems, and *any* difference >> visible to guests will cause things to go wrong. Errata management >> gets in the way, as usual (name *one* integration that isn't broken >> one way or another!). > > Keqian appears to have a use case - but obviously I don't know the details. I guess Keqian needs to convince you of that. Sure, there is use case, but I'm very sorry that it's inconvenient to show the detail. Maybe cross-chip migration will be supported by arm64 eventually, so I think use case is not a key problem. > >> Allowing migration across heterogeneous hosts requires a solution to >> the errata management problem, which everyone (including me) has >> decided to ignore so far (and I claim that not having a constant timer >> frequency exposed to guests is an architecture bug). > > I agree - errata management needs to be solved before LPT. Between restricted subsets of hosts this doesn't seem impossible, but I guess we should stall LPT until a credible solution is proposed. I'm certainly not proposing one at the moment. > >>> Nested virt is indeed more of an issue - we did have some ideas around >>> using SDEI that never made it to the spec. >> >> SDEI? Sigh... Why would SDEI be useful for NV and not for !NV? > > SDEI provides a way of injecting a synchronous exception on migration - although that certainly isn't the only possible mechanism. For NV we have the problem that a guest-guest may be running at the point of migration. However it's not practical for the host hypervisor to provide the necessary table directly to the guest-guest which means the guest-hypervisor must update the tables before the guest-guest is allowed to run on the new host. The only plausible route I could see for this is injecting a synchronous exception into the guest (per VCPU) to ensure any guest-guests running are exited at migration time. > > !NV is easier because we don't have to worry about multiple levels of para-virtualisation. > >>> However I would argue that the most pragmatic approach would be to >>> not support the combination of nested virt and LPT. Hopefully that >>> can wait until the counter scaling support is available and not >>> require PV. >> >> And have yet another set of band aids that paper over the fact that we >> can't get a consistent story on virtualization? No, thank you. >> >> NV is (IMHO) much more important than LPT as it has a chance of >> getting used. LPT is just another tick box, and the fact that ARM is >> ready to ignore sideline a decent portion of the architecture is a >> clear sign that it hasn't been thought out. > > Different people have different priorities. NV is definitely important for many people. LPT may also be important if you've already got a bunch of VMs running on machines and you want to be able to (gradually) replace them with newer hosts which happen to have a different clock frequency. Those VMs running now clearly aren't using NV. > > However, I have to admit it's not me that has the use-case, so I'll leave it for others who might actually know the specifics to explain the details. > >>> We are discussing (re-)releasing the spec with the LPT parts added. If >>> you have fundamental objections then please me know. >> >> I do, see above. I'm stating that the use case doesn't really exist >> given the state of the available HW and the fragmentation of the >> architecture, and that ignoring the most important innovation in the >> virtualization architecture since ARMv7 is at best short-sighted. >> >> Time scaling is just an instance of the errata management problem, and >> that is the issue that needs solving. Papering over part of the >> problem is not helping. > > I fully agree - errata management is definitely the first step that needs solving. This is why I abandoned LPT originally because I don't have a generic solution and the testing I did involved really ugly hacks just to make the migration possible. > > For now I propose we (again) park LPT until some progress has been made on errata management. > > Thanks, > > Steve > . As we have discussed, to support the vtimer part of cross-chip migration, we still face many problems. Firstly, we have no complete solution to realize the basic functionality. For PV solution, LPT just handles Linux kernel, other SW agents are not involved. For non-PV solution, ARMv8.4 ext and ARMv8.6 ext is not enough. Besides the basic functionality, we should concern errata management and NV (I think this is not urgent). Giving above, I agree with Steven that re-park LPT. Thanks, Keqian _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6236C43461 for ; Mon, 7 Sep 2020 02:50:03 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4C41520796 for ; Mon, 7 Sep 2020 02:50:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="s6IhIrL8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C41520796 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=E9YRWK888P3GWgfEJYQH8shnLGoE8eVFFcSTcXbwEO0=; b=s6IhIrL87mIEWlvd8uNTTwXXw jUEI6UrOIwDfcwKW6QgNr4wf28OMhA+91w2BEUX7fCmAiqshKBzuKxe+Ke4F0HlN3x+ndQX9c21NI jGTMmLD3nOYHJonivYfZ5Dino2qkkOnwZW7aM6gtl630J3sYIgeqyWYEshmnVeGTzZwVhLmzxmEBD HSxNp+pYUled6YpkgH/+2zvp5R2LliYmh8yf/KpxAdmURYCGVFfPpJGdbhkdKWdtxgyLPrIcN9rM6 KaCs/5+rOg7gQhWb2ypPVBelQjQdIgeo5/ESLL4fWBN+34R07y8FsXS0tJNbE6z+80ZEgExuYHtT8 CYG2kzwDQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kF7D5-0002oi-Jn; Mon, 07 Sep 2020 02:48:47 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kF7D3-0002nK-07 for linux-arm-kernel@lists.infradead.org; Mon, 07 Sep 2020 02:48:46 +0000 Received: from DGGEMS408-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 58D119CDACDCB4858C94; Mon, 7 Sep 2020 10:48:35 +0800 (CST) Received: from [10.174.187.22] (10.174.187.22) by DGGEMS408-HUB.china.huawei.com (10.3.19.208) with Microsoft SMTP Server id 14.3.487.0; Mon, 7 Sep 2020 10:48:29 +0800 Subject: Re: [RFC PATCH 0/5] KVM: arm64: Add pvtime LPT support To: Steven Price , Marc Zyngier References: <20200817084110.2672-1-zhukeqian1@huawei.com> <8308f52e4c906cad710575724f9e3855@kernel.org> <87h7svm0o5.wl-maz@kernel.org> <75ce4c12-f0e3-32c4-604f-9745980022e0@arm.com> From: zhukeqian Message-ID: Date: Mon, 7 Sep 2020 10:48:28 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <75ce4c12-f0e3-32c4-604f-9745980022e0@arm.com> X-Originating-IP: [10.174.187.22] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200906_224845_278968_E0656391 X-CRM114-Status: GOOD ( 32.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, kvm@vger.kernel.org, Suzuki K Poulose , Catalin Marinas , linux-kernel@vger.kernel.org, James Morse , wanghaibin.wang@huawei.com, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Marc and Steven, On 2020/9/2 18:09, Steven Price wrote: > Hi Marc, > > Sorry for the slow response, I've been on holiday. > > On 22/08/2020 11:31, Marc Zyngier wrote: >> Hi Steven, >> >> On Wed, 19 Aug 2020 09:54:40 +0100, >> Steven Price wrote: >>> >>> On 18/08/2020 15:41, Marc Zyngier wrote: >>>> On 2020-08-17 09:41, Keqian Zhu wrote: > [...] >>>>> >>>>> Things need concern: >>>>> 1. https://developer.arm.com/docs/den0057/a needs update. >>>> >>>> LPT was explicitly removed from the spec because it doesn't really >>>> solve the problem, specially for the firmware: EFI knows >>>> nothing about this, for example. How is it going to work? >>>> Also, nobody was ever able to explain how this would work for >>>> nested virt. >>>> >>>> ARMv8.4 and ARMv8.6 have the feature set that is required to solve >>>> this problem without adding more PV to the kernel. >>> >>> Hi Marc, >>> >>> These are good points, however we do still have the situation that >>> CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I >>> presume the use-case Keqian is looking at predates the necessary >>> support in the CPU - Keqian if you can provide more details on the >>> architecture(s) involved that would be helpful. >> >> My take on this is that it is a fictional use case. In my experience, >> migration happens across *identical* systems, and *any* difference >> visible to guests will cause things to go wrong. Errata management >> gets in the way, as usual (name *one* integration that isn't broken >> one way or another!). > > Keqian appears to have a use case - but obviously I don't know the details. I guess Keqian needs to convince you of that. Sure, there is use case, but I'm very sorry that it's inconvenient to show the detail. Maybe cross-chip migration will be supported by arm64 eventually, so I think use case is not a key problem. > >> Allowing migration across heterogeneous hosts requires a solution to >> the errata management problem, which everyone (including me) has >> decided to ignore so far (and I claim that not having a constant timer >> frequency exposed to guests is an architecture bug). > > I agree - errata management needs to be solved before LPT. Between restricted subsets of hosts this doesn't seem impossible, but I guess we should stall LPT until a credible solution is proposed. I'm certainly not proposing one at the moment. > >>> Nested virt is indeed more of an issue - we did have some ideas around >>> using SDEI that never made it to the spec. >> >> SDEI? Sigh... Why would SDEI be useful for NV and not for !NV? > > SDEI provides a way of injecting a synchronous exception on migration - although that certainly isn't the only possible mechanism. For NV we have the problem that a guest-guest may be running at the point of migration. However it's not practical for the host hypervisor to provide the necessary table directly to the guest-guest which means the guest-hypervisor must update the tables before the guest-guest is allowed to run on the new host. The only plausible route I could see for this is injecting a synchronous exception into the guest (per VCPU) to ensure any guest-guests running are exited at migration time. > > !NV is easier because we don't have to worry about multiple levels of para-virtualisation. > >>> However I would argue that the most pragmatic approach would be to >>> not support the combination of nested virt and LPT. Hopefully that >>> can wait until the counter scaling support is available and not >>> require PV. >> >> And have yet another set of band aids that paper over the fact that we >> can't get a consistent story on virtualization? No, thank you. >> >> NV is (IMHO) much more important than LPT as it has a chance of >> getting used. LPT is just another tick box, and the fact that ARM is >> ready to ignore sideline a decent portion of the architecture is a >> clear sign that it hasn't been thought out. > > Different people have different priorities. NV is definitely important for many people. LPT may also be important if you've already got a bunch of VMs running on machines and you want to be able to (gradually) replace them with newer hosts which happen to have a different clock frequency. Those VMs running now clearly aren't using NV. > > However, I have to admit it's not me that has the use-case, so I'll leave it for others who might actually know the specifics to explain the details. > >>> We are discussing (re-)releasing the spec with the LPT parts added. If >>> you have fundamental objections then please me know. >> >> I do, see above. I'm stating that the use case doesn't really exist >> given the state of the available HW and the fragmentation of the >> architecture, and that ignoring the most important innovation in the >> virtualization architecture since ARMv7 is at best short-sighted. >> >> Time scaling is just an instance of the errata management problem, and >> that is the issue that needs solving. Papering over part of the >> problem is not helping. > > I fully agree - errata management is definitely the first step that needs solving. This is why I abandoned LPT originally because I don't have a generic solution and the testing I did involved really ugly hacks just to make the migration possible. > > For now I propose we (again) park LPT until some progress has been made on errata management. > > Thanks, > > Steve > . As we have discussed, to support the vtimer part of cross-chip migration, we still face many problems. Firstly, we have no complete solution to realize the basic functionality. For PV solution, LPT just handles Linux kernel, other SW agents are not involved. For non-PV solution, ARMv8.4 ext and ARMv8.6 ext is not enough. Besides the basic functionality, we should concern errata management and NV (I think this is not urgent). Giving above, I agree with Steven that re-park LPT. Thanks, Keqian _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel