From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C0C3C43461 for ; Tue, 18 May 2021 17:00:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F30861184 for ; Tue, 18 May 2021 17:00:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346458AbhERRBk (ORCPT ); Tue, 18 May 2021 13:01:40 -0400 Received: from foss.arm.com ([217.140.110.172]:57134 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232496AbhERRBj (ORCPT ); Tue, 18 May 2021 13:01:39 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 09AB66D; Tue, 18 May 2021 10:00:21 -0700 (PDT) Received: from C02TD0UTHF1T.local (unknown [10.57.6.226]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C68D03F73B; Tue, 18 May 2021 10:00:18 -0700 (PDT) Date: Tue, 18 May 2021 18:00:16 +0100 From: Mark Rutland To: Michael Kelley Cc: "will@kernel.org" , "catalin.marinas@arm.com" , "lorenzo.pieralisi@arm.com" , "sudeep.holla@arm.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "linux-hyperv@vger.kernel.org" , "linux-efi@vger.kernel.org" , "arnd@arndb.de" , "wei.liu@kernel.org" , "ardb@kernel.org" , "daniel.lezcano@linaro.org" , KY Srinivasan Subject: Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support Message-ID: <20210518170016.GP82842@C02TD0UTHF1T.local> References: <1620841067-46606-1-git-send-email-mikelley@microsoft.com> <1620841067-46606-4-git-send-email-mikelley@microsoft.com> <20210514123711.GB30645@C02TD0UTHF1T.local> <20210517130815.GC62656@C02TD0UTHF1T.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote: > From: Mark Rutland Sent: Monday, May 17, 2021 6:08 AM > > > > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote: > > > From: Mark Rutland Sent: Friday, May 14, 2021 5:37 AM > > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote: > > > > > Add architecture specific definitions and functions needed > > > > > by the architecture independent Hyper-V clocksource driver. > > > > > Update the Hyper-V clocksource driver to be initialized > > > > > on ARM64. > > > > > > > > Previously we've said that for a clocksource we must use the architected > > > > counter, since that's necessary for things like the VDSO to work > > > > correctly and efficiently. > > > > > > > > Given that, I'm a bit confused that we're registering a per-cpu > > > > clocksource that is in part based on the architected counter. Likewise, > > > > I don't entirely follow why it's necessary to PV the clock_event_device. > > > > > > > > Are the architected counter and timer reliable without this PV > > > > infrastructure? Why do we need to PV either of those? > > > > > > > > Thanks, > > > > Mark. > > > > > > For the clocksource, we have a requirement to live migrate VMs > > > between Hyper-V hosts running on hardware that may have different > > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz > > > requirement). The Hyper-V virtualization does scaling to handle the > > > frequency difference. And yes, there's a tradeoff with vDSO not > > > working, though we have an out-of-tree vDSO implementation that > > > we can use when necessary. > > > > Just to be clear, the vDSO is *one example* of something that won't > > function correctly. More generally, because this undermines core > > architectural guarantees, it requires more invasive changes (e.g. we'd > > have to weaken the sanity checks, and not use the counter in things like > > kexec paths), impacts any architectural features tied to the generic > > timer/counter (e.g. the event stream, SPE and tracing, future features), > > and means that other SW (e.g. bootloaders and other EFI applications) > > are unlikley to function correctly in this environment. > > > > I am very much not keen on trying to PV this. > > > > What does the guest see when it reads CNTFRQ_EL0? Does this match the > > real HW value (and can this change over time)? Or is this entirely > > synthetic? > > > > What do the ACPI tables look like in the guest? Is there a GTDT table at > > all? > > > > How does the counter event stream behave? > > > > Are there other architectural features which Hyper-V does not implement > > for a guest? > > > > Is there anything else that may change across a migration? e.g. MIDR? > > MPIDR? Any of the ID registers? > > The ARMv8 architectural system counter and associated registers are visible > and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is > instantiated by the arm_arch_timer.c driver based on the GTDT in the guest, > and a Linux guest on Hyper-V runs fine with this clocksource. Low level code > like bootloaders and EFI applications work normally. That's good to hear! One potential issue worth noting is that as those pieces of software are unlikely to handle counter frequency changes reliably, and so may not behave correctly if live-migrated. > The Hyper-V virtualization provides another Linux clocksource that is an > overlay on the arch counter and that provides time consistency across a live > migration. Live migration of ARM64 VMs on Hyper-V is not functional today, > but the Hyper-V team believes they can make it functional. I have not > explored with them the live migration implications of things beyond time > consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc. > > Would a summary of your point be that live migration across hardware > with different arch counter frequencies is likely to not be possible with > 100% fidelity because of these other dependencies on the arch counter > frequency? (hence the fixed 1 GHz frequency in ARM v8.6) Yes. In addition, there are a larger set of things necessarily exposed to VMs that mean that live migration isn't all that practical except betweenm identical machines (where the counter frequency should be identical), and the timer frequency might just be the canary in the coalmine. For example, the cache properties enumerated in CTR_EL0 cannot necessarily be emulated on another machine. > > > For clockevents, the only timer interrupt that Hyper-V provides > > > in a guest VM is its virtualized "STIMER" interrupt. There's no > > > virtualization of the ARM arch timer in the guest. > > > > I think that is rather unfortunate, given it's a core architectural > > feature. Is it just the interrupt that's missing? i.e. does all the > > PE-local functionality behave as the architecture requires? > > Right off the bat, I don't know about timer-related PE-local > functionality as it's not exercised in a Linux VM on Hyper-V that is > using STIMER-based clockevents. I'll explore with the Hyper-V > team. My impression is that enabling the ARM arch timer in a > guest VM is more work for Hyper-V than just wiring up an > interrupt. Thanks for chasing this up! Mark. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBA9FC433ED for ; Tue, 18 May 2021 17:02:02 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 77A97610A1 for ; Tue, 18 May 2021 17:02:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 77A97610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vNAW4/cJtSJrfaqa94ZLWNjrCLsGpgORzcxdlC3j8j0=; b=TgZX90YFdPAhBvyB+r/qF0+qN kYA/hETvy8AbGkKk7sv2uxcJ5vyUHaClurZM5swRgu/nEVAgNBfnT+EG/CDpv35zDLeiWkAwarexT 2DFAY/TcIEfuDflVcoRb/HIvrZk00HZ7qFGWApH+urAkoQXTa6SHipkJ5ZbieC/0vC8veb7m8bdmD VADc1+VJhn8T0+nUU7c4lrtV78zNNtXXZZ+ksOSFituDi4t4qdkOUVS1++uJv8TCaDNpsLcMqU7oF Nmw3nkEJ2qzpNFuTUJmwjfJIGQDdKVJt5LvkNgdV9+EwLFTP2aLrdB/IxcgKhwDaHQxb0XL53//MO cIeH/8++Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lj355-001QIF-Ab; Tue, 18 May 2021 17:00:31 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lj351-001QHK-1H for linux-arm-kernel@desiato.infradead.org; Tue, 18 May 2021 17:00:27 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=In-Reply-To:Content-Type:MIME-Version :References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=gIoGC+/Y16/PKvVkz6pE81WFtgUoOOzkHsbnZnKlecw=; b=CY2chtcoGjm7BR3Ok0We/S85LI M2hgcIPthMq7KyUGnVcepTPxLG+zXLvhAUI3ncmVe+9YYPFBheNgTaqZlKC21z9eo6nVewyitg2/f 8GL5H4wzP35j1Fet5l93FRz9p3TS+Fci7PPi8RAgi8pet6WOtl1lopHEPTcfeLz7bi9OxpgkEk4ia 6ZZ+fUK1bPtuxkt+4jbZ0kHMMyxjsqPbpt3K/6FgWCvssdFbU7XD43JHTU0gTB0A8igtFlGccQQUZ Icb0RKjJdMLAbVK22Mmy6aPCnJZqK7TffCxekm458Gag0U3FbTGGw5het0lJhCc/B8ehmQjWc+4KH 2uFS7Rhw==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lj34x-00EpCl-UK for linux-arm-kernel@lists.infradead.org; Tue, 18 May 2021 17:00:25 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 09AB66D; Tue, 18 May 2021 10:00:21 -0700 (PDT) Received: from C02TD0UTHF1T.local (unknown [10.57.6.226]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C68D03F73B; Tue, 18 May 2021 10:00:18 -0700 (PDT) Date: Tue, 18 May 2021 18:00:16 +0100 From: Mark Rutland To: Michael Kelley Cc: "will@kernel.org" , "catalin.marinas@arm.com" , "lorenzo.pieralisi@arm.com" , "sudeep.holla@arm.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "linux-hyperv@vger.kernel.org" , "linux-efi@vger.kernel.org" , "arnd@arndb.de" , "wei.liu@kernel.org" , "ardb@kernel.org" , "daniel.lezcano@linaro.org" , KY Srinivasan Subject: Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support Message-ID: <20210518170016.GP82842@C02TD0UTHF1T.local> References: <1620841067-46606-1-git-send-email-mikelley@microsoft.com> <1620841067-46606-4-git-send-email-mikelley@microsoft.com> <20210514123711.GB30645@C02TD0UTHF1T.local> <20210517130815.GC62656@C02TD0UTHF1T.local> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210518_100024_105769_67D98D4D X-CRM114-Status: GOOD ( 47.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote: > From: Mark Rutland Sent: Monday, May 17, 2021 6:08 AM > > > > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote: > > > From: Mark Rutland Sent: Friday, May 14, 2021 5:37 AM > > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote: > > > > > Add architecture specific definitions and functions needed > > > > > by the architecture independent Hyper-V clocksource driver. > > > > > Update the Hyper-V clocksource driver to be initialized > > > > > on ARM64. > > > > > > > > Previously we've said that for a clocksource we must use the architected > > > > counter, since that's necessary for things like the VDSO to work > > > > correctly and efficiently. > > > > > > > > Given that, I'm a bit confused that we're registering a per-cpu > > > > clocksource that is in part based on the architected counter. Likewise, > > > > I don't entirely follow why it's necessary to PV the clock_event_device. > > > > > > > > Are the architected counter and timer reliable without this PV > > > > infrastructure? Why do we need to PV either of those? > > > > > > > > Thanks, > > > > Mark. > > > > > > For the clocksource, we have a requirement to live migrate VMs > > > between Hyper-V hosts running on hardware that may have different > > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz > > > requirement). The Hyper-V virtualization does scaling to handle the > > > frequency difference. And yes, there's a tradeoff with vDSO not > > > working, though we have an out-of-tree vDSO implementation that > > > we can use when necessary. > > > > Just to be clear, the vDSO is *one example* of something that won't > > function correctly. More generally, because this undermines core > > architectural guarantees, it requires more invasive changes (e.g. we'd > > have to weaken the sanity checks, and not use the counter in things like > > kexec paths), impacts any architectural features tied to the generic > > timer/counter (e.g. the event stream, SPE and tracing, future features), > > and means that other SW (e.g. bootloaders and other EFI applications) > > are unlikley to function correctly in this environment. > > > > I am very much not keen on trying to PV this. > > > > What does the guest see when it reads CNTFRQ_EL0? Does this match the > > real HW value (and can this change over time)? Or is this entirely > > synthetic? > > > > What do the ACPI tables look like in the guest? Is there a GTDT table at > > all? > > > > How does the counter event stream behave? > > > > Are there other architectural features which Hyper-V does not implement > > for a guest? > > > > Is there anything else that may change across a migration? e.g. MIDR? > > MPIDR? Any of the ID registers? > > The ARMv8 architectural system counter and associated registers are visible > and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is > instantiated by the arm_arch_timer.c driver based on the GTDT in the guest, > and a Linux guest on Hyper-V runs fine with this clocksource. Low level code > like bootloaders and EFI applications work normally. That's good to hear! One potential issue worth noting is that as those pieces of software are unlikely to handle counter frequency changes reliably, and so may not behave correctly if live-migrated. > The Hyper-V virtualization provides another Linux clocksource that is an > overlay on the arch counter and that provides time consistency across a live > migration. Live migration of ARM64 VMs on Hyper-V is not functional today, > but the Hyper-V team believes they can make it functional. I have not > explored with them the live migration implications of things beyond time > consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc. > > Would a summary of your point be that live migration across hardware > with different arch counter frequencies is likely to not be possible with > 100% fidelity because of these other dependencies on the arch counter > frequency? (hence the fixed 1 GHz frequency in ARM v8.6) Yes. In addition, there are a larger set of things necessarily exposed to VMs that mean that live migration isn't all that practical except betweenm identical machines (where the counter frequency should be identical), and the timer frequency might just be the canary in the coalmine. For example, the cache properties enumerated in CTR_EL0 cannot necessarily be emulated on another machine. > > > For clockevents, the only timer interrupt that Hyper-V provides > > > in a guest VM is its virtualized "STIMER" interrupt. There's no > > > virtualization of the ARM arch timer in the guest. > > > > I think that is rather unfortunate, given it's a core architectural > > feature. Is it just the interrupt that's missing? i.e. does all the > > PE-local functionality behave as the architecture requires? > > Right off the bat, I don't know about timer-related PE-local > functionality as it's not exercised in a Linux VM on Hyper-V that is > using STIMER-based clockevents. I'll explore with the Hyper-V > team. My impression is that enabling the ARM arch timer in a > guest VM is more work for Hyper-V than just wiring up an > interrupt. Thanks for chasing this up! Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel