From: Michael Kelley <mikelley@microsoft.com>
To: Mark Rutland <Mark.Rutland@arm.com>
Cc: "will@kernel.org" <will@kernel.org>,
"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
"lorenzo.pieralisi@arm.com" <lorenzo.pieralisi@arm.com>,
"sudeep.holla@arm.com" <sudeep.holla@arm.com>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
"linux-efi@vger.kernel.org" <linux-efi@vger.kernel.org>,
"arnd@arndb.de" <arnd@arndb.de>,
"wei.liu@kernel.org" <wei.liu@kernel.org>,
"ardb@kernel.org" <ardb@kernel.org>,
"daniel.lezcano@linaro.org" <daniel.lezcano@linaro.org>,
KY Srinivasan <kys@microsoft.com>
Subject: RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
Date: Tue, 8 Jun 2021 15:36:06 +0000 [thread overview]
Message-ID: <MWHPR21MB1593800A20B55626ACE6A844D7379@MWHPR21MB1593.namprd21.prod.outlook.com> (raw)
In-Reply-To: <20210518170016.GP82842@C02TD0UTHF1T.local>
From: Mark Rutland <mark.rutland@arm.com> Sent: Tuesday, May 18, 2021 10:00 AM
>
> On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM
> > >
> > > On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > > > From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM
> > > > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > > > Add architecture specific definitions and functions needed
> > > > > > by the architecture independent Hyper-V clocksource driver.
> > > > > > Update the Hyper-V clocksource driver to be initialized
> > > > > > on ARM64.
> > > > >
> > > > > Previously we've said that for a clocksource we must use the architected
> > > > > counter, since that's necessary for things like the VDSO to work
> > > > > correctly and efficiently.
> > > > >
> > > > > Given that, I'm a bit confused that we're registering a per-cpu
> > > > > clocksource that is in part based on the architected counter. Likewise,
> > > > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > > > >
> > > > > Are the architected counter and timer reliable without this PV
> > > > > infrastructure? Why do we need to PV either of those?
> > > > >
> > > > > Thanks,
> > > > > Mark.
> > > >
> > > > For the clocksource, we have a requirement to live migrate VMs
> > > > between Hyper-V hosts running on hardware that may have different
> > > > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > > > requirement). The Hyper-V virtualization does scaling to handle the
> > > > frequency difference. And yes, there's a tradeoff with vDSO not
> > > > working, though we have an out-of-tree vDSO implementation that
> > > > we can use when necessary.
> > >
> > > Just to be clear, the vDSO is *one example* of something that won't
> > > function correctly. More generally, because this undermines core
> > > architectural guarantees, it requires more invasive changes (e.g. we'd
> > > have to weaken the sanity checks, and not use the counter in things like
> > > kexec paths), impacts any architectural features tied to the generic
> > > timer/counter (e.g. the event stream, SPE and tracing, future features),
> > > and means that other SW (e.g. bootloaders and other EFI applications)
> > > are unlikley to function correctly in this environment.
> > >
> > > I am very much not keen on trying to PV this.
> > >
> > > What does the guest see when it reads CNTFRQ_EL0? Does this match the
> > > real HW value (and can this change over time)? Or is this entirely
> > > synthetic?
> > >
> > > What do the ACPI tables look like in the guest? Is there a GTDT table at
> > > all?
> > >
> > > How does the counter event stream behave?
> > >
> > > Are there other architectural features which Hyper-V does not implement
> > > for a guest?
> > >
> > > Is there anything else that may change across a migration? e.g. MIDR?
> > > MPIDR? Any of the ID registers?
> >
> > The ARMv8 architectural system counter and associated registers are visible
> > and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
> > instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
> > and a Linux guest on Hyper-V runs fine with this clocksource. Low level code
> > like bootloaders and EFI applications work normally.
>
> That's good to hear!
>
> One potential issue worth noting is that as those pieces of software are
> unlikely to handle counter frequency changes reliably, and so may not
> behave correctly if live-migrated.
>
> > The Hyper-V virtualization provides another Linux clocksource that is an
> > overlay on the arch counter and that provides time consistency across a live
> > migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
> > but the Hyper-V team believes they can make it functional. I have not
> > explored with them the live migration implications of things beyond time
> > consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
> >
> > Would a summary of your point be that live migration across hardware
> > with different arch counter frequencies is likely to not be possible with
> > 100% fidelity because of these other dependencies on the arch counter
> > frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
>
> Yes.
>
> In addition, there are a larger set of things necessarily exposed to VMs
> that mean that live migration isn't all that practical except betweenm
> identical machines (where the counter frequency should be identical),
> and the timer frequency might just be the canary in the coalmine. For
> example, the cache properties enumerated in CTR_EL0 cannot necessarily
> be emulated on another machine.
>
> > > > For clockevents, the only timer interrupt that Hyper-V provides
> > > > in a guest VM is its virtualized "STIMER" interrupt. There's no
> > > > virtualization of the ARM arch timer in the guest.
> > >
> > > I think that is rather unfortunate, given it's a core architectural
> > > feature. Is it just the interrupt that's missing? i.e. does all the
> > > PE-local functionality behave as the architecture requires?
> >
> > Right off the bat, I don't know about timer-related PE-local
> > functionality as it's not exercised in a Linux VM on Hyper-V that is
> > using STIMER-based clockevents. I'll explore with the Hyper-V
> > team. My impression is that enabling the ARM arch timer in a
> > guest VM is more work for Hyper-V than just wiring up an
> > interrupt.
>
> Thanks for chasing this up!
>
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and
I'll resubmit the code so that arm_arch_timer.c provides the
standard arch_sys_counter clocksource. As noted previously, this just
works for a Hyper-V guest. The live migration discussion may come
back later after a deeper investigation by Hyper-V.
For clockevents, there's not a near term fix. It's more than just plumbing
an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
From their perspective there's also benefit in having a timer abstraction
that's independent of the architecture, and in the Linux guest, the STIMER
code is common across x86/x64 and ARM64. It follows the standard Linux
clockevents model, as it should. The code is already in use in out-of-tree
builds in the Linux VMs included in Windows 10 on ARM64 as part of the
so-called "Windows Subsystem for Linux".
So I'm hoping we can get this core support for ARM64 guests on Hyper-V
into upstream using the existing STIMER support. At some point, Hyper-V
will do the virtualization of the ARM64 arch timer, but we don't want to
have to stay out-of-tree until after that happens.
Thoughts?
Michael
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-06-08 15:38 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-12 17:37 [PATCH v10 0/7] Enable Linux guests on Hyper-V on ARM64 Michael Kelley
2021-05-12 17:37 ` [PATCH v10 1/7] asm-generic: hyperv: Fix incorrect architecture dependencies Michael Kelley
2021-05-12 17:37 ` [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities Michael Kelley
2021-05-14 12:52 ` Mark Rutland
2021-05-14 15:14 ` Michael Kelley
2021-05-17 11:44 ` Mark Rutland
2021-05-17 16:41 ` Michael Kelley
2021-05-12 17:37 ` [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support Michael Kelley
2021-05-14 12:37 ` Mark Rutland
2021-05-14 15:35 ` Michael Kelley
2021-05-17 13:08 ` Mark Rutland
2021-05-17 17:27 ` Michael Kelley
2021-05-18 17:00 ` Mark Rutland
2021-06-08 15:36 ` Michael Kelley [this message]
2021-06-10 16:45 ` Mark Rutland
2021-06-14 2:42 ` Michael Kelley
2021-06-16 20:17 ` Michael Kelley
2021-06-22 9:54 ` Mark Rutland
2021-06-23 8:56 ` Marc Zyngier
2021-06-28 2:21 ` Michael Kelley
2021-05-12 17:37 ` [PATCH v10 4/7] arm64: hyperv: Add kexec and panic handlers Michael Kelley
2021-05-12 17:37 ` [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot Michael Kelley
2021-05-13 15:16 ` Wei Liu
2021-05-12 17:37 ` [PATCH v10 6/7] arm64: efi: Export screen_info Michael Kelley
2021-05-12 17:37 ` [PATCH v10 7/7] Drivers: hv: Enable Hyper-V code to be built on ARM64 Michael Kelley
2021-05-13 13:17 ` [PATCH v10 0/7] Enable Linux guests on Hyper-V " Sudeep Holla
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=MWHPR21MB1593800A20B55626ACE6A844D7379@MWHPR21MB1593.namprd21.prod.outlook.com \
--to=mikelley@microsoft.com \
--cc=Mark.Rutland@arm.com \
--cc=ardb@kernel.org \
--cc=arnd@arndb.de \
--cc=catalin.marinas@arm.com \
--cc=daniel.lezcano@linaro.org \
--cc=kys@microsoft.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-efi@vger.kernel.org \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lorenzo.pieralisi@arm.com \
--cc=sudeep.holla@arm.com \
--cc=wei.liu@kernel.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).