From: Marc Zyngier <firstname.lastname@example.org> To: Will Deacon <email@example.com> Cc: Andrew Murray <firstname.lastname@example.org>, Catalin Marinas <Catalin.Marinas@arm.com>, Mark Rutland <Mark.Rutland@arm.com>, Sudeep Holla <Sudeep.Holla@arm.com>, email@example.com, kvmarm <firstname.lastname@example.org>, linux-arm-kernel <email@example.com>, firstname.lastname@example.org Subject: Re: [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full SPE profiling buffer controls Date: Wed, 08 Jan 2020 12:36:11 +0000 Message-ID: <email@example.com> (raw) In-Reply-To: <20200108115816.GB15861@willie-the-truck> On 2020-01-08 11:58, Will Deacon wrote: > On Wed, Jan 08, 2020 at 11:17:16AM +0000, Marc Zyngier wrote: >> On 2020-01-07 15:13, Andrew Murray wrote: >> > On Sat, Dec 21, 2019 at 02:13:25PM +0000, Marc Zyngier wrote: >> > > On Fri, 20 Dec 2019 14:30:16 +0000 >> > > Andrew Murray <firstname.lastname@example.org> wrote: >> > > >> > > [somehow managed not to do a reply all, re-sending] >> > > >> > > > From: Sudeep Holla <email@example.com> >> > > > >> > > > Now that we can save/restore the full SPE controls, we can enable it >> > > > if SPE is setup and ready to use in KVM. It's supported in KVM only if >> > > > all the CPUs in the system supports SPE. >> > > > >> > > > However to support heterogenous systems, we need to move the check if >> > > > host supports SPE and do a partial save/restore. >> > > >> > > No. Let's just not go down that path. For now, KVM on heterogeneous >> > > systems do not get SPE. >> > >> > At present these patches only offer the SPE feature to VCPU's where the >> > sanitised AA64DFR0 register indicates that all CPUs have this support >> > (kvm_arm_support_spe_v1) at the time of setting the attribute >> > (KVM_SET_DEVICE_ATTR). >> > >> > Therefore if a new CPU comes online without SPE support, and an >> > existing VCPU is scheduled onto it, then bad things happen - which I >> > guess >> > must have been the intention behind this patch. >> >> I guess that was the intent. >> >> > > If SPE has been enabled on a guest and a CPU >> > > comes up without SPE, this CPU should fail to boot (same as exposing a >> > > feature to userspace). >> > >> > I'm unclear as how to prevent this. We can set the FTR_STRICT flag on >> > the sanitised register - thus tainting the kernel if such a non-SPE CPU >> > comes online - thought that doesn't prevent KVM from blowing up. Though >> > I don't believe we can prevent a CPU coming up. At the moment this is >> > my preferred approach. >> >> I'd be OK with this as a stop-gap measure. Do we know of any existing >> design where only half of the CPUs have SPE? > > No, but given how few CPUs implement SPE I'd say that this > configuration > is inevitable. I certainly went out of my way to support it in the > driver. > >> > Looking at the vcpu_load and related code, I don't see a way of saying >> > 'don't schedule this VCPU on this CPU' or bailing in any way. >> >> That would actually be pretty easy to implement. In vcpu_load(), check >> that that the CPU physical has SPE. If not, raise a request for that >> vcpu. >> In the run loop, check for that request and abort if raised, returning >> to userspace. >> >> Userspace can always check /sys/devices/arm_spe_0/cpumask and work out >> where to run that particular vcpu. > > It's also worth considering systems where there are multiple > implementations > of SPE in play. Assuming we don't want to expose this to a guest, then > the > right interface here is probably for userspace to pick one SPE > implementation and expose that to the guest. That fits with your idea > above, > where you basically get an immediate exit if we try to schedule a vCPU > onto > a CPU that isn't part of the SPE mask. Then it means that the VM should be configured with a mask indicating which CPUs it is intended to run on, and setting such a mask is mandatory for SPE. > >> > One solution could be to allow scheduling onto non-SPE VCPUs but wrap >> > the >> > SPE save/restore code in a macro (much like kvm_arm_spe_v1_ready) that >> > reads the non-sanitised feature register. Therefore we don't go bang, >> > but >> > we also increase the size of any black-holes in SPE capturing. Though >> > this >> > feels like something that will cause grief down the line. >> > >> > Is there something else that can be done? >> >> How does userspace deal with this? When SPE is only available on half >> of >> the CPUs, how does perf work in these conditions? > > Not sure about userspace, but the kernel driver works by instantiating > an > SPE PMU instance only for the CPUs that have it and then that instance > profiles for only those CPUs. You also need to do something similar if > you had two CPU types with SPE, since the SPE configuration is likely > to be > different between them. So that's closer to what Andrew was suggesting above (running a guest on a non-SPE CPU creates a profiling black hole). Except that we can't really run a SPE-enabled guest on a non-SPE CPU, as the SPE sysregs will UNDEF at EL1. Conclusion: we need a mix of a cpumask to indicate which CPUs we want to run on (generic, not-SPE related), and a check for SPE-capable CPUs. If any of these condition is not satisfied, the vcpu exits for userspace to sort out the affinity. I hate heterogeneous systems. M. -- Jazz is not dead. It just smells funny...
next prev parent reply index Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-12-20 14:30 [PATCH v2 00/18] arm64: KVM: add SPE profiling support Andrew Murray 2019-12-20 14:30 ` [PATCH v2 01/18] dt-bindings: ARM SPE: highlight the need for PPI partitions on heterogeneous systems Andrew Murray 2019-12-20 14:30 ` [PATCH v2 02/18] arm64: KVM: reset E2PB correctly in MDCR_EL2 when exiting the guest(VHE) Andrew Murray 2019-12-21 13:12 ` Marc Zyngier 2019-12-24 10:29 ` Andrew Murray 2020-01-02 16:21 ` Andrew Murray 2019-12-20 14:30 ` [PATCH v2 03/18] arm64: KVM: define SPE data structure for each vcpu Andrew Murray 2019-12-21 13:19 ` Marc Zyngier 2019-12-24 12:01 ` Andrew Murray 2019-12-20 14:30 ` [PATCH v2 04/18] arm64: KVM: add SPE system registers to sys_reg_descs Andrew Murray 2019-12-20 14:30 ` [PATCH v2 05/18] arm64: KVM/VHE: enable the use PMSCR_EL12 on VHE systems Andrew Murray 2019-12-20 14:30 ` [PATCH v2 06/18] arm64: KVM: split debug save restore across vm/traps activation Andrew Murray 2019-12-20 14:30 ` [PATCH v2 07/18] arm64: KVM/debug: drop pmscr_el1 and use sys_regs[PMSCR_EL1] in kvm_cpu_context Andrew Murray 2019-12-20 14:30 ` [PATCH v2 08/18] arm64: KVM: add support to save/restore SPE profiling buffer controls Andrew Murray 2019-12-21 13:57 ` Marc Zyngier 2019-12-24 10:49 ` Andrew Murray 2019-12-24 15:17 ` Andrew Murray 2019-12-24 15:48 ` Marc Zyngier 2019-12-20 14:30 ` [PATCH v2 09/18] arm64: KVM: enable conditional save/restore full " Andrew Murray 2019-12-20 18:06 ` Mark Rutland 2019-12-24 12:15 ` Andrew Murray 2019-12-21 14:13 ` Marc Zyngier 2020-01-07 15:13 ` Andrew Murray 2020-01-08 11:17 ` Marc Zyngier 2020-01-08 11:58 ` Will Deacon 2020-01-08 12:36 ` Marc Zyngier [this message] 2020-01-08 13:10 ` Will Deacon 2020-01-09 11:23 ` Andrew Murray 2020-01-09 11:25 ` Andrew Murray 2020-01-09 12:01 ` Will Deacon 2020-01-10 10:54 ` Andrew Murray 2020-01-10 11:04 ` Andrew Murray 2020-01-10 11:51 ` Marc Zyngier 2020-01-10 12:12 ` Andrew Murray 2020-01-10 11:18 ` Marc Zyngier 2020-01-10 12:12 ` Andrew Murray 2020-01-10 13:34 ` Marc Zyngier 2019-12-20 14:30 ` [PATCH v2 10/18] arm64: KVM/debug: use EL1&0 stage 1 translation regime Andrew Murray 2019-12-22 10:34 ` Marc Zyngier 2019-12-24 11:11 ` Andrew Murray 2020-01-13 16:31 ` Andrew Murray 2020-01-15 14:03 ` Marc Zyngier 2019-12-20 14:30 ` [PATCH v2 11/18] KVM: arm64: don't trap Statistical Profiling controls to EL2 Andrew Murray 2019-12-20 18:08 ` Mark Rutland 2019-12-22 10:42 ` Marc Zyngier 2019-12-23 11:56 ` Andrew Murray 2019-12-23 12:05 ` Marc Zyngier 2019-12-23 12:10 ` Andrew Murray 2020-01-09 17:25 ` Andrew Murray 2020-01-09 17:42 ` Mark Rutland 2020-01-09 17:46 ` Andrew Murray 2019-12-20 14:30 ` [PATCH v2 12/18] KVM: arm64: add a new vcpu device control group for SPEv1 Andrew Murray 2019-12-22 11:03 ` Marc Zyngier 2019-12-24 12:30 ` Andrew Murray 2019-12-20 14:30 ` [PATCH v2 13/18] perf: arm_spe: Add KVM structure for obtaining IRQ info Andrew Murray 2019-12-22 11:24 ` Marc Zyngier 2019-12-24 12:35 ` Andrew Murray 2019-12-20 14:30 ` [PATCH v2 14/18] KVM: arm64: spe: Provide guest virtual interrupts for SPE Andrew Murray 2019-12-22 12:07 ` Marc Zyngier 2019-12-24 11:50 ` Andrew Murray 2019-12-24 12:42 ` Marc Zyngier 2019-12-24 13:08 ` Andrew Murray 2019-12-24 13:22 ` Marc Zyngier 2019-12-24 13:36 ` Andrew Murray 2019-12-24 13:46 ` Marc Zyngier 2019-12-20 14:30 ` [PATCH v2 15/18] perf: arm_spe: Handle guest/host exclusion flags Andrew Murray 2019-12-20 18:10 ` Mark Rutland 2019-12-22 12:10 ` Marc Zyngier 2019-12-23 12:10 ` Andrew Murray 2019-12-23 12:18 ` Marc Zyngier 2019-12-20 14:30 ` [PATCH v2 16/18] KVM: arm64: enable SPE support Andrew Murray 2019-12-20 14:30 ` [PATCH v2 17/18, KVMTOOL] update_headers: Sync kvm UAPI headers with linux v5.5-rc2 Andrew Murray 2019-12-20 14:30 ` [PATCH v2 18/18, KVMTOOL] kvm: add a vcpu feature for SPEv1 support Andrew Murray 2019-12-20 17:55 ` [PATCH v2 00/18] arm64: KVM: add SPE profiling support Mark Rutland 2019-12-24 12:54 ` Andrew Murray 2019-12-21 10:48 ` Marc Zyngier 2019-12-22 12:22 ` Marc Zyngier 2019-12-24 12:56 ` Andrew Murray
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --cc=Catalin.Marinas@arm.com \ --cc=Mark.Rutland@arm.com \ --cc=Sudeep.Holla@arm.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
KVM Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/kvm/0 kvm/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 kvm kvm/ https://lore.kernel.org/kvm \ firstname.lastname@example.org public-inbox-index kvm Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.kvm AGPL code for this site: git clone https://public-inbox.org/public-inbox.git