Re: REGRESSION: Upgrading host kernel from 5.11 to 5.13 breaks QEMU guests - perf/fw_devlink/kvm

From: Marc Zyngier <maz@kernel.org>
To: Will Deacon <will@kernel.org>
Cc: "Russell King (Oracle)" <linux@armlinux.org.uk>,
	linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org,
	kvmarm@lists.cs.columbia.edu, James Morse <james.morse@arm.com>,
	Alexandru Elisei <alexandru.elisei@arm.com>,
	Suzuki K Poulose <suzuki.poulose@arm.com>
Subject: Re: REGRESSION: Upgrading host kernel from 5.11 to 5.13 breaks QEMU guests - perf/fw_devlink/kvm
Date: Mon, 20 Sep 2021 11:23:56 +0100	[thread overview]
Message-ID: <87y27rv937.wl-maz@kernel.org> (raw)
In-Reply-To: <20210920095656.GA11961@willie-the-truck>

On Mon, 20 Sep 2021 10:56:57 +0100,
Will Deacon <will@kernel.org> wrote:
> 
> On Sun, Sep 19, 2021 at 02:36:46PM +0100, Marc Zyngier wrote:
> > From 9c26e3e6bbcbc3a583b3974e7a9017029d31fe29 Mon Sep 17 00:00:00 2001
> > From: Marc Zyngier <maz@kernel.org>
> > Date: Sun, 19 Sep 2021 14:09:49 +0100
> > Subject: [PATCH] KVM: arm64: Fix PMU probe ordering
> > 
> > Russell reported that since 5.13, KVM's probing of the PMU has
> > started to fail on his HW. As it turns out, there is an implicit
> > ordering dependency between the architectural PMU probing code and
> > and KVM's own probing. If, due to probe ordering reasons, KVM probes
> > before the PMU driver, it will fail to detect the PMU and prevent it
> > from being advertised to guests as well as the VMM.
> > 
> > Obviously, this is one probing too many, and we should be able to
> > deal with any ordering.
> > 
> > Add a callback from the PMU code into KVM to advertise the registration
> > of a host CPU PMU, allowing for any probing order.
> > 
> > Fixes: 5421db1be3b1 ("KVM: arm64: Divorce the perf code from oprofile helpers")
> > Reported-by: "Russell King (Oracle)" <linux@armlinux.org.uk>
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > Link: https://lore.kernel.org/r/YUYRKVflRtUytzy5@shell.armlinux.org.uk
> > Cc: stable@vger.kernel.org
> > ---
> >  arch/arm64/kvm/perf.c        |  3 ---
> >  arch/arm64/kvm/pmu-emul.c    | 12 +++++++++++-
> >  drivers/perf/arm_pmu.c       |  2 ++
> >  include/kvm/arm_pmu.h        |  3 ---
> >  include/linux/perf/arm_pmu.h |  6 ++++++
> >  5 files changed, 19 insertions(+), 7 deletions(-)
> > 
> > diff --git a/arch/arm64/kvm/perf.c b/arch/arm64/kvm/perf.c
> > index f9bb3b14130e..c84fe24b2ea1 100644
> > --- a/arch/arm64/kvm/perf.c
> > +++ b/arch/arm64/kvm/perf.c
> > @@ -50,9 +50,6 @@ static struct perf_guest_info_callbacks kvm_guest_cbs = {
> >  
> >  int kvm_perf_init(void)
> >  {
> > -	if (kvm_pmu_probe_pmuver() != ID_AA64DFR0_PMUVER_IMP_DEF && !is_protected_kvm_enabled())
> > -		static_branch_enable(&kvm_arm_pmu_available);
> > -
> >  	return perf_register_guest_info_callbacks(&kvm_guest_cbs);
> >  }
> >  
> > diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c
> > index f5065f23b413..588100c52f34 100644
> > --- a/arch/arm64/kvm/pmu-emul.c
> > +++ b/arch/arm64/kvm/pmu-emul.c
> > @@ -740,7 +740,17 @@ void kvm_pmu_set_counter_event_type(struct kvm_vcpu *vcpu, u64 data,
> >  	kvm_pmu_create_perf_event(vcpu, select_idx);
> >  }
> >  
> > -int kvm_pmu_probe_pmuver(void)
> > +void kvm_host_pmu_init(struct arm_pmu *pmu)
> > +{
> > +	if (pmu->pmuver != 0 &&
> > +	    pmu->pmuver != ID_AA64DFR0_PMUVER_IMP_DEF &&
> > +	    !is_protected_kvm_enabled()) {
> > +		static_branch_enable(&kvm_arm_pmu_available);
> > +		kvm_info("PMU detected and enabled\n");
> > +	}
> > +}
> > +
> > +static int kvm_pmu_probe_pmuver(void)
> >  {
> >  	struct perf_event_attr attr = { };
> >  	struct perf_event *event;
> > diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c
> > index 3cbc3baf087f..295cc7952d0e 100644
> > --- a/drivers/perf/arm_pmu.c
> > +++ b/drivers/perf/arm_pmu.c
> > @@ -952,6 +952,8 @@ int armpmu_register(struct arm_pmu *pmu)
> >  		pmu->name, pmu->num_events,
> >  		has_nmi ? ", using NMIs" : "");
> >  
> > +	kvm_host_pmu_init(pmu);
> 
> Just a nit, but I think this will get called for each PMU we probe
> on a big.LITTLE system which is probably harmless, but possible not
> what you want?

Yeah, it is a bit ugly, but harmless. In the future, it would be
useful to track which PMU is used on which CPUs, and this will give us
a decent hook.

I'll tone the print down though.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.