Re: [PATCH RFC 5/9] arm64: dts: rockchip: rk1808k-toybrick-m0: Suppress vGIC interrupt

From: Marc Zyngier <maz@kernel.org>
To: "Andreas Färber" <afaerber@suse.de>
Cc: linux-rockchip@lists.infradead.org,
	linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org, Rob Herring <robh+dt@kernel.org>,
	Heiko Stuebner <heiko@sntech.de>,
	devicetree@vger.kernel.org
Subject: Re: [PATCH RFC 5/9] arm64: dts: rockchip: rk1808k-toybrick-m0: Suppress vGIC interrupt
Date: Mon, 24 May 2021 16:46:23 +0100	[thread overview]
Message-ID: <87eedww2wg.wl-maz@kernel.org> (raw)
In-Reply-To: <f9523a3c-16a7-16d1-4d2e-9994c3fe03d9@suse.de>

On Mon, 24 May 2021 15:40:22 +0100,
Andreas Färber <afaerber@suse.de> wrote:
> 
> On 17.05.21 11:29, Marc Zyngier wrote:
> > On Mon, 17 May 2021 00:05:47 +0100,
> > Andreas Färber <afaerber@suse.de> wrote:
> >>
> >> Avoid the kernel getting stuck after:
> >>
> >> [    1.175956] kvm [1]: IPA Size Limit: 40 bits
> >> [    1.177164] kvm [1]: vgic-v2@ff320000
> >> [    1.177545] kvm [1]: GIC system register CPU interface enabled
> >>
> >> or when dropping GICV reg entry:
> >>
> >> [    1.176001] kvm [1]: IPA Size Limit: 40 bits
> >> [    1.177191] kvm [1]: GICv3: no GICV resource entry
> >> [    1.177664] kvm [1]: disabling GICv2 emulation
> >> [    1.178115] kvm [1]: GIC system register CPU interface enabled
> >>
> >> Signed-off-by: Andreas Färber <afaerber@suse.de>
> >> ---
> >>  arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts | 4 ++++
> >>  1 file changed, 4 insertions(+)
> >>
> >> diff --git a/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts b/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts
> >> index 2f8075d2391c..15293a8576c6 100644
> >> --- a/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts
> >> +++ b/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts
> >> @@ -48,6 +48,10 @@ &cpu1 {
> >>  	cpu-supply = <&vdd_cpu>;
> >>  };
> >>  
> >> +&gic {
> >> +	/delete-property/ interrupts;
> >> +};
> >> +
> >>  &uart2 {
> >>  	status = "okay";
> >>  	clocks = <&xin24m>;
> > 
> > As I said in my reply to the cover letter, this is not an acceptable
> > outcome. Please add some debug to kvm_vgic_hyp_init() to understand
> > where this is hanging and why.
> 
> Many thanks for that pointer.
> 
> So, as alternative to dropping the DT interrupts property above, I could
> also work around this issue by commenting out
> vgic-init.c:vgic_init_cpu_starting()'s enable_percpu_irq() call.
> 
> Otherwise I am seeing the following call flow:
> 
> cpuhp_setup_state() -> __cpuhp_setup_state_cpuslocked() ->
> cpuhp_issue_call() -> cpuhp_invoke_ap_callback() -> __cpuhp_kick_ap() ->
> wait_for_ap_thread() -> wait_for_completion() --- doesn't return
> 
> With kvm_info() / printk():
> 
> [    1.244079] kvm [1]: IPA Size Limit: 40 bits
> 
> [    1.245205] kvm [1]: vgic-v2@ff320000
> 
> [    1.245584] kvm [1]: GIC system register CPU interface enabled
> 
> [    1.246177] kvm [1]: before cpuhp_setup_state
> 
> [    1.246605] __cpuhp_setup_state_cpuslocked: kvm/arm/vgic:starting
> 
> [    1.247198] __cpuhp_setup_state_cpuslocked: for_each_present_cpu 0:
> state 225
> 
> [    1.247933] __cpuhp_setup_state_cpuslocked: for_each_present_cpu 0:
> before cpuhp_issue_call
> 
> [    1.248745] cpuhp_issue_call: before invoke
> 
> [    1.249154] cpuhp_issue_call: before AP invoke
> 
> [    1.249585] cpuhp_invoke_ap_callback
> 
> [    1.249936] cpuhp_invoke_ap_callback: after cpu_online
> 
> [    1.250435] cpuhp_invoke_ap_callback: before st->thread
> 
> [    1.250944] cpuhp_invoke_ap_callback: after st->thread
> 
> [    1.251445] __cpuhp_kick_ap
> 
> [    1.251731] __cpuhp_kick_ap: not returned
> 
> [    1.252140] vgic_init_cpu_starting: 9
> 
> [    1.252507] vgic_init_cpu_starting: done
> 
> [    1.255538] __cpuhp_kick_ap: wait_for_ap_thread

And you never see any RCU stall after that? It looks like a CPU has
disappeared in the weeds after enabling the per-CPU interrupt.

Please instrument what happens in
drivers/irqchip/irq-gic-v3.c::gic_unmask_irq() when d->hwirq == 9, a
well as vgic_maintenance_handler(), just in case it gets called...

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.