From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 516E0C282C3 for ; Tue, 22 Jan 2019 09:53:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 22CA7218D0 for ; Tue, 22 Jan 2019 09:53:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727846AbfAVJxo (ORCPT ); Tue, 22 Jan 2019 04:53:44 -0500 Received: from foss.arm.com ([217.140.101.70]:49694 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726872AbfAVJxn (ORCPT ); Tue, 22 Jan 2019 04:53:43 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4430AEBD; Tue, 22 Jan 2019 01:53:43 -0800 (PST) Received: from big-swifty.misterjones.org (big-swifty.copenhagen.arm.com [10.32.148.139]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 929923F6A8; Tue, 22 Jan 2019 01:53:36 -0800 (PST) Date: Tue, 22 Jan 2019 09:53:30 +0000 Message-ID: <86fttl9fxh.wl-marc.zyngier@arm.com> From: Marc Zyngier To: Heyi Guo Cc: , Thomas Gleixner , Jason Cooper , Subject: Re: [RFC] irq-gic-v3-its: fix occasional VLPI drop In-Reply-To: <1548071508-11846-1-git-send-email-guoheyi@huawei.com> References: <1548071508-11846-1-git-send-email-guoheyi@huawei.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/25.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: ARM Ltd MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Heyi, On Mon, 21 Jan 2019 11:51:48 +0000, Heyi Guo wrote: > > Every VLPI will temporarily be mapped to the first CPU in system > (normally CPU0) and then moved to the real scheduled CPU later. There > is a time window so a VLPI may be sent to CPU0 instead of the real > scheduled vCPU, in a multi-CPU virtual machine. However, CPU0 may have > not been scheduled as a virtual CPU after system boots up, so the > value of GICR_VPROPBASER still be the reset value. According to GIC > spec, the reset value of IDbits in GICR_VPROPBASER is architecturally > UNKNOWN, and the GIC will behave as if all virtual LPIs are out of > range if it is less than 0b1101. On our platform the GICR will simply > drop the incoming VLPI, which results in interrupt missing in Guest. OK, it took me some time to page this horror back in. So let's see if I can sum-up the issue correctly: - When a VM gets created, all the vPEs are mapped to CPU0's redistributor. - If a device starts emitting VLPIs targeting a vPE that has not run yet, these VLPIs are forwarded to CPU0's redistributor. - If CPU0 has itself never run any vPE, its GICR_PROPBASER is not initialised, meaning that the IDbits field may contain a value that makes the redistributor drop the interrupt on the floor. Is that a correct assessment of the issue you're seeing? If so, I think you have a very good point here, and this looks like a hole in the driver. Comments below: > As no code will clear GICR_VPROPBASER at runtime, we can safely > initialize the IDbits field at boot time for each CPU to get rid of > this issue. > > Signed-off-by: Heyi Guo > Signed-off-by: Heyi Guo > --- > drivers/irqchip/irq-gic-v3-its.c | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) > > diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c > index db20e99..6116215 100644 > --- a/drivers/irqchip/irq-gic-v3-its.c > +++ b/drivers/irqchip/irq-gic-v3-its.c > @@ -2144,6 +2144,20 @@ static void its_cpu_init_lpis(void) > val |= GICR_CTLR_ENABLE_LPIS; > writel_relaxed(val, rbase + GICR_CTLR); > > + /* > + * Temporary workaround for vlpi drop on Hi1620. Why is this specific to this implementation? Isn't this an issue that affects every GICv4 implementations? > + * IDbits must be set before any VLPI is sent to this CPU, or else the > + * VLPI will be considered as out of range and dropped. > + */ > + if (gic_rdists->has_vlpis) { > + void __iomem *vlpi_base = gic_data_rdist_vlpi_base(); > + > + val = (LPI_NRBITS - 1) & GICR_VPROPBASER_IDBITS_MASK; > + pr_info("GICv4: CPU%d: Init IDbits to 0x%llx for GICR_VPROPBASER\n", > + smp_processor_id(), val); I don't think this pr_info is useful to a normal user, as it is only debug information. I'm actually minded to demote a bunch of the GICv3 prints to pr_debug. > + gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER); > + } > + I think we need to clear GICR_VPENDBASER.Valid too (you can probably reuse part of its_vpe_deschedule for that), so that we don't get into a bizarre situation where CPU0's redistributor has some ancient programming left in, and could start corrupting memory. > /* Make sure the GIC has seen the above */ > dsb(sy); > out: > -- > 1.8.3.1 > Can you please respin this quickly with the above changes? Thanks, M. -- Jazz is not dead, it just smell funny.