From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D14B6C433B4 for ; Tue, 13 Apr 2021 09:23:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B3AF1613BA for ; Tue, 13 Apr 2021 09:23:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245638AbhDMJXf (ORCPT ); Tue, 13 Apr 2021 05:23:35 -0400 Received: from mail.kernel.org ([198.145.29.99]:47394 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237403AbhDMJXe (ORCPT ); Tue, 13 Apr 2021 05:23:34 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 72D4A611F2; Tue, 13 Apr 2021 09:23:15 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1lWFGL-007CIx-7A; Tue, 13 Apr 2021 10:23:13 +0100 Date: Tue, 13 Apr 2021 10:23:12 +0100 Message-ID: <871rbeo7wf.wl-maz@kernel.org> From: Marc Zyngier To: Peter Geis Cc: Thomas Gleixner , "open list:ARM/Rockchip SoC..." , linux-kernel@vger.kernel.org Subject: Re: [RFC] ITS fails to allocate on rk3568/rk3566 In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: pgwipeout@gmail.com, tglx@linutronix.de, linux-rockchip@lists.infradead.org, linux-kernel@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Peter, On Mon, 12 Apr 2021 21:49:59 +0100, Peter Geis wrote: > > Good Afternoon, > > I am assisting with early bringup of the rk3566 based quartz64 > development board for mainline linux. > I've encountered a few issues with allocating ITS on their version of > the GIC-V3. > The first issue is the ITS controller can only use 32bit addresses. > This leads to the following error: > [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 > [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode > [ 0.000000] GICv3: 320 SPIs implemented > [ 0.000000] GICv3: 0 Extended SPIs implemented > [ 0.000000] GICv3: Distributor has no Range Selector support > [ 0.000000] Root IRQ handler: gic_handle_irq > [ 0.000000] GICv3: 16 PPIs implemented > [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fd460000 > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > [ 0.000000] ITS@0x00000000fd440000: Devices doesn't stick: > f907000100190600 f907000000190600 Ouch. That looks pretty bad. Bit 32 of the register doesn't stick, and that's right in the middle of the address. The register should be fully writable as far as the address field is concerned. Please dump the distributor and ITS IIDR registers so that I can find the TRM for the exact IP. > [ 0.000000] ITS@0x00000000fd440000: failed probing (-6) > [ 0.000000] ITS: No ITS available, not enabling LPIs > > Downstream fixed this by adding the GFP_DMA32 flag to the memory > allocations. Urgh... this really looks like broken silicon to me. > They also force clear the GITS_BASER_SHAREABILITY_MASK. Why? Does this also apply to the command queue? Are they forcing cache flushing? > Unfortunately while this allowed ITS to allocate on downstream, as > soon as MSIs attempted to use it all interrupts would time out. > > On upstream, we observe this during allocation: > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > [ 0.000000] ITS@0x00000000fd440000: allocated 8192 Devices @3810000 > (indirect, esz 8, psz 64K, shr 1) > [ 0.000000] ITS@0x00000000fd440000: allocated 32768 Interrupt > Collections @3820000 (flat, esz 2, psz 64K, shr 1) > [ 0.000000] GICv3: using LPI property table @0x0000000003830000 > [ 0.000000] GICv3: CPU0: using allocated LPI pending table > @0x0000000003840000 > [ 0.000000] ITS queue timeout (64 1) > [ 0.000000] ITS cmd its_build_mapc_cmd failed > [ 0.000000] ITS queue timeout (96 1) > [ 0.000000] ITS cmd its_build_invall_cmd failed > So the command queue is not making forward progress. Either because the ITS cannot access the commands, or because it cannot use the memory it has been allocated. Please dump GITS_CBASER (and the value that has been written to it), just in case it shows the same brokenness as the GITS_BASER registers... [...] > Any assistance you can provide would be greatly appreciated. I'm not sure there is much we can do without a lot more details about the HW. We need to know the exact GIC implementation they are using (ARM has two versions of the GICv3 IP), and we also need to find out *how* this has been integrated. Only Rockchip can tell you that. Once we know which version they are using, and how it is wired, we can start looking at some IMPDEF debug registers. The Linux driver assumes that the ITS is able to access the whole memory. If there are restrictions on what memory ranges can be used, it is going to be a pain to support :-(. M. -- Without deviation from the norm, progress is not possible.