From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27B06C433EF for ; Tue, 22 Mar 2022 14:56:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238277AbiCVO6K (ORCPT ); Tue, 22 Mar 2022 10:58:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238353AbiCVO6D (ORCPT ); Tue, 22 Mar 2022 10:58:03 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 8DBA88A6EA; Tue, 22 Mar 2022 07:56:35 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 425931042; Tue, 22 Mar 2022 07:56:35 -0700 (PDT) Received: from [10.57.43.230] (unknown [10.57.43.230]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 76C6D3F73B; Tue, 22 Mar 2022 07:56:33 -0700 (PDT) Message-ID: Date: Tue, 22 Mar 2022 14:56:29 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: [PATCH v2 0/2] PCI: xgene: Restore working PCIe functionnality Content-Language: en-GB To: Rob Herring , Marc Zyngier Cc: dann frazier , "linux-kernel@vger.kernel.org" , linux-arm-kernel , PCI , Toan Le , Lorenzo Pieralisi , =?UTF-8?Q?Krzysztof_Wilczy=c5=84ski?= , Bjorn Helgaas , =?UTF-8?Q?St=c3=a9phane_Graber?= , Android Kernel Team References: <20220321104843.949645-1-maz@kernel.org> <87h77rxnyl.wl-maz@kernel.org> <87fsnbxgau.wl-maz@kernel.org> <61809b8f-acaa-bae2-ac5e-aa47c55eea23@arm.com> From: Robin Murphy In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2022-03-22 14:39, Rob Herring wrote: > On Tue, Mar 22, 2022 at 01:16:35PM +0000, Robin Murphy wrote: >> On 2022-03-21 20:06, Robin Murphy wrote: >>> On 2022-03-21 19:21, Marc Zyngier wrote: >>>> On Mon, 21 Mar 2022 18:03:27 +0000, >>>> Rob Herring wrote: >>>>> >>>>> On Mon, Mar 21, 2022 at 11:36 AM Marc Zyngier wrote: >>>>>> >>>>>> On Mon, 21 Mar 2022 15:17:34 +0000, >>>>>> Rob Herring wrote: >>>>>>> >>>>>>> On Mon, Mar 21, 2022 at 5:49 AM Marc Zyngier wrote: >>>>>>>> >>>>>>> For XGene-1, I'd still like to understand what the issue is. Reverting >>>>>>> the first fix and fixing 'dma-ranges' should have fixed it. I need a >>>>>>> dump of how the IB registers are initialized in both cases. I'm not >>>>>>> saying changing 'dma-ranges' in the firmware is going to be required >>>>>>> here. There's a couple of other ways we could fix that without a >>>>>>> firmware change, but first I need to understand why it broke. >>>>>> >>>>>> Reverting 6dce5aa59e0b was enough for me, without changing anything >>>>>> else. >>>>> >>>>> Meaning c7a75d07827a didn't matter for you. I'm not sure that it would. >>>>> >>>>> Can you tell me what 'dma-ranges' contains on your system? >>>> >>>> Each pcie node (all 5 of them) has: >>>> >>>> dma-ranges = <0x42000000 0x80 0x00 0x80 0x00 0x00 0x80000000 >>>> �������������� 0x42000000 0x00 0x00 0x00 0x00 0x80 0x00>; > > This is the same as what St�phane has for Merlin. So c7a75d07827a ("PCI: > xgene: Fix IB window setup") should have fixed Mustang. Unless XGene 1 has some weird implicit requirement on the order in which the registers are programmed, that XGene 2 doesn't. And from looking at the code, I don't see any obvious less-mad possibility to explain the breakage. >>> Hmm, is there anyone other than iommu-dma who actually depends on the >>> resource list being sorted in ascending order of bus address? I recall >>> at the time I pushed for creating the list in sorted order as it was the >>> simplest and most efficient option, but there's no technical reason we >>> couldn't create it in as-found order and defer the sorting until >>> iova_reserve_pci_windows() (at worst that could even operate on a >>> temporary copy if need be). It's just more code, which didn't need to >>> exist without a good reason, but if this is one then exist it certainly >>> may. >> >> Taking a closer look, the Cadence driver is already re-sorting the list >> for its own setup, so iommu-dma can't assume the initial sort is >> preserved and needs to do its own anyway. Does the (untested) diff below >> end up helping X-Gene also? > > There's no IOMMU on X-Gene 1 or 2 based on the upstream dts files, so > how would this matter? Because devm_of_pci_get_host_bridge_resources() is forcing the dma_ranges list to be in a different order from the original DT for iommu-dma's benefit, but whether iommu-dma actually consumes it or not later is immaterial. Robin. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BAFE4C433F5 for ; Tue, 22 Mar 2022 15:02:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=S8ae4nQ2PNnESSC5+J/OEUR1yOecVk5M7epymgwNVm8=; b=D4C6aAmgEN1mNv xuzPh/0BTmhTHsLW/PuDXmKpCM1dR6KcswKRDRTODMsWT/Jk38jS40w2yJO82RGakFD8lqsaon6L/ D2G+Z/tChqLw2am/gCROt0hKnzC14Y1wr8p1Dj5s9JXY8xqVQq/v/f4Hz7gUnUS+WIN2TnmNj8Af7 h/rzJfg11+VWNfvFOyTzn2gt5SoCXt3/8BpC+s56QbIBVyFP81ATIp9Jmxx0qchM/5HHdCPe0gdG0 BHzWzY7l/MTPSapzhZRj8hzmpeO9jQI8AlA+0b7MFrYtBQ7L/8lp1sCnel1ON4ZMVmGcp+iJHynW4 c8t7hg54gDb7zvW+Ta9w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWg0W-00BRPH-Bm; Tue, 22 Mar 2022 15:01:12 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWfw5-00BPJj-4z for linux-arm-kernel@lists.infradead.org; Tue, 22 Mar 2022 14:56:38 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 425931042; Tue, 22 Mar 2022 07:56:35 -0700 (PDT) Received: from [10.57.43.230] (unknown [10.57.43.230]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 76C6D3F73B; Tue, 22 Mar 2022 07:56:33 -0700 (PDT) Message-ID: Date: Tue, 22 Mar 2022 14:56:29 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: [PATCH v2 0/2] PCI: xgene: Restore working PCIe functionnality Content-Language: en-GB To: Rob Herring , Marc Zyngier Cc: dann frazier , "linux-kernel@vger.kernel.org" , linux-arm-kernel , PCI , Toan Le , Lorenzo Pieralisi , =?UTF-8?Q?Krzysztof_Wilczy=c5=84ski?= , Bjorn Helgaas , =?UTF-8?Q?St=c3=a9phane_Graber?= , Android Kernel Team References: <20220321104843.949645-1-maz@kernel.org> <87h77rxnyl.wl-maz@kernel.org> <87fsnbxgau.wl-maz@kernel.org> <61809b8f-acaa-bae2-ac5e-aa47c55eea23@arm.com> From: Robin Murphy In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220322_075637_330100_2D892B41 X-CRM114-Status: GOOD ( 20.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMjAyMi0wMy0yMiAxNDozOSwgUm9iIEhlcnJpbmcgd3JvdGU6Cj4gT24gVHVlLCBNYXIgMjIs IDIwMjIgYXQgMDE6MTY6MzVQTSArMDAwMCwgUm9iaW4gTXVycGh5IHdyb3RlOgo+PiBPbiAyMDIy LTAzLTIxIDIwOjA2LCBSb2JpbiBNdXJwaHkgd3JvdGU6Cj4+PiBPbiAyMDIyLTAzLTIxIDE5OjIx LCBNYXJjIFp5bmdpZXIgd3JvdGU6Cj4+Pj4gT24gTW9uLCAyMSBNYXIgMjAyMiAxODowMzoyNyAr MDAwMCwKPj4+PiBSb2IgSGVycmluZyA8cm9iaEBrZXJuZWwub3JnPiB3cm90ZToKPj4+Pj4KPj4+ Pj4gT24gTW9uLCBNYXIgMjEsIDIwMjIgYXQgMTE6MzYgQU0gTWFyYyBaeW5naWVyIDxtYXpAa2Vy bmVsLm9yZz4gd3JvdGU6Cj4+Pj4+Pgo+Pj4+Pj4gT24gTW9uLCAyMSBNYXIgMjAyMiAxNToxNzoz NCArMDAwMCwKPj4+Pj4+IFJvYiBIZXJyaW5nIDxyb2JoQGtlcm5lbC5vcmc+IHdyb3RlOgo+Pj4+ Pj4+Cj4+Pj4+Pj4gT24gTW9uLCBNYXIgMjEsIDIwMjIgYXQgNTo0OSBBTSBNYXJjIFp5bmdpZXIg PG1hekBrZXJuZWwub3JnPiB3cm90ZToKPj4+Pj4+Pj4KPj4+Pj4+PiBGb3IgWEdlbmUtMSwgSSdk IHN0aWxsIGxpa2UgdG8gdW5kZXJzdGFuZCB3aGF0IHRoZSBpc3N1ZSBpcy4gUmV2ZXJ0aW5nCj4+ Pj4+Pj4gdGhlIGZpcnN0IGZpeCBhbmQgZml4aW5nICdkbWEtcmFuZ2VzJyBzaG91bGQgaGF2ZSBm aXhlZCBpdC4gSSBuZWVkIGEKPj4+Pj4+PiBkdW1wIG9mIGhvdyB0aGUgSUIgcmVnaXN0ZXJzIGFy ZSBpbml0aWFsaXplZCBpbiBib3RoIGNhc2VzLiBJJ20gbm90Cj4+Pj4+Pj4gc2F5aW5nIGNoYW5n aW5nICdkbWEtcmFuZ2VzJyBpbiB0aGUgZmlybXdhcmUgaXMgZ29pbmcgdG8gYmUgcmVxdWlyZWQK Pj4+Pj4+PiBoZXJlLiBUaGVyZSdzIGEgY291cGxlIG9mIG90aGVyIHdheXMgd2UgY291bGQgZml4 IHRoYXQgd2l0aG91dCBhCj4+Pj4+Pj4gZmlybXdhcmUgY2hhbmdlLCBidXQgZmlyc3QgSSBuZWVk IHRvIHVuZGVyc3RhbmQgd2h5IGl0IGJyb2tlLgo+Pj4+Pj4KPj4+Pj4+IFJldmVydGluZyA2ZGNl NWFhNTllMGIgd2FzIGVub3VnaCBmb3IgbWUsIHdpdGhvdXQgY2hhbmdpbmcgYW55dGhpbmcKPj4+ Pj4+IGVsc2UuCj4+Pj4+Cj4+Pj4+IE1lYW5pbmcgYzdhNzVkMDc4MjdhIGRpZG4ndCBtYXR0ZXIg Zm9yIHlvdS4gSSdtIG5vdCBzdXJlIHRoYXQgaXQgd291bGQuCj4+Pj4+Cj4+Pj4+IENhbiB5b3Ug dGVsbCBtZSB3aGF0ICdkbWEtcmFuZ2VzJyBjb250YWlucyBvbiB5b3VyIHN5c3RlbT8KPj4+Pgo+ Pj4+IEVhY2ggcGNpZSBub2RlIChhbGwgNSBvZiB0aGVtKSBoYXM6Cj4+Pj4KPj4+PiBkbWEtcmFu Z2VzID0gPDB4NDIwMDAwMDAgMHg4MCAweDAwIDB4ODAgMHgwMCAweDAwIDB4ODAwMDAwMDAKPj4+ PiDDr8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr3D r8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr3Dr8K/wr0gMHg0MjAwMDAwMCAweDAwIDB4MDAgMHgwMCAw eDAwIDB4ODAgMHgwMD47Cj4gCj4gVGhpcyBpcyB0aGUgc2FtZSBhcyB3aGF0IFN0w6/Cv8K9cGhh bmUgaGFzIGZvciBNZXJsaW4uIFNvIGM3YTc1ZDA3ODI3YSAoIlBDSToKPiB4Z2VuZTogRml4IElC IHdpbmRvdyBzZXR1cCIpIHNob3VsZCBoYXZlIGZpeGVkIE11c3RhbmcuCgpVbmxlc3MgWEdlbmUg MSBoYXMgc29tZSB3ZWlyZCBpbXBsaWNpdCByZXF1aXJlbWVudCBvbiB0aGUgb3JkZXIgaW4gd2hp Y2ggCnRoZSByZWdpc3RlcnMgYXJlIHByb2dyYW1tZWQsIHRoYXQgWEdlbmUgMiBkb2Vzbid0LiBB bmQgZnJvbSBsb29raW5nIGF0IAp0aGUgY29kZSwgSSBkb24ndCBzZWUgYW55IG9idmlvdXMgbGVz cy1tYWQgcG9zc2liaWxpdHkgdG8gZXhwbGFpbiB0aGUgCmJyZWFrYWdlLgoKPj4+IEhtbSwgaXMg dGhlcmUgYW55b25lIG90aGVyIHRoYW4gaW9tbXUtZG1hIHdobyBhY3R1YWxseSBkZXBlbmRzIG9u IHRoZQo+Pj4gcmVzb3VyY2UgbGlzdCBiZWluZyBzb3J0ZWQgaW4gYXNjZW5kaW5nIG9yZGVyIG9m IGJ1cyBhZGRyZXNzPyBJIHJlY2FsbAo+Pj4gYXQgdGhlIHRpbWUgSSBwdXNoZWQgZm9yIGNyZWF0 aW5nIHRoZSBsaXN0IGluIHNvcnRlZCBvcmRlciBhcyBpdCB3YXMgdGhlCj4+PiBzaW1wbGVzdCBh bmQgbW9zdCBlZmZpY2llbnQgb3B0aW9uLCBidXQgdGhlcmUncyBubyB0ZWNobmljYWwgcmVhc29u IHdlCj4+PiBjb3VsZG4ndCBjcmVhdGUgaXQgaW4gYXMtZm91bmQgb3JkZXIgYW5kIGRlZmVyIHRo ZSBzb3J0aW5nIHVudGlsCj4+PiBpb3ZhX3Jlc2VydmVfcGNpX3dpbmRvd3MoKSAoYXQgd29yc3Qg dGhhdCBjb3VsZCBldmVuIG9wZXJhdGUgb24gYQo+Pj4gdGVtcG9yYXJ5IGNvcHkgaWYgbmVlZCBi ZSkuIEl0J3MganVzdCBtb3JlIGNvZGUsIHdoaWNoIGRpZG4ndCBuZWVkIHRvCj4+PiBleGlzdCB3 aXRob3V0IGEgZ29vZCByZWFzb24sIGJ1dCBpZiB0aGlzIGlzIG9uZSB0aGVuIGV4aXN0IGl0IGNl cnRhaW5seQo+Pj4gbWF5Lgo+Pgo+PiBUYWtpbmcgYSBjbG9zZXIgbG9vaywgdGhlIENhZGVuY2Ug ZHJpdmVyIGlzIGFscmVhZHkgcmUtc29ydGluZyB0aGUgbGlzdAo+PiBmb3IgaXRzIG93biBzZXR1 cCwgc28gaW9tbXUtZG1hIGNhbid0IGFzc3VtZSB0aGUgaW5pdGlhbCBzb3J0IGlzCj4+IHByZXNl cnZlZCBhbmQgbmVlZHMgdG8gZG8gaXRzIG93biBhbnl3YXkuIERvZXMgdGhlICh1bnRlc3RlZCkg ZGlmZiBiZWxvdwo+PiBlbmQgdXAgaGVscGluZyBYLUdlbmUgYWxzbz8KPiAKPiBUaGVyZSdzIG5v IElPTU1VIG9uIFgtR2VuZSAxIG9yIDIgYmFzZWQgb24gdGhlIHVwc3RyZWFtIGR0cyBmaWxlcywg c28KPiBob3cgd291bGQgdGhpcyBtYXR0ZXI/CgpCZWNhdXNlIGRldm1fb2ZfcGNpX2dldF9ob3N0 X2JyaWRnZV9yZXNvdXJjZXMoKSBpcyBmb3JjaW5nIHRoZSAKZG1hX3JhbmdlcyBsaXN0IHRvIGJl IGluIGEgZGlmZmVyZW50IG9yZGVyIGZyb20gdGhlIG9yaWdpbmFsIERUIGZvciAKaW9tbXUtZG1h J3MgYmVuZWZpdCwgYnV0IHdoZXRoZXIgaW9tbXUtZG1hIGFjdHVhbGx5IGNvbnN1bWVzIGl0IG9y IG5vdCAKbGF0ZXIgaXMgaW1tYXRlcmlhbC4KClJvYmluLgoKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QK bGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRl YWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=