From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0B90EB64DB for ; Mon, 19 Jun 2023 09:07:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230460AbjFSJHh (ORCPT ); Mon, 19 Jun 2023 05:07:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230190AbjFSJHf (ORCPT ); Mon, 19 Jun 2023 05:07:35 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF507B4; Mon, 19 Jun 2023 02:07:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 441AA60B42; Mon, 19 Jun 2023 09:07:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33A63C433C8; Mon, 19 Jun 2023 09:07:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687165653; bh=14J4WS+8yy469joEoEVl33nkw3ghjN+BWRbdAbpGaU0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=B9tlb1O7jbTOkBZh0+L/dMVuqQVsH5U+Wy/XX47i0cD6T/QRklmp/GNNT2/XtMP80 3YL4FQgWuyrs5jvAHOJ9wyD+9N8FSJuWyY9WB3ZBjYaGpF8NfjjoXsl0p/SNP+aHXg kVtja0B7ictQY+Ue96WUtQmr2jmfLVjdQ5uYIVO/bSWuuJb/TjurGkOd+Qg++4JSfz tiRGlwp/7c9UF6/tOnLRgds404eejjbY3Vu6zAP7SfsZdJUeHKHyyX6eQx/9U/peRZ 3hH4JYTMV8EfDtL2desisfGwrWGJKryttrAQEFp/VLXW3nsV5en8Q+KXKzc1Ez9XMa gNs3HBK7H8XlQ== Date: Mon, 19 Jun 2023 11:07:24 +0200 From: Lorenzo Pieralisi To: Hongxing Zhu Cc: Bjorn Helgaas , "l.stach@pengutronix.de" , "bhelgaas@google.com" , "linux-pci@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "kernel@pengutronix.de" , dl-linux-imx , Serge Semin Subject: Re: [PATCH v2] PCI: imx6: Save and restore MSI control of RC in suspend and resume Message-ID: References: <20230317222436.GA1978818@bhelgaas> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Apr 10, 2023 at 06:48:48AM +0000, Hongxing Zhu wrote: [...] > > I am getting back to this since I am still not convinced and I want to understand > > this once for all. > > > > We do use dw_pcie_find_capability() in most DWC drivers to find and peek/poke > > at eg PCI express capability of the *Root port* (?), > > > > eg dw_pcie_wait_for_link() > > > > so I assume that for iMX6 dw_pcie_find_capability() does just the same, which > > would mean that we are poking the "Message Control" field of the Root port MSI > > capability. > > > > Either that (which would mean that iMX6 has a HW bug because the RP Message > > Control field does not control the delivery of MSIs from endpoints but just for the > > root port itself ) or all DWC controllers modelled the root complex MMIO space as > > a set of PCI/PCIe capabilities that are NOT necessarily mappable to PCI > > specifications defined ones. > > > > Can anyone please shed some light on this ? I don't have DWC HW, we need to > > know before merging this code. > Hi Lorenzo: > Regarding my understanding, DWC HW has the PCI/PCIe capability map when > it works in RC mode and Spec doesn’t specify these Caps for host controller. > And, there are comments describe these callbacks already in pcie-designware.c. > ... > /* > * These interfaces resemble the pci_find_*capability() interfaces, but these > * are for configuring host controllers, which are bridges *to* PCI devices but > * are not PCI devices themselves. > */ > static u8 __dw_pcie_find_next_cap(struct dw_pcie *pci, u8 cap_ptr, > u8 cap) > ... > > I still believe this is an integration bug, more so after reading the commit Serge pointed out: 75cb8d20c112 ("PCI: imx: Enable MSI from downstream components"). The commit above implies that if you have CONFIG_PCIEPORTBUS enabled, you would not need to set the MSI enable bit explicitly because that's done by the port driver while requesting RP services. This means that it is _seen_ by the PCI core as a capability register and it also means that if you have CONFIG_PCIEPORTBUS enabled and that the port driver disables MSIs, all downstream MSIs are disabled, not only the RP ones (as it should be according to the PCI specs). So, this is a HW bug I am afraid - I will merge this patch but AFAICS the HW integration bug is there regardless, however we slice it. Thanks, Lorenzo From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3E05EB64D9 for ; Mon, 19 Jun 2023 09:08:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bkqLZCLDevdoQXxfv96YXi7VIOZaPVhllh4ceeLz0oo=; b=lcfwXhDTl8Lrx0 eY8s3GYKnKsmUZvPG64V5rt0vCEh03pekxGC7+E5XJH0NW9YXVNmVAQXPfGhDKo/b8AZglzKz4YTL utH/p5rABFIb6OnJJul4OOWSSZoCFpP8ipQV2pATaU/n4+WGcQ8fSURV9O6r6Qnn2Q4Iegn9v21LY o2XVxM2V0Ng7NNUqS0Ku4FxuyGpowW3GtAD5AV8aB+W7aIM+dIi14DkXYeVFdTSmFuNyesOFpe6Uy gsnwBL9qwJFb31qOvZNqMlQF/VvFhPhkBULpWHu0E2LK5eM6L2yGvm5S2otu4P6OSPW2Gm49KtcOe +kAejRmXonxWZEzEw/1g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qBArL-007wsB-0c; Mon, 19 Jun 2023 09:07:39 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qBArH-007wqz-2E for linux-arm-kernel@lists.infradead.org; Mon, 19 Jun 2023 09:07:37 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 46DFE60B70; Mon, 19 Jun 2023 09:07:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33A63C433C8; Mon, 19 Jun 2023 09:07:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687165653; bh=14J4WS+8yy469joEoEVl33nkw3ghjN+BWRbdAbpGaU0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=B9tlb1O7jbTOkBZh0+L/dMVuqQVsH5U+Wy/XX47i0cD6T/QRklmp/GNNT2/XtMP80 3YL4FQgWuyrs5jvAHOJ9wyD+9N8FSJuWyY9WB3ZBjYaGpF8NfjjoXsl0p/SNP+aHXg kVtja0B7ictQY+Ue96WUtQmr2jmfLVjdQ5uYIVO/bSWuuJb/TjurGkOd+Qg++4JSfz tiRGlwp/7c9UF6/tOnLRgds404eejjbY3Vu6zAP7SfsZdJUeHKHyyX6eQx/9U/peRZ 3hH4JYTMV8EfDtL2desisfGwrWGJKryttrAQEFp/VLXW3nsV5en8Q+KXKzc1Ez9XMa gNs3HBK7H8XlQ== Date: Mon, 19 Jun 2023 11:07:24 +0200 From: Lorenzo Pieralisi To: Hongxing Zhu Cc: Bjorn Helgaas , "l.stach@pengutronix.de" , "bhelgaas@google.com" , "linux-pci@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "kernel@pengutronix.de" , dl-linux-imx , Serge Semin Subject: Re: [PATCH v2] PCI: imx6: Save and restore MSI control of RC in suspend and resume Message-ID: References: <20230317222436.GA1978818@bhelgaas> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230619_020735_832235_D338BB41 X-CRM114-Status: GOOD ( 28.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gTW9uLCBBcHIgMTAsIDIwMjMgYXQgMDY6NDg6NDhBTSArMDAwMCwgSG9uZ3hpbmcgWmh1IHdy b3RlOgoKWy4uLl0KCj4gPiBJIGFtIGdldHRpbmcgYmFjayB0byB0aGlzIHNpbmNlIEkgYW0gc3Rp bGwgbm90IGNvbnZpbmNlZCBhbmQgSSB3YW50IHRvIHVuZGVyc3RhbmQKPiA+IHRoaXMgb25jZSBm b3IgYWxsLgo+ID4gCj4gPiBXZSBkbyB1c2UgZHdfcGNpZV9maW5kX2NhcGFiaWxpdHkoKSBpbiBt b3N0IERXQyBkcml2ZXJzIHRvIGZpbmQgYW5kIHBlZWsvcG9rZQo+ID4gYXQgZWcgUENJIGV4cHJl c3MgY2FwYWJpbGl0eSBvZiB0aGUgKlJvb3QgcG9ydCogKD8pLAo+ID4gCj4gPiBlZyBkd19wY2ll X3dhaXRfZm9yX2xpbmsoKQo+ID4gCj4gPiBzbyBJIGFzc3VtZSB0aGF0IGZvciBpTVg2IGR3X3Bj aWVfZmluZF9jYXBhYmlsaXR5KCkgZG9lcyBqdXN0IHRoZSBzYW1lLCB3aGljaAo+ID4gd291bGQg bWVhbiB0aGF0IHdlIGFyZSBwb2tpbmcgdGhlICJNZXNzYWdlIENvbnRyb2wiIGZpZWxkIG9mIHRo ZSBSb290IHBvcnQgTVNJCj4gPiBjYXBhYmlsaXR5Lgo+ID4gCj4gPiBFaXRoZXIgdGhhdCAod2hp Y2ggd291bGQgbWVhbiB0aGF0IGlNWDYgaGFzIGEgSFcgYnVnIGJlY2F1c2UgdGhlIFJQIE1lc3Nh Z2UKPiA+IENvbnRyb2wgZmllbGQgZG9lcyBub3QgY29udHJvbCB0aGUgZGVsaXZlcnkgb2YgTVNJ cyBmcm9tIGVuZHBvaW50cyBidXQganVzdCBmb3IgdGhlCj4gPiByb290IHBvcnQgaXRzZWxmICkg b3IgYWxsIERXQyBjb250cm9sbGVycyBtb2RlbGxlZCB0aGUgcm9vdCBjb21wbGV4IE1NSU8gc3Bh Y2UgYXMKPiA+IGEgc2V0IG9mIFBDSS9QQ0llIGNhcGFiaWxpdGllcyB0aGF0IGFyZSBOT1QgbmVj ZXNzYXJpbHkgbWFwcGFibGUgdG8gUENJCj4gPiBzcGVjaWZpY2F0aW9ucyBkZWZpbmVkIG9uZXMu Cj4gPiAKPiA+IENhbiBhbnlvbmUgcGxlYXNlIHNoZWQgc29tZSBsaWdodCBvbiB0aGlzID8gSSBk b24ndCBoYXZlIERXQyBIVywgd2UgbmVlZCB0bwo+ID4ga25vdyBiZWZvcmUgbWVyZ2luZyB0aGlz IGNvZGUuCj4gSGkgTG9yZW56bzoKPiBSZWdhcmRpbmcgbXkgdW5kZXJzdGFuZGluZywgRFdDIEhX IGhhcyB0aGUgUENJL1BDSWUgY2FwYWJpbGl0eSBtYXAgd2hlbgo+ICBpdCB3b3JrcyBpbiBSQyBt b2RlIGFuZCBTcGVjIGRvZXNu4oCZdCBzcGVjaWZ5IHRoZXNlIENhcHMgZm9yIGhvc3QgY29udHJv bGxlci4KPiBBbmQsIHRoZXJlIGFyZSBjb21tZW50cyBkZXNjcmliZSB0aGVzZSBjYWxsYmFja3Mg YWxyZWFkeSBpbiBwY2llLWRlc2lnbndhcmUuYy4KPiAuLi4KPiAvKgo+ICAqIFRoZXNlIGludGVy ZmFjZXMgcmVzZW1ibGUgdGhlIHBjaV9maW5kXypjYXBhYmlsaXR5KCkgaW50ZXJmYWNlcywgYnV0 IHRoZXNlCj4gICogYXJlIGZvciBjb25maWd1cmluZyBob3N0IGNvbnRyb2xsZXJzLCB3aGljaCBh cmUgYnJpZGdlcyAqdG8qIFBDSSBkZXZpY2VzIGJ1dAo+ICAqIGFyZSBub3QgUENJIGRldmljZXMg dGhlbXNlbHZlcy4KPiAgKi8KPiBzdGF0aWMgdTggX19kd19wY2llX2ZpbmRfbmV4dF9jYXAoc3Ry dWN0IGR3X3BjaWUgKnBjaSwgdTggY2FwX3B0ciwKPiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgdTggY2FwKQo+IC4uLgo+IAo+IAoKSSBzdGlsbCBiZWxpZXZlIHRoaXMgaXMgYW4g aW50ZWdyYXRpb24gYnVnLCBtb3JlIHNvIGFmdGVyIHJlYWRpbmcgdGhlCmNvbW1pdCBTZXJnZSBw b2ludGVkIG91dDoKCjc1Y2I4ZDIwYzExMiAoIlBDSTogaW14OiBFbmFibGUgTVNJIGZyb20gZG93 bnN0cmVhbSBjb21wb25lbnRzIikuCgpUaGUgY29tbWl0IGFib3ZlIGltcGxpZXMgdGhhdCBpZiB5 b3UgaGF2ZSBDT05GSUdfUENJRVBPUlRCVVMgZW5hYmxlZCwKeW91IHdvdWxkIG5vdCBuZWVkIHRv IHNldCB0aGUgTVNJIGVuYWJsZSBiaXQgZXhwbGljaXRseSBiZWNhdXNlIHRoYXQncwpkb25lIGJ5 IHRoZSBwb3J0IGRyaXZlciB3aGlsZSByZXF1ZXN0aW5nIFJQIHNlcnZpY2VzLgoKVGhpcyBtZWFu cyB0aGF0IGl0IGlzIF9zZWVuXyBieSB0aGUgUENJIGNvcmUgYXMgYSBjYXBhYmlsaXR5IHJlZ2lz dGVyCmFuZCBpdCBhbHNvIG1lYW5zIHRoYXQgaWYgeW91IGhhdmUgQ09ORklHX1BDSUVQT1JUQlVT IGVuYWJsZWQgYW5kIHRoYXQKdGhlIHBvcnQgZHJpdmVyIGRpc2FibGVzIE1TSXMsIGFsbCBkb3du c3RyZWFtIE1TSXMgYXJlIGRpc2FibGVkLCBub3QKb25seSB0aGUgUlAgb25lcyAoYXMgaXQgc2hv dWxkIGJlIGFjY29yZGluZyB0byB0aGUgUENJIHNwZWNzKS4KClNvLCB0aGlzIGlzIGEgSFcgYnVn IEkgYW0gYWZyYWlkIC0gSSB3aWxsIG1lcmdlIHRoaXMgcGF0Y2ggYnV0IEFGQUlDUwp0aGUgSFcg aW50ZWdyYXRpb24gYnVnIGlzIHRoZXJlIHJlZ2FyZGxlc3MsIGhvd2V2ZXIgd2Ugc2xpY2UgaXQu CgpUaGFua3MsCkxvcmVuem8KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxA bGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xp c3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK