From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8D2FC43215 for ; Thu, 21 Nov 2019 12:52:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AC0DF20872 for ; Thu, 21 Nov 2019 12:52:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726613AbfKUMwl (ORCPT ); Thu, 21 Nov 2019 07:52:41 -0500 Received: from mga01.intel.com ([192.55.52.88]:49563 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726342AbfKUMwl (ORCPT ); Thu, 21 Nov 2019 07:52:41 -0500 X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Nov 2019 04:52:40 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,224,1571727600"; d="scan'208";a="216129127" Received: from lahna.fi.intel.com (HELO lahna) ([10.237.72.163]) by fmsmga001.fm.intel.com with SMTP; 21 Nov 2019 04:52:36 -0800 Received: by lahna (sSMTP sendmail emulation); Thu, 21 Nov 2019 14:52:36 +0200 Date: Thu, 21 Nov 2019 14:52:36 +0200 From: Mika Westerberg To: "Rafael J. Wysocki" Cc: Karol Herbst , Bjorn Helgaas , LKML , Lyude Paul , "Rafael J . Wysocki" , Linux PCI , Linux PM , dri-devel , nouveau , Dave Airlie , Mario Limonciello Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Message-ID: <20191121125236.GX11621@lahna.fi.intel.com> References: <20191120151542.GH11621@lahna.fi.intel.com> <20191120155301.GL11621@lahna.fi.intel.com> <20191121112821.GU11621@lahna.fi.intel.com> <20191121114610.GW11621@lahna.fi.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191121114610.GW11621@lahna.fi.intel.com> Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo User-Agent: Mutt/1.12.1 (2019-06-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Nov 21, 2019 at 01:46:14PM +0200, Mika Westerberg wrote: > On Thu, Nov 21, 2019 at 12:34:22PM +0100, Rafael J. Wysocki wrote: > > On Thu, Nov 21, 2019 at 12:28 PM Mika Westerberg > > wrote: > > > > > > On Wed, Nov 20, 2019 at 11:29:33PM +0100, Rafael J. Wysocki wrote: > > > > > last week or so I found systems where the GPU was under the "PCI > > > > > Express Root Port" (name from lspci) and on those systems all of that > > > > > seems to work. So I am wondering if it's indeed just the 0x1901 one, > > > > > which also explains Mikas case that Thunderbolt stuff works as devices > > > > > never get populated under this particular bridge controller, but under > > > > > those "Root Port"s > > > > > > > > It always is a PCIe port, but its location within the SoC may matter. > > > > > > Exactly. Intel hardware has PCIe ports on CPU side (these are called > > > PEG, PCI Express Graphics, ports), and the PCH side. I think the IP is > > > still the same. > > > > > > > Also some custom AML-based power management is involved and that may > > > > be making specific assumptions on the configuration of the SoC and the > > > > GPU at the time of its invocation which unfortunately are not known to > > > > us. > > > > > > > > However, it looks like the AML invoked to power down the GPU from > > > > acpi_pci_set_power_state() gets confused if it is not in PCI D0 at > > > > that point, so it looks like that AML tries to access device memory on > > > > the GPU (beyond the PCI config space) or similar which is not > > > > accessible in PCI power states below D0. > > > > > > Or the PCI config space of the GPU when the parent root port is in D3hot > > > (as it is the case here). Also then the GPU config space is not > > > accessible. > > > > Why would the parent port be in D3hot at that point? Wouldn't that be > > a suspend ordering violation? > > No. We put the GPU into D3hot first, then the root port and then turn > off the power resource (which is attached to the root port) resulting > the topology entering D3cold. I don't see that happening in the AML though. Basically the difference is that when Windows 7 or Linux (the _REV==5 check) then we directly do link disable whereas in Windows 8+ we invoke LKDS() method that puts the link into L2/L3. None of the fields they access seem to touch the GPU itself. LKDS() for the first PEG port looks like this: P0L2 = One Sleep (0x10) Local0 = Zero While (P0L2) { If ((Local0 > 0x04)) { Break } Sleep (0x10) Local0++ } One thing that comes to mind is that the loop can end even if P0L2 is not cleared as it does only 5 iterations with 16 ms sleep between. Maybe Sleep() is implemented differently in Windows? I mean Linux may be "faster" here and return prematurely and if we leave the port into D0 this does not happen, or something. I'm just throwing out ideas :) From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mika Westerberg Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Date: Thu, 21 Nov 2019 14:52:36 +0200 Message-ID: <20191121125236.GX11621@lahna.fi.intel.com> References: <20191120151542.GH11621@lahna.fi.intel.com> <20191120155301.GL11621@lahna.fi.intel.com> <20191121112821.GU11621@lahna.fi.intel.com> <20191121114610.GW11621@lahna.fi.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <20191121114610.GW11621@lahna.fi.intel.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: "Rafael J. Wysocki" Cc: Karol Herbst , Linux PM , Linux PCI , Mario Limonciello , "Rafael J . Wysocki" , LKML , dri-devel , Bjorn Helgaas , nouveau List-Id: nouveau.vger.kernel.org T24gVGh1LCBOb3YgMjEsIDIwMTkgYXQgMDE6NDY6MTRQTSArMDIwMCwgTWlrYSBXZXN0ZXJiZXJn IHdyb3RlOgo+IE9uIFRodSwgTm92IDIxLCAyMDE5IGF0IDEyOjM0OjIyUE0gKzAxMDAsIFJhZmFl bCBKLiBXeXNvY2tpIHdyb3RlOgo+ID4gT24gVGh1LCBOb3YgMjEsIDIwMTkgYXQgMTI6MjggUE0g TWlrYSBXZXN0ZXJiZXJnCj4gPiA8bWlrYS53ZXN0ZXJiZXJnQGludGVsLmNvbT4gd3JvdGU6Cj4g PiA+Cj4gPiA+IE9uIFdlZCwgTm92IDIwLCAyMDE5IGF0IDExOjI5OjMzUE0gKzAxMDAsIFJhZmFl bCBKLiBXeXNvY2tpIHdyb3RlOgo+ID4gPiA+ID4gbGFzdCB3ZWVrIG9yIHNvIEkgZm91bmQgc3lz dGVtcyB3aGVyZSB0aGUgR1BVIHdhcyB1bmRlciB0aGUgIlBDSQo+ID4gPiA+ID4gRXhwcmVzcyBS b290IFBvcnQiIChuYW1lIGZyb20gbHNwY2kpIGFuZCBvbiB0aG9zZSBzeXN0ZW1zIGFsbCBvZiB0 aGF0Cj4gPiA+ID4gPiBzZWVtcyB0byB3b3JrLiBTbyBJIGFtIHdvbmRlcmluZyBpZiBpdCdzIGlu ZGVlZCBqdXN0IHRoZSAweDE5MDEgb25lLAo+ID4gPiA+ID4gd2hpY2ggYWxzbyBleHBsYWlucyBN aWthcyBjYXNlIHRoYXQgVGh1bmRlcmJvbHQgc3R1ZmYgd29ya3MgYXMgZGV2aWNlcwo+ID4gPiA+ ID4gbmV2ZXIgZ2V0IHBvcHVsYXRlZCB1bmRlciB0aGlzIHBhcnRpY3VsYXIgYnJpZGdlIGNvbnRy b2xsZXIsIGJ1dCB1bmRlcgo+ID4gPiA+ID4gdGhvc2UgIlJvb3QgUG9ydCJzCj4gPiA+ID4KPiA+ ID4gPiBJdCBhbHdheXMgaXMgYSBQQ0llIHBvcnQsIGJ1dCBpdHMgbG9jYXRpb24gd2l0aGluIHRo ZSBTb0MgbWF5IG1hdHRlci4KPiA+ID4KPiA+ID4gRXhhY3RseS4gSW50ZWwgaGFyZHdhcmUgaGFz IFBDSWUgcG9ydHMgb24gQ1BVIHNpZGUgKHRoZXNlIGFyZSBjYWxsZWQKPiA+ID4gUEVHLCBQQ0kg RXhwcmVzcyBHcmFwaGljcywgcG9ydHMpLCBhbmQgdGhlIFBDSCBzaWRlLiBJIHRoaW5rIHRoZSBJ UCBpcwo+ID4gPiBzdGlsbCB0aGUgc2FtZS4KPiA+ID4KPiA+ID4gPiBBbHNvIHNvbWUgY3VzdG9t IEFNTC1iYXNlZCBwb3dlciBtYW5hZ2VtZW50IGlzIGludm9sdmVkIGFuZCB0aGF0IG1heQo+ID4g PiA+IGJlIG1ha2luZyBzcGVjaWZpYyBhc3N1bXB0aW9ucyBvbiB0aGUgY29uZmlndXJhdGlvbiBv ZiB0aGUgU29DIGFuZCB0aGUKPiA+ID4gPiBHUFUgYXQgdGhlIHRpbWUgb2YgaXRzIGludm9jYXRp b24gd2hpY2ggdW5mb3J0dW5hdGVseSBhcmUgbm90IGtub3duIHRvCj4gPiA+ID4gdXMuCj4gPiA+ ID4KPiA+ID4gPiBIb3dldmVyLCBpdCBsb29rcyBsaWtlIHRoZSBBTUwgaW52b2tlZCB0byBwb3dl ciBkb3duIHRoZSBHUFUgZnJvbQo+ID4gPiA+IGFjcGlfcGNpX3NldF9wb3dlcl9zdGF0ZSgpIGdl dHMgY29uZnVzZWQgaWYgaXQgaXMgbm90IGluIFBDSSBEMCBhdAo+ID4gPiA+IHRoYXQgcG9pbnQs IHNvIGl0IGxvb2tzIGxpa2UgdGhhdCBBTUwgdHJpZXMgdG8gYWNjZXNzIGRldmljZSBtZW1vcnkg b24KPiA+ID4gPiB0aGUgR1BVIChiZXlvbmQgdGhlIFBDSSBjb25maWcgc3BhY2UpIG9yIHNpbWls YXIgd2hpY2ggaXMgbm90Cj4gPiA+ID4gYWNjZXNzaWJsZSBpbiBQQ0kgcG93ZXIgc3RhdGVzIGJl bG93IEQwLgo+ID4gPgo+ID4gPiBPciB0aGUgUENJIGNvbmZpZyBzcGFjZSBvZiB0aGUgR1BVIHdo ZW4gdGhlIHBhcmVudCByb290IHBvcnQgaXMgaW4gRDNob3QKPiA+ID4gKGFzIGl0IGlzIHRoZSBj YXNlIGhlcmUpLiBBbHNvIHRoZW4gdGhlIEdQVSBjb25maWcgc3BhY2UgaXMgbm90Cj4gPiA+IGFj Y2Vzc2libGUuCj4gPiAKPiA+IFdoeSB3b3VsZCB0aGUgcGFyZW50IHBvcnQgYmUgaW4gRDNob3Qg YXQgdGhhdCBwb2ludD8gIFdvdWxkbid0IHRoYXQgYmUKPiA+IGEgc3VzcGVuZCBvcmRlcmluZyB2 aW9sYXRpb24/Cj4gCj4gTm8uIFdlIHB1dCB0aGUgR1BVIGludG8gRDNob3QgZmlyc3QsIHRoZW4g dGhlIHJvb3QgcG9ydCBhbmQgdGhlbiB0dXJuCj4gb2ZmIHRoZSBwb3dlciByZXNvdXJjZSAod2hp Y2ggaXMgYXR0YWNoZWQgdG8gdGhlIHJvb3QgcG9ydCkgcmVzdWx0aW5nCj4gdGhlIHRvcG9sb2d5 IGVudGVyaW5nIEQzY29sZC4KCkkgZG9uJ3Qgc2VlIHRoYXQgaGFwcGVuaW5nIGluIHRoZSBBTUwg dGhvdWdoLgoKQmFzaWNhbGx5IHRoZSBkaWZmZXJlbmNlIGlzIHRoYXQgd2hlbiBXaW5kb3dzIDcg b3IgTGludXggKHRoZSBfUkVWPT01CmNoZWNrKSB0aGVuIHdlIGRpcmVjdGx5IGRvIGxpbmsgZGlz YWJsZSB3aGVyZWFzIGluIFdpbmRvd3MgOCsgd2UgaW52b2tlCkxLRFMoKSBtZXRob2QgdGhhdCBw dXRzIHRoZSBsaW5rIGludG8gTDIvTDMuIE5vbmUgb2YgdGhlIGZpZWxkcyB0aGV5CmFjY2VzcyBz ZWVtIHRvIHRvdWNoIHRoZSBHUFUgaXRzZWxmLgoKTEtEUygpIGZvciB0aGUgZmlyc3QgUEVHIHBv cnQgbG9va3MgbGlrZSB0aGlzOgoKICAgUDBMMiA9IE9uZQogICBTbGVlcCAoMHgxMCkKICAgTG9j YWwwID0gWmVybwogICBXaGlsZSAoUDBMMikKICAgewoJSWYgKChMb2NhbDAgPiAweDA0KSkKCXsK CSAgICBCcmVhawoJfQoKCVNsZWVwICgweDEwKQoJTG9jYWwwKysKICAgfQoKT25lIHRoaW5nIHRo YXQgY29tZXMgdG8gbWluZCBpcyB0aGF0IHRoZSBsb29wIGNhbiBlbmQgZXZlbiBpZiBQMEwyIGlz Cm5vdCBjbGVhcmVkIGFzIGl0IGRvZXMgb25seSA1IGl0ZXJhdGlvbnMgd2l0aCAxNiBtcyBzbGVl cCBiZXR3ZWVuLiBNYXliZQpTbGVlcCgpIGlzIGltcGxlbWVudGVkIGRpZmZlcmVudGx5IGluIFdp bmRvd3M/IEkgbWVhbiBMaW51eCBtYXkgYmUKImZhc3RlciIgaGVyZSBhbmQgcmV0dXJuIHByZW1h dHVyZWx5IGFuZCBpZiB3ZSBsZWF2ZSB0aGUgcG9ydCBpbnRvIEQwCnRoaXMgZG9lcyBub3QgaGFw cGVuLCBvciBzb21ldGhpbmcuIEknbSBqdXN0IHRocm93aW5nIG91dCBpZGVhcyA6KQpfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGlu ZyBsaXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVl ZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWw=