From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mika Westerberg Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Date: Thu, 21 Nov 2019 13:46:10 +0200 Message-ID: <20191121114610.GW11621@lahna.fi.intel.com> References: <20191120120913.GE11621@lahna.fi.intel.com> <20191120151542.GH11621@lahna.fi.intel.com> <20191120155301.GL11621@lahna.fi.intel.com> <20191121112821.GU11621@lahna.fi.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: "Rafael J. Wysocki" Cc: Karol Herbst , Bjorn Helgaas , LKML , Lyude Paul , "Rafael J . Wysocki" , Linux PCI , Linux PM , dri-devel , nouveau , Dave Airlie , Mario Limonciello List-Id: dri-devel@lists.freedesktop.org On Thu, Nov 21, 2019 at 12:34:22PM +0100, Rafael J. Wysocki wrote: > On Thu, Nov 21, 2019 at 12:28 PM Mika Westerberg > wrote: > > > > On Wed, Nov 20, 2019 at 11:29:33PM +0100, Rafael J. Wysocki wrote: > > > > last week or so I found systems where the GPU was under the "PCI > > > > Express Root Port" (name from lspci) and on those systems all of that > > > > seems to work. So I am wondering if it's indeed just the 0x1901 one, > > > > which also explains Mikas case that Thunderbolt stuff works as devices > > > > never get populated under this particular bridge controller, but under > > > > those "Root Port"s > > > > > > It always is a PCIe port, but its location within the SoC may matter. > > > > Exactly. Intel hardware has PCIe ports on CPU side (these are called > > PEG, PCI Express Graphics, ports), and the PCH side. I think the IP is > > still the same. > > > > > Also some custom AML-based power management is involved and that may > > > be making specific assumptions on the configuration of the SoC and the > > > GPU at the time of its invocation which unfortunately are not known to > > > us. > > > > > > However, it looks like the AML invoked to power down the GPU from > > > acpi_pci_set_power_state() gets confused if it is not in PCI D0 at > > > that point, so it looks like that AML tries to access device memory on > > > the GPU (beyond the PCI config space) or similar which is not > > > accessible in PCI power states below D0. > > > > Or the PCI config space of the GPU when the parent root port is in D3hot > > (as it is the case here). Also then the GPU config space is not > > accessible. > > Why would the parent port be in D3hot at that point? Wouldn't that be > a suspend ordering violation? No. We put the GPU into D3hot first, then the root port and then turn off the power resource (which is attached to the root port) resulting the topology entering D3cold. > > I took a look at the HP Omen ACPI tables which has similar problem and > > there is also check for Windows 7 (but not Linux) so I think one > > alternative workaround would be to add these devices into > > acpi_osi_dmi_table[] where .callback is set to dmi_disable_osi_win8 (or > > pass 'acpi_osi="!Windows 2012"' in the kernel command line). > > I'd like to understand the facts that have been established so far > before deciding what to do about them. :-) Yes, I agree :) From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89261C432C3 for ; Thu, 21 Nov 2019 11:46:17 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 68C37208A3 for ; Thu, 21 Nov 2019 11:46:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 68C37208A3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 992496EE54; Thu, 21 Nov 2019 11:46:16 +0000 (UTC) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3210D6EE13; Thu, 21 Nov 2019 11:46:15 +0000 (UTC) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Nov 2019 03:46:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,224,1571727600"; d="scan'208";a="216119105" Received: from lahna.fi.intel.com (HELO lahna) ([10.237.72.163]) by fmsmga001.fm.intel.com with SMTP; 21 Nov 2019 03:46:10 -0800 Received: by lahna (sSMTP sendmail emulation); Thu, 21 Nov 2019 13:46:10 +0200 Date: Thu, 21 Nov 2019 13:46:10 +0200 From: Mika Westerberg To: "Rafael J. Wysocki" Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Message-ID: <20191121114610.GW11621@lahna.fi.intel.com> References: <20191120120913.GE11621@lahna.fi.intel.com> <20191120151542.GH11621@lahna.fi.intel.com> <20191120155301.GL11621@lahna.fi.intel.com> <20191121112821.GU11621@lahna.fi.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo User-Agent: Mutt/1.12.1 (2019-06-15) X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Karol Herbst , Linux PM , Linux PCI , Mario Limonciello , "Rafael J . Wysocki" , LKML , dri-devel , Bjorn Helgaas , nouveau Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Message-ID: <20191121114610.4pej9h1acfgIR4ssN2d-E1gUuMfVflmF0-rrgCUQvRo@z> T24gVGh1LCBOb3YgMjEsIDIwMTkgYXQgMTI6MzQ6MjJQTSArMDEwMCwgUmFmYWVsIEouIFd5c29j a2kgd3JvdGU6Cj4gT24gVGh1LCBOb3YgMjEsIDIwMTkgYXQgMTI6MjggUE0gTWlrYSBXZXN0ZXJi ZXJnCj4gPG1pa2Eud2VzdGVyYmVyZ0BpbnRlbC5jb20+IHdyb3RlOgo+ID4KPiA+IE9uIFdlZCwg Tm92IDIwLCAyMDE5IGF0IDExOjI5OjMzUE0gKzAxMDAsIFJhZmFlbCBKLiBXeXNvY2tpIHdyb3Rl Ogo+ID4gPiA+IGxhc3Qgd2VlayBvciBzbyBJIGZvdW5kIHN5c3RlbXMgd2hlcmUgdGhlIEdQVSB3 YXMgdW5kZXIgdGhlICJQQ0kKPiA+ID4gPiBFeHByZXNzIFJvb3QgUG9ydCIgKG5hbWUgZnJvbSBs c3BjaSkgYW5kIG9uIHRob3NlIHN5c3RlbXMgYWxsIG9mIHRoYXQKPiA+ID4gPiBzZWVtcyB0byB3 b3JrLiBTbyBJIGFtIHdvbmRlcmluZyBpZiBpdCdzIGluZGVlZCBqdXN0IHRoZSAweDE5MDEgb25l LAo+ID4gPiA+IHdoaWNoIGFsc28gZXhwbGFpbnMgTWlrYXMgY2FzZSB0aGF0IFRodW5kZXJib2x0 IHN0dWZmIHdvcmtzIGFzIGRldmljZXMKPiA+ID4gPiBuZXZlciBnZXQgcG9wdWxhdGVkIHVuZGVy IHRoaXMgcGFydGljdWxhciBicmlkZ2UgY29udHJvbGxlciwgYnV0IHVuZGVyCj4gPiA+ID4gdGhv c2UgIlJvb3QgUG9ydCJzCj4gPiA+Cj4gPiA+IEl0IGFsd2F5cyBpcyBhIFBDSWUgcG9ydCwgYnV0 IGl0cyBsb2NhdGlvbiB3aXRoaW4gdGhlIFNvQyBtYXkgbWF0dGVyLgo+ID4KPiA+IEV4YWN0bHku IEludGVsIGhhcmR3YXJlIGhhcyBQQ0llIHBvcnRzIG9uIENQVSBzaWRlICh0aGVzZSBhcmUgY2Fs bGVkCj4gPiBQRUcsIFBDSSBFeHByZXNzIEdyYXBoaWNzLCBwb3J0cyksIGFuZCB0aGUgUENIIHNp ZGUuIEkgdGhpbmsgdGhlIElQIGlzCj4gPiBzdGlsbCB0aGUgc2FtZS4KPiA+Cj4gPiA+IEFsc28g c29tZSBjdXN0b20gQU1MLWJhc2VkIHBvd2VyIG1hbmFnZW1lbnQgaXMgaW52b2x2ZWQgYW5kIHRo YXQgbWF5Cj4gPiA+IGJlIG1ha2luZyBzcGVjaWZpYyBhc3N1bXB0aW9ucyBvbiB0aGUgY29uZmln dXJhdGlvbiBvZiB0aGUgU29DIGFuZCB0aGUKPiA+ID4gR1BVIGF0IHRoZSB0aW1lIG9mIGl0cyBp bnZvY2F0aW9uIHdoaWNoIHVuZm9ydHVuYXRlbHkgYXJlIG5vdCBrbm93biB0bwo+ID4gPiB1cy4K PiA+ID4KPiA+ID4gSG93ZXZlciwgaXQgbG9va3MgbGlrZSB0aGUgQU1MIGludm9rZWQgdG8gcG93 ZXIgZG93biB0aGUgR1BVIGZyb20KPiA+ID4gYWNwaV9wY2lfc2V0X3Bvd2VyX3N0YXRlKCkgZ2V0 cyBjb25mdXNlZCBpZiBpdCBpcyBub3QgaW4gUENJIEQwIGF0Cj4gPiA+IHRoYXQgcG9pbnQsIHNv IGl0IGxvb2tzIGxpa2UgdGhhdCBBTUwgdHJpZXMgdG8gYWNjZXNzIGRldmljZSBtZW1vcnkgb24K PiA+ID4gdGhlIEdQVSAoYmV5b25kIHRoZSBQQ0kgY29uZmlnIHNwYWNlKSBvciBzaW1pbGFyIHdo aWNoIGlzIG5vdAo+ID4gPiBhY2Nlc3NpYmxlIGluIFBDSSBwb3dlciBzdGF0ZXMgYmVsb3cgRDAu Cj4gPgo+ID4gT3IgdGhlIFBDSSBjb25maWcgc3BhY2Ugb2YgdGhlIEdQVSB3aGVuIHRoZSBwYXJl bnQgcm9vdCBwb3J0IGlzIGluIEQzaG90Cj4gPiAoYXMgaXQgaXMgdGhlIGNhc2UgaGVyZSkuIEFs c28gdGhlbiB0aGUgR1BVIGNvbmZpZyBzcGFjZSBpcyBub3QKPiA+IGFjY2Vzc2libGUuCj4gCj4g V2h5IHdvdWxkIHRoZSBwYXJlbnQgcG9ydCBiZSBpbiBEM2hvdCBhdCB0aGF0IHBvaW50PyAgV291 bGRuJ3QgdGhhdCBiZQo+IGEgc3VzcGVuZCBvcmRlcmluZyB2aW9sYXRpb24/CgpOby4gV2UgcHV0 IHRoZSBHUFUgaW50byBEM2hvdCBmaXJzdCwgdGhlbiB0aGUgcm9vdCBwb3J0IGFuZCB0aGVuIHR1 cm4Kb2ZmIHRoZSBwb3dlciByZXNvdXJjZSAod2hpY2ggaXMgYXR0YWNoZWQgdG8gdGhlIHJvb3Qg cG9ydCkgcmVzdWx0aW5nCnRoZSB0b3BvbG9neSBlbnRlcmluZyBEM2NvbGQuCgo+ID4gSSB0b29r IGEgbG9vayBhdCB0aGUgSFAgT21lbiBBQ1BJIHRhYmxlcyB3aGljaCBoYXMgc2ltaWxhciBwcm9i bGVtIGFuZAo+ID4gdGhlcmUgaXMgYWxzbyBjaGVjayBmb3IgV2luZG93cyA3IChidXQgbm90IExp bnV4KSBzbyBJIHRoaW5rIG9uZQo+ID4gYWx0ZXJuYXRpdmUgd29ya2Fyb3VuZCB3b3VsZCBiZSB0 byBhZGQgdGhlc2UgZGV2aWNlcyBpbnRvCj4gPiBhY3BpX29zaV9kbWlfdGFibGVbXSB3aGVyZSAu Y2FsbGJhY2sgaXMgc2V0IHRvIGRtaV9kaXNhYmxlX29zaV93aW44IChvcgo+ID4gcGFzcyAnYWNw aV9vc2k9IiFXaW5kb3dzIDIwMTIiJyBpbiB0aGUga2VybmVsIGNvbW1hbmQgbGluZSkuCj4gCj4g SSdkIGxpa2UgdG8gdW5kZXJzdGFuZCB0aGUgZmFjdHMgdGhhdCBoYXZlIGJlZW4gZXN0YWJsaXNo ZWQgc28gZmFyCj4gYmVmb3JlIGRlY2lkaW5nIHdoYXQgdG8gZG8gYWJvdXQgdGhlbS4gOi0pCgpZ ZXMsIEkgYWdyZWUgOikKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Au b3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRl dmVs