From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Rafael J. Wysocki" Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Date: Wed, 20 Nov 2019 23:29:33 +0100 Message-ID: References: <20191120101816.GX11621@lahna.fi.intel.com> <20191120112212.GA11621@lahna.fi.intel.com> <20191120115127.GD11621@lahna.fi.intel.com> <20191120120913.GE11621@lahna.fi.intel.com> <20191120151542.GH11621@lahna.fi.intel.com> <20191120155301.GL11621@lahna.fi.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Karol Herbst Cc: "Rafael J. Wysocki" , Mika Westerberg , Bjorn Helgaas , LKML , Lyude Paul , "Rafael J . Wysocki" , Linux PCI , Linux PM , dri-devel , nouveau , Dave Airlie , Mario Limonciello List-Id: dri-devel@lists.freedesktop.org On Wed, Nov 20, 2019 at 10:40 PM Karol Herbst wrote: > > On Wed, Nov 20, 2019 at 10:37 PM Rafael J. Wysocki wrote: > > > > On Wed, Nov 20, 2019 at 4:53 PM Mika Westerberg > > wrote: > > > > > > On Wed, Nov 20, 2019 at 04:37:14PM +0100, Karol Herbst wrote: > > > > On Wed, Nov 20, 2019 at 4:15 PM Mika Westerberg > > > > wrote: > > > > > > > > > > On Wed, Nov 20, 2019 at 01:11:52PM +0100, Karol Herbst wrote: > > > > > > On Wed, Nov 20, 2019 at 1:09 PM Mika Westerberg > > > > > > wrote: > > > > > > > > > > > > > > On Wed, Nov 20, 2019 at 12:58:00PM +0100, Karol Herbst wrote: > > > > > > > > overall, what I really want to know is, _why_ does it work on windows? > > > > > > > > > > > > > > So do I ;-) > > > > > > > > > > > > > > > Or what are we doing differently on Linux so that it doesn't work? If > > > > > > > > anybody has any idea on how we could dig into this and figure it out > > > > > > > > on this level, this would probably allow us to get closer to the root > > > > > > > > cause? no? > > > > > > > > > > > > > > Have you tried to use the acpi_rev_override parameter in your system and > > > > > > > does it have any effect? > > > > > > > > > > > > > > Also did you try to trace the ACPI _ON/_OFF() methods? I think that > > > > > > > should hopefully reveal something. > > > > > > > > > > > > > > > > > > > I think I did in the past and it seemed to have worked, there is just > > > > > > one big issue with this: it's a Dell specific workaround afaik, and > > > > > > this issue plagues not just Dell, but we've seen it on HP and Lenovo > > > > > > laptops as well, and I've heard about users having the same issues on > > > > > > Asus and MSI laptops as well. > > > > > > > > > > Maybe it is not a workaround at all but instead it simply determines > > > > > whether the system supports RTD3 or something like that (IIRC Windows 8 > > > > > started supporting it). Maybe Dell added check for Linux because at that > > > > > time Linux did not support it. > > > > > > > > > > > > > the point is, it's not checking it by default, so by default you still > > > > run into the windows 8 codepath. > > > > > > Well you can add the quirk to acpi_rev_dmi_table[] so it goes to that > > > path by default. There are a bunch of similar entries for Dell machines. > > > > OK, so the "Linux path" works and the other doesn't. > > > > I thought that this was the other way around, sorry for the confusion. > > > > > Of course this does not help the non-Dell users so we would still need > > > to figure out the root cause. > > > > Right. > > > > Whatever it is, though, AML appears to be involved in it and AFAICS > > there's no evidence that it affects any root ports that are not > > populated with NVidia GPUs. > > > > last week or so I found systems where the GPU was under the "PCI > Express Root Port" (name from lspci) and on those systems all of that > seems to work. So I am wondering if it's indeed just the 0x1901 one, > which also explains Mikas case that Thunderbolt stuff works as devices > never get populated under this particular bridge controller, but under > those "Root Port"s It always is a PCIe port, but its location within the SoC may matter. Also some custom AML-based power management is involved and that may be making specific assumptions on the configuration of the SoC and the GPU at the time of its invocation which unfortunately are not known to us. However, it looks like the AML invoked to power down the GPU from acpi_pci_set_power_state() gets confused if it is not in PCI D0 at that point, so it looks like that AML tries to access device memory on the GPU (beyond the PCI config space) or similar which is not accessible in PCI power states below D0. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACE8AC432C0 for ; Wed, 20 Nov 2019 22:29:47 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8A71B20872 for ; Wed, 20 Nov 2019 22:29:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8A71B20872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 214E36EA5D; Wed, 20 Nov 2019 22:29:47 +0000 (UTC) Received: from mail-ot1-f68.google.com (mail-ot1-f68.google.com [209.85.210.68]) by gabe.freedesktop.org (Postfix) with ESMTPS id E020D6EA5D; Wed, 20 Nov 2019 22:29:45 +0000 (UTC) Received: by mail-ot1-f68.google.com with SMTP id m15so1105163otq.7; Wed, 20 Nov 2019 14:29:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=DGzPyxLr5Ay6yfeJ4SIhL2wKpa5Cl6ARb+65yasor8Y=; b=gWVI+kl9SSYV4c0EEb9MGNK+AEAX2mV1Lz01517/RZvY5rcWIPpXDQqH5PXkQosh6C ifWai8r0A0Cr35DS9A1MpnDbjGygUI37frWfQrrIJ1t5enZgTTGjC3nno/cArlI4Qc5m G8M3USO0OSDp0K6lORDK1G//L4NMzx98UQsjRlMJDT9lW9csu6A8nlHHZ5XcPD+2+RoZ eKZdIV7W33y5aM1/gN6HAgl5I/Tcd2D2yrXb24le7v26gV+FQO5ngPI8v1FP0P2LmmBD /UtPpMxq2GJxRivoFkm9kttl71Sb2xCAjPj/5z7O4C9DNJnTY2X+ojFf1WxE8crWfKYu l5cA== X-Gm-Message-State: APjAAAVt28xLXvILvb3YOOX2pC3vznPZeH03lypFzmp/Wkp0kYEXh6GN JF2WYrx0xDhJpQv2PF7FSRhgpRLREvmPGY5Sw9HYVMdC X-Google-Smtp-Source: APXvYqy+5B1uMvfcEzRSenoFSNtJMDeccEusFuvVsseAsLdM74Z0KG8lam60Ak6v61O2xJDRNnmZ4TlH+5v6N8Y9Z1k= X-Received: by 2002:a05:6830:232a:: with SMTP id q10mr4003045otg.262.1574288985011; Wed, 20 Nov 2019 14:29:45 -0800 (PST) MIME-Version: 1.0 References: <20191120101816.GX11621@lahna.fi.intel.com> <20191120112212.GA11621@lahna.fi.intel.com> <20191120115127.GD11621@lahna.fi.intel.com> <20191120120913.GE11621@lahna.fi.intel.com> <20191120151542.GH11621@lahna.fi.intel.com> <20191120155301.GL11621@lahna.fi.intel.com> In-Reply-To: From: "Rafael J. Wysocki" Date: Wed, 20 Nov 2019 23:29:33 +0100 Message-ID: Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges To: Karol Herbst X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Rafael J. Wysocki" , Linux PCI , Mika Westerberg , Linux PM , "Rafael J . Wysocki" , LKML , dri-devel , Mario Limonciello , Bjorn Helgaas , nouveau Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Message-ID: <20191120222933.-udQMX3KjZ6SOvoyPN4CFAmKvjotOkLszXkZd6YUVAs@z> T24gV2VkLCBOb3YgMjAsIDIwMTkgYXQgMTA6NDAgUE0gS2Fyb2wgSGVyYnN0IDxraGVyYnN0QHJl ZGhhdC5jb20+IHdyb3RlOgo+Cj4gT24gV2VkLCBOb3YgMjAsIDIwMTkgYXQgMTA6MzcgUE0gUmFm YWVsIEouIFd5c29ja2kgPHJhZmFlbEBrZXJuZWwub3JnPiB3cm90ZToKPiA+Cj4gPiBPbiBXZWQs IE5vdiAyMCwgMjAxOSBhdCA0OjUzIFBNIE1pa2EgV2VzdGVyYmVyZwo+ID4gPG1pa2Eud2VzdGVy YmVyZ0BpbnRlbC5jb20+IHdyb3RlOgo+ID4gPgo+ID4gPiBPbiBXZWQsIE5vdiAyMCwgMjAxOSBh dCAwNDozNzoxNFBNICswMTAwLCBLYXJvbCBIZXJic3Qgd3JvdGU6Cj4gPiA+ID4gT24gV2VkLCBO b3YgMjAsIDIwMTkgYXQgNDoxNSBQTSBNaWthIFdlc3RlcmJlcmcKPiA+ID4gPiA8bWlrYS53ZXN0 ZXJiZXJnQGludGVsLmNvbT4gd3JvdGU6Cj4gPiA+ID4gPgo+ID4gPiA+ID4gT24gV2VkLCBOb3Yg MjAsIDIwMTkgYXQgMDE6MTE6NTJQTSArMDEwMCwgS2Fyb2wgSGVyYnN0IHdyb3RlOgo+ID4gPiA+ ID4gPiBPbiBXZWQsIE5vdiAyMCwgMjAxOSBhdCAxOjA5IFBNIE1pa2EgV2VzdGVyYmVyZwo+ID4g PiA+ID4gPiA8bWlrYS53ZXN0ZXJiZXJnQGludGVsLmNvbT4gd3JvdGU6Cj4gPiA+ID4gPiA+ID4K PiA+ID4gPiA+ID4gPiBPbiBXZWQsIE5vdiAyMCwgMjAxOSBhdCAxMjo1ODowMFBNICswMTAwLCBL YXJvbCBIZXJic3Qgd3JvdGU6Cj4gPiA+ID4gPiA+ID4gPiBvdmVyYWxsLCB3aGF0IEkgcmVhbGx5 IHdhbnQgdG8ga25vdyBpcywgX3doeV8gZG9lcyBpdCB3b3JrIG9uIHdpbmRvd3M/Cj4gPiA+ID4g PiA+ID4KPiA+ID4gPiA+ID4gPiBTbyBkbyBJIDstKQo+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ ID4gPiBPciB3aGF0IGFyZSB3ZSBkb2luZyBkaWZmZXJlbnRseSBvbiBMaW51eCBzbyB0aGF0IGl0 IGRvZXNuJ3Qgd29yaz8gSWYKPiA+ID4gPiA+ID4gPiA+IGFueWJvZHkgaGFzIGFueSBpZGVhIG9u IGhvdyB3ZSBjb3VsZCBkaWcgaW50byB0aGlzIGFuZCBmaWd1cmUgaXQgb3V0Cj4gPiA+ID4gPiA+ ID4gPiBvbiB0aGlzIGxldmVsLCB0aGlzIHdvdWxkIHByb2JhYmx5IGFsbG93IHVzIHRvIGdldCBj bG9zZXIgdG8gdGhlIHJvb3QKPiA+ID4gPiA+ID4gPiA+IGNhdXNlPyBubz8KPiA+ID4gPiA+ID4g Pgo+ID4gPiA+ID4gPiA+IEhhdmUgeW91IHRyaWVkIHRvIHVzZSB0aGUgYWNwaV9yZXZfb3ZlcnJp ZGUgcGFyYW1ldGVyIGluIHlvdXIgc3lzdGVtIGFuZAo+ID4gPiA+ID4gPiA+IGRvZXMgaXQgaGF2 ZSBhbnkgZWZmZWN0Pwo+ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gQWxzbyBkaWQgeW91IHRy eSB0byB0cmFjZSB0aGUgQUNQSSBfT04vX09GRigpIG1ldGhvZHM/IEkgdGhpbmsgdGhhdAo+ID4g PiA+ID4gPiA+IHNob3VsZCBob3BlZnVsbHkgcmV2ZWFsIHNvbWV0aGluZy4KPiA+ID4gPiA+ID4g Pgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiBJIHRoaW5rIEkgZGlkIGluIHRoZSBwYXN0IGFuZCBp dCBzZWVtZWQgdG8gaGF2ZSB3b3JrZWQsIHRoZXJlIGlzIGp1c3QKPiA+ID4gPiA+ID4gb25lIGJp ZyBpc3N1ZSB3aXRoIHRoaXM6IGl0J3MgYSBEZWxsIHNwZWNpZmljIHdvcmthcm91bmQgYWZhaWss IGFuZAo+ID4gPiA+ID4gPiB0aGlzIGlzc3VlIHBsYWd1ZXMgbm90IGp1c3QgRGVsbCwgYnV0IHdl J3ZlIHNlZW4gaXQgb24gSFAgYW5kIExlbm92bwo+ID4gPiA+ID4gPiBsYXB0b3BzIGFzIHdlbGws IGFuZCBJJ3ZlIGhlYXJkIGFib3V0IHVzZXJzIGhhdmluZyB0aGUgc2FtZSBpc3N1ZXMgb24KPiA+ ID4gPiA+ID4gQXN1cyBhbmQgTVNJIGxhcHRvcHMgYXMgd2VsbC4KPiA+ID4gPiA+Cj4gPiA+ID4g PiBNYXliZSBpdCBpcyBub3QgYSB3b3JrYXJvdW5kIGF0IGFsbCBidXQgaW5zdGVhZCBpdCBzaW1w bHkgZGV0ZXJtaW5lcwo+ID4gPiA+ID4gd2hldGhlciB0aGUgc3lzdGVtIHN1cHBvcnRzIFJURDMg b3Igc29tZXRoaW5nIGxpa2UgdGhhdCAoSUlSQyBXaW5kb3dzIDgKPiA+ID4gPiA+IHN0YXJ0ZWQg c3VwcG9ydGluZyBpdCkuIE1heWJlIERlbGwgYWRkZWQgY2hlY2sgZm9yIExpbnV4IGJlY2F1c2Ug YXQgdGhhdAo+ID4gPiA+ID4gdGltZSBMaW51eCBkaWQgbm90IHN1cHBvcnQgaXQuCj4gPiA+ID4g Pgo+ID4gPiA+Cj4gPiA+ID4gdGhlIHBvaW50IGlzLCBpdCdzIG5vdCBjaGVja2luZyBpdCBieSBk ZWZhdWx0LCBzbyBieSBkZWZhdWx0IHlvdSBzdGlsbAo+ID4gPiA+IHJ1biBpbnRvIHRoZSB3aW5k b3dzIDggY29kZXBhdGguCj4gPiA+Cj4gPiA+IFdlbGwgeW91IGNhbiBhZGQgdGhlIHF1aXJrIHRv IGFjcGlfcmV2X2RtaV90YWJsZVtdIHNvIGl0IGdvZXMgdG8gdGhhdAo+ID4gPiBwYXRoIGJ5IGRl ZmF1bHQuIFRoZXJlIGFyZSBhIGJ1bmNoIG9mIHNpbWlsYXIgZW50cmllcyBmb3IgRGVsbCBtYWNo aW5lcy4KPiA+Cj4gPiBPSywgc28gdGhlICJMaW51eCBwYXRoIiB3b3JrcyBhbmQgdGhlIG90aGVy IGRvZXNuJ3QuCj4gPgo+ID4gSSB0aG91Z2h0IHRoYXQgdGhpcyB3YXMgdGhlIG90aGVyIHdheSBh cm91bmQsIHNvcnJ5IGZvciB0aGUgY29uZnVzaW9uLgo+ID4KPiA+ID4gT2YgY291cnNlIHRoaXMg ZG9lcyBub3QgaGVscCB0aGUgbm9uLURlbGwgdXNlcnMgc28gd2Ugd291bGQgc3RpbGwgbmVlZAo+ ID4gPiB0byBmaWd1cmUgb3V0IHRoZSByb290IGNhdXNlLgo+ID4KPiA+IFJpZ2h0Lgo+ID4KPiA+ IFdoYXRldmVyIGl0IGlzLCB0aG91Z2gsIEFNTCBhcHBlYXJzIHRvIGJlIGludm9sdmVkIGluIGl0 IGFuZCBBRkFJQ1MKPiA+IHRoZXJlJ3Mgbm8gZXZpZGVuY2UgdGhhdCBpdCBhZmZlY3RzIGFueSBy b290IHBvcnRzIHRoYXQgYXJlIG5vdAo+ID4gcG9wdWxhdGVkIHdpdGggTlZpZGlhIEdQVXMuCj4g Pgo+Cj4gbGFzdCB3ZWVrIG9yIHNvIEkgZm91bmQgc3lzdGVtcyB3aGVyZSB0aGUgR1BVIHdhcyB1 bmRlciB0aGUgIlBDSQo+IEV4cHJlc3MgUm9vdCBQb3J0IiAobmFtZSBmcm9tIGxzcGNpKSBhbmQg b24gdGhvc2Ugc3lzdGVtcyBhbGwgb2YgdGhhdAo+IHNlZW1zIHRvIHdvcmsuIFNvIEkgYW0gd29u ZGVyaW5nIGlmIGl0J3MgaW5kZWVkIGp1c3QgdGhlIDB4MTkwMSBvbmUsCj4gd2hpY2ggYWxzbyBl eHBsYWlucyBNaWthcyBjYXNlIHRoYXQgVGh1bmRlcmJvbHQgc3R1ZmYgd29ya3MgYXMgZGV2aWNl cwo+IG5ldmVyIGdldCBwb3B1bGF0ZWQgdW5kZXIgdGhpcyBwYXJ0aWN1bGFyIGJyaWRnZSBjb250 cm9sbGVyLCBidXQgdW5kZXIKPiB0aG9zZSAiUm9vdCBQb3J0InMKCkl0IGFsd2F5cyBpcyBhIFBD SWUgcG9ydCwgYnV0IGl0cyBsb2NhdGlvbiB3aXRoaW4gdGhlIFNvQyBtYXkgbWF0dGVyLgoKQWxz byBzb21lIGN1c3RvbSBBTUwtYmFzZWQgcG93ZXIgbWFuYWdlbWVudCBpcyBpbnZvbHZlZCBhbmQg dGhhdCBtYXkKYmUgbWFraW5nIHNwZWNpZmljIGFzc3VtcHRpb25zIG9uIHRoZSBjb25maWd1cmF0 aW9uIG9mIHRoZSBTb0MgYW5kIHRoZQpHUFUgYXQgdGhlIHRpbWUgb2YgaXRzIGludm9jYXRpb24g d2hpY2ggdW5mb3J0dW5hdGVseSBhcmUgbm90IGtub3duIHRvCnVzLgoKSG93ZXZlciwgaXQgbG9v a3MgbGlrZSB0aGUgQU1MIGludm9rZWQgdG8gcG93ZXIgZG93biB0aGUgR1BVIGZyb20KYWNwaV9w Y2lfc2V0X3Bvd2VyX3N0YXRlKCkgZ2V0cyBjb25mdXNlZCBpZiBpdCBpcyBub3QgaW4gUENJIEQw IGF0CnRoYXQgcG9pbnQsIHNvIGl0IGxvb2tzIGxpa2UgdGhhdCBBTUwgdHJpZXMgdG8gYWNjZXNz IGRldmljZSBtZW1vcnkgb24KdGhlIEdQVSAoYmV5b25kIHRoZSBQQ0kgY29uZmlnIHNwYWNlKSBv ciBzaW1pbGFyIHdoaWNoIGlzIG5vdAphY2Nlc3NpYmxlIGluIFBDSSBwb3dlciBzdGF0ZXMgYmVs b3cgRDAuCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmRy aS1kZXZlbCBtYWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRw czovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1kZXZlbA==