From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FAKE_REPLY_C,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B853C43215 for ; Tue, 19 Nov 2019 22:57:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 43E792245C for ; Tue, 19 Nov 2019 22:57:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574204246; bh=MfxbIsx2AwLtCUhkamnT0UAwFBGbj1YyWpFNWLIPhMM=; h=Date:From:To:Cc:Subject:In-Reply-To:List-ID:From; b=PyTLYsYukrbwwrOmYrAGapi0q1ByKDPD9n6uJYkEsua1SA2rGpucXee25+c5MK7eF c2Inu9ROTsZFercquTnt+PfqfLb9teNGniT9z6w49PAIWTXa61wE1QgmSOvx9Rw1QR lZqozf9V/Owz94Nu8ZqpxN/FxTqlsxR+4lT9owc4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727378AbfKSW5Z (ORCPT ); Tue, 19 Nov 2019 17:57:25 -0500 Received: from mail.kernel.org ([198.145.29.99]:50628 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726025AbfKSW5Y (ORCPT ); Tue, 19 Nov 2019 17:57:24 -0500 Received: from localhost (unknown [69.71.4.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 41B882245B; Tue, 19 Nov 2019 22:57:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574204243; bh=MfxbIsx2AwLtCUhkamnT0UAwFBGbj1YyWpFNWLIPhMM=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=E52+6cgLfMfo9ceaHCYZ+iAp4zf56dFMAe0NcguaKJCgzSyAD9J19ZM6oLMsh0R0R JOkfl/XLao+RYszehEsWWTShlZeOhUuIiZOAk7lvjnwKmyQlFn7Vd+w0/9CGSoA0i4 Cc2z3+YhUvhSgeBDsRLdaEMY+KWsj10hEhoT7Wz0= Date: Tue, 19 Nov 2019 16:57:21 -0600 From: Bjorn Helgaas To: Karol Herbst Cc: LKML , Lyude Paul , "Rafael J . Wysocki" , Mika Westerberg , Linux PCI , Linux PM , dri-devel , nouveau , Dave Airlie , Daniel Drake , Vidya Sagar , Thierry Reding Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Message-ID: <20191119225721.GA237491@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org [+cc Daniel, Vidya, Thierry; thread starts at https://lore.kernel.org/r/20191017121901.13699-1-kherbst@redhat.com] On Tue, Nov 19, 2019 at 11:26:45PM +0100, Karol Herbst wrote: > On Tue, Nov 19, 2019 at 10:50 PM Bjorn Helgaas wrote: > > On Thu, Oct 17, 2019 at 02:19:01PM +0200, Karol Herbst wrote: > > > Fixes state transitions of Nvidia Pascal GPUs from D3cold into > > > higher device states. > > > ... > > > + * - kernel crashes, as all pci reads return -1, which most code > > > + * isn't able to handle well enough. > > > > More details about these crashes would be useful as we look at > > places that *should* be able to handle errors like this. > > makes sense, I could ,orthogonal to this, make the code more robust > if we hit issues like this in the future. What I am mostly wondering > about is, why pci core doesn't give up if the device doesn't come > back from D3cold? It sounds like, that the most sane thing to do > here is to just give up and fail runtime_resume and report errors > back to userspace trying to make use of the devices. It's possible there's something the core could do here. It's interesting that we have at least three issues in this area in this release: 1) This NVIDIA GPU issue 2) Daniel's issue with AMD Ryzen5/7 XHCI power-on (https://lore.kernel.org/r/20191014061355.29072-1-drake@endlessm.com) 3) Vidya's issue with Intel 750 NVMe power-on (https://lore.kernel.org/r/20191118172310.21373-1-vidyas@nvidia.com) Vidya's current patch is the most generic (calling pci_dev_wait() from pci_power_up()). That will print a warning if the device doesn't respond, but we still don't go as far as returning errors to runtime_resume. This is definitely something we should consider improving in the future. > > > + * - sudden shutdowns, as the kernel identified an unrecoverable error after > > > + * userspace tries to access the GPU. > > > > This doesn't fit with the others and more details might be > > informative here as well. > > yeah.. I try to get more infos on that. But at least for me (and it > might be a distribution thing) if I execute lspci, the system shuts > down, or at least tries to and might fail. The lspci doesn't need to be after the failure occurs. You can do it immediately after boot. If you can capture any part of the dmesg or console log of these sudden shutdowns, that's all I'm interested in at this point. Bjorn From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=FAKE_REPLY_C, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEA96C432C0 for ; Tue, 19 Nov 2019 22:57:27 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E5F22245B for ; Tue, 19 Nov 2019 22:57:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E5F22245B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A33D16E9D3; Tue, 19 Nov 2019 22:57:25 +0000 (UTC) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by gabe.freedesktop.org (Postfix) with ESMTPS id 238AC6E91A; Tue, 19 Nov 2019 22:57:24 +0000 (UTC) Received: from localhost (unknown [69.71.4.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 41B882245B; Tue, 19 Nov 2019 22:57:23 +0000 (UTC) Date: Tue, 19 Nov 2019 16:57:21 -0600 From: Bjorn Helgaas To: Karol Herbst Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Message-ID: <20191119225721.GA237491@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574204243; bh=MfxbIsx2AwLtCUhkamnT0UAwFBGbj1YyWpFNWLIPhMM=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=E52+6cgLfMfo9ceaHCYZ+iAp4zf56dFMAe0NcguaKJCgzSyAD9J19ZM6oLMsh0R0R JOkfl/XLao+RYszehEsWWTShlZeOhUuIiZOAk7lvjnwKmyQlFn7Vd+w0/9CGSoA0i4 Cc2z3+YhUvhSgeBDsRLdaEMY+KWsj10hEhoT7Wz0= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thierry Reding , Linux PM , Linux PCI , Mika Westerberg , "Rafael J . Wysocki" , LKML , dri-devel , Daniel Drake , nouveau , Vidya Sagar Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Message-ID: <20191119225721.3aCQljG34KUrh_UvOgXpA7donrTJEGilO0lk4Ayyq0s@z> WytjYyBEYW5pZWwsIFZpZHlhLCBUaGllcnJ5OyB0aHJlYWQgc3RhcnRzIGF0Cmh0dHBzOi8vbG9y ZS5rZXJuZWwub3JnL3IvMjAxOTEwMTcxMjE5MDEuMTM2OTktMS1raGVyYnN0QHJlZGhhdC5jb21d CgpPbiBUdWUsIE5vdiAxOSwgMjAxOSBhdCAxMToyNjo0NVBNICswMTAwLCBLYXJvbCBIZXJic3Qg d3JvdGU6Cj4gT24gVHVlLCBOb3YgMTksIDIwMTkgYXQgMTA6NTAgUE0gQmpvcm4gSGVsZ2FhcyA8 aGVsZ2Fhc0BrZXJuZWwub3JnPiB3cm90ZToKPiA+IE9uIFRodSwgT2N0IDE3LCAyMDE5IGF0IDAy OjE5OjAxUE0gKzAyMDAsIEthcm9sIEhlcmJzdCB3cm90ZToKPiA+ID4gRml4ZXMgc3RhdGUgdHJh bnNpdGlvbnMgb2YgTnZpZGlhIFBhc2NhbCBHUFVzIGZyb20gRDNjb2xkIGludG8KPiA+ID4gaGln aGVyIGRldmljZSBzdGF0ZXMuCj4gPiA+IC4uLgoKPiA+ID4gKyAqICAtIGtlcm5lbCBjcmFzaGVz LCBhcyBhbGwgcGNpIHJlYWRzIHJldHVybiAtMSwgd2hpY2ggbW9zdCBjb2RlCj4gPiA+ICsgKiAg ICBpc24ndCBhYmxlIHRvIGhhbmRsZSB3ZWxsIGVub3VnaC4KPiA+Cj4gPiBNb3JlIGRldGFpbHMg YWJvdXQgdGhlc2UgY3Jhc2hlcyB3b3VsZCBiZSB1c2VmdWwgYXMgd2UgbG9vayBhdAo+ID4gcGxh Y2VzIHRoYXQgKnNob3VsZCogYmUgYWJsZSB0byBoYW5kbGUgZXJyb3JzIGxpa2UgdGhpcy4KPiAK PiBtYWtlcyBzZW5zZSwgSSBjb3VsZCAsb3J0aG9nb25hbCB0byB0aGlzLCBtYWtlIHRoZSBjb2Rl IG1vcmUgcm9idXN0Cj4gaWYgd2UgaGl0IGlzc3VlcyBsaWtlIHRoaXMgaW4gdGhlIGZ1dHVyZS4g V2hhdCBJIGFtIG1vc3RseSB3b25kZXJpbmcKPiBhYm91dCBpcywgd2h5IHBjaSBjb3JlIGRvZXNu J3QgZ2l2ZSB1cCBpZiB0aGUgZGV2aWNlIGRvZXNuJ3QgY29tZQo+IGJhY2sgZnJvbSBEM2NvbGQ/ IEl0IHNvdW5kcyBsaWtlLCB0aGF0IHRoZSBtb3N0IHNhbmUgdGhpbmcgdG8gZG8KPiBoZXJlIGlz IHRvIGp1c3QgZ2l2ZSB1cCBhbmQgZmFpbCBydW50aW1lX3Jlc3VtZSBhbmQgcmVwb3J0IGVycm9y cwo+IGJhY2sgdG8gdXNlcnNwYWNlIHRyeWluZyB0byBtYWtlIHVzZSBvZiB0aGUgZGV2aWNlcy4K Ckl0J3MgcG9zc2libGUgdGhlcmUncyBzb21ldGhpbmcgdGhlIGNvcmUgY291bGQgZG8gaGVyZS4g IEl0J3MKaW50ZXJlc3RpbmcgdGhhdCB3ZSBoYXZlIGF0IGxlYXN0IHRocmVlIGlzc3VlcyBpbiB0 aGlzIGFyZWEgaW4gdGhpcwpyZWxlYXNlOgoKICAxKSBUaGlzIE5WSURJQSBHUFUgaXNzdWUKICAy KSBEYW5pZWwncyBpc3N1ZSB3aXRoIEFNRCBSeXplbjUvNyBYSENJIHBvd2VyLW9uCiAgICAgKGh0 dHBzOi8vbG9yZS5rZXJuZWwub3JnL3IvMjAxOTEwMTQwNjEzNTUuMjkwNzItMS1kcmFrZUBlbmRs ZXNzbS5jb20pCiAgMykgVmlkeWEncyBpc3N1ZSB3aXRoIEludGVsIDc1MCBOVk1lIHBvd2VyLW9u CiAgICAgKGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL3IvMjAxOTExMTgxNzIzMTAuMjEzNzMtMS12 aWR5YXNAbnZpZGlhLmNvbSkKClZpZHlhJ3MgY3VycmVudCBwYXRjaCBpcyB0aGUgbW9zdCBnZW5l cmljIChjYWxsaW5nIHBjaV9kZXZfd2FpdCgpIGZyb20KcGNpX3Bvd2VyX3VwKCkpLiAgVGhhdCB3 aWxsIHByaW50IGEgd2FybmluZyBpZiB0aGUgZGV2aWNlIGRvZXNuJ3QKcmVzcG9uZCwgYnV0IHdl IHN0aWxsIGRvbid0IGdvIGFzIGZhciBhcyByZXR1cm5pbmcgZXJyb3JzIHRvCnJ1bnRpbWVfcmVz dW1lLgoKVGhpcyBpcyBkZWZpbml0ZWx5IHNvbWV0aGluZyB3ZSBzaG91bGQgY29uc2lkZXIgaW1w cm92aW5nIGluIHRoZQpmdXR1cmUuCgo+ID4gPiArICogIC0gc3VkZGVuIHNodXRkb3ducywgYXMg dGhlIGtlcm5lbCBpZGVudGlmaWVkIGFuIHVucmVjb3ZlcmFibGUgZXJyb3IgYWZ0ZXIKPiA+ID4g KyAqICAgIHVzZXJzcGFjZSB0cmllcyB0byBhY2Nlc3MgdGhlIEdQVS4KPiA+Cj4gPiBUaGlzIGRv ZXNuJ3QgZml0IHdpdGggdGhlIG90aGVycyBhbmQgbW9yZSBkZXRhaWxzIG1pZ2h0IGJlCj4gPiBp bmZvcm1hdGl2ZSBoZXJlIGFzIHdlbGwuCj4gCj4geWVhaC4uIEkgdHJ5IHRvIGdldCBtb3JlIGlu Zm9zIG9uIHRoYXQuIEJ1dCBhdCBsZWFzdCBmb3IgbWUgKGFuZCBpdAo+IG1pZ2h0IGJlIGEgZGlz dHJpYnV0aW9uIHRoaW5nKSBpZiBJIGV4ZWN1dGUgbHNwY2ksIHRoZSBzeXN0ZW0gc2h1dHMKPiBk b3duLCBvciBhdCBsZWFzdCB0cmllcyB0byBhbmQgbWlnaHQgZmFpbC4KClRoZSBsc3BjaSBkb2Vz bid0IG5lZWQgdG8gYmUgYWZ0ZXIgdGhlIGZhaWx1cmUgb2NjdXJzLiAgWW91IGNhbiBkbyBpdApp bW1lZGlhdGVseSBhZnRlciBib290LgoKSWYgeW91IGNhbiBjYXB0dXJlIGFueSBwYXJ0IG9mIHRo ZSBkbWVzZyBvciBjb25zb2xlIGxvZyBvZiB0aGVzZQpzdWRkZW4gc2h1dGRvd25zLCB0aGF0J3Mg YWxsIEknbSBpbnRlcmVzdGVkIGluIGF0IHRoaXMgcG9pbnQuCgpCam9ybgpfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0 CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3Rv cC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWw=