From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61CE3C33CA9 for ; Mon, 13 Jan 2020 15:32:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 348E4207FD for ; Mon, 13 Jan 2020 15:32:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ooaz6XlF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728767AbgAMPcp (ORCPT ); Mon, 13 Jan 2020 10:32:45 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:59670 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727222AbgAMPco (ORCPT ); Mon, 13 Jan 2020 10:32:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1578929563; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ElsdEkFyMX9PyqBKUyWZEu82iD4Q0YddLseGJka/eAI=; b=Ooaz6XlFmNrVJKmco00BzFbmOs7B+48bssYuHOkNAAc1PzDKN0hdLlgJvgLnBUc/n1xz/M X3Jd4I/iD9dQvNez1/gNr12YOpMEuhCeozBKDOTQUgMZO0e5vbYWqxsSrIh6mdfWt1wNtM KKArvcBTW+5ngXyzgAwsDm+UXV30GfM= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-166-PlcLqXMtPVCMZaGERpaRRw-1; Mon, 13 Jan 2020 10:32:40 -0500 X-MC-Unique: PlcLqXMtPVCMZaGERpaRRw-1 Received: by mail-qt1-f197.google.com with SMTP id m30so6768583qtb.2 for ; Mon, 13 Jan 2020 07:32:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ElsdEkFyMX9PyqBKUyWZEu82iD4Q0YddLseGJka/eAI=; b=IYm+OyL17B/+2r7oH8DBoeXhGSql/Nmr/33PZYjsT1i1UpuJsxf6ZjWfJy/zRg8wzm 21TUjpkOtAZokPIm1TDjN9pycXlac7JV8p9LTwNcKMO6OZfpAY4Ac0M4bTKxDuxloIT+ CGntPB8uFibJG/0VGapiZZW4KNfUq2JacQbDjFinKZl8pcXhF5jAbFFKcQ6N29+pJkRD jdclfQo2ILeUH0YrOJRX/N/l01bsdUlzW9X/8l6pygKIH9Serx1/Tj+Rn678yowmsT2S C5219WDGLfRGyVqeF0wMOtq/25wYKs8rkszZI+PUeMbPdRtU4lfn9iHl4M2uZfiEg6q2 6ihg== X-Gm-Message-State: APjAAAUyAkayUnzdhGDJtsnXGgKNFf/HQxHYi8SNjg+5Au7hWQXTjtn/ aiAIQllLvY2KJZNbhlkd0lbiie1yUPyfG+JjhEEFdZACmsauo+UpbAoaYf4/0VOXVCweYsGsBZj Y4s9qjcZV/pChLzhrPcBwx64DZ4vppYuHeUMDfXOG X-Received: by 2002:a37:9245:: with SMTP id u66mr12325311qkd.102.1578929560037; Mon, 13 Jan 2020 07:32:40 -0800 (PST) X-Google-Smtp-Source: APXvYqytL56xMC0vF1glxTM38WsY2Ix40Z1naBdqh2T+YigaQEOs8RJRhNMIOMubvldRvknv5G4xQqo3cppMYY9Rex0= X-Received: by 2002:a37:9245:: with SMTP id u66mr12325263qkd.102.1578929559700; Mon, 13 Jan 2020 07:32:39 -0800 (PST) MIME-Version: 1.0 References: <20191121112821.GU11621@lahna.fi.intel.com> <20191121114610.GW11621@lahna.fi.intel.com> <20191127114856.GZ11621@lahna.fi.intel.com> In-Reply-To: From: Karol Herbst Date: Mon, 13 Jan 2020 16:31:50 +0100 Message-ID: Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges To: Dave Airlie Cc: "Rafael J. Wysocki" , Lyude Paul , Mika Westerberg , Bjorn Helgaas , LKML , "Rafael J . Wysocki" , Linux PCI , Linux PM , dri-devel , nouveau , Mario Limonciello Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org okay.. so checking whatever is the difference with _REV being 5 (meaning the firmware uses the legacy paths) doesn't help in any way. It's using a different method to turn the link of and the other ACPI variables touched either point to undocumented registers on the PCI bridge or internal ACPI memory... so, anybody with any other ideas? I really wished the nvidia driver would enable runpm on pre turing GPUs, but that's sadly not the case and on Turing things seem to be totally different, so it wouldn't help to check there as well... *sigh* On Tue, Dec 10, 2019 at 9:49 PM Karol Herbst wrote: > > On Tue, Dec 10, 2019 at 8:58 PM Dave Airlie wrote: > > > > On Mon, 9 Dec 2019 at 21:39, Rafael J. Wysocki wrote: > > > > > > On Mon, Dec 9, 2019 at 12:17 PM Karol Herbst wrote: > > > > > > > > anybody any other ideas? > > > > > > Not yet, but I'm trying to collect some more information. > > > > > > > It seems that both patches don't really fix > > > > the issue and I have no idea left on my side to try out. The only > > > > thing left I could do to further investigate would be to reverse > > > > engineer the Nvidia driver as they support runpm on Turing+ GPUs now, > > > > but I've heard users having similar issues to the one Lyude told us > > > > about... and I couldn't verify that the patches help there either in a > > > > reliable way. > > > > > > It looks like the newer (8+) versions of Windows expect the GPU driver > > > to prepare the GPU for power removal in some specific way and the > > > latter fails if the GPU has not been prepared as expected. > > > > > > Because testing indicates that the Windows 7 path in the platform > > > firmware works, it may be worth trying to do what it does to the PCIe > > > link before invoking the _OFF method for the power resource > > > controlling the GPU power. > > > > > > > Remember the pre Win8 path required calling a DSM method to actually > > power the card down, I think by the time we reach these methods in > > those cases the card is already gone. > > > > Dave. > > > > The point was that the firmware seems to do more in the legacy paths > and maybe we just have to do those things inside the driver instead > when using the new method. Also the _DSM call just wraps around the > interfaces on newer firmware anyway. The OS check is usually what > makes the difference. I might be wrong about the _DSM call just > wrapping though, but I think I saw it at least in some firmware at > some point. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Karol Herbst Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Date: Mon, 13 Jan 2020 16:31:50 +0100 Message-ID: References: <20191121112821.GU11621@lahna.fi.intel.com> <20191121114610.GW11621@lahna.fi.intel.com> <20191127114856.GZ11621@lahna.fi.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: nouveau-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org Sender: "Nouveau" To: Dave Airlie Cc: "Rafael J. Wysocki" , Linux PCI , Mika Westerberg , Linux PM , "Rafael J . Wysocki" , LKML , dri-devel , Mario Limonciello , Bjorn Helgaas , nouveau List-Id: nouveau.vger.kernel.org okay.. so checking whatever is the difference with _REV being 5 (meaning the firmware uses the legacy paths) doesn't help in any way. It's using a different method to turn the link of and the other ACPI variables touched either point to undocumented registers on the PCI bridge or internal ACPI memory... so, anybody with any other ideas? I really wished the nvidia driver would enable runpm on pre turing GPUs, but that's sadly not the case and on Turing things seem to be totally different, so it wouldn't help to check there as well... *sigh* On Tue, Dec 10, 2019 at 9:49 PM Karol Herbst wrote: > > On Tue, Dec 10, 2019 at 8:58 PM Dave Airlie wrote: > > > > On Mon, 9 Dec 2019 at 21:39, Rafael J. Wysocki wrote: > > > > > > On Mon, Dec 9, 2019 at 12:17 PM Karol Herbst wrote: > > > > > > > > anybody any other ideas? > > > > > > Not yet, but I'm trying to collect some more information. > > > > > > > It seems that both patches don't really fix > > > > the issue and I have no idea left on my side to try out. The only > > > > thing left I could do to further investigate would be to reverse > > > > engineer the Nvidia driver as they support runpm on Turing+ GPUs now, > > > > but I've heard users having similar issues to the one Lyude told us > > > > about... and I couldn't verify that the patches help there either in a > > > > reliable way. > > > > > > It looks like the newer (8+) versions of Windows expect the GPU driver > > > to prepare the GPU for power removal in some specific way and the > > > latter fails if the GPU has not been prepared as expected. > > > > > > Because testing indicates that the Windows 7 path in the platform > > > firmware works, it may be worth trying to do what it does to the PCIe > > > link before invoking the _OFF method for the power resource > > > controlling the GPU power. > > > > > > > Remember the pre Win8 path required calling a DSM method to actually > > power the card down, I think by the time we reach these methods in > > those cases the card is already gone. > > > > Dave. > > > > The point was that the firmware seems to do more in the legacy paths > and maybe we just have to do those things inside the driver instead > when using the new method. Also the _DSM call just wraps around the > interfaces on newer firmware anyway. The OS check is usually what > makes the difference. I might be wrong about the _DSM call just > wrapping though, but I think I saw it at least in some firmware at > some point. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 431CCC33CA9 for ; Mon, 13 Jan 2020 15:32:49 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 16F49207FD for ; Mon, 13 Jan 2020 15:32:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IQI0GF8g" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 16F49207FD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 67C9289330; Mon, 13 Jan 2020 15:32:46 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id A494489131 for ; Mon, 13 Jan 2020 15:32:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1578929563; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/ugGRc3ZSBpA03LgnAY27K4MxINi4CuEFoptsSyvb9I=; b=IQI0GF8gLlN2KalGmdI54eAVK/wV9BfsrkO0cTQ4lI+2WMox4vZAo7f3jSkWBWw4JwkcMB CJMjJzG+CWRAXv693nXk4E/NmXuHeXeJFEDKZr8mYQ+yY09XBspvx7OH9J4mb2t6PxyKA8 8Ml9TJUPSSyLveeMVaYjcpydPBWVNVc= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-182-n5yRsVh1MCK6Boo5pY5epQ-1; Mon, 13 Jan 2020 10:32:40 -0500 Received: by mail-qt1-f197.google.com with SMTP id d9so6741773qtq.13 for ; Mon, 13 Jan 2020 07:32:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ElsdEkFyMX9PyqBKUyWZEu82iD4Q0YddLseGJka/eAI=; b=BmvrD1mgvr4vuu14kSyOkT/uhij857D2Ukh9P9+3di4yo8B0eiMqIcWI4ifAL+580s UEKP7Z9ScdH/e8omgn4oXPorpeo3WuennS8MKF5gcBenYP/38W7nOl63dfG0ug/zr2vP VmGIDiFqK6Ttts0E+nfDtjazBJqx2OG804SA9MtRnl8nd8q/xAaJi3Rm2TJdfkl6BIFk HBckuV1NIGL+KUiTnqZfZeJnEphL33hg7i6SLgiDCO4d8PoGhhrLC1geUxYZPn+l119N opH9pzL09bEsyItkn1vuhQxzLNOwfI7WOPKvTUoFQVGfn1c60uiASrFJGNnCQlyABSZz blAA== X-Gm-Message-State: APjAAAVFPJcvGHihDg12DfGolNM4AbmvPeA5OqFEub9S6Kh3GtMl5nus fO+P8fPYCTzUYIIJpXARjyXFIoWc0zC6Ttl3Hd7ETS3ZGJ5CTMgwdGU9VfCr10x+nkpb1N6CRws 0gpkRJQlMMjjjTOOBlAw13nQxr0cvbOaqXuUuqIHSvSTS X-Received: by 2002:a37:9245:: with SMTP id u66mr12325310qkd.102.1578929560032; Mon, 13 Jan 2020 07:32:40 -0800 (PST) X-Google-Smtp-Source: APXvYqytL56xMC0vF1glxTM38WsY2Ix40Z1naBdqh2T+YigaQEOs8RJRhNMIOMubvldRvknv5G4xQqo3cppMYY9Rex0= X-Received: by 2002:a37:9245:: with SMTP id u66mr12325263qkd.102.1578929559700; Mon, 13 Jan 2020 07:32:39 -0800 (PST) MIME-Version: 1.0 References: <20191121112821.GU11621@lahna.fi.intel.com> <20191121114610.GW11621@lahna.fi.intel.com> <20191127114856.GZ11621@lahna.fi.intel.com> In-Reply-To: From: Karol Herbst Date: Mon, 13 Jan 2020 16:31:50 +0100 Message-ID: Subject: Re: [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges To: Dave Airlie X-MC-Unique: n5yRsVh1MCK6Boo5pY5epQ-1 X-Mimecast-Spam-Score: 0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Rafael J. Wysocki" , Linux PCI , Mika Westerberg , Linux PM , "Rafael J . Wysocki" , LKML , dri-devel , Mario Limonciello , Bjorn Helgaas , nouveau Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" okay.. so checking whatever is the difference with _REV being 5 (meaning the firmware uses the legacy paths) doesn't help in any way. It's using a different method to turn the link of and the other ACPI variables touched either point to undocumented registers on the PCI bridge or internal ACPI memory... so, anybody with any other ideas? I really wished the nvidia driver would enable runpm on pre turing GPUs, but that's sadly not the case and on Turing things seem to be totally different, so it wouldn't help to check there as well... *sigh* On Tue, Dec 10, 2019 at 9:49 PM Karol Herbst wrote: > > On Tue, Dec 10, 2019 at 8:58 PM Dave Airlie wrote: > > > > On Mon, 9 Dec 2019 at 21:39, Rafael J. Wysocki wrote: > > > > > > On Mon, Dec 9, 2019 at 12:17 PM Karol Herbst wrote: > > > > > > > > anybody any other ideas? > > > > > > Not yet, but I'm trying to collect some more information. > > > > > > > It seems that both patches don't really fix > > > > the issue and I have no idea left on my side to try out. The only > > > > thing left I could do to further investigate would be to reverse > > > > engineer the Nvidia driver as they support runpm on Turing+ GPUs now, > > > > but I've heard users having similar issues to the one Lyude told us > > > > about... and I couldn't verify that the patches help there either in a > > > > reliable way. > > > > > > It looks like the newer (8+) versions of Windows expect the GPU driver > > > to prepare the GPU for power removal in some specific way and the > > > latter fails if the GPU has not been prepared as expected. > > > > > > Because testing indicates that the Windows 7 path in the platform > > > firmware works, it may be worth trying to do what it does to the PCIe > > > link before invoking the _OFF method for the power resource > > > controlling the GPU power. > > > > > > > Remember the pre Win8 path required calling a DSM method to actually > > power the card down, I think by the time we reach these methods in > > those cases the card is already gone. > > > > Dave. > > > > The point was that the firmware seems to do more in the legacy paths > and maybe we just have to do those things inside the driver instead > when using the new method. Also the _DSM call just wraps around the > interfaces on newer firmware anyway. The OS check is usually what > makes the difference. I might be wrong about the _DSM call just > wrapping though, but I think I saw it at least in some firmware at > some point. _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel