From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from new2-smtp.messagingengine.com (new2-smtp.messagingengine.com [66.111.4.224]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F8D5173 for ; Sat, 22 Jan 2022 02:16:28 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.nyi.internal (Postfix) with ESMTP id 552795801FA; Fri, 21 Jan 2022 21:16:27 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Fri, 21 Jan 2022 21:16:27 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=turner.link; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=2Ks4YRmEPcljHJ erAZWS515icKiNKayB8uN7anMTDAI=; b=VXV7Y3cRNuHVmR+RH4wwstQhJk2Bwc B9kFO0ff6gkZhbATxca7ckbcB1HgUaScqPuaOsC+XQEqQ3EBB4MwgAmBEHvs4Van PVKq6k1E8vHKEOGS4vRJsw3vtnP+Ad5RwYam5+PU4uUV1Pp0OqkJGf6Sx7jd4OKy emJ4j2NQf3OUxINlxklg0zqj0s9rMgjVd/AfZ22VsxX6iWzodR2IdRHNn5g+CALQ 1UQj29WHG4H97rpa86NsczYXK0pZsPAZu9VjeHjMzdpi1ZnN2LrC5LBlIRglsyyn DNd9+K+rjMFmAnVMKHHeu51xKGDSYbsStUkRmN9lILNAXYCQxp+Hgh0g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=2Ks4YRmEPcljHJerAZWS515icKiNKayB8uN7anMTD AI=; b=SnNXXdHkvOhaFcl+6uNNIwqzdwMIs5+H29f1NBFswg8RZumE44DooKJkb eCyN5r+xDC61wHCjySvRcRgrSydTQ2DW+IQtVbA4PY4LXZUeRw0LhMFbXf49wngf qoaXdYumEJ8aCQDsleutFUjV0QkipnX7rjOnSfGSXnh4Jv+QadDYzOs6xagnp5+Q MfSQ7H52/E+aUq23fvv+xM0jWcm1M05Xi+17m9pIWuBcTuoGy98FBthhpQl7Xjho 4+gApkXpy9V4Zhq2UUY2dfWKIHtEhtsS3LrR2vT9sBSgHBaV+igtLlCWC9pwY3T/ BKvFTNAHMF96GeX01CM7ybK2p5AAQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrvddugdegfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpehfhffvufffjgfkgggtgfesthhqredttddtjeenucfhrhhomheplfgrmhgvshcu vfhurhhnvghruceolhhinhhugihkvghrnhgvlhdrfhhoshhssegumhgrrhgtqdhnohhnvg drthhurhhnvghrrdhlihhnkheqnecuggftrfgrthhtvghrnhepvdetfffgieevhedvgfei ieeiffdtvdegtdduieffvdejjeduudejgedvveeljeehnecuvehluhhsthgvrhfuihiivg eptdenucfrrghrrghmpehmrghilhhfrhhomheplhhinhhugihkvghrnhgvlhdrfhhoshhs segumhgrrhgtqdhnohhnvgdrthhurhhnvghrrdhlihhnkh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 21 Jan 2022 21:16:26 -0500 (EST) References: <87ee57c8fu.fsf@turner.link> <87a6ftk9qy.fsf@dmarc-none.turner.link> <87zgnp96a4.fsf@turner.link> From: James Turner To: Alex Deucher Cc: Thorsten Leemhuis , Alex Deucher , Lijo Lazar , regressions@lists.linux.dev, kvm@vger.kernel.org, Greg KH , "Pan, Xinhui" , LKML , "amd-gfx@lists.freedesktop.org" , Alex Williamson , Christian =?utf-8?Q?K=C3=B6nig?= Subject: Re: [REGRESSION] Too-low frequency limit for AMD GPU PCI-passed-through to Windows VM Date: Fri, 21 Jan 2022 19:51:11 -0500 In-reply-to: Message-ID: <87czkk1pmt.fsf@dmarc-none.turner.link> Precedence: bulk X-Mailing-List: regressions@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > Are you ever loading the amdgpu driver in your tests? Yes, although I'm binding the `vfio-pci` driver to the AMD GPU's PCI devices via the kernel command line. (See my initial email.) My understanding is that `vfio-pci` is supposed to keep other drivers, such as `amdgpu`, from interacting with the GPU, although that's clearly not what's happening. I've been testing with `amdgpu` included in the `MODULES` list in `/etc/mkinitcpio.conf` (which Arch Linux uses to generate the initramfs). However, I ran some more tests today (results below), this time without `i915` or `amdgpu` in the `MODULES` list. The `amdgpu` kernel module still gets loaded. (I think udev loads it automatically?) Your comment gave me the idea to blacklist the `amdgpu` kernel module. That does serve as a workaround on my machine =E2=80=93 it fixes the behavi= or for f9b7f3703ff9 ("drm/amdgpu/acpi: make ATPX/ATCS structures global (v2)") and for the current Arch Linux prebuilt kernel (5.16.2-arch1-1). That's an acceptable workaround for my machine only because the separate GPU used by the host is an Intel integrated GPU. That workaround wouldn't work well for someone with two AMD GPUs. # New test results The following tests are set up the same way as in my initial email, with the following exceptions: - I've updated libvirt to 1:8.0.0-1. - I've removed `i915` and `amdgpu` from the `MODULES` list in `/etc/mkinitcpio.conf`. For all three of these tests, `lspci` said the following: % lspci -nnk -d 1002:6981 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD= /ATI] Lexa XT [Radeon PRO WX 3200] [1002:6981] Subsystem: Dell Device [1028:0926] Kernel driver in use: vfio-pci Kernel modules: amdgpu % lspci -nnk -d 1002:aae0 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin = HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0] Subsystem: Dell Device [1028:0926] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel ## Version f1688bd69ec4 ("drm/amd/amdgpu:save psp ring wptr to avoid attack= ") This is the commit immediately preceding the one which introduced the issue. % sudo dmesg | grep -i amdgpu [ 15.840160] [drm] amdgpu kernel modesetting enabled. [ 15.840884] amdgpu: CRAT table not found [ 15.840885] amdgpu: Virtual CRAT table created for CPU [ 15.840893] amdgpu: Topology: Add CPU node % lsmod | grep amdgpu amdgpu 7450624 0 gpu_sched 49152 1 amdgpu drm_ttm_helper 16384 1 amdgpu ttm 77824 2 amdgpu,drm_ttm_helper i2c_algo_bit 16384 2 amdgpu,i915 drm_kms_helper 303104 2 amdgpu,i915 drm 581632 11 gpu_sched,drm_kms_helper,amdgpu,drm_ttm_he= lper,i915,ttm The passed-through GPU worked properly in the VM. ## Version f9b7f3703ff9 ("drm/amdgpu/acpi: make ATPX/ATCS structures global= (v2)") This is the commit which introduced the issue. % sudo dmesg | grep -i amdgpu [ 15.319023] [drm] amdgpu kernel modesetting enabled. [ 15.329468] amdgpu: CRAT table not found [ 15.329470] amdgpu: Virtual CRAT table created for CPU [ 15.329482] amdgpu: Topology: Add CPU node % lsmod | grep amdgpu amdgpu 7450624 0 gpu_sched 49152 1 amdgpu drm_ttm_helper 16384 1 amdgpu ttm 77824 2 amdgpu,drm_ttm_helper i2c_algo_bit 16384 2 amdgpu,i915 drm_kms_helper 303104 2 amdgpu,i915 drm 581632 11 gpu_sched,drm_kms_helper,amdgpu,drm_ttm_he= lper,i915,ttm The passed-through GPU did not run above 501 MHz in the VM. ## Blacklisted `amdgpu`, version f9b7f3703ff9 ("drm/amdgpu/acpi: make ATPX/= ATCS structures global (v2)") For this test, I added `module_blacklist=3Damdgpu` to kernel command line to blacklist the `amdgpu` module. % sudo dmesg | grep -i amdgpu [ 14.591576] Module amdgpu is blacklisted % lsmod | grep amdgpu The passed-through GPU worked properly in the VM. James