Re: [regression, bisected, pci/iommu] Bug 216865 - Black screen when amdgpu started during 6.2-rc1 boot with AMD IOMMU enabled

[Vasant Hegde <vasant.hegde@amd.com>]

On 1/4/2023 12:24 PM, Baolu Lu wrote:
> On 2023/1/4 2:55, Matt Fagnani wrote:
>> I reproduced the problem with 6.2-rc1 in a Fedora 37 installation with early
>> kdump enabled as described
>> athttps://fedoraproject.org/wiki/How_to_use_kdump_to_debug_kernel_crashes 
>> https://github.com/k-hagio/fedora-kexec-tools/blob/master/early-kdump-howto.txt 
>> I panicked the kernel with sysrq+alt+c. The dmesg saved with the kdump showed
>> warnings at drivers/pci/ats.c:251 pci_disable_pri+0x75/0x80 and at
>> drivers/pci/ats.c:419 pci_disable_pasid+0x45/0x50 involving AMD IOMMU and
>> amdgpu functions in the trace. Since those warnings' were
>> if (WARN_ON(!pdev->pri_enabled)) and if (WARN_ON(!pdev->pasid_enabled)),
>> pci_disable_pri and pci_disable_pasid looked like they were called when
>> pdev->pri_enabled and pdev->pasid_enabled were both false.
>> A null pointer dereference occurred right after that which made amdgpu crash.
>>
>> [   13.132368] [drm] amdgpu kernel modesetting enabled.
>> [   13.133766] amdgpu: Topology: Add APU node [0x0:0x0]
>> [   13.137596] Console: switching to colour dummy device 80x25
>> [   13.143717] amdgpu 0000:00:01.0: vgaarb: deactivate vga console
>> [   13.143970] [drm] initializing kernel modesetting (CARRIZO 0x1002:0x9874
>> 0x103C:0x8332 0xCA).
>> [   13.144205] [drm] register mmio base: 0xF0400000
>> [   13.144209] [drm] register mmio size: 262144
>> [   13.144310] [drm] add ip block number 0 <vi_common>
>> [   13.144316] [drm] add ip block number 1 <gmc_v8_0>
>> [   13.144320] [drm] add ip block number 2 <cz_ih>
>> [   13.144324] [drm] add ip block number 3 <gfx_v8_0>
>> [   13.144328] [drm] add ip block number 4 <sdma_v3_0>
>> [   13.144332] [drm] add ip block number 5 <powerplay>
>> [   13.144336] [drm] add ip block number 6 <dm>
>> [   13.144340] [drm] add ip block number 7 <uvd_v6_0>
>> [   13.144343] [drm] add ip block number 8 <vce_v3_0>
>> [   13.144347] [drm] add ip block number 9 <acp_ip>
>> [   13.144388] amdgpu 0000:00:01.0: amdgpu: Fetched VBIOS from VFCT
>> [   13.144397] amdgpu: ATOM BIOS: 113-C75100-031
>> [   13.144425] [drm] UVD is enabled in physical mode
>> [   13.144431] [drm] VCE enabled in physical mode
>> [   13.144435] amdgpu 0000:00:01.0: amdgpu: Trusted Memory Zone (TMZ) feature
>> not supported
>> [   13.144491] [drm] vm size is 64 GB, 2 levels, block size is 10-bit,
>> fragment size is 9-bit
>> [   13.144503] amdgpu 0000:00:01.0: amdgpu: VRAM: 512M 0x000000F400000000 -
>> 0x000000F41FFFFFFF (512M used)
>> [   13.144511] amdgpu 0000:00:01.0: amdgpu: GART: 1024M 0x000000FF00000000 -
>> 0x000000FF3FFFFFFF
>> [   13.144524] [drm] Detected VRAM RAM=512M, BAR=512M
>> [   13.144529] [drm] RAM width 64bits UNKNOWN
>> [   13.144623] [drm] amdgpu: 512M of VRAM memory ready
>> [   13.144630] [drm] amdgpu: 3572M of GTT memory ready.
>> [   13.144653] [drm] GART: num cpu pages 262144, num gpu pages 262144
>> [   13.144705] [drm] PCIE GART of 1024M enabled (table at 0x000000F400600000).
>> [   13.158820] amdgpu: hwmgr_sw_init smu backed is smu8_smu
>> [   13.175036] [drm] Found UVD firmware Version: 1.91 Family ID: 11
>> [   13.175097] [drm] UVD ENC is disabled
>> [   13.186675] [drm] Found VCE firmware Version: 52.4 Binary ID: 3
>> [   13.187879] amdgpu: smu version 27.18.00
>> [   13.193760] [drm] DM_PPLIB: values for Engine clock
>> [   13.193773] [drm] DM_PPLIB:     300000
>> [   13.193776] [drm] DM_PPLIB:     480000
>> [   13.193779] [drm] DM_PPLIB:     533340
>> [   13.193781] [drm] DM_PPLIB:     576000
>> [   13.193784] [drm] DM_PPLIB:     626090
>> [   13.193786] [drm] DM_PPLIB:     685720
>> [   13.193788] [drm] DM_PPLIB:     720000
>> [   13.193791] [drm] DM_PPLIB:     757900
>> [   13.193793] [drm] DM_PPLIB: Validation clocks:
>> [   13.193796] [drm] DM_PPLIB:    engine_max_clock: 75790
>> [   13.193799] [drm] DM_PPLIB:    memory_max_clock: 93300
>> [   13.193802] [drm] DM_PPLIB:    level           : 8
>> [   13.193806] [drm] DM_PPLIB: values for Display clock
>> [   13.193809] [drm] DM_PPLIB:     300000
>> [   13.193811] [drm] DM_PPLIB:     400000
>> [   13.193814] [drm] DM_PPLIB:     496560
>> [   13.193816] [drm] DM_PPLIB:     626090
>> [   13.193819] [drm] DM_PPLIB:     685720
>> [   13.193821] [drm] DM_PPLIB:     757900
>> [   13.193823] [drm] DM_PPLIB:     800000
>> [   13.193825] [drm] DM_PPLIB:     847060
>> [   13.193828] [drm] DM_PPLIB: Validation clocks:
>> [   13.193830] [drm] DM_PPLIB:    engine_max_clock: 75790
>> [   13.193833] [drm] DM_PPLIB:    memory_max_clock: 93300
>> [   13.193836] [drm] DM_PPLIB:    level           : 8
>> [   13.193839] [drm] DM_PPLIB: values for Memory clock
>> [   13.193842] [drm] DM_PPLIB:     667000
>> [   13.193844] [drm] DM_PPLIB:     933000
>> [   13.193847] [drm] DM_PPLIB: Validation clocks:
>> [   13.193849] [drm] DM_PPLIB:    engine_max_clock: 75790
>> [   13.193852] [drm] DM_PPLIB:    memory_max_clock: 93300
>> [   13.193854] [drm] DM_PPLIB:    level           : 8
>> [   13.193973] [drm] Display Core initialized with v3.2.215!
>> [   13.309967] [drm] UVD initialized successfully.
>> [   13.511031] [drm] VCE initialized successfully.
>> [   13.515217] kfd kfd: amdgpu: Allocated 3969056 bytes on gart
>> [   13.515442] amdgpu: sdma_bitmap: f
>> [   13.515549] ------------[ cut here ]------------
>> [   13.515555] WARNING: CPU: 0 PID: 477 at drivers/pci/ats.c:251
>> pci_disable_pri+0x75/0x80
>> [   13.515571] Modules linked in: amdgpu(+) drm_ttm_helper ttm iommu_v2
>> hid_logitech_hidpp crct10dif_pclmul drm_buddy crc32_pclmul gpu_sched
>> crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3
>> drm_display_helper wdat_wdt serio_raw hid_multitouch sp5100_tco
>> hid_logitech_dj r8169 cec video wmi scsi_dh_rdac scsi_dh_emc scsi_dh_alua fuse
>> dm_multipath
>> [   13.515620] CPU: 0 PID: 477 Comm: systemd-udevd Kdump: loaded Not tainted
>> 6.2.0-0.rc1.14.fc38.x86_64 #1
>> [   13.515628] Hardware name: HP HP Laptop 15-bw0xx/8332, BIOS F.52 12/03/2019
>> [   13.515634] RIP: 0010:pci_disable_pri+0x75/0x80
>> [   13.515642] Code: 54 24 06 89 ee 48 89 df 83 e2 fe 66 89 54 24 06 0f b7 d2
>> e8 1d e1 fc ff 80 a3 4b 08 00 00 fd 48 83 c4 08 5b 5d e9 2b 8b 69 00 <0f> 0b
>> eb b6 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90
>> [   13.515651] RSP: 0018:ffffbaf4407ab8e8 EFLAGS: 00010046
>> [   13.515658] RAX: 0000000000000000 RBX: ffff90aa00ac4000 RCX: 0000000000000009
>> [   13.515663] RDX: 0000000000000000 RSI: 0000000000000014 RDI: ffff90aa00ac4000
>> [   13.515668] RBP: ffff90aa0e0c3810 R08: 0000000000000002 R09: 0000000000000000
>> [   13.515673] R10: 0000000000000000 R11: ffffffffade4e430 R12: ffff90aa011a8800
>> [   13.515678] R13: ffff90aa0e0c3800 R14: ffff90aa011a8800 R15: ffff90aa0e0c3960
>> [   13.515683] FS:  00007fabd67feb40(0000) GS:ffff90aaf7400000(0000)
>> knlGS:0000000000000000
>> [   13.515689] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [   13.515695] CR2: 00007f5689ff54c0 CR3: 0000000100f16000 CR4: 00000000001506f0
>> [   13.515700] Call Trace:
>> [   13.515704]  <TASK>
>> [   13.515710]  amd_iommu_attach_device+0x2e0/0x300
>> [   13.515719]  __iommu_attach_device+0x1b/0x90
>> [   13.515727]  iommu_attach_group+0x65/0xa0
>> [   13.515735]  amd_iommu_init_device+0x16b/0x250 [iommu_v2]
>> [   13.515747]  kfd_iommu_resume+0x4c/0x1a0 [amdgpu]
>> [   13.517094]  kgd2kfd_resume_iommu+0x12/0x30 [amdgpu]
>> [   13.518419]  kgd2kfd_device_init.cold+0x346/0x49a [amdgpu]
>> [   13.519699]  amdgpu_amdkfd_device_init+0x142/0x1d0 [amdgpu]
>> [   13.520877]  amdgpu_device_init.cold+0x19f5/0x1e21 [amdgpu]
>> [   13.522118]  ? _raw_spin_lock_irqsave+0x23/0x50
>> [   13.522126]  amdgpu_driver_load_kms+0x15/0x110 [amdgpu]
>> [   13.523386]  amdgpu_pci_probe+0x161/0x370 [amdgpu]
>> [   13.524516]  local_pci_probe+0x41/0x80
>> [   13.524525]  pci_device_probe+0xb3/0x220
>> [   13.524533]  really_probe+0xde/0x380
>> [   13.524540]  ? pm_runtime_barrier+0x50/0x90
>> [   13.524546]  __driver_probe_device+0x78/0x170
>> [   13.524555]  driver_probe_device+0x1f/0x90
>> [   13.524560]  __driver_attach+0xce/0x1c0
>> [   13.524565]  ? __pfx___driver_attach+0x10/0x10
>> [   13.524570]  bus_for_each_dev+0x73/0xa0
>> [   13.524575]  bus_add_driver+0x1ae/0x200
>> [   13.524580]  driver_register+0x89/0xe0
>> [   13.524586]  ? __pfx_init_module+0x10/0x10 [amdgpu]
>> [   13.525819]  do_one_initcall+0x59/0x230
>> [   13.525828]  do_init_module+0x4a/0x200
>> [   13.525834]  __do_sys_init_module+0x157/0x180
>> [   13.525839]  do_syscall_64+0x5b/0x80
>> [   13.525845]  ? handle_mm_fault+0xff/0x2f0
>> [   13.525850]  ? do_user_addr_fault+0x1ef/0x690
>> [   13.525856]  ? exc_page_fault+0x70/0x170
>> [   13.525860]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>> [   13.525867] RIP: 0033:0x7fabd66cde4e
>> [   13.525872] Code: 48 8b 0d e5 5f 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e
>> 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 af 00 00 00 0f 05 <48> 3d
>> 01 f0 ff ff 73 01 c3 48 8b 0d b2 5f 0c 00 f7 d8 64 89 01 48
>> [   13.525878] RSP: 002b:00007ffdd89bc6a8 EFLAGS: 00000246 ORIG_RAX:
>> 00000000000000af
>> [   13.525884] RAX: ffffffffffffffda RBX: 0000563e4d23f0a0 RCX: 00007fabd66cde4e
>> [   13.525887] RDX: 00007fabd6817453 RSI: 000000000174fb66 RDI: 00007fabd3bd4010
>> [   13.525890] RBP: 00007fabd6817453 R08: 0000563e4d237c70 R09: 00007fabd672f900
>> [   13.525893] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000020000
>> [   13.525896] R13: 0000563e4d239060 R14: 0000000000000000 R15: 0000563e4d23e450
>> [   13.525900]  </TASK>
>> [   13.525902] ---[ end trace 0000000000000000 ]---
>> [   13.525964] ------------[ cut here ]------------
> 
> This (including the following) kernel traces are triggered by the
> following code.
> 
> 1698 static int pdev_pri_ats_enable(struct pci_dev *pdev)
> 1699 {
> 1700         int ret;
> 1701
> 1702         /* Only allow access to user-accessible pages */
> 1703         ret = pci_enable_pasid(pdev, 0);
> 1704         if (ret)
> 1705                 goto out_err;
> 
> [--cut for short--]
> 
> 1724 out_err:
> 1725         pci_disable_pri(pdev);
> 1726         pci_disable_pasid(pdev);
> 1727
> 1728         return ret;
> 1729 }
> 
> pci_disable_pri() and pci_disable_pasid() are called with PCI PASID and
> PRI not enabled. There are WARN_ON()s in the pci code for such cases.

Yeah. Error path needs to be fixed.

> 
> This happens in the domain attach device path. I haven't figured out why
> the failure of PASID or PRI enabling will cause the domain attach device
> to fail. And also why pci_pasid_features() and pci_pri_supported() are
> not called before pci_enable_pasid/pri().

PASID/PRI support is verified in amd_iommu_device_info().
For AMD GPUs (PASID/PRI supported devices)
  - We allocate new domain called V2 domain and then attach device(s).
    amd_iommu_init_device() - > iommu_attach_group()
    In attach devices path :
      amd_iommu_attach_device() -> attach_device()
      If domain is v2 domain and device is PASID/PRI capable, then we try to
enable PASID/PRI. This is where we are hitting WARN_ON.

I think if attach device fails then we should put the device/group back to
default domain so that we don't hit these warnings.

Matt,

Can you please test below patch? (its not a fix to original issue, but to avoid
kernel warnings/traces).

> 
> commit 201007ef707a ("PCI: Enable PASID only when ACS RR & UF enabled on
> upstream path") requires ACS P2P Request Redirect and Upstream
> Forwarding are enabled for the path leading to the device when enabling
> PASID because PCIe fabric routes Memory Requests based on the TLP
> address, ignoring any PASID. I guess this is the reason why
> pci_enable_pasid() returns failure and discovers above buggy code.

Can we get the lspci -vvv output. It will tell whether ACS request support.


-Vasant

-----
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index cbeaab55c0db..f81ab787eee1 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -1702,27 +1702,26 @@ static int pdev_pri_ats_enable(struct pci_dev *pdev)
 	/* Only allow access to user-accessible pages */
 	ret = pci_enable_pasid(pdev, 0);
 	if (ret)
-		goto out_err;
+		return ret;

 	/* First reset the PRI state of the device */
 	ret = pci_reset_pri(pdev);
 	if (ret)
-		goto out_err;
+		goto out_pasid;

 	/* Enable PRI */
 	/* FIXME: Hardcode number of outstanding requests for now */
 	ret = pci_enable_pri(pdev, 32);
 	if (ret)
-		goto out_err;
+		goto out_pasid;

 	ret = pci_enable_ats(pdev, PAGE_SHIFT);
-	if (ret)
-		goto out_err;
-
-	return 0;
+	if (!ret)
+		return 0;

-out_err:
 	pci_disable_pri(pdev);
+
+out_pasid:
 	pci_disable_pasid(pdev);

 	return ret;
diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c
index 864e4ffb6aa9..4228e44b4950 100644
--- a/drivers/iommu/amd/iommu_v2.c
+++ b/drivers/iommu/amd/iommu_v2.c
@@ -815,6 +815,7 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
 	return 0;

 out_drop_group:
+	iommu_detach_group(dev_state->domain, group);
 	iommu_group_put(group);

 out_free_domain: